How to Transition to Agile Software Development
By Lawrence Mantrone
Would you accept a 60 percent failure rate for your software development projects?
According to the Standish Group’s 2011 CHAOS report, which studied projects undertaken between 2002 and 2010, fewer than 40 percent of software development projects are likely to succeed in delivering desired features on time and under budget.
You are not alone if you find these statistics unacceptable. Organizations are demanding better technology project execution, but not just due to cost overruns, missed schedules and quality issues. Management also wants to harvest the business value expected from new software sooner while using fewer resources, as the Scrum Alliance noted in its report.
Technology managers have tried for decades to improve software delivery by applying project management best practices originally developed for construction or engineering projects. These approaches rely on predictive planning and tight control over requirements and schedule changes. Their mixed results in improving project success have led more organizations to turn to agile methodologies in an effort to do better.
Agile methods treat software development as an innovation process. They accept the resulting uncertainties and reject predictive techniques in which teams attempt to plan, define and build software in preconceived sequential steps. They favor building software iteratively and incrementally, and the results of early phases influence the decisions and planning of later phases.
Agile approaches empower individual team members by promoting close, unstructured collaboration within self-managed teams. This contrasts with top-down management approaches in which senior managers dictate assignments to junior team members.
Conventional software development processes define requirements in detailed documentation, which are then handed off to designers and developers for implementation. Agile teams start with brief feature descriptions and work out details through repeated cycles of discussion, development, testing and review.
The use of agile practices can help teams better manage software development project risks. Incremental development produces functional software earlier and eliminates reliance on often-unreliable predictive estimation techniques. Software testing and review take place throughout the development process rather than being concentrated at its end. Therefore, project sponsors can quickly make informed decisions based on actual progress.
The 2011 CHAOS report suggests the use of agile methods might be paying off. The study showed projects were three times more likely to succeed when agile methodologies were employed. Organizations that have tried agile methodologies tend to keep using them. The Scrum Alliance reported that 47 percent of the respondents to its 2013 survey are using the popular agile framework known as Scrum “frequently” or “always.”
Agile methodologies enable companies to become superior competitors by improving their chances of completing projects within cost and schedule constraints. Companies also find that the use of agile approaches allows them to better meet the requirements of their internal and external customers. Faster software deployment accelerates innovation, efficiency and revenue, the Scrum Alliance found.
Today, companies must be able to innovate continually to survive. This has led many to consider alternatives to predictive planning processes deployed within conventional, pyramid-shaped management structures in favor of agile product development, which can reduce product development life cycles from years to months.
Many organizations find traditional processes and management structures to be too time-consuming, less able to deal with complex problems, and not responsive enough to new product requirements that are dictated by changes in business or regulatory environments.
Agile methodologies are well-suited to address the demand for continual learning and the psychological needs of today’s workforce, especially Millennial employees (those born after 1980). Agile processes emphasize positive work relationships based on mutual respect. They are designed to enable self-managed teams to create products without being tied to unrealistic goals and schedules. Agile teams are expected to be multidisciplinary, and their members are encouraged to master new skills.
Flexibility and Change
While agile methods have proven to yield benefits for many companies, their successful adoption requires significant changes in processes and attitudes.
Gene Gendel, an experienced agile coach, points out that “managers have to give up their commanding/controlling power and rights to dictate best practices to their subordinates,” as agile approaches emphasize self-management. He further notes that “conventional project management heavily relies on centralized control and assumes a clearly defined hierarchical reporting structure,” which may cause managers to question their roles when supporting self-managing teams.
Dave Prior, an agile project management professional at BigVisible Solutions, advises organizations to “walk into an agile transition with eyes open. Patience and management commitment are required. Positive outcomes are possible, but it takes a while and a lot of trial and error.”
Agile approaches stress transparency and openness, which, he observes, may not sit well with senior managers who dislike sharing bad news. He warns that despite their benefits, agile transitions are difficult and painful, as every part of an organization is affected.
Software architects, developers, business analysts and users will also have to learn new ways of doing their work if they are assigned to projects that use agile approaches. Staff accustomed to working in conventional environments based on top-down management philosophies may not always be comfortable working without explicit written requirements or design documentation. They may resist working in a highly collaborative environment if either they or their fellow team members don’t possess the interpersonal skills required for comfortable working relationships.
While the Scrum framework emphasizes collaborative effort, it requires a single individual to be designated to make final product decisions and work closely with the development team. Gendel has found that management may struggle to empower someone to handle this “product ownership” role if no single person can be found who has the necessary time and skills. He says a common but often ineffective fallback is a “product owner committee” approach, but involving more people often results in a lack of accountability and decision making, which can inhibit progress.
William Bridges warns in his book Managing Transitions that “the single biggest reason organizational changes fail is [that] no one thought about endings or planned to manage their impact on people.” He recommends applying transition management best practices carefully.
Clear reasons for trying agile approaches must be defined and articulated to affected employees. For example, an organization might decide to introduce such approaches to build a product when there is a lot of uncertainty about the product’s requirements.
An agile process would allow a team to build functionality and features in stages. Stakeholders would be given regular opportunities to review what has been built and suggest changes.
It is critical to include all key stakeholders in planning for your change initiative. You should understand their opinions about existing processes and get them to buy in to the new approach before moving forward with any process improvement initiative.
This gives those charged with implementing process changes a better sense of issues worthy of attention during the transition. Any affected group needs to be consulted. This would include managers in finance departments, project management offices and enterprise risk management staff.
Kaplan Test Prep's Ilio Krumins-Beens, executive director, Agile Practices, and Mariya Breyter, director of Project Management and Agile Practices, implemented agile processes at the organization. With executive support, they worked with their peers to identify and acknowledge the problems in their development processes. By raising awareness of the problems throughout the organization, they were able to build a desire for change.
They took care not to oversell the potential benefits of using agile methods. Breyter shared a comment she heard at a conference: “Agile is not a silver bullet. It is a silver mirror.”
This observation helped Kaplan Test Prep employees understand that every problem wouldn’t be eliminated overnight, but the transition would help them learn about their strengths, weaknesses and flexibility. Even if it failed, the attempt at transformation would highlight their potential for change.
Choose First Project Carefully
Organizations must carefully consider which projects will be first to use agile processes. Managers may have heard of cases where agile methodologies have revitalized large, mission-critical projects on the verge of disaster. They might advocate using a large troubled project for their company’s first agile effort.
A safer approach would be to start with one or two small, noncritical projects and learn from the results. The experience gained from the pilot projects can be used to adjust processes or conduct remedial training as needed.
Gendel recommends breaking larger projects into components, with each component assigned to a different agile team and—ideally—represented by a single product owner. If the organization decides to create a product owner committee, a single individual should be designated as the committee’s voice to each agile team. This eliminates the possibility of agile teams being provided with conflicting information from different people.
Prior strongly advises getting a good agile coach. He believes coaches can provide critical training and guidance to agile team members and the rest of the organization. He suggests looking for “someone who has been through agile adoptions multiple times with multiple companies.”
An experienced coach has the ability to anticipate emerging situations and can help deal with cultural issues and backlash. Prior warns against hiring coaches who claim a perfect record of success, as they may not have dealt with adversity during a transition project.
Organizations should consider applying agile best practices to how they adopt agile techniques. Agile methods emphasize gauging progress during and after iterations. Decisions are made based on frequent measurements of performance.
For example, the Scrum framework includes regular team self-assessments of how well Scrum is being applied. Agile teams are either encouraged by seeing positive improvements in how products are delivered, or are challenged to find ways to improve unsatisfactory performance during the next iteration. The Kaplan Test Prep team applied these principles by defining a set of goals for its agile transition. These included delivering higher quality, greater value, improved time to market and increased satisfaction within the organization.
The team conducted a baseline survey before starting the agile transition to capture perceptions of how well it was achieving these goals and followed up with quarterly reassessments as the transition proceeded. Breyter reported that within six to eight months after the transition had started, significant improvement was shown in all areas, including faster time to market.
Subsequently, the survey metrics did not show as great an additional rate of incremental change. But, by adjusting the training approach, the team was able to regain the previous pace of improvement. Rather than mandate a predefined training regimen, the team empowered its agile teams to request their own topics, and it promoted a culture of empowerment, networking and collaboration.
Krumins-Beens believes the team’s forthrightness and the use of an incremental rollout strategy were critical to Kaplan Test Prep’s success. The team members admitted up front that agile methods might not be best for the company, but they would at least use the methodology to assess strengths and weaknesses in their software development methods. They realized that when process improvement initiatives attempt to change an entire company overnight, such initiatives often fail.
By embracing a pilot project approach to flesh out and demonstrate the effectiveness of the new agile processes, the team gained management buy-in and grassroots support. As a result, the rest of the Kaplan Test Prep organization adopted agile processes within six months.
Lawrence Mantrone is a management consultant who teaches graduate and continuing education courses in business at the NYU School of Continuing and Professional Studies. He is a project management professional (PMP) and a Certified Scrum Master. You can reach him at firstname.lastname@example.org.