Over the past three years, some of the
Almost all had the same pedigree when it came to the development and delivery of the core business applications software that sit at the heart of their businesses. More often than not, such bespoke development or major adjustments to packaged applications were over budget, years in gestation and late, and when they did arrive (assuming they did) the software was either far removed from what the business had originally conceived or had been overtaken by changes in business circumstances.
What they all had in common was a pressing need to fix that. And for companies as diverse as BT, Standard Life and Tarmac the experimental remedy was the Agile methodology of software development.
Despite early acceptance issues – from both IT and business stakeholders – scalability challenges, and what amounts to a change in internal IT culture, the results have been highly positive. No-one is saying that Agile is the basis of these companies’ success in their markets, but plenty are pointing to the delivery of software in rapid cycles, software that meets the business’s needs and lower development cost as key factors.
In the eyes of companies like these, that spells an end to three decades of bad IT practice – and bad business project failure.
What that has taken is vision, commitment and a willingness to learn along the way.
The BT story
When Al-Noor Ramji took over as CIO of BT Group in May 2004, a major part of his remit was to shake up the development of IT projects within the company.
At that point, BT was desperate to break out from its telecoms heritage and move into areas such as corporate networking, broadband, IT services and media, with scores of new products for corporate customers and consumers. And it needed to rapidly develop the IT to underpin that expansion.
One thing was clear to Ramji: the speed of traditional software delivery within the company – and more importantly, how well the end result fitted dynamic business needs – was woefully mismatched to supporting those business goals.
“The style of the [software development] organisation had been carefully, well-thought through engineering; very large projects, enormous scale, long delivery times,” says Roger Leaton, who now takes the title ‘Agile Advocate’ at BT.
“We were chasing a new type of business. And in that new world, responsiveness – an ability to react – is more important than heavyweight engineering geared to deliver efficiency.”
As at businesses everywhere, the old way was a classic, rigorous ‘waterfall’ or plan-driven approach: the assumption was that any project would take a large-scale, upfront commitment from both the business side and IT. The scope would be defined over many meetings between mostly senior executives, documented across many volumes and then ‘thrown over the wall’ to the IT designers and architects, who in turn would pass their work to developers and testers.
In his first six months of fact-finding, Al-Noor Ramji developed a clear perspective on whether that was working. First he broke down existing programme barriers, grouping together all the large IT developments BT had under way under the ‘One IT’ directorship. And then he dropped a bombshell.
“I want you to deliver a working product of value to the business in 90 days… and then every 90 days after that until the project is complete,” was the message.
Ramji did not dictate how that should be done; rather, he deliberately left a gap so that the creative response would come from the organisation.
At the time, the typical project model within BT was based on a deliverable every 18 months – but in many of the larger projects, that time frame was regarded as unfeasibly short. “There was a conceptual leap for some,” says Leaton. “‘90 days? We can’t get through the requirements work in 90 days,’ was the initial reaction.”
Some of those reacted by trying ‘death-march programming’. Cram classic requirements capture and design into 30 days, follow that by 30 days of round the clock coding and then 30 days of testing.
That route burnt out quite a few, says Leaton.
But within BT were a small group of individuals who had been trying (not with much success) to push a number of maverick methodologies for several years – DSDM, Extreme Programming, SCRUM and Agile – many with overlapping threads but none that resembled the existing approach.
“They were all on the fringe at BT, but they were now thrust onto the agenda,” says Leaton, as educators, evangelists, coaches and project leaders.
Within months, it became clear that Agile was the best bet (perhaps the only bet) that was going to allow the development organisation to meet the demands that Ramji was putting on them.
Stories and iterations
The principles that BT has embraced – and encapsulated in its Agile Cookbook – draw heavily on the Agile Manifesto developed by a group of visionary developers in 2001, and now being increasingly widely applied:
- Customer involvement – IT needs to be in near constant collaboration with the ultimate recipients and users of the technology it is developing. At BT, each 90-day period is preceded by a three-day collaborative workshop known as a ‘Hothouse’, during which business users and executives get together with people from all levels of IT to define – and redefine – functionalities and priorities.
- User stories – In order to set out the scope of the project, Agile encourages descriptions of how the resulting system will need to look and feel to the users and what it will do for the business – effectively these stories are slices of the overall scope.
- Iterative development – Rather than apply the waterfall, Agile embodies iterative development. BT, for example, segments its 90-day cycles into two-week iterations, with each phase designed to demonstrate progress to the business users, and fully functional deliverables at the end of each cycle. A key measure of each post-90-day deliverable is the Programme Implementation Review.
- Automate testing and continuous integration – Rather than testing code and integrating the final product after it is complete, software is tested by IT and by users constantly over the project.
- The iterative approach – and the necessary and close IT/business collaboration that flows from that – is perhaps the most controversial shift.
“Agile is different. It assumes a realistic world in which customers don’t know what they ultimately want, and IT teams start to apply technologies that contain some surprises. So there is no longer any point in trying to work everything out in advance,” says Leaton. “You approach the requirement in thin slices.”
In the three years since the first initial projects got under way, Agile has become the underpinning principle in 60% to 70% of BT’s software development assignments, affecting the work of most of its 8,000 internal IT staff to some degree and becoming part of the agenda of thousands of contactors in the UK, Asia and elsewhere.
Agile life
BT is not alone in its enthusiasm for Agile. Three years ago, pension company Standard Life took a long hard look at how its IT and its business organisation worked together.
At the time, the two maintained a healthy suspicion of and separation from each other. From the business side, Standard Life’s customer service director, Garry Morrison, paints an all-too-familiar picture: “IT would only start building an application if the business had thought of and documented every single conceivable requirement and had it signed off in the blood of the sponsoring director.
“We recognised that there was perhaps a better way; that working together might benefit the business as a whole. We then began a journey of developing applications and services in an Agile manner.”
To foster trust and understanding, Standard Life took a bold step towards close collaboration, by embedding the relevant IT team within its ecommerce pensions group.
There is now one ecommerce development team, made up of pension specialists and IT professionals, says Morrison. IT staff are no longer far removed from the business customers that they serve. “By being embedded in the business area, by listening to what [external] customers want, we have been developing applications and services that are relevant to customer needs, and doing it much more quickly than we ever did before – but without losing any of the rigour of the development processes.
“And contrary to popular belief, business and IT people [can] actually talk the same language. They discuss the problems and arrive at possible solutions together. Instead of spending time in bureaucracy, they simply get on and deliver.”
His colleague, chief architect of the group’s IT Ian Muir, also points to the role of the new methodology: “In our experience, it is Agile that gets IT close to the business.”
New challenges
Notwithstanding such progress, companies like Standard Life and BT have not lost sight of the challenges they have faced in embracing Agile methods – and those they are still coming up against. The degree of cultural change should not be underestimated.
“You need to improve the collaboration right across the team, so everyone thinks of themselves as the delivery team. And that is a huge cultural change for many,” says Leaton.
Software developers are not the only source of scepticism. Business people who have been used to specifying every feature of a desire system up front struggle to get away from the notion that they only have a single chance to input all their requirements.
“The business was used to a requirements meeting every 18 months where they would load the document with anything anyone has thought they might need – much of which turns out not to be wanted,” says Leaton.
“Initially, you need to get them to understand that they no longer only get one go at this, that this is only the first of a series of 90-day slices.”
But even then there can be resistance. “Some parts of the organisation have said, ‘we really like this new way of working’, and others are saying, ‘why do we have to keep talking to IT, why don’t they just take the documents that we have thrown at them and do what we ask?’.”
At the same time on the IT side there is a whole set of social skills that the higher degree of collaboration demands – not always the developer fraternity’s natural skill set.
As part of its education, BT runs large-scale simulations of the agile experience, coaching and roadshows through technology education specialists such as Exoftware. “Talking about it and reading books about it is great, but it is the actual experience of doing it that makes all the difference,” says Leaton.
Beyond the walls
Changes to the internal culture are critical. But few organisations today develop all their own systems in house. And, indeed, one of the tenets of the Agile movement – “co-located, self-governing team with customer involvement” – is not an easy fit when the developers and business team might be spread across the
Analysts at Forrester Research have advised Standard Life that Agile can be made to work even in highly distributed or offshoring circumstances with the implementation of collaborative technologies.
Indeed, BT has worked with application development tools company Borland to build a team collaboration environment, TeamFocus, which provides both a development hub for Agile teams and a planning capability that means Agile can be effectively managed across geographically dispersed teams.
As that might suggest, Agile development comes with a bill. “A huge change like this doesn’t immediately give you benefits.” As a result of the steep learning curve, the chance of working practices and uncertainty in the work environment, there is a drop-off in performance initially, reports Leaton.
But when that is weighed against the alternative of developing code that is a mismatch to requirements, late and over budget, Agile seems a sound investment.
Standard Life certainly thinks so. Between 2003 and 2006, the company halved its IT project development budget, and that has been maintained since, even as the business has grown. Ian Muir puts that saving – currently in the vicinity of £20 million – down to two factors: “Over the past few years, perhaps the two most important directions we have established within IT are our service-oriented architecture and, more latterly, the adoption of Agile development techniques.”
In 2007, Standard Life asked analysts at Gartner to benchmark its development activities against the top quartile of its industry peers. On a productivity level its development teams were two and a half to three times ahead of industry rivals.
And Muir does not hesitate to make the wider connection: “A strong focus on Agile techniques has helped Standard Life to be number one in the corporate pension market and help drive growth and efficiency.”
BT does not link Agile to any business numbers. The metric there comes from users. The post-implementation reviews show that internal business partner satisfaction rates have risen to over 80%. Even if the company had measured user satisfaction prior to the arrival of Al-Noor Ramji, it would never have been anywhere near that region – no matter how well engineered the waterfall code.
Further reading:
Embedded value – Standard Life has transformed its relationship to IT by embedding the team within its pensions operation and exploiting agile development and an advanced service-oriented architecture
