Managing an Agile project is worse than herding cats, because at least cats know what they need. They need food, shelter and affection, but an Agile software project defends its right to keep its requirements sketchy.
No wonder this makes senior management uncomfortable, because they want in-depth plans that state ‘how long’ and ‘how much’. Yet the Agile team refuses to commit themselves because they insist the requirements must be ‘discovered’ as the project progresses. So even though Agile sounds interesting and looks compelling, there is just too much uncertainty.
It is important to understand the useful distinction between project governance and software engineering. To be effective it is necessary to have one of each. The expectation is that the governance regime will deliver for senior management an external view of actual progress.
Agile methods such as Scrum are ways of engineering software that many people have heard of, some understand and few completely trust. But Agile methods and governance, such as Prince2 or PMBOK, are seldom mentioned in the same breath. It is a myth that Agile is without structure, but because the structure is internal to the method, there is very little exposed outwardly for the governance method to measure.
On first inspection the concepts of governance and Agile appear to be incompatible. As long as this is the case, the adoption of Agile will be limited.
But, what if it were possible to combine the two? If there were a different kind of governance structure available, the problem would be mitigated, and Agile methods would become more acceptable to larger organisations.
Project control exists on two levels: internal control within the engineering method, and external control (management control) represented by governance. Governance measures
the effectiveness of engineering progress. Senior management argue that it is not only progress that matters, but also the evidence of progress. The Agile camp defines progress as working code, whereas the governance camp defines it as completed milestones on a project plan. Yet what both can agree upon is that real progress is what matters, and not simply indications that give an illusion of progress.
When the differences are described in terms of the way progress is demonstrated, a way forward may be found. First it is necessary to review what is meant by ‘agility’.
Scrum down
Agile is a word that has a particular connotation in software engineering circles. For instance, in Scrum (a term derived from rugby) business requirements are accepted as being only partially understood. Successful Scrum implementations are characterised by small, motivated teams, being comfortable with changing requirements and little traditional project governance. Therefore an organisation that is familiar with large teams, ‘complete’ specifications and significant governance will often be resistant to Scrum principles.
The process that underpins Scrum is straightforward. The project is divided into units of time known as Sprints. A Sprint is an iteration of time that typically lasts 30 days after which there is a software release that is shown to the stakeholders and evaluated. Based on what they see the stakeholders may modify their requirements, this is known as requirements churn.
The requirements delivered in a Sprint are taken from the list of all known requirements, known as the Product Backlog. Those in the current Sprint are catalogued in a subset of the Product Backlog that is termed the Sprint Backlog.
Because changes are to be expected upon the release of a Sprint, it is only possible to do limited planning. At the end of each Sprint there is a retrospective meeting. In this longer meeting, the changes and challenges the team face are discussed. Changes are factored into the Product Backlog and the next Sprint Backlog is defined. The process then continues. Clearly the Sprint process has internal structure.
Agile governance
Agile planning incorporates a strategy for dealing with change and uncertainty, whereas a ‘command and control’ governance structure such as Prince2 views change as an ‘issue’ to be managed in an ‘issues list’. Issues indicate problems that must be quantified to understand the risk posed.
Firm planning gives management a sense of security at the beginning of a project. As the project progresses however, changes are seen as jeopardising the plan, leaving the project in a constant state of having to react. It is common that events constantly show the real situation to be at variance with the plan. Another anomaly occurs where progress metrics indicate conformance with the plan, yet the reality of the progress is something else entirely. Therefore a plan is only as good as it is a representation of the true state of affairs and the ability of the team to adapt it to changing circumstances.
It is natural that organisations crave certainty. Senior management want to be able to rely on estimates of time and cost. They want to be able to quantify their risk; to understand their uncertainty. Therefore there is structural resistance in many organisations to accepting Agile engineering because of its lack of guarantees. Yet, in the light of experience of projects that fall behind in their deliverables, this resistance may be misplaced because a guarantee that cannot be enforced provides no comfort.
The great unknown in Agile projects is the business requirements. It would be preferable to somehow pin the requirements down in a meaningful way without believing they are set in stone. The ability to do this has proved elusive. What happens in Scrum is that requirements are indeed tied down, but only for a single Sprint (i.e. 30 days effort).
All aspects of project governance can be understood in terms of requirements. Progress reporting and cost estimation all depend on how successful the team has been in delivering appropriate function-ality in each of its iterations. Therefore it is possible to express delivered functionality in terms of requirements. The variable that must be measured then depends on the Agile team’s willingness to express those requirements in an appropriate formalism. This formalism must be acceptable to both the agile and governance teams.
Instead of using a blunt term like ‘business requirements’, one might instead talk of high-level and low-level requirements as a way of reconciling Agile engineering and Agile governance. It might then be possible to say that the high-level requirements were stable, but the low-level requirements were liable to change. This would only work if the high- and low-level requirements were directly related ie if the high-level requirement was a container for the low-level requirements. In this way the low-level requirements serve as the detail that describes the accomplishment of the user’s real goal. It is reasonably safe to say that a set of high-level user goals, and their satisfaction is a fairly safe and stable picture of the system that is being constructed.
An example might help to make the point. A system to manage a hire fleet of vehicles needs the high-level requirement to ‘book a vehicle to a customer’. The low-level implementation might be required to ‘record the driver’s license number’, ‘record the driver’s address’, ‘take the driver’s credit card number’, and so on. While the system was being developed, the management decide that instead of a car hire company, they now want to run a community car club where members are already known. The requirements to ‘book a vehicle to a customer’ however remains valid, whereas the low-level requirements have now changed dramatically, because the ‘customer’ is already known by virtue of being a member. The example illustrates the point of how low-level requirements can change without it affecting the validity of the high-level parent requirement.
The uptake of Agile methods is hampered by organisational cultures that are uncomfortable with the uncertainty of business requirements. But their acceptability would be enhanced by combining them with an Agile governance method that expresses business requirements in a structure of hierarchical abstraction that insulates them from change at the management level.
Dr Peter Merrick is head of Active.requirements, a specialist in the transformation of business requirements into system requirements. www.princelite.co.uk, peter@activerequirements.co.uk
Further reading
The Agile revolution
How a radical rethink of development processes has aligned software with the business
Embedded value
Standard Life has transformed its relationship to IT by embedding the team within its pensions operation and exploiting agile development and an advanced service-oriented architecture
Find more stories in the SOA & Development Briefing Room
