Foreword: This blog entry was originally intended to be a single article. In writing it I found that I had trouble trying to communicate all the information in one sitting. As a result I’ve decided that this will be a 2 part article. The first part (presented here) discuss the Iterative Waterfall pattern and the reasons why this pattern of software development process is detrimental. In the second part I’ll discuss possible approaches to moving away from this pattern.
Update: Dave Nicolette has written a great article that provides more context, from the both organizational and team point of view. It’s well worth reading and you can find it here.
Over the last 3 years I’ve noticed a reoccuring anti-pattern amongst companies trying to implement Agile. This has happened with development teams that are particularly experienced with RUP [for reasons that I'll discuss towards the end of this article].
There are two variations of this anti-pattern, which I’ve called Iterative Waterfall and Staggered Iterative Waterfall. The Staggered Iterative Waterfall anti-pattern is simply the Iterative Waterfall anti-pattern taken to it’s extreme. Throughout this article I’ve used the term Iterative Waterfall to refer to both variations except in situations where I need to distriguish between the two.
So, what are the characteristics of these anti-patterns?
A description of the problem
The Iterative Waterfall anti-pattern may arise in when consecutive timeboxed iterations are devoted to consecutive phases from the traditional waterfall software lifecyle (SDLC).
As an example, consider a project that has been started with iterations that have been timeboxed to 2 week intervals. Each iteration is dedicated to one phase from the traditional waterfall SDLC; either Analysis, Design, Construction or Testing. Frequently Design and construction are combined. So in our example Analysis is done in the first iteration, Design and Contruction in the second iteration and Testing in the third iteration. This is then repeated for the duration of the project.
A Gantt chart for this project is shown below:
Project managers will look at this Gantt chart and will immediately notice that there is room for improvement. The Business Analysts that worked in iteration 1 are not fully productive again until iteration 4. Similarly for the developers and testers. If we were to stagger iterations we could utilise every team member at 100%!
Here’s what a Gantt chart that shows the Staggered Iterative Waterfall anti-pattern:
Reasons why this approach is undesirable
The Iterative Waterfall model of software development has a number of drawbacks. The most serious of these problems is the impact on the teams ability to communicate and collaborate. The separation of the understanding of the problem (Analysis), from the solution (Design/Coding), and from the quality (Testing) is arbitrary. One of the most significant advantages of Agile methods is that this seperation of knowledge is elimated, or at the very least blurred. This is done by having multi-skilled teams and by having continuous communication between team members. To re-introduce this separation of knowledge into an Agile project is counter-productive.
Let me list the reasons why I believe this model to be very dangerous:
- Breaks the collaboration model of Agile software development. There is a contract in any handoff point, in this case the requirements documention is the contract document (although it may be implicit). Agile software development is more effective in environments where there is continuous dialog between the customer and the developer. Introducing a written document into this dialog is, at best, not helpful.
- Does not improve the visibility of progress made. Measuring a teams progress can be difficult at the best of times. The Iterative Waterfall model complicates the measurement of progress. When is a particular piece of work complete; at the end of Construction? at the end of Testing? How much progress has been made at the end of Analysis?
- Deminishes flexibility for changing business requirements. If I were to start analysing a problem immediately (iteration 1), I would not recieve functional software until the end testing (iteration 3). In our example (where we had iterations timeboxed to 2 weeks) this would equate to an elapsed time of 6 weeks. This may be an improvement on a Waterfall model but if you have the option of delivering every 2 weeks, why would you not want to do that?
How does this anti-pattern come about
Part of the problem is that there are several terms that are common between RUP and Agile that are overloaded. That is, they have different meanings when used in different contexts. The term “Iteration” is especially confusing.
Let us look at how RUP defines an “Iteration”:
“Each pass, which is like a mini waterfall, explores a new portion of the requirements set and offers a chance to correct defects and rework the result of the previous iteration.” – Introducing the IBM Rational Unified Process essentials by analogy.
The diagram from this page is particularly illuminating. A RUP “Iteration” is a complete cycle of each of the waterfall phases.
The Agile concept of an “Iteration” is quite different. It’s more appropriate to say that Agile iterations are timeboxes, during which work is completed. Within the Agile “Iteration” there is no separation of the phases. Even though the development team will perform some of the same activities, the separation and ordering is not there. My collegue Victor said that, “We take waterfall and put it in the blender.” I personally think that’s quite a good analogy.
So, how does this anti-pattern evolve? I believe its comes about when a team tries to introduce agile with a preconception of what is meant by the word “Iteration”.
I’ve outline two variations of an anti-pattern that I’m calling Iterative Waterfall and Staggered Iterative Waterfall. These patterns of software development have all the disadvantages of traditional Waterfall without any of the advantages of Agile.
But recognising an anti-pattern and actively addressing it are two very different things. In my next blog entry I’ll discuss some approaches to mitigating the Iterative Waterfall model of software development, and putting Agile back on track.