Agile is Not Magic
A nine question test to decide if your next team should be agile or waterfall
Well-meaning consulting firms pitch clients using PowerPoints rammed full of buzzwords and transformational ideas. One frequent suspect is Agile project management. But what does Agile really mean for your business? Re-imagined IT functions are happily filled with Agile champions. LinkedIN profiles and CVs are peppered with the latest Agile badges and guru statuses. All good, but where, oh where are the stalwart defenders of waterfall? Agile is great; Agile is changing the world. Agile empowers teams to pursue the creation of unlimited value.
Some claim every program should be run agile!
But…..Suppose you are comfortably settled into a flight somewhere between New York and London. The captain comes on the loud speaker, “Ladies and gentleman, please enjoy this flight, arrival time into Heathrow will be 705 am. Just to let you know, we did have a small engine warning light come on. This plane was designed with agile so we don’t have any documentation to reference. But don’t worry, sit back and relax, the co-pilot and I will be conducting agile sprints until we solve the problem. Enjoy your flight.” Even the most seasoned jet setter would turn white knuckled. Who would trust their life to such a piece of agilely designed equipment at 30000 ft.
Large technology projects and teams invariably debate the best delivery method. Agile vs Waterfall. Agile and waterfall have many formal branded methods. Along with the methods come sufficient consultants and gurus to champion(sell); the scope of this article is exploring the philosophical roots and differences. With this background you should be well equipped with a foundational view of the competing approaches. You will be ready to organize your next initiative. The savvy leader finds both the off trend waterfall methods and the flashy agile methods a place in the management tool belt. Each is a tool suited to a different purpose. This article offers guidance to help you select the best philosophy suited to your programme, team or upcoming initiative.
Part One: A Battle of Philosophy
Waterfall(planning) approaches rose to prominence in the 1970s during a time of science led technical innovation. The U.S. Defense Department formalized waterfall in a document helpfully titled DOD-STD-2167A. It contained “uniform requirements for the software development that are applicable throughout the system life cycle.” Anyone involved in business and technology projects has seen more waterfall plans than they care to remember. Waterfall looks something like this:
Consider the major applied technological advances in the early decades of last century: bridges, telephones, buildings, rockets, and radios. These technologies assumed a predictable target environment. For example, a rocket is designed to maintain flight along a trajectory. Once successfully built, the use case of the rocket remains equally valuable today, tomorrow, and for all time. The environment (the rules of physics) are stable, not co-evolutionary, with the rocket technology. Design once, use for a very long time. Once a rocket is designed, the design remains valid. The laws of physics do not suddenly change and render the rocket and its trajectory calculations useless. This stable environment paradigm is the key assumption. The assumption is woven throughout waterfall approaches.
The 1990s was a new era. The web, social technologies, and startup businesses dramatically exposed a completely different environmental paradigm. Building for the web is markedly different than designing rockets. The new wave delivered technologies into a chaotic, co-evolutionary world. Co-evolutionary means the environment changes. Imagine designing rockets where one day earth’s gravity is 9.8 m/s² while tomorrow there is a 50% chance gravity will be 100 m/s² or a 50% chance it will be 1 m/s². In such a universe designing rockets is much harder.
Designing technology in a co-evolutionary environment means a great design for today’s requirements will not be competitive tomorrow. Here is another example, if you launch a shiny new America Online or Yahoo web portal style business in 2021, I predict you will fail catastrophically. Lets go back in time shall we:
The old stable environment was ordered and Newtonian, the social environment is chaotic and co-evolutionary. In response, new technology development approaches became more adaptive. They became agile. Development work was designed in shorter bursts and timelines were truncated to release small pieces into the wild quickly. Testing, aka colliding with reality, is the surest way to obtain feedback. Teams implement course corrections to faulty design much sooner and thus save project cost. Co-evolving environments magnify the value of early testing and feedback.
I see threads of the agile philosophy in the work of John Boyd. He was a fighter pilot who came up with the idea of the OODA loop. He believed the difference in aerial combat between vanquished and vanquisher was the pilot who executed the OODA loop faster. If one pilot makes a turn, but during that time the opposing pilot makes five maneuvers (5 cycles through OODA), which do you think will be victorious?
On the web, whoever adapts faster typically wins. The formal recognition of this philosophy applied to software development was the agile manifesto in 2001:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Agile fixes resource inputs and challenges those fixed resources to create the maximum value possible.
Waterfall(planning) fixes a targeted value outcome and plans resources accordingly.
Agile is bottom up.
Planning is top down.
Agile believes desired future outcomes are unpredictable.
Planning believes outcomes can be achieved by the application of know-how, experience, and planning.
Agile embraces failure as a low cost experiment.
Waterfall avoids costly failure.
Agile believes the world is chaotic and unknowable.
Waterfall believes the world is ordered and repeatable.
Part Two: Agile vs Waterfall Selection Test
Use this short quiz to help you think through which approach suits your next programme or initiative. Add up the points for each question below.
- Does your project or program have explicit known requirements which must be delivered? No +1
A piece of software for a regulatory tax summary would be a known requirement. Your new product initiative to launch the “uber for diamonds” would be an unknown requirement. Support teams are very much a no because, by definition, problems are not known in advance!
2. Is the environment you will deliver into rapidly evolving(co-evolutionary). Yes +2
The rules for dual entry accounting are unlikely to change, so an accounting system might be a no. A new iPhone app for a new set of customers or product line might be a yes. Any product targeting a dynamic consumer market is likely co-evolutionary.
3. Do you already have agile methods and teams actively succeeding within your organization Yes +1
If your organization already uses one type of approach, momentum will be on your side. Agile approaches bring ways of working and “language” that requires a non trivial effort to adopt.
4. Can the team or project be fully empowered to own outcomes? Yes +1
Waterfall seeks to reduce friction by planning and early exposure of “dependencies” between actors and resources internal and external to an initiative. Agile is best when the ability to create sits with actors within the agile team/structure. An agile team dependent on external parties will be hamstrung.
5. Has this type of project been executed elsewhere? If No +1
By definition, Software as a Service is bringing your organization a solution which many other organizations also use. I tend to think the higher in the stack you are working, the more likely a project should be waterfall. Also, I tend to think systems of record are more aligned to waterfall, whilst systems of experience tend to look more agile. If you are starting from a blank window and a C compiler, agile sounds very interesting.
6. Can incentives be aligned so that all parties benefit from “excess” value created by the project. If Yes + 3
Agile fixes inputs(people, time) and seeks to inspire really massive value creation. Organizations going for outside help from consultants or other vendors often avoid t & m due to the agent principal problem. Waterfall is a much more natural fit for contracts because your partner commits to an outcome, and the partner takes the risk of how much input resources are required for delivery. This more certain fixed outcome is great for selling an initiative to the board, but it does impose an upper bound on the value you are going to receive. The popular PMP certification calls this “gold plating” and projects are discouraged from delivering “extra goodies”. Accordingly, in waterfall the outcome value tends to be capped. Designing a contract with your partner to be both accountable for outcomes and agile, can be asking for two philosophically opposed approaches. A number of hybrid approaches, such as scaled agile(SAFe), are designed with an eye to address these tradeoffs. Whether they succeed or not is outside the scope of this article.
7. Is it possible to craft small pieces of delivery which are useful before the whole is delivered? Yes +2
Going back to an airplane, the whole thing needs to be functional before the end user finds it of much value. An airplane delivered with no engines would create little value. On the other hand, a small enhancement to your website or an A/B test on a marketing advertisement might be very useful.
8. Is the cost of failure catastrophic? If No +1
9. Does this program involve a business capability which is a competitive differentiator? If Yes +2
Non critical capabilities which your business executes tend to be okay with “good enough”. Waterfall program and project outcomes are more predictable. However, when it comes to building capabilities central to the business model, agile seeks to offers a handsome exposure to significant upside.
Add up all your answers and then check against the following
0–4 — Your initiative may be very well suited for Waterfall
5–10 — Waterfall or Agile could be a fit
10 + — Your initiative could benefit in an Agile sweet spot