The Miracle of Agile

I am a disciple of Agile as a software development methodology for the simple reason that it was brought to Engineering teams through divine intervention. That statement might contain a bit of hyperbole but only a bit. Agile has revolutionized the way in which software is being developed by many teams across many industries though it is still unknown or a mystery to many more.

What is Agile? Agile is an iterative approach to software engineering who's precepts are: collaborative teams consisting of cross functional members; frequent validation of requirements, designs, and implementations; self organized teams; and, unambiguous individual accountability.

How is Agile different?
I'm sure everyone reading this has heard of processes such as requirements gathering, analysis and design, development, QA testing, and user acceptance testing. Agile as well as many of the tradition approaches, Waterfall being by far the most common, embrace these principles. The differences lie in the implementation of those principles. Where Waterfall will attempt to describe the the entire application upfront through documentation which is then followed by development, QA testing, and user acceptance testing, Agile bundles those principles into iterative life cycles called "sprints". Sprints are of a fixed duration, commonly between 2-4 weeks, and repeated over and over again until all the features are developed, the project budget is exhausted, or time runs out.

Plan-Driven (Waterfall) vs Value-Driven (Agile)
The way in which the principles are implemented is the manifestation of their difference in philosophy. Where waterfall is plan-driven, Agile is value-driven. A plan-driven methodology is heavy on documentation and strives to fully describe the application before developing it through requirements, analysis, and design. That process provides a framework to methodically develop what's been documented. Typically there is a gargantuan project plan that is associated with the effort and everyone marches to the plan with little regard for course corrections and re-evaluation.

A value-driven methodology stresses an empirical engineering process using an inspect and adapt approach with frequent feedback loops, i.e. sprints. The reason for this is that Agile believes that requirements gathering, analysis, design, development and testing should happen together when the feature is ready to be implemented. This approach provides flexibility with project changes such as feature deprecation, adding new features, and changing the requirements of existing features. As these events occur in a Waterfall project the documentation becomes cumbersome to maintain. The most efficient time to write about it is when it's time to develop it. During an Agile project, it's easy to make course corrections such as changing requirements as a result of what's been developed previously or a reconfiguring development priorities because of extenuating circumstances.

Although it's comforting to know that so much thought went into determining what the customer wants and how their requirements should be implemented, there are significant inefficiencies with the level of detail that goes into the upfront analysis. The reasons are simple, some of which are:

• Users aren't always clear in their own minds what they want and how they want it.
• Large documents are not handled well by many people. They tend to be too abstract for people to fully grasp.
  
• Things inevitably change in the minds of many users once they can see and feel their requirements implemented.
• The business environment often changes and directly impacts the priorities of the project.
  

In my mind, it makes perfect sense to get your customer to see and feel the application features as soon as possible. This doesn't mean abandoning requirements, analysis, and design. It just means that upfront planning should be limited to:

• Project Initiation: Planning the sprints, project infrastructure, team members, communication matrix, etc.
  
• Developing a Prioritized List of Features: This is a list of features, prioritized by importance, with user-stories on how the features will be used.
• High Level Architecture: A high level architecture should be developed. This phase determines things like the application tiers, technologies employed, thick or thin client, etc.
  

Once those steps are taken then development should start. I prefer two week sprints at the end of which my team gets immediate feedback from the customer. Catching mistakes with interpreting the user requirements, or users realizing that their requirements need to change is far cheaper to address at this point in development than at the end of the project during user acceptance testing. The recurring customer demonstrations also builds user confidence because they see progress. From the developer perspective, sprints provide a clear mission and the immovable customer demo at the end provides a constant sense of urgency.

Why isn't everyone doing it?
There isn't a single answer as to why everyone isn't using Agile but the most common is - you guessed it - resistance to change. Here are a few reasons for resistance:

• The Devil You Know: People are comfortable with what they know and uncomfortable with what they don't know. In my experience, once the change is made, most team members see more similarities to their previous approach than they thought were there otherwise. The apprehension of the change is greater than the actual change itself.
• Personal Fear: Can I do this? Will I like it? Will I learn it fast enough? Will I look foolish? How will my job be affected?
  
• Risk Aversion: The chances of switching methodologies without feeling some amount of pain is low. It will take at least one, but probably several, projects to change behaviors where Agile processes are implemented and feel natural. The speed of the transition is dependent on the fervency and commitment of the transition evangelist.
  
• Existing Team(s): Agile is highly collaborative and tends to minimize attention on documentation, thus an inherent lack of detailed specifications. If your projects rely heavily on coding to specs, then the transition to Agile could be challenging. However, if your team is already working from prototypes, has close involvement by the customer, and is highly communicative, then you are already working in an Agile-like style and the transition will likely provide change in the form structure not in approach.
• The Iron Triangle: There is contention with each side of the iron triangle - features, cost, and schedule - and it is virtually impossible to fix commitments to all of them simultaneously. Agile makes clear that commitments can only be applied to a maximum of two of the sides. One side always needs to be fluid. For example, company x has a demo at a convention in 3 months where it must show y set of features, therefore the the schedule and scope are fixed so the cost needs to be variable. With Waterfall projects the commitments to date, scope, and budget are typically, and irrationally, fixed at the beginning of a project. This makes people feel comfortable. However, in reality one or more of those commitments will inevitably be violated because it is unlikely that enough is known to allow for committing to all three.

Although it is unlikely that Agile was brought to us through divine intervention, the genius in it's approach makes me wonder what took so long to get here.