Estimating Defect Production

Estimating the number of defects a project should expect to produce and remove is one of the least talked about subjects in software development. In this article I'll provide some industry statistics to help with that estimation process and hopefully convince you of why Agile will likely reduce the level of effort associated with defect fixing.

There have been studies performed to estimate the number of defects that a software development effort will likely encounter during its life cycle. Obviously the larger the project the greater number of defects one should expect to encounter. If you follow this blog, you’ll know Steve McConnell’s book "Software Estimation: De-mystifying the Black Art” is one of my favorites and is always by my side. One of the chapters discusses estimating defects. In his book, McConnell references a Capers Jones (2000) study that indicates a reasonable expectation is to have 50 defects per 1000 lines of code (LOC).

However, a more granular look will show that smaller projects will experience fewer defects than larger ones; for example a project with fewer than 2K LOC will likely have 0-25 defects per 1K LOC where a project with over 512K LOC will likely have 4-100 defects per 1K LOC. Keep in mind that factors such as your programming language and other technologies will effect this estimate. It’s always more accurate to use historical data to estimate effort but in lieu of that, these data are better than nothing at all.

Here are a few more factors to consider:

- Defects occur at all points during development e.g. requirements, architecture, development, documentation, etc.

- There are best practices for defect removal such as design reviews, code reviews, prototyping, unit testing, system testing, and various levels of beta-testing – each of which have different removal rates. The highest removal rates come from formal code reviews (45-70%), prototyping (35-80%), and high volume beta testing (60-85%). Surprisingly (at least to me) one of the lowest removal ratings comes from regression testing (15-30%).

Here is where I transition from conveying data to providing my opinion on why Agile allows a development team to reduce the effort associated with defect removal when compared to Waterfall.

With Waterfall, the life cycle stages occur in a sequential manner,i.e. requirements, design, development, and test. Although it could be reasonably argued that the number of defects may not significantly change between Waterfall and Agile I think it is unreasonable to assume that the effort to remove them will remain the same. Defects introduced during the requirements gathering and design stages that are not found until development or even later have a snow ball affect because they pile up on one another.

Because of it’s iterative nature, Agile allows for requirements, design, and development defects to show themselves virtually immediately after they have been introduced. Dealing with defects introduced over a single two week sprint is a lot easier to manage than untangling a slew of defects that occurred over several months. We’ve all been on Waterfall projects where our integration testing revealed flaws that have required rewriting methods and even entire components. I contend that those types of wholesale defect removal efforts are mitigated substantially through continuous integration, daily stand-ups, two week sprints with customer tests immediately thereafter, as well as the other feedback loops inherent in Agile.

I’m interested in your thoughts.

Why Is Software Estimation Always In the Backseat?

I don't understand why software estimation doesn't take a more prominent role in software development.

Virtually everyone who's livelihood is related to software development has experienced projects where team members have worked a zillion hours, the project went over budget, had features ripped out to meet a date, was delivered with poor quality, etc.

Here are some data to provide context into our experiences in relation to the software development space as a whole (thanks again to Steve McConnell's book "Software Estimation: Demystifying the Black Art"):

• Many studies have concluded results that state 25% of projects will be canceled, 25% will be on time and within budget, and 50% will be late and/or over budget.
• A project with 10,000 function points, which is a mid-sized project, has a 1% chance that it will be delivered early and a 20% chance that it will be canceled. As the number of function points goes up so does the failure rate.
  
• On average, a late project is 120% late and an over budget project is 100% over budget.
  

It is likely that the percentages for late/over-budget is only a piece of the picture and the actual experience is much worse. When a project is 120% late and/or 100% over budget, there is severe urgency to deliver. When that occurs functionality is often stripped out of the original scope and teams are forced into a death march to get the project done. Most teams do not pay overtime so whether a developer works 8 hours or 18 hours, the effect on the budget is the same. As a result, the overage stats do not reflect that what is delivered is likely less than what was scoped and the hours applied are far more than the budget overrun indicates. It is also reasonable to assume that in that environment design time suffers causing a more difficult application to maintain, extend, and scale.

The moral of the story is that the our industry is dysfunctionally addicted to under estimating our projects! One major reason for this, believe it or not, is expert judgment. There are a lot of smart people with loads of experience in our industry and many will estimate tasks, and thus projects, based on gut instinct instead of quantifiable data. This is by far the least reliable estimation technique.

In the next article, I'll provide a few methodologies that you can use that are fairly easy to implement and will increase your likelihood of success.

Software Estimation v.1

I recommend to everyone who is involved in any way with software development to read the book "Software Estimation: Demystifying the Black Art" by Steve McConnell. I laughed, I cried, it changed my life. Well maybe not the first two, but it did change the professional side of my life.

There's so much to software estimation that one article cannot do it justice; especially if the objective of the article is to keep it short and concise. This first of probably many blog entries on software estimation will be dedicated to distinguishing between an estimate and a commitment. This is a vital point because the two are often used interchangeably.

Estimating is an unbiased analytical process. With regards to software development, the objective is to approximate the amount of time and/or cost to develop software to solve a problem or satisfy a specific need. As an approximation, an estimate should be communicated as a range, e.g. "it is likely that this project will be delivered in 3-5 weeks".

A commitment is much more definitive. It is an agreement to deliver specific functionality on a specific date for a specific cost, e.g. "the project will be delivered in 6 weeks."

Related to a commitment is a target. A target is a biased process based on the goals of the business, e.g. "We need this software delivered by June 15 to demo at the premier industry convention!".

In my experience, more often than not estimates are expressed as commitments which are influenced by the business driven target.

Communicating estimates as a single point number vs as a range is misleading (in most cases unintentionally). Every software development effort is, in actuality, an invention and an invention cannot be guaranteed to be completed on a certain day for a certain cost. As we all know, our estimates are not 100% accurate. However, if we communicate estimates as a single point number we are implying 100% accuracy.

Developing high/low cost estimates is fairly common and as you can probably guess I recommend communicating both to your customers. There are many techniques to quantify, thus making credible, high/low estimates. Although estimation by intuition is one estimation technique - and is probably the most frequently used - it is the least reliable. I'll be writing about some of many techniques I garnered from Steve McConnell's book in future blog entries.

On the other hand, dates are rarely communicated as high/low estimates. Typically they are communicated as commitments (single point date) which often result in very long days for developers, negotiating with the client to deliver at a later date, removing features from scope, providing a poor quality product, or a combination of some or all of those. I have a question for you, when was the last time you presented your delivery dates as probabilities, e.g. "there is a 25% probability of us delivering the software in 11 weeks and a 98% probability that we'll deliver it in 15 weeks."? The techniques for quantifying those probability statements will come in later articles too.

I'm interested in hearing about your issues/resolutions to software estimation problems.