Estimating Defect Production

Estimating the number of defects a project should expect to produce and remove is one of the least talked about subjects in software development. In this article I'll provide some industry statistics to help with that estimation process and hopefully convince you of why Agile will likely reduce the level of effort associated with defect fixing.

There have been studies performed to estimate the number of defects that a software development effort will likely encounter during its life cycle. Obviously the larger the project the greater number of defects one should expect to encounter. If you follow this blog, you’ll know Steve McConnell’s book "Software Estimation: De-mystifying the Black Art” is one of my favorites and is always by my side. One of the chapters discusses estimating defects. In his book, McConnell references a Capers Jones (2000) study that indicates a reasonable expectation is to have 50 defects per 1000 lines of code (LOC).

However, a more granular look will show that smaller projects will experience fewer defects than larger ones; for example a project with fewer than 2K LOC will likely have 0-25 defects per 1K LOC where a project with over 512K LOC will likely have 4-100 defects per 1K LOC. Keep in mind that factors such as your programming language and other technologies will effect this estimate. It’s always more accurate to use historical data to estimate effort but in lieu of that, these data are better than nothing at all.

Here are a few more factors to consider:

- Defects occur at all points during development e.g. requirements, architecture, development, documentation, etc.

- There are best practices for defect removal such as design reviews, code reviews, prototyping, unit testing, system testing, and various levels of beta-testing – each of which have different removal rates. The highest removal rates come from formal code reviews (45-70%), prototyping (35-80%), and high volume beta testing (60-85%). Surprisingly (at least to me) one of the lowest removal ratings comes from regression testing (15-30%).

Here is where I transition from conveying data to providing my opinion on why Agile allows a development team to reduce the effort associated with defect removal when compared to Waterfall.

With Waterfall, the life cycle stages occur in a sequential manner,i.e. requirements, design, development, and test. Although it could be reasonably argued that the number of defects may not significantly change between Waterfall and Agile I think it is unreasonable to assume that the effort to remove them will remain the same. Defects introduced during the requirements gathering and design stages that are not found until development or even later have a snow ball affect because they pile up on one another.

Because of it’s iterative nature, Agile allows for requirements, design, and development defects to show themselves virtually immediately after they have been introduced. Dealing with defects introduced over a single two week sprint is a lot easier to manage than untangling a slew of defects that occurred over several months. We’ve all been on Waterfall projects where our integration testing revealed flaws that have required rewriting methods and even entire components. I contend that those types of wholesale defect removal efforts are mitigated substantially through continuous integration, daily stand-ups, two week sprints with customer tests immediately thereafter, as well as the other feedback loops inherent in Agile.

I’m interested in your thoughts.

The Balance Between Talent and Team

I came across an old Joel on Software (Joel Spolsky) article from 2005 called Hitting the High Notes that I thought was particularly thought provoking. One of the premises of the article is that a single brilliant developer is more valuable for innovation and invention than an army of mediocre ones. His reasoning is that the brilliant developer is capable of thinking things and creating things that are virtually impossible for mediocre developers. The best line that captures the essence of the article is "Five Antonio Salierie's won't produce Mozart's Requiem. Ever. Not if they work 100 years."

But how often does the average organization require the elegance and brilliance of Mozart to compose the software equivalent of his Requiem? When viewed from the context of what the overwhelming majority of software engineers do every day, the answer is “almost never”.

Like Joel Spolsky, I’m a big fan of greatness. Real life accomplishments can often be better than fiction. However, software is as prolific and ubiquitous as there are people and it is because it doesn’t require Mozarts to create useful software to serve an organization and save tons of time and money and provide insight into business and industry trends.In fact, the effect of hiring purely on programming talent is at best problematic.

Let’s agree that software engineering is intellectually demanding. You can't be a dolt and do this job effectively. Technologies and subject matter change too quickly so having intellectual horsepower is a must. It’s also my opinion that for those same reasons one needs to understand that being a software engineer is not a 9-5 job.

As an organization, having intelligent and committed individuals is still not enough. The New York Yankees proved that by trying to buy World Series championships year after year through the assembly of the most talented players alone and failed miserably (As a Red Sox fan I consider that a success story).

That talent-alone strategy doesn’t work in software engineering either. The whole doesn’t necessarily have to be more than the sum its parts to be effective but in those cases one should expect mediocrity and not a whole lotta fun for those involved.

To accomplish cool things, if not world changing things, and have fun doing it there has to be synergy. Included with talent, intellect, and commitment, everyone should respect one another as people and professionals, understand what each other needs to be effective, compliment each others talents, and together fill all pieces of the engineering pie. If you think getting all these pieces to fit together properly sounds really, really, hard - I'm in full agreement!

Accomplishing the assembly of the aforementioned team is certainly a non-trivial task. As with everything in life, it requires compromise and balance. In this case, the compromise and balance is between the individual and team attributes needed, weighting them accordingly, and then hoping you’re right when hiring your next engineer.