Why Is Software Estimation Always In the Backseat?

I don't understand why software estimation doesn't take a more prominent role in software development.

Virtually everyone who's livelihood is related to software development has experienced projects where team members have worked a zillion hours, the project went over budget, had features ripped out to meet a date, was delivered with poor quality, etc.

Here are some data to provide context into our experiences in relation to the software development space as a whole (thanks again to Steve McConnell's book "Software Estimation: Demystifying the Black Art"):

• Many studies have concluded results that state 25% of projects will be canceled, 25% will be on time and within budget, and 50% will be late and/or over budget.
• A project with 10,000 function points, which is a mid-sized project, has a 1% chance that it will be delivered early and a 20% chance that it will be canceled. As the number of function points goes up so does the failure rate.
  
• On average, a late project is 120% late and an over budget project is 100% over budget.
  

It is likely that the percentages for late/over-budget is only a piece of the picture and the actual experience is much worse. When a project is 120% late and/or 100% over budget, there is severe urgency to deliver. When that occurs functionality is often stripped out of the original scope and teams are forced into a death march to get the project done. Most teams do not pay overtime so whether a developer works 8 hours or 18 hours, the effect on the budget is the same. As a result, the overage stats do not reflect that what is delivered is likely less than what was scoped and the hours applied are far more than the budget overrun indicates. It is also reasonable to assume that in that environment design time suffers causing a more difficult application to maintain, extend, and scale.

The moral of the story is that the our industry is dysfunctionally addicted to under estimating our projects! One major reason for this, believe it or not, is expert judgment. There are a lot of smart people with loads of experience in our industry and many will estimate tasks, and thus projects, based on gut instinct instead of quantifiable data. This is by far the least reliable estimation technique.

In the next article, I'll provide a few methodologies that you can use that are fairly easy to implement and will increase your likelihood of success.