Half a decade ago I did some fairly exhaustive research into software time and cost estimation techniques for Microsoft Services. I evaluated the effectiveness of both formal and informal techniques used within Microsoft, across the software development industry, and even in other industries. I looked at everything from Estimation by Analogy to COCOMO II.
After months of research I discovered that the most commonly used estimation techniques were Estimation by Analogy and Wideband Delphi, often done in combination. However, most of the teams using these techniques were doing so informally, and had never heard of either of these terms.
I also discovered that there were no software estimation techniques that consistently produced accurate estimates, other than for small projects, even those sophisticated parametric estimation techniques like COCOMO II.
At the time I was doing the research the failure rate for IT projects was around 70%, and many of those failures were due to catastrophic budget and schedule overruns. It quickly became evident to me that the way the industry had historically thought about estimating software projects was deeply broken; The Emperor had no clothes!
My final recommendation to Microsoft Services was that they adopt Estimation by Analogy and Wideband Delphi, and formally train their staff on the use and limitations of these techniques. My other recommendation was that they also invest in maturing their risk management processes, given the historical inaccuracy of software estimation in general. It was on this latter area that I focused until I left Microsoft.
Our obviously apparent inability to do accurate estimation bugged me for years after I completed this research. This itch that I could not seem to scratch motivated me to do some reading about predication in general, and in particular Quantitative Analysis and its use of Stochastic models; one would imagine that if any industry had worked out how to predict the future it would be the Financial Industry given their huge monetary incentive to do so. Obviously recent events have shown that not to be the case, but this was before all that nastiness.
And then I read “The Black Swan: The Impact of the Highly Improbable” by Nassim Nicholas Taleb, and it all made sense. It is not that we have yet to find an accurate technique to do software estimation; it is that accurate estimation of large, complex software development projects is simply NOT POSSIBLE! (given our current understanding of the laws of physics anyway). Not only does the Emperor not have clothes, but he will remain naked for the foreseeable future.
I suspect that the move to Agile software development practices is a natural response to our inability to do accurate estimation, but I am continually amazed at how many people in the industry and how many of its customers are still in denial about this, what should now be self-evident, fact. I still encounter many projects where development teams are held to early estimates, which are typically done by non-technical project managers. And this is particularly true on fixed-cost\budget projects.
Surely it is time for the entire industry and its customers to acknowledge that this whole approach is simply broken? The illusion is that fixed-cost\budget projects shift ownership of the risk from the customer to the organization who is developing the software; but given the percentage of IT projects that still fail today, this is obviously just that: an illusion; and a pernicious one at that.
We need to start from the premise that accurate software estimation is currently, and probably forever, impossible, and go from there. Agile practices have made a good start but we obviously need to do more. And most importantly we need to educate our customers!
Here is an interview with Nassim Nicholas Taleb wherein he talks specifically about our inability to estimate IT projects:
