Emergent Complexity Management

In a previous post I asserted that accurate estimation of even moderately-complex software projects is impossible. This stems from our inability to predict the probability and impact of emergent complexity, due to our cognitive biases and the current human-centric method of generating software. Our inability to accurately predict and estimate is evident in almost every domain; the Great Recession being a perfect and painful example. Despite the fact that every decade or so a Black Swan Event occurs in the financial markets, Black-Scholes is still being used to predict the future price of options, and the markets still seem to be in denial about our inability to make accurate long-term predictions (even when we use sophisticated mathematical models). The difference between the markets and your average medium- to high-complexity software project is that in software projects you are almost guaranteed to have one, and typically multiple, Black Swan Events!

Despite the current significant failure rate of software projects, Professional Project Managers still believe that they can mitigate the risks associated with emergent complexity by building in a “buffer” or “contingency”. The data show unequivocally that this is, at best, pure self-delusion, and at worst criminal negligence; no matter how large this buffer is made, there is no way to ascertain upfront whether or not it is big enough to cover overruns that may arise due to emergent complexity. And obviously making it too large will cause project failure for purely financial reasons.

It is fairly obvious that the emergence of Agile Practices is an attempt to address the obvious shortcomings in software estimation specifically, and in our software development processes in general. Recursive refactoring of Epics into relatively-low-complexity User Stories and Tasks, and using techniques like Planning Poker, do make vaguely-accurate estimation possible, but I suspect that the fact that many Agile Gurus suggest abandoning estimation (in hours as opposed to an abstraction like Story Points) entirely, is an indication that this does not substantially improve the overall temporal predictability of Software projects.     

Selling Agile Practices to developers is like selling Gatorade to people who have been lost in the desert for days without a canteen. They get it instantly. The problem however is selling Agile to other “stakeholders”; whether they are internal Management, Sales, Product Management or Marketing teams, or a client for whom one is building software. This is where the real Agile battle is being fought.

The status quo is our, i.e. software engineers, own fault; for nearly half a century now we have been telling our customers that we know how long it is going to take, and implying that we have high confidence in our estimates. One has to wonder why, after we have been shown to be so utterly untrustworthy in this regard, that our customers and managers have continued to let us near computer keyboards.

So why do customers and managers keep buying the snake oil? Obviously they believe that they are limiting their exposure to the risks associated with emergent complexity (and sometimes incompetence unfortunately),  by passing that risk on to the development team or company responsible for the software development. Sure, if they make the penalty clauses punitive enough, and the developers agree to those clauses, it may partially mitigate the risks, but it does not address the fact that, in this day and age of rapidly-and-always changing competitive landscapes, a missed software release may very well mean the demise of an organization. So instead of mitigating the risk, it is simply being hidden and made binary.

That said, and despite what the Gurus say, it is not practical to simply leave software project costs and schedules open-ended. We need a way to roughly size software projects, but we also need a way to manage emergent complexity and minimize it’s negative impact  to the schedule and ultimately to the value of the software to the user.

I am obviously one of a multitude of software professionals grappling with this, and I have yet to find a solution that is totally satisfying (and sellable). I suspect that the answer includes motivating stakeholders to abandon the illusion of certainty for real transparency and a daily opportunity to inflect the engineering process; and a more scientific approach to product management along the lines of The Lean Startup.