There seems to be a lot of chatter on the Internet recently about the evils of Agile, and Scrum specifically. Some of these posts make a few valid points, but they all seem to have been written by people who have been burned because they were doing Fragile (see You might be doing Fragile), or who simply want to get a few extra page views on their blogs by posting content that is designed to provoke an emotional response. I won’t get into the specific flaws I see in their arguments, rather I will describe one single practice, that falls under the Agile banner, that can mean the difference between sustained success, and failure on any project; regardless of whether the people involved claim to be practicing Agile or any other methodology for that matter.
Despite the mind-blowing progress that has been made in software development technologies, tools, processes and practices in the last half century, more than half of the software development projects undertaken by organizations of all sizes still land up in some or other failure state, often related to poor execution rather than a misconceived vision. Though complex software development remains a relatively difficult undertaking, this does not justify or excuse the high rate of failure. Surely after creating software for over half a century, we would have worked out how to do it in such a way that most attempts were successful, at least in execution? Why have we not learned from our collective mistakes?
I assert that the software industry has already stumbled upon a simple practice that is almost a silver bullet for successful software project outcomes. Over my career, I have been on teams that have used BDEUF (Big Design and Estimation Up-Front), proto-agile Iterative methods, formal Agile methods, including Scrum, eXtreme Programming and Kanban; novel and ad-hoc processes, and some that have just practiced Seat-of-the-pants. And across all of the aforementioned, the one practice that seems to have an almost binary impact on successful execution, is The Retrospective (though it is not always named such).
The Retrospective represents the ultimate confluence of the warm and fuzzy of human psychology and the hard and practical of the Scientific Method. It creates a simple, though formal process, from which a team can learn from its own collective experience, continuously experiment with new practices (with managed risk), and gives every member of the team an equal opportunity to shape the process that they are following.
Effective human-centric process engineering is hard; human psychology is complex and volatile. A process that works perfectly well for one person or team may fail dismally when adopted by another, even if the formalisms of that process are held to with high fidelity. And given the ever-more-rapidly changing environment, a formal process that works perfectly today, will almost certainly not be optimal tomorrow.
And most teams who attempt to adopt formal process frameworks, either don’t implement all the practices and disciplines required to make those frameworks effective, and then let them mutate organically; or institute overly-cumbersome change control processes in an attempt to minimize the cost and productivity impact of changes. Processes can quickly go from being assets to liabilities; reducing organizational productivity and agility, rather than improving them.
What organizations and teams need is a continuous process improvement process as the meta-process for the organization; one which permits and encourages continuous optimization in the face of the ever-changing environment. Managing the continuous improvement of processes needs to be baked into the organizational culture, and it must be owned by everyone from the intern to the CEO; everyone needs to be able to suggest changes to the process, and those changes should be considered if the proponent can formalize a credible hypothesis of how the change will make a net improvement in the process. The organization or team also needs to be able to measure the effectiveness, or lack thereof, of all tactics that it adopts, or experiments it attempts.
The Scientific Method is inherently empirical; for any hypothesis, there needs to be defined:
- One or more tests that will prove the hypothesis
- One or more tests that will disprove the hypothesis
- A description of the data that need to be collected over which the tests will be evaluated
- A mechanism to collect those data
- A description of how other potentially un-related environmental factors might affect those data
There is a misconception that the Scientific Method is complex. It’s not; it can be simply reduced to cycles of Reflect, Execute, Measure, and Inflect. The Retrospective is the primary mechanism for making the Scientific Method the basis for a continuous process improvement process, while also providing the heartbeat of the process.
I will save a detailed description for how I run Retrospectives for a future post, but the rough structure is as follows: every member of the team, in no specific order, gets an opportunity to comment on practices that should be continued, practices that should be stopped or modified, and new practices that should be adopted. All comments, though particularly those related to abandoning and adopting a practice, must be supported by a description of the expected effect that the proposed practice will have on the key metrics that the team uses to measure their performance, and how those effects will be measured. If the hypothesis is consistent and coherent, then the practice is adopted for one or more iterations or milestones (depending on how long it will be before the expected impact should be measurably and unambiguously identifiable). The proponent is then accountable for making sure that the required data are collected and analyzed, and for providing a conclusion in the Retrospective that coincides with the end of the experiment. If the experiment is successful then the practice is adopted as part of the formal process, until someone makes a supportable case for why it should be modified or abandoned in a Retrospective.
The person who is running the Retrospective also must ensure that argument, debate and defensiveness are kept to a minimum, that egos are managed appropriately, and that the Retrospective does not run over the allocated time. As is common with other Agile practices it is a best practice that the person running the retrospective is not a Boss. The importance of creating an atmosphere where participants feel empowered to speak their minds, are confident that their opinions are valued, and that all hypotheses will be evaluated solely on the merits of said hypotheses (rather than the seniority or the source), cannot be overemphasized.
Even if a team is not following frequentative process cadence, regular Retrospectives provide a vital heartbeat for the team and the process. The cost of adopting the Retrospective is low, and this simple practice will have a significant positive impact on the productivity and morale of any team, regardless of the process that they are using, or the organizational culture.
