You Might Be Doing Fragile

Many years ago I attended a talk by Steve McConnell, the author of a number of seminal software engineering tomes including Code Complete, Rapid Development and Software Estimation: Demystifying the Black Art. I don’t remember the exact year, but it was around the time that the Agile Manifesto was published, and though the term Agile had yet to become an entry in the mainstream software developer's lexicon, many of the practices that we associate with Agile today were already being used in the industry.

 

During Steve’s presentation he said (and I am paraphrasing since I do not have a transcript) that in his experience the most important contributor to success in software projects was not which formal software development methodology a team followed, but rather on how disciplined they were in following the methodology or process that they had. You will note the use of the term formal in that last sentence; by this I am referring to mature methodologies that have evolved and survived in the crucible that is large-scale real-world software development.

 

Based on my own experience at the time, I immediately recognized Steve’s observation as deep wisdom, and this is as true today as it was at the time. Whether a team is following a Big Design and Estimation Up Front (BDEUF) or an Agile process, it is more important that they are disciplined in the adoption of that process, than which process they use.

 

Unfortunately, a lot of engineering teams who attempt to adopt Agile practices, fail miserably. And nine times out of ten they fail because they are undisciplined about their adoption, and they land up practicing what I call Fragile. Many managers don’t seem to get that there is no formal (there's that word again!) process called Agile; rather there are formal processes called Scrum, XP, Kanban, Lean, MSF, etc., that follow the Agile philosophy.

 

Though the aforementioned methods do lend themselves to customization, and in many cases demand it in the name of continuous process improvement, their initial adoption requires disciplined adherence to the specific principles and practices of the method. For example, just doing daily stand-up meetings, will not result in higher developer productivity, product quality or predictability. Stand-up meetings are just one part of Scrum and need to be combined with the other Scrum practices to realize the real benefits of the methodology.

 

And like any organizational change, the adoption of a formal Agile process also requires perseverance; it can take half a year before a team hits its stride with the new process. Unfortunately many failed attempts at adopting Agile become classic cases of throwing the baby out with the bathwater - after continued failure many teams abandon Agile practices and swing back to BDEUF, and then typically continue to fail with that approach.

 

I have heard it said that using Agile is just an excuse for poor planning or a justification for laziness, and that may sometimes be the case, but these are just examples of practicing Fragile. I have seen it time and time again where a team that has been doing BDEUF (please lets stop calling it Waterfall, since that moniker was used to describe how NOT to do software development!) resorts to Agile (read Fragile) when it becomes clear that there is no way they are going to meet their dates, requirements or features. It doesn’t help them, or stakeholders’ perceptions of the effectiveness of Agile practices.

 

So, with a nod to Jeff Foxworthy, here are a few ways you can identify if you are doing Fragile.

 

If you got part way through your BDEUF project and after realizing that there was no way that you were going to make your dates or scope you resorted to an Agile process…

…then you might be doing Fragile.

 

If you adopted a couple of Agile practices, e.g. a daily stand-up meeting, but haven't seen any improvement in your engineering team's productivity, predictability or general satisfaction; and are thinking that Agile is a load of hogwash…

…then you might be doing Fragile.

 

If you are not doing retrospectives or post mortems after every milestone…

…then you might be doing Fragile.

 

If your stakeholder’s don’t have complete transparency into your process and progress…

…then you might be doing Fragile.

 

If you are not using an Application Lifecycle Management  platform like Team Foundation Server to support your adoption of Agile…

…then you might be doing Fragile.

 

If you are not doing Build and Test Automation…

…then you might be doing Fragile.

 

If you don’t require that developers write tests for the code that they write…

…then you might be doing Fragile.

 

If you believe that human beings can estimate complex tasks (greater than 8 hours) with any degree of accuracy…

…then you might be doing Fragile.

 

If you are not refactoring user stories and work items into smaller user stories and work items to mitigate the above human limitation…

…then you might be doing Fragile.

 

If you think that you can mitigate the unquantifiable risks associated with emergent complexity with a fixed-size contingency buffer…

…then you might be doing Fragile.

 

If your Scrum Master (or whatever you call her or him) is A Boss…

…then you might be doing Fragile.

 

If anyone on your team asks, "what should I be doing next?”…

…then you might be doing Fragile.

 

If any stakeholder has to call up a lead or manager and ask "when will my feature be ready?"…

…then you might be doing Fragile. 

 

If anyone in a meeting does not know whether they are a Pig or a Chicken, or does not know what that even means…

…then you might be doing Fragile.

 

If you are not doing sprints or iterations, or if those iterations are measured in months as opposed to weeks…

…then you might be doing Fragile.

 

If you are not doing backlog grooming with stakeholders multiple times during each iteration…

…then you might be doing Fragile.

 

If the total daily time commitment required by each developer in the development process, that is not related to designing software or writing code, is more than about 15 minutes…

…then you might be doing Fragile.

 

If your development leads spend more time managing the process than writing code…

…then you might be doing Fragile.

Please don’t do Fragile.

.NET Iteration Performance

Last night I gave a talk to the .NET User Group of British Columbia on .NET Performance Engineering. During the presentation I demonstrated the performance characteristics of four different C# iteration mechanisms: incremented for, decremented for, foreach and LINQ.

Here (as promised to the audience) is the code for the test application and the individual test cases:

namespace PerformanceMeasurement
{
     using System;
     using System.Collections.Generic;
     using System.Diagnostics;
     using System.IO;
     using System.Linq;

     public class Program
    {
         private enum TimerUnits
        {
            Ticks,
            Milliseconds
        }

         private enum ReturnValue
        {
            TotalTime,
            AverageTime
        }

         public static void Main(string[] args)
        {   
             if (Debugger.IsAttached)
            {                  Console.WriteLine("A debugger is attached! JIT-compiled code will not be optimized ");
            }
            
            Console.Write("Press any key to warm up tests");
            Console.ReadKey();
            var testData = InitAndWarmUpIterationTests();
            Console.Write("Press any key to start tests     ");
            Console.ReadKey();
            Console.WriteLine("\rStarting Performance Tests      ");

#if DEBUG
            Console.WriteLine("This is a Debug build! JIT-compiled code will not be optimized ");
#endif
        
            Console.WriteLine();

            const int IterationCount = 1000;

            RuntIterationTests(IterationCount, testData);
            RuntCollectionTests(IterationCount);

            Console.Write("Tests Complete. Press any key to exit.");
            Console.ReadLine();
        }

        private static List<List<string>> InitAndWarmUpIterationTests()
        {
            var testData = CreateIterationTestData();

            var warmupResults = new List<int>
                                    {
                                        WordCountWithLinq(testData),
                                        WordCountWithForEach(testData),
                                        WordCountWithFor(testData),
                                        WordCountWithForDownward(testData)
                                    };

            Trace.WriteLine(warmupResults.Count);
            
            return testData;
        }

        private static List<List<string>> CreateIterationTestData()
        {
            var entries = new List<List<string>>();

            const string Path = @"data\";

            foreach (var fileName in Directory.EnumerateFiles(Path))
            {
                using (var stream = File.OpenRead(fileName))
                using (var bufferedStream = new BufferedStream(stream))
                {
                    var reader = new StreamReader(bufferedStream);
                    var lines = new List<string>();
                    string current;
                    while ((current = reader.ReadLine()) != null)
                    {
                        lines.Add(current);
                    }

                    entries.Add(lines);
                }
            }

            return entries;
        }

        private static void RuntIterationTests(int iterationCount, List<List<string>> testData)
        {
            Console.WriteLine("Iteration Performance Tests:");

            const TimerUnits Units = TimerUnits.Ticks;
            const ReturnValue ReturnValue = ReturnValue.AverageTime;

            Console.WriteLine("Creating Test Data...");
            var results = new List<int>();

            Console.WriteLine("Running Tests...");
            var t1 = TimeMe(WordCountWithLinq, testData, iterationCount, ref results, Units);
            var t2 = TimeMe(WordCountWithForEach, testData, iterationCount, ref results, Units);
            var t3 = TimeMe(WordCountWithFor, testData, iterationCount, ref results, Units);
            var t4 = TimeMe(WordCountWithForDownward, testData, iterationCount, ref results, Units);

            Console.WriteLine("Result Count: {0}", results.Count);
            Console.WriteLine("WordCountWithLinq \t\t{0}\t: {1} {2}", ReturnValue, t1, Units);
            Console.WriteLine("WordCountWithForEach \t\t{0}\t: {1} {2}", ReturnValue, t2, Units);
            Console.WriteLine("WordCountWithFor \t\t{0}\t: {1} {2}", ReturnValue, t3, Units);
            Console.WriteLine("WordCountWithForDownward \t{0}\t: {1} {2}", ReturnValue, t4, Units);
            Console.WriteLine();
        }

        private static void RuntCollectionTests(int iterationCount)
        {
            const TimerUnits Units = TimerUnits.Ticks;
            const ReturnValue ReturnValue = ReturnValue.AverageTime;

            var results = new List<int>();

            const int MaxValue = 10000;
            var t1 = TimeMe(CreateAndEnumerateList, MaxValue, iterationCount, ref results, Units);
            var t2 = TimeMe(CreateAndEnumerateGenericList, MaxValue, iterationCount, ref results, Units);

            Console.WriteLine("Collection Performance Tests:");
            Console.WriteLine("CreateAndEnumerateList \t\t{0}\t: {1} {2}", ReturnValue, t1, Units);
            Console.WriteLine("CreateAndEnumerateGenericList \t{0}\t: {1} {2}", ReturnValue, t2, Units);
            Console.WriteLine();
        }

        private static float TimeMe<TArg, TReturn>(
            Func<TArg, TReturn> me,
            TArg arg,
            long iterationCount,
            ref List<TReturn> results,
            TimerUnits units = TimerUnits.Milliseconds,
            ReturnValue returnValue = ReturnValue.AverageTime)
        {
            var timer = new Stopwatch();

            var currentIteration = 0L;
            var time = 0F;

            do
            {
                timer.Start();
                results.Add(me(arg));
                timer.Stop();
                time += (units == TimerUnits.Milliseconds) ? timer.ElapsedMilliseconds : timer.ElapsedTicks;
                currentIteration += 1;
                timer.Reset();
            }
            while (currentIteration < iterationCount);

            if (returnValue == ReturnValue.AverageTime)
            {
                time = time / iterationCount;
            }

            return time;
        }

        private static int WordCountWithFor(List<List<string>> entries)
        {
            var wordcount = 0;

            for (var i = 0; i < entries.Count; i++)
            {
                for (var j = 0; j < entries[i].Count; j++)
                {
                    wordcount += entries[i][j].Length;
                }
            }

            return wordcount;
        }

        private static int WordCountWithForDownward(List<List<string>> entries)
        {
            var wordcount = 0;
            for (var i = entries.Count - 1; i > -1; i--)
            {
                for (var j = entries[i].Count - 1; j > -1; j--)
                {
                    wordcount += entries[i][j].Length;
                }
            }

            return wordcount;
        }

        private static int WordCountWithForEach(List<List<string>> entries)
        {
            var wordcount = 0;

            foreach (var entry in entries)
            {
                foreach (var line in entry)
                {
                    wordcount += line.Length;
                }
            }
            return wordcount;
        }

        private static int WordCountWithLinq(List<List<string>> entries)
        {
            return entries.SelectMany(entry => entry).Sum(line => line.Length);
        }

        private static int CreateAndEnumerateList(int maxValue)
        {
            var list = new System.Collections.ArrayList(maxValue);

            foreach (var val in Enumerable.Range(0, maxValue))
            {
                list.Add(val);
            }

            var total = 0;

            foreach (int val in list)
            {
                total += val;
            }

            return total;
        }

        private static int CreateAndEnumerateGenericList(int maxValue)
        {
            var list = new List<int>(maxValue);

            foreach (var val in Enumerable.Range(0, maxValue))
            {
                list.Add(val);
            }

            var total = 0;

            foreach (var val in list)
            {
                total += val;
            }

            return total;
        }
    }
}

Note: in order to run this code you will need to create a data folder that contains a number of multi-line text files and modify the code to point at that folder.

 

On my workstation I get the following output when directly executing the application:

 Starting Performance Tests
 
 Iteration Performance Tests:
 Creating Test Data...
 Running Tests...
 Result Count: 4000
 WordCountWithLinq               AverageTime     : 3697.65 Ticks
 WordCountWithForEach            AverageTime     : 916.269 Ticks
 WordCountWithFor                AverageTime     : 833.674 Ticks
 WordCountWithForDownward        AverageTime     : 736.798 Ticks
 
 Collection Performance Tests:
 CreateAndEnumerateList          AverageTime     : 582.813 Ticks
 CreateAndEnumerateGenericList   AverageTime     : 232.494 Ticks
 
 Tests Complete. Press any key to exit.
        

As you can see, the decremented for loop is the fastest, and approximately 5 times faster than the LINQ implementation! It is also about 13% faster than the incremented for loop, which is significant. After showing the code and results above, I walked the audience through how I discovered why this is the case. I won't document the entire process here, but after looking at the disassembly of the optimized native code generated by the CLR x86 JIT, I discovered that in the decremented case the evaluation of the condition uses the test and jl (Jump If Less) instructions, rather than the cmp (Compare) and jae (Jump if Above or Equal) instructions, which are used in the incremented case. Obviously the former combination of instructions executes faster than the latter. The latter combination of instructions are also used in the un-optimized, i.e. Debug, version of the decremented for, which is even slower than the un-optimized foreach, which is why it is important to make sure you always performance-test the Release version of the code.

 

So why should anyone care about this?

Well if you are using LINQ in a performance-sensitive code path then you should STOP doing that immediately, and in cases that you are iterating over very large arrays with incremented for, you should try the decremented for and see what you gain. Note that the overall performance gain will vary depending on how much work you are doing in the body of the loop.

And lastly, don't be afraid to look at the disassembly of your managed code; it's right there in Visual Studio and it represents the ultimate truth of how your code is executing on the hardware.

Applied Architecture Patterns on the Microsoft Platform, Second Edition

The book that I have been working on for the last year, with Andre Dovgal and Dmitri Olechko, has been published and is now available for purchase on Amazon.   Applied Architecture Patterns on the Microsoft Platform, Second Edition, published by Packt Publishing, builds upon the framework that was presented in the first edition and includes new chapters on recent Microsoft technologies, including .NET 4.5, SQL Server and SharePoint.

The book is aimed at software architects, developers and consultants working with the Microsoft Platform, who want to extend their knowledge of the technologies that make up said platform, and learn how those technologies can be composed into complex systems.

I decided to contribute to this book because of how useful I found the first edition. Hopefully this updated version will be as useful to those who read it.

Everyone Jumps, Everyone Fights

I have been in a lot of interviews recently and the topic of my leadership style has come up in many of them. My 20-year career has given me numerous opportunities to observe a wide spectrum of leadership styles, some very effective, and others not so much. My own leadership style has percolated from the styles of the aforementioned leaders, and more specifically from the philosophies, practices and techniques that I have seen be consistently successful. Though most are self-evident I think they are worth writing about, and I will do so in a number of posts.

My informal study of leadership began while I was serving in the South African Defense Force with 1 Parachute Battalion, then part of the 44 Parachute Brigade. In airborne units typically everyone jumps, and everyone fights; everyone from the private working in the mess to the commanding officer is jump-qualified and is trained for active combat. The commanding officer while I was at 1 Parachute Battalion was Colonel J.R. Hills. Colonel Hills was a seasoned and decorated veteran of the Angolan Bush War, and was easily one of the most capable soldiers in a unit of highly capable warriors. He without exception commanded the respect and loyalty of every member of the unit, by demonstrating time and again that he was a leader who not only could jump and fight, but who would lead from the front.

Being in the Infantry, airborne or otherwise, means that your skill as a soldier is primarily predicated on your endurance. An infantryman needs to be able to march or run a significant distance burdened with equipment and weaponry and arrive at the objective with enough reserves to be able to fight immediately. And the physical and emotional endurance that is then required for combat is staggering.

One of the ways that Colonel Hills demonstrated his willingness to lead from the front was his annual running of the Comrades and Washie ultra-marathons, and his challenge to the entire unit to beat him in the former marathon, and simply finish the latter. The reward offered was always a much sought-after weekend furlough. Though I do recall some members of the unit beating his Comrades time, I can’t recall a single soldier taking the Washie challenge. The Colonel demonstrated time and again that he had more raw endurance than anyone in the unit.

By the time I served in the unit South Africa was not involved in any active conflicts, but if there was anyone I would have wanted as my commanding officer in active combat it would have been Colonel Hills. I would have followed him into Hell had it been required.

Lead from the front is probably my most core leadership principle. Software engineering teams are meritocracies, so effective leadership of these teams requires that every member of the team respects the leader’s technical knowledge and skill. Leading from the front also means that a leader should not expect anything from their team members that they are not prepared to do themselves. This does not imply that the leader needs to master every discipline of every role in the team they lead, but they should at least be able to roll up their sleeves and make a contribution in every role if necessary, and certainly understand the day-to-day requirements and challenges of each role.

I am lucky that I have had opportunities to work in most sub-disciplines within the broader Software Engineering discipline, including Product Management, Program Management, Project Management, Architecture, Development, User Interface\Experience Design, Quality Assurance, Release\Configuration Management, and even Support and Operations to a lesser degree; so I have a good understanding of the minutia of most of these. I also endeavour to write code every day; partly because I just love to write code, but also because it means I never lose touch with the reality of being a software engineer. In my last role I made sure that I was actively contributing to the product, by taking Architecture, Development and QA work items in every sprint. I believe that this made me a much more effective leader, earned me the technical respect of my team, and gave me some of the context necessary to build rapport with everyone on that team.

I should note that not all the leadership practices I observed being successful in the military translate to leading software engineering teams. Military organizations employ a Command and Control management style by necessity, which is almost always a disaster if used with modern software engineers. I favour a Facilitation management style, which I will elaborate on in a follow-up post.

Peak SharePoint?

Although SharePoint Server has historically been one of Microsoft’s fastest growing products, it may just be that the world is approaching, or perhaps has even reached, Peak SharePoint [Adoption]. I am not naively suggesting that SharePoint adoption will drop off precipitously, but I think the case can be made that SharePoint’s best days may be behind it.

I am currently co-authoring a new edition of Applied Architecture Patterns on the Microsoft Platform for PACKT Publishing. The new chapters that I have written include a SharePoint Primer, and one of the first points I make is that SharePoint is not a product, but rather a business application platform. It is the breadth of capabilities in SharePoint (and a fiendish viral sales and marketing strategy) which have made it such a popular platform, but it has also made the platform incredibly complex, and difficult and costly to manage, administer and maintain. I posit that it is this complexity that will ultimately bring about SharePoint’s slow demise.

SharePoint is a distributed ASP.NET web application, which uses complex HTML, CSS, and JavaScript, and a .NET-based API, which  makes COM Interop calls to COM components in an ISAPI Extension, which  accesses data stored in a hyper-normalized SQL Server database and XML files stored in the Windows file system. And that is just the core system; SharePoint also exposes a plethora of other services, each with their own complexities and idiosyncrasies.

Before the 2010 release, SharePoint was not considered a serious contender in the Enterprise Content Management space, and adoption was generally limited to the departmental level. SharePoint 2010 hit a sweet spot, providing sophisticated ECM capabilities at a price point that was impossible to ignore. SharePoint 2010 was also necessarily much more complex than SharePoint 2007, and upgrades to the new version were painful and costly for enterprises, and created an entirely new and very lucrative IT consulting sub-sector – SharePoint Wrangling. 

I have worked with multiple organizations who have SharePoint Site Collections that have 100GB or more of content stored in them. Many of these Site Collections have been in existence for years and have been subject to significant entropy and Bit Rot. Administrators of these Site Collections have no simple way to differentiate relevant and useful content from trash, and often need to use 3rd-party discovery, management and migration tools, to get a  handle on their SharePoint infrastructure.  By the second law of thermodynamics  the entropy will be ever increasing, while the potential value of the Site Collections increase as content is added; value that is going to become more difficult and expensive to extract as these Site Collections age.

Clearly Microsoft’s plans for SharePoint are in the cloud. I was told by an ex-colleague who attended the recent SPC14 conference that the sessions were dominated by topics related to Office 365 and SharePoint Online. Microsoft is also regularly releasing new versions of SharePoint Online, while the on-premises product lags behind; SharePoint Online already reports its version as 16.x, while the on-premises version is still 15.x. Despite Microsoft’s focus on SharePoint in the cloud however, enterprises have not rushed to move all of their on-premises content to Office 365. I suspect that this is because of the current price-performance/scale characteristics of SharePoint Online and a lingering distrust of The Cloud. I image that Microsoft will address the former soon and the latter will dissipate over time, but I do not believe that a move to the cloud will drive a new wave of SharePoint adoption and upgrades.

Organizations who have a significant existing investment in SharePoint have not rushed to migrate to SharePoint 2013, either on-premises or in the cloud. 

Migration from SharePoint 2007 to 2010 was so painful for many who accomplished it that they are probably reticent to attempt it again with SharePoint 2013, and the new features in SharePoint 2013 do not seem to be compelling enough to justify the effort or risk. In particular, the new Social Computing features seem half-baked when compared to what is available in Yammer. The Yammer acquisition obviously happened too late to influence SharePoint 2013’s feature set. The Mobile Device features also don't seem particularly compelling, given the super-high-resolution screens and highly-usable browsers that are now the norm on mobile devices.

So what could Microsoft do to drive a new wave of SharePoint adoption and upgrades?

The promise of ECM is that it can turn any organization into a Learning Organization by enabling the extraction of significant Knowledge and Wisdom from all the content they generate and store. This has been the Knowledge Management Holy Grail. Unfortunately, like the elusive chalice in the Arthurian Legend, it has in almost all cases remained unfound. In my opinion the failure to deliver the promised Return On Investment of ECM is because Knowledge Management is currently heavily predicated on human classification of content, and because to date ECM systems have heavily relied on Search capabilities to extract Knowledge. It is abundantly clear that in general humans are terrible at consistently  categorizing and classifying content (probably due to innate human cognitive biases). Machine Learning and Natural Language Processing technologies have now reached a point where they can, on average, do a far better job at automatically classifying content (in a computationally cost-effective manner), and can even improve the accuracy of their classification over time.

With SharePoint, Microsoft brought Enterprise Content Management to organizations of every size, by providing a powerful, usable, feature-rich platform at a price point that almost any company could afford. Perhaps if Microsoft did the same for sophisticated Machine Learning- and Natural Language Processing-based automatic content classification, they might give the SharePoint platform significantly longer legs, and allow all of their customers to realize the promise of ECM and Knowledge Management. The substantial value that this would add to the SharePoint platform should also offset any issues associated with the increased complexity of the platform.

Given the staggering potential value of the content currently stored in SharePoint instances the world over I don’t doubt that it will be around for decades to come, but I also suspect that it has grown overly complex, and difficult to manage and maintain. If Microsoft doesn’t do something disruptive with the platform soon I strongly suspect that we will shortly reach Peak SharePoint, if we haven’t already.

The Mythical Story Point

I fairly recently became embroiled in an argument about whether Story Points or hours are better for estimating the effort associated with Software Engineering tasks. Having helped a lot of teams adopt Scrum and other agile practices, this is not the first time I have danced this dance, and my experience has left me with a strong preference for using hours rather than an abstraction.

The motivations for using Story Points (or any other abstraction for that matter, e.g. T-shirt Sizes) to estimate effort, seem very reasonable and arguments for their use are initially very compelling. Consistent high-accuracy software estimation is probably beyond current human cognitive capability, so anything that results in improvements, even small ones, constitutes a good thing.

The primary motivation for using Story Points is that they represent a unit of work that is invariant in the face of differing levels of technical skill, differing levels of familiarity with the code or domain, the time of day or year, or how much caffeine either the estimator or developer doing the work has ingested. They also provide a typically more course-grained unit of estimation than hours, which by necessity will result in more apparently accurate estimates. By combining this course-grained unit of work, with mandatory refactoring of Stories (or Epics, or Product Backlog Items, or whatever nomenclature you choose to use) larger than a particular effort size, a team is bound to improve the accuracy of their estimates.

The use of estimation abstractions also seem to be beneficial when a team follows the Principle of Transparency, which is espoused by most Agile philosophies. When a team follows this principle, they make the team’s velocity, estimates, actuals and other data available to all stakeholders (e.g. Sales, Marketing, Support and Management), who almost invariably care a great deal about the work items that they have a stake in, and particularly when those work items will be DONE. By using Story Points for estimates one initially avoids setting unrealistic expectations with stakeholders, who may not necessarily understand the prevalence of emergent complexity in the creation of software.

I would imagine that the human brain has a lot of deep circuitry designed exclusively to deal with time. It was clearly highly adaptive for an early hominid to be able to predict how far in the future a critical event might occur; whether that was knowing when the sun might go down and nocturnal predators appear, or when a particular migratory species would be in the neighbourhood. We are clearly genetically hard-wired for understanding course-grained time, e.g. circadian rhythms, synodic months, the 4 seasons, and the solar year. And human cultural evolution has yielded many powerful and ubiquitous time-related memes, which have added a deep and fine-grained temporal awareness to the human condition, measured in seconds, minutes and hours. Almost every modern electronic device’s default screen displays the time and date, including phones, microwaves, computers, thermostats etc. Time is so ubiquitously available in our modern digital lives that the site of an analog wall clock will require a Tweet or post to Instagram. And everyone has a calendar of some sort that they use to manage their futures. We have clearly become the Children of Time.

Unfortunately, being the Children of Time has obviously not made us capable of even vaguely accurate estimation of the time any task of significant complexity will take. However, we are also terrible at estimating pretty much everything else, so I suspect this is not indicative of a specific limitation of our time-related neural circuitry. 

It is also our aforementioned parentage that limits the usefulness of Story Points and similar abstractions for estimating effort in general.  After some, typically short, period of time everyone on the team and all the stakeholders unconsciously construct a model in their minds that maps the unit of the abstraction back to time in hours or days. And as soon as this  model has been constructed they ostensibly go back to thinking in hours or days, though they now require an extra cognitive step to apply the necessary transformation.

So why bother with using the abstraction in the first place?

I have experimented with the use of estimation abstractions with teams in the past and I can confidently say that using abstractions has proven to be a distraction in the long run. I have settled on an approach that uses course-grained time buckets for initial estimates, e.g. 1, 5, 10, 25, 50 and 100 hours. The key performance indicator for estimation should a be a steady improvement in estimation accuracy over time, rather than the absolute accuracy of the estimate.

Accuracy in the software estimation process is emergent and improvements are predicated on iteration, increased familiarity with the code and the domain, and visibility into the historical data (and analysis thereof). Showing team members how far their estimates have been off over time, just before they estimate other work, is a good way to prime them, and give them an appropriate anchor.

I suspect that I will dance this dance again in the future.

There’s No Going Back in The Box

I have always had a Hobbesian view of the human condition, but over the last couple of years the fine and rather disturbing details of that view have been elucidated by the The Black Swan by Nassim Nicholas Taleb, Nudge by Richard Thaler and Cass Sunstein, Free Will by Sam Harris, a number of books by Steven Pinker, Risk : The Science and Politics of Fear by Daniel Gardner, and most recently by Thinking, Fast and Slow by Daniel Kahneman.

I have also always had a strong suspicion that the human condition would be completely codified in my lifetime. What I did not suspect however is just how thoroughly depressing the coalescence of that codification would be; indubitable scientific research shows that we are pathologically, recursively and absolutely  deluded as a species, and that the vast majority of judgements and decisions that we make individually and collectively are non-optimal in the best cases, and patently wrong in the worst.  This goes a long way to explaining why the planet is in the state that it is in.

As I have attempted to internalize the codification contained within the aforementioned books, I have struggled with how one can turn the awareness of the limitations of one’s meatware to one’s advantage. Basically it would require the insertion of the equivalent of a  perhaps-you-should-think-hard-about-that  hardware interrupt into all of one’s cognitive processes, or at least those that don’t involve having to respond to a flesh and blood Panthera leo that has put Homo sapiens sapiens on the menu. Achieving this level of mental discipline is hard enough in its own right, but then once the interrupt has fired there is the not-insignificant challenge of identifying and adjusting for the cognitive biases and heuristics that might be in play; and there are A LOT of them and their effects are most often very subtle and  totally counter intuitive (as their name implies). It may simply be beyond my, and most human’s, cognitive abilities to avoid making decisions with a “sub-system” evolved for avoiding the inadvertent consumption of toxic plants, hungry predators and randomly falling trees in The Savannah.       

Having an awareness of this codification also leads to the uncomfortable realization that humanity is utterly vulnerable to manipulation by amoral agents (sociopaths, politicians or artificial intelligences)  who have deep knowledge of this research and the motivation to use/misuse it.  

If we are to survive as a species we are going to have to overcome the significant limitations of our own neurology. Given that most humans are seemingly unable to even overcome their petty prejudices, I don’t have a high degree of confidence that we are going to be around long enough to speciate. I can only hope that as awareness of this research becomes more widespread that we will become collectively less vulnerable to the limitations of our own mental processes, and the manipulation of those processes by unscrupulous forces. We could start by requiring that all high school students and politicians learn about cognitive biases and how they negatively impact our ability to make optimal decisions and accurate judgements.

Some or other ancient Greek philosopher suggested that one should “Know Thyself”. They should have followed this gem with a warning that one cannot “Un-know Oneself” after the fact. There is no going back in the box.  

CODE

I am a huge fan of both Barak Obama and Shepard Fairey. I have one of Fairey’s classic Obama 2008 “Change” posters hanging in my home. For the hell of it I did a self-portrait in the style of Fairey’s classic poster.  Obama has undoubtedly brought HOPE, PROGRESS and CHANGE to the world. I thought I should choose a less ambitious slogan; something aligned with my capabilities and chosen profession, and self-fulfilling of course; so “CODE” it is.

Trapped in the Knowledge Value Network

In an effort to drive innovative thinking in my engineering team I have had them read "The Innovators Dilemma" by Clayton M. Christensen. One of the key concepts in the book is how Value Network’s prevent otherwise successful companies from initiating or responding to Disruptive Innovation. 

Though this term is overloaded, Christensen defines a Value Network as:

"The collection of upstream suppliers, downstream channels to market, and ancillary providers that support a common business model within an industry. When would-be disruptors enter into existing value networks, they must adapt their business models to conform to the value network and therefore fail that disruption because they become co-opted."

Value Networks are inherently self-reinforcing and  cause often-catastrophic organizational inertia. Counter-intuitively, the more successful an organization is (in the traditional sense anyway), the more valuable the network is, and the greater the inertial effect that network has on the organization’s ability to innovate. Being in a  Value Network seems to amount to signing up for collective-blinker-wearing in the name of short- to mid-term success. Not only do the members of the network not want disruptive innovation, they also don’t seem to see it  coming; and when they do, they are often powerless to do anything about it because of the need to maintain short- to mid-term revenue.  

"The Innovators Dilemma" was first published in 1997, making it a relatively old book, and its position as the state of the art in Innovation Management has been usurped by "The Lean Startup" and its ilk, but most of the ideas contained within are still highly relevant. I am routinely amazed, and not in a good way, by how few professionals, in either explicit or tacit Innovation Management roles, have read this book, or are even aware of the principles it expounds, despite how accurate they have been shown to be. This book should be required reading for everyone involved in Innovation Management, which is just about everyone involved in commercial software development in my opinion. 

I have found that the notion of inertia-causing value networks is applicable to more than just organizations; it is also a very useful metaphor for planning one’s career as a software professional. As a software engineer one develops one’s own Knowledge Value Network simply by using languages, tools, platforms and technologies on a day-to-day basis.

Microsoft SharePoint is a very good case in point; a SharePoint Developer is not just a .Net Developer with some SharePoint knowledge; SharePoint has enough breadth and depth that it is a domain in its own right, and becoming expert in all its APIs, services, models, schemas, languages (CAML anyone?), and other technical concepts, is at least a fulltime job and maybe two.  One could have spent the last decade just doing SharePoint development, making lots of money and getting ahead in the world of software development, and never have heard of Hadoop, Node.js, Scala, or even other Microsoft technologies like ASP.NET MVC.

So what happens when SharePoint's star starts to burn through the heavier elements? When does one become aware of the fact that one’s future success is no longer assured by being a SharePoint expert? Probably too late in one’s career to do anything about it, spending one’s technical twilight years doing legacy SharePoint application support. Old software does not die; it simply becomes covered under different clauses in the support policy.

I am not making any assertions about the specific future of SharePoint; this is most definitely a product with long legs, but it is a truism that those legs will get shorter at some point in the not-too-distant future and then disappear entirely.

I have also seen this phenomenon time and time again in large-scale enterprise development; a team of talented young engineers are hired to develop, enhance and maintain a complex line of business application. After a few years, and dozens of releases, the software has become so complex that it is a knowledge domain in its own right. New developers on the team take six months to get up to speed, despite being experts in the tools and technologies with and on which the application is built. Knowledge of the product’s minutia becomes the developers’ personal value proposition and source of job security. They live and breath the technical intricacies of the application  every day, to the exclusion of emergent  and alternative technologies that are simply not part of the technology-mix used by the application. The line of business application, its tools and technologies become their personal  Knowledge Value Network. And like organizational value networks it is closed and self-reinforcing. These same developers often struggle to find financially and intellectually rewarding work when they literally emerge from under the rock.

Developers who do not recognize this will find themselves technically irrelevant at a critical point in their careers. One hopes that they have decided to follow the management track before this point, otherwise they are going to find themselves grey-haired individual-contributors with technical skills only relevant for supporting legacy applications. That it not to say that this will be a poor source of income, but no amount of money is worth the soul-death that becoming a professional manager means for many software engineers.

So how does one avoid this fate? Set aside time each week to research emergent and competitive technologies, products, platforms, tools, methodologies, languages, architectures, designs, metaphors, patterns, anti-patterns, philosophies and domains. And don't just be reading edge; if you can actually play with these technologies, do so, even if you can't leverage that work in your day-job. Not only will it give you real job security and maintain your sanity, it will also make you a better software engineer.

A personal example was learning Dynamic and then Functional Programming. A few years ago I decided to learn Python and then F#, despite the fact that my day job at the time required neither. Learning these languages has made me a significantly better C# programmer, though I still can’t tell you exactly why a space suit is like a monad; or is that the other way round? No matter. I also find that I am now able to solve programming problems far more elegantly and simply that I would have before learning these languages.

My personal practice is to set aside a few hours per week and let my curiosity drive me. My most recent endeavour has been to work through  Daniel Shiffman’s excellent new book, “The Nature of Code”, which is about simulating real-world phenomena in code. The examples are all in Processing and I have had a lot of fun creating random animated 3D surfaces using Perlin Noise. And no, it doesn’t have anything to do with SharePoint; but that is the point.

Stay curious. Disrupt thyself.

Tour of the International Space Station

If you are a space nerd then this has got to be the coolest video EVER!