Why Software Development is like Ironing a Thneed

 “I’m being quite useful. This thing is a Thneed.
A Thneed's a Fine-Something-That-All-People-Need!
It's a shirt. It's a sock. It's a glove. It's a hat.
But it has OTHER uses. Yes, far beyond that.
You can use it for carpets. For pillows! For sheets!
Or curtains! Or covers for bicycle seats!"

                                         – from “The Lorax” by Dr. Seuss

During my time in the military two decades ago I become highly skilled at ironing. And not only shirts and pants, but beds, hats, sheets and other items one would not normally consider “ironable”. I continue to this day to do my own ironing. I actually find it rather therapeutic.

Since having children I have also become reacquainted with the works of Theodor Geisel, more affectionately known as Dr. Seuss. It struck me the other day while ironing a particularly pesky shirt that software development is very much like ironing a Thneed.

So let’s make some assumptions about a Thneed based on the description above. Some poetic licence and imagination will be required.

1. Nobody is entirely sure what a Thneed is, not even the manufacturer.
2. When customers buy a Thneed they have only a vague idea of what they need it for and how it is going to make their lives better.
3. No two Thneed’s are exactly alike; they are the snowflakes of garments.
4. Thneed producers create new and improved Thneed’s all the time.
5. A Thneed is too big and awkwardly shaped to fit on your ironing board and Dr. Seuss makes no mention of a Thneed-press.
6. It is hard, if not impossible, to estimate how long it will take you or a highly trained team of Thneed-ironers to iron a Thneed, with any level of accuracy or confidence in your estimate.
7. The market for Thneed-ironing accessories is confusing in its profusion, pace and super-competitiveness.
8. Thneed-ironing methodologies were derived from shirt and pants ironing methodologies, but they are actually poorly suited.
9. A Thneed is like most other garments in that it occasionally requires ironing.
10. Thneed manufactures’ own Thneed’s are usually the worst ironed.

So based on the assumptions above, how does one go about ironing this Thneed thing? Well the trick is known by every person who has ever had to iron a shirt.

Manipulate the garment so that a small piece of it is flat on the ironing board and then iron that piece. Then get another piece flat and iron that piece, and so on and so on, until you have ironed the entire garment. If you are foolish enough to try to iron large sections of the garment at the beginning, you will become very frustrated and will probably run out of steam before you are done. You have to “divide and conquer” when it comes to ironing a Thneed; that is the winning strategy.

I don’t think that I need to actually spell out why this is like software development; if you have done any software development in your life you will know that I am right (or mostly so).

Maybe someday I will post “Why Architects are like the Lorax, and Users like the Once-ler”. On second thoughts maybe I won’t.

Happy Ironing!

Good Software Design is Honest

Ten Principles of Good Design Redux | Part 6

The Software Industry has been lying to itself and its customers since its emergence. At some point in the early evolution of the Software Industry someone must have noticed that the emerging development lifecycle model, originally proposed described by Winston Royce and now known as the “Waterfall” or “Big Design Up Front” model, was better suited to Aeronautical Engineering than it was to software development. Designing and building software is not the same as designing and building aircraft.

[Update (2011-10-07): Winston Royce did not propose Waterfall; he was the first to formally describe it, and used it as an example of how not to do software development. He was the “someone” I was referring to in the previous paragraph. Though this error makes my assertion that Aeronautical Engineering is better suited to the Waterfall Model seem a little arbitrary, it does not invalidate the point of the post. Mea culpa for poor fact checking.]

I am not so naïve that I would suggest that they are different merely because of the complexity of the problem domain or because of the amount of uncertainty innate to the process. Though I can only imagine that having the Laws of Physics as a significant source of constraints must reduce the uncertainty somewhat in the design and implementation of an aircraft, I would assert that they are most different in the degrees to which they are subject to the more capricious aspects of Human Nature.

When an aircraft manufacturer designs a new aircraft they typically know precisely what the majority of the requirements are from the outset, e.g. carry this many Wi-Fi-connected, grumpy adults and screaming infants from this point on the globe to this other point, using this much fuel, and oh, don’t crash. The tolerances and constraints are well understood; the laws of physics are essentially immutable, and you can only “comfortably” cram so many humans into a given space (though as someone who is over six foot and flies regularly I have to say that I am not so convinced of my last point).

The aircraft manufacturer can plug all these requirements and constraints into a model and calculate whether or not, given existing and emerging technology, they can feasibly build the aircraft, or if and where the invention of new technology is going to be required. Much of the problem domain is known and is relatively constant, i.e. the Laws of Physics and the current state of the Material Sciences. They also know what they don’t know, and they can mostly quantify the risk\cost associated with that uncertainty thanks to good historical data. A lifecycle model that is dominated by a discrete design phase is clearly effective in bringing a new aircraft to market, though obviously prototyping and innovation are also required during this initial phase. The cost associated with this protracted design phase is accepted as necessary by the, now mature, aviation industry, probably because the cost of failure is so high.

So why is Software development different? One of the most significant differences between Aeronautical and Software Engineering is that users’ of software products and systems are very rarely able to articulate detailed requirements at the outset. The constraints and requirements have to be extracted out of the minds of the intended users, in the best case, or out of the mind of one or more analysts who thinks they understand the users’ requirements, in the worst. And though the Laws of Physics are at play in the hardware that the software runs on, they almost never have to be considered as constraints in the design of the software. Typically the requirements gathering process is a bootstrapping exercise that continues well into the actual development of the system. Users have to have usable examples of what they don’t want to lead them to understand what they actually want, and the engineers have to attempt to solve some of the known hard technical problems in order to reveal the initially-unknown, usually harder, ones.

And it is impossible to estimate how long it is going to take to extract the real requirements and then develop the software to meet those requirements, at the beginning of the project. That is not to say that a waterfall model, that included exhaustive prototyping during the design phase, would not work for software, but it would require that everyone acknowledge that the requirements gathering phase would need to be completed before a fixed-cost time and effort estimate of the development could be provided, and that the duration of that initial phase could only be roughly estimated. There are simply more unknown unknowns in Software Development than in classical Aeronautical Engineering. I say “classical” because modern aircraft probably require as much Software Engineering as they do Aeronautical Engineering.

Another difference is that because so much of the software that powered the growth of the industry was developed by under- or un-paid geeks, it has created a mass hallucination about how much [paid for] time and effort it actually takes to develop high-quality software. Everyone has now come to accept this hallucination despite the number of software projects it has caused to run over time and budget, or to fail completely.

But thankfully all of this is changing; Agile software development practices are an attempt to address this innate dishonesty, and acknowledge that we as software developers simply don’t know what we don’t know. And though Agile has become mainstream, there are still those who doggedly cling to the old dishonest and delusional ways.

It is every software architect’s and developer’s responsibility to promote and champion Agile as an honest approach to the design and development of great software, particularly in the face of grumblings from old-school project managers and customers who want the illusion of fixed risk.

Note: I could write an entire book on the topic above, and was well on the way to doing so before I reminded myself that this was just a blog post destined to be read by 10s of my friends. I am sure though that the above makes the point I was trying to make.

Good Software Design is as Little Design as Possible

Ten Principles of Good Design Redux | Part 10

I think it was the eminent Rico Mariani who coined the term ”OOPoholic” to describe a software engineer who is addicted to adding gratuitous indirection and complexity to their code in the name of Object Oriented Design. After doing many code and architecture reviews I now develop a speech impediment every time I hear the word “facade”.

Einstein is credited with saying "Everything should be made as simple as possible, but not simpler." I think this holds especially true for software. Design patterns were originally proposed to make designing and describing designs easier, not harder. I propose the following razor:

If using a particular design pattern makes it harder to describe the the overall design to ones grandmother then one probably shouldn’t be using it.

Note: Replacing “grandmother” with “Project Manager” or “Client” in the aforementioned, does not reduce its utility in any way.

Good Software Design is Environmentally Friendly

Ten Principles of Good Design Redux | Part 9

This is the principle that I have taken the most liberty with. Rams’ original meaning was self evident; the manufacturing processes and materials used to realise a design should be environmentally sustainable and generally “Green”. I would agree that software should be designed in such a way that it uses hardware, and thus energy, efficiently; but I think there is a much broader application for this principle.

In a recent post I wrote about Gestalt Driven Development and Gestalt Driven Architecture. I defined Software Architecture as “the process of designing and developing software in such a way that it will remain in harmony with the significant contexts within which it is created and runs over its entire lifetime.” The “significant contexts” are ostensibly the software’s environment, and it is this environment that should be the software’s BFF.

Good Software Design is Thorough Down to the Last Detail

Ten Principles of Good Design Redux | Part 8

This principle speaks to one of my pet peeves; Software Architects are first and foremost Software Engineers, and therefore need to be able to map any high level design they create to at least one feasible concrete implementation. Some people seem to believe that once you become a Software Architect that you are excused from understanding the technology all the way down to the last turtle. I have witnessed too many once-technical architects design solutions that are impractical or inappropriate, because they have lost touch with the underlying technologies.

Good Software Design is Long-Lasting

Ten Principles of Good Design Redux | Part 7

Note: This post is technically out-of-order, but if modern microprocessors can do it then so can I.

I very recently designed a system to replace a “legacy” application that has been running for over 25 years. I have no doubt that this is not unusual and that there are still millions, if not billions, of lines of code still running that are well into their 20s. One would imagine that we learned our lesson with the whole Y2K nastiness, or simply by observing just how long some code manages to stay running.

Unfortunately I have seen time and time again how ostensibly competent Software Architects tacitly design software systems for a limited life-span. And it is the clients of these architects who have to bare the costs, since by the time the software starts to display the symptoms of its accelerated decrepitude they are usually heavily locked into the software, and have to do significant surgery to make critically-needed changes. Typically these changes add significant technical debt to the system and make subsequent changes harder still.

This can all be avoided if one designs the software with the assumption that it might live for a quarter of a century or longer. Of course you may be given a specific lifespan as a formal requirement. It is up to you to decide just how much salt to take those requirements with given your understanding of the client or domain. The case might also be made that designing relatively short-lived software is a way to guarantee your own job security. That may hold true in some cases but I would assert that if you are morally so flexible then you should be in politics rather than software development.

Good Software Design is Unobtrusive

Ten Principles of Good Design Redux | Part 5

If I were to apply this principle as it was originally stated by Rams, it would contradict the assertion that I made in an earlier post that a software design needs to be a work of art. Ram’s suggests that products are “like tools and are neither decorative objects nor works of art”. I wholeheartedly agree that Form should always follow Function in software design, but I also believe that there is a genetic component to aesthetic that is useful in establishing the quality and suitability of a software design. 

One would naturally assume that the form follows function principle would be universally adhered to by all software developers and architects. And that is generally a safe assumption. It is not however safe to assume that this form-pursued function is merely the set of formal requirements for the system; often the system is also designed to address the personal requirements of the designer.   

Many Software Architects are rabidly righteous; “Tiger Architects” defending their favoured technologies, design patterns and methodologies with ferocity and zeal, even in cases where those memes are obviously not optimal for a design. Despite the fact that Software is a memeplex on the leading edge of memetic evolution, the aforementioned phenomena is mostly a manifestation of  significantly baser stuff; mammalian territoriality and human ego (though the former may be considered the precursor of the latter).

I have been guilty of this in the past, but over the years I have learned to look at my own software designs through a deconstructivist lens; separating the fundamental requirements of the system from those that I have introduced because of my own biases, preferences and comfort zones. That is not to say that I simply reject all parts of the design that display evidence of the aforementioned, but I attempt to make sure that none of these personal requirements subsumes any of the fundamental requirements of the system. One might say that I practice Deconstructive Software Architecture, though that might be misapprehended if it is confused with the similarly-named movement in meatspace architecture. 

I am motivated to add the following rider to this principle:

Good Software Design is Egoless.

Good Software Design Makes a Product Understandable

Ten Principles of Good Design Redux | Part 4

This principle is applicable in two areas; firstly to User Interface Design and User Experience Design, and secondly to the Conceptual, Logical and Physical Designs of a software system. In the case of the latter the comprehensibility of a software system design is so obviously vital to the successful realisation of that design that I will focus this post entirely on the former. 

 

The ease with which a user comes to understand the visual metaphors utilized in a user interface, and thus the speed with which they can accomplish necessary tasks and operations, defines how usable that user interface is. From a user experience perspective “understandable” and “usable” are synonymous.

 

In 1995 I worked for a progressive IT company who were in the process of setting up a full usability testing lab. The plan for this lab called for the installation of cameras for recording video of the user’s facial and body language, a key and mouse logger to monitor and capture user input data, a pulse monitor to measure the user’s stress levels, and a high resolution screen-capture of the user’s interaction with the application itself. The plan even called for having statisticians and clinical psychologists on staff to structure the usability tests and interpret the integrated data streams. The goal was to not only test the usability of clients applications, but also gain a deep understanding of the Laws of Usability. I left the company before the plan was realized, to go and work on a new shiny thingy called the “World Wide Web”, so unfortunately I never got to see the lab completed, but it did introduce me to Usability as a sub-domain of Software Design. Even in 1995 the importance of Software Usability and Interaction Design, and their significant impact on the successful development and adoption of software, was well understood, though the Laws themselves had not yet fully emerged.

 

In the mid 2000s, while at Microsoft,  I was fortunate enough to interact with some of the brilliant User Experience folks in the Windows Product Group and Microsoft Research. They were on the very bleeding edge of User Interface and Experience Design. Despite all of the UI innovation that happens in these teams, Microsoft has to be conservative in its adoption of these UI innovations because of the very broad use of their client operating systems; I saw many mind-blowing UI prototypes during that time, and I have seen a small percentage of the elements of those prototypes show up in shipping Microsoft products. Working with these teams gave me some idea of just how far the User Experience Design discipline had come in a relatively short time; Usability had gone from being a Craft to a fully-fledged Science, with its own sub-disciplines, laws and formalisms.

 

The Web has had a massive impact on our understanding of what makes a user interface usable and understandable. Because of their rapid mutation and evolution in response to users explicit and tacit feedback, web pages are to Usability Engineers what fruit flies are to Geneticists. User interface and experience design on the web has been highly experimental because of the breadth of its accessibility and the relative simplicity of web technologies. It has brought about a “democratization of [UI] design”. I am continually amazed at the brilliant UI that I see online.    

 

A couple of years back I read a great book that further formalised the way I approach User Interface design. It is called “Super Crunchers: Why Thinking-by-Numbers Is the New Way to Be Smart” . It obviously is not a UI Design book; It is actually about the hidden knowledge in the Exabytes of digital data that we generate, which can be revealed by applying various statistical analysis techniques. There is a great chapter about the use of randomized testing to improve the usability of web sites. The study documented in the book suggests that there are in fact no Laws of Usability, and that there is no such a thing as a user interface that will be usable and comprehensible by all users. It further suggests that the key to a usable interface is adaptability and customizability.

 

Ideally a user interface should adapt or be adaptable to each individual user. A user should not have to learn new visual and interaction metaphors to get a task done. “Fortunately”, the software industry has forced users to learn a lot of metaphors, e.g The Desktop, The Hyperlink, The Tree View, etc., which can now be used relatively freely, though we should certainly not feel obliged to use them. If possible a user interface should simply adapt without requiring that the user explicitly set a number of configuration options. At a minimum a user interface should give the user a simple mechanism for customizing the user interface to suit their preferences or preferred interaction “style”.

 

An example of an excellent user interface is Microsoft’s Visual Studio 2010. For a product with so many features and capabilities, I find it incredibly easy to use and fully configure to my personal preferences. Obviously this is not a general purpose product, but for those who use it, i.e. software geeks, it is easy to understand and is highly usable (except for the occasional performance issue). I have used Eclipse in anger (pun entirely intended) over the course of the last decade and it offers a good Usability counterpoint to Visual Studio 2010. 

 

So one would image, given what we already know about designing easily understood and usable user interfaces and experiences, that all modern user interfaces would be highly usable. Sadly that is not the case; I still see product after product, solution after solution, and web site after web site, designed so poorly that only a Visual Basic programmer from 1993 could love them! There really isn’t any excuse.

Good Software Design is Aesthetic

Ten Principles of Good Design Redux | Part 3

I usually get rather testy when software developers insist that Software Development and Architecture are more Art than Science. The testiness is usually exacerbated by the poor quality of the code that is generally written by these very same software developers. If you need to do a depth-first traversal of a binary tree it is extremely unlikely that you will need to invent a new algorithm for doing so. Somewhere in the Noosphere you will undoubtedly find an exhaustive academic study on optimal algorithms for tree-traversal. I have lost track of the times that I have had to scold developers for attempting to implement their own crypto or thread-synchronization primitives.

Is traditional Meatspace Architecture an Art or a Science? It is unarguably a creative discipline, but it is fundamentally underpinned by the material sciences and psychology*. Buildings that do not respect the laws of physics don’t stand very long, regardless of how beautiful they are. Meatspace Architecture is a creative process that is governed by a set of predominantly invariant constraints. It is true that new materials are constantly being invented, which alter those constraints, but I don’t imagine gravity is going away any time soon (until we start living in space anyway). I believe that Software Architecture (in the small and large) is no different from its Meatspace namesake.

So now that I have pooh-poohed Software Development and Architecture as Art I am going to do some legerdemain, some prestidigitation, and apparently contradict myself. Great Software Architecture is Art! A great software design, either in the small or large, can, and must in my opinion, be a thing of beauty. As a Software Architect you must be able to show a design to a colleague and have them say “Wow, that is awesome!”; “awesome” being equivalent to “beautiful” in High-Geek.  

Is it not true that budding artists of all types spend years and years learning, and in some rare cases developing, formal techniques? Only when they have truly mastered these techniques do they go on to create truly magnificent works of art, channelling their raw creativity and talent through a set of deeply inculcated techniques. Yes, yes, I know there are some great works of art painted by cats, but they are the exceptions rather than the rule.

And there is great hidden value in creating aesthetically pleasing software designs. The human brain is genetically tuned to recognise certain levels of complexity and other naturally occurring patterns as either aesthetically pleasing or not depending on their utility. By ensuring that your designs are aesthetically pleasing you are tapping into a pattern recognition system 2.5 million years in the making. An aesthetically pleasing software design is going to be more understandable, more enjoyable to implement and maintain, and most definitely easier to sell to a client or your management.

Quod erat demonstrandum (I hope anyway).

* Historically psychology has been considered a pseudo-science but I think folks like Steven Pinker and his ilk are dragging it kicking and screaming into the realm of “pure” Science.

Good Software Design Makes a Product Useful

Ten Principles of Good Design Redux | Part 2

“Of course software makes a product useful! Software is useful by design!” I hear you say. But is that really always the case?

At the beginning of this year I received a Blackberry smartphone from my employer. It was not one of the current-generation touchscreen devices, and I can’t speak directly to how useful those devices are, but the device that I received was not useful in the least. The combination of the form factor, the trackball input device and the very poorly designed user interface rendered the device practically useless to me. Despite the fact that this was a paid-for smartphone, with both voice and data covered, it sat in my bag unused for months. I eventually just sent it back and decided to cover my own communication costs rather than be subjected to the Blackberry user experience. I never even bothered to find out the model number, but it looked something like this.

I do know people who swear by their RIM devices, but I think RIM’s recent poor performance speaks in part to their inability to design and ship useful devices; this is the company who shipped a great tablet platform hobbled by its dependency on a Blackberry smartphone to enable its PIM features. And the sad thing is, there was a time when they were thought to be the embodiment of  “useful”; I recall getting a Blackberry pager when I first joined Microsoft Corp. in 2000. It looked like this.

It became the most useful piece of technology I owned. So what happened between then and now? The form factor of the phone that I found unusable in 2011 was the logical evolution of the pager that I found indispensible in 2000.

 

This happened.

And the iPhone reset the “useful” bar. Dieter Rams is a huge Apple fan, and so am I. I want to quote the description of Rams’ principle as it appears on Wikipedia because it so precisely describes Apple’s apparent approach to designing useful products.

“A product is bought to be used. It has to satisfy certain criteria, not only functional, but also psychological and aesthetic. Good design emphasises the usefulness of a product whilst disregarding anything that could possibly detract from it.”

When I use any feature of my iPhone I can visualize the team of designers and engineers brutally redesigning and refactoring that feature to make it as useful to the user as possible. Though RIM’s devices are functional, they were clearly not designed to be beautiful and give their users a deeply satisfying visceral experience. And though there are things about the iPhone that bug me, for example the lack of ability to “skin” the user interface, they do not detract from the overall experience, because it is just so bloody great!

As an aside, I recently saw Microsoft’s new mobile operating system, codenamed “Mango”, running in the wild. It is beautiful. That is not a word that I use to describe Microsoft products very often, but it is totally apropos in this case. Though the predominantly monochromatic tile user interface metaphor is very bold, it is offset by a number of subtle visual elements and the Metro typography, which makes the whole experience highly pleasing, usable and ultimately useful. There is definitely a Windows Phone in my future.

I think Rams’ “useful” principle is applicable to far more than just User Interface and User Experience Design; Software Architects should deeply consider the usefulness of the technology that they design and build, even if that software has no user interface to speak of. Software should ultimately always be designed with the actual living and breathing humans who are going to be affected by it, not just use it, in mind. This is the core principle of Gestalt Driven Development.

Good Software Design is Innovative

Ten Principles of Good Design Redux | Part 1

If I were to propose a software system design based on a 40-year-old model for distributed computing, i.e. a classic 2-tiered Client/Server Model, would that be a good design, even if I used current technologies, e.g. the .NET Framework 4 or the latest JDK?

The case can be made that innovation is motivated by a company’s financial and competitive umwelt. That is definitely true, but I would assert that this is an effect rather than a cause. I believe that humans are driven by an innate instinct to improve and optimize our tools to maximize our survivability as a species. That is not to say that all innovation always moves us forward; some innovation can be positively retrogressive in the context of the survivability of our species. On the whole though I do believe that innovation moves us, in the large, to a better place. And innovation in software technology is no exception.

So to answer my earlier question; there might be some very niche cases where a design based on the classic 2-tiered Client/Server Model might represent a good design, but in general I would assert that it would not. Having tested the model thoroughly over the last 40 years we have discovered its obvious, and even subtle, flaws and weaknesses and have innovated new and improved models to address those shortcomings. The case can also be made that the problems that we are trying to solve have themselves significantly evolved and mutated, and that the old models have limited to no utility in light of these transmogrifications.

One of the most glaring limitations of the classic 2-tiered Client/Server model is that software designed using this model does not age well; because of the typically high degree of coupling between the domain and presentation logic, it becomes more and more difficult to modify the application as it ages, and it makes replacement of the user interface technology almost impossible. The model seems to have a innate “drag” that accelerates the accrual of technical debt over time, inevitably leading to the technical bankruptcy of the system or application.

To be clear, I do not espouse innovation for innovation sake, or using emerging software technologies just because they are new. I do espouse keeping one lazy eye on the bleeding edge of innovation, and always considering how these emerging technologies might simplify or improve an application or system.

Ten Principles of Good Design Redux

A few weeks back I wrote a teaser about Dieter Rams’ “ten principles of good design” and mentioned that I believe these principles are very applicable to Software Architecture. I did not however elaborate on how these principles are applicable. I hope to address that with a number of posts over the next few weeks, each covering one principle. I will endeavour to not deviate too far from Mr. Rams’ original definitions, given their recursive elegance, but in some cases I will need to invoke Poetic Licence.

Gestalt-Driven Development: Memes, Alleles and Architectural Fitness

Claiming that Evolution is merely a “theory” is akin to claiming the same about Gravity. Evolution is a beautiful and elegant idea, which also happens to precisely describe and predict how living organisms change and diversify over time in Meatspace. It is also a fabulously useful metaphor outside the realm of biology, and I have appropriated it, along with Memetics as cornerstones of Gestalt-Driven Development.

Note: I have only provided definitions of new terms I have introduced or terms that I have subtly changed. For formal definitions of common terms follow the links to the appropriate entries in the Encyclopaedia Galactica.

A Software Architect typically has a number of products, technologies, approaches, patterns, practices, processes, methodologies, frameworks, libraries, algorithms and languages in their Software Architect’s “toolbox”. All of the aforementioned have a lifecycle; they are added  as new software paradigms emerge or become fashionable, refined and expanded as they mature, then used less as they go out of fashion, and finally relegated to the bottom of the toolbox or discarded entirely when they are no longer relevant. The contents of this toolbox can be thought of as Architectural Memes.

Architectural Memes that are repetitively used together form an Architectural Meme Complex or Memeplex. A simple example of a Memeplex would be C#, WF, WCF, OOP, Windows Server AppFabric, SQL Server and Visual Studio. Another would be HTML5, SVG, CSS3, DRY, JavaScript, jQuery, and AJAX.

Architectural Memes and Memeplexes will be more or less appropriate for a system given how well they meet the requirements of that system or part thereof. The measure of how well a given Architectural Meme or Memeplex meets the requirements of a system, sub-system or component is measured against the Fitness Criteria for that system, sub-system or component. The Fitness Criteria represent the requirements of the software system, expressed in such a way that the relative fitness of any given Architectural Meme or Memeplex can be quantified.

Obviously there exist overlapping or competitive Architectural Memes and Memeplexes, i.e. they can be used to address the same requirements. These are Meme Alleles. Meme Alleles are Memes or Memeplexes that can be used to meet similar requirements, and who’s Architectural Fitness will be evaluated using the same set of Fitness Criteria. An over-simplified example of Architectural Alleles would a Relational Database (RDBMS) implementation and a NoSQL database implementation given the following fitness criterion:  Enterprise-scale storage and retrieval of structured data. RDBMS products from different companies would also be Memetic Alleles in this case.

Note: A Gene and a Meme are not entirely analogous. A Meme is technically analogous to an instance of a Gene, an Allele, rather than the Gene itself. Hopefully scholars of Genetics and Memetics will forgive the minor poetic licence that I am taking with my definition of Meme Allele.

As I mentioned before Architectural Memes are not limited to technologies; they include processes and practices e.g. Waterfall and Agile, which will have better or worse fitness depending on the nature of the software being developed, management or the client’s ability/willingness to embrace the uncertainty innate in software development, the geographic and cultural distribution of the team, etc. Architectural Memes also include approaches, patterns and frameworks, e.g.  OOP, SOLID, GRASP, ESB, SOA, MVC, MVP, MVVM, BDD, FDD, DDD, TDD, TOGAF, ITIL, and many, many more wonderfully hermetic acronyms.

In Gestalt-Driven Development it is the task of the Software Architect (and the whole software engineering team) to identify the Memes and Meme Complexes that might be applicable to the system, ascertain the fitness of those Memes to derive the set of appropriate building blocks, and then compose those building blocks into a System Architecture, or Phenotype. Obviously this requires that the requirements have been transformed and refactored into one or multiple sets of Fitness Criteria. This is a shared task that starts with requirements analysis and ends with the engineering team.  

Using Gestalt-Driven Development and the metaphor outlined above a Software Architect is able to test and verify the suitability of an architecture to the requirements of the system early in the development process and continuously evolve that architecture throughout the process as requirements mutate and emerge.

Note: Obviously random mutation of code and data are not a good thing in all but the rarest cases. GDD proposes a high level approach to software development and architecture, not the use of genetic algorithms (though it does not preclude their use).

I am still in the process of formalizing GDD so I will definitely post a lot more about it in the future.

A Taoist Definition of Software Architecture

The question that I am asked most often in job interviews is “What is a Software Architect?” I am never sure whether I am being asked this question so that the interviewer can compare my answer with their own understanding of the role, or because they don’t really have a clue and would like someone to explain it to them. I assume the former, though I have sometimes suspected the latter. Since I have had to answer this question so often, one would imagine that I have cogitated a suitable job-winning answer. One would imagine rightly (though it has not always won me the job). 

So what is a Software Architect then? Here are some possible definitions:

• A Senior Senior Software Engineer.
• A Software Engineer who can reconcile the business and technical requirements of a complex software system.
• A Software Engineer who manages a team and owns a budget i.e. knows how to drive Microsoft Project.
• An accomplished Software Engineer who has grown weary of writing code (Oh, say it isn’t so!).
• A Software Engineer with an IASA, TOGAF, FEAC or similar certification.
• A Senior Software Engineer who has a SOLID GRASP of reading-edge TLA’s including OOP, SOA, AOP, UML, ALM, BDD, DDD, TDD and RUP.

Though there is some truth, and obviously a little humour, in the above definitions, I don’t think any of them hit the mark. Note that all of the above definitions contain the words “Software Engineer”; I believe strongly that Software Architects should be accomplished Software Engineers.

In my humble opinion the key attribute that differentiates the Software Architect from the Senior Software Engineer is the number of contexts that they take into consideration when designing software solutions and the processes to deliver and operate that software.

Or if you prefer a more Taoist definition:

Software Architecture is the process of designing and developing software in such a way that it will remain in harmony with the significant contexts within which it is created and runs over its entire lifetime.

Typically, when software is designed the contexts that are considered are the functional and non-functional requirements, and the financial and schedule constraints. The requirements are captured either upfront or over the course of the development process in a list of feature narratives, use cases, user stories, epics, diagrams or formal specification language artefacts; and typically only reflect the Technical and Business [Process] Contexts of the system.

The Financial Context is often expressed in the contract or project plan and is limited to an estimate of the software development cost and duration, an estimate of the infrastructure cost, and an, often limited, estimate of the operational cost of the system.

I say “limited” because the financial model often does not include how the operational, maintenance and support costs will change over the lifetime of the software. This includes how skills, and demand for resources with those skills, will change over time, e.g. Microsoft solutions that used C++ and COM half a decade ago, in the belief that .Net was a passing fad, or could not deliver the required performance, are having a hard time finding skilled developers with domain expertise to maintain and extend their code. Available skilled resources with C++ and COM skills are getting very expensive.

Outside of operating system and game development C++ is fast becoming the “new” COBOL, or more precisely what COBOL was at the turn of the century. That said, in light of some recent speculation about Windows 8 development, C# might be about to become the new C++, but I digress.

As an[other] aside, many requirements that typically make it into the non-functional category, including performance, scalability, security, privacy, maintainability, etc., clearly belong in the functional category because they all require architecture in the small; i.e you have to code for them at the lowest level; you can’t successfully bolt them on later. I am not sure I even believe in non-functional requirements anymore; if it is non-functional then it is not a requirement at all.

The context outlined above is already a lot of context. Surely given all this, one can design and develop usable, high quality software, that meets the clients requirements? Perhaps, with a little bit of luck and a team of stars; but given that the number of software development projects that still fail despite taking all this context into consideration (and all the advances that have been made in software development practices, processes, tools and technology), the odds are stacked against a successful outcome. So what contexts are not being considered that could turn failure into success?

I posit that it is the Human Context; or more accurately, the Social, Political and Cultural contexts in which the software has to exist. It is these often ignored contexts, that I actually believe are the most important; they should subsume all other thinking about how the software should be designed and developed.

Let me summarise and describe the contexts that  I believe should be considered when designing software (in priority order).

The Social Context

Software is ultimately about improving people’s lives. One should always know whose lives are going to be affected by the software, and how. This also includes how the development of the software is going to affect people’s lives. This aspect is often overlooked, even by the most enlightened software architects; the software industry is infamous for eating its own young in an effort to deliver on time, on budget, to spec.

This also goes way beyond usability, user acceptance, and project governance. This is a “people first” approach to software design and development, from the people who develop the software and the various stakeholders, to the end users, and everyone in between.

The Political Context 

How often is it that a project fails because its executive sponsor is either promoted, moves to another division, or leaves the company? Was the project initiated to primarily further the political aims of the sponsor or the corporate cabal of which she is a member? What happens if and when the executive sponsor achieves her actual goal, or looses her influence or position to a rival? Are there any stakeholders that have a vested political interest in a project failing? It may turn out that the client/customer is optimizing for a political outcome rather than the successful delivery of the solution. If this is the case one should modify ones architecture and execution plan to support their goal (assuming of course that it is not morally reprehensible!).

I have observed this enough that I now endeavour to ascertain the often subtle and hidden political driving forces on a project, and factor those into the design, development, and operation of the software. It always helps to optimize for the same thing that the [actual] client/customer is optimizing for.

Note that this is technically a sub-context of the Social Context above, but is important enough that it should be considered as a context in its own right.

The Cultural Context

This is best illustrated by an anecdote. In the late 90s I was a Development Consultant on a massive web portal project for a European telecommunications company. Despite the system’s complexity and the available budget, the project was poorly managed and there was no significant architecture for the system, and as a result the project was haemorrhaging cash. I could not understand why the client was allowing this to happen, until the executive sponsor took me aside and explained it to me; what they really required was the appearance that they were going to be first to market with a new technology. The quality of the solution, and its longevity, were mostly irrelevant. In the culture that the solution was being developed and deployed, it was the perception that was most important, not the actual product itself. Once I understood this I could optimize appropriately.

I have worked on software projects in Africa, Europe, North and South America, Australia and Asia and I have come to recognize the significant role that culture plays in the success or failure of a project. Ignore it at your own peril!

Note again that this is technically a sub-context of the Social Context above, but is important enough that it should be considered as a context in its own right.

The Financial Context

This is often a context that gets a lot of consideration, but as I have already alluded to above, it should cover the entire lifetime of the software.

The Business Context

This context represents the superset of Business Requirements. Nothing new here, but I continue to be amazed at how hard it is to extract the Business Requirements from stakeholders, regardless of whether waterfall or agile practices are being used.

The Technical Context

The usual suspects; Applications, Data and Technical Architectures (TOGAF) and the requirements they address.

Do note my comment above about functional versus non-functional requirements. 

A good software architecture will take all of the above into account. Obviously there is the risk of information overload. The number of contexts and the contents of each that need to be considered can quickly become overwhelming, and it can be very hard to filter the noise from the data. It is also easy to run out of time by fixating on a single sub-context, e.g. the minutia of the technical requirements (my favourite). That is not an argument for ignoring any particular context; it just means that one needs to have a formal process for comparing and prioritizing elements from different contexts. I will describe the process that I use in a later post.

To keep in line with the current trend in naming software development methodologies and practices I have named the approach described above “Gestalt-Driven Development”. I plan to elaborate on GDD in future posts.