On Software Architecture and Creativity

I recently became embroiled in a discussion about what exactly Software Architecture, Software Design and Software Engineering are, and if and how they are different. Despite how often these terms are used in the course of creating software, there seem to be as many definitions as there are people who use them. I have had reasonably durable definitions of these concepts for decades, so I thought I might discuss them in a post.

Firstly, I generally think that in the context of Software Development the terms Architecture and Design, as both verbs and nouns, are interchangeable, with the exception of  User Experience Design and  Interface Design which I consider different disciplines and domains entirely.

A useful generalized definition of Architecture is the process of envisioning and planning the implementation of a coherent system or systems to meet the static and dynamic constraints imparted by the broadest possible contexts in which those systems will operate over their lifetimes. Not all systems have direct users per se, but when they do Architecture should always prioritize the constraints imparted by its human users.

A foundational dichotomy in traditional Architecture seems to be the relationship between Form and Function, and which of these should be the prime mover. It would seem that the current zeitgeist has Function in the clear lead, and that Form should follow (though you might disagree after looking at the utterly magnificent buildings designed by Zaha Hadid or Frank Gehry).

My opinion is that Form is in fact just Function with a human face.      

In meatspace Architecture the human emotional, social and cultural contexts are fundamental to creating buildings that are optimized for their human occupants. Novelty, for its own sake, seems to be something that humans are neurologically hard-wired to receive neurochemical reward for. It does not require a massive amount of imagination to see how, from an evolutionary perspective, desiring new and different might be highly adaptive, particularly when balanced against a similar hard-wiring to need things to remain the same (though they might not be embodied equally in the same individual).

Humans are also significantly emotionally impacted by certain forms. Whether this is culturally informed, or by Genetic or Evolutionary Aesthetics, the apprehension of structural, non-structural, and environmental elements of buildings can and do have very significant psychological impact on the humans who live or work within or around them. For example, the unfortunate emotional impact that Le Corbusier’s architecture had on some of the people who had to live in his buildings is well documented, and one cannot help but be awed by Antoni Gaudí's Sagrada Familia.

I would assert that Form can be thought of as the sum of several Functional contexts related to Human psychology.

So how does this relate to Software Architecture? Where does Form, artistry and creativity fit in to the design of software systems?

I think it is safe to say that all human endeavors involve some component of creativity. No matter how dry and formal a domain, humans are continually finding new and often “better” ways to do things. And Software Architecture is no exception. Great software architecture does most definitely require and benefit from creativity, though I would assert that unlike meatspace Architecture, creativity for the sake of novelty is a very bad idea in most cases (that I have seen anyway). Recall that I am not including User Interface and Experience in this; I am referring to the design of software systems. And that is also not to say that a computer system, described in diagrams, algorithms or code, cannot be things of stunning beauty. But that beauty is a function of simplicity,  rather than novelty in my opinion.

Knowing when and where to be creative, and where to explicitly not be, is the mark of a great Software Architect. All too and painfully often I have observed Software Architects add novelty to a system that ultimately makes that system less stable, usable, maintainable, scalable, perfomant, future-proof and cost effective. Software Architecture is a sub-discipline of Software Engineering after all. When faced with a problem and a set of constraints on the solution, an engineer’s first action must be to see if there exists a canonical solution that holds over the same constraints. If there is, then there is no discussion; the most efficient and cost-effective solution is to use what already exists. Only in the case that the problem and its constraints are truly novel is creativity required. And that creative process must be heavily informed by existing bodies of work that relate to solutions to similar problems.

I like to think that I am an artist, so I fully acknowledge the value of and need for creativity, but we need to endeavor to deploy it judiciously in the design of software systems.  We should recognize and reward Software Architects and Engineers for how much code lands up NOT being written!

Applied Architecture Patterns on the Microsoft Platform, Second Edition

The book that I have been working on for the last year, with Andre Dovgal and Dmitri Olechko, has been published and is now available for purchase on Amazon.   Applied Architecture Patterns on the Microsoft Platform, Second Edition, published by Packt Publishing, builds upon the framework that was presented in the first edition and includes new chapters on recent Microsoft technologies, including .NET 4.5, SQL Server and SharePoint.

The book is aimed at software architects, developers and consultants working with the Microsoft Platform, who want to extend their knowledge of the technologies that make up said platform, and learn how those technologies can be composed into complex systems.

I decided to contribute to this book because of how useful I found the first edition. Hopefully this updated version will be as useful to those who read it.

Why Software Development is like Ironing a Thneed

 “I’m being quite useful. This thing is a Thneed.
A Thneed's a Fine-Something-That-All-People-Need!
It's a shirt. It's a sock. It's a glove. It's a hat.
But it has OTHER uses. Yes, far beyond that.
You can use it for carpets. For pillows! For sheets!
Or curtains! Or covers for bicycle seats!"

                                         – from “The Lorax” by Dr. Seuss

During my time in the military two decades ago I become highly skilled at ironing. And not only shirts and pants, but beds, hats, sheets and other items one would not normally consider “ironable”. I continue to this day to do my own ironing. I actually find it rather therapeutic.

Since having children I have also become reacquainted with the works of Theodor Geisel, more affectionately known as Dr. Seuss. It struck me the other day while ironing a particularly pesky shirt that software development is very much like ironing a Thneed.

So let’s make some assumptions about a Thneed based on the description above. Some poetic licence and imagination will be required.

1. Nobody is entirely sure what a Thneed is, not even the manufacturer.
2. When customers buy a Thneed they have only a vague idea of what they need it for and how it is going to make their lives better.
3. No two Thneed’s are exactly alike; they are the snowflakes of garments.
4. Thneed producers create new and improved Thneed’s all the time.
5. A Thneed is too big and awkwardly shaped to fit on your ironing board and Dr. Seuss makes no mention of a Thneed-press.
6. It is hard, if not impossible, to estimate how long it will take you or a highly trained team of Thneed-ironers to iron a Thneed, with any level of accuracy or confidence in your estimate.
7. The market for Thneed-ironing accessories is confusing in its profusion, pace and super-competitiveness.
8. Thneed-ironing methodologies were derived from shirt and pants ironing methodologies, but they are actually poorly suited.
9. A Thneed is like most other garments in that it occasionally requires ironing.
10. Thneed manufactures’ own Thneed’s are usually the worst ironed.

So based on the assumptions above, how does one go about ironing this Thneed thing? Well the trick is known by every person who has ever had to iron a shirt.

Manipulate the garment so that a small piece of it is flat on the ironing board and then iron that piece. Then get another piece flat and iron that piece, and so on and so on, until you have ironed the entire garment. If you are foolish enough to try to iron large sections of the garment at the beginning, you will become very frustrated and will probably run out of steam before you are done. You have to “divide and conquer” when it comes to ironing a Thneed; that is the winning strategy.

I don’t think that I need to actually spell out why this is like software development; if you have done any software development in your life you will know that I am right (or mostly so).

Maybe someday I will post “Why Architects are like the Lorax, and Users like the Once-ler”. On second thoughts maybe I won’t.

Happy Ironing!

Architecting Simplicity

I am amazed at the plethora of products and technologies that are required to deliver a best-of-breed, leading-edge, enterprise-scale, line-of-business software system.

As an example, a system that I am currently working on has an architecture that uses the following products and technologies:

• jQuery,  and a number of jQuery plugins
• ASP.NET MVC 3.0
• Internet Information Server (IIS) 7.5
• Enterprise Library 5.0
• Unity Application Block
• AutoMapper
• NHibernate
• SQL Server (SSIS, SSRS & SSAS)
• Windows Server AppFabric
• Windows Communication Foundation (WCF)
• Windows Workflow Foundation (WF)
• Windows Identity Foundation (WIF)
• MOQ mocking library
• EMC Documentum

None of the aforementioned technologies are being used gratuitously; the architecture that aggregates all of the above is necessarily complex, given the requirements. A senior developer working on this project needs to understand all of these technologies and will be writing “code” in HTML, CSS, JavaScript, XML, C# and TSQL on a daily basis. That’s a lot of tech to wrap one’s head around. And that does not include understanding the domain “problems” that the system needs to solve, which are typically complex in their own right. And this amount of technology is not atypical for enterprise line-of-business application development.

Does software that solves complex problems really need to have so many moving parts? Isn’t Simplicity one of the core tenets of great software design? 

I recently had the opportunity to chat with Rob Boyes, a Technical Director at airG, about the technologies that they are using for their latest product and service offerings. airG is a leading mobile social entertainment provider based in Vancouver, and they have millions of users from across the planet using their software. Rob told me that, though in the past they have used the LAMP stack for their backend platform, they are now using node.js and mongoDB. Though I knew of the existence of mongoDB, I had to admit to Rob that I had not heard of node.js. Since I love nothing more than tinkering with new software technology, this conversation motivated me to do a little hands-on research into these technologies, and I have to say that I have been super-impressed; these two technologies are, in a word, “awesome”! And much of that awesomeness derives from their elegant simplicity.   

node.js, or just “node”, is a server-side JavaScript runtime based on Google’s V8 JavaScript Engine; the same JavaScript engine that is in Google Chrome. It includes built-in HTTP support (though it is not limited to the HTTP protocol for network IO).

Here is a very simple example of node JavaScript code:

var http = require('http');
http.createServer(function (request, response) {
   response.writeHead("Content-Type", "text/html");
   response.end('<html><body><h1>The Barbarian Programmer</h1></body></html>');
}).listen(8000, "127.0.0.1");

Obviously being able to write JavaScript on the server gives web applications elegant symmetry, but node’s execution model is also very simple; the runtime, which will run on just about any modern operating system, runs user code on a single thread (though the runtime itself is multi-threaded). Request processing is non-blocking and is based on an asynchronous event/call-back model, so the entire server is super-scalable, and developers do not need to concern themselves with pernicious thread synchronization issues. There is also nothing to stop you from running multiple hardware-thread-affined instances of the node runtime on a single box  and using a load-balancer, probably also running on node, if you want to take advantage of multiple cores or CPUs for additional scalability and/or throughput. How to Node is a good place to learn about all things node.  

There are also a lot of additional JavaScript libraries for node, including Connect and express, which further simplify the development of web applications. There is also a great package manager available for node called npm, which makes installing these libraries dead easy. It is currently “experimental” on Windows.

mongoDB is a super-scalable “document-oriented” database, which natively supports the storage and retrieval of JSON(ish) documents. This makes it the perfect choice for use with node.js. A node.js driver is available for mongoDB, and drivers are also available for just about every platform under the sun, including the .NET Framework. mongoDB binaries are available for Windows, Linux, OS X and Solaris, and since it is open source, so is the source code. You can install the node.js drivers using npm.

When node.js and mongoDB are combined with HTML50, CSS3.0, JQuery and client-side JavaScript they represent a super-scalable, simple, consistent, and powerful web application platform. And it’s JavaScript (and JSON) all the way down! Obviously these technologies are not going to be suitable for every type of application, but I will most definitely be looking for opportunities to use them in upcoming systems that I design. Perhaps you should too.

Mad props to Rob, for reminding me that there are indeed new things under the sun.

Good Software Design is Honest

Ten Principles of Good Design Redux | Part 6

The Software Industry has been lying to itself and its customers since its emergence. At some point in the early evolution of the Software Industry someone must have noticed that the emerging development lifecycle model, originally proposed described by Winston Royce and now known as the “Waterfall” or “Big Design Up Front” model, was better suited to Aeronautical Engineering than it was to software development. Designing and building software is not the same as designing and building aircraft.

[Update (2011-10-07): Winston Royce did not propose Waterfall; he was the first to formally describe it, and used it as an example of how not to do software development. He was the “someone” I was referring to in the previous paragraph. Though this error makes my assertion that Aeronautical Engineering is better suited to the Waterfall Model seem a little arbitrary, it does not invalidate the point of the post. Mea culpa for poor fact checking.]

I am not so naïve that I would suggest that they are different merely because of the complexity of the problem domain or because of the amount of uncertainty innate to the process. Though I can only imagine that having the Laws of Physics as a significant source of constraints must reduce the uncertainty somewhat in the design and implementation of an aircraft, I would assert that they are most different in the degrees to which they are subject to the more capricious aspects of Human Nature.

When an aircraft manufacturer designs a new aircraft they typically know precisely what the majority of the requirements are from the outset, e.g. carry this many Wi-Fi-connected, grumpy adults and screaming infants from this point on the globe to this other point, using this much fuel, and oh, don’t crash. The tolerances and constraints are well understood; the laws of physics are essentially immutable, and you can only “comfortably” cram so many humans into a given space (though as someone who is over six foot and flies regularly I have to say that I am not so convinced of my last point).

The aircraft manufacturer can plug all these requirements and constraints into a model and calculate whether or not, given existing and emerging technology, they can feasibly build the aircraft, or if and where the invention of new technology is going to be required. Much of the problem domain is known and is relatively constant, i.e. the Laws of Physics and the current state of the Material Sciences. They also know what they don’t know, and they can mostly quantify the risk\cost associated with that uncertainty thanks to good historical data. A lifecycle model that is dominated by a discrete design phase is clearly effective in bringing a new aircraft to market, though obviously prototyping and innovation are also required during this initial phase. The cost associated with this protracted design phase is accepted as necessary by the, now mature, aviation industry, probably because the cost of failure is so high.

So why is Software development different? One of the most significant differences between Aeronautical and Software Engineering is that users’ of software products and systems are very rarely able to articulate detailed requirements at the outset. The constraints and requirements have to be extracted out of the minds of the intended users, in the best case, or out of the mind of one or more analysts who thinks they understand the users’ requirements, in the worst. And though the Laws of Physics are at play in the hardware that the software runs on, they almost never have to be considered as constraints in the design of the software. Typically the requirements gathering process is a bootstrapping exercise that continues well into the actual development of the system. Users have to have usable examples of what they don’t want to lead them to understand what they actually want, and the engineers have to attempt to solve some of the known hard technical problems in order to reveal the initially-unknown, usually harder, ones.

And it is impossible to estimate how long it is going to take to extract the real requirements and then develop the software to meet those requirements, at the beginning of the project. That is not to say that a waterfall model, that included exhaustive prototyping during the design phase, would not work for software, but it would require that everyone acknowledge that the requirements gathering phase would need to be completed before a fixed-cost time and effort estimate of the development could be provided, and that the duration of that initial phase could only be roughly estimated. There are simply more unknown unknowns in Software Development than in classical Aeronautical Engineering. I say “classical” because modern aircraft probably require as much Software Engineering as they do Aeronautical Engineering.

Another difference is that because so much of the software that powered the growth of the industry was developed by under- or un-paid geeks, it has created a mass hallucination about how much [paid for] time and effort it actually takes to develop high-quality software. Everyone has now come to accept this hallucination despite the number of software projects it has caused to run over time and budget, or to fail completely.

But thankfully all of this is changing; Agile software development practices are an attempt to address this innate dishonesty, and acknowledge that we as software developers simply don’t know what we don’t know. And though Agile has become mainstream, there are still those who doggedly cling to the old dishonest and delusional ways.

It is every software architect’s and developer’s responsibility to promote and champion Agile as an honest approach to the design and development of great software, particularly in the face of grumblings from old-school project managers and customers who want the illusion of fixed risk.

Note: I could write an entire book on the topic above, and was well on the way to doing so before I reminded myself that this was just a blog post destined to be read by 10s of my friends. I am sure though that the above makes the point I was trying to make.

Good Software Design is as Little Design as Possible

Ten Principles of Good Design Redux | Part 10

I think it was the eminent Rico Mariani who coined the term ”OOPoholic” to describe a software engineer who is addicted to adding gratuitous indirection and complexity to their code in the name of Object Oriented Design. After doing many code and architecture reviews I now develop a speech impediment every time I hear the word “facade”.

Einstein is credited with saying "Everything should be made as simple as possible, but not simpler." I think this holds especially true for software. Design patterns were originally proposed to make designing and describing designs easier, not harder. I propose the following razor:

If using a particular design pattern makes it harder to describe the the overall design to ones grandmother then one probably shouldn’t be using it.

Note: Replacing “grandmother” with “Project Manager” or “Client” in the aforementioned, does not reduce its utility in any way.

Good Software Design is Environmentally Friendly

Ten Principles of Good Design Redux | Part 9

This is the principle that I have taken the most liberty with. Rams’ original meaning was self evident; the manufacturing processes and materials used to realise a design should be environmentally sustainable and generally “Green”. I would agree that software should be designed in such a way that it uses hardware, and thus energy, efficiently; but I think there is a much broader application for this principle.

In a recent post I wrote about Gestalt Driven Development and Gestalt Driven Architecture. I defined Software Architecture as “the process of designing and developing software in such a way that it will remain in harmony with the significant contexts within which it is created and runs over its entire lifetime.” The “significant contexts” are ostensibly the software’s environment, and it is this environment that should be the software’s BFF.

Good Software Design is Thorough Down to the Last Detail

Ten Principles of Good Design Redux | Part 8

This principle speaks to one of my pet peeves; Software Architects are first and foremost Software Engineers, and therefore need to be able to map any high level design they create to at least one feasible concrete implementation. Some people seem to believe that once you become a Software Architect that you are excused from understanding the technology all the way down to the last turtle. I have witnessed too many once-technical architects design solutions that are impractical or inappropriate, because they have lost touch with the underlying technologies.

Good Software Design is Long-Lasting

Ten Principles of Good Design Redux | Part 7

Note: This post is technically out-of-order, but if modern microprocessors can do it then so can I.

I very recently designed a system to replace a “legacy” application that has been running for over 25 years. I have no doubt that this is not unusual and that there are still millions, if not billions, of lines of code still running that are well into their 20s. One would imagine that we learned our lesson with the whole Y2K nastiness, or simply by observing just how long some code manages to stay running.

Unfortunately I have seen time and time again how ostensibly competent Software Architects tacitly design software systems for a limited life-span. And it is the clients of these architects who have to bare the costs, since by the time the software starts to display the symptoms of its accelerated decrepitude they are usually heavily locked into the software, and have to do significant surgery to make critically-needed changes. Typically these changes add significant technical debt to the system and make subsequent changes harder still.

This can all be avoided if one designs the software with the assumption that it might live for a quarter of a century or longer. Of course you may be given a specific lifespan as a formal requirement. It is up to you to decide just how much salt to take those requirements with given your understanding of the client or domain. The case might also be made that designing relatively short-lived software is a way to guarantee your own job security. That may hold true in some cases but I would assert that if you are morally so flexible then you should be in politics rather than software development.

Ten Principles of Good Design

A few years ago I learned about Dieter Rams’ ten principles of “good design” and was immediately struck by how applicable these principles are to Software Architecture. I recently watched an amazing documentary film by Gary Hustwit called “Objectified” which is about “our complex relationship with manufactured objects and, by extension, the people who design them”.

This clip from the film, in which Dieter Rams talks about his design principles, and a recent discussion I was part of concerning whether one should optimize for making a product “perfect” or for meeting committed ship dates, reminded me of how rarely products, including software, are designed using these principles. This film should be required viewing for all Software Architects.

Note: Gary Hustwit made another excellent documentary film called “Helvetica” which is a must see for anyone doing user interface design.

Gestalt-Driven Development: Memes, Alleles and Architectural Fitness

Claiming that Evolution is merely a “theory” is akin to claiming the same about Gravity. Evolution is a beautiful and elegant idea, which also happens to precisely describe and predict how living organisms change and diversify over time in Meatspace. It is also a fabulously useful metaphor outside the realm of biology, and I have appropriated it, along with Memetics as cornerstones of Gestalt-Driven Development.

Note: I have only provided definitions of new terms I have introduced or terms that I have subtly changed. For formal definitions of common terms follow the links to the appropriate entries in the Encyclopaedia Galactica.

A Software Architect typically has a number of products, technologies, approaches, patterns, practices, processes, methodologies, frameworks, libraries, algorithms and languages in their Software Architect’s “toolbox”. All of the aforementioned have a lifecycle; they are added  as new software paradigms emerge or become fashionable, refined and expanded as they mature, then used less as they go out of fashion, and finally relegated to the bottom of the toolbox or discarded entirely when they are no longer relevant. The contents of this toolbox can be thought of as Architectural Memes.

Architectural Memes that are repetitively used together form an Architectural Meme Complex or Memeplex. A simple example of a Memeplex would be C#, WF, WCF, OOP, Windows Server AppFabric, SQL Server and Visual Studio. Another would be HTML5, SVG, CSS3, DRY, JavaScript, jQuery, and AJAX.

Architectural Memes and Memeplexes will be more or less appropriate for a system given how well they meet the requirements of that system or part thereof. The measure of how well a given Architectural Meme or Memeplex meets the requirements of a system, sub-system or component is measured against the Fitness Criteria for that system, sub-system or component. The Fitness Criteria represent the requirements of the software system, expressed in such a way that the relative fitness of any given Architectural Meme or Memeplex can be quantified.

Obviously there exist overlapping or competitive Architectural Memes and Memeplexes, i.e. they can be used to address the same requirements. These are Meme Alleles. Meme Alleles are Memes or Memeplexes that can be used to meet similar requirements, and who’s Architectural Fitness will be evaluated using the same set of Fitness Criteria. An over-simplified example of Architectural Alleles would a Relational Database (RDBMS) implementation and a NoSQL database implementation given the following fitness criterion:  Enterprise-scale storage and retrieval of structured data. RDBMS products from different companies would also be Memetic Alleles in this case.

Note: A Gene and a Meme are not entirely analogous. A Meme is technically analogous to an instance of a Gene, an Allele, rather than the Gene itself. Hopefully scholars of Genetics and Memetics will forgive the minor poetic licence that I am taking with my definition of Meme Allele.

As I mentioned before Architectural Memes are not limited to technologies; they include processes and practices e.g. Waterfall and Agile, which will have better or worse fitness depending on the nature of the software being developed, management or the client’s ability/willingness to embrace the uncertainty innate in software development, the geographic and cultural distribution of the team, etc. Architectural Memes also include approaches, patterns and frameworks, e.g.  OOP, SOLID, GRASP, ESB, SOA, MVC, MVP, MVVM, BDD, FDD, DDD, TDD, TOGAF, ITIL, and many, many more wonderfully hermetic acronyms.

In Gestalt-Driven Development it is the task of the Software Architect (and the whole software engineering team) to identify the Memes and Meme Complexes that might be applicable to the system, ascertain the fitness of those Memes to derive the set of appropriate building blocks, and then compose those building blocks into a System Architecture, or Phenotype. Obviously this requires that the requirements have been transformed and refactored into one or multiple sets of Fitness Criteria. This is a shared task that starts with requirements analysis and ends with the engineering team.  

Using Gestalt-Driven Development and the metaphor outlined above a Software Architect is able to test and verify the suitability of an architecture to the requirements of the system early in the development process and continuously evolve that architecture throughout the process as requirements mutate and emerge.

Note: Obviously random mutation of code and data are not a good thing in all but the rarest cases. GDD proposes a high level approach to software development and architecture, not the use of genetic algorithms (though it does not preclude their use).

I am still in the process of formalizing GDD so I will definitely post a lot more about it in the future.

A Taoist Definition of Software Architecture

The question that I am asked most often in job interviews is “What is a Software Architect?” I am never sure whether I am being asked this question so that the interviewer can compare my answer with their own understanding of the role, or because they don’t really have a clue and would like someone to explain it to them. I assume the former, though I have sometimes suspected the latter. Since I have had to answer this question so often, one would imagine that I have cogitated a suitable job-winning answer. One would imagine rightly (though it has not always won me the job). 

So what is a Software Architect then? Here are some possible definitions:

• A Senior Senior Software Engineer.
• A Software Engineer who can reconcile the business and technical requirements of a complex software system.
• A Software Engineer who manages a team and owns a budget i.e. knows how to drive Microsoft Project.
• An accomplished Software Engineer who has grown weary of writing code (Oh, say it isn’t so!).
• A Software Engineer with an IASA, TOGAF, FEAC or similar certification.
• A Senior Software Engineer who has a SOLID GRASP of reading-edge TLA’s including OOP, SOA, AOP, UML, ALM, BDD, DDD, TDD and RUP.

Though there is some truth, and obviously a little humour, in the above definitions, I don’t think any of them hit the mark. Note that all of the above definitions contain the words “Software Engineer”; I believe strongly that Software Architects should be accomplished Software Engineers.

In my humble opinion the key attribute that differentiates the Software Architect from the Senior Software Engineer is the number of contexts that they take into consideration when designing software solutions and the processes to deliver and operate that software.

Or if you prefer a more Taoist definition:

Software Architecture is the process of designing and developing software in such a way that it will remain in harmony with the significant contexts within which it is created and runs over its entire lifetime.

Typically, when software is designed the contexts that are considered are the functional and non-functional requirements, and the financial and schedule constraints. The requirements are captured either upfront or over the course of the development process in a list of feature narratives, use cases, user stories, epics, diagrams or formal specification language artefacts; and typically only reflect the Technical and Business [Process] Contexts of the system.

The Financial Context is often expressed in the contract or project plan and is limited to an estimate of the software development cost and duration, an estimate of the infrastructure cost, and an, often limited, estimate of the operational cost of the system.

I say “limited” because the financial model often does not include how the operational, maintenance and support costs will change over the lifetime of the software. This includes how skills, and demand for resources with those skills, will change over time, e.g. Microsoft solutions that used C++ and COM half a decade ago, in the belief that .Net was a passing fad, or could not deliver the required performance, are having a hard time finding skilled developers with domain expertise to maintain and extend their code. Available skilled resources with C++ and COM skills are getting very expensive.

Outside of operating system and game development C++ is fast becoming the “new” COBOL, or more precisely what COBOL was at the turn of the century. That said, in light of some recent speculation about Windows 8 development, C# might be about to become the new C++, but I digress.

As an[other] aside, many requirements that typically make it into the non-functional category, including performance, scalability, security, privacy, maintainability, etc., clearly belong in the functional category because they all require architecture in the small; i.e you have to code for them at the lowest level; you can’t successfully bolt them on later. I am not sure I even believe in non-functional requirements anymore; if it is non-functional then it is not a requirement at all.

The context outlined above is already a lot of context. Surely given all this, one can design and develop usable, high quality software, that meets the clients requirements? Perhaps, with a little bit of luck and a team of stars; but given that the number of software development projects that still fail despite taking all this context into consideration (and all the advances that have been made in software development practices, processes, tools and technology), the odds are stacked against a successful outcome. So what contexts are not being considered that could turn failure into success?

I posit that it is the Human Context; or more accurately, the Social, Political and Cultural contexts in which the software has to exist. It is these often ignored contexts, that I actually believe are the most important; they should subsume all other thinking about how the software should be designed and developed.

Let me summarise and describe the contexts that  I believe should be considered when designing software (in priority order).

The Social Context

Software is ultimately about improving people’s lives. One should always know whose lives are going to be affected by the software, and how. This also includes how the development of the software is going to affect people’s lives. This aspect is often overlooked, even by the most enlightened software architects; the software industry is infamous for eating its own young in an effort to deliver on time, on budget, to spec.

This also goes way beyond usability, user acceptance, and project governance. This is a “people first” approach to software design and development, from the people who develop the software and the various stakeholders, to the end users, and everyone in between.

The Political Context 

How often is it that a project fails because its executive sponsor is either promoted, moves to another division, or leaves the company? Was the project initiated to primarily further the political aims of the sponsor or the corporate cabal of which she is a member? What happens if and when the executive sponsor achieves her actual goal, or looses her influence or position to a rival? Are there any stakeholders that have a vested political interest in a project failing? It may turn out that the client/customer is optimizing for a political outcome rather than the successful delivery of the solution. If this is the case one should modify ones architecture and execution plan to support their goal (assuming of course that it is not morally reprehensible!).

I have observed this enough that I now endeavour to ascertain the often subtle and hidden political driving forces on a project, and factor those into the design, development, and operation of the software. It always helps to optimize for the same thing that the [actual] client/customer is optimizing for.

Note that this is technically a sub-context of the Social Context above, but is important enough that it should be considered as a context in its own right.

The Cultural Context

This is best illustrated by an anecdote. In the late 90s I was a Development Consultant on a massive web portal project for a European telecommunications company. Despite the system’s complexity and the available budget, the project was poorly managed and there was no significant architecture for the system, and as a result the project was haemorrhaging cash. I could not understand why the client was allowing this to happen, until the executive sponsor took me aside and explained it to me; what they really required was the appearance that they were going to be first to market with a new technology. The quality of the solution, and its longevity, were mostly irrelevant. In the culture that the solution was being developed and deployed, it was the perception that was most important, not the actual product itself. Once I understood this I could optimize appropriately.

I have worked on software projects in Africa, Europe, North and South America, Australia and Asia and I have come to recognize the significant role that culture plays in the success or failure of a project. Ignore it at your own peril!

Note again that this is technically a sub-context of the Social Context above, but is important enough that it should be considered as a context in its own right.

The Financial Context

This is often a context that gets a lot of consideration, but as I have already alluded to above, it should cover the entire lifetime of the software.

The Business Context

This context represents the superset of Business Requirements. Nothing new here, but I continue to be amazed at how hard it is to extract the Business Requirements from stakeholders, regardless of whether waterfall or agile practices are being used.

The Technical Context

The usual suspects; Applications, Data and Technical Architectures (TOGAF) and the requirements they address.

Do note my comment above about functional versus non-functional requirements. 

A good software architecture will take all of the above into account. Obviously there is the risk of information overload. The number of contexts and the contents of each that need to be considered can quickly become overwhelming, and it can be very hard to filter the noise from the data. It is also easy to run out of time by fixating on a single sub-context, e.g. the minutia of the technical requirements (my favourite). That is not an argument for ignoring any particular context; it just means that one needs to have a formal process for comparing and prioritizing elements from different contexts. I will describe the process that I use in a later post.

To keep in line with the current trend in naming software development methodologies and practices I have named the approach described above “Gestalt-Driven Development”. I plan to elaborate on GDD in future posts.