How I did not (yet) learn how to write architectural documentation

As I promised in the last posting, here are some thoughts on architectural documentation. They crept out of my mind while I read chapters 6 and 7 of Ian Gorton's "Essential Software Architecture". For such a central topic as the documentation of an architecture, the chapters are astonishingly short. To be fair, the book is not only about the documentation of software architecture, but about all of architecture. Yet, we would expect a solid foundation of how to arrive at a good documentation. Do we get this?

Essentially, chapter 6 of the book is structured like this:

• One and a half pages of introduction
• One page about what to document
• Eight pages of introduction to UML 2.0
• Half a page on having an "Architecture Documentation Template", and a extremely high-level example of one
• Finally, a one-page summary

As I and all my architects and all our analysts know UML 2.0, what remains are four pages of information.

Let's try to dissect the information in them:

1. The first two paragraphs tell the true story that both feeble and massive documentation can be, and "sometimes"(?) is, "out-of-date, inappropriate and not very useful."
2. "But there are many good reasons why we want to document our architectures, for example" so that others, "most commonly members of the design and development team" "can understand and evaluate the design"; "we [whoever we are, in contrast to the development team] can understand the design" later; others can "learn from the architecture by digesting the thinking behind the design" [I do not understand this; isn't that "evaluating the design"?]; "we can do analysis on the design, perhaps to assess its likely performance, or to generate standard metrics like coupling and cohesion."
3. But it's hard to document, and the predominant tools are Word, and Visio and PowerPoint ("along with their non-Microsoft equivalents"), and the notation is "informal 'block and arrow' diagrams". "We should be able to do better."
4. The second section, "What to Document", starts with an example that shows, in my opinion, a huge misunderstanding of what software architecture is:

   A two-tier client server application with complex business logic may actually be quite simple architecturally. It might require no more than an overall “marketeture” diagram describing the main components, and a perhaps a structural view of the major components (maybe it uses a model-view-controller architecture) and a description of the database schema, no doubt generated automatically by database tools. This level of documentation is quick to produce and routine to describe.

I'll show, a few postings down my ramblings, that all the standard books and texts on software architecture notation were, up to now, not able to provide and explain a notation that can even capture this "simple architecture". At the moment, let's just take it that some high-level diagrams seem to be the goal of an archtectural documentation. Why? Because we are told so?
5. However, the section tries to sum up the reasons for more extensive documentation: Complexity ["in the eye of the beholder"? measured somehow? agreed by a team based on gut feeling?]; longevity [again: how evaluated?]; needs of stakeholders. However, this still does not tell us "what" to document; only that it must be "more".
6. Final sentence: "It’s therefore important to think carefully about what documentation is going to be most useful within the project context". I thought and think carefully, and three of us discuss—sometimes more heated than carefully—what documentation is going to be the most useful. We are not very successful at this, I have to admit.
7. The short template, after the UML 2.0 section gets its meat from the case study in chapter 7—an interesting approach dating back to the Babylonians (teach by example, not by abstraction) which, for me, does not really work, because the key point of learning, namely how to apply it in one's own environment, is lacking.
8. The summary again tries to morally uplift us:

   Generating architecture documentation is nearly always a good idea. The trick is to spend just enough effort to produce only documentation that will be useful for the project’s various stakeholders. This takes some upfront planning and thinking. Once a documentation plan is established, team members should commit to keeping the documentation reasonably current, accurate and accessible.

   I remain sceptical and am unable to heed this advice in our team, I have to admit.
   
9. Finally, there is a summary with two main ideas: First, and I agree wholeheartedly for—right now—lack of something better, it favors UML 2.0 as the notation to use, for sketches, for "closely" modelling components and objects, and for "exact" modelling for model-driven development.
10. The other idea in the summary is to have a repository for documentation, with some sort of "automatic documentation production", for whatever purpose and with an unknown functionality.

10 items of information about how to document a software architecture. I do not want to dispute most of them. I have indicated above where I think the text is wrong (and I hope to explain this in more detailed later postings). But here, I want to find out how to proceed with our documentation: And for this, I have not gotten many clues. Why? See my next posting.

Is there something wrong with software architecture - or with us?

I am a software architect (one of a few) for a 20-million LOC business software, with currently a few thousand installations, developed and delivered in a Scrum process with monthly iterations, a development team of more than 40 developers, analysts, and testers. The software is in a reasonable state—we have no problems delivering new features, we have no problems integrating new team members, we envision major overhauls of the software (ranging from virtualization to running the hitherto GUI-driven application server as a SaaS system and integrating more DWH functionality into a database designed for a pure—and partly time-critical—OLTP load).

We do not have a valid software architecture documentation.

We have, I think, a solid software architecture: We are able to immerse new team members into our culture in a way that keeps our main architectural constraints in place—layers, tiers, synchronous and async calls, normalized and denormalized data, communication with central systems about licences and failures. The performance, the usability, the stability of the whole system has never been a fundamental problem: Standard design and implementation activities have kept everything in place more or less nicely: Adding covering indexes to critical database tables, rewriting small as well as large chunks of business objects—and their underlying database structures—or swapping out complete workflows with their GUIs, even when they were referenced from other parts of the system, was "work to be done", never a "crisis".

But we do not have a valid software architecture documentation.

And we are not happy about it, because we would like to base some upcoming decisions on both

• what actually goes on in our software (might it be that the application server actually calls, at some obscure place, some client-side software directly? or can we rely on the fact that this does not happen?)—what is commonly called the descriptive architecture; 
• and on our rules of our software architecture (did we actually agree, 10 years ago, that we will forever only access a single database behind the application server?)—what is sometimes called the prescriptive architecture.

At one time, we had a project Wiki. It worked somewhat nicely for maybe 3 or 4 year of our 12 year product history, then one of the main contributors left the company, and the other one—me—got more productive, i.e., wrote more code. The Wiki, which was designed in a somewhat hierarchical fashion (it always had a large catalogue of "TBD" documents), was more and more referenced as "don't read it, because it's uncontrollably wrong as well as right". We tried to revive it half a year ago, with let's say small success, even though we still remain hopeful—but this is a separate story.

What should we do?

I started reading about software architecture documentation, and software architecture in general. And I found the problem. But the solution will be hard, and most probably require serious out-of-the-box thinking. I will tell you about the problem in the next postings.

Let me start with what I learnt from "the books"—two of them, actually, and a host of articles I found on the web.

The first book I started to read was "Essential Software Architecture" by Ian Gorton, which was published in 2006. It is a good book—read it (it is somewhere on the web, whether legally or not I do not know), and take everything away with you that you didn't know, or saw differently before.

• I agree with everything it says in chapters 1 to 3.
• Chapter 4, a guide to middleware technologies, is already outdated—not so much because the described technologies have evolved so much, but rather because they have been superseded, mostly by the cloud and the REST revolution.
• With chapter 5, I disagree: This is a disguised waterfall process. Unfortunately, this is still the mainstream view in the software architecture literature, sometimes a little beautified by talk about agile cycles, but without real depth. Writers in the field will probably disagree. I will make my case in later postings.
• Then comes chapter 6: "Documenting a Software Architecture". This would be it. But it wasn't. And then there is a "case study" in chapter 7: Interesting—we should expect that if we do as is done there, we would be perfect, or at least very good. I'll look at this shortly—but let me go quickly over the rest of the chapters, to find whether there is something interesting in there.
• With Chapter 8, a second part is introduced which collects contributions for various topics from different authors. From my (probably limited) perspective, they are not relevant: Product lines in chapter 9 may be interesting, but are, in my humble opinion, just a special case of restructuring software in the long run. Aspect orientation, in chapter 10, has not fulfilled its promises—or it has been integrated in all sort of frameworks so seamlessly that we didn't notice that it has already arrived; anyway, it is, as of today, not relevant as a major architectural concept. Model driven architecture, in chapter 11, is equally absent from industrial practice. SOA, in chapter 12, is in some sense here: With web services, in the cloud, and then with micro-services, we have services everywhere. The "web service standards" shown on p.226, on the other hand, seem to lose ground: XML (still strong) declines against JSON; messaging and especially reliable messaging is nowhere seen, as are (distributed) transactions; on metadata, I do not dare say anything; but WSDL support for typical B2B services is sketchy at best. The semantic web, from chapter 13, has not arrived. And also chapter 14's software agents are a niche architectural style, at least in all IT systems I know.
• The fascinating future of chapter 15, at least, has arrived: But, when viewed from a decade after the publication, more in the sense that on the one hand, many concrete technologies and the architectural patterns and styles they were based on or promoted, have been defeated by completely new ones, for which the old diagrams do not really work. But on the other hand, this fascinating future has, as far as I can see, not brought much progress in the wide-spread use of distinctly architectural techniques and methods in many many software projects ...

... which brings me back to my original question: Is there something wrong with software architecture—or with us? ... and to chapters 6 and 7 of Gorton's book, about documentation of an architecture, and a case study.

Let me tackle these in my next posting.

Structure? Really?

This blog will, if I find time and motivation, explain what I think is a pragmatic approach to documenting and keeping software architecture for everyone—every developer, every software project (big and small), and of course every software architect (if you think that there are or should be such people as software architects).

The postings will be presented as a sequential, doubly-linked list that grows over time—most probably together with my understanding of what I really want to explain and say. But of course, it's a blog—so you can jump to whichever posting you like ...

It will take some time until I explain that—simple, and straightforward—approach. In the meantime, I will hopefully find time to look at various misconceptions in and around software architecture—misconceptions, of course, in my view. Others may, and will, think differently.

For example, in What is a software architecture? (from 2006), Peter Eeles claims:

If you were to ask anyone to describe "architecture" to you, nine times out of ten, they'll make some reference to structure.

and, therefore

It should not surprise you then that if you ask someone to describe the architecture of a software system [...], you'll probably be shown a diagram that shows the structural aspects of the system.

Now, I hasten to say that I find Peter Eeles's article very good as a very short, but still comprehensive introduction to the standard view on software architecture. Read it!

However, to return to the question of "structure," Peter Eeles's text does what unfortunately quite a few texts in software architecture do, namely to venture into non-software territory, and then haphazardly drawing shaky conclusions. "Structure," when used in civil engineering, is not that much related to some "structural aspect": Rather, it is simply another word for "object" or "building," in a general sense. So if people use the word "structure" when uttering their thoughts about architecture, this does not at all mean that a completely different architecture—namely that of software—should therefore or similarly or analogously be concerned with the structural aspects of a system.

Of course, software architecture is concerned with structural aspects of a software system. But this has not much to do with civil engineering and its architecture.

And on the whole, this issue is not really important on its own.

However, the same mistake—looking at other disciplines like architecture of buildings or electrical engineering; and then drawing conclusions from what they do—is also made in quite a few software architecture texts with respect to diagrams and drawings.
And there, the consequences are much more grave; my personal opinion is that we all have been brain-washed (ok: "brain-washed") to believe that software architecture inherently requires diagrams. Now, diagrams are useful—but not in the way they are useful in most other engineering disciplines; and, almost obviously if you look at them, not in the way they are mostly presented in software architecture texts.
To find out why, you should, at least as an experiment, try hard to document your software architecture once without diagrams. You'll learn something.

But this is another topic, for somewhen later.

But let me "start at the beginning", in the next posting: Why don't we have a useful software architecture documentation?