Archichect's package architecture checked with Archichect

"Eat your own dog food", they say—last call for Archichect's package (or rather, namespace) rules to be checked by Archichect itself.

Here is the intended structure—it is essentially what I have shown in that first posting about Archichect, but enhanced with

• two more sets of feature packages: "Calculating", which allows for helper plugins to "compute something"; and "WebServing", for a small webserver intended for browser access.
• five helper packages: A general constraint solver for linear inequalities, the caching facility Gibraltar, support for markers ("Markers"), dependency and item matching ("Matching"), and matching of complete paths of dependency sequences ("PathMatching")
• and a sneaked-in class diagram for the main data structure of Archichect:

In Archichect's rule syntax, the package dependencies one can derive from this diagram might be written as follows:

// Nested feature packages can access their master package(e.g.CheckDeps->Transforming)
(Archichect.(Calculating|Reading|Rendering|Transforming)).** ---> \1

// Allow nested feature packages access to common classes (e.g. CheckDeps->Dependency)
(Archichect.(Calculating|Reading|Rendering|Transforming)).** ---> Archichect
Archichect.WebServing ---> Archichect

// Allow access to helper packages from everywhere
Archichect.** ---> Gibraltar
Archichect.** ---> Archichect.(ConstraintSolving|Markers|Matching|PathMatching)

However, that's not all. I know of some groups of dependencies that do not follow these high-level patterns.

The first group of them stems from some of these "utility bottoms" classes mentioned already in the previous posting. Let me state the relevant rules here—but I mark all of them as "questionable" (with the ---? declaration), so that I can distinguish them from "ok" rules.

Archichect.** ---? Archichect:Log
Archichect.** ---? Archichect:DictionaryExtensions

Also, I allow the Matching helper package to back-reference Archichect's ItemType, Item and Dependency classes, including their base classes. That's an example of the "utility bottom" explained in the last posting:

Archichect.Matching ---? Archichect:(ItemType|Item|AbstractItem|ItemSegment|Dependency|AbstractDependency)

The same helper package also contains a support class for options for setting up matchers, which is needed by quite a few transformers. For parsing the options, this class may access the utility Option class in the main package. Come to think of it, I would allow this from everywhere if some other helper package also would like to provide such a reusable option support:

Archichect.* ---? Archichect:Option*

A second set of "other" dependencies goes from Archichect's top level class Program to the constructors of some feature classes:

• Archichect's -write-dip option writes a .dip file (file with textual representation of dependencies), so Archichect's top level class Program directly references the DipWriter class.
• When Archichect is only passed a CheckDeps configuration (in addition to input files), it will do dependency checking with the CheckDeps transformer and then write the resulting violations with RuleViolationWriter. Also these two classes are, therefore, directly referenced by Program (this is, essentially, a useless behavior—saving a few trivial command line options is not the path to enhanced usability. I should remove this).
• Finally, Archichect's -http-run option creates a WebServer object.

I consider these to be features, so the following "down dependencies" are ok, even though they create cycles in the current package layout:

Archichect:Program ---> Archichect.Rendering.TextWriting:DipWriter
Archichect:Program ---> Archichect.Transforming.ViolationChecking:CheckDeps
Archichect:Program ---> Archichect.Rendering.TextWriting:RuleViolationWriter
Archichect:Program ---> Archichect.WebServing:WebServer

Is this small static architecture ok? Well, yes and no: It is certainly (I hope you agree) a tidy architecture which knows what goes where. However, while writing the rules, it has become clear to me that there are two types of classes that are referenced from many places, yet they "hide" in the top Archichect package—there should be separate packages for these:

• A few utility classes (e.g. DirectoryExtensions or the Option classes)
• The fundamental Item and Dependency class structure.

A third problem is that the Program class forms a sort of "top level spider" which needs access to more classes than others—also this should be honored with a separate package (which would contain only one class!). Some work to do—and it might have unintended consequences: A perfect example for exploring an experimental architecture—I'll try that in the next posting.

But now, let me check how far Archichect is away from its intended architecture. I run it on itself ...

... and get 2 violations. Well, that should have been zero, but I got quite close. What did I overlook? Number one is

Bad dependency detected by global rule group:
DOTNETMETHOD:Archichect.PathMatching:AbstractRegexDepthFirstPathTraverser:Archichect;0.0.0.5;:Traverse
  -- ;;1'_usesmember
  -> DOTNETMETHOD:Archichect:AbstractDependency:Archichect;0.0.0.5;:get_UsedItem
  (at C:\PT\github\Archichect\src\Archichect\PathMatching\AbstractRegexDepthFirstPathTraverser.cs/127)

Mhm—I should have considered that the PathMatching helper needs AbstractDependency, similar to the Matching helper. I could just allow this:

Archichect.Matching ---? Archichect:AbstractDependency

With moving AbstractDependency to a separate package, I would also remove the question mark, and I would be happy. Or would I? That there is only one (1!) dependency from PathMatching—a package with about 900 lines of code—to AbstractDependency hints at an opportunity: Namely, to completely get rid of it! To-do item number three (after extracting one or two utility packages and a top-level package): Decouple PathMatching from AbstractDependency.

Last, there is

DOTNETMETHOD:Archichect.Matching:DependencyMatchOptions:Archichect;0.0.0.5;:Parse'...
  -- ;;1'_declaresparameter
  -> DOTNETPARAMETER:Archichect:GlobalContext:Archichect;0.0.0.5;:globalContext'_in
  (at archichect.exe/Archichect.Matching.DependencyMatchOptions.Parse)

The GlobalContext is—well, the global context. Yes, it is passed around (too) heavily, but it should not go into helper packages. However, option parsing, right now, always needs the global context, so it would be hard to get rid of this. Maybe I should move DependencyMatchOptions to the main package, after all? Or to the proposed options helper package? This looks very much like another opportunity for exploring an "experimental architecture"! Hopefully, I can show how to do a little bit in the next posting.

Two underrated simple design patterns - "Helper Package" and "Feature Package"

After having dealt with the generic dependencies to the .Net framework and those to specific feature-related .Net or external libraries, I am finally left with Archichect's internal dependencies.^Note

A Foreword

For the roughly 4700 dependencies in my focus I want to write descriptive rules that capture my architectural decisions, so that I can then check the actual code against them. But before I do that, let me make one thing clear: Even though the last few postings used the Archichect tool for formulating and checking rules, I am not so much interested in propagating this specific tool and the notations it provides, but rather in detecting and explaining various concepts that make up this idea of a useful, yet precise language for handling software architectures.

Just like notation, tooling is something that must provide much more than only "the right functionality": At the least, it must tie in with existing infrastructures (a tool that runs only on Windows; or only with Microsoft's TFS, is not very helpful in a Unix development environment). But, moreover, it seems that it must also appeal to its users, and this typically requires the build-up of a "fan community" which needs to like the principles and the concrete implementation (including all its bugs and hiccups and inadvertent effects) and, last but not least, also itself, i.e., the community—how its members communicate among themselves about the open problems, the evolution path to follow, and how the problems and its resolution alternatives are perceived. I am, right now, not really prepared to act a great deal in these areas—and would therefore also encourage you to take over anything you might have learned from my ponderings to the toolset of your liking: Modern tools are typically quite open, with configurable models and all sorts of scripting or even plugin capabilities, so hopefully you can add all you like also there.

On another level, my "relativism" about the "right tooling" also means that in each of my postings, I want to present a concept that is of more general interest than whether Archichect has—or lacks—features supporting that concept. I hope that each text makes it clear enough where this boundary between the interesting problem or solution, on the one hand, and the workings of Archichect or any other tool, on the other hand, is located.

Those patterns ...

Ok then. After the preliminaries above, let me write in this posting about two design patterns: Namely ways to separate out a sub-package from a large package. I need this to explain the intended static architecture of Archichect (and other programs) more succinctly, so that I can then explain, in another step, some more prescriptive rules that will capture many of those dependencies that still look like "violations".

Most probably, the following patterns that describe how, and why, packages can be subdivided have already been described somewhere else. However, I did not find anything on the internet or in my (few) books, except a short paragraph on Martin Fowler's refactoring.com by Gerard M. Davison, with an additional note by Fowler himself—but both only describe it in terms of a refactoring and not as a group of patterns. So let me present my descriptions of the two patterns I have identified, using a shortened GoF-like description template.

Pattern name & classification: Helper Subpackage (or Helper Package), a structural pattern

Intent:

When a package becomes too large, split off some part into a new subsidiary package.

Motivation, forces, scenario:

A package "search" for searching in some data set contains a set of search controllers (e.g. for selecting various sets to search, various outputs for receiving search results) and also genuine algorithm classes, e.g. for field-wise equality or unequality matching of search parameters to the searched objects. Over time, the package is enhanced with more and more refined and complex algorithms, e.g. some variant of regular expressions, or fuzzy matching. The number of all classes—search controllers as well as algorithms—in the package increases to a point where it becomes hard to separate the controllers (and possibly other groups of classes) from the algorithm classes; and where it becomes hard to manage the dependencies between the controllers and the algorithms. Both these problems can be reduced by moving the algorithm classes into a subpackage, say "search.algorithms", and then enforcing and maintaining that controllers may use classes from the algorithms package, but not vice versa.

Structure & participants:

• main package
• helper package
• external API users

Instead of having a single package, move "more internal stuff" into a helper package. The pattern allows for the possibility that some of the helper classes are part of a public API: In such a case, the external API caller will access both the main and the helper package.

Consequences:

• The number of classes per package is reduced.
• Classes are sorted into named "drawers"; only the main "drawer" does not get a special qualifying name.
• If the helper classes access resources common with the main classes, a cyclic dependency from the helper package to the main package will occur (see the discussion on "practical cycles" below).
• If external API callers should not access the helper package, a Facade may encapsulate both the main and the helper classes.

Known Uses:

JDK's javax.swing.border (some more explanations are necessary here)

Related Patterns:

In the Helper Package pattern, the single or main dependency is from the main package to the helper package: "main uses helper". In the Feature Package pattern, on the other hand, the majority of the dependencies is from the feature subpackages to the main package: "the features use the (common) main algorithm base". This difference is the main rationale to distinguish the two patterns: If it is not clear why a sub-package is extracted from a main-package, both purposes might overlap, resulting in a tangle of cycles between sub-package(s) and main package which may be hard to clean up or even understand later.

So much for the first pattern. The second one is:

Pattern name & classification: Feature Subpackage (or Feature Package), a structural pattern

Intent:

When a package becomes too large, split off some feature-related parts into new packages, with one sub-package per feature.

Motivation, forces, scenario:

A package "parallelcollections" exposes multiple thread-safe collection types, e.g. bags and dictionaries. The number of common and type-specific classes in the package is so large that recognizing which class uses which other classes becomes difficult if all classes resided in the same package. Thus, a separate sub-package is provided for each collection type, e.g. "parallelcollections.bag", "parallelcollections.dictionary" and so on. The classes in each such feature package may access common algorithms in the main package, whereas the main package typically does not access the feature packages.

Structure & participants:

• main package
• feature packages
• external API users

Instead of having a single package, move stuff related to some feature into a corresponding feature package. Typically, the external API users primarily access the feature packages.
The pattern allows for the possibility that some of the classes in the main package are also part of the public API: In such a case, the external API caller will access both the feature packages and the main package.

Consequences:

• The number of classes per package is reduced.
• Classes are sorted into named "drawers"; only the supporting main "drawer" does not get a special qualifying name.
• If the main package contains references to the feature packages (e.g. for a  factory that creates various feature class instances), cyclic dependencies between the feature packages and the main package may occur (see the discussion on "practical cycles" below).
• If external API callers should not access the main package, each feature together with some part form the common API may be encapsulated behind a Facade.

Known Uses:

JDK's javax.swing.table, .Net's System.Collections.Generic (also this needs elaboration...)

Related Patterns:

Helper Subpackage; see there for the main difference between Feature and Helper Package.

These two patterns are certainly not rocket science. But I think it's worthwhile to point out the fundamental difference between the "helper" and the "feature" approach. The main reason why I think it's useful to have these two concepts is described above in the "Related Patterns" paragraph of the "Helper Package" pattern: If classes were moved to a single sub-package for both reasons at the same time—factor out "a common helper" and also "separate features"—, you will probably get a considerable volume of both "up" and "down" dependencies, which tightly couple the packages together. Thus, important additional benefits of separating out the package, like improved testability, are more or less lost. To prevent this, deciding on exactly one of the two patterns seems useful.

Practical cycles

The desire for cyclefreeness nonwithstanding, I think (and see in examples) that, for pragmatic reasons, a few cyclic dependencies might not be that harmful. I would call such cases "shackled", as in "shackled helper package" and "shackled feature package": In both cases, one is not really free to use the referenced package on its own; but as the package came from a larger one anyway, and its primary cause was not decoupling, but organizing a large set, this is not a disadvantage a priori. Still, you have to deal with these cycles, which will make testing more difficult in parts, and reuse on the whole. Here is a discussion what you can do with the cycles of a "shackled helper package"; for "shackled feature packages", similar considerations can be made:

a) You backtrack, don't extract the subpackage and continue to live with the monolith. Of course, you "got rid" of the cycle only on the face of it, because the class dependencies have not changed a bit—but it might look good to your management or even your architect. This is most probably not acceptable—therefore, continue with option b).

b) You use some standard method for cycle elimination, e.g., interface extraction and maybe an abstract factory.
Pros: No cycles.
Cons: Added complexity that does not seem to have any benefit - after all, you just extracted the package to organize your code better, not to introduce new abstractions. If this is not acceptable, go to c).

c) You analyze the original package to find out whether you can split it to remove the cycle. Quite often, classes in a package have an internal hierarchy akin to the following:

Typically, the "worker part" is the heavy-weight part, which you (completely or partly) want to extract into the new package. But this creates that ominous cycle:

However, the diagram hints at what can be done to remove the cycle: Split the old package into a "lower utility" and an "upper facade" part, and live happily ever thereafter:

Pros: No cycles, no added complexity on class level.
Cons: Three packages; this may be especially troublesome if both the "utility" and the "facade" parts contain APIs that have been exposed to the user, who expects these related classes to remain in a single package. If your package structure is part of the API (e.g. because you use .Net namespaces as your package structure), this will not work out. What about option d)?

d) You might be able to map classes to different packages without a change to the external API. The most flexible way is, of course, to have an explicit mapping table. With Archichect rules, you might for example write:

INTEGRATION := mainpackage:(IntegrationClass1|IntegrationClass2|IntegrationClass3)
UTIL        := mainpackage:(UtilClass1|UtilClass2|UtilClass3)

with cyclefree and clearly understandable rules

INTEGRATION        ---> mainpackage.helper
mainpackage.helper ---> UTIL

Pros: No cycles on package level, no added complexity on class level.
Cons: Alternate, specialized package mapping. If you are not happy about this, i.e., you always want (or have to) map packages to your language's publicly visible name groups (like Java packages or .Net namespaces), go to option e).

e) You live with the cycle.
After all, it's only an internal detail of exactly this module. This is not something to lose sleepless nights about.
Pros: You solved the problem in the most straightforward fashion.
Cons: ... but that cycle, oh that cycle ...

To sum it up succinctly:

• Helper package = sub-package into which the main package looks down
• Feature packages = sub-packages with features which themselves look up into the main package

With these concepts in mind, let me describe Archichect's intended package architecture, and the rules to describe it, in the next posting.

Note: To be precise, I am considering here only direct, static, namespace dependencies: In other words, I am currently not interested in

• indirect dependencies (e.g. which classes implement which interfaces indirectly),
• dynamic dependencies (e.g., which objects create or have communication channels to which other objects, and when),
• other static dependencies (e.g. .Net assembly dependencies).

One step at a time! Maybe I have some ideas about some of these dependencies later.

Broad or narrow-spectrum prescriptions - that's the question

In my last posting, I have shown that writing precise rules for dependencies requires formulating a standard set of rules that allow using all the self-evident parts of the chosen runtime environment.

However, after we did that for the Archichect tool itself, there were still more than six thousand dependencies that violate the first, simplistic set of rules that I derived from a rough overview. What are the reasons for these violations? Essentially, there are two:

• First, Archichect uses some libraries to provide its features—e.g. Mono.Cecil for reading assemblies, or System.IO for reading and writing files.
• Second, the first attempt of describing Archichect's internal architecture was too simple-minded—thus, there are many dependencies between parts of Archichect that are not captured by my first set of rules.

Here, I will concentrate on the first item, and derive an interesting and important question from it. The next posting will then deal with the second topic.

Before I start with the concrete rules, I would like to know how many of the present violations are of the first kind—i.e., referring external classes—, and how many of the second. For this, I simply add a rule

Archichect.** ---? Archichect.**

This rule allows every item inside Archichect to use any other Archichect item. Of course, I must take care to remove this rule from the final rule set afterwards. That's the reason why I chose the "question mark arrow", which will mark all matching dependencies as "questionable"—hopefully, I will remember to remove this rule later (an additional comment might also be helpful ...).

With this rule added, Archichect's output is Writing 1438 violations and 4717 questionable dependencies. Thus, we will deal in this step with around 1400 dependencies—the rest will be tackled in the next posting.

Now, what about these "violations"? Among other things, Archichect can do the following (I will explain the reasons for these features later, maybe much later—right now, assume that these features are just a "given"):

1. It can extract dependencies from .Net assemblies (".DLLs" and ".EXEs");
2. it can watch files (e.g. rule files or source files) and automatically rerun a check in a new thread when a file changes;
3. it can create some architectural diagrams;
4. of course, it accesses files and directories at many places;
5. it provides a small web server for accessing output via a browser;
6. it can write rule violations in an XML format (but right now not JSON, which is an unforgivable oversight; I will repair this soon);
7. it can load plugins from assemblies for reading, transforming, writing, and more;
8. it caches items and strings internally for more efficient memory usage.

These functionalities are located in the following classes:

1. Reading .Net assemblies: Classes in namespace Archichect.Reading.AssemblyReading;
2. file watching: Class FileWatcher in namespace Archichect;
3. diagram creation: Classes in namespace Archichect.Rendering.GraphicsRendering;
4. file and directory access: many classes need this for reading (of dependencies or configurations) and writing (of violations, logs etc.); so I allow it just for all of Archichect.
5. webserver: Classes in namespace Archichect.WebServing;
6. XML writing: Class RuleViolationWriter in namespace Archichect.Rendering.TextWriting;
7. loading plugins: Classes GlobalContext and ItemAndDependencyFactoryList in namespace Archichect;
8. Caching of items: Classes in namespace Archichect and child namespaces Archichect.Reading and Archichect.Rendering; in turn, the classes implementing the cache use System.Threading.

The following rules capture the intention that these packages are allowed to use certain namespaces from external libraries:

Archichect.Reading.AssemblyReading     ---> Mono.(Cecil.**|Collections.Generic)
        // For reading .Net assemblies, Archichect uses Mono.Cecil,
        // including some general Mono collection classes
Archichect:FileWatcher                 ---> System.ComponentModel:Component
        // .Net's FileSystemWatcher derives from Component
Archichect:FileWatcher                 ---> System.Threading
Archichect.Rendering.GraphicsRendering ---> System.Drawing.**
Archichect.**                          ---> System.IO
Archichect.WebServing                  ---> System.Net
Archichect.Rendering.TextWriting:RuleViolationWriter.** ---> System.Xml.**
Archichect(.Reading.**|.Rendering.**)? ---> Gibraltar
        // String caching I copied from the web and generalized for arbitrary objects
Gibraltar                              ---> System.Threading

And now—ta-ta-ta-taa: Running Archichect with these rules leaves us with a mere 40 violations, down from the 1438 just minutes ago! It seems I have described an interesting part of Archichect's architecture almost perfectly (but, well, I invented the tool and wrote its code, so it would be fatal if I couldn't accomplish this, wouldn't it?)

What about these last 40? A quick scan of them exposes the following two reasons:

• A somewhat widespread use of System.Reflection.MemberInfo, where its Name property is accessed. This results from code like ...GetType().Name, which I find totally ok. On the other hand, I do not want to allow use of all of System.Reflection everywhere, therefore I add the specific rule

Archichect.** ---> System.Reflection:MemberInfo::get_Name

• The GlobalContext class uses System.Threading's CancellationTokenSource and CancellationToken as well as System.Reflection's Assembly and AssemblyName classes. I could now add another two rules going from GlobalContext to ... but wait: Am I still doing architecture here, or is this going into nitty-gritty details that concern no one?

That's an interesting question: Should the prescriptive architecture be

• "narrow", i.e., allow only access from and to items where we positively need this for current features?
• Or should it be "broad", i.e., allow the use of large swaths of foreign libraries in most of the product so that future extensions and modifications can use them freely?

I have to confess that I have never managed to keep a "narrow regime" up for a long time: Developers feel or actually are constrained to solve the problems they are supposed to solve. It seems to me that dependency rules should mostly "forbid what has to be forbidden", i.e., prevent definitely problematic design decisions. But they should not restrict something just because "there might be a reason against it, if we only think hard enough". Also, developers will find ways to subvert rules that are not common-sense or simply obstructive: So, my advice is, don't be narrow, be broad.

Still, threads and reflection are not something to use lightly. So, I will restrict the use of these two packages to the cancellation and assembly related classes, respectively; but allow that they be used everywhere in Archichect.

Of course, in your environment, there might be hard and fast rules that must be followed no matter what—so much better so that one can define and check precise dependency rules with tools like Archichect (when and if it is completed).

However, these are still lacking for the more than 4000 open "violations" flagged inside Archichect. The next posting is a stepping stone to tackling them.

27000 violations - that, by all means, is a huge ticket ...

In the previous posting, I gave a first set of example rules for the Archichect tool, with the sad result that almost 27000 of the 35500 or so dependencies read in from Archichect.exe were flagged as bad. I want to find out what's wrong here. In other words, we are now looking at the descriptive architecture, or whatever Archichect considers this to be.

If you are curious how to run Archichect on itself, here is a short how-to: Put the rules in some text file, say ArchichectRules.dep (.dep is the standard extension of Archichect rule files). Then, run the following command on the command line:

Archichect.exe -read Archichect.exe -configure CheckDeps { -rule-defaultfile ArchichectRules.dep } -transform CheckDeps -write RuleViolationWriter { -newline } ArchichectViolations.txt

There are nicer ways to do this (putting the commands into an .arch file is one), but I will explain them somewhen later.

After running the command above, ArchichectViolations.txt contains all those 27-and-something-thousand bad dependencies. Let's take a look at the first (I have replaced some trailing information that we are not yet interested in with ellipses ...):

Bad dependency detected by global rule group:
DOTNETTYPE::<>f__AnonymousType0:Archichect;0.0.0.2;:...
  -- ;;1'_usesmember
  -> DOTNETITEM:System.Runtime.CompilerServices:CompilerGeneratedAttribute:...
  (at archichect.exe/.<>f__AnonymousType0)

Oh—the compiler produces things by itself! This is not an architectural artifact, and not even a design artifact, but an internal artifact of the concrete execution infrastructure—we simply do not want to see such things (typically). Before diving deeply into this complicated area, we simply write a catch-all rule:

** ---> System.Runtime.CompilerServices

Rerunning Archichect now gives us 26281 violations—that's about 800 fewer than before. Slow going. What's the next?

Bad dependency detected by global rule group:
DOTNETTYPE::<>f__AnonymousType0:Archichect;0.0.0.2;:...
  -- ;;1'_usesmember
  -> DOTNETITEM:System.Diagnostics:DebuggerDisplayAttribute:mscorlib;4.0.0.0;:.ctor
  (at archichect.exe/.<>f__AnonymousType0)

A use of System.Diagnostics:DebuggerDisplayAttribute should certainly be allowed:

** ---> System.Diagnostics

Come to think of it, we also should allow using many more things from the System libraries, like int and bool and string. At least the whole System namespace should be ok—but not its sub-namespaces: We might want to control whether our software uses System.Windows.Forms, for example. Let's therefore add all of System:

** ---> System

11207 is the new violation count—more than half of the former "violations" were simply uses of the .Net framework. What else is there? Before stumbling over all these problems one by one, let me show here all the namespace rules that capture useful and certainly allowed dependencies and which I therefore allow everywhere^{Note 1}:

// Dependencies allowed everywhere in .Net applications

** ---> System
** ---> System.Collections
** ---> System.Collections.Generic
** ---> System.Collections.Specialized
** ---> System.Diagnostics
** ---> System.Diagnostics.CodeAnalysis:ExcludeFromCodeCoverageAttribute
** ---> System.Globalization
** ---> System.Linq
** ---> System.Runtime.CompilerServices
        // CompilerGeneratedAttribute, ExtensionAttribute,
        // IteratorStateMachineAttribute, RuntimeHelpers
** ---> System.Text
** ---> System.Text.RegularExpressions
** ---> -:<>f__AnonymousType*
        // Anonymous types, e.g. in Linq expressions
** ---> -:<PrivateImplementationDetails>/*
        // the C# compiler emits such things from time to time

** ---> JetBrains.Annotations
        // If you use ReSharper's [NotNull], [CanBeNull]

(**) ---> \1
        // A namespace can use all of itself
        // (but excluding child and parent namespaces)

And instead of copy-pasting these rules everywhere, it makes sense to put them into a file e.g. named StandardRules.dep, and then include this file from other rule files (this is done with a simple + line). The ArchichectRules.dep rule file now looks as follows:

$ DOTNETITEM ---> DOTNETITEM

+ StandardRules.dep

Archichect.Reading      ---> Archichect
Archichect.Transforming ---> Archichect
Archichect.Rendering    ---> Archichect

Running Archichect a last time for this posting, we are still left with 6155 violations, which I leave for the next posting.

But let me add an important, if obvious remark: In the process above, I fell deep into the claws of the descriptive model, i.e., into "what is".
But of course, with any software, there will always be some more prescriptive rules, i.e. "what we want", or at least "what we expect". It makes very much sense to write down such rules early.

Usually, a large part of the software will follow these rules, or rules that are just a little bit different from them: And during such an "architecture reconstruction", the rules, but soon also the software, can change so that the prescriptive and the descriptive models start to converge. But how do we make sure that they do not start to diverge the next day? I'll think a little bit about this in yet another posting.

Note 1: Of course, there are situations where these "minimal dependency rules" need tuning, e.g. for portable libraries. I trust that you can adapt them yourself in such cases.

A first example of rules - introducing Archichect

It's high time for a few examples. I'll draw them from two programs:

• First, the medium-sized "Archichect" tool that I wrote for checking and exploration purposes (with substantial input some years ago from Thomas Freudenberg; and requests and ideas by colleagues at Pharmatechnik in Germany, my employer). Archichect is right now a proof-of-concept software, which means, among other things, that I change anything quite freely on a daily basis (if you nevertheless want to take a look, you can search it on GitHub and look into it).
• Second, the large flagship IXOS product of our company, which I will use later to demonstrate large-scale architectural aspects.

Let me start with a small example from the prescriptive architecture of Archichect. Here is a rough sketch of some part of its intended (therefore, prescriptive) architecture:

Obviously, this is a "type 1 sketch": It is a mixture of definitive information ("there are three subpackages called Reading, Transforming, and Rendering") and illustrative, but incomplete information ("There is a class called DependencyChecker implementing interface ITransformer, but probably there are more implementors of this interface"). There is nothing at all wrong with such diagrams—except that one must be extremely careful to draw conclusions from them: The "existential assumption" is usually ok ("All the things in the diagram will be there in reality"), but even this is sometimes risky ("oh, I just meant that as an example").

So let us write some rules for this part of Archichect that lift the information from the diagram to a "type 3 declaration". Here they are, in Archichect syntax:

$ DOTNETITEM ---> DOTNETITEM

(**)                    ---> ::\1
Archichect.Reading      ---> ::Archichect
Archichect.Transforming ---> ::Archichect
Archichect.Rendering    ---> ::Archichect

(You are not happy with the notation?—you would like to express this in code? Well, as I said, I am not too stubborn about notation, so I will show how to write the same rules with code in one of the next postings).

One can actually run these rules over Archichect itself and get a nice and very long result telling us that Archichect (in its current version) has 35505 dependencies, 26690 of which violate the rules above: "Sad", as some well-known guy would tweet. Obviously, transferring an architectural diagram to strict rules requires a little more work than just more or less faithfully copy it to text—but it's only a little more, I promise (and will show you).

Before that, however, let me explain the rules above, and the assumptions behind them, a little more.

First of all, the diagram did not spell out how the concept of a UML package is mapped to the language. Most modern languages have at least one concept of nestable groups for naming things; for example, Java has packages, and the .Net languages have namespaces. In addition to these logical nestable constructs, the physical units of runtime environments, e.g. JAR files or .NET assemblies, are typically named with filenames, which can again use a hierarchical naming concept. For example, in .Net, there are assemblies named System.dll, System.Threading.dll, System.Threading.Tasks.dll, System.Threading.Tasks.Dataflow and System.Threading.Tasks.Parallel.dll etc.

In Archichect, I chose the standard approach of mapping packages to the naming concept of the implementation language, i.e., to .Net's (and C#'s) namespaces. Thus, I would put

• Program, Item, Dependency as well as the three base interfaces into namespace Archichect;
• DotNetAssemblyReader in namespace Archichect.Reading,
• DependencyChecker in namespace Archichect.Transforming, and
• ViolationsWriter in namespace Archichect.Rendering.

The type DOTNETITEM defined by Archichect's DotNetAssemblyReader nicely defines the fields Namespace, Class, Assembly.Name, Assembly.Version, Assembly.Culture, and Member.Name, and therefore the rules above refer to the Namespace field without any further syntactical ado.

Still, an alternative architecture decision could be to distribute the classes and interfaces into multiple assemblies, and then, the rules would have to be written differently, e.g. (there are more possibilities) as follows:

$ DOTNETITEM ---> DOTNETITEM

::(**)                    ---> ::\1
::Archichect.Reading      ---> ::Archichect
::Archichect.Transforming ---> ::Archichect
::Archichect.Rendering    ---> ::Archichect

The two colons indicate that the strings are assumed to refer to the third field of DOTNETITEM, i.e., the Assembly.Name field. Assembly rules collect, at least in .Net, also important architectural information and should therefore be part of the "rule set"; but I will ignore them for the moment and continue with namespace-based rules.

Secondly, regarding the diagram, it does not say anything about any dependency restrictions inside each package. A typical implicit assumption is that, on this level of granularity, each item inside a package may use any other item in the same package. To allow such dependencies, the rule

(**) ---> \1

is added. The \1 notation here is borrowed from regular expressions (and actually, internally, the rule checking is mostly done by creating regexes from the rules and then matching the read-in dependencies against them).
This Archichect rule says that items from some namespace can use items from exactly the same namespace, but not from a child or parent namespace. We will see in a later posting that this would, in some cases, prevent the useful organization of namespaces, and therefore the actual architecture rules of Archichect are somewhat different. For the moment, we leave it at that.

Still, we should understand why our apparently so useful rules gave us so many invalid dependencies, shouldn't we? That's some stuff for another posting.

Prescriptive, descriptive and experimenal architectures - what you want, what you have, what you like

In the previous posting, I described a frugal model for architectural descriptions; and ended with the question for which purposes it can be used. This should have been followed by examples—but I decided that before that, I want to explain a very important high-level view concerning architectural descriptions.

The software we have at a concrete point of time during its evolution is often not the software we want. That is true for external attributes, i.e. features and non-functional qualities, but it is also true for the many architectural aspects of a software or system. From this alone it follows that we have to deal with two different sets of architectural descriptions:

• The prescriptive architecture is the set of rules that we want some software to follow. Typical prescriptive constraints are "the GUI models must not access the database directly", "all event handlers must be aynchronous", "there must be no cycles in the dependencies of modules of type X" and the like.
• The descriptive architecture is the set of constraints that is actually adhered to in the software. These constraints are typically much more muddy than the prescriptive ones—"the generated GUI models do not access that database, but in the startup module, a model directly reads configuration from the database, and for some plugin modules, we actually do not know whether they access the database or not" might be an honest description of some set of dependencies.

In a well-functioning architectural process,

• the relevant aspects of the prescriptive architecture are known and documented unambiguously;
• also the corresponding aspects of the descriptive architecture are routinely extracted from the software and documented;
• and both are compared to detect when the latter diverges from the former at some critical point.

Maybe it should be noted that there are two categories of reasons why the prescriptive and the descriptive architecture might differ: On the one hand, the prescriptive architecture might be stable, but for whatever reasons actual development does not follow it. On the other hand, the prescriptive architecture might change, because architecture-driving requirements change. This should only highlight that it is not necessarily someone's "fault" if the two do not match.

Are we done with the architectural process?

No, we are not: When a critical difference emerges, something has to happen to align them. This might be a change to the prescriptive architecture, or a change to the actual software (which changes the descriptive architecture), or changes to both of them. In many cases, this alignment will be painful. After all, both the reasons why the prescriptive architecture is as it is, and the reasons why the actual software is as it is are profoundly embedded in the requirements and processes and people building the software. In almost all cases I have seen or taken part, planning for the reconciliation of "what we want" and "what we have" was hard and frustrating. Typically, the consequences of such a "re-architecting effort" were, and are, not at all clear for two very important factors:

• How much would the modification cost? The potentially recursive ripple effects that one change creates could lead to a nightmare of subsequent changes, and that prospect alone often considerably reduces the possibility to get the "funding" for such a change.
• How much benefit would the modification yield? Aligning the software with some prescriptive architecture may sound great, but there are typically good (but maybe not well-understood) reasons why the software is as it is; and so "following the rules" might actually make the software worse. The same is also true in the opposite direction: Just changing the prescriptive architecture to "what we have" may result in a set of "rules" that is so large and chaotic that following them is practically impossible.

One important reason for these uncertainties is that we mostly approach such alignment tasks with only two tools:

• "Dive in": This works by direct modification of the software (where the goal is to keep the prescriptive architecture) or the prescriptive documents (when the software, or some aspects of it, should be kept, but the prescriptive architecture should change). For changes in the software, this is typically (and hopefully) done in a feature branch to shield the productive software from modifications whose adverse and potentially fatal consequences are seen only later in the modification enterprise.
  For changes to the prescriptive architecture, the same should be true—which requires versioned handling of architectural documents, including "branching" and "merging". As I understand it, current tools and notations are not well-prepared for this—I would shudder to find out what an automatic merge of two UML diagrams might produce. But maybe I am too feeble here.
  In sum, irrelevant of which side has to change, "dive in" is an expensive undertaking.
• "Panorama": This approach works using informal knowledge and notations that try to capture only the essential aspects and consequences of modification variants. From these, often shaky grounds, decisions on how to proceed are derived. Typically, these are very conservative, and often limit themselves to "pilot projects" or some "drill-downs" which are supposed to be fed into another loop of the modification process.
  In sum, "panorama" often requires many iterations to get a useful result, and is therefore also arbitrarily expensive.

Both the "dive in" and the "panorama" approaches are valuable tools. However, they seem to work only with changes of a limited size. For larger systems and changes, their "sort-of-quadratic effort" (practically try out a subset of all interaction pairs between any two components) limits their usefulness and possibilities.

Thus, there should be a third possibility, namely to simulate changes on an abstract representation of the software. I think we would like to do something like the following:

If we move all the controllers into a new package, then we could separately unit-test them. Let's do it ...
... Ah, but now we see that some controllers have dependency loops with their models, and others dont. We do not want a rule 'allow loops between models and controllers'; but cleaning up all these loops right now is no option.
... But wait, it seems that the loops are mainly on trivial controllers that do not have service dependencies; whereas the controllers on top of services are typically cycle-clean ... so let's make two groups, the 'simple controllers' and the 'service controllers'. Ok, we do it ...
... and now there are only two service controllers with loops. But could we carve out their 'looping' code into a simple controller—let's try it ...
... Ok, so we can agree on new rules for controllers: 'Simple controllers must not access services, but can have looping dependencies with their models' and 'Service controllers may access services, but must not have loops with their models'.
etc.etc.

Such an exploration into the possible changes and their consequences obviously needs to be done on a model of the architecture—a model that can be trusted, i.e., mimics the actual software; is simple, i.e. can be intuitively understood by at least the architects; can be efficiently handled by tools; and what not else (I'll have to come back to these properties later, won't I?).
To distinguish this model from the two introduced at the very beginning, I call it the (or rather, an) experimental architecture.

So, in a nice(r?) world, we end up with three architectural models of a software:

• The prescriptive architecture—what we want (or believe that we want).
• The descriptive architecture—what we have (for some interesting abstractions).
• Experimental architectures—where we try what we might want and have.

But don't we have all these right now, you might ask? After all, the first is in some documents and the minds of all architects and developers, the second is "in the software", the third is on flipcharts during discussions about architecture. Of course, that's true: But I would argue that we need to be able to move information—and this will be loads of information; remember the "telephone directory property"!—between these models, and move it reliably and quickly. That's why a common notation and tooling for all three would appear worthwhile, at least to me.

To coin another term, I will call this the "escalator property" of languages for architectural descriptions: Namely that they can be "escalated" from the descriptive (what we have) to the prescriptive (what we should have) to the experimental (what we might have). And just as an escalator can also go downwards, it should be possible to map the results of an architectural experiment easily to a prescriptive architecture, i.e., to an enhanced set of rules for the system under consideration.

Examples are now really what is needed!

Models, notations, and languages

In previous postings, I used the terms "notation" and "language" too sloppily. Here is a short explanation how I intend to use them in subsequent postings—I hope that this is in line with common usage:

• Whenever we want to work with some real world things, we need a model of it. The model is a more or less rigid (mathematical) abstraction of the object(s) under consideration (which are called the "universe"). For example, a real CPU might be modelled via an abstract processor, which only considers its assembly-level commands, but not e.g. its heat emission. A model of an SQL database might only consider tables, columns and views, but "abstract away" triggers, stored procedures and everything else the vendor might have added as a feature.
• A notation is some sort of symbols that adhere to some syntax. Many notations are linear text notations (all programming languages I know of), but there are graphical notations like UML's notation.
• A language is a combination of a model and a notation, where the notation is mapped to the model or modifications of it.

The last definition implies that one can have many languages for the same model. Here is a simple example of this: Let our model (and also universe) be expressions of integral numbers, with e.g. subtraction, multiplication, and evaluation. Three possible notations are

• parenthesized infix expressions, e.g. "(5 – 3) * (5 – 2)"
• postfix expressions, e.g. "5 3 – 5 2 – *"
• and a tree notation that shows the expression tree.

All three (and many other) notations can be mapped to the model in a way that they "compute the result" of an expression correctly.

Having a good notation is important, but getting notation right is better done by a series of experiments with real people than a conceptual process. I will, therefore, not be very stubborn about notations for architectural problems. On the other hand, I will try to find a single small representation and manipulation model for architectural problems; and then try to argue that the chosen model is sufficient and practical and, well, good.

In spite of my laissez-faire approach to notation, I do hold a few beliefs about notation that I will try to argue more or less emphatically.

The most important is that any notation must scale to large descriptions. Thus, it must be possible to describe, in a manageable and legible way, a system that consists of a "flat 1000 different parts". By "flat 1000 different parts", I mean that the notation must not force the writer and the reader to introduce any sort of abstractions solely because the notation becomes unwieldy. I call this the "telephone directory property": A useful notation must be capable of practically notating a large, boring list of slightly different things "just so".

As a special case, I will not consider any diagrammatic notations for the moment (later, I'll come back to diagrams). For almost the complete history of software engineering, people—intelligent people—have tried to come up with a graphical replacement for formal textual languages like programming languages. There is a complete theory and much practical experience with two-dimensional diagram languages—but on the whole, they have never replaced textual languages in anything but small, and often not-too critical, software systems. The reason is exactly that diagram notations do not have the "telephone directory property"—diagrams describing a 1000-part system are, for all purposes, unusable: They cannot be viewed easily (especially if they contain longer, winding line paths), cannot be printed easily, and they cannot be manipulated easily. The morale: Designing and maintaining diagrams that are not useless from the outset is very hard.

(If you think "UML", and especially "UML according to all the software architecture textbooks out there", I remind you that my focus is not the use of diagrams for informal or "semi-formal"—whatever that means—purposes. For this, many diagram notations are perfectly fine. But I consider only what I called "use case no.3", i.e., languages for describing and maintaining architectures that have a strict semantics that can be used to prove or maintain something interesting in a software system).

There are a few more aspects—important aspects— that will influence all the many parts I want to assemble for useful "rule-based architecturing", but in order to keep the suspense low, I will now immediately give away what my proposed model is: (Finite) directed graphs with labelled edges and nodes. I will not restrict this quite general model much more, except that I have to define the allowed labels. They are:

• A node is identified by a label that is a tuple of strings. In addition, a node can have informational tags, each of which consist of a name and a real number.
• An edge is only identified by the nodes at its ends. It has three counts that are called the overall count, the questionable count, and the bad count. In addition, it can also have name+number tags like nodes.
• Both nodes and edges can have an arbitrary source information that is intended to find the object from which the node or edge were derived at some time.

For "historical reasons" (we invented the basics of this model some 10 years ago), I use the following terms:

• Nodes are called items.
• Edges are called dependencies, and the two items at the end of a dependency are called the using item and the used item.
• The identifying strings of items are called values, and the non-identifying tags of items and dependencies are called markers.

I hope that these terms do confer a rough meaning of the purposes for which they are used.

But—for which purposes are they used?