More notes on toolsets for EA

Various notes following on from my recent post ‘New toolsets for enterprise-architecture‘. Most of this is about some great interactions on the ‘EA-toolset’ themes with Phil Beauvoir (@ArchiToolkit), Peter Bakker (@mapbakery) and Gene Hughson (@GeneHughson), and also, if somewhat peripherally, with Simon Brown (@simonbrown). Yet it’s also about some other themes that have come up somewhat in parallel, that in some ways interweave with those conversations. I’ll summarise all of this under four headings:

• Semantics of connections
• Structure and search
• Model-types and object-ownership
• Support for story

Before we start on those, let’s do a quick recap on some of the key themes and concerns that underpin this whole ‘new EA-toolsets’ concept.

First, the purpose of all of this is to provide better support for how we work, as enterprise-architects and the like. More specifically, it’s about toolsets to support how we work – ‘tools’ both in the sense of frameworks to help guide our thinking, and software-based applications and repositories that support those frameworks and help with recording, cross-referencing, linking, retrieving and reporting on the information and ideas that we collect and collate during that work. (For more on this, see, for example, the posts ‘How do you think?‘ and ‘The toolset ecosystem‘, and the further posts linked to from those.)

What we also need is full support for the whole of the development-process, as classically depicted in Damien Newman’s ‘the Squiggle’:

The catch, for us, is that just about all that most of our current ‘EA’ tools will support is the ‘easy bit’ over at the far right-hand end of the Squiggle, where, in essence, the models become little more than static ‘decision-records‘. One of the key problems that arises from this over-emphasis on completions is that whilst we do have a useful ‘decision-record’ in each case, we’ll have almost certainly lost any connection with the ‘why’ or ‘how’ via which we arrived at that decision – including trade-offs and choices that might have had little to no connection with any real-world needs. This not only forces us to start all over again, every time, whenever Reality Department indicates or demands some form of future change, but – by depending too much on the seeming certainties of those former decision-records – can all too easily lead us once again down pathways that may be misleading, downright dangerous or just plain wrong. Not A Good Idea… – we urgently need something better than that.

So how do we support all of the Squiggle? – not just the tail-end of ‘Design’, but ‘Concept / Prototype’ and ‘Research’ and ideation too? Although software-tools and the like will always be limited, more providing decision-support rather than decision-making, what I’ve suggested so far is that we could perhaps usefully build around what I’d termed the Burnett Query – that for every concept or idea or ‘thing’, we should be able to derive useful answers from three related sets of questions:

• Tell me about yourself? – and why you’re the way that you are?
• Tell me what you’re associated with? – and why?
• Tell me how you have changed and are expected to change? – and why?

In that sense, frameworks and model-types such as UML, BPMN, Archimate and the like are potentially-useful ways to constrain how we describe and view various types of ‘things’ and their interrelationships and changes. But we need to remember that in essence Archimate and the like are just sets of views, associated with specific needs and choices: using those frameworks, we see things in terms of those views, but the views do not and, in most cases, cannot describe the totality of the things themselves – that distinction is, as usual, somewhat subtle, but extremely important. Hence we need our toolsets not only to be able to support just about any framework or view that we might need, yet also allow each of the ‘things’ – and, as we’ll see, the nominal links between them – to be somewhat independent of how we would view or model them within the terms of any one framework.

And we also need this toolset to support the ‘messiness’ in the earlier stages of the Squiggle – which is exactly what most of those frameworks are designed to prevent. (And the latter for good reasons, of course, because we do usually need to trim out the uncertainties as we move towards any real-world implementation. But attempting to squash uncertainty is exactly what not to do during the earlier period of inherent-uncertainty – and that’s the point that we need to emphasise here.)

Which brings us to the first topic here…

Semantics of connections

The main set of responses that I had around the previous ‘New toolsets’ post were around how to implement describing the connections between things, as can be seen in the following tweets:

• mapbakery: what about a Semantic Mediawiki in combination with RDF?
• tetradian: thanks – useful themes to follow up (though does RDF require a predefined vocabulary? – if so, may not work for this…)
• mapbakery: I’m not sure I understand the question correctly but “RDF is a framework for modeling all forms of data” (see Bergman, ‘Advantages and Myths of RDF‘)
• ArchiToolkit: RDF is the implementation of a semantic model. Think in terms of an abstract semantic model, not its representation.
• tetradian: specific-implementation vs abstract model – exactly the distinction I need – thanks Phil!
• ArchiToolkit: Semantic model is about triples. Could be RDF, OWL, Turtle. See for example Stackoverflow (on the difference between RDF and OWL) but I’m no expert!
• mapbakery: Me neither, but I’ve worked with a Semantic Mediawiki  >good test platform for Tom’s ideas…

And yes, a semantic-wiki or somesuch would be one way to make a start on this: I’ve long since thought that a wiki-type would be the best way to model a glossary and thesaurus, for examples, and in a sense everything in context is a kind of glossary-item in its own right.

But there’s a catch – a show-stopper of a catch, in fact. Current semantic-frameworks seem to be geared primarily around static perceived-relationships: and that seems especially to apply to RDF/OWL (from my somewhat-cursory reading so far). Which puts us, again, right over at the right-edge of the Squiggle, with nothing to support exploration and experimentation around concerns around inherent-uncertainty, such as requisite-fuzziness and ‘variety-weather‘ – the variability of variety itself – which is what we must be able to work with during the earlier stages of the process, over towards the left-side of the Squiggle.

We could perhaps illustrate this with a cross-map to SCAN:

(Okay, we need to note that the layout of SCAN is the opposite way round to the Squiggle – certainty on the left, versus to the right on the Squiggle – but we can live with that, I presume?)

The Squiggle says that, to arrive at a usable design, we first need to go through a period of uncertainty. I’d agree with that. Yet the deeper catch – and the point that SCAN does help us to see more easily – is that all we’ve done during the process through the Squiggle is filter out from ‘the Everything’ various useful bits of sort-of seeming-certainty, enough to actually build something that might support and (maybe) resolve some specific need. What SCAN shows us is the whole context, certainty and uncertainty blended together in recursive, fractal form – and that all of it is always there, whether or not we exclude it from view. We’ve used the Squiggle to help us filter out and select out what we need – or, in SCAN terms, move the focus steadily further and further towards the left-hand side of the frame. Yet the uncertainty itself never goes away: it’s still there, always, whether we like it or not. All we’ve done is selectively exclude out from view – which is very different from actually removing it from the real-world. If we give ourselves the illusion that everything is certain when it in fact is not, we can design ourselves into serious trouble.

And that’s especially true if we allow ourselves to settle too early into pseudo-certainty – which is precisely what tends to happen with all of those static ‘has-a’ and ‘is-a’ and suchlike associations in RDF and the existing ‘EA’-tools. Once again, it’s Not A Good Idea… By contrast, the kind of relationships we actually need to work with in the left-hand and central stages of the Squiggle are much more fluid, such as:

• “this model might use that type of server”
• “some of the options we could use are A, B, C and possibly D (if it’s available)”
• “the vendor suggests that version 2.1 should be released by that time – otherwise we might have to go with the other vendor’s option, which may need us to change these parts of our infrastructure”
• “David quite likes that option, Charley and Hazel don’t like it at all, and Andy is still sitting on the fence but won’t say why”
• “there’s a big hole right there marked ‘Don’t Know’ – we’ve no idea what to do about it as yet”

At the minimum, we need an equivalent of the MoSCoW set – Must-have, Should-have, Could-have, can-Wait. (Or, for that matter, the SCAN framing: Simple, Complicated, Ambiguous, Not-known.) For semantics, the only frame that I know (thank you Erik Proper!) that comes even somewhere close to supporting this kind of need is ORM (Object Role Modelling): perhaps we could usefully focus some of our attention there for a while?

Structure and search

In another separate set of tweets, Simon Brown talked about software-architecture in terms of what he called ‘Shneiderman’s Mantra‘ – “Overview first; zoom and filter; then details on demand”. Simon mapped this to his ‘4C’ software-architecture hierarchy-model, as follows:

And as Gene Hughson noted, the same obviously also applies to the way in which we’d look for, and at, ‘things’ and collections of ‘things’, within the toolset itself:

• GeneHughson: indeed…potentially lots of recursion, so that’s a given

Yet once again there’s a really important catch: within our kind of context, almost all ‘containers’ are virtual. Or, to be pedantic, a nominal container is itself just another kind of ‘thing’ that happens to contain links to other things – links that we’d describe as representing a ‘membership-of’ relationship for the respective referenced-thing. A ‘thing’ is also arguably a container for its attributes (though even that assertion can be questioned, as we’ll see in a moment); but beyond that, ‘containment’ is little more than a synonym for ‘included in something’s list somewhere’ – a very different concept compared to the kind of absolute ‘is-owned-by’ relationship that we’d see in, say, the classic organisational-silo structures, or the ‘Business / Application / Technology’ layering of Archimate and the like.

Anyone with a reasonable experience of enterprise-architecture should recognise straight away the problem here: if our structure mandates that all we have is things and containers, with every ‘thing’ belonging to a single container, how do we describe ‘things’ that naturally sit between containers, that link all those other things together into a (possibly)-unified whole? How do we describe how and why and when such ‘things’ move between containers? If something is ‘owned’ by a project, how does it stop being part of that project, and become part of operations instead – where it might itself be used and re-used in different configurations, each itself a kind of container? If we’re not careful, concepts of ‘containers’ in a toolset can quickly become more of a hindrance than a help…

In practice, containers represent potentially-intersecting sets that could be organised in whatever way we choose. A query-result is actually a type of container, though one that exists only for as long as we maintain that particular query – and its effective-content might vary over time, of course, as is probably true for many other kinds of containers.

We also need to be wary of arbitrary assumptions about set-membership, and, in particular, arbitrary conflations that can cause huge problems as soon as we move outside of the nominal scope within which those conflations might seem to be valid. As described back in the post ‘On layers in enterprise-architecture‘, the unquestioned-assumptions embedded in the layering in TOGAF and Archimate results in a set of arbitrary-conflations that are truly horrible:

• ‘Business’ = anything-human + anything-relational + principle-based (‘vision/values’) decisions + higher-order abstraction + intent
• ‘Application’ = anything-computational + anything-informational + ‘truth-based’ (algorithmic) decisions + virtual (lower-order ‘logical’ abstraction)
• ‘Technology’ = anything-mechanical + anything-physical + physical-law (‘rule-based’) decisions + concrete (‘physical’/tangible abstraction)

Yes, it’s just-about possible to describe a meaningful architecture for classic ‘big-IT’ with that kind of mess. But it soon becomes clear how much of a disaster-area it really is when we try to describe routine real-world concerns such as human-based ‘applications’ (such as Amazon’s ‘Mechanical Turk‘) or physical-based information-systems (kanban-board, or ‘block’ system in single-track rail-working). And what about entirely new kinds of technology, such as programmable-hardware or shape-memory materials? – we cannot get away with the TOGAF/Archimate conflations there.

“Overview first; zoom and filter; then details on demand” – that’s a really useful pattern to support search and suchlike. And in turn, virtual-containers also represent another really-useful organisational-pattern to support a framing of that overview, zoom and detail-view, as ways to conceptualise ‘pre-package’ certain routine types of queries: membership of a project, for example, or the items associated with a particular work-session, with ‘the project’ or ‘the session’ providing the frame for the overview. But the moment we think that those containers are ‘real’, that they somehow ‘own’ other items in a supposedly absolute-exclusive sense? Or, even more, that all ‘things’ must have a single owning container, in a hierarchy-tree of containers with ‘things’ as the final leaf-nodes in the tree? – well, that’s where we’re likely to find ourselves in trouble, even just within the confines of a data-repository, let alone anywhere else. Whilst developing towards a more-useful type of toolset for our work, we do need to be careful about this point – because misplaced assumptions about ‘containers’, categories, classes and ‘owners’ still seem all too common in most mainstream thinking about enterprise-architecture and the like.

Which brings us to our next topic…

Model-types and object-ownership

It’s easy enough to talk about all of these ideas – but if it’s ever to have a chance of becoming something more than mere talk, at some point we do need to work out a way to implement it for real. I’m really pleased, and really grateful, that that part of the discussion is starting to happen – or, at least, that it’s now more than just me that’s talking about it. 🙂 Hence, for example, a trio of tweets respectively from Phil Beauvoir, Peter Bakker and Gene Hughson:

• ArchiToolkit: I’m still thinking RDF/OWL. But the model can manifest in many ways depending on where it is implemented
• mapbakery: I’m trying to implement some of @tetradian’s ideas on a personal (phone) scale with NoteLynx Pro http://t.co/KTI3kZVuaj
• GeneHughson: for discussion-will add comment on “New toolsets for enterprise-architecture”

The tail-end of that last tweet from Gene (which unfortunately I’ve managed to lose…) pointed to this entity-relationship diagram:

And I’ll admit that that’s close to the kind of model I roughed out when I first started on this. Yet as I discovered the hard way – and I clearly haven’t explained well enough so far – there are two crucial points we need to address. The first is as in the section just above, the point about ‘containers’ always being virtual, not ‘real’. The second is what I’d sort-of hinted at in the tweet below, before Gene’s reply:

• tetradian: (for now, crucial point to note is that, by definition, links are between ‘things’ – they don’t belong to one ‘thing’ only)
• GeneHughson: @tetradian Right…used ‘Owner’ & ‘Target’ but definitely open to better terminology

What Gene’s shown here is a one-to-many link from Thing both to Attribute and to Link – in other words, that both Attribute and Link are contained within and ‘owned’ by Thing. (He’s also showing that Links can have their own Attributes too – with which I strongly agree.) On top of that, Links and Attributes can each be associated with one or more Profiles – the term he’s using for what I’d described as ‘editors’ in the previous ‘New Toolsets’ post.

To me, all of this is fair enough, as far it goes – and yes, it is possible to implement it this way, and make it work. Sort of – enough to use it as a start-point, anyway. Yet it doesn’t go far enough, in terms of breaking away from risks of inappropriate ‘ownership’, in several different senses – and I do need to explain those points better than I have done to date.

First point is that, in principle at least, everything is an ‘equal citizen’. Everything is a Thing – and that not only includes the more-obvious Things and container-like Things as well, but also Links, and, technically, Attributes too. For the latter, it’s probably more as an abstract ideal rather than feasible in practice: every possible Attribute, and even every possible value, for example, could technically be a Thing in its own right – but it would be impossible to implement it as such, because by definition there could or would be an infinity of them. Yet it’s certainly true that we should be able to create links to specific Attributes as much as to their respective parent Thing – and links to Links, too. And that requirement makes things a lot more complex – yet also a lot more simple, if we can think creatively enough about that complexity…

In the first attempts I’d made at this, several years ago now, I developed the idea of what I called a ‘mote‘:

A mote is tiny – right down at what we might describe as an atomic scale. It consists solely of an item-identifier, a role-identifier, one parameter, and any number of links. These links point to other motes, which in turn point to other motes, which together build up towards even bigger molecule-like clusters of motes… – well, you get the idea. It’s a nice idea, and it actually does describe what’s needed. The catch is that it just isn’t implementable – not with current data-technologies, anyway. (As a table-structure it’s actually very easy to implement – it’s just one SQL table – but the scale of recursion it would need in search, retrieval and update, and the probable complexities of polymorphic mapping for different data-types, would make it horrendously inefficient and slow.) Even purpose-built graph-databases would struggle with this. Somehow, we do need to preserve that concept of ‘everything is an equal-citizen’: but in practice, we’re going to need a different trade-off than that too-theoretical ideal.

After a lot of research and exploration and heartache and thought-experiments, the best trade-off I’ve found so far could be summarised like this:

• every Thing is an equal-citizen
• every Thing will share an identical set of mandatory parameters – in particular, an item-ID, role-ID and access-rights controls
• every Thing may carry any number (0..n) of non-editable, searchable tags
• every Thing may carry any number (0..n) of optional parameters
• every Thing may carry any number (0..n) of pointers to other Things
• every parameter will be defined from an editor-module, identified by searchable tag
• every pointer may be associated with an optional parameter defining the reason for the link

All of that is implementable quite easily in a NoSQL document-database such as MongoDB, and (I believe?) also possible, if with some serious table-juggling, to implement in SQL as well. It’s doable, anyway. But with various combinations of the above, what this gives us is as follows:

• a basic tag-Thing with no editable parameters and no links, but can be pointed to by other Things’ links
• a standard-Thing that carries its own editable-parameters
• a parameter-Thing that exists primarily to carry some parameter for another Thing, but for various reasons it’s useful to treat as a Thing in its own right – for example, a Thing that carries the content and edit-history of a wiki-page
  (an alternative implementation would be for the parameter-Thing to have one pointer that points back to its ‘parent’ Thing)
• an extended-Thing that has its own parameters (like a standard-Thing) but also has pointers to ‘external’ parameter-Things
• a link-Thing that has pointers to (usually) two other Things, and that therefore describes an identifiable association between those Things
• a container-Thing that carries any number of pointers to other Things that are conceptually ‘contained’ by it in some way

There might perhaps be a few other variations on that overall theme, but the point is that the data-structures that we need do all seem to be implementable just through that one small set of elements: mandatory-parameters, non-editable tags, editable-parameters, and pointers.

The tricky part is in how those Things are defined, and their content edited and amended – because to make it work, we have to break away from one of the most common assumptions of all in current toolset-design: that an object is defined and ‘owned’ by some predefined model-type. Instead, we shift priority from the model-type to the object-instance: a Thing is ‘owned’ by nothing other than itself.

Let’s take a practical example, and imagine we want to implement support for Archimate in this toolset. (Which we do, of course, but it may take us a while to get that far – we’d start implementation with something simpler first!) Archimate defines a bunch of object-types, and the relationship-types that are permitted between each possible pairing of objects. The usual way to do this is to define everything in a formal metamodel, starting with the object-types and their parameters, and then distinct pairings for each relationship-type permitted for that object-type. It’s a complex piece of work – and it grows almost exponentially each time we add either a new object-type or relationship-type, so we’d always want to keep both of those to a minimum. Also, and importantly, every object created from this Archimate metamodel is ‘owned’ as an ‘Archimate-only’ item: in general, it can’t be used for anything else.

(That’s the way it’s usually implemented, anyway, although some of the existing toolsets – most notably Abacus Avolution – do allow us to add and edit further parameters to objects, if usually at a class-level only.)

What we do here instead is all but turn the whole thing upside-down. There’s still the same concept of a metamodel that defines objects and relationships: but it’s done in a crucially-different way. The object-definition is used as a template, either to create a new Thing, or to specify parameters and tags that can be applied, edited and used when an existing Thing is explored in context of this specific editor. (There’s a quite close analogy here with the ‘Open with…’ functionality in Windows or OS-X, though it applies at instance-level rather than at class-level – think in terms of individual-file rather than file-type.) And linking-rules relate to tags assigned to and associated with object-types – rather than object-types themselves – which greatly simplifies how linking works, and also obviates that near-exponential expansion of complexity as new object-types or link-types are added.

Hence, for example, we might find via a search-query a Thing that was originally created via a BPMN editor to represent an Activity called ‘Accept Room-Booking’, in a process-model for a hotel system. We decide we want to represent this in Archimate as a Business-Service. So having found this Thing with all of its embedded BPMN-parameters, we open it in the Archimate editor. We tag it as a ‘Business-Service’, and for the moment, keep its name of ‘Accept Room-Booking’. But having tagged it as a Business-Service, we can now edit parameters appropriate to an Archimate Business-Service; and we can also link it to other Archimate-tagged things in terms of Archimate’s linking-rules. Those Archimate-compatible parameters and tags then become a permanent part of the data of and for that Thing. Yet the respective Thing is not ‘owned’ by Archimate, any more than it is owned by BPMN, or any other editor. We work on it with the Archimate editor (in this case), to treat it as-if it’s an Archimate object, but that’s not the same as saying that it ‘is‘ an Archimate-only object – a subtle yet crucial distinction.

In that sense, each Thing may have a data-structure that is different from that of every other Thing – in effect reflecting its own distinct edit-history, as worked-on by any number of different editors, each of which assign their own parameters and tags to that Thing as required and appropriate. Yet because the core-structure and internal framing – mandatory-IDs, tags, editable-parameters, link-pointers – is the same for every Thing, they’re still searchable and suchlike as if they do have the same data-structure. Think for a while of the power and flexibility that comes from that – and yet it can all still be maintained under whatever formal-rigour may be required by any editor that applies.

There’s quite a lot more I need to write about this – particularly about the structure and content for editor-specifications – but that’s the basic idea, anyway.

And that theme of flexibility of content and ‘ownership’ and linking for Thing-entities leads us on to the final topic for this post…

Support for story

The start-point here is a tweet from Peter Bakker with a quote from the section ‘It’s all about story’ in my post ‘Enterprise Canvas as service-viability checklist‘:

• mapbakery: “So whenever we look at an architecture, whenever we look at structure, we need also to look at, and look for, the stories”

This is a common theme in other forms of architecture too. For example, in the post ‘Two points of view on (enterprise) architecture‘, I pointed to a quote from Matthew Frederick’s 101 Things I Learned In Architecture School:

Two points of view on architecture

ARCHITECTURE IS AN EXERCISE IN TRUTH. A proper building is responsible to universal knowledge and is wholly honest in the expression of its functions and materials.

ARCHITECTURE IS AN EXERCISE IN NARRATIVE. Architecture is a vehicle for the telling of stories, a canvas for relaying societal myths, a stage for the theater of everyday life.

Or, to put it another way round, architecture is about structure, but also is about story: we need both of these in place if we’re to create an architecture that works and ‘makes sense’ in real-world practice and real-world use.

(There’s more on this in other posts here, such as ‘The enterprise is the story‘ and the NOTES series, and also the slidedecks ‘The enterprise is the story‘ and ‘Staging the story‘ on Slideshare).

Hence, as Peter says in another tweet:

• mapbakery: we need a dialogue mapping tool like DesignVue 🙂 http://t.co/y0vhcORmFA

I don’t know DesignVue, but I do know the story-builder in the Southbeach Modeller toolset – and yes, that’s something that we’re going to need. In Southbeach, it takes the embedded content of a model – the identifiers and parameters of the objects and the links between them – and builds a textual description that can be used in a report. (I believe that some of the existing ‘EA’-toolsets can also do something like that, though I don’t have any details as yet.) Although in Southbeach it’s made simpler by the fact that that toolset only allows a limited range of object-types and relations between them, it’s still something that should be doable with the data-structure I’ve described in the section above.

In Southbeach, we can generate a textual ‘story’ for a single object at a time, or the set of objects displayed in the current view. For the earlier stages of the Squiggle, though, we’d want to be able to build a story about any set of objects, that could be of nominally-disparate types, created by different editors, quite possibly at different levels of abstraction or aggregation or composition or whatever, that we choose to link together in some way.  In other words, not just a simple:

A > B > C > D > E

But something that could weave in all manner of different directions, and even loop back through itself to show how different insights have arisen from the process of connection:

A > B > C > J > Q > A > C > K > B

And so on – we choose the story, rather than having the story-structure imposed on us by the built-in rules of a single editor-specification.

Another theme to explore, I’d say – but definitely something that we’re going to need, anyway.

Best leave it at that for now: over to you for comments, perhaps?