Architecting the enterprise backbone

Software-architect extraordinaire Simon Brown kindly pointed me to the InfoQ article ‘Agile and Architecture Conflict‘, which summarised the views of various folks on the ‘agile vs architecture’ debate, including myself, Simon and another of my regular co-creators (co-conspirators? 🙂 ), Jan van Til, all of us looking at different aspects of the idea that agility needs a backbone in order to serve the needs of the enterprise well.

The InfoQ article included a really important question: “But how does it all tie up in the real world?” In other words, what is this ‘backbone’? How does it work? And how do we define it, build it, maintain it, use and reuse it to support enterprise agility? So from my admittedly architecture-oriented side of the fence, here are some suggestions towards a practical answer.

Step 1: It’s all about the enterprise. In effect, the backbone represents the core of the organisation’s relationship with the enterprise. Agility only makes sense if it directly supports the enterprise, and that relationship with the extended-enterprise. So to make any sense of this, we need to identify what that extended-enterprise is – in other words the deep-‘Why’ for the organisation and its relationship with others, as expressed in its business-model, system-interfaces and so on. There are quite a few ways to do this: I typically use a workshop based on the sequence Vision, Role, Mission, Goal (there’s more detail on this in various of my books, such as Doing EA and Mapping the Enterprise), but there are plenty of other options.

What we need to end up with at the end of this step is some kind of vision-descriptor (about the extended-enterprise, not solely the organisation!), and an overview of the various players and key assets, activities and resources in use throughout that extended-enterprise. That becomes our starting-point for the next step.

Step 2: What part do we play in the enterprise? For what assets, activities and resources do others come to us and our organisations, or which we need from others? In the IT part of the organisation, this should tell us the types of information of which we need to keep track. We need to take a note at this point to try to distinguish between information whose structure remains stable and is used by many different areas of the organisation and its partners – because those items are likely candidates for the ‘backbone’.

Step 3: What enterprise items are unique to this organisation, and/or to its immediate partners or competitors? For example, the national mail-provider (Australia Post, Royal Mail, USPS etc) will usually have the responsibility ‘on behalf of the nation’ to maintain an up-to-date list of addresses, locations, post-codes and even probable addressees; Google maintains huge physical data-storage capacity and unique search-algorithms; Amazon builds and maintains a vast index and repository of opinions about products. Any such ‘unique items’ are almost certain candidates for the ‘backbone’. Note that these items can be much more than data alone: for example, the armed forces in principle represent unique capabilities for the nation, not just weaponry, but communications, logistics, field-hospitals and much more besides. We may also be able to identify some of these items by reviewing business-processes and other activities: for example, a national mail-provider will also maintain street post-boxes, which represent a ‘unique enterprise asset’.

In some ways what we’re looking for here are similar to ‘core competencies‘ and the like – yet it’s also sufficiently different that we need to take care not to get too sidetracked down that route. For example, equivalents for some data-items, such as industry-wide product- or service-identifiers, may also be held by competitors, by suppliers, or even by customers. The point here is that they’re unique within the shared-enterprise – which again makes them key candidates for inclusion in the ‘backbone’.

Step 4: What enterprise items are essential to our work for and contribution to this enterprise, and will need to be shared across our organisation? For example, if we are a commercial organisation, we’re very likely to have customer-records and service- or inventory records; almost every organisation will have HR records, organisational-structure records, account-codes, and so on. The point is that these are items that will need to be shared by everyone, in a consistent way – because if it’s not described and handled consistently, it’s almost certain to cause problems. If they’re essential to the business of the organisation, and shared across the organisation, they need to be in the ‘backbone’.

Step 5: Using all of the items identified in the previous steps, define the ‘backbone’. We would typically do this through the tools and techniques of the various domain-architectures: data-architecture, applications-architecture, process-architecture, organisational-architecture, brand-architecture and so on. (Notice, by the way, that not everything needs to be in the backbone! We don’t try to record everything in ‘excruciating detail’, in classic Zachman style – instead, this applies only to the items that really are core to the organisation and enterprise as a whole.) We may well need to do a ‘clean-up’ at this point to improve consistency and reduce duplication, such as described in TOGAF and the like. Once any item is included in the backbone, it should be placed under strict governance and change-control, with an explicit item-owner. Each backbone item should also be included in an explicit glossary and thesaurus, which itself should be subject to strict governance and change-control. The scope of governance depends on the context: in some cases it may be constrained to a single business-function or business-discipline, but in others the scope may need to be organisation-wide, or even broader.

Step 6: Define the interfaces to the ‘backbone’. This is where service-oriented architectures come into their own – IT-SOA, service-based process interfaces, and so on – with explicit service-catalogues and the like. Anyone who uses the information or other items (process-functions, assets etc) will need clear rules and, in many cases, development-support, on how to use the item, and the conditions under which it may be changed. There’s nothing particularly new in this: it’s fairly straightforward service-architectures, except that it should not necessarily be constrained only to information or IT.

Step 7: Define the spectrum of governance from waterfall-control to free-form agile, and the conditions that apply to change-projects and experiments at each point along that spectrum. In other words, define a flexible form of governance across the change-space, in which, for example, ‘Shadow IT’ would be explicitly encouraged as long as it’s outside of clearly-defined bounds. Provide support to enable agile projects to identify non-negotiable constraints such as regulation or mandatory standards.

Step 8: Define governance needed to migrate new items into the backbone. This is an important part, because the core will slowly change – some things will be dropped, but new processes and information-stores and the like that have proven both useful and reusable out in the agile space will be valuable in the core. As they move more towards the backbone – i.e. become shared by more people, and those crosslinks become more business-critical – they would also move ‘upward’ in terms of strictness of governance, from free-form agile towards waterfall-control.

The end-point of all of this is that we end up with a layered structure: a core that has a set of defined content with defined interfaces that can be be accessed anywhere required, and all tightly controlled; then a mid-layer in which interfaces tend to be shared across a more limited range; and finally agile hacks and mashups that may well be used only for one business-unit or even for one short- to medium-term purpose, but which still accesses the backbone for anything that is and needs to be common across the whole organisation. Some people might see this as somewhat hierarchical, but in fact it isn’t: mashups can connect to other mashups as required, perhaps bypassing the backbone completely. The point is that the backbone interfaces are guaranteed not to break (or, if they do need to change, then anyone who might be affected by the change would know about it in advance); the mid-range interfaces are not guaranteed; whilst the agile-layer interfaces – if they exist at all – can be all but guaranteed that they will break at some point. And it doesn’t matter that they might break – that’s the whole point. In other words, whilst the backbone must usually be as ‘fail-safe’ as we can make it, and the mid-layer must ‘fail gracefully’, the agile layer is intentionally designed to be ‘safe-fail’ – we often want it to fail, so as to learn from the ‘failure’, and adapt and retry accordingly.

This also resolves some of the classic fights between ‘the IT department’ – the conventional big-IT systems – and the ‘shadow-IT’ of little Excel quick-and-dirty kludges and local databases and HTML5/JSON/NoSQL hacks. In this scheme, each has their explicit place, with their own governance, their own role: each needs the other, and hence needs to respect the other, too. And it also clarifies the role and place of upcoming technologies: cloud, for example, has a really strong place in the agile context – but is often likely to be a very bad idea for the backbone.

So that’s my understanding of all of this: a way to unify architecture and agile, linking process-management, asset-management, information-management and everything else via a systematic enterprise-scope architecture, to support agility wherever it’s needed, and in whatever form is needed.

Something to play with, anyway: comments/feedback/suggestions, anyone?