Tackling uniqueness in enterprise-architectures

There’s a core theme that reaches right to the heart of every enterprise-architecture: what is the appropriate tradeoff between sameness versus uniqueness?

The classic Taylorist solution has been to emphasise extreme sameness: to force everything – and everyone – to be the same, because it keeps things simple, controllable and easily replicable. But we now know that it’s too simple to work well with the real complexities of the real world. In fact it only ‘works’ as long as we can maintain the delusion that it does work: and whenever it fails – which eventually it always does – there’s a tendency to collapse into complete chaos. Which is not much of a ‘solution’ at all.

Yet to focus only on uniqueness all but takes us back to a pre-industrial age where everything is custom-made, everything is different, nothing is actually designed to work with anything else, there are no possible economies of scale, and no certain means to communicate with each other – all of which would seem to be the antithesis of architecture. Which is likewise not a good idea.

Clearly there is an architectural tradeoff there: and hence we need something – some conceptual framework – to help us tackle it.

For at least the past half-decade, and probably longer – see, for example, Andrew Johnston’s 2005 article ‘Architects: Masters of Order and Unorder?‘ – enterprise-architects have turned to Kurtz & Snowden’s Cynefin framework for help on this. For many of us, the Cynefin categorisation of Simple [aka ‘Known’], Complicated [aka ‘Knowable’], Complex and Chaotic has proven extremely valuable, such as for identifying structural themes and potential problems in conceptual misalignment. One example of the latter, as Dave Snowden has also often pointed out, is the misuse of Simple-domain techniques such as Six Sigma: by definition, these depend on very high degrees of repeatability – literally millions of identical events, in Six Sigma’s case – yet there are frequent attempts to apply them in contexts that have little or no repeatability (‘Complex’ or ‘Chaotic’ respectively, in Cynefin terms), which makes no sense at all.

Beyond that basic categorisation, though, attempting to use Cynefin in enterprise-architecture can itself be problematic, particularly where we need to tackle inherent uniqueness. The explicit focus of Cynefin, and Snowden’s Cognitive Edge consultancy, is the application of techniques derived from complexity-science to inherently-complex areas such as policy. (Which, from a cross-map of Cynefin to the ISO9000 quality-standard ‘stack’, is exactly what we should expect: ‘Complex’ maps to the ISO9000 ‘Policy’ layer, in the same way that ‘Simple’ maps to ISO9000’s ‘Work Instruction’.) Yet whilst Cynefin uses the sciences to tackle complexity, what we also need in enterprise-architecture is some means to use complexity to tackle ‘chaotic’ uniqueness – which is not the same at all. Therein lie some serious problems – and some potentially-serious mistakes – if we try to re-use Cynefin concepts in contexts for which it was not designed.

I’ll admit that I’ve probably made some of those mistakes myself. Over the past couple of years I’ve written a number of articles on Cynefin on enterprise-architectures, which made a lot of practical sense to many people, but unfortunately also led to some extremely unpleasant arguments that I have no wish to revisit. What’s become clear to me over the past few months is that the beguiling simplicity of the Cynefin categorisation can blind us to the fact that although its descriptions of the Complex and Complicated domains are essentially the same as we would use for context-space mapping in enterprise-architectures, its definitions and usage of the Chaotic and Simple domains are fundamentally different to those that are needed to tackle uniqueness and sameness in architecture. It’s like comparing a cross-head screwdriver with a flat-head one: at a cursory glance they may seem to be the same, and it’s clear that they are related in the sense that they have similar functions – but in practice they’re not interchangeable, and trying to use them as such will cause a lot of frustration and possibly a lot of damage too. Not a good idea.

So I’d like here to explore what aspects of Cynefin can be used in enterprise-architectures, how and why and where it should not be used, and what we could use as an alternative in those contexts. [I perhaps need to emphasise here that this is not a critique about Cynefin itself, but solely about certain (mis-)uses of Cynefin in enterprise-architecture.]

This again will need to be quite long – apologies – but at least this time there’ll be a fair number of diagrams to break the verbiage into more manageable chunks. 🙂

What is the Cynefin framework?

The basic ‘Cynefin framework’ is a way to categorise and interpret different types of cause-effect relationships in a given context. Starting from an initial position where we cannot identify what type of cause-effect relationships exist (the domain of ‘disorder’, shown as the unlabelled grey region in the centre of the diagram below), we categorise the relationships as follows:

The right side is about order, certainty and repeatability; the left-side is about ‘unorder’, where things do not repeat in an ordered way. (Note that this early diagram from Johnston’s ‘Masters of Order and Unorder’ article uses the original labels of ‘Known’ and ‘Knowable’ for the two ‘order’ domains; for several practical reasons, explained elsewhere by Dave Snowden, these domains are nowadays labelled ‘Simple’ and ‘Complicated’ respectively.)

One important point is that Cynefin is not a simple two-axis matrix – hence the curved lines separating the domains. But it’s also important to note that the purpose of Cynefin is to describe and act on themes that apply primarily in the Complex domain – contexts in which “cause and effect are only coherent in retrospect and do not repeat”. The ways in which we would act on those themes are familiar to architects: patterns, perspectives and the use of exploratory experiments to probe for appropriate options. This contrasts with the ‘Knowable’ or Complicated domain, in which it’s assumed that if we can identify all the factors in play – the analytical or so-called ‘hard-systems thinking’ approach – it will lead us automatically to the single repeatable ‘right answer’ to any problem.

That specific contrast is extremely important in enterprise-architectures, because there’s a very common misperception – especially amongst IT-oriented folks – that ‘complexity’ is an exact synonym for ‘very complicated’. All we have to do to remove complexity, it’s said, is to break it down into smaller, simpler parts, and the complexity will go away of its own accord. The Cynefin framework makes it clear that this is a fallacy – particularly when the system includes real people in any real, non-machine-like way, but also in many, many types of ‘network-effects’ and the like. The blunt fact is that systems that are designed solely on ‘ordered’ principles – Simple and/or Complicated – will inevitably fail in any real-world context: hence the importance of something like Cynefin.

What’s different about architecture?

As I understand it, the primary purpose and use of Cynefin is in development of policy, and related programmes of action around large-scale themes involving complex issues that engage or need inputs from a large number of people. It’s been used a lot recently in the defence and anti-terrorism contexts, for example, including new ways to work with narrative-enquiry and the like.

Enterprise-architectures do indeed work on some themes that are similar to those: yet the real point is that we’re also strongly focussed on individual responses to specific, unique requirements – which is not the same as Cynefin’s focus on policy, which is more about the collective than the individual. In architectures we also make very strong use of uniqueness in the ideation process; and for IT-architectures especially, we have a strong need for ‘solutions’ that are simple enough for IT-systems to operate in real-time.

And real-time itself is another strong focus: for much of what we do, we don’t have time for analysis or experiment – and yet whatever solutions we build have to be effective, to work well in accordance with the real needs of the context. That too is different from the needs of policy: recent work by Cognitive Edge has brought complexity-based tactics much closer to real-time, but it still is not actually operating in real-time – time is important there, but it’s not the real focus.

So there are two key distinctions here:

• real-time is a key concern in enterprise-architecture, yet much less so in Cynefin
• the Chaotic and Simple domains (real-time unorder and order) seem all but undesirable in Cynefin, yet are central to enterprise-architecture

Cynefin provides a (very) good match for enterprise-architecture’s needs in the Complex and Complicated [Knowable] domains; but it does not fit well in the Chaotic and Simple [Known] domains. Using Cynefin for enterprise-architecture in the latter types of contexts would not only yield misleading results, but would also be a misuse of Cynefin itself. Not recommended.

So there’s a strong suggestion here that we do need something else that looks much the same as Cynefin in the non-time-critical contexts, and is compatible with it in those domains – yet takes a different approach as we move closer to real-time.

Dynamics in context-space

The same themes come up when we consider Cynefin’s dynamics – the pathways via which we move between conceptual-domains. To illustrate this, I’ll again use a diagram from Johnston’s ‘Masters of Order and Unorder’ article:

As shown by the orange pathways, most new ideas arise in the Chaotic domain. The dead-ends of some of these lines in the Complex domain and the continuation of others (via blue lines) into the Knowable/Complicated domain aligns well with the classic sequence of scientific development: Idea (Chaotic) to Hypothesis (Complex) to Theory (Complicated) to Law (Simple). The back-and-forth between Chaotic and Complex (path 7), Complex and Complicated (paths 4 and 5), and Complicated and Simple (path 3), are also typical – and necessary – in scientific development. Yet the only links shown between Chaotic and Simple – the key areas of concern for real-time enterprise-architectures – are failure-modes: the collapse of the over-simplistic (path 1) or the collapse of access to new ideas (path 2).

This too is exactly what we would expect in social contexts where a probably-spurious sense of certainty (Simple order) is privileged over everything else: the cycle of scientific-development dead-ends at ‘law’, preventing any further progress or adaptation until there’s some form of catastrophic collapse (a ‘paradigm shift’, as in Kuhn’s ‘scientific revolutions’).

Snowden & Kurtz describe the opposite collapse (path 2 in the ‘dynamics’ diagram above) as ‘imposition of order’ – such as the point at which a dictator ‘takes control’ to enforce their own brand of order upon apparent social chaos. In more mundane form, we see this pattern very often in enterprise-architectures and project-management, such as in the ‘cart before house’ attempts to implement a quick pre-packaged ‘solution’ without giving sufficient space to identify the actual requirements. This particularly occurs when perceived time-constraints remove the apparent option for the more conventional requirements-to-implementation loop via pilot-projects (Complex) and analysis (Complicated).

So the standard Cynefin model above shows us only the dysfunctional pathways between Chaotic and Simple, without any functional paths as such. It also tends to describe both of these domains as context-spaces to avoid: the Chaotic because it is too unstable for any science-based cause-effect approach to sensemaking, and the Simple because it’s so often too simple – and hence dangerously simplistic. This limitation of standard Cynefin is perhaps not surprising, because the main focus of the model is on relationships and differences between Complicated and Complex (such as ‘hard-systems’ versus ‘soft-systems’), where time is important but not a key factor. Yet it does also mean that we’ll need an alternate yet related approach for architectures at near-real-time scales, that does describe both dysfunctional and functional paths between them, and that gives us a richer and perhaps more balanced view of the roles of each of the domains.

An alternate frame

One option for an alternate approach comes from classic Jungian psychology. At first glance it’s just a simple two-axis matrix: ‘truth’ (order) versus ‘value’ (unorder – or related to Cynefin’s ‘unorder’, anyway), and outer-world/’objective’ versus inner-world/’subjective’:

Yet we should also apply that key Cynefin insight that each of these domains is simply a means to ‘make sense’ of the totality of what’s going on and what we’re experiencing, and hence that there’s also a ‘none-of-the-above’ domain that precedes any form of sensemaking, which we might label ‘reality’ or ‘disorder’. And we can also derive other labels from elsewhere in the Jungian-related traditions, of which probably the most useful set are the Artist (idea/Chaotic), Technologist (hypothesis/Complex), Scientist (theory/Complicated) and Priest (law/Simple):

One of the most important points that this adds to the standard Cynefin set (Chaotic, Complex, Complicated, Simple) is that the Priest domain is not solely about the real-time simplicity of law – it’s also about certainty. Each of the domains has its own form of ‘certainty’, and its own form of passion too; but in the Simple domain, the Priest is also a passion that certain things are true – and that everything else is not ‘true’. And it’s that passion that seeks to prevent change – even when change is needed. It needs things to be simple, to be certain, to be clear, to be defined forever in some kind of ‘divine Order’ – and it guards that order and certainty with a deep, unyielding, passion that has what we could accurately describe as a religious intensity.

This is a side-effect of the push to simplify, and is also a key driver in the process by which, as Dave Snowden has again commented, valid Simple-domain techniques such as Six Sigma or BPR (Business Process Reengineering) are always at risk of becoming ‘cult-like’. Techniques become cult-like when their proponents forget that the real world cannot all be reduced to Simple rules: it will always also include components that are Complicated, Complex and/or Chaotic. It’s not just that “order is real”, as Dave Snowden asserted in one of our unfortunate arguments, because it is equally true that “unorder is real”: both are true, for a given sense of ‘true’ – which in a sense also means that there are circumstances where each of them is ‘not-true’. What often matters most, perhaps, is which of them is useful in any given context; which has more value.

And so on, and so on – which can quickly recede into a mind-bending kind of recursion. One of the real dangers that I’ve seen in techniques that place their main roots in science – such as Six Sigma or BPR, or Cynefin itself, for that matter – is that the push for Simple laws can sometimes cause a withdrawal into ‘scientism’, a bizarre ‘religion of science’ that is actually closer to an antithesis of science itself. (The antics of many so-called Skeptics represent some of the more infamous examples; likewise people who consider that they’ve experienced a ‘conversion’ to Dawkins-style atheism.) Often key cues arise from language used: phrases such as “must be” or “it’s obvious” are common warnings of a collapse into the over-Simple. The recursion here means that we need to use the frame to review the frame itself: so Cynefin, for example, is oriented towards the Complex, but its roots are in complexity-science (a Complicated-domain modality), and its dependence on scientific ‘law’ (Simple) means that it must always guard itself against becoming overly-simplistic – and also against the resultant ‘natural’ tendency to regard any new ideas and interpretations (the Chaotic domain) as ‘heresy’, a ‘threat’.

Yet the Simple-domain (the Priest) is also one end – the ‘ordered’ end – of a spectrum that implements action in real-time, where there is no time available for thought and reflection. No matter how Complex or Complicated the behaviour may appear be, most real-world processes are anchored in Simple rules: that’s one of the key foundations of all the sciences – including chaos-science – and we ignore that fact at our peril. It applies just as much in business too: hence the everyday foundation of every quality-system is the plain, ordinary, everyday work-instruction, describing the routine choices in routine action. Most things are Simple; most things, most of the time, do indeed need to happen in much the same way – and we ignore that fact at our peril, too.

The Simple goes wrong – often badly wrong – when it makes the mistake of assuming that ‘most’ is the same as ‘all’, or that ‘most of the of time’ is the same as ‘always’. The point here is that there are times when we need to move to the other end – the ‘value/unorder’ end – of that real-time spectrum: and we need to move there as an intentional choice, rather than as the enforced result of an unintended collapse.

For example, every time we start a new project or a new architecture-cycle, we need to intentionally block out everything we believe to be ‘true’, so as to be able to identify the real needs and real requirements in the context. When we first start on a new design, we must deliberately face ‘the unknown’ – otherwise the best we’ll get is another iteration of the known. One excellent description of this process comes from the Indian film director Shekhar Kapur, in a presentation at TED India in late 2009:

When I go out to direct a film, every day we prepare too much, we think too much. Knowledge becomes a weight upon wisdom. You know, simple words lost in the quicksand of experience. So I come up, and I say, “What am I going to do today?” I’m not going to do what I planned to do, and I put myself into absolute panic. It’s my one way of getting rid of my mind, getting rid of this mind that says, “Hey, you know what you’re doing. You know exactly what you’re doing. You’re a director, you’ve done it for years.” So I’ve got to get there and be in complete panic. So it’s a symbolic gesture. I tear up the script. I go on to the set. I panic myself. I get scared. I’m doing it right now. You can watch me. I’m getting nervous. I don’t know what to say. I don’t know what I’m doing. I don’t want to go there.

And as I go there, of course, my AD says, “You know what you’re going to do, sir.” I say, “Of course, I do.”

And … inside … I’m allowing myself to go into chaos because out of chaos, I’m hoping some moments of truth will come. All preparation is preparation. I don’t even know if it’s honest. I don’t even know if it’s truthful. The truth of it all comes on the moment, organically, and if you get five great moments of great, organic stuff in your storytelling, in your film, your film audiences will get it. So I’m looking for those moments, and I’m standing there and saying, “I don’t know what to do”.

Here we deliberately go into the Chaotic – a structured serendipity where we use principles rather than pre-packaged ‘truths’ as our guide. The principles act as a sort-of constraint, but more as a guiding-star to a direction, rather than a boundary or ‘attractor’ as it is in the Complex domain. It creates a space in which the new, the unexpected-yet-appropriate, can happen, can create itself out of nowhere, using us as the means by which it does so – or that’s how it will typically feel, anyway. And the ‘enemy’ here – the ‘giggle-wrecker’ that destroys that bizarrely timeless state of ‘flow’ – is ‘control’, the state of possibly-spurious ‘certainty’ that’s so desirable (and, in many ways, so necessary) in the Simple.

This is very different from Cynefin, in which the Chaotic is essentially somewhere that we might perhaps visit for a brief moment, but otherwise aim to get away from as quickly as possible. Instead, for this purpose, the Chaotic is often somewhere we need to stay in for as long as possible, though it often takes a great deal of courage to rest with the ‘panic’ that this will naturally entail.

More on recursion

Recursion (and likewise its inverse, reflexion – ‘the whole seen within any part’) is a key theme in enterprise-architectures, not least because it can aid in creating simplicity without falling into the over-Simple. One of the problems we do see with careless use of Cynefin in enterprise-architecture is a failure to make appropriate use of the recursion/reflexion pair, in part because whilst recursion is a key pattern that’s used in Cynefin – especially in the Complex domain – it’s not inherent in the framework itself. In effect, the basic Cynefin diagram is often used solely as a Simple frame, as a straightforward single-layer checklist and not much more.

Yet once we introduce recursion as an inherent attribute of everything within the frame, some interesting options open up. [I perhaps need to reiterate that this is not ‘standard Cynefin’ here, though it is one way in which the Cynefin-style frame can be (re)used.] One example above was the recursion in which Cynefin’s focus on the Complex is based on the Complicated ‘truth’-frames of complexity-science, which in turn are grounded in scientific concepts of Simple laws, which in part loop back to the Chaotic in that there are elements of complexity-science that are inherently uncertain.

Complexity-science and its scientific siblings such as chaos-mathematics provide us with powerful tools to make sense of the uncertain at a larger scale – such as in Cognitive Edge’s work on narrative interpretation and abductive reasoning. But the closer we move towards uniqueness, the individual event, the complexity-science breaks down – or more accurately, moves outside of the scope to which it can appropriately apply. When dealing with inherently unpredictable events such as weather-systems or radioactive decay, chaos-mathematics can quantify the unpredictability, to make the degree of unpredictability more predictable, yet it still does not make any individual event any less unpredictable: we can often measure a fissile half-life, or a mean-time-between-failure, very accurately indeed, yet still have no means whatsoever to predict when an individual atom will split, or when an individual unit will fail. And it’s often those individual events that matter most in the architecture – and certainly so in the ideation or sensemaking processes at the start of a new project.

We can see much the same recursion if we use a Cynefin-like frame on methods and methodologies. For example, we could use it to categorise methods and the extent to which we could modify those methods:

• Simple (‘Priest’): the method is fixed, predetermined, invariant – ‘true for all cases’
• Complicated (‘Scientist’): the method can be configured, in accordance with predefined (Simple) parameters or (Complicated) algorithms or (Complex) usage patterns, etc
• Complex (‘Technologist’): the method must be adapted to the context, and ideally should be self-adapting (i.e. Complex Complex) to each context
• Chaotic (‘Artist’): the method cannot be predicted in advance, and may indicate itself without apparent warning, though pre-seeding with (Complex) patterns to provide direction will usually help

Also notable in many Chaotic-domain methods is the exact inverse of the warning that applies so strongly elsewhere. In the Complicated-domain, it’s (rightly) considered crazy to repeat the same action and expect different results; yet in the Complex-domain, doing the same thing will usually lead to different results, whilst in a true Chaotic-domain doing the same thing will always lead to different results. Hence in the Chaotic-domain, constant repetition of Simple actions – as typified by meditation and the like, or the musician’s endless practice of scales and sequences – will often provide ‘just-enough’ predictability to make the unpredictability usable in practice.

It’s also essential to note that whilst Complex-domain patterns are useful in all skills, ultimately every true skill is personal – in other words, must always in part be Chaotic. We cannot learn on others’ behalf; there is no substitute for personal experience. Which also leads to another key point in every architecture: it’s always in part about values – sometimes collective but also always personal – which are, by definition, ‘subjective’: so we can’t and shouldn’t expect that all of it will necessarily ‘make sense’ to anyone else. That’s one aspect of our work that will always be ‘unscientific’ – yet also a strange blend of scientific law and the science and art of inherent uncertainty.

In the actionable world, order is real, yet so is unorder: a real-world, real-time spectrum that wanders between predictability and unpredictability, engineering and imagination, sameness and difference. We can quantify uncertainty in the enterprise, but we can never make it certain. Getting that balance right across all of those multitudinous interdependencies and layers and complexities is, to me, the real essence of enterprise-architecture.