Control, complex, chaotic

What exactly is ‘the chaotic’ in enterprise-architectures? How do we work with it, design for it rather than ‘against’ it?

Yeah, I know this is a theme I’ve visited often here, but to me it’s a challenge that’s right at the core of all of our current approaches to enterprise-architectures and the like. What triggered it this time was a great conversation with Nigel Green, in our back-and-forth comments to the post ‘More keywords for SCAN‘, and also in that, in looking for a worked-example for an upcoming conference, I came back to the seemingly-endless problems with the UK National Health Service (NHS).

I won’t go into any detail on the NHS’ travails here: many of them are well-known, such as the ongoing disaster-area of its would-be information-systems, and the disastrous misadventure of trying to measure and ‘control’ health-delivery performance in terms of predefined targets. The NHS started out some 65 years ago with the powerful vision of ‘health-care for all, free at the point of delivery‘, yet that vision has been so much whittled away by politicians and others over the years that it’s now best described as a highly-politicised mess of competing interests in which, all too often, the health-needs of patients have ended up at the bottom of the stack. Oh well…

There’s a lot that needs to be done to make the NHS workable again, of which perhaps the most important would be to properly refocus on that original vision. Over the next few weeks, I’ll use this as a worked-example on how to apply enterprise-architecture thinking and methods, at very large scale, and beyond IT-alone, to tackle a major strategic review for any organisation, and how to link all the way from strategy to execution and back again, so as to bring that organisation ‘back on track’ to its real aims.

For here, though, there’s one point that’s really striking, and which is more about mindset than method. Many people have (correctly, to my mind) identified some of the core problems of the NHS as arising from misguided attempts to ‘control’ the inherently-complex: the ever-increasing managerialisation of the service, the disastrous impacts of misguided ‘targets’, or the all-too-frequent failures to deal with inherent wicked-problems as wicked-problems – for example, pharmaceuticals-manufacturers wanting to maximise profits, against the the Service’s desperate need to manage spiralling costs; or the diagnostic capabilities of new technologies creating a spiralling problem of ‘too much medicine‘, “the threat to human health posed by overdiagnosis and the waste of resources on unnecessary care”. Attempting to ‘control’ complexity just doesn’t work: we need to treat the complex as complex, not as a ‘controllable problem for which we don’t quite know all the rules (but will know them all Real Soon Now, honest…)’.

Yet I’m also noticing another deeper problem: misguided attempts to apply complexity-theory to things that are neither rule-bound control nor pattern-based complexity, but are inherently ‘chaotic’ – a ‘market-of-one‘. Although we can identify definite patterns in health and health-care – that’s the whole basis of epidemiology, for example – neither rules nor statistics can help us deal with the blunt fact the everyone is different. The kind of patterns that we’d use in a complexity-model – probabilities, Bell-curve distributions, outliers, all that kind of thing – can all too easily mask the real underlying fact of uniqueness, from which that supposed ‘pattern’ will actually arise: somewhat like the barely-visible deep-randomness that underlies the visible patterns of Brownian-motion.

This is actually where the difference between complexity-theory and real-world practice ended up forcing me to part company with complexity-oriented frameworks such as Cynefin. They seemed to interpret ‘chaos’ solely as a feed into their own models of complexity, whereas I was finding uniqueness – or, to use Common Ground‘s terms, particularity or distinctiveness – as an inescapable fact that I needed to deal with in its own right. Uniqueness – the ‘chaotic’ – cannot be reduced to complexity or control.

One example of the ‘gotchas’ here, that’s commonly used both in concepts of ‘control’ and of complexity, is ‘Occam’s Razor‘:

“among competing hypotheses, the one that makes the fewest assumptions should be selected”

Yet on its own, as Steve Nimmons warned some while ago, Occam’s Razor is a proven anti-pattern in enterprise-architecture. To make it work in real-world conditions, we need to balance it with a better awareness of uniqueness and difference, such as in the ‘anti-razor’ of ‘Hickam’s dictum‘:

“patients can have as many diseases as they damn well please”

Yes, every patient will have similarities with others; yet they are also different to all others, unique, themselves. ‘Similar’ is not the same as ‘the same’ – it’s ‘same and different’, both at the same time. Which means that whilst the needs of each patient may be similar to others, and hence need treatment that’s similar to those of others, they’re also different, unique, not the same as any other. Which, in turn, means that any health-delivery mechanism that assumes that everyone is the same (‘control’) or fits with some kind of predeterminable pattern (‘complexity’) is going to deliver the wrong treatment for that unique person, quite a lot of the time.

The same is true in sales: every sale is ultimately a market-of-one. The same is true in business-models: the whole point there is to differentiate from others, to not be the same as everyone else – even though we need to trade via the same standards and suchlike. I’ve come across this same theme time after time in many other guises: for example, Australian researchers are world-leaders on ‘ageing-aircraft syndrome’ – not just corrosion, or fatigue, or mechanical-damage, but the ways in which each of these and other relatively-predictable factors interact in each unique case. Attempting to apply ‘control’ or ‘complexity’ to the chaotic just doesn’t work: we need to treat the unique as unique, not as something that’s resolvable by a ‘control’-based or complexity-pattern-based ‘solution’.

In sense, what we need here is a kind of uniqueness-oriented equivalent of complex-adaptive systems: we might call it chaotic-adaptive systems, perhaps. One possible source of ideas here is ‘‘pataphysics‘:

“‘Pataphysics seeks … to view each event in the universe as completely unique, subject to no laws but its own … the science of the particular [rather than] the only science … of the general … not the rules governing the general recurrence of a periodic incident (the expected case) as the games governing the special occurrence of a sporadic accident (the excepted case).”

The catch is that, by intent, ‘pataphysics is more of a parody of science than a true science in its own right – hardly surprising, given that its origins were in Absurdist theatre… Yet there are distinct disciplines around it, such as the psychogeography work of the Situationists and, more recently, the writer Will Self – or, again, as documented in Common Ground’s work on ‘particularity’ and ‘local distinctiveness‘. Perhaps not a science in the conventional sense, then, but still capable of explicit disciplines that are fully-compatible with the sciences. Probably quite a lot more that needs to be done there to make it more accessible to others, though.

At this point it might also be useful to look at this in terms of the SCAN framework:

Another way to look at the axes in SCAN, that’s more compatible with what we’re working with here, is to redescribe them as follows:

• horizontal-axis: from same to different – identical (far left) to unique (far right)
• vertical-axis: from theory to practice – infinite available-time-before-action (‘infinity’ – e.g. analysis-paralysis) to ‘NOW!‘ moment-of-action (baseline)

Which, in terms of the domains, gives us:

• Simple: real-time rule-based decisions, suitable only for identical or near-identical contexts
• Complicated: non-real-time algorithmic decisions (with computer-automation enabling it to get closer to but not at real-time), suitable for high-variety yet still predictable contexts
• Ambiguous: non-real-time pattern-based decisions (with computer-based pattern-matching enabling it to get closer to but not at real-time), suitable for ‘complex’-type contexts
• Not-known: real-time principle-based decisions, suitable for potentially-unique or ‘chaotic’ contexts

Note that every real-world context will have a mix of Simple, Complicated, Ambiguous and Not-known: which means that we’ll need an equivalent mix of matching techniques to ‘solve’ and/or resolve the whole of the context. (We don’t say that a context ‘is’ Simple or whatever – solely ‘in’ one domain or another – because that all but guarantees some form of mismatch-failure further down the line.)

The catch, in practice, is that whilst in most enterprise-architecture contexts we do have many well-identified techniques for ‘control’ (Simple and Complicated), and increasing awareness of techniques for ‘complex’ (Ambiguous) – if perhaps still too much reluctance to use those complexity-oriented techniques – we still don’t have much if any awareness either of techniques to work on the ‘chaotic’ (Not-known), or of how and where and when to use them. That’s a real challenge that I’d suggest enterprise-architects and others definitely do need to start looking at right now.

Comments, anyone?