More on reframing entropy in business

My previous post on ‘Reframing entropy in business‘ kinda triggered off a veritable storm of correspondence, both in the comments and offline – hence seems it’s worth summarising and revisiting those themes here.

Perhaps the first point is that yes, it’s confusing – to say the least. There are several different ways to view entropy in business, and often they clash horribly with each other. Probably the best advice here came in a comment from Ivo Velitchkov:

it seems we are not dealing with entropy but with the “story of entropy”, and our co-creating based on our mental models is what matters

In other words, think of it not as about establishing which theory or view is ‘true’, but much more about what kind of story is useful for enterprise-architecture and the like. A rather important difference…

The real source of confusion is that there are at least two fundamentally-different things going on:

• the strict physics view of entropy as ‘the winding-down of the universe’, a steady(ish) leveling-out and decay of (usable) energy-difference
• the notion of ‘order’ in the universe, often contrasted with ‘disorder’

Hence we have the notion of (desirable) order decaying to (undesirable) disorder, as part of a loss of available energy. That’s a useful way to describe some of the issues that enterprise-architecture would need to address in business – a useful story of entropy. Yet that’s only one interpretation we can derive from that mix of themes: there are quite a few others. And sorting out which story is which, and which story to use in each type of context, is not helped by the fact that ‘order’ is just one of several key terms here that each have at least two radically-different meanings or interpretations:

• ‘order’ as something inherent, something that naturally exists – the ‘laws of science’ and suchlike – and that describes and enables predictability and certainty
• ‘order’ as something artificial, a constraint or filter arbitrarily imposed upon the broader view of the whole
• ‘disorder’ as the collapse of order, the decay after a breakdown in order
• ‘disorder’ as an unfiltered condition before sensemaking or imposition of order-constraints takes place
• ‘chaos’ as the absence of order, or the extreme end-point after disorder-as-breakdown-of-order, a state of total differentiation where nothing is possible
• ‘chaos’ as the extreme start-point before order, a state of total differentiation where anything is possible
• ‘variety’ as the total number of possible states that a context can present (context-oriented view – hence ‘requisite-variety’, the total number of states that a system would need in order to ‘control’ that context)
• ‘variety’ as the total number of states that a system of order can impose on the context (system-oriented view)

Different combinations of these definitions can imply very different stories. For example, the classic deterministic view, typified by Taylorism, would view order as inherent, natural, certain and desirable. Disorder is a failure of order, and undesirable; chaos is the absence of order, and highly undesirable. The quest is always to create a system with sufficient control-variety as to render the context under complete, perfect and predictable order, viewed always as an ideal yet at least theoretically-achievable goal. In this story, entropy is essentially a story of failure: the loss of certainty, the loss of control.

Contrast that with the story implied by the Cynefin framework. As I understand it (and I may be wrong, of course), ‘disorder’ here is not the collapse of order, but more a label to apply to a context before any sensemaking or ‘ordering’ takes place. Order is not necessarily something inherent, but more a view or filter that’s imposed on a context – particularly as seen in Cynefin’s ‘simple’ and ‘complicated’ domains. Yet deterministic-order itself can at times be misleading, because contexts may often include elements whose variety can change in often-unpredictable ways, and that can be inherently non-linear, self-altering (such as in ‘wicked problems‘), random or unique. These latter elements are not amenable to conventional linear concepts of ‘order’ – hence why Cynefin co-originator Cynthia Kurtz coined the term ‘unorder‘ – and are described in Cynefin as ‘complex’ or ‘chaotic’.

In essence, the Cynefin story both includes and extends the Taylorist story. Entropy could still be seen as a natural decay of order, yet not necessarily a ‘failure’, because the existence of unorder – the Complex and the Chaotic – always presents the possibility of counter-entropy refresh.

To me, as an enterprise-architect, this view as suggested by Cynefin is a more useful story than the classic Taylorist one of a futile struggle for ‘control’ against the tides of decay.

The view or story of entropy is different again in the SCAN framework. Here the emphasis is probably more on the relationship between the effective-variety and ‘variety-weather‘ in the context-in-scope (the first meaning of ‘variety’ in that bullet-list above), versus the available variety in the means we have to interact with that context (the second meaning of ‘variety’) – and hence whether the would-be system of ‘control’ has the requisite-variety to impose its control on the context.

In SCAN, the effective limit of requisite-variety is represented by the ‘Inverse-Einstein boundary’ – the vertical red-line. To the left of the line, doing the same thing leads to the same results: the control-system has requisite-variety to deliver the desired outcomes in the context. To the right of the line, doing the same thing may lead to different results, or we may need to do something different in order to achieve the same results. There, the control-system either does not have requisite-variety, or else it needs assistance from ‘outside’ of itself in order to adapt the control-mechanisms so as to create a localised sort-of-requisite-variety for that specific context.

There’s also a crucial distinction between theory and practice, that’s often expressed in terms of the time-dimension in SCAN – the horizontal dotted-line boundary between ‘considered’ intent (above the line) versus real-time action (below the line):

What theory expects or asserts is ‘controllable’ can often omit or ignore elements of variety that make the context much less controllable in real-practice: the variety of the control-system will seem sufficient to provide requisite-variety for the context, but there’s more variety in the real context than theory expects. This often gives rise to a skewed perception at the theory-level – the Complicated domain – of just how much ‘control’ is available from the current variety in the would-be control-system, and/or a dangerous delusion either that everything ‘should’ be in our control and that the real-world (or the operator who has to work on that real-world context) is somehow ‘wrong’ for failing to align itself with our expectations, or that we have control anyway because anything that doesn’t fit our expectations can be safely ignored:

The only way out of that trap is to pay rigorous attention to what the real-world is telling us about the actual variety in the context, and to keep a careful rein on our expectations and wishful-thinking in the desire for certainty and ‘order’. (Both of these challenges are represented respectively by the two cross-cutting boundaries in the SCAN frame: the horizontal boundary between theory (above) and practice (below); and the vertical Inverse-Einstein boundary between repeatable (left) versus potentially or actually non-repeatable (right).) At times we need to be very careful to remember that ‘science’ is just another belief-system, with all the limitations and constraints that that implies – especially where people’s emotions and feelings are facts in the context.

In terms of entropy, I’d argue the same principle as in the previous post, that entropy represents a decay from chaos to ‘useful-order’ to ‘non-useful order’ and onward to decrepitude. In a sense, any form of ‘order’ is over on the left-side of the SCAN frame – and as soon as it gets there, it’s already decaying towards ‘non-useful order’. The distinction between ‘useful’ versus ‘non-useful’ is less about the variety of the context in itself, but more about the available-variety in the system for ‘control’ of the context – so it’s actually more that the relationship between the ‘control’-system and the real variety in the context, rather than that the context itself is decaying.

In many cases the variety in the context actually increases during this apparent decay  – for example, parts within the control-system become more worn (hence capable of more variety of movement etc), or raw materials have more variance than the control-system expects. The decay into so-called ‘disorder’ or ‘chaos’ is more that the control-system can’t deliver the desired results any more – not something inherent in the context itself.

So as soon as we have ‘order’, or ‘control’, over the context, the fact of entropy should warn us that we’re already starting to lose it. Once the loss of ‘order’ or ‘control’ moves far enough towards a tipping-point, we’re likely to be pushed over the Inverse-Einstein boundary into uncertainty and ‘unorder’, whether we like it or not. The key here is to realise that that far side of the Inverse-Einstein boundary is the only space where counter-entropy becomes possible –  in other words, a place where we can leverage the uncertainty itself to reframe the structure and capability of the variety in our ‘control’-system, and thence to revitalise our ‘useful-order’ in the context.

In the longer term, what entropy really tells us is that the only way to maintain order is to let go of order – and to know when (and under what conditions, and so on) to hold onto order, and when to let it go. That’s a real skill in itself… for which the key trick is to choose to let go before it’s forced upon us by that decay of entropy. For example, at a more conceptual or theoretical level, we can explore ‘It depends…‘, which places us for the while over in the ‘Ambiguous but Actionable’ space (and respective tactics) of the SCAN frame; or we can face the deeper challenges of ‘I don’t know‘, making use of the tactics summarised for the ‘Not-known, None-of-the-above’ space in the frame.

I’d emphasise again, though, that it’s best to assume that none of these views into entropy and the like are ‘the truth’, or suchlike – merely that they can be useful, in different ways, in different contexts, for different people, and so on.

What I’ve found that definitely is useful, in enterprise-architecture practice and elsewhere, is to be able to switch between these views or ‘lenses’ according to need – what Richard Veryard, Nigel Green and others describe as ‘lenscraft’. I hope this is useful to you in that sense at least, perhaps?