Complexity, chaos and enterprise-architecture
Courtesy of a link by fellow enterprise-architect Sally Bean, I’ve just spent the past couple of hours viewing and then reviewing an online seminar on complexity by one of the thought-leaders on complexity-theory and practice, Dave Snowden:
From Induction to Abduction: a new approach to research and productive enquiry
This seminar will provide a summary of both the theory and practice of a new approach to research based on the large scale capture of self-interpreted micro-narrative. The approach has been described as the first technique for distributed ethnography and has been developed over the past decade with project based funding from the US, UK and Singapore Governments in the context of risk assessment, horizon scanning, cultural mapping and weak signal detection. It allows the linkage of research with knowledge management and impact based measurement. Current projects involve measuring the impact of development projects in Africa, narrative based knowledge management for the US Army in Afghanistan and cultural mapping of various inner city communities within the UK.
The theoretical origins lie in the application of complex adaptive systems theory to social systems together with new understanding about the nature of human decision making from the cognitive sciences. The seminar will summarise the theory, but will also use a series of projects to combine theory with practice. One of the goals is to create learning systems that work on continuous capture of material in the field as it happens linked with a capacity for feedback loops and sophisticated representations that allow people to learn by doing, building on the micro-narratives of day to day experience. Narrative forms of knowledge lie between the experiential and the symbolic, allowing complex interactions and interventions in multiple social situations.
Abductive reasoning is, as Dave explains, “the logic of hunches”, and plays a key role in helping to develop understanding of how themes emerge in social contexts such as in business and elsewhere. It’s all fascinating stuff – very strongly recommended. The depth and versatility of the techniques will be a real eye-opener to anyone who hasn’t previously seen Dave’s work, and its applicability to whole-of-enterprise architecture and the like should be self-evident.
I will admit I do have mixed feelings about the way Dave develops and presents his work. On the one hand, he has a brilliant mind and is a brilliant presenter, and there’s no doubt at all that his tools and techniques, such as Cynefin and the theory and practice behind his Sensemaker software-suite represent real paradigm-shifts in the way we think about organisations and enterprises (in the broadest sense of those terms). But I do find it beyond tedious that he spends so much effort denigrating other people’s work – for example, Nonaka, Weick and Six Sigma (endless derided by Dave as ‘sick stigma’) all come in for attack in the first few minutes of the seminar. And I too have been on the receiving end of that same… well, I would have to describe it as an odd kind of sort-of-scientific bigotry… which is more than just annoying at times. And annoying not least because pretty much everything I’d tried to explain to him and that he’d dismissed with such vehemence – such as the nature of ‘magical’ processes and the role of ritual – Dave in fact now incorporates (though probably unconsciously) as significant if unacknowledged sub-themes in his work (as can be seen in various places in the video). But we have to take the ‘Dave Snowden’ package as a whole, I guess: and most of the contents of that package are important – definitely.
But there’s one slide, right at the start of the presentation, that I find especially fascinating:
There is indeed a clear historical sequence here, paralleling the shifts in the underlying scientific paradigms, from Newtonian to hard-systems to complex-systems in the present day. But there are two important points that are easy to miss here:
- in each case the old dominant paradigm remains useful, though is seen to describe a distinct set of special-cases rather than a grand ‘Theory of Everything’;
- the sequence does not stop here, with Dave’s ‘sense-making’ – it continues on to at least one more layer, and possibly two.
The reasoning for those assertions comes from Cynefin itself, plus a cross-map to this diagram above. As Shawn Callahan of Australian consultancy Anecdote explains in his excellent intro on the same web-page, Cynefin has a central (and ‘initial-state’) domain of ‘disorder’, and has four distinct domains of sensemaking and action:
- Simple: assumes simple cause-effect rules; sensemaking tactic is ‘sense -> categorise -> respond’
- Complicated: assumes linear causality, but accepts that these may include many factors, delays, feedback-loops etc; sensemaking tactic is ‘sense -> analyse -> respond’
- Complex: accepts that cause and effect are intertwined, leading to non-linearity and non-reversibility; sensemaking tactic is ‘probe -> sense -> respond’
- Chaotic: no identifiable cause-effect relationships; sensemaking tactic is ‘act -> sense -> respond’
Dave links the first three of these to the respective S-curves: Scientific Management is Simple, ‘classic’ Systems-Theory is Complicated, and his version of Sense-Making is Complex. Yet he provides no equivalent linkage for the Chaotic domain, and the listed tactic of ‘act -> sense -> respond’ literally consists of running away from the problem. Which is hardly a valid approach if the chaos insists on being sustained. Which, in the real world, it all too often does…
It’s often struck me that Dave’s great strength in the Complex domain seems also to create a real inability to describe any means to tackle the Chaotic domain. In this sense it does seem that, to use his own words, “the old dominant paradigm suppresses the new idea” – where the ‘new idea’ is that the Chaotic domain does need to be respected in exactly the same depth as we now do for Complex and the others.
To me there are two key clues here.
The first is a direct warning from the classical tradition and elsewhere that running away is not a viable response in the Chaotic domain. The clue here is the Greek-derived word ‘panic’, which is what many people will experience when facing any kind of chaos. But the ‘pan-‘ root literally means ‘the everything’ (hence panorama, pandemonium, and so on): what’s happening in panic is that everything and nothing is true at the same time. Yet that’s exactly what we need when we’re striving for innovation, or any other kind of search for new ideas: we need the ability to bring apparently unrelated themes together in new ways. And in practice we do that by not running away, but instead by ‘holding to the centre’, ‘finding the still-point’, ‘the calm at the centre of the storm’, and so on.
To find that centre, we turn to the other clue in what’s not in Dave’s diagram above.
The Simple domain is about control of Function, says Dave – in other words, the physical world, the physical dimension. Hence scientific-management came to the fore in the heyday of the assembly-line – and it still makes perfect sense within that type of context, where everything remains exactly predictable, exactly the same, just like most physical objects do.
The Complicated domain, says Dave, is about control of Information, the conceptual dimension. Hence ‘hard-systems’ thinking came to the fore in the heyday of the mainframe and the supercomputer, massive number-crunching and the like – and it still makes perfect sense in that context, of massively complicated cause-effect relationships between information-items.
The Complex domain, says Dave, is about control (or ‘situating’, rather) of the Network – otherwise known as the relational dimension. Hence Dave’s ‘Sensemaker’ and the like come to the fore in genuinely-complex social contexts, where meaning and ‘truth’ emerge from the interweaving between the individual and the collective in the respective physical, conceptual, social and aspirational milieu, in which everything and anything may become both cause and effect of everything else.
But what’s not there in Dave’s model is any consistent framework to tackle the Chaotic domain – instead, we’re just told to run away back to the safety of one of the other domains. And yet, following that same logic above, we can see straight away what its base would be: the aspirational dimension, the explicit choice of meaning and purpose – otherwise known in the enterprise-architecture as vision, values and principles.
From my discussions with him, Dave seems to dismiss this whole domain because to him it appears to have no identifiable science behind it. Yet I would suggest that this too may be the result of a too-close identification with his own concept of ‘science’, because as soon as we allow ourselves to move outside of the constraints of Western tradition of science, we will immediately find other traditions with at least the same levels of precision and discipline, if not more. The Australian Aboriginal concept of the Dreaming is one obvious example, an extremely sophisticated study of relationship with the land that is only now beginning to be understood in Western terms; likewise the Tibetan research into the period immediately before and after death; or, to give a more tangible example, the Polynesian science of navigation:
‘The Wayfinders’ [lecture-series] is a profound celebration of the wonder of human genius and spirit as brought into being by culture.
The entire science of wayfinding is based on dead-reckoning. You only know where you are by knowing where you have been and how you got to where you are…that your position at any one time is determined solely on the basis of distance and direction travelled since leaving the last known point… If you took all of the genius that allowed us to put a man on the moon and applied it to an understanding of the ocean, what you would get is Polynesia.
The consistent theme in each of these traditions is a very strong sense of purpose, intentionally embedded by and within the individual to act as a personal ‘guiding star’ that provides a known, certain ‘still-point’ under conditions on uncertainty and chaos. Hence, although Dave somewhat characteristically dismisses and debunks vision in the business context, I do believe he’s missed the point. Those are genuine skills, genuine sciences, every bit as valid as as the sciences behind scientific-management, systems-thinking and complexity-theory – and their niche of greatest applicability is the Chaotic domain.
Which, once we think of it that way, makes Cynefin complete.
Or rather, there’s one more layer to this. Each of the Cynefin domains has its own respective science, its own technologies and so on. But there’s also a need for a ‘meta-discipline’ to switch between the Cynefin domains, linking them together into a unified whole.
Checklists can provide some of that discipline; likewise a consistent iterative methodology such as the extended TOGAF-like cycle that I use in my own enterprise-architecture work. An explicit multi-dimensional model such as the tetradian can also help in this. And we have much that we could learn from the many non-Western traditions – or even from a better understanding of how science really works in practice.
But perhaps more, for here, we perhaps need to note that whilst Dave’s complexity-theory is useful – very useful indeed – it’s unlikely to be ‘the last word’ in the sciences that we need in enterprise-architecture. There’s still some way to go: and a more consistent, more honest approach to how we handle the Chaotic domain would seem to be the necessary next step in that journey.
Thanks for recommending the broadcase Tom. Some points of clarification:
I happily plead guilty to deriding six sigma in all its manifestations; its a cult and it damages people
I have a huge respect of Weick and regard him as one of the sources for my work. I disagree with his work (with Sutcliffe) that fails to take account of context in seeking to transfer capability. I use a metaphor to illustrate this (I can get the same behaviour as firefighters if I burn the office down every morning). It think you are in danger of confusing disagreement and distinction making with derogation which would be a pity.
Ditto Nonaka although to a lesser degree. He wrote a good book about Japanese manufacturing, but a very bad KM book. The misreading of Polayani and the SECI model that arose was just the sort of start that KM didn’t need.
I don’t think I have ever called any of them a bigot, or accused them of taking a bigoted perspective by the way (to use your words)
Chaos in the Cynefin framework is a state of randomness. Taking any action means that you establish connection of some sort which moves it out of the chaos domain. I find your “runaway to the safety of other domains” a misleading comment. OK you want to make something more of the chaos domain – feel free, but don’t misrepresent the model
Shamanistic knowledge (which I have studied in depth) is complex not chaotic, its based on connectivity that we don’t fully understanding. Its concrete not symbolic so we cannot articulate it in the way that we can outline rules of accounting. Polynesian navigation, dreamtime are all examples of complex systems in practice, not chaotic ones. The practice of a spirt journey is very similar to the exercises of Loyola and other meditation techniques. I take this sort of thing seriously, both the practice and the growing body of scientific insight. I get very concerned about western romanticization of such practice, and also the retreat from reason which often accompanies it. I’m sure you share that view. The disagreement is that I see it as part of the richness of complexity you on the other hand ….
…. want to call this sort of thing magi. That is of course your own affair. However, given the common meaning of that term I think its use does a disservice to indigenous knowledge and would result (I know this is not your intent) in a consequential lack of respect.
Having a vision is very different from vision statements in corporate and government speak. Please don’t confuse my attacks on the meaningless second form with my respect for the first.
Shawn’s otherwise excellent video does not deal with the role of disorder. A lot of what you mean by chaos is in my terms disordered (subject to the qualification on indigenous knowledge above. Exploring disorder and its boundaries is one of the areas where I am doing a fair amount of development at the moment
I don’t see how I can have subconsciously picked up on the use of ritual, when it is an explicit part of the method set. I first formally used it over two decades ago in a major project (and many times since) so my use and acceptance of it is long standing and explicit. One of the reasons for this is that the cognitive science understanding of how ritual triggers change in the brain allows scalability.
Interesting reading as ever, I look forward to more
Hi Dave – Many thanks for replying, and yes, I’d like to see the webinar get as much attention as possible, because I do believe that what you’re describing there is extremely important.
I’ll skip over your interpretation of the comments about Nonaka et al, other than to suggest that bigotry occurs when someone mistakenly assumes that because they know they are right, therefore everyone else is wrong. That’s the kind of crude two-valued ‘either/or’ logic that is so popular in the Simple or Complicated domains, but in the Complex domains is often dangerously misleading – instead, as you know, a context-driven ‘both/and’ logic is preferable there. So, for example, even Six Sigma _does_ have its value – though I’d agree that there aren’t that many contexts that have many millions of exactly identical events, which is the only type of context where a literal ‘six-sigma’ would make much sense.
As for the rest, I believe that we agree for the most part – and I’ll happily acknowledge that you have a far better grasp of the formal science of complexity than I will ever achieve. 🙂 Yet as for how and where it connects with the Chaotic domain – following the stated logic of Cynefin, that is, which may not quite how you originally envisioned it – I believe there’s still a lot more that we can do there. I certainly believe at present that simply treating it as a source for Complex-domain data, whilst obviously valid, is not making anything like the best use of its possibilities. I also suspect that abductive reasoning, as you’ve described it, may more properly belong in the Chaotic domain rather than the Complex. I’ll develop some more ideas on that for another post over the next couple of days.
In the meantime, once again, many thanks.
Thanks for a generally positive response.
There is nothing you can legitimately do with Six Sigma that you can’t do with a cheaper BPR tool, hence my comments. I am sure Mussolini did good work to make the trains run on time, but that does not mean that wanting to have the trains run on times means that there is some good in fascism.
On Chaos: the only way something can sit in that domain is if all agent behaviour is random. In all work on complexity science the key learning is that connections emerge very quickly in chaotic environments, the systems become complex at which point (in human systems) meaning is possible. So if you can find some example of human action or knowledge which is purely random then it would be there. However I doubt you can and the examples you have given so far of indigenous knowledge are far from random in nature.
I hope you are not saying that I am bigoted in the sense you define it above. If you are then come out in the open and say it, otherwise please clear up any potential misunderstanding
On Six Sigma and the like, I look mainly at the principle rather than solely at the execution – and the underlying principles seem sound enough, though limited in their application (because there aren’t all that many contexts with many millions of exactly identical events). It’s quite probable “there is nothing … with Six Sigma that you cant do with a cheaper BPR tool” (though I was under impression – presumably wrongly – that Six Sigma was a collection of techniques rather than an automation tool as such), yet, again, the principle itself is sound enough. And yes, I too share the same strong dislike of layered certifications that may well be meaningless in practice – we likewise suffer from the same problem in enterprise-architecture. But again, I’m looking always first at the principle, then for contexts where that principle can be used well, and only _then_ start to critique areas where they may be inappropriate.
On Chaos, I take your point, yet I wonder whether you’re perhaps placing complexity at ‘the Theory of Everything’ in much the same way as Taylor did with his simple Newtonian concepts. Rather than the somewhat Spiral-like linear development you’ve presented, I tend to think of it more as occupations of conceptual/methodological space, kind of like a four-axis version of your triangle-mapping. That’s what I’m aiming to explore in the post I’m writing now.
On bigotry, I would much prefer to keep this professional and politely discreet, but you insist that I must say this out loud. So I do apologise, yet it _is_ necessary to say this to you. I will never say anyone ‘is’ bigoted, because that precludes the possibility of change; yet unfortunately it is indeed true that – as evidenced all too often in that seminar, and even in your replies here – you do tend to display behaviours which would be experienced by many as “an odd kind of sort-of scientific bigotry” in the sense described above: an over-certainty in self combined with perhaps too much eagerness to assert that others are flat-out ‘wrong’, without much if any acknowledgement or respect for the contexts in which they may well be right, in their own terms at least. I too have been on the receiving of this from you, and can tell you that it is unpleasant in the extreme: it certainly suppresses willingness to engage with you, because, to use your own phrase, it does feel very much feel that “the old dominant paradigm suppresses the new idea” – with ‘the old dominant paradigm’ in this case (somewhat ironically, given the struggles you’ve had in establishing the value of complexity-thinking) being your own personal dominance in Cynefin and the related contexts of complexity. To be frank, I am literally afraid of how you will respond to this: you will appreciate, I hope, that it does not make it easy to engage with you in what should otherwise be a strictly technical discussion on the merits and flaws of various ideas – especially when those ideas are, of necessity, often in a half-formed and uncertain state. So I do ask you to face that unfortunate tendency in the way you engage with others, because it would make it much easier for us in turn to engage with you and your work. Having said this, I can only hope that you take this as it is indeed meant, as respectful critique of professional actions and professional behaviours. And I do emphasise, strongly, that I say all of this with the deepest respect for you, in every sense, as one of the great thought-leaders and great men of this field.
Beyond that, I hope that we can now return the focus to the underlying concepts and possibilities implied in the relationships between the Complex and the Chaotic – because I do believe that there is much that we (collectively, not just you and I) can learn. Will you help us in this?
In any case, once again, many thanks.
Complexity is the theory of systems where constraint and action co-evolve, its that simple. Given that some systems have random action and others have constrained then it is hardly a theory of everything or a linear model. If you are looking for a system beyond those then you need to find more than three ways of perming two variables.
I much prefer directness to the innuendo of your previous comments. Given your dislike of criticism such name calling would be hypocritical at best. Some things however are flat out wrong (the dichotomy between tacit and explicit knowledge, creationism and many others) and no one does any one any favours by saying anything else. Some theories and people have made a huge contribution but are clearly wrong in some aspects (the failure to appreciate context in Weick and Sutcliffe for example); for the avoidance of doubt I suspect this will be the case with my work as well. Indeed I hope it is or there will be nothing new to work through. Basically if you think I am being bigoted then say so and say why and there is no problem. However don’t be surprised to get a response in kind. If you can’t take it don’t dish it out is a good motto. 🙂
In this respect I think you need to understand that if you want to contribute to the development of a field or a model you must expect rigorous debate and challenge. This happens all the time in debates I have with Boisot, McKelvey and many others in the field (compared to which our couple of disputes are a minor storm in an insignificant tea cup). Without such rigor there is little chance of progress. If you find this unpleasant then you are going to have problems, but you are not afraid to dish it out you know (look at the above) so I think you can probably cope.
I have a strong bias towards the natural sciences and the Cynefin framework is built from a science based position. However natural science is not longer in the Newtonian paradigm and it seems to me that you characterise it in that way. I can study and value indigenous knowledge without the need to resort to a concept of an unknowable other (such as magic). To take another example, I have seen two many examples of dowsing not to believe it works in some way, I can also see that in all the cases it is a deeply embodied skill that cannot be taught. That leads me to look at the way in which the left, autonomic brain builds patters and the sensor networks. I also have to respect the fact that all controlled tests have failed to establish authenticity. This provides an interesting dilemma. On the one hand I have seen it work with water engineers, and with the man/jcb symbiosis that dug out the drive to the side of my house, on the other hand controlled tests have failed to validate. That means we have a really interesting anomaly that requires investigation – but it does not allow a strong claim for authenticity and the solution will be scientific.
I hope that helps
Dave – Many thanks indeed – this hasn’t been easy, but that’s exactly the response I’d hoped for. Thank you.
We have different styles, I guess. I find it immensely difficult to create and work on new ideas in an atmosphere of violent clash, but I can understand (just…) that others may prefer it the latter way. (Wild Welsh versus Insipid English, perhaps? 🙂 ) In my experience, though, there are ways to critique that work well, and others that don’t. So I don’t like to ‘dish it out’ not because I “can’t take it” but because I see the damage that it does. Yet each to their own, really.
On dowsing, you probably know that that’s been one of my main methodological research-fields for several decades now. I agree that it cannot be taught, but as with any skill it is possible to provide conditions under which it can be learnt. I also agree strongly with you that it present “a really interesting anomaly that requires investigation” – but a conventional scientific approach to such investigation will always have trouble, because science requires repeatability, which by definition is in short supply in such contexts. More on that in the post that I’m working on at the moment.
Once again, though, many thanks – is much appreciated.
Pleased we have arrived there. Only two minor points (i) from my perspective, if you look at your language you are more than happy to “dish it out”, and (ii) a dispute does not need to be violent, vigorous criticism/defense can be interpreted in that way. I’ve seen a lot of violence done to people by facilitators who try and silence people with a dissenting view because a harmonious agreement within a group and take the “any idea is a valid idea” approach.