Can complex-systems be ‘architected’?
A great question from Jeff Sussna, as a follow-on to my ‘manifesto‘ post:
- RT @jeffsussna: @tetradian question: can complex systems like your ‘enterprise’ actually be ‘architected’?
And yeah, it’s also a tricky question, because a lot of any answer depends on what’s meant by ‘complex systems’, ‘enterprise’ and ‘architected’…
In terms of what’s commonly meant by each of those terms, and what most commonly claims to be ‘enterprise’-architecture, the short-answer is ‘Probably no‘. What you’d end up with, at best, is something a bit like an odd mixture of random hype and rigid Taylorism – where endless attempts are made to force-fit not just the organisation, but everything that it touches, into line with the latest technology-laden fad. True, that kind of mess might even hold together for a short while, in some cases; but its internal contradictions will cripple it at best, and would more usually lead to catastrophic collapse, more likely sooner than later.
Yet to me, though, using the definitions and techniques that I do, the short-answer to Jeff’s question would be ‘Probably yes‘ – it can. The somewhat-longer-answer is that it’s definitely a qualified ‘Yes’, with an awful lot of ‘It depends‘ and suchlike… which is where the choices for those definitions and techniques really start to matter…
The first part of the ‘It depends’ in Jeff’s question relates to what’s meant by ‘complex systems‘. To some people – perhaps especially in the IT-related trades – ‘complexity’ is just a more extreme version of ‘complicated’: a quantitative difference, “complicated that we haven’t as yet quite pinned down the rules and algorithms for”. To me, though, I’d agree with those who argue that there’s a qualitative difference between ‘complicated’ and ‘complex’: for example, the kind of complexities that we see in wicked-problems, where even the act of looking at a context can itself change the context.
To use the SCAN diagram, the ‘complexity that’s just very complicated’ sits over on the left-side of the frame; inherent-ambiguity and other forms of ‘wicked-complexity’ sit more over to the right. That qualitative-difference between ‘very-complicated’ and ‘real-complexity’ in effect occurs at the point indicated by the red-line ‘boundary of effective-certainty’:
That boundary does sort-of move dynamically, and hence can give the illusion that it should be possible to push the boundary indefinitely to the right. In practice, though, there are real limits as to how far to the right ‘the complicated’ can go – for example, even just in a physics sense it’s limited by non-negotiable constraints such as the speed of light.
The second part of the ‘It depends’ in Jeff’s question relates to what’s meant by ‘enterprise‘. For this, I’d first point to one of the themes from the ‘manifesto’: that we create an architecture for an organisation, about its enterprise. In essence, ‘the organisation’ is what we want to architect – which, remember, includes both ‘order’ and ‘unorder’ – whereas ‘the enterprise’ is the effective outer-limit of the context for that ‘the organisation’.
It’s often useful to split the ‘not-the-organisation’ aspects of this ‘the enterprise‘ into two parts – ‘the market’, emphasising direct-transactions, and ‘the shared-enterprise’, more emphasising indirect-interactions – but that’s just a convenience, not a requirement. Anyway, here’s a visual summary of how I view ‘the enterprise’ in relation to ‘the organisation’:
To link this to the common ‘three architectures’ structure popularised in TOGAF and other mainstream ‘enterprise’-architecture frameworks:
- ‘organisation’: IT-Infrastructure Architecture / Technology Architecture
- ‘market’: Information-Systems Architecture (Data Architecture plus Applications Architecture)
- ‘shared-enterprise’: Business Architecture
[The crucial point to understand about that ‘three-architectures’ structure above is that in essence it is usable only for the architecture of IT-infrastructures: everything else is just context. In particular, what’s described as ‘Business Architecture’ is only peripherally about the architecture of the business-of-the-business: instead, it’s really just a descriptor of ‘anything else not-IT that might affect IT-infrastructure’. Trying to use this structure to guide an actual business-architecture is extremely unwise, because there’s then no means to describe the broader-context of that business other than as part of ‘the business’ itself – a common yet potentially-dangerous conflation of ‘the organisation’ and ‘the enterprise’. To make the ‘three-architectures’ usable for that purpose, we would need to recurse upward such the business-architecture becomes the ‘the organisation’ base-level for the structure, equivalent to IT-Infrastructure Architecture above.]
Crucially, anything beyond ‘the organisation’ is also, by definition, beyond our ‘control’. This can get especially complex where outsourced services are ‘inside’ the organisation’s boundary-of-identity – what others see as the effective boundary of ‘the organisation’ – but ‘outside’ the organisation’s own explicit boundary-of-control:
Within a well-constructed architecture, we do have plenty of options to influence the broader-shared enterprise – the ‘everything’ that’s beyond the organisation’s own boundary-of-control. But the notion that the organisation could simply demand that things happen, and they will, is definitely a delusion there (and even within the organisation itself, of course). In that sense, relations between ‘the organisation’ and ‘the enterprise’ will always be somewhat complex, somewhat emergent, even somewhat chaotic at times – and the architecture must be able to support that fact.
Hence the third part of the ‘It depends’ in Jeff’s question relates to what’s meant by ‘architected‘. One place we could usefully start would be the definition of ‘architecture’ used in the IEEE1471 and ISO42010 standards:
⟨system⟩ fundamental concepts or properties of a system in its environment embodied in its elements, relationships, and in the principles of its design and evolution
Sure, it’s a very structure-oriented view of architecture – there’s no reference to the narrative or story-aspects of architecture – and we have to hunt deeper into the standard to find other essentials such as the place of people in general. Even so, it’s probably usable enough for our purposes here.
Remember what I said earlier above that every real-world context will include elements ‘in’ all of the SCAN domains, and that, in turn, any architecture that relates to that context must be able to address all of those elements. So perhaps the simplest way to summarise what the ‘It depends’ here most depends on, is to use the categorisation of techniques and methods from the version of the SCAN diagram that I used in the ‘manifesto’:
That mapping gives us two keywords for typical assessment-techniques (‘rotation’ etc), and one keyword for typical decision-support deliverables (‘rules’ etc), for each of the SCAN domains:
– Simple (rules)
- regulation – reliance on rules and standards
- rotation – systematically switch between different elements in a predefined set, such as in a work-instruction or checklist (or in the elements in a framework such as SCAN itself)
– Complicated (algorithms)
- reciprocation – interactions should balance up, even if only across the whole
- resonance – systemic delays and feedback-loops may cause amplification or damping
– Ambiguous (patterns / guidelines)
- recursion – patterns may be ‘self-similar’ at every level of the context
- reflexion – intimations of the nature of the whole may be seen in any part of or view into that whole
– Not-known (principles)
- reframe – switch between nominally-incompatible views and overlays, to trigger ‘unexpected’ insights
- rich-randomness – use tactics from improv and suchlike to support ‘structured serendipity’
There are also a few other points that aren’t in that diagram, but are worth noting here. For example:
— We’d typically expect to do analysis in the Complicated domain (certainty-oriented, distant from the moment-of-action), versus experimentation in the Ambiguous domain: the former would expect to deliver a ‘knowable’ and repeatable result, whereas the latter quite probably would not.
— At the moment-of-action – where there’s no time to think, yet decisions do still have to be made – we’d expect a range of decision-support tools across the length of that horizontal-axis, ranging from step-by-step work-instructions over on the far left, to aircraft-industry-style action-checklists that bounce across the known / unknown boundary, to principles that can help keep decisions ‘on purpose’ even in the most extreme of uncertainty.
— Although it’s likely we would ultimately use all of these techniques and suchlike in every part of an architecture, we should expect to use ‘left-side’ techniques more often within the organisation, and ‘right-side’ techniques more often beyond the organisation, because the boundary of the organisation also represents its boundary-of-control, and hence the effective limits of its own certainty.
Remember also, from the ‘manifesto’, that all of the same concerns and principles apply to the architecture itself – including recursion, reframe, rich-randomness and so on.
Without this latter point about self-recursion and suchlike, then no, it’s probable that the architecture would not be able to work with, or even fully describe, any kind of complex-system.
Yet with this in place, and used appropriately, then yes, I’d suggest that we are able to architect any, and maybe even all, of the organisation’s structure, story and relationships with itself and the complex-systems within which it operates.
In other words, yes, we do need to accept and acknowledge that there’s a fair amount of ‘It depends’ involved here, as above: but given that, then yes, we can architect a complex-system – even as complex a system as a shared-enterprise.
I hope that adequately answers the question, perhaps?
Over to all of you to add your comments and suggestions, anyway.
Very important question. Thank you for this…..I think it is possible, provided everyone (in community) is involved and politically mature enough (e.g. radically optimistic enough) to transcend possible tyranny of hierarchical management. We managed to achieve holarchy with teams of carefully matched designer-types in several experiments.
Many thanks for this, John.
Also looks like it might be useful to meet up somewhen, if only online? – many parallels between our approaches to the work (including the ‘tetrahedra’ theme). Over to you on that, anyway.
Hi Tom – Enjoyed your recent group of posts.
I’m not sure isolating the quadrants as separate approaches/solutions is completely accurate. They really are a continuum. Further, I’d suggest that Recursive/Reflective strategies are really about interpretation, which allows variable solution strategies based on the reasoning.
Recursion/Reflection are also software concepts that support similar ends. They were implemented in SmallTalk to limited effect, but we are extending these concepts in Cloud/Web architecture.
Best Regards,
Dave
Hi Dave – thanks, as usual. 🙂
@Dave: “I’m not sure isolating the quadrants as separate approaches/solutions is completely accurate.”
They’re not ‘isolated’, at all – that’s the whole point about a framework that’s designed to apply to itself. If we take a Simple categorisation-type view, then yes, they can be viewed as isolated quadrants; but in a Complicated view, the boundaries move (as described explicitly in the post above – “that boundary does sort-of move, dynamically”); in an Ambiguous view, the boundaries are kind of porous anyway; and in a Not-known view the boundaries are there and not-there at the same time. Remember that kind of recursion here – it’s important.
Interested to hear more about “we are extending these [recursion etc] concepts Cloud/Web architecture” – any pointers/links on that, please?
Hi Tom,
Your blog is one of the few places I’ll ‘go there’ for the same reason you articulated below – “because I confuse people enough already by often starting from places where they aren’t and assumptions that they don’t (yet) hold.” 🙂
Here are some articles and conference papers –
http://www.infoq.com/interviews/duggal-ideate
http://www.infoq.com/articles/social-lean-agile/
http://www.scribd.com/doc/48094484/Putting-Work-in-Context-Duggal-Malyk-COCOA2010-120610b
http://www.slideshare.net/dduggal/consilience-intl-ws-rest2011-resourceoriented-frameworkfinal-abr
plus our website http://www.enterpriseweb.com – we have a series of whitepapers coming out over next few months
They touch on the nature of our recursion without diving deep – we try to make it accessible, albeit the details are diluted.
I saw your point regarding dynamic movement, but we’d start at far right and move left as appropriate for ‘best-fit’ of problem-solution space. This way affordance of our System is optimal and utilization is contextual.
At some point, let’s catch up on a call, as your conceptual models and our IT implementation are very well aligned.
Best,
Dave
[Posted by Tom Graves on behalf of Charlie Alfred, via email – if anyone else has difficulty with the comments-posting mechanism on this blog, please let me know?]
Nice post. I agree with your views on enterprise vs. organization, complicated vs. complex and controllable vs. influence-able. I thought you did a good job of communicating them.
I thought that your description of architecture was consistent with the prevailing view. In other words, architecture as a mechanism to design the structure and behavior of a system.
I’d like to suggest that you have an opportunity to think/talk about architecture with a broader perspective that’s consistent with your other points of view.
Definitions:
Scope is the boundary around an intentional system for which a person has responsibility, and the ability to act with both decision making authority and influence.
Architecture of the Solution: architecture focused on making significant decisions (about structure, capabilities, resources, behavior, etc.) *within* the scope of the system.
Architecture of the Problem: shifts the focus so that the system scope is one element in a larger ecosystem. The perspective here is on discovering one’s role and responsibilities. This involves partnering with some stakeholders in order to create value for other stakeholders in the ecosystem.
The Architecture of the Solution determines *what* the system’s capabilities, resources, and priorities will be.
The Architecture of the Problem determines *why* these assets matter and against *which* challenges they will be deployed.
Clayton Christensen writes “consumers hire a product for a job.” Here, “job” is more than a function, it is a function plus a challenge. I could drive from Boston, MA to Chicago, IL using my car. It would be convenient and would take 15 hours. Southwest Airlines reduces the travel time by 75%. My challenge is Southwest Airline’s opportunity.
The architecture of the solution includes how Southwest plans its flight schedule, trains its flight crew, acquires and maintains its air fleet, etc.
Thanks, Charlie.
@Charlie: “I thought that your description of architecture was consistent with the prevailing view. In other words, architecture as a mechanism to design the structure and behavior of a system.”
Uh, yeah, you’re right – and in part that was (necessarily?) the intention, because I confuse people enough already by often starting from places where they aren’t and assumptions that they don’t (yet) hold.
Your subsequent description of a view more aligned with how I probably should describe it – yes, very much so. They only bit I’d add is that (especially in a wicked-problem or similar complex-system) there is no explicit separation of ‘problem’ and ‘solution’ – they interact so much with each other that they need to be understood as ‘the same thing’.
(I do like your point about “it is a function plus a challenge”, though – I like that a lot. 🙂 )
[Posted by Tom Graves on behalf of Charlie Alfred, who’s still having difficulty posting here – we still don’t know why.]
I urge you to use your broader view of architecture, even at the risk of confusing people. It’s a lot more consistent with how you (and I) describe the overall process. Either that, or we need to ditch the word architecture in favor of another word that won’t have double meaning.
I agree that architecture of the problem and solution are a continuum. What separates them is the choice(s) made be the organization about what challenges to address and how.
If this process is truly done in a continuum, then it is much harder to prioritize and assess the impact of solution decisions on each other.
Every decision made in solution space creates new constraints on subsequent decisions, which is why it is so critical to make decisions on the highest leverage challenges as early as possible.
Finally, the thing I like about challenges is how they trace from problem to solution spaces (i.e. what are my main challenges and how do they link to the challenges of the contexts I want to serve.
What is the thinking behind the selection of complexity as a function of time over certainty?
It does rather give the impression that levels of disorder are a psychological product. That doesn’t scan (no pun intended) with the characterisation of complexity as a state of affairs distinct from complication or confusion.
How does one make an objective assessment of the levels of certainty that pertain to any organisational system (leaving aside for a moment the deeply interesting question you have discussed about where to draw the boundaries of the system itself)?
Ouch – Ric is back… 🙂 (and yes, I do very much mean that as a compliment! 🙂 )
This one’s going to need a separate post (again…). But a possibly-useful-for-now short-answer would probably be as follows:
“selection of complexity as a function of time over certainty” etc – This suggests that I haven’t explained clearly enough the real role of SCAN. It’s not that it claims to describe where something ‘is’ – i.e. that ‘so-and-so is in the Complex domain’ – but more about where our understanding (sensemaking) and response (decision-making) would reside (or, more precisely, transition through). In that sense, it’s entirely correct to say “It does rather give the impression that levels of disorder are a psychological product” – it’s about (and/or includes) perception of ‘disorder’, not some purported ‘absolute measure’ of ‘disorder’.
Which brings us to “How does one make an objective assessment of the levels of certainty that pertain to any organisational system” – the short-answer is that we don’t. Almost by definition, ‘objective assessment’ is somewhere between unlikely and impossible within a sociotechnical complex-system. The purpose of a framework such as SCAN or CLA (Causal Layered Analysis) is to give us a means to work with that inherently-problematic fact, rather than pretend that it doesn’t exist (as in most existing ‘enterprise’-architecture frameworks).
More later, anyway – and thanks again for the challenge!
I like this statement: “Almost by definition, ‘objective assessment’ is somewhere between unlikely and impossible within a sociotechnical complex-system.” and yet, it is possible to create a consensus based ‘subjective assessment’ where all people in the room agree to the statement, which is close enough to objective to be really useful, because the involved people are buying into the assessment. You have written yourself about it in the “SCAN as decision dartboard” post (http://weblog.tetradian.com/2013/06/15/scan-as-decision-dartboard/) which I use by (mis)using SCAN to come to a fast assessment on Past, Present and Future to drive decisions in the needed speed (typically time boxed to one hour, plenty challenges to look at in that one hour). I do though never try to look for the perfect truth, just a barely good enough version works.
I think, that variants 3, 4 and 5 (see http://improving-bpm-systems.blogspot.com/2010/12/illustrations-for-bpm-acm-case.html) will handle it. Of course, variant 5 is the best choice.
Thanks,
AS
Thanks, Alexander – and yes, agreed re those variants in your diagrams.
I’d suggest, though, that it might go even further: your Variant 5 shows a ‘futures’-type view, yet what I’m looking at is more interactive than even that – kind of proactive + feedforward + feedback + principle-driven intent? There are some real elements where future can sort-of directly interact with both past and present – very difficult to describe without going kinda ‘mystical’ or suchlike, though… 🙂
Places are the elements that directly connect the past, the now and the future. See the book “Consensus Design socially inclusive process by Christopher Day” for the architecture point-of-view and the “Healing the land, healing ourselves” article at http://pishwanton.org/files/2010/10/Caduceus-Article1.pdf by Margaret Colquhoun for the more mystical point of view 🙂
Thanks, Tom. Trying to translate your idea into my illustrations – do you mean that the other (i.e. non-selected) scenarios can be changed during execution of the selected scenario?
Sure, the future can interact with the past and present – something like “if you said A then say B”.
Thanks,
AS
@Alexander: “do you mean that the other (i.e. non-selected) scenarios can be changed during execution of the selected scenario?”
Sure, they can do – that’s part of the nature of a complex system.
Simple example: a little-known technology starts to get taken up, or an ‘unimportant’ social-network takes off in a hurry – that happens quite a lot. You’d ignored it before, or noticed it and did a quick contingency-plan but left it at that, now you have to switch over to it fast, regardless of what you’d planned to do.
An opposite example: for your phone-management, you looked at both Apple and Android, your business committed itself to Blackberry, but a year later the company hits the skids – you now need to revisit your choices.
In both cases the context for all of the other options continues on in the background, with or without your knowledge or intervention. One of your diagrams in the ‘variant 5’ set sort-of covers this, but doesn’t show how the trade-offs can keep changing, in some cases so much (as in both those examples) that you need to switch options in mid-stride.
Your blogpost was mentioned in http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&discussionID=5812680199795388416&gid=70120&commentID=5814708677097070592&trk=view_disc&fromEmail=&ut=0mBdLrr-x0A601
Thanks,
AS