The stench of systemic decay

It was the smell that caught my attention first, I guess – the smell of chemicals as I walked through through the front door of their supposedly upmarket offices. But it’s something I’ve come to recognise, to watch for, as a first crucial warning – not so much the smell itself, but more just a feeling of wrongness, or lostness, of things coming apart at the seams but that can not yet be seen as such with the visible eye. A quiet, subtle precursor to catastrophic collapse; the stench of systemic decay.

That was way back now, almost three decades ago, when I was still working in the pre-press industry. A company was having quality problems with their typesetting, and had asked me to assess their overall operation and give some advice on what to do. And that’s when I first hit up against that stench – in an all too literal sense, in that context, hit in the face by the reek of photochemistry as I came through the door.

I’d long been an admirer of the work of Gerry Weinberg, and his advice in his book The Secrets of Consulting that “I always get the answer in the first five minutes – but it can take me hours or days to recognise what it was that I saw in the first five minutes”. That was exactly true there: the smell of photo-processing chemistry was a key indicator for their whole quality-problem.

As typesetters, they were excellent: their design-sense was superb, their attention to language and placement was almost unparalleled, their handling of the IT side of the business was second to none. Yet when it came to dealing with anything physical, they were pathetic, appalling, worse than useless. The rolls of photo-paper – often each worth thousands of dollars – were allowed to scrunch up and glue themselves together irretrievably on the output side of the photoprocessor: but the despatchers sent the rolls out to the clients in that state anyway, blithely unaware that they were unusable. The stench that had hit me coming through the door was because the processing-chemicals hadn’t been changed in well over a week – they should have been refreshed every day – and the processor-rollers were filthy, so bad that they frequently left long streaks down the photo-rolls: but again, no-one noticed. Yet the clients definitely did: and hence work was coming back, and back, and back, to be redone, again and again and again. At huge expense in cost, time and reputation.

Once they’d reviewed my report, the management pretty much ‘read the Riot Act‘ at the operators; and it did make a difference, for a while – perhaps a week or two. But after that… well, they soon slid back to their old ways of (not) doing things. And that loss of quality eventually killed the whole company, only a month or two later.

The real problem was that the operators couldn’t see the system as a system: only the IT, which they liked, and therefore did, with pleasure, and the not-IT, which they disliked, and avoided as much as practicable, on the basis that it wasn’t part of their work and hence didn’t exist. They regarded themselves as ‘computer people’, and only computer-people: they viewed it as kinda beneath their dignity to deal with anything physical, like rollers or chemistry – to actually risk actually getting their hands dirty… – yuk!!! And they could always find a way to make it Somebody Else’s Problem – which meant that it just didn’t get done. With disastrous results, as above.

In short, if we don’t view a system as a system – as a unified whole – then we all but guarantee failure.

Yet if we watch for them, we can get crucial hints that the system is at risk – that subtle stench of the early signs of systemic failure.

And the key one of those signs is evidence of a failure to view a system as a system – in particular, a tendency to view one subset of a system as the ‘whole’ of that system, and myopically ignore the rest.

Which is why it so much worries me that enterprise-architecture is still so much dominated by IT-centrism and ‘IT-only-ism’, such as I described in the section on ‘certification and professionalisation’ in my previous post.

As enterprise-architects, we need to hammer home the blunt fact that there is far more to an enterprise than just its IT – and that an ‘enterprise-architecture’ cannot, should not and must not be described as enterprise-architecture unless it does explicitly cover the whole of the systemic scope in context.

For example, even something as seemingly focussed on IT as a data-centre always involves much more than the IT alone – and failure to view that overall system as a system will invariably have unfortunate consequences. Consider the huge new NSA data-centre in Utah, for example, plagued with still-unresolved ‘arc-failure’ electrical faults in its power-supply; or consider what can happen with cooling-systems:

Few data center operators understand the biology and chemistry of open or closed loop cooling systems. … Each spring and fall the system would experience water flow blockages, so a chemical cleaning agent was added to the pipes to remove scale build-up. Before the cleaning agent could be diluted or removed, the heating system was turned on. Thanksgiving night, the 4-inch lines let loose. Chemically treated 180-degree water flooded down 26 stories of the tower. Because no one was on site [who] knew how to shut the system down, it ran for two hours before being stopped.

Again and again we see the same distinctions:

• companies and architectures that do think in terms of whole-systems do plan for and survive even severe disasters
• companies and architectures that don’t think in terms of whole-systems don’t understand what’s actually possible or actually going on, hence don’t plan, and sometimes don’t survive – or at the least, will cause significant disruption.

Watching all those presentations on ‘certification’ and ‘professionalisation’ at Open Group London the other day, I kept on sensing that same stench of systemic-decay. I should emphasise, though, that it’s not just TOGAF or Archimate: virtually all of what currently passes for ‘enterprise-architecture’ is riddled with same obsessively-myopic IT-centrism, from top to bottom – and what little is not, most of that is now beginning to be infected by a perhaps even more dysfunctional ‘business-centrism’. It picks out one arbitrarily-selected domain as ‘the centre’, and blocks out the view from any other domain – destroying any possible sense of system-as-system. And doing that guarantees disaster – maybe sooner, maybe later, but every single time.

So let’s repeat this again:

— The only way that any architecture will work is when it is able to address (and does address) the system as a unified whole – system-as-system.

— In any viable architecture, everywhere and nowhere is ‘the centre’, all at the same time.

And unless those bald facts are embedded right into the deepest roots of every aspect of our enterprise-architectures – well, watch out for that subtle yet sickly stench of systemic decay…