Enterprise-architect – applied-scientist, or alchemist?

Is enterprise-architecture more akin to alchemy than applied-science? And if so – and to what extent as so – what would this imply to how we study, discuss, teach, learn and apply enterprise-architectures in real-world practice?

Kinda suggests perhaps, that we might think of our organisations and enterprises less in terms of the IT-obsessive’s over-conceptual cloud, but more like the grime and dust and uncertainty of the late-mediaeval alchemist’s fire-and-fume-filled ‘manufactories’…

What triggered this off was a comment by Alexander Samarin to my post ‘‘Which EA school do you belong to”‘:

RE “And that’s because the real point here is that by definition, real-world EA is not something that could ever fit comfortably into an ‘academic’-style model.” – We are still alchemists who are, fanatically, doing theory, practice and methodology at the same time. Unfortunately, we have to end this to make EA as an applied science.

I’d agree that in EA we are often “doing theory, practice and methodology at the same time” – and perhaps too often “fanatically” as well… Yet do we have to end this? Or even should we end this, to move to something more like ‘applied science’? I’m not so sure.

Part of the reasons for those doubts I’ve previously described in the post ‘The science of enterprise-architecture‘. In essence, it’s a straightforward SCAN-type trade-off: the more repeatable a context is – over to the left side of the SCAN frame – the more value a classical ‘science’-type approach would deliver. Yet as repeatability drops, so does the value of the ‘science’ – so much so that once we cross over SCAN’s ‘boundary of effective certainty’, classical approaches to ‘science’ increasingly become more of a hindrance than a help:

An enterprise in which everything is perfectly predictable, under strict ‘scientific’ control: that’s the ideal that Taylorists and IT-obsessives drool and dream about. Unfortunately for them – and perhaps fortunately for us – it’s a fantasy, a delusion: it’s simply not feasible in any real real-world context. Reality Department is a lot messier than that – and we need to work with that fact, rather than pretend that it doesn’t exist.

But… alchemy? Are you kiddin’ me? I mean, that’s been discredited long ago – utter nonsense, isn’t it, unscientific rubbish, like astrology and all that stuff?

Hmm… Yeah, this is where this gets kinda difficult…

Let’s take a real historical example. Isaac Newton has just about the most solid scientific credentials one can get: his Principia pretty much laid the foundations for much of modern science, certainly right up into the middle of the last century, and much of it well into today. All of that is very well-known indeed. Yet what’s perhaps less well-known – though now described in some detail in Michael White’s biography Isaac Newton: The Last Sorcerer – is that Newton studied and wrote far more on alchemy and astrology than he did on what we would understand today as ‘science’. His studies there directly informed and are interwoven with his science – they’re all of a piece, and can’t really be separated one from the other without destroying the sense of the whole. And, in a sense, he did discover gold, too: in his later life, he was appointed Master of the Mint (manufacture of state coinage), and changed the de-facto basis of the currency from silver to gold – the so-called ‘gold standard‘.

However strange those studies might sound to us in the present day, Newton was by no means unusual, in that sense at least: many of his contemporaries did much the same. And it should certainly not be forgotten that all of that detailed experimentation and analysis for alchemy and astrology provided key foundation-stones for what we’d now know as chemistry and astronomy respectively; likewise the formal basis and practices for ‘experimental method‘ that underpin much of the science of today.

Okay, that’s in the past: but alchemy now? You’re joking, right?

Actually, no, I’m not joking at all. I’d think immediately of a close friend back in Australia: his day-job was as the lead systems-administrator for a fairly large IT-department, but back at home he was a practising alchemist – and had been so for more than thirty years. When I last visited him at home, his alchemy-lab, in a purpose-built building out the back of his house, looked exactly like what any industrial chemist would know and use – but what came out of that lab was definitely different. For example, most people would know pure copper-sulphate as bright-blue crystals, or a bright-blue solution; but his copper-sulphate – equally pure – was bright-green, and with some significantly-different chemical-properties, too. Seems, then, that Reality Department still ain’t quite so sewn-up and certain as some scientists and self-styled ‘Skeptics’ would have us believe…

In fact, there’s quite a thriving alchemical community worldwide, typically basing their work on classics such as Michael Maier’s 1617 opus Atalanta Fugiens. Except to call it a community is stretching it a bit, because one of the fundamentals of that not-really-community is that they don’t share ‘best-practices’ and the like – in fact they’re a notoriously secretive lot, often to extremes. In part it’s because some of the practices are necessarily personal (I, uh, won’t bother to explain why…). But also, I asked my friend whether he shared and discussed his processes and results with other alchemists – to which he snorted, almost derisively, “Of course not! I’ve spent more than thirty years on this: why on earth would I give it away to anyone else?” He might eventually take on an apprentice or two, he said, especially if or when he got closer to developing genuinely-useful results for the health context, but that was about it.

For enterprise-architects, and especially for business-architects, some of this may already start to sound all too familiar: the effects and impacts of local-distinctiveness, of individual, collective and corporate culture, of ‘intellectual-property’ and ‘trade-secrets’ and all that… Recognise it? Yeah, you do, don’t you? 🙂

In SCAN terms, the more repeatable that something is, and the more it’s openly shared, the more it moves us over towards the left-side of the frame, towards ‘applied-science’. Yet the more unique it is, and/or the more secretive and private it is, the more it moves over to the right-hand side of the frame – and towards something that looks a lot more like alchemy than ‘applied-science’.

And the reality is that a lot of what happens in organisations and enterprises necessarily sits well over towards that right-hand side of the frame – whether we want it to, or not. Which means that – perhaps somewhat strangely, in this so-called ‘scientific’ age – there’s a lot we could learn from the old alchemists.

Alchemy Lesson #1 might be plain old health-and-safety, of course. If we’re going to be mucking around with things that, metaphorically speaking, can go off ‘BANG!’ in our faces, it’s wise to have a fair idea of how to handle it…

[When Isaac Newton died, he was found to have seriously-high levels of mercury in his system, accidentally inhaled during his alchemical experiments – and mercury can do really nasty things to the brain in particular (hence the Mad Hatter). Somewhat explains Newton’s notorious irascibility in later life, in fact.]

But it’s kinda tricky to get that ‘fair idea of how to handle it’ when everything’s somewhat unique and people are being secretive about it all. Somehow, we have to get it over to people that they need to tell us about anything that’s a potential hazard – and that includes things that can be hazardous in not-so-obvious ways, such as at risk of damaging trust.

Alchemy Lesson #2 is about use of patterns or metaframeworks, rather than mechanistic ‘recipes’. We need to work with the uniqueness, rather than pretend that it doesn’t exist – and patterns provide us with a means to bridge between repeatability and uniqueness. In a sense, our equivalents of the old alchemists’ classics are the frameworks we know and sometimes love, such as TOGAF and Zachman and the like. The catch is that we need to think of them more as patterns, not ‘frameworks’: whenever we use them, we need to keep the focus on ‘adapt, then adopt’.

[I remember a session at an Open Group conference where one of the TOGAF leads stated flatly that “no-one uses TOGAF out-of-the-box – it’s always adapted for that organisation”. There’s some material on this in the TOGAF specification, but it perhaps needs a much stronger emphasis than at present – though that’s part of what training-courses are supposed to deliver, of course.]

Alchemy Lesson #3 is about the real need for precise, disciplined observation. What we’re really aiming for here is best described by the German military term fingerspitzengefühl, literally ‘finger tips feeling’, a continuous, overarching sense of the whole as whole, a knowing of where the fine-detail matters, and where it doesn’t. This is about more than just looking, more than just digital-information: often it requires embracing our own inner-‘weirdness‘, using all of our senses and more.

We don’t get this from studying just the theory: in fact – as the history of alchemy proved all too often – if we get too hung up on theory, and deviate too far from real-world practice, that’s when things go badly wrong. The 16thC alchemist and astrologer Tycho Brahe perhaps best illustrates both sides of this: many years of precise observation – far more accurate than any of his European contemporaries – that established him as one of the first ‘real’ astronomers in the West, yet also clinging on to Aristotelian theory and cosmology until way too far beyond its ‘use-by date’. Again, enterprise-architects and suchlike will no doubt recognise much the same latter symptoms in their own organisations… especially in terms of Taylorism and technology-centrism.

Perhaps the most important thing here is to notice things that don’t fit our expectations – they’re often very easy to miss, especially given Gooch’s Paradox, that “things not only have to be seen to believed, but also have to be believed to be seen”.

Alchemy Lesson #4 is that we need to work consistently, to allow time for things to mature. Tycho Brahe again illustrates this well: decades of precise observation, every night, without fail, eventually delivering a dataset that was large enough, accurate enough and consistent enough to form a key foundation for what we now think of as the science of astronomy.

For enterprise-architecture, the key point here is that – despite the sales-pitch of those darned big-consultancies – little to no real value will be delivered if we treat the development of the architecture as a ‘one-off project’. Instead, it’s about real-world practice, day after day, month after month, year after year, working with the whole-as-whole. If we’re too uncomfortable with the idea of alchemy, we could learn a lot from the ‘down-and-dirty’ DevOps crew: they’re doing this right now much better than most of us have ever done…

Alchemy Lesson #5 is about ‘just enough documentation’. In a sense, ‘just enough documentation’ – or Just Enough Detail – is another variant of fingerspitzengefühl: we need enough documentation and detail to keep our finger on the pulse, but not so much as to slow us down. It’s a tricky balance…

One of the reasons it’s tricky is because of Lesson #4, about working consistently over long periods of time – certainly for years, quite possibly for decades. Over those kinds of timescales, memory alone will not suffice: we’re going to need to write it down, or document it in some other way. And in ways that we can still access and search years or decades later – which in these current times of rapid obsolescence is not as as easy as it might at first seem.

We also need to remember that whilst scribbled sketch notes might well be adequate if it’s just for ourselves, and just for a handful of years at most, it definitely won’t be enough if it’s across an entire enterprise or industry, and over decades or more. That’s where the theme of patterns comes into the story: it’s often a lot easier to document the adaptation of a pattern, than trying to document everything. Yet it’s also where we come across the next ‘lesson from alchemy’…

Alchemy Lesson #6 is that some secrecy is often unavoidable, and even necessary. Every trade has its own ‘trade-secrets’; most businesses have some kind of ‘secret sauce’ behind their story (or would like to have such, anyway). And there are often very good reasons for this, beyond just protecting and privatising a source of monetary income: for example, many domains are just too darn dangerous, in many different senses, to risk making everything fully open.

When I asked my friend about the cryptic nature of alchemy-classics such as Atalanta Fugiens, he reminded me that much of what the old alchemists worked with was still very little-understood in the modern sense, yet often either seriously-poisonous (mercury was one of the milder problems…) or straight-out explosive – potentially more than just dangerous, not just for them, but for everyone around them. Making things cryptic was a way of slowing people down ‘just enough’ whilst they developed the basic skills and their own essential safe-practices. We still see the same in present-day health-and-safety hazard-management: we don’t let first-level apprentices play around with power-tools, for example, let alone poisons or explosives.

And some contexts require a subtle form of secrecy just to do their job. In a telco, a customer-service unit, and a social-work group, I’ve seen coded tags for client-records which might seem innocuous-enough on the surface, but actually act as warnings to the initiated: ‘VIP’, for example, which meant ‘difficult customer’. In each of those cases, it was a simple form of protection against lawsuits and suchlike, in case the record needed to be shown to the client under Freedom Of Information laws, but no doubt you’ll have much the same in many other places and contexts in your own organisation and beyond. Secrecy for its own sake is a darn nuisance, or worse, but there are times where it is essential: so in our enterprise-architectures, we’ll need not only to document and support those cases, but also why they’re needed.

Alchemy Lesson #7 brings together all of the above, and is about accepting that we are, of necessity, “doing theory, practice and methodology at the same time“. Whenever and wherever our work passes over to the right-hand side of the SCAN frame – the Ambiguous or the Not-known – then by definition theory alone will not be enough: which by definition places us outside of a purely-predictable model of ‘applied-science’. In practice, ‘applied-science’ actually bounces back-and-forth across the ‘boundary of effective-certainty’, from imposed-certainty to inherent-uncertainty: in effect, the ‘science’ part on the left, and the ‘applied’ part – or finding out how to apply it – over on the right. But when the theory we’d need doesn’t really even exist, and we’re forced to work things out as we go along – which happens a lot within our enterprises, especially where any kind of innovation or experiment or one-off instance is involved – well, yeah, that’s where we’d be in a world that works much less like ‘applied science’ and lot more like alchemy: creating our own metaphoric form of the Philosopher’s Stone from little more than skills and hope and that strange awareness of fingerspitzengefühl.

Then there’s the other SCAN dimension to consider – the its ‘vertical-dimension’ transition from theory to practice, from plan to action in the real-time ‘NOW!‘, and back again, and back again, continually, recursively, fractally. The point here is that – just as with present-day DevOps and the like – alchemy wasn’t about abstract theories: it was about doing something, making something, in real, tangible form. And when we look at the current practice of enterprise-architecture, I must admit I get very frustrated when I see self-styled ‘enterprise-architects’ dismiss all implementation as Somebody Else’s Problem, as the role of ‘lesser beings’ such as solution-architects, developers, process-designers and operations-staff. Apart from being extraordinarily insulting, it also demonstrates a serious lack of awareness or understanding of how things actually work – because it’s only when we get things working in the practice, in the real world, that our work has any real value at all.

So yeah, don’t be quite so quick to hide away in illusions of ‘applied-science’: much of our work does, of necessity, more closely resemble alchemical practice – an alchemy of the enterprise.

And if that’s the case, might it not be a wise idea to get good at doing it?