Enterprise Debt and the Shirky Principle

Just how much are organisations themselves ‘their own worst enemy’ for the enterprise?

Have been thinking about this one for quite a while, following up on some great conversations with Kevin Smith (of PEAF fame) and Nigel Green (of VPEC-T fame) about Kevin’s concept of Enterprise Debt – an expansion into the whole-enterprise scope of Ward Cunningham’s concept of Technical Debt:

Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation.

In classic IT-oriented ‘enterprise’-architecture, Enterprise Debt occurs with any ‘quick-fix’ IT solution, or any solution that breaches the architecture guidelines. A lot of ‘shadow-IT’ fits into this category, of course. But exactly as with a credit-card, for example, the ‘debt’ doesn’t matter much if it’s cleared quickly: short-term debt is rarely much of a problem or risk, and if managed well does help us to achieve short-term goals. Likewise for longer-term debt, such as a mortgage or a car-loan: as long as we know that the debt is there, and that we’ve allowed for it and managing it, it’s not a problem. Yes, it’s a risk, but it’s a managed risk, and one that allows us to reach towards our longer-term goals. So, to take the metaphor back to IT-architectures, a large-scale solution that breaches the architecture-principles may indeed be a risk – but it’s probably not too much of a risk if we’re aware of it, are managing it and do have specific plans in place to ‘write down the debt’.

For enterprise-architectures and any of the other non-IT-oriented architectures, we take the same principles, and adapt them to the respective scope to identify and address the overall Enterprise Debt and concomitant risks. It’s really no different to Technical Debt, other than in its scope: the tasks to identify and address it are certainly much the same.

So far, so good. But it’s the hidden debt – the debt that we don’t notice or don’t know about – that can be the real killer here. Because we don’t know or notice that it exists, it keeps on building up and building up in the background, until suddenly the metaphoric bill is presented, and we have little or no way to pay it off. A classic example was Y2K – otherwise known as the ‘short-term thinking bug’ which comes back to haunt us again and again. In that context, the use of two-digit dates was a design decision that was sensible enough given the technical constraints of the time, but whose short-term expediency was never redressed, and came back to bite us big-time… a good example of a kurtosis risk.

In enterprise-architectures and the like, there are many sources for hidden risk of this kind, often quietly building up quite terrifying amounts of Enterprise Debt. Some examples to watch for:

• small ‘knock it up over a weekend’ applications and databases becoming used in business-critical processes [no documentation, lack of links to enterprise standards]
• a database for a single team, developed using office software tools such as MS Access or cloud apps, becoming used across a wider spread [security concerns, database stability in a true multi-user context]
• business-processes continuing indefinitely without review [loss of ‘fitness for purpose’, high efficiency but low effectiveness]
• anything that depends on or assumes specific silo boundaries or organisational structures [because those boundaries or structures may change at any time]
• anything that depends on a single individual or small team [serious maintainability-risks, risk of inability to review design-decisions]

We can tackle some of this with proper governance that provides an appropriate balance between the competing needs for agility and stability, such as in a ‘backbone’ governance-model. We also need a long-term view – often reaching out into decades, for many types of business-critical information and processes – and strategies and tactics for systematic ‘entropy-reduction‘ in the overall architecture.

To me, this is one of the most important tasks of an enterprise-architecture unit: addressing the issues of the past so as to support the needs of the future.

Yet there’s a really nasty booby-trap here, a huge source of hidden Enterprise Debt, of which we definitely need to be aware at every level, especially at an enterprise-wide scope. It’s a variant of Parkinson’s Law, often nicknamed the Shirky Principle after Clay Shirky’s aphorism: “Institutions will try to preserve the problem to which they are the solution“. This is a classic wicked-problem whose symptoms include:

• people whose incomes and identities depend on ‘the problem’ not being resolved [very common in social-work contexts]
• ‘hero’-managers and other ‘fire-fighters’ whose prestige depends on having metaphoric fires to fight [hence have vested-interest against preventive strategies or tactics]

To say this is ‘political’ is an understatement… 🙁 And unfortunately, by its business-anarchist nature, enterprise-architecture tends to highlight the inherent absurdities of these kinds of situations – which means that, when accidentally exposing this, architects themselves often tend to be attacked, often with extreme ‘irrational’ anger, in a classic shoot-the-messenger ‘defence’.

This does mean, though, that many organisations – especially in the government or non-profit sectors – are actually ‘their own worst enemy’. The people, the processes and even the structures will often in effect act to perpetuate the problem they purport to resolve. Tricky… especially for the enterprise-architects who work in those organisations…

There are some practical options, though, for enterprise-architects unfortunate enough to be faced with that kind of ‘mess’. These include:

• be careful never to describe any of this as anyone’s ‘fault’ – it’s a natural tendency at the systemic level
• use narrative or story-based methods such as SEMPER or Causal Layered Analysis to ‘surface’ the structural issues
• always reframe motivations and requirements in terms of being ‘for‘ something – never ‘against’ something

As enterprise-architects, we have to deal with the painful realities of Enterprise Debt as best we can. But first we have to learn to become aware of it: and warnings such as the Shirky Principle can be a lot of help in this.

Hope this helps, anyway – comments/suggestions, as usual?