From tools to toolset
Okay, I’ll admit to a certain amount of frustration here…
Yes, I’ve been talking a lot about toolsets for enterprise-architecture and the like, the ‘Why’ for a new type of toolset, the workflows within which such toolsets would appear and how those toolsets could be used within such workflows, the overall toolset-ecosystem from handheld to ‘war-room’ and more – real architectural stuff like that, of real, direct, personal importance to enterprise-architects and their work.
What I’ve also described in these posts is a set of interlinked, interdependent requirements that, to my knowledge, no existing software-based toolset or data-structure can currently satisfy.
It’s a challenge that, as enterprise-architects, we really need to resolve – if only as a demonstrator of ‘eating our own dogfood’ or the like.
Instead, much of what I’ve been getting back seems to consist of variations on a theme of “this app is The Answer, you should have known that already”, “this toolset is wonderful!”, “I really like this tool”, “you could do it all with RDF/OWL”, “it sounds like you’re trying to recreate MOF”, “why don’t you just use Visio?”, “you can describe it all with ontologies” – on and on and on and on.
All of which is no doubt well-intentioned – and I do appreciate that, and thank you.
Yet all of those suggestions completely miss the point…
Yes, each of those tools or whatever might well do a good job on some part of the requirements – but not the whole. Yet because the requirements are tightly interlinked and interdependent, what I’m describing will only work when we get the whole working together as a whole. Anything less than that will merely make things worse, adding yet more to the current mess of fragmentation across the toolset-space. Not helpful.
But yeah, okay, I do acknowledge that if the message isn’t getting through, in part it’ll be because I haven’t explained it well enough.
In other words, my fault.
So, okay, let’s start again. One more reframe, in the hope that it might make more sense this time.
(If it doesn’t, we can always park it back on the back-shelf for another few years, wait until there’s a better time to try again. The catch for me, of course, is that I may not be around to do much about it by then. Oh well.)
In essence, here’s a real question to start from: what’s the difference between a bunch of tools, versus a toolset?
Or, perhaps more usefully, what do we need to change a bunch of disparate tools into a unified toolset?
At the moment, those questions might sound a bit abstract. So let’s make it more concrete, more immediate, more practical. Right now, go to any place where you have a bunch of tools that are related in some way, towards some shared task – the kitchen, the garage, the sewing-room, the garage, that storage-space under the stairs, maybe the hardware-store:
And ask the same questions again: what is it that would make this disparate bunch of tools into a unified toolset?
There are two answers that I’d hope you’d get straight away:
- yourself – the fact that you use them
- the task – the purpose towards which you use them
Yet there’s another answer that’s a bit more subtle, that a lot of people might miss first time:
- at least one of those tools acts as a platform for the others – providing a space or context through which the other tools can work together
What’s interesting is that often those ‘platform’-tools will seem to be almost invisible. Things like the work-bench in the garage, the embroidery-frame in the sewing-kit, the chopping-board in the kitchen:
Often in a quite literal sense, these are things that hold things together, to be worked on by other tools.
In short, the short-answer to those questions above is ‘a platform’. Yet that platform itself often doesn’t look much like any of the tools that are used on that platform, that unifies those tools into a unified toolset – an interesting characteristic that we could perhaps note for later…
The distraction is that there’ll also be quite a few sets of things-that-work-together: the power-drill and its set of attachments, the food-processor likewise; the vacuum-cleaner likewise. Yet notice that whilst we’ll often hear such things described as a ‘toolset’, in most cases the attachments will only work with the respective often-very-much-proprietary ‘parent’-device – and they often won’t play nicely with anything else at all. (Apple and your wretched five-sided screws, we’re looking at you…) So yes, whilst the parent-device provides a platform of a kind, it isn’t the platform that we need that would hold all of those disparate tools together, in the toolshed or wherever. Each of those is a sort-of-toolset, a sort-of-platform – but we’d need to go a bit more ‘meta-‘ from there, a couple more recursions at least, before we’d reach the kind of all-tools-together-as-unified-toolset that’s really meant here.
So, to our own case, our own work-context.
We have lots and lots of tools in enterprise-architecture.
Some of those tools we’ll use a lot – the metaphoric equivalents of the kitchen-knives, the cross-head screwdriver and that #2 needle that you like so much.
Some of those tools are a bit odd, that can do just one thing well, that you maybe bought on a whim and still haven’t gotten round to using for anything real – the metaphoric equivalents of those weird crochet-needles, that thing for producing curls of orange-peel, or the infamous ‘tool for removing stones from horses’-hooves’ penknife-attachment that probably no-one’s used in the last half-century.
Some of these tools will describe themselves as ‘toolsets’ – but they’re the metaphoric equivalents of the food-processor, the vacuum-cleaner and the garden-rakes that only work well within their own ‘walled-garden’, and won’t connect up with anything else at all.
For each of us in enterprise-architecture, some of those tools we use might be the same as everyone else uses in our business-environment, but some of them will definitely be different. And if they’re different, there is at present almost no way to link them together. (Other than – as Peter Bakker put it in a recent comment – by “creating little input-output scripts”: which, however, will often increase the fragmentation across the toolset-space, because everyone’s scripts are different, as well as the tools themselves.)
Hence the real problem, right now, is that there is no platform to link all of our tools together into a unified toolset – certainly not across the whole of the development-process depicted by the Squiggle, which is what we really need.
In fact, right now, for that purpose, the nearest we have to such a platform – such as it is – is us: and we’re not doing it all that well, if we’re honest about it. We urgently need something better.
(It’s been urgent for decades, actually – but if nothing’s been happening for decades, then the best time to make a proper start on this is right now…)
So let’s make a start on this.
The core business-need that we’re aiming to address here is our own variant of what I’ve elsewhere termed the Burnett Query – that as enterprise-architects, we’re responsible for helping our stakeholders make sense of three key questions, or sets of questions:
- What are the ‘things’ of the enterprise? – data, assets, products, processes, services, structures, capabilities, whatever? How do we describe those things? – their attributes and suchlike? And why should we describe them that way?
- What are the relations between those things? How do we describe those relations? And why should we describe the relations in that way?
- What are the changes, over time, for each of these ‘things’ and the relations between them? In what ways do or would they change? In what ways would we describe this in terms of time? – past, present, future, whatever? And why should the changes be that way?
And we need to be able to do this across the entirety of the Squiggle – though along a pathway that’s probably also different every time:
The catch is that each of the tools that we have is like a single dot on the context-space of the Squiggle: it will usually cover only one small part of a Squiggle’s path, with no easy means to connect its outcomes to anything else.
Worse, most of the tools that are currently described as ‘Enterprise Architecture Tools’ are like the tools marked in red dots on the later part of the Squiggle. They’re good at dealing with anything certain, but often very poor at modelling or managing uncertainty or change – in fact some are so bad at it that they’ll only allow us to describe a context in terms of a single static ‘As-Is’ or ‘To-Be’, with no means at all to describe how to link or move between one so-called ‘state’ to another…
Reality is that we need to do enterprise-architecture work along the whole of the Squiggle – not just the ‘easy-bit’ at the far end.
Which means that we need to be able to do sensemaking and modelling about the architecture anywhere and everywhere in the context-space denoted by the Squiggle, such that we can make sense of how everything depends on and connects with everything else throughout the context-space of our current Squiggle.
Which means that we need to be able to share the information about the ‘things’, the relations between ‘things’, and the changes to those ‘things’ and relations, across and along the entire length of the Squiggle.
Which means we need to be able to connect together all of our tools that work on that information, across and along the whole of the Squiggle.
And we may also need to connect any or all of this between any instance of the Squiggle, and any other instance of the Squiggle – maybe done by someone else, anywhere, anywhen, maybe separated in time by months, years, decades.
Which is where we come back to the need for those core elements that I’ve been ranting about through all of these posts:
- a common file-format, or common content-structure, to share information between tools
- a database-structure that can hold that content within some kind of database-technology, such that editors can work on it
- a common way to describe ‘editors‘ (tools) that can act on that content, in whatever way we need
- software apps that can support the content, the editors and the underlying data-structures, across the whole of the device-ecosystem from hand-helds to virtual-reality ‘war-rooms’
The crux of this – the one, core, inescapable requirement for this – is that all of these elements must be able to work together, as a unified platform, without being blocked anywhere by anyone’s ‘walled-garden’. If we don’t have that, it doesn’t work: we’d be stuck instead with yet another ‘dotting of the joins’ within the existing fragmented mess.
Sure, there are variants of all of those that sort-of exist already, in myriad forms. But in current implementations, again, they each only cover one small part of the space, and they don’t work together – and that’s the whole point, the whole problem. For example, when we look at the file-format, the content-structure:
- it’s sort-of like HTTP MIME-types – it has to be able to reference any type of context
- it’s sort-of like what we could do in a SQL database – yet it needs to be accessible, storable and sharable at a literally global scale
- it’s sort-of like what we could do in a graph-database
- it’s sort-of like what we could do in document-database
- it’s sort-of like Git or Subversion, in terms of version-control and rollback
- it’s sort-of like OWL or RDF, in terms of support for ontologies and semantics
- it’s sort-of like a wiki – some aspects of it, anyway, particularly readability, editability, versioning and rollback
- it’s sort-of like a topic-map – yet one that needs version-history and access-control as well
- it’s sort-of like DNS, in that we’ll need to be able to translate a true unique-ID into something more accessible and meaningful
- it’s sort-of like Java namespaces, in that we’ll need to support contextualisation, yet whilst still supporting truly-global connectivity
- and so on, and so on…
The catch, of course, is that no single one of these ‘sort-of like’ examples would be enough: it needs to be able to support all of those things, and more.
All in the one structure, that we can move between any tool to any other tool.
Which doesn’t exist. Not yet. Not in any non-proprietary form, certainly – which it must be, for this purpose.
That’s the problem; that’s the reason for working on all of this.
- why data-entities for ‘things’ need to be independent of any editor that works on them – in other words, that a ‘thing’ is not is ‘owned’ by or dependent on a single editor-tool, but is ‘owned’ by nothing but itself
- that a corollary is that a ‘thing’ accretes information from any number of editors – and hence needs to extensible indefinitely at an instance-level
- that we would need several entity-types, including tag, standard ‘thing’, extended-‘thing’, link and container
- why all entity-types – including links and tags – need to be ‘equal-citizens’ with and as ‘thing’-entities
- that we can satisfy all of the requirements for the content-structure with a single data-entity structure that (including cross-references to other ‘thing’-entities) could look like this:
Given various configurations of those parameters and arrays, that content-structure should be able to carry all of the content-types and instance-level content needed to satisfy the Burnett Query.
On its own, of course, the description of the data-structure isn’t of much use: it needs real-world implementations, in at least two forms: file-format, and database-structure.
The file-format can be implemented in any form that can handle the instance-level variation that’s needed here. For example, the data-content is essentially just a list of objects and their content, so even JSON would be sufficient – though preferably a variant that would support some form of comments or content-identification. probably should be implementable as a plain-text format (though compressed or encrypted formats should also become available as soon as practicable). It does also need to be human-readable (which pretty much rules out XML?) – maybe even one of the ‘literate programming‘ formats such as Literate CoffeeScript. Whatever we decide on, the file-format(s) must be open and non-proprietary – otherwise we’d be in an even worse mess than we are already.
The database-structure can, again, be implemented in any way that works – that it fully preserves the content-structure, handling import and export appropriately. As long as we have at least one implementation that’s non-proprietary and, preferably, open-source available to us, then proprietary implementations would be fine, too. To give one example, the content-structure above could be implemented directly into a single MongoDB ‘collection’ (schema-less table) – including query-types that could consistently search that layered structure without serious performance-hit. Initial implementations could probably get away without much access-control, but later down the track we would probably need explicit wrappers around the database itself, and a query-language optimised for this specific type of data-structure, in order to support the access-control constraints. (There’s some description of those constraints in the previous posts, but I’ll probably need to do a post specifically on that topic fairly soon, so that we can gather there the comments on those needs.)
Given the content-structure and the file-format, we’d have some means to move content between tools; given the file-format, the database-structure and some kind of software-utilities to read and write those files to and from the database, we’d have some means to store and swap such content in usable form, for whatever tools we need.
But which tools? That’s where we come to the next part, about editors and editor-definitions.
The fundamental point here is that if we’re going to be able to connect all of our tools all the way across all of the Squiggle, we will need, quite literally, to support any type of tool that anyone needs to use, that could usefully work on the content-structure described above.
I’ll say that again: any type of tool.
Right now, we don’t know what those tools will be. Every enterprise-architect I know has dozens of them – maybe hundreds – and uses them in different ways, according to their own style, context and way of working. For enterprise-architects, it’s our equivalent of the toolkit in the garage, or the sewing-kit, or all the paraphernalia of the kitchen. Each tool we use in enterprise-architecture does a different job, plays a different role in our iterations and explorations of the Squiggle – in exactly the same way that carving-knives and saucepans and spatulas and the pepper-pot all have different roles, as different tools, within the kitchen.
So because we don’t know beforehand what that content of the enterprise-architecture toolkit will be – or often even how each tool will be used – we need to design, right from the start, that the toolkit will need to be extensible. Arbitrarily extensible. Maybe even infinitely-extensible – or close to it, anyway.
Which is where editors and editor-definitions come into the picture: they’re what would make it possible to define and specify what a tool is, and how it will work on that content and the underlying data-structure. To make sense of this in practice, though, and clarify how it could be implemented, we first need to step back a bit, and look at the context in which the work on that data-structure would happen.
The usual approach to this is that we take some kind of standard – Archimate or BPMN or whatever – and build a software-application around that specification. All the information-entities are ‘owned’ by that one application – which in turn probably runs only on a small subset of the overall device-ecosystem. But we need to remember here that we’ve already turned this approach on its head: the information-entities are independent of any individual tool or standard. And we also need any given standard or whatever to work on the content-structure in the same way on every different application-platform, and, to the maximum extent practicable, every type of device in the device-ecosystem. (Small devices won’t be able to handle the same amounts of information as a larger device, but it should still be possible for them do so in some form at least.) To make that feasible, we need first to jump up a level of abstraction, and then come back down again with the various elements in a somewhat different configuration:
- device-independent description of tool-functionality – specifies tool-specific data-content, data-content, editing-rules, linking-rules, display-constraints, menu-structures, etc
- device-specific implementation of tool-functionality, compatible with any required tool-types
In effect, both the tool-specifications and the implementation-functionality are different types of ‘plug-ins’ to a shared platform, often with a many-to-many relationship between them.
(It’s true that some existing EA-toolsets can sort-of do this already – Abacus perhaps being the best examplar of this right now. But remember the key point here, that we need to connect between every type of platform: Abacus may be powerful, but it can’t handle everything we’d want to throw at it, and it’s also a proprietary platform, which breaks another of our key requirements. ‘Sort-of’ is good for examples and ideas, but it’s not ‘The Answer’: we do need to be careful not to allow ourselves to be sucked into that kind of mistake.)
Ideally, we’d want to build editors through some kind of drag-and-drop interface, something easy, something visual. But to start with, we could usefully focus on a text-based definition-language, because that’s what we’re going to need to as a file-format for sharing of editors, anyway – especially if we treat editors simply as another type of ‘thing’ within the overall content-structure, as we probably should.
To be honest, that part is going to take a while: there’s a lot of thinking I still need to do on this, and it needs to be developed in parallel with at least a demonstrator for the edit-functionality itself. Defining a simple-enough text-language is going to take a while just in itself, especially given the scope of what we’ll need it to describe. For example, this is a list of some of the functionality that I know I’d need just for my own current toolset – let alone anyone else’s – in approximate order of implementation:
- basic fields-only (any field-type as per HTML5), no graphics [e.g. requirements]
- basic ‘contain’ one-to-many linking [e.g. sessions, groups, clusters]
- basic ‘link’ many-to-many or controlled one-to-one linking [e.g. requirement-to-requirement etc]
- basic fields plus auto- or manual-links [e.g. glossary/thesaurus]
- basic with non-editable base-graphic [e.g. SWOT, SCORE]
- basic with drag-and-drop fields [e.g. Business Model Canvas, basic SCAN]
- basic with drag-and-drop regions and fields [e.g. extensible holomap]
- crossmap – base-graphic with overlays [e.g. SCAN crossmap]
- structured-assess – fields from template plus generated-graphic [e.g. SEMPER]
- graphic structured-assess – templated-fields, auto-links, manual-links, scrollable/zoomable generated-graphic [e.g. This]
- concept-map – simple fields, manual-links, auto-links, manual scrollable graphic [e.g. CMap, mind-map]
- media-attach – photo, audio, video, MIME-types
- knowledge-base – complex fields, auto- or manual-links, content-import
- graphic-model – objects with graphics / fields, auto / manual links, formal validation [e.g. Archimate, UML, BPMN]
- sketch – freeform draw, fields, tags, region-select as subsection / object-within-object
And that’s before we define any kind of analytics or suchlike that would work on the collections of ‘things’ and relations-between-things. So yeah, it’ll take a while… But it is all doable, and extensible, yet also with appropriate separation between device-independent (the editors) and device-dependent (the specific functionality) – that’s the point here.
(The separation-of-concerns provided by the two types of plugins – editors and functionality-plugins – is crucially important. Creating that separation means that whilst the functionality called for by the editor is the same in each case, the implementation of that functionality can be completely different, depending on what we need.
For example, a query returns a list of ‘things’. The editor specifies that the list should be shown, and what should happen when we select a single item from that list, defining the decision-logic, the event-response logic and suchlike – but that’s where it stops. The functionality-plugin determines how each of these will be displayed, the core user-interface metaphor and so on. Hence an item-list could be displayed in very different ways – as an HTML5 table, an Apple-style carousel, a holographic ball which we could reach into in 3D-space, a dynamically-tactile surface – but from the editor‘s perspective, they’re all exactly the same. That’s the separation we need here, for the extensibility that we need.)
The final part of the puzzle – in effect, what people will see as ‘the platform’ – is the need for apps that would connect all of the other elements together: the content-structure, file-format, database-structure, device-independent editor-definitions, and device-specific functionality, plus a few other concerns such as user-interface metaphor and user-logon.
There are two reasons we’ll need this.
One is that we’re going to need a ‘demonstrator’ that, yes, might well be a bit clunky at first, but can fully demonstrate the power of the core concepts here: the exchangeable, transferrable content-structure that can fully support any appropriate aspect of the Burnett Query with any tool on any device-type; and the extensibility of the editors, all within the same consistent toolset-as-toolset, each on the same device-as-platform, across any appropriate part of the device-ecosystem. Until we have that demonstrator, most people simply probably still won’t understand what this is all about, or why it’s so important to our work.
And the other reason is that – to be blunt – the existing tool-vendors do at present have quite a strong vested-interest against this happening – which means that we need to give them strong practical reasons and incentives as to why they should change their minds about this. We need to show them that, at the very least, it’s in their interest to be able to import and export the file-format, otherwise their own proprietary will find itself bypassed. And also, whilst, yes, we must have at least one non-proprietary implementation of the overall platform, the reality is that the commercial tool-vendors do have the funds and expertise and panache to build a toolset that’d likely be a lot more polished and easier to use than the more basic community-built tools – so there is still a very good business-opportunity there, and one that isn’t being properly addressed at present.
As I’ve said before on this blog, the first commercial tool-vendor who really gets what I’ve been saying here would, frankly, dominate the commercial market for tools that actually do support the way that enterprise-architects really work.
(And not just enterprise-architects, of course: the same would apply to product-developers, process-designers, graphic-designers, interface-designers, business-modellers, strategists, futurists, and many others – in fact anyone who needs to work across their respective version of the context-space denoted by the Squiggle. That’s a very big market that’s very poorly served at present… – a huge business-opportunity for the right developer and vendor. I’ve a lot more notes and research on this that I haven’t published yet, so if anyone’s interested, perhaps get in touch?)
As I said in the post ‘And more on toolsets for EA‘, the probable structure that we’d need for that demonstrator-app is fairly straightforward – in essence, a standard MVC (Model-View-Controller) type of software-architecture that we could summarise visually as follows:
Relative to a standard event-driven MVC, the only significant complications would be around the data-interface and user-manager, both of which might need a selection of wrappers to work with different database-types and authentication-mechanisms; and the plugin-mechanisms to support the editor-library (device-independent editor plugins) and action-library (device-specific functionality-plugins).
Yet none of it should be all that hard to do. For this part of the platform, there’s nothing much that’s new as such: the software-architecture is pretty straightforward, and all of the basic technology that we’d need for a usable toolset-platform already exists. There’s just a lot of it to do, that’s all… 🙂 And for that, as Phil Beauvoir put it some while back, the best tactic would be “MVP by MVP” – step-by-step, moving steadily onward from one Minimum Useful Product to another. Once we do have a ‘walking skeleton‘ that does link together all of the key architectural components, the rest should come together without too much difficulty: it’d be an awful lot of detail-work to do, yes, but as far as I can see it at present it would all be just more-of-much-the-same.
So, to wrap-up:
— For this purpose, ‘tool’ and ‘toolset’ are analogous to the individual tools in the kitchen, versus the entirety of the set of tools and suchlike that make up ‘the kitchen’. Metaphorically, it’s about how we use our tools together, as a unified set, in our workflows within our respective enterprise-architecture ‘kitchen’.
— A single tool is not the whole of the set; neither is a collection of tools that may work together amongst themselves but won’t work directly with anything else. For this purpose, a toolset is something that supports the entirety of a workflow in a unified way.
— Often the most important tool in a toolset is also the least-visible – a platform that holds the work together and provides a space or context upon which other tools can act.
— Our key problem here is fragmentation of the context-space, with each of our current tools yet further ‘dotting the joins‘ into near-incomprehensibility.
— Hence the need here is to move from fragmentary tools to unified toolset.
— Hence the aim here is to identify a type of platform that could re-unify our tools and the ways in which we use those tools, and help in ‘un-dotting the joins‘ in our enterprise-architectures and their development.
— The ultimate core of that platform is a common means to share information about any type of architectural ‘thing’; about the perceived relations between ‘things’; and about the ways in which those ‘things’ and relations might change.
— The toolset / platform needs to be able to support variance right down to the instance-level, and to connect between a set of tools that may be almost arbitrarily extensible.
— Focussing too much on a single existing tool or tool-cluster will only trap us back into further ‘dotting the joins’. Whatever we develop has to be able to support the whole of the core requirements: stopping at some halfway-house of a ‘solution’ will only make things worse.
— This post, and other posts in this series, provide suggestions and pointers as to how we could make a practical start on implementing such a platform, for use across the whole of enterprise-architecture and beyond.
I hope all of that makes more practical sense now? Comments and suggestions welcome, anyway.