There’s no short-cut to experience
At least he was open about it, I guess. “Tell you what I’ll do”, he says to my colleague here in Guatemala, “I’ll find you a client, then I’ll sit in, learn everything you do, and then I’ll apply it in my own business. How does that sound to you?”
Uh, no. Not a good idea. Not just because it’s a really bad deal from our perspective, but much more that Reality Department really doesn’t work that way: there’s no short-cut to experience.
Yes, it all looks simple enough – in fact that’s the whole point. A lot of simple visual summaries, and surprisingly simple-seeming methods, too. Yet it only looks simple because we’ve been through a heck of lot of hard work to make it that way. Hard-won experience, won the hard way through years and years of practice in many, many different contexts, getting it ‘wrong’ time and time again, in many, many different ways in order to get it right.
The real trap is that these simple-seeming ideas and methods aren’t simple rules, prepackaged sense-making and decision-making that will always work in every context. These are simple principles, simple guidelines, the kind of easy-to-memorise information that helps decision-making in real-time, in circumstances that are subtly different every time. (See my SCAN posts for more on these distinctions.) If you try to use them as ‘rules’ for inherently-uncertain contexts, without understanding why those principles apply, or how they need to be tweaked every time to match each different context, you’re going to be in real trouble – along with everyone else around you. Not a good idea…
The same often applies even to things that really are ‘rules’. Take that example of perhaps the greatest simplification ever made: e=mc2. All the core information you need to build a nuclear power-station is right there in that equation: but there’s a heck of a long way – a heck of a lot of engineering-experience – to go from that one equation to building a nuclear-power station that actually works.
Same with everything else, really: simplification is essential, but can also be deceptive – especially when people mistake ‘simple’ for ‘easy’…
Which is also why I get a bit hot-under-the-collar about the current proliferation of ‘certification-schemes’ in enterprise-architecture and elsewhere. Some of them are genuinely valuable, but others – to be blunt – are little better than money-spinning scams, in terms of the actual value that they (don’t) deliver. And the crucial distinction revolves around the role and recognition of experience.
For example, the TOGAF Foundation and Archimate Foundation certifications have real value. They verify that the respective person has a credible command of the terminology and language – a requirement that matters a lot for communication across a dispersed and disparate team.
Likewise the ATAC certifications should have real value, because each explicitly tests practical experience in the respective area.
But unless they’ve changed it in the past year or so, the full TOGAF certification is delivered through the absurdly-inappropriate mechanism of a multiple-choice test. And to my mind, that’s not merely useless, it’s actually worse than useless, because it’s exactly how not to test for the kind of experience that that type of competence requires. (When I did the TOGAF 8 exam some years back, I almost failed because I answered several key questions correctly in terms of real-world experience, rather than the theory-based wrong-assumptions that the test thought were ‘right’.) The result of that kind of pseudo-test is a bevy of people who can wave a certificate around, but have no idea how to do the work in any real-world context.
A good training-course can make all the difference, and the better training-providers do take up some of the slack here. (I’ll wave a flag at this point for John Polgreen at Architecting The Enterprise, who’s been doing sterling work for years on adapting TOGAF for the US-government context.) Yet none of that competence carries through anywhere into the actual test: so unless we know each of the training-providers, we have no way to tell whether a candidate does actually know what they’re doing, or merely that they have a piece of paper to prove that they know just enough to get into real trouble, but not enough to get out of it again.
In effect, right now, the full TOGAF certification is of less real-world value than the Foundation certification – which is both bizarre and sadly pointless. And I’ll hasten to add that I’m using TOGAF here merely as one example amongst many: it may well be that most of the so-called ‘certifications’ in this field are even more meaningless than that. And the results can be seen everywhere in the trade…
In short, it’s a mess.
What we need to be testing for is genuine understanding of a context, and the ability to adapt for uniqueness. And that calls for much, much more than can be covered in a crude multiple-choice test delivered through a mindless machine. Sure, that kind of test is cheap, and relatively easy to administer: but it’s also all but meaningless for anything than foundation-level rote-knowledge. It really does take years of painful practice to develop the experience needed to do this work well: and if this trade is to gain the credibility that it needs, we need to stop pretending that we don’t need to test for that experience.
Time to re-think how we do this, and how we respect this, too: there’s no short-cut to experience.
Agree totally. Include the “understanding” of a context to include beyond IT!
Include that John Polgreen (& you I may add) are doing GREAT work.
Many certifications are purely a memorization of opinionated facts (or filtered facts). It does take experience as well as a full understanding of what really is “the enterprise”.
Tom, you wrote: “But unless they’ve changed it in the past year or so, the full TOGAF certification is delivered through the absurdly-inappropriate mechanism of a multiple-choice test. And to my mind, that’s not merely useless, it’s actually worse than useless, because it’s exactly how not to test for the kind of experience that that type of competence requires. (When I did the TOGAF 8 exam some years back, I almost failed because I answered several key questions correctly in terms of real-world experience, rather than the theory-based wrong-assumptions that the test thought were ‘right’.)
When we developed the TOGAF 9 certification program we tried to address what we saw as the key shortcoming of the TOGAF 8 exam – the use of simple multi-choice questions for the whole test. So, TOGAF 9 has two exams. The first for Foundation uses simple multiple choice – there’s a question and a number of proposed answers, one of which is right – 40 questions in 60 minutes. As you say, that’s an OK way to test knowledge.
But for level 2, the Certified level, we wanted to test understanding and touch on ability to apply. We use scenario based complex multiple choice questions, with gradient scoring – 8 questions in 90 minutes. And you get access to the text of TOGAF 9 during the test. This is the way it’s been since we introduced TOGAF 9 certification in 2009.
I’m not claiming it’s perfect – far from it – and it certainly does not test competence as an architect! But it’s a lot better than the TOGAF 8 exam.
From what you write you haven’t taken a TOGAF 9 part 2 test, yet you’re lambasting the program on the basis of your TOGAF 8 exam experience.
If you are interested in what it’s really like, why not take the exam and find out?
As you are already TOGAF 8 certified, I suggest the Bridge exam – it contains an abbreviated part 1 followed by the full open book part 2 (8 complex multi-choice questions). Or you could take the two parts separately. Your choice.
Let me know if you are interested and I’ll send you a Prometric voucher
James de Raeve
VP Certification at The Open Group
James – Many thanks for the courteous reply!
I’m very glad to see that you-plural did re-think the TOGAF 8 exam, because it really was not-good. That multiple-choice structure would (and, in mine and many other architects’ experience, did) weed out both those who hadn’t read the book and also those who actually did know what they were doing – especially in a whole-enterprise-context – leaving only those who’d read the book and had only limited-experience of real-world IT-architecture. In other words, what that mechanism did was award certification primarily to those with just enough knowledge to really get into trouble, but not enough to get out of it again… not a good idea, from anyone’s perspective?
I emphasise again that this is generic to all attempts to use machine-based multi-choice exam to verify skills – it’s not at all specific to TOGAF. (In a sense, I’m more questioning Prometrics’ business-model than Open Group’s here.)
I’m glad we agree that multiple-choice test is a reasonably-appropriate mechanism to test rote-learning – which in essence is what the TOGAF 9 Foundation is about. I think we also agree that such ‘foundation’ testing is useful, for everyone, as it verifies familiarity with a necessary shared-language. No questions there – from my side, anyway.
Yet I do still have very severe doubts about any use of multiple-choice for skills-assessment: I appreciate that you’ve done a lot of work towards making it better, yet it’s still fundamentally the wrong way to go about it. (My Masters was in design for skills-education, and I’ve been involved ever since then in skills-education in a wide variety of skills, so I do have some idea of what I’m talking about here.) And I do appreciate the practical quandary you face: you have a literally global scope, so you need to do some form of distributed testing – for which Prometrics and their ilk would be a good fit. And yet the fundamental mechanism of multiple-choice only works well for the ‘rote-learning’ part of that problem – not the skills-learning part. Hence, to be blunt, I just don’t trust it: no matter how much tweaking you do, it’s still the wrong mechanism to test competence in inherently-unique contexts – which is the case for most real-world applications of TOGAF.
Many thanks for the offer of the Prometrics coupon – much appreciated, indeed – yet also aware of the several real challenges behind that offer! 🙂
— One is that, yes, I darn well ought to put my money (or action, rather) where my mouth is: in a sense I shouldn’t critique the current version of the test unless I’ve done it myself.
— Which brings me to the second challenge: because yes, to be blunt about myself, that brings up plenty of real fears – Would I pass, or not? What would happen if I fail? – and so on, just like anyone else facing a formal test, though there’s perhaps a fair bit of subtle back-implications around this one, for me and for you-plural.
— The third challenge, though, is whether TOGAF is actually relevant to what I do. Even with the offered coupon, there’s still a lot of investment in time and study-materials before I could take the test: but is it worth it? Is it what I need to do for the work that I’m doing? And the short answer there is ‘probably not’: it’s more likely to be a distraction than a help.
For me, the key around that last challenge is that I still feel/think that the current structure of TOGAF is fundamentally the wrong way to approach enterprise-architecture. ‘Architecture of the enterprise’ is, literally, the architecture of the enterprise as a whole; whereas TOGAF is, very obviously, the architecture of the enterprise-IT, with a severely-constrained inside-out view of the rest of the enterprise – which in systems-terms is a classic anti-pattern, guaranteed to cause more problems than it solves at the broader enterprise-wide scope. We did hope that that fundamental flaw would have been resolved in TOGAF 9: it wasn’t. (There’s still some hope that it might at last be addressed in TOGAF X?) TOGAF 9 still insists on an IT-centric view of the enterprise – which means that people who are trained and certified only in the current versions of TOGAF are not likely to be capable of doing ‘the architecture of the enterprise’ in the literal sense of the term.
Agreed, I haven’t seen the current certification-exam: but there’s nothing in the TOGAF 9 specification that explains that problem, and instead a great deal that reinforces it. Yet until that is resolved, there are huge educational, operational, ethical, and even legal issues around the Open Group’s usage of the term ‘enterprise-architecture’ for TOGAF: for example, as soon as it becomes clear that that usage of the term ‘enterprise-architecture’ is an invalid ‘term-hijack‘, by definition you fall foul of the Trade Descriptions Act and its equivalents elsewhere. (“You have been warned”, etc?)
So yes, in one sense I ‘ought’ to do the TOGAF 9 exam; in another, it’s exactly what I should not do – not least because a sizeable part of my work is about re-educating IT-centric EAs to be able to work at a true whole-enterprise scope. And there’s also the blunt fact that, however you tweak it, multiple-choice predefined-answer is a fundamentally-wrong method to assess not just EA-skills, but any form of skills and experience: so I would be submitting myself to a type of testing that’s inherently invalid even before we start. Tricky, that…
You can see my quandary there: damned if I don’t, damned if do. And I appreciate your quandary too: you need some form of certification, all your trainees need some form of certification, all the prospective employers need some form of certification – and yet you know (or should do, by now) that the only easy method for delivering the certification-test will not and cannot do the task. A proper peer-review such as in ITAC would be a lot better, but is much more expensive and much more difficult to administer – and where the contexts are genuinely unique, as they often are in whole-enterprise EA, who then are our peers? And if they don’t know the realities of the world we deal with, how could they claim to stand in judgment of us? Again, definitely a tricky problem… no easy answers, certainly.
In that sense, we’re both a bit stuck here: what we have right now does not work, yet there’s no obvious way forward. Discuss, perhaps?
And thanks again, anyway – much appreciated.