where all sorts of errors in human judgment are analyzed. They wrote a joint paper in an effort to compare these apparently competing views. But what they found was that in important respects, they did not disagree, so they decided to name the paper, “Conditions for Intuitive Expertise: A Failure to Disagree.”8
They found they agreed that developing expert intuition in any field is not an automatic outcome of experience. Experts needed “high-validity” feedback so that the outcomes of estimates and decisions could be learned from. Our feedback should be consistent (we get feedback most, if not all, of the time), quick (we don't have to wait long for it), and unambiguous.
Risk management simply does not provide the type of consistent, immediate, and clear feedback that Kahneman and Klein argue we need as a basis for learning. Risk managers make estimates, decisions, or recommendations without knowing what the effect is for some time, if ever. If risk went down after the implementation of a new policy, how would you know? How long would it take to confirm that the outcome was related to the action taken? How would you determine whether the outcome was just due to luck?
What we will not do to measure the performance of various methods is rely on the proclamations of any expert regardless of his or her claimed level of knowledge or level of vociferousness. So even though I may finally have some credibility in claiming experience after thirty years in quantitative management consulting, I will not rely on any appeals to my authority regarding what works and what does not. I will, instead, resort to using published research from large experiments. Any mention of anecdotes or quotes from “thought leaders” will only be used to illustrate a point, never to prove it.
I don't think it is controversial to insist that reason and evidence are the way to reach reliable conclusions about reality. The best source of evidence is large, random samples, clinical trials, unbiased historical data, and so on. The data should then be assessed with proper mathematical methods to make inferences.
AN ASSESSMENT OF SELF-ASSESSMENTS
The potential existence of an analysis placebo, the difficulty of learning from experience alone in risk management, and the general lack of objective measurements of performance in risk management means that we should be wary of self-assessments in this field. We should bear in mind one particular statement in the previously mentioned article by Daniel Kahneman and Gary Klein:
True experts, it is said, know when they don't know. However, nonexperts (whether or not they think they are) certainly do not know when they don't know. Subjective confidence is therefore an unreliable indication of the validity of intuitive judgments and decisions. (p. 524)
Risk managers are still humans and, not surprisingly, there is a tendency for most of us humans to have at least a slightly inflated opinion of ourselves (friends and family will confirm that I'm no exception). For example, 87 percent of Stanford MBA students rate their academic performance to be in the top half of their class.9 Other surveys show that a clear majority of people rate themselves as being more popular,10 better looking,11 healthier,12 and better drivers13 than at least half of the population.
These are examples of the Dunning-Kruger effect, the tendency for the least competent in a given area to also be the most overconfident in his or her capabilities. I first mentioned this phenomenon in the first edition of this book but it has become more widely recognized (if a bit overused) since then. It comes from the work of Cornell psychologists Justin Kruger and David Dunning. They published their research in the area of self-assessments in the harshly titled article, “Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments.”14 They show that about two-thirds of the entire population will rate themselves as better than most in reasoning skills, humor, and grammar. Although the last few studies I just mentioned are not focused on management, if you think that C-level management and trained business professionals are more realistic and their confidence is more justified, wait until you read part 2 of the book.
There is no reason to believe risk management avoids the same problems regarding self-assessment. As we saw in the surveys, any attempt to measure risk management at all is rare. Without measurements, self-assessments in the effectiveness of risk management are unreliable given the effect of the analysis placebo, the low validity problem described by Kahneman and Klein, and the Dunning-Kruger effect.
There is an old management adage that says, “You can't manage what you can't measure.” (This is often misattributed to W. E. Deming, but is a truism, nonetheless.) Management guru Peter Drucker considered measurement to be the “fourth basic element in the work of the manager.” Because the key objective of risk management—risk reduction or at least a minimized risk for a given opportunity—may not exactly be obvious to the naked eye, only deliberate measurements could even detect it. The only way organizations could be justified in believing they are “very effective” at risk management is if they have measured it.
Risk professionals from Protiviti and Aon (two of the firms that conducted the surveys in chapter 2) also have their suspicions about the self-assessments in surveys. Jim DeLoach, a Protiviti managing director, states that “the number of organizations saying they were ‘very effective’ at managing risks was much higher than we expected.” Recall that in the Protiviti survey 57 percent of respondents said they quantify risks “to the fullest extent possible” (it was slightly higher, 63 percent, for those that rated themselves “very effective” at risk management). Yet this is not what DeLoach observes first-hand when he examines risk management in various organizations: “Our experience is that most firms aren't quantifying risks … I just have a hard time believing they are quantifying risks as they reported.”
Christopher (Kip) Bohn, an actuary, fellow of the Casualty Actuarial Society, and formerly a director at Aon Global Risk Consulting, is equally cautious about the findings in surveys about how common quantitative methods may be in risk management. Bohn observes, “For most organizations, the state of the art is a qualitative analysis. They do surveys and workshops and get a list of risks on the board. They come up with a ranking system with a frequency and impact, each valued on a scale of, for example, one to five.” This is not exactly what an actuary like Bohn considers to be quantitative analysis of risk.
My own experience also seems to agree more with the personal observations of DeLoach and Bohn than the results of the self-assessment surveys. I treat the results of the HDR/KPMG survey as perhaps an upper bound in the adoption of quantitative methods. Whenever I give a speech about risk management to a large group of managers, I ask those who have a defined approach for managing risks to raise their hands. A lot of hands go up, maybe half on average. I then ask them to keep their hands up only if they measure risks. Many of the hands go down. Then I ask them to keep their hands up only if probabilities are used in their measurements of risks (note how essential this is, given the definition of risk we stated). More hands go down and, maybe, one or two remain up. Then I ask them to keep their hands up if they think their measures of probabilities and losses are in any way based on statistical analysis or methods used in actuarial science. After that, all the hands are down. It's not that the methods I'm proposing are not practical. I have used them routinely on a variety of problems. (I'll argue in more detail later against the myth that such methods aren't practical.)
Of
