us, we are easy to fool.
A harder question to answer is, “What is the evidence for the belief that it works?” For any firm that hasn't asked that question before, it should be an immediate priority. If the firm can't answer that question, then it has no reason to believe that efforts to manage risks are working or, for that matter, are even focusing on the right risks. The standard must be some objective measure that could be verified by other stakeholders in the organization or outside auditors.
Most (69 percent according to the HDR/KPMG survey) don't even attempt to measure whether risk management is working. Of those who say they do measure risk, most (63 percent) are merely using a survey of staff with questions such as, “How would you rate the effectiveness of risk management?” It may not be obvious now, but there are ways to measure risk management objectively even though such measurements are uncommon.
This chapter will describe the difficulties in conducting measurements of risk management and some solutions for overcoming them. But first, to highlight the importance of measuring risk management, let's look at one example involving the health and safety of large numbers of people.
ANECDOTE: THE RISK OF OUTSOURCING DRUG MANUFACTURING
In 2007, I was asked to speak at a conference organized by the Consumer Health Products Association (a pharmaceutical industry association). The event organizers were specifically interested in my contrarian views on common risk management methods. After my keynote, I was asked by the event organizers to attend another session on a new risk management method for outsourcing drug manufacturing and provide my comments to the audience. They thought it would be interesting if I could start a conversation by offering an on-the-spot evaluation of the new method.
To control costs, this large pharmaceutical manufacturer was more frequently outsourcing certain batch processes to China. Virtually all of this manufacturer's competition were doing the same. But although the costs were significantly lower, they had a concern that batches from China might have additional quality control issues over and above those of batches manufactured here in the United States. These concerns were entirely justified.
Earlier that year there had already been several widely publicized product safety incidents with goods produced in China. In June, there was a toxin found in toothpaste and lead found in toys produced in China. Then there was tainted pet food that killed as many as 4,000 pets. There was even the disturbing case of “Aqua Dots,” the children's craft-beads that stuck together to make different designs. The coating of these beads could metabolize in the stomach to produce gamma-hydroxybutyrate—the chemical used in date-rape drugs.
So, clearly, assessing the risk of outsourcing was a major area of interest at the conference, and the room was at capacity. The presenter—a very respected chemical engineer—began to describe a risk assessment method based on a subjective weighted score.1 In it, several “risk indicators” were each scored on a scale of 1 to 5. For example, if the manufacturer already produces a similar but not identical drug, it might get a low risk score of 2 on the indicator called proven technical proficiency. If it was inspected by and got a positive evaluation from the Chinese health agency, but was not yet inspected by the Food and Drug Administration, then it might get a 4 on the formal inspections indicator.
Then these scores were each multiplied by a weight of somewhere between 0.1 and 1.0 and then all of the weighted scores were totaled. The total of the weighted score might be 17.5 for one outsourcing strategy, 21.2 for another, and so on. The team that chose the scores also chose the weights and, again, it was based only on subjective judgments. The team further separated the resulting scores into various stratifications of risk that would, apparently, have some bearing on the decision to use a particular China-based source for a drug. For example, risk scores of over 20 might mean “extremely high risk: Find an alternative”; 10 to 19 might mean “high risk: Proceed only with increased quality assurance”; and so on.
When the presenter had finished, I was expected to provide my two cents on the method. I decided I could neither endorse nor reject the approach outright. To be perfectly fair, neither position could yet be positively justified at that point without knowing a few more details (although there is a good chance it shared the flaws of many weighted scores, which I discuss later). I simply asked, “How do you know it works?” This is the most important question we could ask about a risk analysis and risk management approach. Once I knew the answer to that question, then I could legitimately take a position.
The presenter seemed to struggle with this question, so I then suggested to the presenter that the engineers in this field could be as scientific in their approach to this problem as they are in any other aspect of their profession. I pointed out that, for one, there was no need to start from scratch. If they were developing a new process for pharmaceutical manufacture, I'm sure they would examine existing research in the area. Likewise, there is quite a lot of literature in the general area of assessing risks in a mathematically and scientifically sound manner. It would be helpful to know that they don't have to reinvent any of the fundamental concepts when it comes to measuring risks.
Then I pointed out that in the design of processes in drug production, once they had thoroughly reviewed the literature on a topic, no doubt they would design empirical tests of various components in the process and measure them in a way that would satisfy the peer-reviewed journals and the FDA inspectors alike. Again, this same philosophy can apply to risk.
In fact, a much more sophisticated method is often already used to assess a different risk in the drug industry. Stop-gate analysis (also variously referred to as phase-gate and stage-gate analysis) is used to determine whether a candidate for a new product should advance from formulation to animal testing, then from animal testing to human trials, until finally the company decides whether to go to market. Many drug companies use proven statistical methods at each step in the stop-gate analysis. But, somehow, none of the basic concepts of stop-gate analysis were built on to assess the risks of outsourcing production to China.
I was already fairly sure that they had no objective measure for the effectiveness of this method. If they had known to create such measures, they would probably have been inclined to create a very different approach in the first place. When it came to designing a method for assessing and managing risks, these scientists and engineers developed an approach with no scientific rigor behind it. Although the lack of such rigor would be considered negligent in most of their work, it was acceptable to use a risk assessment method with no scientific backing at all.
Of course, this wasn't deliberate; they just didn't know it could be scientific. They just didn't think of this new risk in the same way as they thought of the substances and processes they use to manufacture drugs in a highly regulated industry. The chemicals they process and the vessels they use are concrete, tangible things and, to the engineers, risk might seem like an abstraction. Even the methods they use in stop-gate analysis might take on an air of concreteness simply because, by now, they have a lot of experience with using it. Perhaps to them, the process of managing an unfamiliar risk seems like an intangible thing that doesn't lend itself to the same methods of validation that a drug manufacturing process would have to undergo for FDA approval. Applying the type of scientific reasoning and testing used on the production of a drug to the risk analysis of producing that same drug in China is a leap they had not considered.
The presenter and the audience felt that the weighted scoring method they described was something close to “best practices” for the industry. When I asked, nobody in the room claimed to have an approach that was any more sophisticated. Most had no risk analysis at all for this problem.
Fortunately for the company that was presenting its risk management solution, it had not yet seen the worst-case scenarios that might result from unsound risk analysis. But with an entire industry approaching the outsourcing problem
