and over 30 is critical). This sort of method can sometimes be informed by the previously mentioned checklists and audits.
Mathematical and Scientific Methods
The most sophisticated risk analysts will eventually use some form of probabilistic models in which the odds of various losses and their magnitudes are computed mathematically. It is the basis for modeling risk in the insurance industry and much of the financial industry. It has its own flaws but just as Newton was a starting point for Einstein, it is the best opportunity for continued improvement. It could use subjective inputs, as do the other methods, but it is also well-suited to accept historical data or the results of empirical measurements. This includes the probabilistic risk analysis used in engineering as well as quantitative methods used in finance and insurance. This means that uncertainties are quantified as a probability distribution. A probability distribution is a way of showing the probability of various possible outcomes. For example, there may be a 5 percent chance per year of a major data breach. If the breach occurs, there is a 90 percent chance the impact is somewhere between $1 million and $20 million.
As the previous survey showed, quantitative methods usually involve Monte Carlo simulations. This is simply a way of doing calculations when the inputs themselves are uncertain—that is, expressed as probability distributions. Thousands of random samples are run on a computer to determine the probability distribution of an output (say, the total losses due to cyberattacks) from the inputs (the various possible individual types of cyberattacks and their impacts).
These methods also include various types of statistical analysis of historical data. Although the lack of data is sometimes perceived as a problem in risk analysis (16 percent of HDR/KPMG survey respondents said this was a problem), statistical methods show you need less data than you think, and, if we are resourceful, you have more data than you think. There are a couple of categories of methods that are not strictly based on statistical methods or probabilities, but may get lumped in with mathematical or scientific methods, at least by their proponents. One is deterministic financial analysis. By deterministic I mean that uncertainties are not explicitly stated as probabilities. Readers may be familiar with this as the conventional cost-benefit analysis in a spreadsheet. All the inputs, although they may be only estimates, are stated as exact numbers, but there are sometimes attempts to capture risk analysis. For example, a discount rate is used to adjust future cash flows to reflect the lower value of risky investments. One might also work out best-case and worst-case scenarios for costs and benefits of various decisions.
One final approach that sometimes gets grouped together with mathematical methods in risk management includes expected utility theory, which gives us a way to mathematically make trade-offs between risk and return. These methods combine to create a quantitative method broader than risk analysis: decision analysis. As mentioned in chapter 1, risk analysis is only part of decision analysis. We will be spending a lot more time discussing these approaches.
Other methods under the umbrella of “preference theory” were originally created as derivatives of the previously mentioned expected utility theory, but instead of trading off risk and return, they purport to mathematically assist in the trade-offs of multiple different objectives. Variously named but similar methods include multi-attribute utility theory (MAUT), multi-criteria decision-making (MCDM), and analytic hierarchy process (AHP). They claim more mathematical validity than simple weighted scores but ultimately rely on statements of preferences, not forecasts or estimates, of experts. In the case of AHP, a more sophisticated method is used to determine whether the expert judgments are at least internally consistent. As with the other methods listed so far, these have been used on lots of decision analysis problems that might not strictly be risk assessments, but they are included here because they have been used to evaluate decisions according to their risks.
Whatever the chosen method may be, it should be used to inform specific actions. Many of those actions will involve choices regarding whether and how to mitigate risk in some way. You may decide to invest in new cybersecurity controls, keep tighter control over your supply chain, diversify production processes, increase the number of auditors, require new training, and so on. If they were free you would do them all. If all risk mitigation options were equally costly and equally effective, you could do them in any random order you like. But neither of those is the case. You will have more risks than you can realistically control for and the bang for the buck will vary widely. You will have to prioritize and make choices.
If these methods were used for no more than assessing corporate art for the reception area or where to have the company picnic, then the urgency of this evaluation would not be nearly as high. But, as I have already pointed out, these methods are being used for many of the biggest and riskiest decisions in the corporate world and government. Fortunately, some of these can be modified to produce an approach that can be shown to be a significant improvement on the baseline condition of expert intuition alone. Instead of improving on expert intuition, some apparently add error to expert intuition. Until this gets sorted out, improvements in risk management will not be possible.
NOTES
1 1. “Fall Guys: Risk Management in the Front Line,” Economist Intelligence Unit, 2010, https://advisory.kpmg.us/content/dam/advisory/en/pdfs/risk-assurance/risk-management-front-line.pdf; “Best Practice in Risk Management: A Function Comes of Age,” Economist Intelligence Unit, 2007, http://graphics.eiu.com/files/ad_pdfs/eiu_Risk_Management.pdf.
2 2. “Global Risk Management Survey 2017,” Aon Corporation, 2017; “Global Enterprise Risk Management Survey,” Aon Corporation, 2010; “Global Risk Management Survey 2007,” Aon Corporation, 2007, https://www.aon.com/getmedia/d95563c6-a3b8-4ff1-bb45-0ed511c78f72/2017-Global-Risk-Management-Survey-Report-rev-120318.aspx.
3 3. “Executive Perspectives on Top Risks for 2018,” Protiviti & NC State Poole College of Management, 2018; “2007 U.S. Risk Barometer: Survey of C-Level Executives with the Nation's Largest Companies,” Protiviti, 2007, https://www.protiviti.com/sites/default/files/united_states/insights/nc-state-protiviti-survey-top-risks-2018.pdf.
CHAPTER 3 How Do We Know What Works?
Leaders get out in front and stay there by raising the standards by which they judge themselves—and by which they are willing to be judged.
—FREDRICK SMITH, CEO, FEDEX
The first principle is that you must not fool yourself, and you are the easiest person to fool.
—RICHARD P. FEYNMAN, NOBEL PRIZE–WINNING PHYSICIST
According to some risk management surveys, organizations are very often satisfied with their risk assessment and risk management methods. For example, a survey by the major consulting firm Deloitte in 2012 found that 72 percent of organizations rate themselves as “extremely effective” or “very effective” at managing risks (up slightly from 66 percent in 2010). In other words, a majority believe their risk management is working. But,
