often outside the contexts for which they were designed. The main goal of this book is to encourage people designing, creating, and operating computer systems to be conscious and explicit in their actions around trust.
Trust as a Way for Humans to Manage Risk
Risk is a key concept to be able to consider when we are talking about security. There is a common definition of risk within the computing community, which is also shared within the business community:
In other words, the risk associated with an event is the likelihood that it will occur multiplied by the impact to be considered if it were to occur. Probability is expressed as a number between 0 and 1 (0 being no possibility of occurrence, 1 being certainty), and the loss can be explicitly stated either as an amount of money or as another type of impact. The point of the formula is to allow risks to be compared; and as long as the different calculations use the same measure of loss, it is generally unimportant what measure is employed. To give an example, let us say that I am interested in the risk of my new desktop computer failing in the first three years of its life. I do some research and discover that the likelihood of the keyboard failing is 4%, or 0.04, whereas the likelihood of the monitor failing is only 1%, or 0.01. If I were to consider this information on its own, it would seem that I should worry more about the keyboard than the monitor, until I take into account the cost of replacement: the keyboard would cost me $15 to replace, whereas the monitor would cost me $400 to replace. We have the following risk calculations then:
It turns out that if I care about risk, I should be more concerned about the monitor than the keyboard. Once we have calculated the risk, we can then consider mitigations: what to do to manage the risk. In the case of my desktop computer, I might decide to take out an extended manufacturer's warranty to cover the monitor but just choose to buy a new keyboard if that breaks.
Risk is all around us and has been since before humans became truly human, living in groups and inhabiting a social structure. We can think of risk as arising in four categories:
| ASSESSMENT | MITIGATION |
|---|---|
| Easy If there are predators nearby, they might kill us … | Easy … so we should run away or hide. |
| Easy If our leader gets an infection, she may die … | Difficult … but we don't know how to avoid or effectively treat infection. |
| Difficult If the river floods, our possessions may be washed away … | Easy … but if we camp farther away from the river, we are safer. |
| Difficult If I eat this fruit, it may poison me … | Difficult … but I have no other foodstuffs nearby and may go hungry or even starve if I do not eat it. |
For the easy-to-assess categories, both the probability and the loss are simple to calculate. For the difficult-to-assess categories, either the probability or the loss is hard to calculate. What is not clear from the simple formula we used earlier to calculate risk is that you are usually calculating a risk against something that is generally a benefit. In the case of the risk associated with the river, there are advantages to camping close to it—easy access to water and ability to fish, for example—and in the case of the fruit, the benefit of eating it will be that it may nourish me, and I do not need to trek further afield to find something else to eat, thereby using up valuable energy.
Many of the risks associated with interacting with other humans fit within the last category: difficult to assess and difficult to mitigate. In terms of assessment, humans often act in their own interests rather than those of others, or even of a larger group; and the impact of an individual not cooperating may be small—hurt feelings, for example—or large—inability to catch game—or even retribution towards a member of the group. In terms of mitigation, it is often very difficult to guess what actions to take to encourage an individual, particularly one you do not already know, to ensure that they interact with you in a positive manner. You can, of course, avoid any interactions at all, but that means you lose access to any benefits from such interactions, and those benefits can be very significant: new knowledge, teamwork for hunting, more strength to move objects, safety in numbers, even having access to a larger gene pool, to name just a few.
Humans developed trust to help them mitigate the risks of interacting with each other. Think of how you have grown to know and trust new acquaintances: there is typically a gradual process as you learn more about them and trust them to act in particular ways. As David Clark points out when discussing how we develop trust relationships, this “is not a technical problem, but a social one”.10 We see here both time and various other contexts in which trust relationships can operate. Once you trust an individual to act as a babysitter, for instance, you are managing the risks associated with leaving your children with that person. An alternative might be that you trust somebody to make you a cup of tea in the way that you like it: you are mitigating the chance that they will add sugar to it or, in a more extreme case, poison you and steal all of the loyalty points you have accrued with your local cafe.
Trust is not, of course, the only mitigation technique possible when considering and managing risk. We have already seen that you can avoid interactions altogether,11 but two alternatives that are different sides of the same coin are punishment and reward. I can punish an individual if they do not interact with me as I wish, or I can reward them if they do. Many trust relationships between individuals are arguably built up over time with a combination of these mitigations, even if the punishment is as little as a frown and the reward as little as a smile. What is even more interesting is that the building of the trust relationship is two-way in this case, as the individual being rewarded or punished needs to trust the other individual to be consistent with rewards or punishments based on the behaviour and interactions presented.
Risk, Trust, and Computing
Risk is important in the world of IT and computing. Organisations need to know whether their systems will work as expected or if they will fail for any one of many reasons: for example, hardware failure, loss of power, malicious compromise, poor software. Given that trust is a way of mitigating risk, are there opportunities to use trust—to transfer what humans have learned from creating and maintaining trust relationships—and transfer it to this world? We could say that humans need to “trust” their systems. If we think back to the cases presented earlier in the chapter, this fits our third example, where we discussed the bank trusting its IT systems.
Defining Trust in Systems
The first problem with trusting systems is that the world of trust is not simple when we start talking about computers. We might expect that computers and computer systems, being less complex than humans, would be easier to consider with respect to trust, but we cannot simply apply the concept of trust the same way to interactions with computers as we do
