real attacks scale and how the incentives facing different players lead to unsafe equilibria.
At the end of the first workshop on security and human behavior in 2008, the psychologist Nick Humphrey summed up a long discussion on risk. “We're all agreed,” he said, “that people pay too much attention to terrorism and not enough to cybercrime. But to a psychologist this is obvious. If you want people to be more relaxed in airports, take away the tanks and guns, put in some nice sofas and Mozart in the loudspeakers, and people will relax soon enough. And if you want people to be more wary online, make everyone use Jaws as their screen saver. But that's not going to happen as the computer industry goes out of its way to make computers seem a lot less scary than they used to be.” And of course governments want people to be anxious about terrorism, as it bids up the police budgets and helps politicians get re-elected. So we give people the wrong signals as well as spending our money on the wrong things. Understanding the many tensions between the demands of psychology, economics and engineering is essential to building robust systems at global scale.
Research problems
Security psychology is one of the hot topics in 2020. In the second edition of this book, I noted that the whole field of security economics had sprung into life since the first edition in 2001, and wrote ‘We also need more fundamental thinking about the relationship between psychology and security’. Security usability has become a discipline too, with the annual Symposium on Usable Privacy and Security, and we've been running workshops to bring security engineers together with anthropologists, psychologists, philosophers and others who work on risk and how people cope with it.
My meta-algorithm for finding research topics is to look first at applications and then at neighbouring disciplines. An example of the first is safe usability: as safety-critical products from cars to medical devices acquire not just software and Internet connections, but complex interfaces and even their own apps, how can we design them so that they won't harm people by accident, or as a result of malice?
An example of the second, and the theme of the Workshop on Security and Human Behaviour, is what we can learn from disciplines that study how people deal with risk, ranging from anthropology and psychology to sociology, history and philosophy. Our 2020 event is hosting leading criminologists. The pandemic now suggests that maybe we should work with architects too. They're now working out how people can be physically distant but socially engaged, and their skill is understanding how form facilitates human experience and human interaction. There's more to design than just hacking code.
Further reading
The Real Hustle videos are probably the best tutorial on deception; a number of episodes are on YouTube. Meanwhile, the best book on social engineering is still Kevin Mitnick's ‘The Art of Deception’ [1327]. Amit Katwala wrote a short survey of deception detection technologies [1027] while Tony Docan-Morgan has edited a 2019 handbook on the state of deception research with 51 chapters by specialists on its many aspects [569].
For how social psychology gets used and abused in marketing, the must-read book is Tim Wu's ‘The Attention Merchants’ which tells the history of advertising [2052].
In the computer science literature, perhaps a good starting point is James Reason's ‘Human Error’, which tells us what the safety-critical systems community has learned from many years studying the cognate problems in their field [1592]. Then there are standard HCI texts such as [1547], while early papers on security usability appeared as [493] and on phishing appeared as [978]. As we move to a world of autonomous devices, there is a growing body of research on how we can get people to trust robots more by Disneyfication – for example, giving library robots eyes that follow the direction of travel, and making them chirp with happiness when they help a customer [1690]. Similar research on autonomous vehicles shows that people trust such vehicles more if they're given some personality, and the passengers are given some strategic control such as the ability to select routes or even just to order the car to stop.
As for behavioral economics, I get my students to read Danny Kahneman's Nobel prize lecture. For more technical detail, there's a volume of papers Danny edited just before that with Tom Gilovich and Dale Griffin [770], or the pop science book ‘Thinking, Fast and Slow’ that he wrote afterwards [1007]. An alternative view, which gives the whole history of behavioral economics, is Dick Thaler's ‘Misbehaving: The Making of Behavioural Economics’ [1877]. For the applications of this theory in government and elsewhere, the standard reference is Dick Thaler and Cass Sunnstein's ‘Nudge’ [1879]. Dick's later second thoughts about ‘Sludge’ are at [1878].
For a detailed history of passwords and related mechanisms, as well as many empirical results and an analysis of statistical techniques for measuring both guessability and recall, I strongly recommend Joe Bonneau's thesis [290], a number of whose chapters ended up as papers I cited above.
Finally, if you're interested in the dark side, ‘The Manipulation of Human Behavior’ by Albert Biderman and Herb Zimmer reports experiments on interrogation carried out after the Korean War with US Government funding [240]. Known as the Torturer's Bible, it describes the relative effectiveness of sensory deprivation, drugs, hypnosis, social pressure and so on when interrogating and brainwashing prisoners. As for the polygraph and other deception-detection techniques used nowadays, the standard reference is by Aldert Vrij [1974].
Notes
1 1 The story is told in detail in chapter 9 of the second edition of this book, available free online.
2 2 Very occasionally, a customer can confuse the bank; a 2019 innovation was the ‘callhammer’ attack, where someone phones up repeatedly to ‘correct’ the spelling of ‘his name’ and changes it one character at a time into another one.
3 3 Our university's auditors wrote in their annual report for three years in a row that we should have monthly enforced password change, but couldn't provide any evidence to support this and weren't even aware that their policy came ultimately from NIST. Unimpressed, we asked the chair of our Audit Committee to appoint a new lot of auditors, and eventually that happened.
4 4 NIST SP 800-63-3
5 5 This was done to undermine an argument by then FBI Director James Comey that the iPhone was unhackable and so Apple should be ordered to produce an operating system upgrade that created a backdoor; see section 26.2.7.4.
6 6 Government attempts to set up single sign-on for public services have been less successful, with the UK ‘Verify’ program due to be shuttered in 2020 [1394]. There have been many problems around attempts to entrench government's role in identity assurance, which I'll discuss further in the chapter on biometrics, and which spill over into issues from online services to the security of elections. It was also hard for other private-sector firms to compete because of the network effects enjoyed by incumbents. However in 2019 Apple announced that it would provide a new, more privacy-friendly single sign-on mechanism, and use the market power of its app store to force websites to support it. Thus the quality and nature of privacy on offer is becoming a side-effect of battles fought for other motives. We'll analyse this in more depth in the chapter on economics.
7 7 This doesn't work for branchless banks like Monzo; but they do take a video of you when you register so that their call centre can recognise you later.
8 8 There's been pushback from users who see a ReCAPTCHA saying ‘click on all images containing a helicopter’ and don't want to help in military AI research. Google's own staff protested at this research too and the military program was discontinued. But other users still object
