WHAT DO THEY WANT US TO THINK?
H.1Using Short-Term Turnover to Measure Long-Term Problems
H.2Sudden Turns for the Worse
H.3Designating Myths
H.4Rhetorical Flourishes
I.DEBATES: WHAT IF THEY DISAGREE?
I.1Causality Debates
I.2Equality Debates
I.3Policy Debates
PART 3STAT-SPOTTING ON YOUR OWN
J.SUMMARY: COMMON SIGNS OF DUBIOUS DATA
K.BETTER DATA: SOME CHARACTERISTICS
L.AFTERWORD: IF YOU HAD NO IDEA THINGS WERE THAT BAD, THEY PROBABLY AREN’T
M.SUGGESTIONS FOR THOSE WHO WANT TO CONTINUE STAT-SPOTTING
ACKNOWLEDGMENTS
NOTES
INDEX
PREFACE TO THE 2013 EDITION
I was introduced to thinking about bad statistics when I read Darrell Huff’s How to Lie with Statistics as a first-year college student. By the time I checked out the library’s copy, that book was more than ten years old, and the volume showed a lot of wear. Huff’s examples no longer seemed current, and I particularly remember that the graphs he reproduced from newspaper articles struck me as somehow old-fashioned. Nonetheless, that book made a bigger impression on me than anything else I read during my first year in college. I understood that, even if the book’s examples were no longer timely, Huff’s lessons were timeless. As the years went by, I found myself recalling How to Lie with Statistics when I encountered dubious numbers in my reading. I discovered that people continued to get confused by statistics in the same ways Huff had identified.
Updating Stat-Spotting has forced me to think about what’s timely and what’s timeless. This book offers a catalog of common problems that can be found in statistics, particularly the sorts of numbers that pop up when people are debating social issues. These problems—the errors in reasoning people make when they present statistics—don’t go away; people keep making the same mistakes, sometimes because they don’t know any better, sometimes because they’re hoping to mislead an audience that isn’t able to spot what’s wrong with the numbers. These timeless lessons—knowing how to recognize some common ways statistics can be flawed—are what I hope you will take from this book.
The book also contains examples meant to illustrate each of the errors I describe. I chose most of these examples because they seemed timely (and, I hoped, engaging). Inevitably, as the years pass, these examples age; they no longer seem to have been “ripped from the headlines.” So, given the opportunity to update Stat-Spotting, I had to decide what needed to be replaced.
I chose to make some selective changes: (1) I revised all the benchmark statistics because a benchmark that isn’t up-to-date isn’t that useful; (2) I updated the list of resources at the end of the book to give readers a better sense of where they could find current information; (3) I added an entire section on the rhetorical uses of statistics, complete with new problems to be spotted and new examples illustrating those problems; and (4) in a few cases where I knew the debate had evolved in important ways, I revised the discussion of an example or added newer, better references. However, most of the examples remain unchanged. They don’t strike me as antiquated, and I continue to feel that a lot of them are pretty interesting. More importantly, I hope that the people who read this book will focus more on the timeless lessons and less on the timeliness of the examples.
| PART 1 | |
| GETTING STARTED |
A
SPOTTING QUESTIONABLE NUMBERS
The billion is the new million. A million used to be a lot. Nineteenth-century Americans borrowed the French term millionnaire to denote those whose wealth had reached the astonishing total of a million dollars. In 1850, there were 23 million Americans; in the 1880 census, New York (in those days that meant Manhattan; Brooklyn was a separate entity) became the first U.S. city with more than one million residents.
At the beginning of the twenty-first century, a million no longer seems all that big. There are now millions of millionaires (according to one recent estimate, about 8.6 million U.S. households have a net worth of more than $1 million, not counting the value of their principal residences).1 Many houses are priced at more than a million dollars. The richest of the rich are billionaires, and even they are no longer all that rare. In fact, being worth a billion dollars is no longer enough to place someone on Forbes magazine’s list of the four hundred richest Americans; some individuals have annual incomes exceeding a billion dollars.2 Discussions of the U.S. economy, the federal budget, or the national debt speak of trillions of dollars (a trillion, remember, is a million millions).
The mind boggles. We may be able to wrap our heads around a million, but billions and trillions are almost unimaginably big numbers. Faced with such daunting figures, we tend to give up, to start thinking that all big numbers (say, everything above 100,000) are more or less equal. That is, they’re all a lot.
Envisioning all big numbers as equal makes it both easier and harder to follow the news. Easier, because we have an easy way to make sense of the numbers. Thus, we mentally translate statements like “Authorities estimate that HIV/AIDS kills nearly three million people worldwide each year” and “Estimates are that one billion birds die each year from flying into windows” to mean that there are a lot of HIV deaths and a lot of birds killed in window collisions.
But translating all big numbers into a lot makes it much harder to think seriously about them. And that’s just one of the ways people can be confused by statistics—a confusion we can’t afford. We live in a big, complicated world, and we need numbers to help us make sense of it. Are our schools failing? What should we do about climate change? Thinking about such issues demands that we move beyond our personal experiences or impressions. We need quantitative data—statistics—to guide us. But not all statistics are equally sound. Some of the numbers we encounter are pretty accurate, but others aren’t much more than wild guesses. It would be nice to be able to tell the difference.
This book may help. My earlier books—Damned Lies and Statistics and More Damned Lies and Statistics—offered an approach to thinking critically about the statistics we encounter.3 Those books argued that we need to ask how numbers are socially constructed. That is, who are the people whose calculations produced the figures? What did they count? How did they go about counting? Why did they go to the trouble? In a sense, those books were more theoretical; they sought to understand the social processes by which statistics are created and brought to our attention. In contrast, this volume is designed to be more practical—it is a field guide for spotting dubious data. Just as traditional field guides offer advice on identifying birds or plants, this book presents guidelines for recognizing questionable statistics, what I’ll call “stat-spotting.” It lists common problems found in the sorts of numbers that appear in news stories and illustrates each problem with an example. Many of these errors are mentioned in the earlier books, but this guide tries to organize them around a set of practical questions that you might ask when encountering a new statistic and considering whether it might be flawed. In addition, all of the examples used to illustrate the various problems are new; none appear in my other books.
This
