Certainly, the article’s Author didn’t ask many probing, critical questions about the CDF’s claim. Impressed by the statistic, the Author repeated it—well, meant to repeat it. Instead, by rewording the CDF’s claim, the Author created a mutant statistic, one garbled almost beyond recognition.
But people treat mutant statistics just as they do other statistics—that is, they usually accept even the most implausible claims without question. For example, the Journal Editor who accepted the Author’s article for publication did not bother to consider the implications of child victims doubling each year. And people repeat bad statistics: the Graduate Student copied the garbled statistic and inserted it into the dissertation prospectus. Who knows whether still other readers were impressed by the Author’s statistic and remembered it or repeated it? The article remains on the shelf in hundreds of libraries, available to anyone who needs a dramatic quote. The lesson should be clear: bad statistics live on; they take on lives of their own.
This is a book about bad statistics, where they come from, and why they won’t go away. Some statistics are born bad—they aren’t much good from the start, because they are based on nothing more than guesses or dubious data. Other statistics mutate; they become bad after being mangled (as in the case of the Author’s creative rewording). Either way, bad statistics are potentially important: they can be used to stir up public outrage or fear; they can distort our understanding of our world; and they can lead us to make poor policy choices.
The notion that we need to watch out for bad statistics isn’t new. We’ve all heard people say, “You can prove anything with statistics.”* My title, Damned Lies and Statistics, comes from a famous aphorism (usually attributed to Mark Twain or Benjamin Disraeli): “There are lies, damned lies, and statistics.”2 There is even a useful little book, still in print after more than forty years, called How to Lie with Statistics.3
Statistics, then, have a bad reputation. We suspect that statistics may be wrong, that people who use statistics may be “lying”—trying to manipulate us by using numbers to somehow distort the truth. Yet, at the same time, we need statistics; we depend upon them to summarize and clarify the nature of our complex society. This is particularly true when we talk about social problems. Debates about social problems routinely raise questions that demand statistical answers: Is the problem widespread? How many people—and which people—does it affect? Is it getting worse? What does it cost society? What will it cost to deal with it? Convincing answers to such questions demand evidence, and that usually means numbers, measurements, statistics.
But can’t you prove anything with statistics? It depends on what “prove” means. If we want to know, say, how many children are “gunned down” each year, we can’t simply guess—pluck a number from thin air: one hundred, one thousand, ten thousand, 35 trillion, whatever. Obviously, there’s no reason to consider an arbitrary guess “proof” of anything. However, it might be possible for someone—using records kept by police departments or hospital emergency rooms or coroners—to keep track of children who have been shot; compiling careful, complete records might give us a fairly accurate idea of the number of gunned-down children. If that number seems accurate enough, we might consider it very strong evidence—or proof.
The solution to the problem of bad statistics is not to ignore all statistics, or to assume that every number is false. Some statistics are bad, but others are pretty good, and we need statistics—good statistics—to talk sensibly about social problems. The solution, then, is not to give up on statistics, but to become better judges of the numbers we encounter. We need to think critically about statistics—at least critically enough to suspect that the number of children gunned down hasn’t been doubling each year since 1950.
A few years ago, the mathematician John Allen Paulos wrote Innumeracy, a short, readable book about “mathematical illiteracy.”4 Too few people, he argued, are comfortable with basic mathematical principles, and this makes them poor judges of the numbers they encounter. No doubt this is one reason we have so many bad statistics. But there are other reasons, as well.
Social statistics describe society, but they are also products of our social arrangements. The people who bring social statistics to our attention have reasons for doing so; they inevitably want something, just as reporters and the other media figures who repeat and publicize statistics have their own goals. Statistics are tools, used for particular purposes. Thinking critically about statistics requires understanding their place in society.
While we may be more suspicious of statistics presented by people with whom we disagree—people who favor different political parties or have different beliefs—bad statistics are used to promote all sorts of causes. Bad statistics come from conservatives on the political right and liberals on the left, from wealthy corporations and powerful government agencies, and from advocates of the poor and the powerless. In this book, I have tried to choose examples that show this range: I have selected some bad statistics used to justify causes I support, as well as others offered to promote causes I oppose. I hope that you and everyone else who reads this book will find at least one discomforting example of a bad statistic presented in behalf of a cause you support. Honesty requires that we recognize our own errors in reasoning, as well as those of our opponents.
This book can help you understand the uses of social statistics and make you better able to judge the statistics you encounter. Understanding this book will not require sophisticated mathematical knowledge. We will be talking about the most basic forms of statistics: percentages, averages, and rates—what statisticians call “descriptive statistics.” These are the sorts of statistics typically addressed in the first week or so of an introductory statistics course. (The remainder of that course, like all more advanced courses in statistics, covers “inferential statistics,” complex forms of reasoning that we will ignore.) This book can help you evaluate the numbers you hear on the evening news, rather than the statistical tables printed in the American Sociological Review and other scholarly journals. Our goal is to learn to recognize the signs of really bad statistics, so that we won’t believe—let alone repeat—claims about the number of murdered children doubling each year.
_________
*For reasons that will become obvious, I have decided not to name the Graduate Student, the Author, or the Journal Editor. They made mistakes, but the mistakes they made were, as this book will show, all too common.
*For instance, since only child victims are at issue, a careful analysis would control for the relative sizes of the child population in the two years. We also ought to have assurances that the methods of counting child gunshot victims did not change over time, and so on.
*This is a criticism with a long history. In his book Chartism, published in 1840, the social critic Thomas Carlyle noted: “A witty statesman said you might prove anything with figures.”
1
THE IMPORTANCE OF SOCIAL STATISTICS
Nineteenth-century Americans worried about prostitution; reformers called it “the social evil” and warned that many women prostituted themselves. How many? For New York City alone, there were dozens of estimates: in 1833, for instance, reformers published a report declaring that there were “not less than 10,000” prostitutes in New York (equivalent to about 10 percent of the city’s female population); in 1866, New York’s Methodist bishop claimed there were more prostitutes (11,000 to 12,000) than Methodists in the city; other estimates for the period ranged as high as 50,000. These reformers hoped that their reports of widespread
