statistics that have serious flaws. This is important, because most of us have a tendency to equate numbers with facts, to presume that statistical information is probably pretty accurate information. If that’s wrong—if lots of the figures that we encounter are in fact flawed—then we need ways of assessing the data we’re given. We need to understand the reasons why unreliable statistics find their way into the media, what specific sorts of problems are likely to bedevil those numbers, and how to decide whether a particular figure is accurate. This book is not a general discussion of thinking critically about numbers; rather, it focuses on common flaws in the sorts of figures we find in news stories.
I am a sociologist, so most of the examples I have chosen concern claims about social problems, just as a field guide written by an economist might highlight dubious economic figures. But the problems and principles discussed in this book are applicable to all types of statistics.
This book is divided into major sections, each focusing on a broad question, such as: Who did the counting? or What did they count? Within each section, I identify several problems—statistical flaws related to that specific issue. The discussion of each problem lists some things you can “look for” (that is, warning signs that particular numbers may have the flaw being discussed), as well as an example of a questionable statistic that illustrates the flaw. (Some of the examples could be used to illustrate more than one flaw, and in some cases I note an example’s relevance to points discussed elsewhere in the book.) I hope that reading the various sections will give you some tools for thinking more critically about the statistics you hear from the media, activists, politicians, and other advocates. However, before we start to examine the various reasons to suspect that data may be dubious, it will help to identify some statistical benchmarks that can be used to place other figures in context.
B
BACKGROUND
Having a small store of factual knowledge prepares us to think critically about statistics. Just a little bit of knowledge—a few basic numbers and one important rule of thumb—offers a framework, enough basic information to let us begin to spot questionable figures.
B1Statistical Benchmarks
When interpreting social statistics, it helps to have a rough sense of scale. Just a few benchmark numbers can give us a mental context for assessing other figures we encounter. For example, when thinking about American society, it helps to know that:
The U.S. population is something over 300 million (about 312 million in 2011).
Each year, about 4 million babies are born in the United States (the 2011 total was 3,953,593).1 This is a surprisingly useful bit of information, particularly for thinking about young people. How many first graders are there? About 4 million. How many Americans under age 18? Roughly 4 million × 18, or 72 million. Young people are about evenly divided by sex, so we can calculate that there are around 2 million 10-year-old girls, and so on.
About 2.5 million Americans die each year (there were 2,513,171 deaths recorded in 2011). Roughly one in four people dies of heart disease (23.7 percent in 2011), and cancer kills nearly as many, so that about half (1,171,652 deaths in 2011, or 46.6 percent) die of either heart disease or cancer. In comparison, some heavily publicized causes of death are much less common: for instance, traffic accidents killed roughly 35,000 people in 2011, breast cancer 41,000, suicide 38,000, homicide 16,000, and HIV/AIDS 8,000. That is, each of these causes accounted for less than 2 percent of all deaths.2
Statistics about race and ethnicity are complicated because these categories have no precise meaning. In general, however, people who identify themselves as blacks or African Americans account for just about 13 percent of the population—about one person in eight. (Remembering that the overall population is more than 300 million, we can figure that there are about 40 million black Americans: 300 million ÷ 8 = 37.5 million.) Slightly more—over 16 percent, or about one in six—identify themselves as Hispanic or Latino. But people cannot be divided neatly into racial or ethnic categories. Most government statistics treat Hispanic as an ethnic rather than a racial category, because Hispanics may consider themselves members of various races. Thus, in a 2007 press release announcing that “minorities” now accounted for one-third of the U.S. population, the census bureau announced that “the non-Hispanic, single-race white population [is] 66 percent of the total population.”3 Note the awkward wording: “non-Hispanic” is used because some people who classify their ethnicity as Hispanic also list their race as white; “single-race” because some people report mixed ancestry (such as having an American Indian ancestor). In short, the bureau is classifying as minority-group members some people who may consider themselves white. No single, authoritative method exists for classifying race and ethnicity. Still, a rough sense of the ethnic and racial makeup of the U.S. population can be useful.
Having this small set of basic statistical benchmarks for the overall population can help us place the numbers we hear in context. Sometimes, when we compare a statistic to these benchmarks, alarm bells may ring because a number seems improbably large or small. For instance, all other things being equal, we might expect blacks to account for about one-eighth of people in various circumstances: one-eighth of college graduates, one-eighth of prison inmates, and so on. If we learn that the actual proportion of blacks in some group is higher or lower, that information might tell us something about the importance of race in that category.
It isn’t necessary to memorize all of these figures. They are readily available. One of the most useful sources for basic statistics—just crammed full of official figures—is the annual Statistical Abstract of the United States. It is accessible online, and most libraries have a printed copy.4 Whether you can remember these basic numbers or whether you need to look them up, they can help you critically evaluate new statistics. We will have occasion to use these benchmarks (and we will identify a couple of others) later in this book.
| LOOK FORNumbers inconsistent with benchmark figures |
EXAMPLE: BATTERING DEATHS
A Web site claims that “more than four million women are battered to death by their husbands or boyfriends each year.”5 Right away, our benchmarks help us recognize that this number can’t be correct. With only about 16,000 homicides annually, there is no chance that there could be 4 million women killed in battering incidents. In fact, 4 million exceeds the nation’s annual 2.4 million death toll from all causes. We have no way of knowing what led the creator of the Web site to make this error, but there can be no doubt that this number is simply wrong.
Although this particular figure is clearly outlandish, I have seen it repeated on a second Web site. Statistics–both good and bad–tend to be repeated. People assume that numbers must be facts; they tell themselves that somebody must have calculated the figures, and they don’t feel obliged to check them, even against the most obvious benchmarks. For example, neither whoever created the 4-million-battering-deaths statistic nor the people who repeated that figure thought to ask: “Does this number for battering deaths exceed the total number of deaths from all causes?” Instead, folks feel free to repeat what they understand to be factual information. As a result, bad numbers often take on a life of their own: they continue being repeated, even after they have been thoroughly debunked. This is particularly true in the Internet age, when it is so easy to circulate information. A bad statistic is harder to kill than a vampire.
B2Severity and Frequency
In addition to having our small set of statistical benchmarks, it is useful to keep in mind one rule of thumb: in general, the worse things are, the less common they are.
Consider
