impression.
We can suspect that the ease with which graphic artists can use computer software to manipulate the sizes of images and fonts contributed to this mangled image. Attractive graphs are preferable to ugly graphs–but only so long as artistic considerations don’t obscure the information the graph is supposed to convey.
C4Careless Calculations
Many statistics are the result of strings of calculations. Numbers—sometimes from different sources—are added, multiplied, or otherwise manipulated until a new result emerges. Often the media report only that final figure, and we have no easy way of retracing the steps that led to it. Yet when statistics seem incredible, when we find ourselves wondering whether things can possibly be that bad, it can be worth trying to figure out how a number was brought into being. Sometimes we can discover that the numbers just don’t add up, that someone almost certainly made a mistake.
| LOOK FORAs with other sorts of blunders, numbers that seem surprisingly high or lowNumbers that seem hard to produce–how could anyone calculate that? |
EXAMPLE: DO UNDERAGE DRINKERS CONSUME 18 PERCENT OF ALCOHOL?
A 2006 study published in a medical journal concluded that underage and problem drinkers accounted for more than a third of the money spent on alcohol in the United States.9 The researchers calculated that underage drinkers (those age 12–20) consume about 18 percent of all alcoholic drinks–more than 20 billion drinks per year. Right away, we notice that that’s a really big number. But does it make sense?
Our benchmarks tell us that each recent age cohort contains about 4 million people (that is, there are about 4 million 12-year-olds, 4 million 13-year-olds, and so on). So we can figure there are about 36 million young people age 12–20. If we divide 36 million into 20 billion, we get more than 550 drinks per person per year. That is, young people would have to average 46 drinks per month. That sure seems like a lot.
Of course, many underage people don’t drink at all. In fact, the researchers calculated that only 47.1 percent were drinkers. That would mean that there are only about 17 million underage drinkers (36 million × .471): in order for them to consume 20 billion drinks per year, those young drinkers would have to average around 1,175 drinks per year–nearly 100 drinks per month, or about one drink every eight hours.
But this figure contradicts the researchers’ own data. Their article claims that underage drinkers consume an average of only 35.2 drinks per month. Let’s see: if we use the researchers’ own figures, we find that 17 million underage drinkers × 35.2 drinks per month equals a total of just under 600 million drinks per month, × 12 months per year = equals 7.2 billion drinks by underage drinkers per year–not 20 billion. Somehow, somewhere, someone made a simple arithmetic error, one that nearly tripled the estimate of what underage drinkers consume. According to the researchers, Americans consume 111 billion drinks per year. If youths actually drink 7.2 billion of those, that would mean that underage drinkers account for about 6.5 percent–not 18 percent–of all the alcohol consumed.
The fact that we can’t make the researchers’ own figures add up to 20 billion drinks is not the end of the story.10 One could go on to question some of the study’s other assumptions. For example, although there are some young people who drink daily, we might suspect that drinking–and frequency of drinking–increases with age, that even a large proportion of youths who are “current drinkers” find their opportunities to drink limited mostly to weekends. One might suspect that young drinkers average less than 35 drinks per month. Reducing the estimate by only 5 drinks per month would cut our estimate for total drinks consumed in a year by underage drinkers by another billion. The assumptions that analysts make–even when they don’t make calculation errors–shape the resulting figures.
D
SOURCES: WHO COUNTED–AND WHY?
While it is sometimes possible to spot obvious blunders, most statistics seem plausible—at least they aren’t obviously wrong. But are they right? In trying to evaluate any number, it helps to ask questions. A good first question is, Who produced this figure? That is, who did the counting—and why?
Numbers don’t exist in nature. Every number is a product of human effort. Someone had to go to the trouble of counting. So we can begin by trying to identify the sources for our numbers, to ask who they are, and why they bothered to count whatever they counted.
Statistics come from all sorts of sources. Government agencies crunch a lot of numbers; they conduct the census and calculate the crime rate, the unemployment rate, the poverty rate, and a host of other statistics. Then there are the pollsters who conduct public opinion polls: sometimes they conduct independent surveys, but often they are working for particular clients, who probably hope that the poll results will support their views. And there are researchers who have collected data to study some phenomenon, and who may be more objective—or not. All sorts of people are sources for statistics.
Typically, we don’t have direct access to the folks who create these numbers. Most of us don’t receive the original reports from government agencies, pollsters, or researchers. Rather, we encounter their figures at second or third hand—in newspaper stories or news broadcasts. The editors and reporters who produce the news winnow through lots of material for potential stories and select only a few numbers to share with their audiences.
In other words, the statistics that we consume are produced and distributed by a variety of people, and those people may have very different agendas. Although we might wish them to be objective sources, intent on providing only accurate, reliable information, in practice we know that some sources present statistics selectively, in order to convince us that their positions are correct. They may have a clear interest in convincing us, say, that their new drug is effective and not harmful, or that their industry deserves a tax break. These interests can inspire deliberate attempts to deceive, as when people knowingly present false or unreliable figures; but bad statistics can also emerge for other, less devious reasons.
When researchers announce their results, when activists try to raise concern for some cause they favor, or when members of the media publish or broadcast news, they all find themselves in competition to gain our attention. There is a lot of information out there, and most of it goes unnoticed. Packaging becomes important. To attract media coverage, claims need to be crafted to seem interesting: each element in a story needs to help grab and hold our attention, and that includes statistics. Thus, people use figures to capture our interest and concern; they emphasize numbers that seem surprising, impressive, or disturbing. When we see a statistic, we should realize that it has survived a process of selection, that many other numbers have not been brought to our attention because someone deemed them less interesting.
This competition for public notice affects all sorts of numbers, even those produced by the most reputable sources. When government agencies announce the results of their newest round of number-crunching, they may be tempted to issue a news release that highlights the most interesting, eye-catching figures. Researchers who hope to publish their work in a visible, high-prestige journal may write up their results in ways intended to convince the journal’s editor that theirs is an especially significant study. In the competition to gain attention, only the most compelling numbers survive.
And, as we have already seen in the section on blunders, people sometimes present numbers they don’t understand. They may be sincere—fully convinced of the validity of their own dubious data. Of course this is going to be true for people with whom we disagree—after all, if they’ve come to the wrong conclusions, there must be something wrong with their evidence. But—and this is awkward—the same is often true for those who agree with us. Their hearts may be in the right place,
