whether there is enough data to draw a conclusion. Testing for significance is one of the tools we use in analyzing A/B tests, and Chapter 2 will show you how to do it. As we will explain in the next few sections, we need more than just the lift numbers to perform the significance test.
Most website testing managers will tell you that more than half of the website tests that they run are not significant, meaning that they cannot conclude that one version is better than the other. For example, in the video icon test in Figure 1.7, there were no significant differences in the % of users who viewed the product detail pages or the average sales per session. If we looked at the raw data, there were probably some small differences, but that difference was not great enough to rise to the level of significance. The analyst has wisely chosen not to report the lift numbers, and instead simply said, “there was no significant difference.” While the manager who came up with this video icon idea might not be too happy to find that it doesn't work, it is important to know that it doesn't work so that attention can be shifted to more promising improvements to the website. Smart testing managers realize that it is important to run many tests to find the features of the website that really do change user behavior.
Exercises
1 1.5.1 Visit a retail website and identify five opportunities for A/B tests on the website. For each test, clearly define the A and B treatments that you would test and identify a response variable to measure performance.
2 1.5.2 Find an article that reports the results of a medical experiment, a business experiment, or a psychological experiment. How are the results reported? Do they use a graph to display the data? Does the article indicate whether the difference between treatments was significant?
3 1.5.3s. Visit a retail store and identify five opportunities for A/B tests. For each test, clearly define the A and B treatments that you would test and identify a response variable to measure performance.
1.6 A Brief History of Experiments
Experiments are as old as the bible. From The Book of Daniel (1, 11‐16),
Daniel then said to the guard whom the chief official had appointed over Daniel, Hananiah, Mishael, and Azariah, ”Please test your servants for ten days: Give us nothing but vegetables to eat and water to drink. Then compare our appearance with that of the young men who eat the royal food, and treat your servants in accordance with what you see.” So he agreed to this and tested them for ten days. At the end of the ten days they looked healthier and better nourished than any of the young men who ate the royal food. So the guard took away their choice food and the wine they were to drink and gave them vegetables instead.
The first clinical trial was conducted in 1747 by the Scottish physician James Lind, who was trying to find a cure for scurvy. Scurvy was a serious problem, since it killed more British sailors than the French and the Spanish combined. After two months at sea, when the men were afflicted with scurvy, Lind divided 12 sick sailors into six groups of 2. Each day the groups were administered cider, 25 drops of sulfuric acid, vinegar, a cup of seawater, and barley water, and the final group received two oranges and one lemon. After six days the fruit ran out, but one sailor was completely recovered and the other was almost recovered.
Randomization was introduced into experimental design in the nineteenth century by Peirce and Jastrow (1885) (many people incorrectly attribute this to R. A. Fisher in the twentieth century, but they are wrong). Designed experiments that were not randomized, but instead were balanced, were developed by William Gosset, pseudonymously writing as “Student” (who also invented Student's
The development of greatly increased agricultural productivity in the twentieth century has rested largely on field experiments in which new varieties of crops (and new agricultural practices) are compared to standard ones. So important is this empirical testing to agricultural progress that a large part of modern statistical design of experiments actually grew up in the context of agricultural experimentation.
Figure 1.8 US agricultural output and input in the twentieth century.
In agriculture, experimental design is used to maximize output by determining the proper amount of fertilizer, growing conditions, etc. and also in the breeding of plants that have higher yields. The increase in agricultural production can be seen in Figure 1.8; note the rapid rise beginning in about 1940, after experimental methods began to spread throughout the agricultural sector.
The next field to be revolutionized by the design of experiments was manufacturing. Chemical production greatly increased as a result of the design of experiments and this spread to other process industries. Variability in successive batches of output was decreased, allowing for a more uniform, higher quality product and a concomitant decrease in waste. In the 1950s, the statistical pioneer W. Edwards Deming taught statistical methods to Japanese manufacturers at a time when “made in Japan” was synonymous with “low quality.” Deming taught them to greatly increase quality and output, especially in the automotive and electronics industries.
Figure 1.9 US car imports from Japan.
By the 1970s, Japanese exports of autos and electronics to the United States threatened the American auto industry and decimated its electronics industry. American consumers began to realize that the Japanese cars were much better made: “The Japanese vehicle surpassed the American car in measurable, definable, observable quality” (Lightstone et al., 1993, p. 777). American cars had lifespans of 50 000–75 000 miles, while Japanese cars could be counted on to last more than 100 000 miles. Figure 1.9 shows the remarkable growth of American imports of Japanese cars from the 1960s through the 1980s. (After the 1980s, Japan started building car plants in the United States to avoid import restrictions, so it kept selling lots of cars in the United States, but was no longer exporting them from Japan to the United States.) About this time Six Sigma and Total Quality Management, both of which emphasize designed experiments, were taken up by many companies, the former's most famous adherent being General Electric. Medical research adopted experimental design and embraced clinical trials. Previously, research had been of the variety: “Dr. Smith saw six patients and this is what he thinks.” Capital One came to dominate the credit card industry by relying on experimental design to optimize its postal credit card solicitations. In the present day, design of experiments continues to revolutionize various fields including economics with its field experiments, marketing with its web analytics, and even art conservation, as well as many others too numerous to mention here.
1.7 Chapter Exercises
1 1.7.1 Define the following terms:independent variable /
