example comes from Ackoff (1978).
• The number of conditions necessary to establish causality varies from discipline to discipline and even author to author. For example, the epidemiologist Hill (1965) gave nine rules. We stick with just three.
• While observational data can be useful, they are no substitute for an experiment (if an experiment can be conducted!):
But even if done perfectly, an observational study can only approach, but never reach, the credibility of randomization in assuring that there is no missing third variable that accounts for the differences observed in the experimental outcome. (Wainer, 2016, p. 48)
• The book by Schrage (2014) is entertaining and describes numerous business experiments; most of these are small, inexpensive experiments. The book by Holland and Cochran (2005), which is written for laymen, describes several larger, more complicated experiments. The article by Ganguly and Euchner (2018) describes the approach of the Goodyear Tire Company to experimentation and gives many interesting examples, including one where they conducted an experiment to determine whether tire‐pressure monitoring equipment could generate more than enough savings to pay for the monitoring equipment by reducing roadside breakdowns of tractor‐trailers. It is worthwhile to read materials such as these, for it is important for the novice experimenter to develop an idea of what has been done and what is possible.
• The idea that small sample sizes, small effect sizes, and lots of noise can lead to false positives and even sign reversals (truly positive coefficients being estimated as negative) is discussed in terms about as nontechnical as possible in Gelman and Carlin (2014), but you'll have to know what “power” is to follow the argument, so maybe you should wait until after Chapter 2 to read it.
Section 1.5 “Improving Website Design”
All the web tests, in particular the results in Table 1.6, are real. Due to difficulties obtaining permissions for the original web ads, some of the web ads were mocked‐up to simulate the real ads. The photograph in Figure 1.5 is from pixabay.com, and the photograph in Figure 1.7 is from pexels.com.
GuessTheTest.com is a resource for digital marketers who want objective A/B test case studies and helpful information to get split‐testing ideas, insights, and best practices. There are many aspects of A/B testing on the web that are not covered in this book, and the interested reader may profitably spend some time at this website. Also, if you think you're any good at predicting the outcome of an A/B web test, to disabuse yourself of such an errant notion, try guessing at a dozen or so of the many cases presented at this website and see if you can beat 50% accuracy by a statistically significant amount.
• We barely scratched the surface of A/B testing, which, according to two recent surveys, is the most important topic in business: a survey of online marketers found “Conversion Rate Optimization” to be a top priority for the foreseeable future (SalesForce.com, 2014); a survey of businesses that engage in conversion rate optimization used A/B testing more than any other method (Econsultancy, 2015).
• An entertaining layman's article on the rise of A/B testing can be found in Wired magazine (Christian, 2012). On A/B testing and the Obama presidential campaigns, see the interesting article in Bloomberg Businessweek by Joshua Green (2012). This is of historical interest because the Obama campaign was the first to really use analytics for fundraising and get‐out‐the‐vote activities. For those who want to learn more about the technology behind website testing and the types of tests that are possible on websites, we recommend the chapter on web testing in Waisberg and Kashuk's book titled Web Analytics 2.0 (Waisberg and Kaushik, 2009) or the succinct book by McFarland (2012) with the catchy title Experiment!.
There are also numerous other books with more in‐depth coverage of website experiments such as Siroker and Koomen (2013). Technology tools for website testing are rapidly evolving, and the evaluation of software tools is a critical first step in any website testing program. A good article on the mechanics of A/B testing on the web is Kohavi et al. (2009b). He also co‐authored an informative article, “Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained” (Kohavi et al., 2012), as well as “Seven Rules of Thumb for Web Site Experimenters” (Kohavi et al., 2014).
Section 1.6 “A Brief History of Experiments”
• An entertaining history of experimental design that gives proper due to Gosset is Ziliak (2014). To see how the methods that Gosset developed were used for commercial purposes at the Guinness Brewery, see Ziliak (2008).
• Through the 1980s, American cars didn't last long. In the 1960s, American automobile manufacturers offered warranties for 4000 miles or three months, whichever came first. By the 1990s, these warranties had been increased to 70000 miles or seven years (Lightstone et al., 1993, p. 774).
• W. E. Deming was a statistics professor at NYU in the late 1930s when he developed “statistical quality control.” He took his idea to the Detroit automakers, but they didn't want his advice. After WWII, he then took his ideas to Japan, which embraced them wholeheartedly. Deming is largely credited for the postwar Japanese economic miracle. How would the course of history have been altered, if Detroit had embraced Deming instead of rejecting him?
• As a general rule, observational results can't be trusted until they have been verified by a well‐designed experiment. Medical practices often become popular as a result of observational studies, only to be overturned years later by experiments, long after much damage has been done. Some examples are hormone replacement therapy for menopausal women, stenting for coronary disease, and a specific medicine thought to retard heart disease (fenofibrate to treat hyperlipidemia). See Huded et al. (2013) for further examples.
1.9 Statistics Refresher
If these questions utterly confuse you, you probably shouldn't be reading this book.
Which of the below questions you can answer depends on the statistics course you took. You should be able to answer many of these questions. If you can answer them using the statistical software you already know but not with R, that's okay. We give R code for everything we do in this book. If you can't answer a particular question, don't worry. The methods will be explained as needed later in the book.
1 What are the following percentiles for a ‐distribution?0.900.950.975
2 For a ‐distribution, what value of gives the following proportion in the upper tail?0.010.00050.08
3 What are the following percentiles for a ‐distribution with the given degrees of freedom (df)?0.90, df=100.95, df=200.975, df=30
4 For a ‐distribution with the given degrees of freedom (df), what value of gives the following proportions in the upper tail?0.01, df=100.0005, df=200.08, df=30
5 Use the data in file SR1.csv and perform a two‐sided test of the null hypothesis that for Specifically state the non‐rejection region and the rejection region.
6 Use the data in file SR1.csv and produce a 95% confidence interval for the population
