believes a theory except its originator.
Everyone believes an experiment except the experimenter. 1
The decision to design a credible experiment sets you on a path to research with impact. Along this path you will make many decisions. This book prepares you to anticipate the choices you will face to plan and achieve an experiment that you the experimentalist also can believe.
There are two kinds of material to consider with respect to experimental methods: the mechanics of measurement and the strategy of experimentation. This book emphasizes the strategy and tactics of experiment planning.
The fact is that laboratories need people who can answer difficult questions experimentally. We offer this text to answer this need, to promote balanced competence in the field of experimental work. Abraham Lincoln is attributed with the saying “Give me six hours to chop down a tree and I will spend the first four sharpening the axe.”2 If there is anything like a “law of compound interest” in experimental work, then the effort spent to improve an experiment plan will return a bigger payoff than the same effort applied to fine‐tuning transducers.
It is relatively easy to deal with sensors, calibrations, and corrections: those are concrete, factual bits of technology. The basic mechanisms are well known and tested. Could this be why there are so many technical references to transducers and so few to experiment planning?
Strategy is not concrete, however; it contains large elements of opinion, experience, and personal preference. One cannot prove that a particular strategy is best! What seems to be a “clever insight” to one person may seem “dull and pedestrian” to another.
The ideas presented in this text were developed over 60 years of teaching and consulting on experimental methods. They provide an outline of a systematic way of designing experiments of provable accuracy. Not the only way, certainly, but at least one way. Each reader who has much experimental experience will have techniques to add to the list – we ask and welcome your feedback.
Novices might be inclined to take this text too literally, as though experiment planning were a quantitative discipline with rules that always worked. That would be a mistake. One must be flexible in the laboratory – following a sound basic philosophy – but taking advantage of the specific opportunities each experiment offers.
1.1 The Responsibility of an Experimentalist
People have always been impressed by data, as though it could never be wrong. As a young experimentalist, the saying that leads this chapter impressed me greatly. At first, I thought it was a clever play on words. Then I began to take it more seriously. It is true. People do seem to put more credence to experimental results than in analysis. I cannot tell you how many times I have heard someone say, with a tone of absolute finality, “Well, you can't argue with the data!” Fact is, you can and should.
This places a heavy responsibility on the experimenter. One may respond to such responsibility by taking great pains to establish the credibility of the data before actually taking data “for the record.” Another response is to simply crash ahead and take data on a plausible but not proven experiment, because you can't think of anything you have done wrong. After all, these latter folks seem to think, “If I don't like the data, I will do it again, differently.”
That latter view undermines credibility. If you feel free to ignore data because you don't like it or don't understand it, then you haven't run an experiment, you have just been playing around in the lab.
Many of the suggestions offered in this work are related to establishing credibility before the production data are taken: calibration of instruments, running baseline tests, etc. In this respect, an experiment is like an iceberg – 90% of the effort is “unseen,” whereas only about 10% of the effort is invested in taking the production data.
Practically all of the experimentalists we know find the saying as a unique privilege and honor and as exhortation to produce the very best science. It gives us pause that people may base their design choices, their engineering, or even their lifestyle choices on their trust of our reported results and conclusions. This is a weighty responsibility worthy of doing our very best.
1.2 Losses of Credibility
Sadly, not all reported science is credible. In 2005, the Journal of the American Medical Association published “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research,” by J.P.A. Ioannidis (now at Stanford University) (Ioannidis 2005). Among his findings, “Five of six highly‐cited nonrandomized studies had been contradicted or had found stronger effects” than other, better‐designed studies. In other words, only one in six (17%) of these highly influential medical studies was credible.
Some scientific studies lost credibility due to poorly designed experiments. We offer this text to our readers to help them plan and execute experiments that are credible. There is no excuse for poorly designed experiments.
Retractions of scientific studies are no longer rare. The New York Times reported in 2011, that a “well‐known psychologist…whose work has been published widely in professional journals falsified data and made up entire experiments” (Carey 2011). In his case, more than 50 articles were retracted. A site devoted to retracted science, Retraction Watch, can be found at http://retractionwatch.com.
A whole industry of fact‐checkers has come into existence, purportedly to expose the false and reveal the true. We know, however, even fact‐checkers must be checked. Let this motivate us.
1.3 Recovering Credibility
Science and engineering recently found a champion in Ioannidis.
By careful planning and execution of our own experiments, we too become champions of credibility.
As experimentalists, we must be our own front‐line fact‐checker, tackling errors as they arise. We put effort into uncovering every reason to not believe our method, equipment, results, and especially our modeling. We diligently report the uncertainty of our measurements.
When you have completed an experiment, you must have assembled so much evidence of credibility that, like it or not, you have to believe the data. The experimenter must be on guard all the time, looking for anomalies, looking for ways to challenge the credibility of the experiment. If something unexpected happens, the diligent experimenter will find a way to challenge it and either confirm it or refute it. Let “except the experimenter” spur you to create experiments you can believe and defend.
1.4 Starting with a Sharp Axe
In this text, we reintroduce one of the sharpest tools for designing an experiment, the computer program Gosset. Gosset was developed at AT&T Bell Laboratories in the 1980s. Although a few companies and researchers have used Gosset extensively, experimentalists across the world can benefit from its powerful features. The developers of Gosset released it for free to the public domain in 2018. We demonstrate Gosset in Chapter 9. With skill using Gosset, your axe will be sharper.
We also encourage use of one of the sharpest tools for data analysis, the R language. The R language is
