an article or book. As this chapter argues, there is an enormous difference between the credibility of a “journey story” about one person’s experience—a sample size of one—and the highest level of research, the preponderance of evidence.
In this chapter, we explore five levels of educational research. These include (1) personal beliefs, (2) personal experiences, (3) collective experience, (4) systematic comparisons, and (5) preponderance of the evidence. When legislators and other policymakers, leaders, or teachers state, “Research shows that …,” they might be referring to any one of these types of research. However, as citizens hoping to implement effective change in our schools, we can only truly rely on the higher levels of research—systematic comparison and preponderance of the evidence—to produce results that could feasibly be applicable and transferrable to our own schools. It is, therefore, essential that when we hear claims about research, we identify which type of research the people making the claims are using as the basis for their conclusions.
Level 1: Personal Beliefs
People are entitled to their own beliefs. That’s a guarantee in the United States (thanks to the First Amendment to the Constitution), as well as in many other nations around the world. The United States is hardly unique in protecting freedom of beliefs. Indeed, the founding documents of the United States drew heavily from thought leaders in England, France, and many other countries (Lepore, 2018) where laws prohibit governments from interfering with the free exercise of anyone’s religious beliefs, right to free speech, and freedom of assembly. As long as our beliefs do not interfere with the rights of others, we are free to believe whatever we wish.
Hand-in-hand with this freedom of personal beliefs, however, is the ability to believe something that is categorically wrong—and to persist in that belief due to personal preference, even in the face of evidence to the contrary. When presented with scientific evidence that defies personal beliefs, it is not unusual for someone to respond, proudly, “Your research may say that out-of-school suspension, punishment for missing homework, corporal punishment for behavioral infractions, and forcing students to stand in the corner with a dunce cap doesn’t work, but it sure worked for me!” These claims are contrary to research and common sense (Reeves, 2011a), but in a free society, we tolerate them—at least as long as the actions associated with these claims do not harm others.
Whether the forum is a meeting of teachers, a public comment portion of a school board meeting, a legislative hearing, or a speech at a political rally, listeners must ask, “What type of research is this?” When the answer is a personal belief, unburdened by factual evidence, then we can accept it as a by-product of the constitutional guarantee of free speech and exclude it from the realm of evidence-based claims, no matter how sincerely held the belief may be. We cannot debate beliefs any more than we can debate sincerely held religious views. Rather, when encountering strongly held personal beliefs not based on evidence, one should respond respectfully, “Thank you for sharing your views. I want to assure you that I respect your beliefs and your right to hold them. I hope that you’ll also respect other beliefs that are different from your own. Please don’t confuse personal beliefs with evidence.”
Level 2: Personal Experiences
The first level, personal beliefs, is based on sincerely held views that are sometimes supported by examples. Some teachers, administrators, and parents—and indeed some students—believe that grading as punishment is effective, even though a century of evidence undermines the veracity of that belief. The second level, personal experience, extends beyond belief statements and relies instead on what many educators regard as the most compelling evidence of all—their individual encounters with students. Beliefs do not spring from a void; they are often based on one’s own learning and experiences. Many people who did well in school attend college and become educators or administrators. They formulate their beliefs about grading and discipline policies based on a system that is clearly effective for them. “It worked for me!” the opponents of grading and discipline reform will say. They are, of course, correct. Grading systems practices that are, according to the evidence, damaging (such as grading as punishment, the use of zeros on the hundred-point scale, and the use of the average; Reeves, 2011a) do, in fact, motivate some students to higher levels of performance. In my conversations with educators and administrators, I have asked, “For whom were these grading practices effective?” The most frequent answers are, “It worked for me!” and “It worked for my own children!” This is not surprising, coming from the viewpoint of college-educated professionals who are now public school educators. They loved school. They were good at school. They figured out the grading and discipline systems and responded well to rewards and punishments based on grades. The question is, What percentage of today’s students do we expect to become public school educators? If the answer is less than 100 percent, then perhaps we should reconsider attributing to all students the same motivational scheme—grades as rewards and punishments—that were effective for many teachers. Personal anecdotal evidence may be powerful when forming our belief systems, but we must be careful about generalizing our personal experiences to all students. It is preferable to adopt an attitude of “There is always something to learn,” and, indeed, I have seen teachers who, even in the final months of their careers, continue to improve their practices, learn from research, and change their techniques in order to better serve the cause of student learning.
Level 3: Collective Experience
Irving L. Janis (1982) coined the term groupthink to describe the tendency of people to acquiesce to the group and submerge their own better judgment. Sometimes the consequences are trivial, such as in psychological experiments in which subjects are shown two lines, clearly of different lengths, but other observers (confederates of the researchers) claim the lines are of equal length. If there is only one other person in the room, the research subjects hold their own, maintaining the lines are clearly different lengths. But as more people in the room claim the lines are of equal length, the research subjects cave to the inaccurate observations of the group (Heath & Heath, 2013). The consequences are less benign when the conversation participants are debating issues of greater importance.
Although an individual teacher may seek to change his or her practice—perhaps by calling on students randomly or by simply rearranging desks—professional isolation can be terribly lonely. Group experience provides the illusion of certainty, of unanimity, of proof: “It’s not just me who believes this—it’s the entire mathematics department! It’s the entire third-grade team! Our faculty voted, and we are unanimous.” Thus, popularity supplants reason. It is critical to remember, however, that even multiple instances of personal experiences or beliefs are still just personal experiences or beliefs. They remain anecdotal, and as such, are fallible in the absence of more credible research.
Level 4: Systematic Comparisons
Systematic comparisons produce some of the most effective and persuasive research that can directly influence professional practices. Consider the example of a teacher who makes an alteration to her teaching practice in the second semester. She can subsequently make three comparisons to form a very strong basis for research conclusions. The first is a comparison of the same group of students before and after the intervention. For example, the teacher can compare student performance in the first semester, when there was no change in professional practices, with student performance in the second semester, when the teacher altered a particular aspect of her practice. Perhaps it was a change in when students do homework—from at-home practice to in-school practice. Perhaps it was a change in the way the students revise and respond to teacher feedback. Perhaps it was the use of student self-assessment before submitting work to the teacher. The beauty of a systematic comparison is all other variables are held constant; the only change is the change in teacher professional practices. The students are the same, as are the schedule, curriculum, assessment, and teacher. If there is a change in results, it is very likely due to the change in teacher practice. In the second case, the teacher might make year-to-year comparisons, comparing the attendance, behavior, or academic performance in the first semester of one year to the first
