about intervention effects. For example, suppose we want to learn what, if any, types of interventions may be effective in preventing risky sexual behavior among high school students. Suppose we know that in some places the students receive sex education programs that emphasize abstinence only, while in other places the emphasis is on safe-sex practices. Suppose we also know that some settings provide faith-based programs, others provide secular programs, and still others provide no sex education. We could conduct a large-scale survey with many students in many different schools and towns, asking them about the type of sex education they have received and about the extent to which they engage in safe and unsafe sex. If we find that students who received the safe-sex approach to sex education are much less likely to engage in unsafe sex than the students who received the abstinence-only approach, that would provide preliminary evidence as to the superior effectiveness of the safe-sex approach.
Correlational studies typically also analyze data on a variety of other experiences and background characteristics and then use multivariate statistical procedures to see if differences in the variable of interest hold up when those other experiences and characteristics are held constant. In the sex education example, we might find that the real explanation for the differences in unsafe-sex practices is the students' socioeconomic status or religion. Perhaps students who come from more affluent families are both more likely to have received the safe-sex approach as well as less likely to engage in unsafe sex. In that case, if we hold socioeconomic status constant using multivariate statistical procedures, we might find no difference in unsafe-sex practices among students at a particular socioeconomic level regardless of what type of sex education they received.
Suppose we had found that students who received the abstinence-only sex education approach, or a faith-based approach, were much less likely to engage in unsafe sex. Had we held religion constant in our analysis, we might have found that students of a certain religion or those who are more religious are both more likely to have received the abstinence-only or faith-based approach and less likely to engage in unsafe sex. By holding religion or religiosity constant, we might have found no difference in unsafe-sex practices among students who did and did not receive the abstinence-only or a faith-based approach.
Although correlational studies are lower on the hierarchy than experiments and quasi-experiments (some might place them on a par with or slightly above or slightly below single-case experiments on an effectiveness research hierarchy – there is not complete agreement on the exact order of hierarchies), they derive value from studying larger samples of people under real-world conditions. Their main drawback is that correlation, alone, does not imply causality. As illustrated in the sex education example, some extraneous variable – other than the intervention variable of interest – might explain away a correlation between type of intervention and a desired outcome. All other methodological things – such as quality of measurement – being equal, studies that control statistically for many extraneous variables that seem particularly likely to provide alternative explanations for correlations between type of intervention and outcome provide better evidence about possible intervention effects than studies that control for few or no such variables.
However, no matter how many extraneous variables are controlled for, there is always the chance of missing the one that really matters. Another limitation of correlational studies is the issue of time order. Suppose we find in a survey that the more contact youths have had with a volunteer mentor from a Big Brother/Big Sister program, the fewer antisocial behaviors they have engaged in. Conceivably, the differences in antisocial behaviors might explain differences in contact with mentors, instead of the other way around. That is, perhaps the less antisocial youths are to begin with, the more likely they are to spend time with a mentor, and the more motivated the mentor will be to spend time with them.
Thus, our ability to draw causal inferences about intervention effects depends on not just correlation, but also on time order and on eliminating alternative plausible explanations for differences in outcome. When experiments randomly assign an adequate number of participants to different treatment conditions, we can assume that the groups will be comparable in terms of plausible alternative explanations. Random assignment also lets us assume that the groups are comparable in terms of pretreatment differences in outcome variables. Moreover, most experiments administer pretests to handle possible pretreatment differences. This explains why experiments using random assignment rank higher on the hierarchy for assessing intervention effectiveness than do correlational studies.
At the bottom of the hierarchy are the following types of studies:
Anecdotal case reports
Pretest-posttest studies without control groups
Qualitative descriptions of client experiences during or after treatment
Surveys of clients asking what they think helped them
Surveys of practitioners asking what they think is effective
Residing at the bottom of the hierarchy does not mean that these studies have no evidentiary value regarding the effectiveness of interventions. Each of these types of studies can have significant value. Although none of them meet the three criteria for inferring causality (i.e., establishing correlation and time order while eliminating plausible alternative explanations), they each offer some useful preliminary evidence that can inform practice decisions when higher levels of evidence are not available for a particular type of problem or practice context. Moreover, each can generate hypotheses about interventions that can then be tested in studies providing more control for alternative explanations.
Table 3.1 is an example of a research hierarchy representing the various types of studies and their levels on the evidentiary hierarchy for answering EIP questions about effectiveness and prevention. Effectiveness evidence hierarchies are the most commonly described hierarchies in research, but we could create an analogous list for each of the different types of EIP questions.
TABLE 3.1 Evidentiary Hierarchy for EIP Questions about Effectiveness
| Level | Type of study |
|---|---|
| 1 | Systematic reviews and meta-analyses |
| 2 | Multisite replications of randomized experiments |
| 3 | Randomized experiment |
| 4 | Quasi-experiments |
| 5 | Single-case experiments |
| 6 7 | Correlational studies Pretest/posttest studies without control groups |
| 8 | Other:Anecdotal case reportsQualitative descriptions of client experiences during or after treatmentSurveys of clients about what they think helped themSurveys of practitioners about what they think is effective |
Note: Best evidence at Level 1.
Notice that we have not yet discussed the types of studies residing in the top two levels of that table. You might also notice that Level 3 contains the single term randomized experiment. What distinguishes that level from the top two levels is the issue of replication. We can have more confidence about the results of an experiment if its results are replicated in other experiments conducted by other investigators at other sites. Thus, a single randomized experiment is below multisite replications of randomized experiments on the hierarchy. This hierarchy assumes that each type of
