suggested that if there are no alternatives or if a decision is already predetermined, the entire evaluative situation—including economic analysis—becomes moot. Further, if sufficient time or other resources are lacking or if CE types of data will not alter decisions, there is probably not a strong case for doing cost analysis. Even when all the prerequisites for implementing and using cost analysis are present, it is important that such an evaluation be worth doing in the first place.
Moreover, economic evaluation techniques might themselves be interrogated using efficiency criteria: What is the economic payoff to performing CE analysis? Do the benefits of performing BC analysis outweigh the costs? In response, we refer to Scriven’s (1974, pp. 85–93) notion of cost-free evaluation. In considering the choice of an analytic approach and supportive design for implementation, one must ask what will be gained by finding a better alternative. If the gains will be relatively small, only a small investment in evaluation would be merited. Indeed, if the potential gains of a good decision are minuscule, it is possible that no formal analytic study should be undertaken. That is, intuition and present knowledge should suffice, given that little will be gained from a more formal and extended evaluation. However, when the value of selecting a new alternative can be very great, it may be worth making a large investment in evaluation and analysis. Scriven would say that this situation meets the cost-free evaluation criterion if the probable gains from the evaluation will be in excess of the costs of the evaluation. It is cost-free in that more is saved by a good and appropriate evaluation than is expended in resources on that evaluation. Put simply, the more costly the intervention is to implement, the more likely it is that an economic evaluation will pay for itself by changing the allocation of resources. Prima facie, the economic case for economic evaluation appears reasonable: Society devotes a substantial amount of resources to education programs, and the information obtained from an economic evaluation is probably substantial and salient.
Nevertheless, in order to know how much economic evaluation to do, one needs a sense of what research effort is involved. We describe the research tasks in detail in the following chapters, so here we offer very general comments. First, it is important to recognize that costs are estimated, just in the same way that impacts are estimated. Cost estimation is therefore a substantial research activity, just as identification of impacts is. When estimating impacts, the researcher is immediately concerned with potential sampling error—that is, the likelihood that the sample chosen reflects the population. Sampling error is ameliorated by properly designing the sampling frame and drawing a sample of sufficient power. This logic also holds when estimating costs: The researcher should draw from the population of sites (e.g., schools or classrooms or students) to ensure that the sample reflects the population. A priori, it is difficult to say if the sample for estimating costs should be larger or smaller than the sample for estimating impacts; it depends on the variation of costs within the population. Overall, whatever research tasks are important for impact evaluations are likely to be required of economic evaluators.
Second, in most BC or CE evaluations, some expertise will be needed from persons who are specialists in these kinds of analyses. Although BC analysis is a standard component of the economics of public finance, many academic economists have not had practical training in either cost analysis or shadow pricing techniques. Also, one should not confuse expertise in accounting with expertise in BC or CE analysis. While BC and CE analysts must have a good understanding of cost accounting, cost accountants need have no understanding of BC and CE analysis. Often, the correct approaches for cost accounting for business firms are inappropriate for estimating the costs of social projects. In fact, one of the rationales of public economics is that, frequently, social costs and benefits differ considerably from private ones. Cost accountants are not usually trained to address social costs and benefits.
That said, economic evaluations do not just rely on economic principles. In order to perform CE analysis, one has to determine what effectiveness means in each educational context. For instance, for reading programs it might be comprehension or vocabulary or fluency; for dropout prevention programs it might be keeping students in school each year, getting them to finish 12 grades, or getting them to complete a high school diploma. The answers to these questions cannot be found in economic theory. Experts in the respective fields are needed to determine which specific outcomes are most valid.
Overall, there is no presumption that an economic evaluation is necessary nor is there a fixed amount of effort that an expert might contribute to perform such an evaluation. Resources would need to cover three discrete research tasks: (1) calculating costs, (2) identifying effects or estimating benefits, and (3) combining the two to report an economic metric. Potentially, such studies might require resources equivalent to those needed for an impact evaluation. The main guiding principle is the proportionality principle—that is, the idea that the amount of effort required should correspond to the value of the knowledge acquired and how that knowledge influences the results of the analysis. Suffice it to say, we hope to establish in this book that the knowledge gained from most economic evaluations is unlikely to be trivial and that more, rather than fewer, such evaluations are worthwhile.
2.5. Conclusions
Before undertaking an economic evaluation, it is important to understand the research question in detail. This understanding should start with the formal identification and understanding of the problem and the theory of change that each educational intervention adverts to. It should include explicit consideration and selection of alternatives to be evaluated; once these alternatives are set, the economic evaluation will be directed to determining which alternative is the most cost-effective or efficient. The analyst must recognize which audiences will use the evidence. Cognizant of this audience, the analyst must determine the perspective. We propose that each economic evaluation should begin with the presumption that a social perspective should be considered as the first priority even as other perspectives—fiscal, private, or per agency—are valid and important. Next, the analyst must choose the appropriate mode of analysis; the more detailed the investigation of the research question, the easier this choice will be. Finally, the analyst should perform an appraisal as to whether an economic evaluation is worthwhile.
This process requires a substantial research commitment: Each item needs to be decided in tandem, and these decisions will significantly shape the research inquiry. However, once these decisions are made, it is possible to proceed to the next stage of the analysis—how to undertake cost analysis.
Discussion Questions
1 A nonprofit agency wishes to introduce a mentoring program for at-risk youth and wants to perform BC analysis. What are some of the important issues to decide before beginning any data collection?
2 What types of assistance would you need in doing an economic evaluation, and what types of expert could contribute to the activity?
3 Describe potential audiences for different types of cost analyses. What would be the purpose of cost analysis for each audience? What type of analysis would be appropriate for each audience and why?
4 Federal policy in the United States constrains the use of BC analysis to only “economically significant” federal regulations. How should the term economically significant be defined?
Exercises
1 For the following interventions, describe how you would determine whether formal cost analysis is worthwhile.The education department in a particular state has mandated that school districts implement a specific math curriculum, including textbooks, teacher training, computer software, and other materials.A team of second-grade teachers approached their principal with a proposal to spend 10 minutes each morning performing mindfulness exercises with the students to help reduce stress, increase focus, and increase learning throughout the day.A community college is considering expanding its online education program to offer a wider range of courses and possibly even full degree programs.
