the audience and perspective for the evaluation.3 Select the appropriate type of economic analysis and research design in relation to the theory of change.
4 Establish the need for chosen economic evaluation.
Before beginning an economic evaluation, it is important to establish the analytical framework that will be utilized. This framework consists of identifying the nature of the problem and the theory of change, clarifying the specific alternatives that should be considered in the analysis, establishing the identity of the audiences—and the perspective—for the analysis, and selecting the type of cost analysis to use. These tasks are all preparatory to the actual research inquiry, data collection, and analysis, which are addressed in subsequent chapters. At the initial stage, it is important to consider the extent and scope of the inquiry to ensure that the economic evaluation is appropriate and justifiable.
In this chapter, we discuss these issues in turn. Although they might seem outside the scope of a direct economic evaluation, establishing the appropriate structure for the research inquiry is critical. It is going to be very difficult to rederive new estimates of costs and benefits if the research questions change. Importantly, the research findings will depend on how the research is framed. Hence, we describe key issues in establishing the analytic framework for economic evaluation.
2.1. Identifying the Problem
One of the critical, but sometimes neglected, areas of evaluation design is proper identification of the problem. By proper identification, we mean that the nature and scope of the problem should be posed to justify the intervention and to determine which alternatives for reform appropriately address the issue. To accomplish this, aspects of the problem must be documented to understand who is affected and how (for an overview of evaluation methods and practice, see Rossi, Lipsey, & Freeman, 2004).
For example, it is widely accepted as important that children acquire reading skills progressively through school based on age or grade level. If it is found that children are not reading at the appropriate level, it is important to know how many children are behind in reading, at the age or grade level, how severe the gap is between the skills children have and the skills needed to be on target, and where the children are located who need the most assistance. Based on this information, one should ascertain potential reasons to explain the deficiency. The origins may be community or family based and related to insufficient skills upon school entry, reading deficiencies among the population, or lack of books for children outside of school. Another possibility is that the teaching strategies or materials may not be conducive to the variety of skill in the classroom consistent with up-to-date research on pedagogy for teaching reading. A third possibility could be related to attendance. A fourth could be that the children with lagging reading skills are English-language e-learners. Each of these causes would require a different intervention to sufficiently impact the outcome of interest.
We can also review another example where the problem is one at the system level rather than with student skill development. As noted in Chapter 1, the question that is often posed by school districts facing financial exigencies because of declining enrollments is this: Which school or schools do we close? The real problem that must be faced, however, is how to address a shrinkage in resources that does the least damage to the education system. The alternatives to consider include the possibilities of school closure, but they also include the options of reducing personnel, cutting specific offerings, increasing class size, leasing excess space in existing schools, and taking a variety of other potential routes to cutting the budget or raising school revenues. Narrowing the question to which schools are to be closed is to rule out options that may be more appropriate when economic criteria are used. That is, there can obviously be no economic analysis among alternatives that are not considered.
To select the most efficient program or intervention, all relevant alternatives should be identified and considered. The alternatives for addressing particular problems are those potential interventions that might respond to the problem and improve the situation being addressed. It is important to ask whether all the pertinent alternatives have been placed on the agenda for consideration. Obviously, the classes of alternatives that ought to be considered are those that are most responsive to the problem. Again, this will require a sensitive search for ways of meeting the challenge that has been posed. Although one may wish to draw upon traditional responses as well as those that other entities have used in facing similar problematic situations, one should not be limited to these. In fact, sometimes they may not be the most responsive approaches.
In conducting an economic evaluation of alternative interventions, one must ask not only whether the alternatives that will be considered are responsive to the problem but also whether the “responsive” ones have been included. Often, both administrators and evaluators will rule out alternatives before analyzing them when such options are politically sensitive. That is, the pragmatic aspects of daily life suggest that one avoid pitched political battles by keeping politically sensitive issues off the agenda if at all possible. While one can appreciate the pressures on both evaluators and administrators in this regard, there are two reasons that all of the relevant alternatives should be analyzed.
First, it is a matter of professional integrity to provide information on all of the pertinent alternatives, while letting the decisionmaking and political processes eventually determine the choice among them (Ross, Barkaoui, & Scott, 2007). If those processes are not adequately informed about possible responses, they can never consider the costs and impacts of many of the pertinent alternatives. There is an appropriate place for analysis and one for decisionmaking. If certain alternatives are precluded from consideration by their political sensitivity, then the political and decisionmaking processes have taken place before the information and analyses have been derived. Clearly, the two stages are interrelated, but good decisionmaking should be based upon informed choices rather than ones that eliminate potential options before they are ever analyzed and considered.
A second reason for considering even those alternatives that are politically sensitive is that such sensitivity or opposition may be dependent upon circumstances. That is, while some alternatives are indeed “untouchable” in the normal course of events, they may become salient for consideration under more dire circumstances. If a school district is facing serious budgetary problems, it must consider all possibilities that would reduce the budget. If student proficiencies in certain academic areas are woefully inadequate, then a wide range of programs for improvement begin to enter the realm of consideration. It is important to consider the strengths and weaknesses or the costs and effects of selecting from all of the pertinent alternatives. It should also be borne in mind that the retention of existing practices is always an alternative.
Indeed, this leads to a final comment on alternatives. Cost analysis is premised on the view that decisionmakers have choices. The objective is to make the best selection from competing alternatives. Economic evaluation is done in order to aid in the selection among alternatives. If there are no alternatives, there is no point in doing an analysis. That is, no matter how competent the evaluation, it will simply lack usefulness if one cannot do anything with what is learned. Equally importantly, the analyst should evaluate a given intervention against a reasonable alternative that might otherwise be implemented. The evaluation should not compare the intervention against “nothing” if there is a reasonable expectation that students would receive “something”; the intervention should be compared against the other “something.” (The analogy here is with drug trials where a new drug is compared against a placebo rather than a generic version of the drug.) In fact, in order to properly perform an incremental analysis, an economic evaluation should focus as much inquiry on the comparison group as it does on the treatment group.
Ultimately, the goal of an economic evaluation is to help policymakers decide on an alternative. It is to help them choose among alternatives. Of necessity,
