information on all the resources used in the intervention. Contributed resources such as volunteers, donated equipment and services, and other “unpaid” inputs are not included in budgets.
Second, when resources have already been paid for or are included in some other agency’s budget, they will not be easily observed. For example, a building that is provided to a school district by some other unit of government or one that is fully paid for will not be found in the budget of a school district. Many education interventions receive ingredients from multiple levels. For example, Reading Partners, a reading program coordinated through a national office, operates through reading centers within schools and utilizes resources from AmeriCorps, the national community service organization, as well as relying on volunteers (Jacob, Armstrong, Bowden, & Pan, 2016). To accurately estimate costs, all resources used to implement Reading Partners should be calculated.
Third, the standard budget practices may distort the true costs of an ingredient. Typical public budgets and expenditure statements charge the cost of major repairs or facility reconstruction only to the year in which the cost was incurred. Thus, when the roof or heating system of a school is replaced, the expenditures are found in the budget for the year in which the repairs were financed—typically the year in which the reconstruction was done rather than being amortized properly. Yet, a new roof or heating system may have a 30-year life so that only about one-thirtieth of it should be charged to the cost of programs in any given year. Budgetary conventions would typically charge the costs of such capital investments to a single budgetary year, overstating the true costs for that year and understating the costs of operating the program for the 29 subsequent years.
Fourth, the costs of any particular intervention are often embedded in a budget or expenditure statement that covers a much larger unit of operation. Therefore, it may be difficult to isolate the unique costs of a new reading program in a school district budget, since the budget is not constructed according to the costs of particular interventions or activities. In fact, most educational budgets are “line item” classifications of expenditures according to functions and objects. Examples of functions include administration, instruction, and maintenance. Examples of objects include teachers, supplies, clericals, and administrators. It is not only difficult to tie such budget listings to particular activities or interventions but also often impossible even to ascertain what the costs are for a given school or broad instructional program such as a language program, since no such breakdowns are usually provided.
Finally, most budgetary documents represent plans for how resources will be allocated rather than classifying expenditures after they have taken place. This means that, at best, they refer to planned disbursements rather than actual ones. Accordingly, beyond all their other limitations for cost analysis, budgets may not provide precise figures for actual resource use. Actual statements of expenditures may be more accurate, but they are still subject to the shortcomings mentioned before.
Finally, budgets rarely help elucidate how a program is implemented—or at least not in a way that is related to the theory of change or the counterfactual. This point is illustrated in the cost analysis of Read 180 by Levin, Catlin, and Olson (2007). Using the ingredients method, the researchers calculated program costs at three selected sites where Read 180 was being delivered; they also calculated the program costs based on the developer’s recommended version of Read 180 from interviews and documents. Given that Read 180 is a reasonably formulaic reading intervention, we might anticipate that the actual costs at the three sites would be close to the recommended cost. The developer’s recommended version of Read 180 required changes in both personnel and materials. However, as implemented, some sites relied on adding more personnel than others while others yet varied in materials. The result was a significant difference in average costs for each site.
For these reasons, cost analysis cannot place primary reliance on budgetary or expenditure documents to ascertain the costs of interventions. Of course, these documents may still provide supplementary data that will be very useful. However, they cannot serve as a principal source for constructing cost estimates.
3.5. Motivation for Cost Analysis
Understanding the costs of an intervention is of course a prerequisite for performing CE and BC analyses. In itself, cost analysis is also valuable. It allows us to perform a cost study and a cost feasibility analysis (we provide examples of these in subsequent chapters). Even before that, however, cost analysis helps the analyst learn about the intervention in two key respects. These are the main motives for performing a cost analysis per se.
First, cost analysis is a way of describing an intervention. Many educational interventions are complex or multifaceted; by providing details on the resources, we are explaining what is required for the intervention. Information on costs can indicate the quality or intensity of a program. For example, the number of counselors per student may indicate the quality of counseling programs, as would data on the number of days of training required for counselors before the program begins operation. As well, the cost per student for a program will reflect the dosage of the treatment each student receives. For example, the annual cost of a mentoring program might be $500, but the cost per student will depend on how many years each student participates.
Information on costs sheds light on the types of resources needed to implement a given program. Some interventions may require a substantial commitment of in-kind resources; these include volunteer-intensive youth supports such as Big Brothers Big Sisters of America and elementary school literacy programs such as Time to Read (Grossman & Tierney, 1998; Miller & Connolly, 2012). Other interventions may require significant capital investments; for example, massive open online courses (MOOCs) may necessitate investment in an expensive technology infrastructure before any students have enrolled (Bowen, Chingos, Lack, & Nygren, 2014). For some interventions, there may be reorganizational costs that are not obvious. For example, implementation of Success for All requires cross-grade grouping—that is, putting students from different grades in the same classroom; this grouping may have implications for how the school day is organized (Quint, Zhu, Balu, Rappaport, & DeLaurentis, 2015). Finally, some programs may have substantial induced costs such that the biggest cost item is not the intervention itself but the extra resources that flow from implementing the program. A series of examples, showing how costs change as students progress through community college at different rates, is modeled by Belfield, Crosta, and Jenkins (2014).
Relatedly, cost analysis can be applied to check for fidelity of implementation of an intervention. For example, high-quality preschool programs are expected to have a trained lead teacher; the cost analyst can verify if sufficient funds were allocated to ensure that this is affordable. Subject to the caveats just noted, the analyst can compare actual resource use with an ex ante budget to see whether the program resources match the intended design.
The second intrinsic value of cost analysis is that it helps the researcher with prediction and modeling. It is important to recognize that costs are estimated, just as effects are in an impact evaluation. The researcher is estimating an expectation based on the sample units of the intervention. In some cases, the analyst will want to estimate the actual costs of the program as it was delivered in a particular setting for a specific group of students. It is more likely that the analyst will prefer to estimate expected costs—that is, the best prediction of what the program will cost if implemented again. These two estimates will be different: For expected costs, the analyst will use actual ingredients but general prices; for actual costs, the analyst will apply the actual (or as close as possible) prices each site paid for a particular ingredient. This latter approach is similar to a budget approach and so must be applied cautiously.
The use of expected costs allows the analyst to more accurately model costs. One concern for education research is how to take a successful small or pilot intervention and implement it at scale with the same level of success (Quint, Bloom, Black, Stephens, & Akey, 2005). For instance, the policymaker might mandate a class size reduction policy such that all classes must be 10% smaller. What resources would
