“big data” increasingly available, it is hoped that the creative examples identified in this review can help to inspire new impact evaluation studies that address evidence gaps on transport. This work is more relevant than ever, as transport investments increasingly seek to experiment with high level technologies, such as intelligent transport systems, and increasingly focus on behavior change interventions, such as transport demand management, pricing, driving restrictions, and road safety measures. Impact evaluation can play a critical role to ensure that these innovations can be tested to help transport operations improve over time. This review helps to show where future impact evaluation investments can best fill evidence gaps and the methods that may be applied to do so.
Yasuyuki Sawada
Chief Economist and Director General
Economic Research and Regional Cooperation Department
Asian Development Bank
Acknowledgments
This publication reflects the contributions of many individuals within and outside of the Asian Development Bank (ADB). It has been produced under the overall guidance of Edimon Ginting, Deputy Director General of the Economic Research and Regional Cooperation Department and Rana Hasan, Director of the Economic Analysis and Operational Support Division. The publication has also benefited from overall orientation by ADB’s interdepartmental Impact Evaluation Committee chaired by Bernard Woods, Director of the Results Management and Aid Effectiveness Division. David Raitzer, Economist, Economic Analysis and Operational Support Division; Nina Blöndal, independent consultant; and Jasmin Sibal, Economics Officer, Economic Analysis and Operational Support Division, authored the contents.
Sakiko Tanaka provided valuable insights and contributions to early versions of the report. Independent consultant Howard White also provided valuable suggestions. Additional inputs and contributions were provided by ADB consultants Daryll Naval, Denise Encarnacion, and Reneli Gloria. Administrative support has been provided by Lilibeth Poot, Jindra Samson, Amanda Mamon, Gee Ann Burac, Rica Calaluan, Roslyn Perez, and Glennie Amoranto.
Valuable peer reviews have been provided by Ari Kalliokoski, Shinichiro Nagao, Anders Pettersson, and Francesco Tornieri. Their input has helped to improve the content.
Copy editing was performed by Tuesday Soriano and layout was performed by Joe Mark Ganaban. This publication is produced under regional technical assistance 0012 for Developing Impact Evaluation Methodologies, Approaches and Capacities in Selected Developing Member Countries.
Key Messages
There are important untested assumptions within traditional economic models of transport. For example, ex ante economic analysis models often mechanistically project that road improvements lead to increased vehicular speed, more productive time use, and reduced transport costs. Ex post analyses are usually based on before–after comparisons that assume that all changes over time are due to the road projects and not other trends or factors. This means that assumptions about how the roads are used and how users behave are often not directly tested. Understanding these behavioral aspects is critical to making transport investments more effective.
Impact evaluation can bring new evidence to transport design choices. The approach can test basic assumptions about effects of transport on travel speeds, transport costs, time savings, employment, prices, productivity, and welfare. It can also test new ways of doing transport interventions better and shed light on the role of complementary interventions, such as contracting arrangements, maintenance schemes, road safety initiatives, and incentives to shift to more sustainable modes of transportation. It also can test transport policies and effects on pollution and other environmental outcomes.
Transport investments have been subject to a growing number of impact evaluations. Transport has been a major sector for official development assistance. Yet, the evidence on the causal effects of transport sector interventions is relatively limited, compared with the social sectors. This is particularly true for sustainable transport investments, which often seek to change transport modes from private vehicles to public transit, make traffic behavior more efficient, or reduce excess travel. At the same time, a number of new and innovative impact evaluations have emerged in recent years, and the body of studies is growing rapidly. This review identifies 91 transport impact evaluations in developing countries, of which more than 65% have been published after 2012.
Impact evaluations to date have found effects on a range of outcomes. These include effects on income, poverty, employment, education, health, gender disparities, land prices, firm productivity, migration, and much more. Both significant expected effects have been found, as well as unanticipated outcomes. A striking pattern is that effects are variable across studies and contexts, and that impacts are more heterogenous than transport programming often assumes. The evidence offered is limited to specific contexts, periods, and types of interventions, and much of the evidence on these outcomes pertains to rural roads. More studies are needed to better understand factors conditioning effects observed.
Impact evaluations have focused increasingly on transport policies, in addition to infrastructure. A body of recent work has emerged, which evaluates a broader set of interventions, such as vehicle usage restrictions, safety interventions, and toll pricing on outcomes including driving behavior, congestion, local air pollution, and health. This new generation of impact evaluation often utilizes “big data,” multiple data sources, and automated data collection to capture new variables in innovative ways.
Impact evaluation of transport has special challenges. The nature of transport is inherently challenging for impact evaluation. Not only are there problems of nonrandom placement and large facilities with few treated units, but there are also other unique problems. Transport conditions the effects of location, so it has inherently heterogenous impacts. It can also condition location and cause populations of interest to shift. At the same time, a number of methods show promise for transport sector applications, which have not been used frequently. These include regression discontinuity designs, encouragement designs, and in some contexts, synthetic controls.
Randomized evaluations can offer evidence on a broader set of transport interventions. Although large transport infrastructure often cannot be randomly assigned, encouragement designs can introduce random variation in the use of transport facilities. These designs have untapped potential to enable testing of an expanded array of assumptions and interventions, especially through pricing encouragements.
“Open data” can enable new impact evaluation possibilities. High frequency, geospatial, and “crowdsourced” data are increasingly available, and many of these data sets are relevant to transport. For example, remote sensing can now detect road network development and pollution, and ride-sharing applications can enable vehicle speed monitoring in real time. Many recent impact evaluations have started to use automatically collected data from traffic detection systems, air pollution monitors, public transit monitors, and even license plate recognition technology. These developments expand the number of impact evaluation possibilities.
There are important transport evidence gaps that future studies can address. Impact evaluation has given relatively little coverage to major areas of investment, such as urban and sustainable transportation, major transport corridors, and efficiency enhancing measures. Effects via access to public services and prices have also had little investigation. Even where evidence exists, it is often limited to a few countries and programs. With new methods and increasing openness of geospatial data, there is scope for generating more innovative impact studies.
