action so that targeted changes for improvement happen?
• How are actions and their impacts monitored?
• How can OED play a more strategic role? Specifically, how can the influence of independent evaluations on ADB’s long-term strategic thinking be increased?
• How can the DEC play a more strategic role and increase the value it adds to ADB’s learning process?
The specific steps that the DEC has taken to support OED include (i) suggesting topics of strategic interest for inclusion in OED’s work program, (ii) helping to get the timing right for policy studies, (iii) ensuring that country assistance program evaluations are discussed by the DEC before country partnership strategies are finalized, and (iv) initiating measures to strengthen ADB Management responses.
a Available: www.adb.org/bod/dec.asp
To ensure the independence and transparency of evaluation reports, a new approval, review, and disclosure process has been established. Since evaluation reports are final upon approval by the director general of OED, they are made publicly available immediately upon circulation to ADB’s Management and the Board. ADB’s Management is given a specific period in which to respond to lessons and recommendations. ADB Management responses and the DEC chair’s summaries of discussion are disclosed as they become available.9
The policy paper to enhance the independence and effectiveness of OED saw the need to formalize practices for avoiding conflicts of interest in evaluations. In March 2005, ADB became the first member of the Evaluation Cooperation Group (ECG)10 of the multilateral development banks to formally adopt guidelines11 to avoid conflict of interest in independent evaluations, which specify conditions under which OED management, staff, and consultants must recuse themselves.
OED has also strengthened the process used to formulate its work program. Previously, only annual work programs were prepared, but starting in 2005, a 3-year rolling framework was developed.12 Wide-ranging consultations took place for the preparation of the 2006–2008 work program. Discussions on institutional priorities for evaluations were held with the DEC and all vice-presidents. Inputs were also sought from ADB’s operations departments as in the past. The evaluation program has become more strategic and more integrated, with one evaluation feeding into another. For instance, evaluations of the effectiveness of ADB’s technical assistance (TA), microcredit operations, energy policy, governance and anticorruption policies, approaches to partnering and harmonization, policy-based lending, adoption of managing for development results, safeguard policies, and accountability mechanism were programmed to feed into reviews of these by ADB’s Management.
The influence of evaluations on ADB’s operations, policies, and strategies was assessed in the 2006 Annual Evaluation Review.13 Their influence was evident in (i) the sector priorities for ADB’s operations under the new model for selectivity and focus set out in the second medium-term strategy, (ii) ADB Management’s agreement to implement an action program to improve portfolio performance in response to the DEC’s recommendation following its discussion of the 2005 Annual Report on Loan and Technical Assistance Portfolio Performance,14 and (iii) changes in new country strategies and programs directly related to lessons and recommendations from previous country assistance program evaluations.15
Through its oversight of OED, the DEC is now helping to ensure that actions are taken on lessons and recommendations that it considers to be of high priority and that ADB’s Management has accepted. In its annual report to the Board in 2005, the DEC included a specific, candid assessment of the system for taking actions. The DEC considered that there was room for ADB to become a full-fledged learning organization using evaluation results more systematically. The DEC reported on actions taken by OED and by ADB’s Management on its recommendations:
| (i) | Traditionally, the intended end-users were consulted during and near the completion of an evaluation. Nowadays, OED reviews the scope and methodology with the main stakeholders, particularly ADB’s operations departments, before starting. |
| (ii) | OED has introduced the new step of discussing draft findings and recommendations from evaluation with ADB’s operations departments and, in the case of country assistance program evaluations, with the government before the evaluations are finalized. The objectives are to ensure that those who will be responsible for implementing the recommendations understand them, to find out which are acceptable and feasible, and to build early commitment. |
| (iii) | The actions that ADB’s Management commits to in its formal responses to evaluation reports have generally been more specific and time bound. |
To improve consistency and quality, guidelines for the evaluation of public sector projects, program loans, TA, and country assistance programs were issued in 2005. Guidelines for evaluating private sector operations were completed in 2007.16 The new modus operandi has brought institutional advantages, but it also raises the question of how OED itself should be evaluated. To address this issue, the ECG has begun to examine the feasibility of establishing a peer review process of the evaluation function in its members.
Evaluating Operations Performance
There is a long feedback loop from approval to completion of an operation, when development results can be assessed. It takes 7 years or more to implement projects or programs after Board approval and for completion and evaluation reports to be prepared. Thus, virtually all projects or programs that were approved in the mid-1990s were in various stages of implementation at the turn of the millennium. Indeed, many are still under implementation. The completion and evaluation reports for most of the projects and programs approved in the 1990s were prepared between 2000 and 2005. Analysis of project success was undertaken based on the year of approval. Figure 1 shows that the success rates of Asian Development Fund (ADF)-funded projects approved before the 1990s tended to be considerably lower than those funded by ordinary capital resources (OCR),17 and average ADF ratings were much more volatile. In the 1990s, however, the OCR and ADF ratings converged, and both improved. Given the development challenges faced by ADF-eligible countries, this is a positive finding. The long-term trend of the ratings suggests that the design and implementation of ADB-assisted projects have progressively improved, whether funded by OCR or by ADF. This indicates successful, gradual institutional learning from lessons of past operations. This improvement in project outcomes needs to be continuously monitored to ensure that the trend is sustainable.
Figure 1: Trends in Project Ratings by Year of Approval and Source of Fundinga (Combined Completion and Evaluation Results)
a Success rate for ADF and OCR based on 3-year moving average.
ADF = Asian Development Fund, OCR = ordinary capital resources.
Source: OED.
While project performance has varied significantly across countries, it has not varied significantly by country classification (i.e., degree of eligibility for ADF funding).18 The analysis by country classification in Table 1 demonstrates improvement from the 1970s to the 1990s across all country groupings, though patterns of progress differ. By the 1990s, the success rate of each group with access to ADF had gradually increased, and countries in Group A (with the exception of Nepal)
