for the application of CAE principles exist. Here we comment on six main applications; no doubt others exist. These are prospective planning, framing, and doing; retrospective analysis and critique; designing and delivering education and training; reviewing and developing evaluation policy; translating and applying in cross-cultural contexts; and conceptual framing of RoE.
Prospective Planning, Framing, and Doing
Perhaps the most direct and obvious use of the principles would be to guide practice in a prospective manner. We would envision evaluators collaboratively working with members of the program community to engage in planning and implementing CAE on the basis of guidance from the principles. The indicator document provides some clues about actions to take and questions to ask to adhere to the tenets of the principles. We would expect that the actual collective exercise of familiarizing and internalizing would be at the same time instructive and inspiring of consideration of alternative courses of action. As a given CAE project unfolds over time, the principles could be used to stimulate ongoing reflection and dialogue, perhaps leading to alternative actions and/or decisions to reconsider strategies. Following implementation, it would be useful to retrospectively collectively reflect on the process and debrief about lessons learned in terms of both what went well and what challenges require attention. The suggestion is a natural segue into the next envisioned application.
Retrospective Analysis and Critique
To suggest that hindsight is 20:20 is a bit optimistic in our view given that collectively everyone exposed to a particular experience will, in fact, have experienced and remembered it differently. Another useful application of the principles, we believe, would be to use them as a guide to reflection after CAE projects have been completed. To the extent that such analyses can be systematic, it is our opinion that they will be more fruitful. For example, participants in a CAE project may wish to carefully identify and recruit other participating members for a structured dialogue about the process. Obviously, such a conversation would be based on individual and collective memories of what transpired, but such memories can be aided by artifacts and other clues. Nevertheless, it would be important not to delay the opportunity to engage in such reflection too long after a project has been completed since memories fade and people move on to other things. The primary benefit of such analyses would be to generate lessons learned that could inform future practice.
Designing and Delivering Education and Training
To date, we have delivered a range of full-day, half-day, and two-hour workshops and seminars using the CAE principles as a framework (see Appendix 1). One can easily imagine their use for structuring more protracted educational experiences such as a graduate-level evaluation course. Each of the eight principles could be used, for example, to structure a specific module, which could be augmented with introductory and integration modules. Given the requirement for evaluators to employ a range of interpersonal soft skills, in the ideal, a course on such approaches would involve some practical experience in the form of exercises and activities and/or authentic practice.
Our discussion to this point has focused on the professional development of evaluators, but of course, we should not overlook that of program community stakeholders. Another option for training could be seminars and workshops for program community members and organization members with an interest in applying evaluation in a self-directed way. Some of our colleagues (e.g., Alkin & Vo, 2018) have successfully provided highly accessible and readable texts on evaluation that have been quite useful in engaging persons who are, for all intents and purposes, uninitiated in evaluation matters. It is not difficult to imagine the development of support materials framed by the CAE principles that can serve the same purpose.
Reviewing and Developing Evaluation Policy
Hind Al Hudib, one of our COVE research team members, has been conducting in-depth research into interconnections between evaluation policy and evaluation capacity building (ECB) over the past few years. Her research includes the review and analysis of a large sample of organizational evaluation policies as well as an interview study with several contributors to the research knowledge base (Al Hudib, 2018). Suffice to say that the linkage between policy and ECB is not a strong one. In fact, organizational evaluation policies are generally seen to be symbolic and benign. However, from her research, there is some evidence to show that organizations are motivated to revise their evaluation policies to make them more engaging and useful. One of the chief concerns arising from this research is that evaluation policies appear to lean heavily toward supporting accountability-oriented approaches to evaluation. Yet, there is evidence to show that policies and practices that privilege learning as a central and desirable function of evaluation are more likely to connect with organization and program community personnel (Al Hudib, 2018). In our opinion, it would be entirely worthwhile to consider augmenting the content of evaluation policies with due treatment of learning-oriented approaches, CAE being exemplary in this respect. It is likely to be through the direct experience of success with the evaluation that organizational actors will become more willing to embrace evaluation as leverage for change.
Translating and Applying in Cross-cultural Contexts
Many international events have underscored the rapidly growing global interest in evaluation. One such series of events sponsored by EvalPartners was held in 2015 The International Year of Evaluation,9 intended to raise awareness and foster organizational and individual capacity building on a global scale. Much of the work of international development evaluation, as we have observed above, has been heavily weighted toward the interests and needs of bi- and multilateral donor agencies as well as public sector governance institutions. Yet, there is growing interest in the evaluation field building (e.g., Hay, 2010), which implicates the engagement of a much wider range of stakeholder interests in evaluation. We are inclined to think that the CAE principles could help to move this field building agenda. Required would be the official translation of the CAE principles into different languages of interest and applying them retrospectively or prospectively to evaluation projects at the local level. As we discuss below, we have already translated the principles and support documents into Spanish and French. Translation and application into other languages and contexts is most certainly possible and desirable. A caveat, however, is that the CAE principles reflect a western set of underlying assumptions and ways of thinking; translation into other languages is one thing, it would be quite another to actually apply the process in quite different cultural circumstances.
9 www.evalpartners.org/evalyear/international-year-of-evaluation-2015
In western culture, it seems we often equate development contexts with international development, but of course, many of the considerations and principles we have in mind apply to indigenous populations in our own jurisdictions. Such contexts provide yet another cross-cultural opportunity to work with and apply the principles. Regardless, whether international or local/indigenous, applications are not likely to be straightforward given variance and differences in cultural norms. It will be of high interest to see, for example, the extent to which the principles as we have laid them out integrate with indigenous and other ways of knowing.
Conceptually Framing Research on Evaluation
A while ago, an extensive review of 121 empirical studies on CAE was conducted, and it was observed that the vast majority of them took the form of reflective case narratives (Cousins & Chouinard, 2012). The review was extended to CAE in development contexts and similar results were found (Chouinard & Cousins, 2015). While reflective narratives offer considerable value
