are important and valuable because they help actors to understand not only which practices or behaviors are likely to lead to desirable consequences but also to help them avoid practices that could be in some sense detrimental or counterproductive. Therefore, with a set of principles, it must be possible for actors to subscribe to or follow all of the principles in the set and in doing so avoid the potential for contradictory processes or actions that could be counterproductive. Effectiveness principles generally derive from a careful examination of, or reflection on, experience with effective practice. They could be attributed to the wisdom of an expert practitioner or the result of serious processes of consultation, dialogue, and deliberation. They may also be grounded in empirical evidence, which is the process that we selected for the development of CAE principles.
But how can we assess the quality of effectiveness principles? Patton (2017), in his book Principles-Focused Evaluation,7 has given some serious treatment to this question. Ultimately, he came up with a set of criteria useful for this purpose. This set of criteria is captured in the acronym GUIDE and is described in Box 3.
7 In Principles-Focused Evaluation, Patton does not describe effectiveness principles for evaluation per se, rather he elaborates on using principles of practice within the substantive domain of the target intervention to inform the evaluation of the intervention.
Box 3: GUIDE Criteria for Evaluating Effectiveness Principles
Guidance: The principle is prescriptive. It provides advice and guidance on what to do, how to think, what to value, and how to act to be effective. It offers direction. The wording is imperative: The guidance is sufficiently distinct that it can be distinguished from contrary or alternative guidance.
Useful: A high quality principle is useful in informing choices and decisions. Its utility resides in being actionable, interpretable, feasible, and pointing the way toward desired results for any relevant situation.
Inspiring: Principles are values-based, incorporating and expressing ethical premises, which is what makes them meaningful. They articulate what matters, both in how to proceed and the desired result. That should be inspirational.
Developmental: The developmental nature of a high-quality principle refers to its adaptability and applicability to diverse contexts and over time. A principle is thus both context sensitive and adaptable to real-world dynamics, providing a way to navigate the turbulence of complexity and uncertainty. In being applicable over time, it is enduring (not time-bound), in support of ongoing development and adaptation in an ever-changing world.
Evaluable: A high quality principle must be evaluable. This means it is possible to document and judge whether it is actually being followed, and document and judge what results from following the principle. In essence, it is possible to determine if following the principle takes you where you want to go.
Source: Patton, M. Q. (2017). Principles-focused evaluation: The GUIDE. New York, NY: Guildford.
Warrants for CAE Principles
As mentioned above, five years ago, we published an article titled “Arguments for a Common Set of Principles for Collaborative Inquiry in Evaluation” (Cousins et al., 2013). In that paper, we identified three interrelated warrants for developing a set of principles to guide CAE practice. First, there is a growing corpus of CAE family members suggesting their appeal as valued evaluation options is substantial. The list appearing in Box 1 is incomplete. CAE is on the rise in a range of evaluation contexts including international development evaluation, cross-cultural evaluation, and DE contexts. In response to mainstream privileging of the statistical counterfactual as the gold standard for impact evaluation, there is growing concern for the development of alternative approaches, many of which could be considered CAE (e.g., Rugh, Steinke, Cousins, & Bamberger, 2009). In North America, CAE is the most commonly used approach for the evaluation of interventions with indigenous peoples (Chouinard & Cousins, 2007; Hoare, Levy, & Robinson, 1993). In the evaluation of social innovation, DE is most commonly used (Milley, Szijarto, Svensson, & Cousins, 2018); as health care innovations such as patient engagement develop, CAE becomes a much better fit than traditional approaches to evaluation in this sector (Gilbert & Cousins, 2017). All of these approaches share a common theme: evaluators work in partnership with members of the program community to produce evaluative knowledge. As such, it is both reasonable and desirable to develop a set of effectiveness principles to guide CAE practice.
A second warrant relates to a recent development in the field, specifically, that Fetterman and colleagues (2018) have not only framed collaborative, participatory, and empowerment approaches as being comprehensive, but they have taken it upon themselves to nuance the specific dimensions distinguishing these three approaches. They concluded that control of evaluation decision-making (one of the dimensions of process in Figure 1) is the essential dimension along which the three approaches can be differentiated.
Collaborative evaluators are in charge of the evaluation, but they create an ongoing engagement between evaluators and stakeholders…. Participatory evaluators jointly share control of the evaluation…. Empowerment evaluators view program staff members, program participants, and community members as the ones in control of the evaluation. (Fetterman, Rodriguez-Campos, Zukoski, & Contributors, 2018, p. 2, emphasis in the original)
The authors cited a long list of colleagues whom they argued recommend that stakeholder involvement evaluation approaches be differentiated. Yet, we observe that some of these publications provided critiques of only empowerment evaluation and suggested it to be, in practice, indistinguishable from other CAE approaches (e.g., Miller & Campbell, 2006; Patton, 2005); that is to say, they did not explicitly advocate differentiating among collaborative, participatory, and empowerment evaluation. Our main concern with this line of reasoning is that it runs the risk of evaluators self-identifying with particular approaches in seeking to apply them wherever they seem appropriate. In the foregoing excerpt, for example, we see reference to collaborative evaluators, participatory evaluators, and empowerment evaluators. From our perspective, decisions about i) whether CAE is warranted in the first place, ii) what will be its purposes, and iii) what will it look like in practice, will depend on the context within which the program is operating and the information needs that present. Perhaps this is why Miller and Campbell (2006) discovered that a wide range of alleged empowerment evaluations in their sample did not align well with theoretical tenets of the approach and instead resembled other CAE family members. While there may be some value in compartmentalizing different members of the CAE family, we remain somewhat opposed to this direction on the grounds that (i) it runs the risk of privileging method/approach over context, and (ii) it is exclusive of a plethora of related collaborative approaches (Cousins et al., 2013; Cousins, Whitmore, & Shulha, 2014).
The indispensable role of context in shaping evaluation approaches is, in fact, a third warrant for principles to guide CAE practice. In our view, a thorough analysis of the social, historical, economic, and cultural context within which focal programs operate, as well as the impetus for evaluating the program in the first place, are critical considerations for deciding i) whether a collaborative approach would be an appropriate alternative, ii) and if so, what will be its purposes, and iii) what form it should take (see Figure 2). Recent work by colleagues such as Alkin, Vo, and Hansen (2013) and Harnar (2012) to develop visual representations of theory has great value in our view. By representing theories in this way, readers are provided with an accessible overview of a given theory on which to build their deeper understandings. They may also use such representations to draw comparisons among given evaluation theories. This work has great potential to help bridge the gap between theory and practice in the evaluation community. However, despite this inherent value, we remain somewhat skeptical
