visual representation. These two features in the diagram imply that while each principle contributes something unique, there is expected to be a degree of overlap among them. That is to say, they are not to be thought of as being mutually exclusive. Third, we make the claim that the principles are in no specific order although it may be argued that there is a loose temporal ordering beginning with clarify motivation for collaboration and ending with follow through to realize use. Important to note is that we intend for the CAE principles to require an iterative process, as opposed to a lockstep sequential one. Many of the principles described below require ongoing monitoring and adjustments to the evaluation and collaboration as time passes. For example, foster meaningful relationships requires continuous attention and may reassert itself as a priority during a clash of values or a change in stakeholder personnel. Finally, it might be noted that some of the principles laid out in Figure 3 might apply as equally to mainstream approaches to evaluation as they do to CAE. This may be true, but it is important to recognize that (i) these principles emerged from detailed data from evaluators practicing CAE, and (ii) each is somehow unique in its application to the collaborative context, as we elaborate below.
Figure 3 ■ Evidence-based CAE principles (adapted from Shulha et al., 2016).
We now turn to a brief description of each of the principles. Readers interested in a more detailed description and commentary may wish to consult Shulha et al. (2016). In the text to follow, supportive factors for each principle, which were derived from themes in our data, are identified in parentheses (following the title) and through the use of italics (in the descriptive text).
Clarify Motivation for Collaboration (evaluation purpose; evaluator and stakeholder expectations; information and process needs): Evaluators should be able to describe and justify why a CAE was selected in the first place. Why use CAE as opposed to a conventional or another alternative approach to evaluation? The principle encourages the development of a thorough understanding of the justification for the collaborative approach based on a systematic examination of the context within which the intervention is operating.
Clarity on these issues will help to ensure CAE is both called for and appropriate as a response to the evaluation challenge. Program improvement, opportunities for individual and organizational learning, and organizational capacity building were among the evaluation purposes suggested to be most conducive to CAE. On the other hand, accountability-oriented and legitimizing purposes could be counterproductive. Clarifying evaluator and stakeholder expectations for collaboration early on can be quite beneficial and can potentially lead to stakeholders leveraging networks and resources to help. CAE processes that are somehow mandated are less likely to be successful. Finally, clarification about information needs and priorities is an important supportive factor; evaluators can work with organizational or program stakeholders to help generate such clarity. Such activity helps to focus the evaluation and ensure that it will generate information that will be valued.
Foster Meaningful Relationships (respect, trust and transparency; structured and sustained interactivity; cultural competence): The principle inspires the conscious development of quality working relationships between evaluators and program stakeholders and among stakeholders, including open and frequent communication. Successful CAE projects benefit from “highly cooperative and collaborative organizational context, with abundant positive peer/professional relations and a wholesome, trusting, organizational climate” (study participant). Trust and respect are not givens and must be developed through ongoing interaction and transparency. While there is certainly a role for evaluators here, efforts on behalf of program and organizational stakeholders are implicated as well. Trust and respect can be leveraged through ongoing sustained interactive communication where evaluators learn to avoid “too many unspoken assumptions” (study participant). Close and constant contact can be instrumental to real-time communication, relationship building, and expectation clarification. The constructive exploration of differences and search for solutions that go beyond one’s own limited vision are at the crux of cultural competency. In CAE, building respectful sustainable relationships is essential.
Develop a Shared Understanding of the Program (program logic, organizational context): Is the program commonly understood among program and organizational community members and evaluators? Is everyone in agreement about intended program processes and outcomes? The principle promotes the explication of the program logic situated within context. Involving program stakeholders in the program description process is a useful way to deepen understanding about program logic. “The involvement of stakeholders provides a more accurate definition of the terms, problems, and population needs [and] culture” (study participant). Focusing on a mutual understanding of what is being evaluated can reduce the likelihood of stakeholders moving forward in the evaluation with unrealistic expectations. Organizational context is also a significant consideration in this regard. It is important for stakeholders to feel comfortable and confident in the capacity of the organization to embrace the process. Disruptive forces such as a change in administration can diminish this capacity. Evaluators need to monitor the organizational context as the project unfolds.
Promote Appropriate Participatory Processes (diversity of stakeholders; depth of participation; control of decision-making): What does it mean for stakeholders to participate in a CAE? The principle encourages deliberate reflection on the form that the collaborative process will take in practice with regard to specific roles and responsibilities for the range of stakeholders identified for participation. Collaboration in CAE can be operationalized in a contextually responsive way. It is important for evaluators to consider diversity in stakeholder participation, particularly with members or groups who might not otherwise have been involved. A challenge, however, is not just identifying such diversity but negotiating participation. The benefits of involvement to organization and program stakeholders and relatively deep levels of participation in the evaluation process can pay off rather significantly, as suggested by this survey respondent:
Participants were close to—and ultimately owned—the data. They helped design the tools, collect the data, analyze the data, interpret the data, and present findings. It wasn’t just buy-in to the processes and outcomes; it was implementing the process themselves (not being led through) and generating (not been given and asked for their thoughts about) and owning the outcomes.
An important consideration is control of decision-making about the evaluation, which may be difficult to manage. The evaluator being open to sharing the control of evaluation—in terms of instrument choice, data collection, and the interpretation of findings—is an important strategy. On the other hand, complications can easily arise around the control of decision-making, particularly when power issues among stakeholders are present.
Monitor and Respond to Resource Availability (budget, time, personnel): Issues of time and money are challenges for any evaluation but in CAE, important interconnections are associated with personnel. Participating stakeholders are a significant resource for CAE implementation. In addition to fiscal resources, the principle warrants serious attention to the extent to which stakeholder evaluation team members are unencumbered by competing demands from their regular professional roles. If the collaboration is identified as part of the job for
