Validation of the evaluation capacity in organizations questionnaire.

The purpose of this study was to test the construct validity of the Evaluation Capacity in Organizations Questionnaire (ECOQ). Conceptually, the ECOQ examines the role of evaluation in organizational development and, most notably in organizational learning. In this model, evaluation capacity building (ECB) initiatives are assumed to contribute to the development of a culture of systematic self-assessment and reflection, which, in turn, leads to increased organizational learning. Our sample consisted of internal evaluators within the federal, provincial or municipal government, not-for-profit organizations, private firms, and colleges or universities in Canada. Exploratory factor analysis (EFA) and latent path analysis (LPA) were conducted to better understand the underlying structural aspect of the organizational capacity to do and use evaluation construct as measured by the ECOQ. The results of our study indicate that the ECOQ effectively assesses an organization's capacity to do and use evaluation. Furthermore, evidence provided by the LPA statistical analysis suggests that an organization's capacity to learn is enhanced by the relationships among the various factors. Implications of using a validated model of an organization's capacity to do and use evaluations in both research and practice are discussed.

On the evaluation of social innovations and social enterprises: Recognizing and integrating two solitudes in the empirical knowledge base.

Social innovation (SI) is billed as a new way to address complex social problems. Interest in SI has intensified rapidly in the last decade, making it an important area of practice for evaluators, but a difficult one to navigate. Learning from developments in SI and evaluation approaches applied in SI contexts is challenging because of 'fuzzy' concepts and silos of activity and knowledge within SI communities. This study presents findings from a systematic review and integration of 41 empirical studies on evaluation in SI contexts. We identify two isolated conversations: one about 'social enterprises' (SEs) and the other about non-SE 'social innovations'. These conversations diverge in key areas, including engagement with evaluation scholarship, and in the reported purposes, approaches and use of evaluation. We identified striking differences with respect to degree of interest in collaborative approaches and facilitation of evaluation use. The findings speak to trends and debates in our field, for example how evaluation might reconcile divergent information needs in multilevel, cross-sectoral collaborations and respond to fluidity and change in innovative settings. Implications for practitioners and commissioners of evaluation include how evaluation is used in different contexts and the voice of evaluators (and the evaluation profession) in these conversations.

How Do Evaluators Differentiate Successful From Less-Than-Successful Experiences With Collaborative Approaches to Evaluation?

OBJECTIVES: In this exploratory study, we wanted to know how evaluators differentiate collaborative approaches to evaluation (CAE) perceived to be successful from those perceived to be less-than-successful. METHOD: In an online questionnaire survey, we obtained 320 responses from evaluators who practice CAE (i.e., evaluations on which program stakeholders coproduce evaluation knowledge). Respondents identified two specific CAE projects from their own experience-one they believed to be "highly successful" and another they considered "far less successful than [they] had hoped."-and offered their comments and reflections about them. They rated the respective evaluations on 5-point opinion and frequency scales about (i) antecedent stakeholder perspectives, (ii) the purposes and justifications for collaborative inquiry, and (iii) the form such inquiry takes. FINDINGS: The results showed that successful evaluations, relative to their less-than-successful counterparts, tended to reflect higher levels of agreement among stakeholders about the focal program; higher intentionality estimates of evaluation justification and espoused purposes; and wider ranges and deeper levels of stakeholder participation. No differences were found for control of technical decision-making, and evaluators tended to lead evaluation decision making, regardless of success condition. DISCUSSION: The results are discussed in terms of implications for ongoing research on CAE.

Government and voluntary sector differences in organizational capacity to do and use evaluation.

Research on evaluation capacity is limited although a recent survey article on integrating evaluation into the organizational culture (Cousins, Goh, Clark, & Lee, 2004) revealed that interest in the topic is increasing. While knowledge about building the capacity to do evaluation has developed considerably, less is understood about building the organizational capacity to use evaluation. This article reports on the results of a pan-Canadian survey of evaluators working in organizations (internal evaluators or organization members with evaluation responsibility) conducted in 2007. Reliability across all constructs was high. Responses from government evaluators (N=160) were compared to responses from evaluators who work in the voluntary sector (N=89). The former were found to self-identify more highly as 'evaluators' (specialists) whereas the latter tended to identify as 'managers' (non-specialists). As a result, government evaluators had significantly higher self-reported levels of evaluation knowledge (both theory and practice); and they spent more time performing evaluation functions. However, irrespective of role, voluntary sector respondents rated their organizations more favorably than did their government sector counterparts with respect to the antecedents or conditions supporting evaluation capacity, and the capacity to use evaluation. Results are discussed in terms of their implications for evaluation practice and ongoing research.

Making program evaluation activities family-centered: a qualitative study.

PURPOSE: This study explored ways in which program evaluation activities in pediatric rehabilitation settings can become congruent with family-centered service (FCS) philosophy. METHODS: Two Canadian pediatric rehabilitation centers participated in this study, which included focus groups with staff members and interviews with parents. RESULTS: Participants identified seven ways in which program evaluation practices could be made congruent with FCS. Suggestions included: (a) the inclusion of a diverse group of program recipients, (b) the use of processes that facilitate family involvement, (c) the recruitment of family champions, (d) the involvement of families in program development, (e) the establishment of evaluations that are relevant to families (f) the development of center-wide statements about family involvement in evaluation and (g) the compensation of families for their active participation in evaluation. CONCLUSION: For program evaluation practices to be useful and relevant, they should be improved and made consistent with FCS philosophy. Those evaluating pediatric rehabilitation programs need to use approaches and activities that respect the needs, characteristics, cultures and diversity of the program recipients. Such actions will help to improve the quality of care provided, the nature of program evaluation activities, as well as the overall level of FCS in pediatric rehabilitation settings.

A survey of program evaluation practices in family-centered pediatric rehabilitation settings.

Program evaluation is becoming increasingly important in pediatric rehabilitation settings that adhere to the family-centered service (FCS) philosophy. However, researchers know little about the specific evaluation activities occurring in these settings or the extent to which evaluators/service providers uphold FCS in their program evaluation activities. Through a questionnaire survey, this study aimed to document evaluators/service providers' perceptions of the level of program evaluation occurring in their Canadian pediatric rehabilitation centers. It also investigated the extent to which evaluators/service providers perceive program evaluation practices at their centers to be consistent with the FCS context of Canadian pediatric rehabilitation settings. The findings suggested that the amount of evaluation activities occurring within the respondents' centers is variable; that the majority of individuals working in program evaluation do not have formal training in it; and that the respondents' centers have limited resources for evaluation. The study also showed that staff members believe their centers' evaluation activities are somewhat consistent with FCS philosophy, but that improvements are needed.

When does a conceptual framework become a theory? Reflections from an accidental theorist.

In this short paper I reflect on the application of a program logic modeling approach to visualizing evaluation theory as it relates to practical participatory evaluation (PPE). The work of Hansen, Alkin and associates presented in this volume is well-thought out, carefully done, rigorous, and important. I found that their application of the approach resonated well with my understanding of PPE with a few notable exceptions. Specifically, the essence of the partnership aspect of the approach was underemphasized as was stakeholders' role in augmenting the credibility of evaluation findings and PPEs contribution to conceptual use. I then reminisce about the development and evolution of the participatory evaluation framework that we have been working with for so many years. I raise the point that we think of it as a theoretical framework to bound inquiry, not as an evaluation theory per se, which raises an interesting question: when does a theoretical framework become a theory?

The organizational learning survey: a re-evaluation of unidimensionality.

An earlier study by Chakrabarty and Rogé evaluated the dimensionality of the Organizational Learning Survey for assessing the learning capability of organizations developed by Goh and Richards. In this paper, the survey was re-evaluated for unidimensionality using both exploratory and confirmatory factor analysis. Retaining 18 of the original 21 items was sufficient to establish unidimensionality. A number of different confirmatory models validated the unidimensional measurement model on two independent data sets. Implications for further empirical research using this inventory are discussed.