Doing better in theory-based evaluation, from A to Z.

This paper presents an alphabetical, select listing of concepts related to program theory-informed evaluation. Taken together, the concepts are relevant for understanding the fundamentals of program theory-based evaluation and especially for achieving a more beneficial future practice of such evaluations. The paper is offered with the hope of contributing to and helping stimulate further discussion of ways to improve theory-informed evaluation practice.

Surfacing, as well as testing, "elliptical assumptions" in a theory of change: Principled discovery.

While the testing of known assumptions in a theory of change is important, so too is the discovery or surfacing of previously unrecognized assumptions. This paper describes and illustrates the surfacing of "elliptical assumptions," which involve the unknown ingredients that are necessary for a program to be effective. Identifying the ingredients of program success is important for several reasons, which include (a) guiding the development of an improved theory of change, which in turn can guide program improvement, and (b) informing efforts to transfer the program to other settings and populations. However, when an observed pattern, such as differential program effects, points to the existence of a previously unidentified but important ingredient, this might be a just-so story, a seemingly compelling but inaccurate account. Accordingly, the testing of previously unidentified elliptical assumptions is recommended and illustrated.

Stakeholder Involvement in Evaluation: Does it Affect Observers' Perceptions of an Evaluation? And Which Stakeholder Group(s) Do People Think Should to Participate?

BACKGROUND: Stakeholders are often involved in evaluation, such as in the selection of specific research questions and the interpretation of results. Except for the topic of whether stakeholder involvement increases use, a paucity of research exists to guide practice regarding stakeholders. OBJECTIVES: We address two questions: (1) If a third-party observer knows stakeholders were involved in an evaluation, does that affect the perceived credibility, fairness, and relevance of the evaluation? (2) Among individuals with a possible stake in an evaluation, which stakeholder group(s) do they want to see participate; in particular, do they prefer that multiple stakeholder groups, rather than a single group, participate? RESEARCH DESIGN: Six studies are reported. All studies address the former question, while Studies 3 to 5 also focus on the latter question. To study effects of stakeholder involvement on third-party views, participants read summaries of ostensible evaluations, with stakeholder involvement noted or not. To examine a priori preferences among potential stakeholders, participants completed a survey about alternative stakeholder group involvement in an evaluation in which they would likely have an interest. RESULTS AND CONCLUSIONS: Across studies, effects of reported stakeholder participation on third-parties' views were not robust; however, small effects on perceived fairness sometimes, but not always, occurred after stakeholder involvement and its rationales had been made salient. All surveys showed a large preference for the involvement of multiple, rather than single stakeholder groups. We discuss implications for research and practice regarding stakeholder involvement, and for research on evaluation more generally.

Adding Adaptation/Modification Size and Scope to Frameworks for Classifying Changes to an Intervention.

Frameworks for classifying adaptations and modifications to an intervention have been developed and continue to evolve. These frameworks capture a range of attributes of adaptations and modifications. However, the size and the scope of a change, relative to the intervention's overall size and structure, are not currently included. This omission can have negative consequences, especially for conclusions about how different kinds of changes to an intervention may influence implementation quality and intervention outcomes. Coding the relative size and scope of an adaptation or modification should be simple when the original intervention is described in detail. For interventions that are not manualized, coding may alternatively be based on rater judgments. Attending to the size and the scope of adaptations and modifications could have an ancillary benefit, generating increased conceptual and empirical work on procedures for identifying the distinct components and sub-elements of an intervention.

Logic models and content analyses for the explication of evaluation theories: the case of emergent realist evaluation.

We comment on a set of papers in this journal issue that have examined three evaluation theories by constructing a logic model for each and conducting a set of comparative content analyses. We consider the set of papers as an instance of research on evaluation. For instance, we question the extent to which linear logic models convey the contingent decision making emphasized by some evaluation theories. We also offer specific comments about the papers' treatment of one theory, so-called emergent realist evaluation.

"I couldn't have seen it coming": the impact of negative self-relevant outcomes on retrospections about foreseeability.

We examined a phenomenon related to hindsight bias, specifically, retrospective judgements about the foreseeability of an outcome. We predicted that negative, self-relevant outcomes would be judged as less foreseeable by the recipient of the outcome than by others, unlike either positive outcomes or outcomes that are not self-relevant. In the context of a "stock market decision-making game", the hypothetical stock selected by one of two players showed an extreme increase or decrease. As predicted, the player who received an extreme negative outcome reported that this outcome was less foreseeable than did the opponent and an observer, for whom the outcome was less self-relevant. For no other kind of outcome was there a difference between the recipient of an outcome, the opponent, and the observer. The findings have several implications, including the possibility that hindsight bias should be considered as a special case of retrospective foreseeability.

Mood as information in making attributions to discrimination.

Previous research demonstrates that people use their mood as information when making a variety of judgments. The present research examines the extent to which people use their current mood as information when making attributions to discrimination. Women were given a positive or negative mood induction and either provided with an external attribution for their current mood state or not. They then reported on discrimination occurring to themselves and other women. When an external attribution for induced mood was not provided, women in positive moods were less likely to report discrimination across three measures than were women in negative moods. When an external attribution was provided, mood had no effect. Implications for understanding the effects of context and individual differences in the perception and reporting of experiences with discrimination are discussed.