Consequences of survey modification in a program evaluation: An exploratory research on evaluation study.

The purpose of this exploratory research on evaluation study was to examine how modified and unmodified scales of critical thinking and interest in science careers would affect the evaluation conclusions. Surveys measuring various outcomes that are used in program evaluation are frequently modified from their original versions in response to the unique context of programs. Modifying existing published surveys by removing or adding items can affect the psychometric properties of the original scale and may produce differing results. The results of the comparisons found that unmodified and modified surveys had similar reliabilities; however, one of the scales produced contradictory evaluation findings. Lessons learned from this study suggest that scales can be modified in evaluation, but great care is needed to address the potential strengths and limitations of the modified scale and balance the technical needs with responsiveness to program context.

Large-scale evaluation efforts and their implications for the field.

The BUILD initiative is part of the Diversity Program Consortium, which the National Institutes of Health funded to increase diversity in biomedical research. This chapter aims to identify implications for the field from the multisite evaluation of BUILD initiative programs by reviewing the work undertaken by the authors of the other chapters in this issue. Given the complexities involved in multisite evaluations, innovative approaches and methods were used to balance the needs of each site with the overall objectives of the broader initiative. These approaches included a flexible orientation to the evaluation, mixed-methods designs that prioritized understanding the context before measuring it, and innovative analytic techniques (e.g., meta-analysis) to recognize the uniqueness of each site while providing insights about their cumulative impact. The BUILD initiative evaluation also offered many other valuable lessons about engaging stakeholders, focusing on use, and responding to changing priorities over time.

The manifestations of politics in evaluation: An exploratory study across the evaluation process.

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it's time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.

The role of titles in enhancing data visualization.

Many in the data visualization and evaluation communities recommend conveying the message or takeaway of the visualization in the visualization's title. This study tested that recommendation by examining how informative or generic titles impact a visualization's visual efficiency, aesthetics, credibility, and the perceived effectiveness of the hypothetical program examined. Furthermore, this study tested how simple or complex graphs, and positive, negative, or mixed results (i.e., valence of the results) affected outcomes. Participants were randomly assigned to one of 12 conditions, representing a 2 (graph: simple or complex) x 2 (title: generic or informative) x 3 (valence: positive, negative, mixed) between-subjects study. The results indicated that informative titles required less mental effort and were viewed as more aesthetically pleasing, but otherwise did not lead to greater accuracy, credibility, or perceived effectiveness. Furthermore, titles did not interact with graph type or the valence of the findings. While the results suggest it is worthwhile to consider adding an informative title to data visualizations as they can reduce mental effort for the viewer, the intended goal of the visualization should be taken into consideration. Considering the goal of the visualization can be a deciding factor of the type of graph and title that will best serve its intended purposes. Overall, this suggests that data visualization recommendations that impact evaluation reporting practices should be scrutinized more closely through research.

Incorporating public values into evaluative criteria: Using crowdsourcing to identify criteria and standards.

At its core, evaluation involves the generation of value judgments. These evaluative judgments are based on comparing an evaluand's performance to what the evaluand is supposed to do (criteria) and how well it is supposed to do it (standards). The aim of this four-phase study was to test whether criteria and standards can be set via crowdsourcing, a potentially cost- and time-effective approach to collecting public opinion data. In the first three phases, participants were presented with a program description, then asked to complete a task to either identify criteria (phase one), weigh criteria (phase two), or set standards (phase three). Phase four found that the crowd-generated criteria were high quality; more specifically, that they were clear and concise, complete, non-overlapping, and realistic. Overall, the study concludes that crowdsourcing has the potential to be used in evaluation for setting stable, high-quality criteria and standards.

The effects of stakeholder involvement on perceptions of an evaluation's credibility.

This article presents a study of the effects of stakeholder involvement on perceptions of an evaluation's credibility. Crowdsourced members of the public and a group of educational administrators read a description of a hypothetical program and two evaluations of the program: one conducted by a researcher and one conducted by program staff (i.e. program stakeholders). Study participants were randomly assigned versions of the scenario with different levels of stakeholder credibility and types of findings. Results showed that both samples perceived the researcher's evaluation findings to be more credible than the program staff's, but that this difference was significantly reduced when the program staff were described to be highly credible. The article concludes with implications for theory and research on evaluation dissemination and stakeholder involvement.

Evaluative feedback delivery and the factors that affect success.

This study examines the factors that can affect the credibility, influence, and utility of evaluative feedback. These factors include the delivery strategy, accuracy, and type (positive/negative) of feedback provided. In this study over 500 participants were asked to complete a task, and were then randomly assigned to different conditions with varied feedback delivery methods, feedback accuracy, and types of feedback (positive/negative). Then they were asked questions about the feedback's credibility, influence, and utility.

Towards program theory validation: Crowdsourcing the qualitative analysis of participant experiences.

This exploratory study examines a novel tool for validating program theory through crowdsourced qualitative analysis. It combines a quantitative pattern matching framework traditionally used in theory-driven evaluation with crowdsourcing to analyze qualitative interview data. A sample of crowdsourced participants are asked to read an interview transcript and identify whether program theory components (Activities and Outcomes) are discussed and to highlight the most relevant passage about that component. The findings indicate that using crowdsourcing to analyze qualitative data can differentiate between program theory components that are supported by a participant's experience and those that are not. This approach expands the range of tools available to validate program theory using qualitative data, thus strengthening the theory-driven approach.

Methodological Credibility: An Empirical Investigation of the Public's Perception of Evaluation Findings and Methods.

BACKGROUND: When evaluations are broadly disseminated, the public can use them to support a program or to advocate for change. METHODS: To explore how evaluations are perceived and used by the public, individuals in a sample of 425 people in the United States were recruited through an online crowdsourcing service called Mechanical Turk (www.mturk.com). Participants were randomly assigned to receive different versions of a press release describing a summative evaluation of a program. Each condition contained a unique combination of methods (e.g., randomized controlled design) and findings (positive or negative) to describe the evaluation and its findings. Participants in each condition responded to questions about their trust in the content of the evaluation findings and their attitudes toward the program. RESULTS: Results indicated that the type of evaluation methods and the direction of the findings both influenced the credibility of the findings and that the credibility of the findings moderated the relationship between the direction of the evaluation findings and attitudes toward the evaluated program. Additional evaluation factors to explore in future research with the public are recommended.

Crowdsourcing for quantifying transcripts: An exploratory study.

This exploratory study attempts to demonstrate the potential utility of crowdsourcing as a supplemental technique for quantifying transcribed interviews. Crowdsourcing is the harnessing of the abilities of many people to complete a specific task or a set of tasks. In this study multiple samples of crowdsourced individuals were asked to rate and select supporting quotes from two different transcripts. The findings indicate that the different crowdsourced samples produced nearly identical ratings of the transcripts, and were able to consistently select the same supporting text from the transcripts. These findings suggest that crowdsourcing, with further development, can potentially be used as a mixed method tool to offer a supplemental perspective on transcribed interviews.

Politics in evaluation: Politically responsive evaluation in high stakes environments.

The role of politics has often been discussed in evaluation theory and practice. The political influence of the situation can have major effects on the evaluation design, approach and methods. Politics also has the potential to influence the decisions made from the evaluation findings. The current study focuses on the influence of the political context on stakeholder decision making. Utilizing a simulation scenario, this study compares stakeholder decision making in high and low stakes evaluation contexts. Findings suggest that high stakes political environments are more likely than low stakes environments to lead to reduced reliance on technically appropriate measures and increased dependence on measures better reflect the broader political environment.

Anxiety and hostility to an "outsider," as moderated by low perceived power.

The hostile and anxious responses of young adults to an immigrant (vs. native-born) stranger were examined as a function of participants' self-perceived power. In Studies 1 and 2, individuals with low perceived social power (males, in particular) showed high anxiety toward an "outsider" but more so if that individual was an immigrant (and thus posed an ambiguous threat to their position in the hierarchy). In Study 3, young adult males competed on a reaction time test with an immigrant or native-born rival. With immigrant rivals, males with low perceived social power showed relatively high aggression toward an immigrant rival and derogation of the rival's formidability; however, they showed a more deferential pattern with native-born rivals.