A Faculty Development Program to Reduce Rater Error on Milestone-Based Assessments.

BACKGROUND: Rater errors, such as halo/reverse halo, range restriction, and leniency errors, are frequently cited as threats to the validity of resident assessment by faculty. OBJECTIVE: We studied whether participation in faculty development on the use of a new Milestone-based assessment tool reduced rater error for participants compared to individuals who did not participate. METHODS: We reviewed evaluations of resident Milestones completed by faculty at the end of rotations between July 2012 and June 2013. The 2 Milestones in each competency with the greatest number of ratings were selected for analysis. RESULTS: A total of 412 evaluations were analyzed, including 217 completed by faculty who participated in the development activity, and 240 completed by nonparticipant faculty. All evaluations that contained identical scores for all Milestones (16%) were completed by nonparticipant faculty (χ(2)  =  37.498, P < .001). Faculty who had participated in development assigned a wider range of scores and lower minimum scores to residents, and provided the highest ratings for residents less frequently (P < .001) than nonparticipants. CONCLUSIONS: Faculty who participated in education about the Milestones demonstrated significantly less halo, range restriction, and leniency errors than faculty members who did not participate. These findings support a recommendation to develop a cadre of "core faculty" by training them in the use of Milestone assessment tools, and making them responsible for a significant portion of resident assessments.

It's difficult to change the way we teach: lessons from the Integrative Themes in Physiology curriculum module project.

The Integrative Themes in Physiology (ITIP) project was a National Science Foundation-funded collaboration between the American Physiological Society (APS) and the Human Anatomy and Physiology Society (HAPS). The project goal was to create instructional resources that emphasized active learning in undergraduate anatomy and physiology classrooms. The resources (activity modules and professional development) addressed two factors thought to be limiting science education reform: instructors' knowledge of how to implement active learning instruction and time to design innovative curricula. Volunteer instructors with a strong interest in using active learning in their classrooms were recruited to use the ITIP modules and provide ease-of-use feedback and student assessment data. As the study unfolded, instructor attrition was higher than had been anticipated, with 17 of 36 instructors withdrawing. More surprisingly, instructors remaining with the project failed to use the modules and reported specific obstacles that precluded module use, including lack of support from academic leadership, unplanned class size increases and heavy teaching loads, a union strike, insufficient time to develop a mindset for change, inadequate technology/funding, an adverse human subjects ruling, incompatibility of modules with instructors' established content and expectations, and personal factors. Despite the lack of module use and obstacles, 8 of 19 site testers began independently to introduce new active learning instruction into their classrooms. In the larger picture, however, it is important to note that only 8 of the initial 36 volunteers (22%) actually ended up changing their instruction to include opportunities for student active learning. These findings underscore the difficulty of implementing instructional change in college classrooms.