Feedback to support examiners' understanding of the standard-setting process and the performance of students: AMEE Guide No. 145.

The ratings that judges or examiners use for determining pass marks and students' performance on OSCEs serve a number of essential functions in medical education assessment, and their validity is a pivotal issue. However, some types of errors often occur in ratings that require special efforts to minimise. Rater characteristics (e.g. generosity error, severity error, central tendency error or halo error) may present a source of performance irrelevant variance. Prior literature shows the fundamental problems in student performance measurement attached to judges' or examiners' errors. It also indicates that the control of such errors supports a robust and credible pass mark and thus, accurate student marks. Therefore, for a standard-setter who identifies the pass mark and an examiner who rates student performance in OSCEs, proper, user-friendly feedback on their standard-setting and ratings is essential for reducing bias. This feedback provides useful avenues for understanding why performance ratings may be irregular and how to improve the quality of ratings. This AMEE Guide discusses various methods of feedback to support examiners' understanding of the performance of students and the standard-setting process with an effort to make inferences from assessments fair, valid and reliable.

Evaluation of a revised curriculum: a four-year qualitative study of student perceptions.

Following curricular revisions at the Virginia Commonwealth University School of Dentistry, this longitudinal study was designed to determine students' perceptions of their educational experience in the revised curriculum. A SWOT (Strengths, Weaknesses, Opportunities, and Threats) open-ended response questionnaire was administered to students in the class of 2011 (N=89) in January of each academic year, 2008 through 2011, followed by focus groups three months prior to graduation. The overall response rate for the questionnaire was 69 percent, and a total of fourteen students participated in four focus groups. Cumulatively, 1,382 responses (SWOT=984 and focus groups=398) were qualitatively analyzed, and five themes emerged: 1) early clinical experiences led to a perceived readiness for direct patient care; 2) the pace and organization of the revised condensed preclinical curriculum were perceived as hectic yet were appreciated as necessary preparation for patient care; 3) most faculty members were seen as committed to student learning, but a few were reported to have poor teaching skills and attitudes when interacting with students; 4) a perceived lack of patients led to fewer clinical experiences and a decrease in student confidence; and 5) some curricular content was seen to be redundant and irrelevant to future practice. The results indicate that the students were satisfied with aspects of their educational experience, suggesting the revised curriculum's preliminary success in meeting its goals of earlier patient care, a condensed preclinical curriculum, and a student-friendly environment. As the curriculum is adapted in response to student feedback, ongoing evaluation is necessary and should be complemented by other evaluation indicators such as faculty perceptions and student learning outcomes.

Factor analysis methods and validity evidence: a review of instrument development across the medical education continuum.

PURPOSE: Instrument development consistent with best practices is necessary for effective assessment and evaluation of learners and programs across the medical education continuum. The author explored the extent to which current factor analytic methods and other techniques for establishing validity are consistent with best practices. METHOD: The author conducted electronic and hand searches of the English-language medical education literature published January 2006 through December 2010. To describe and assess current practices, she systematically abstracted reliability and validity evidence as well as factor analysis methods, data analysis, and reported evidence from instrument development articles reporting the application of exploratory factor analysis and principal component analysis. RESULTS: Sixty-two articles met eligibility criteria. They described 64 instruments and 95 factor analyses. Most studies provided at least one source of evidence based on test content. Almost all reported internal consistency, providing evidence based on internal structure. Evidence based on response process and relationships with other variables was reported less often, and evidence based on consequences of testing was not identified. Factor analysis findings suggest common method selection errors and critical omissions in reporting. CONCLUSIONS: Given the limited reliability and validity evidence provided for the reviewed instruments, educators should carefully consider the available supporting evidence before adopting and applying published instruments. Researchers should design for, test, and report additional evidence to strengthen the argument for reliability and validity of these measures for research and practice.

Patient safety attitudes and behaviors of graduating medical students.

As North American medical schools reformulate curricula in response to public calls for better patient safety, surprisingly little research is available to explain and improve the translation of medical students' knowledge and attitudes into desirable patient safety behaviors in the clinical setting. A total of 139 fourth-year medical students at Virginia Commonwealth University, School of Medicine, 96% of the 2010 graduating class, completed the Attitudes toward Patient Safety Questionnaire and a self-report of safety behaviors. The students were exposed to informal discussions of patient safety concepts but received no formal patient safety curriculum. Most students recognized errors and responded with attitudes supportive of patient safety but desired behaviors were less common. In particular, errors went unreported, owing, in part, to the relationships of power and social influence undergirding the traditional authority gradient in the culture of medicine. A deeper understanding of patient safety attitudes, behavior, and medical culture is required to better inform instructional design decisions that influence desired patient safety behaviors and improve patient care.

Measuring medical students' orientation toward lifelong learning: a psychometric evaluation.

BACKGROUND: The principle of lifelong learning is pervasive in regulations governing medical education and medical practice; yet, tools to measure lifelong learning are lagging in development. This study evaluates the Jefferson Scale of Physician Lifelong Learning (JeffSPLL) adapted for administration to medical students. METHOD: The Jefferson Scale of Physician Lifelong Learning-Medical Students (JeffSPLL-MS) was administered to 732 medical students in four classes. Factor analysis and t tests were performed to investigate its construct validity. RESULTS: Maximum likelihood factor analysis identified a three-factor solution explaining 46% of total variance. Mean scores of clinical and preclinical students were compared; clinical students scored significantly higher in orientation toward lifelong learning (P < .001). CONCLUSIONS: The JeffSPLL-MS presents findings consistent with key concepts of lifelong learning. Results from use of the JeffSPLL-MS may reliably inform curriculum design and education policy decisions that shape the careers of physicians.