Using audiotapes to monitor student-preceptor interactions in a decentralized clerkship.

BACKGROUND: Community-based preceptors are increasingly being used to provide clinical experiences for medical students; therefore, the issue of when and how to monitor the appropriateness of teacher-learner interactions in the clinical setting has become increasingly complex and urgent. This paper describes how an audiotaping program in a department of family medicine was designed to account for preceptor and student needs and what this audiotaping experience revealed about the teaching strengths and weaknesses of full-time and volunteer faculty. METHODS: Audiotaped case discussions between third-year students and preceptors were evaluated using a structured instrument that reflected important responsibilities of clerkship preceptors. Reviews were compared using chi-square to evaluate for significant differences between university and community preceptors in overall quality of interaction with students and in specific deficiency areas. Ninety-five percent confidence intervals were used to evaluate for significant differences in frequency of occurrence among deficiencies. RESULTS: A total of 376 audiotaped case discussions were reviewed between July 1992 and June 1994. University preceptors had significantly more "superior" interactions with students than their community peers (P < .05). However, university and community preceptors had nonsignificant differences when specific deficiency areas were compared. Deficiencies in students who were allowed to present cases and assessments occurred significantly less frequently, while deficiencies in preceptors using open-ended questions occurred significantly more frequently than deficiencies in other areas. CONCLUSIONS: This study shows that volunteer faculty performed comparably to full-time university faculty in both the frequency of deficiencies in teacher-learner interactions identified and in the types of deficiencies identified. Both university and volunteer faculty had the most deficiencies in higher-order teaching skills--those involving synthesizing and assessing student knowledge and selecting appropriate questions based on that information. Audiotaped case discussions have a potentially significant role in monitoring teacher-student interactions, particularly on decentralized clinical rotations.

Increasing inter-rater agreement on a family medicine clerkship oral examination--a pilot study.

BACKGROUND AND OBJECTIVES: Although oral examinations are a traditional measure of student performance during clinical clerkships, concerns have been expressed about poor inter-rater reliability on these examinations. This study examines inter-rater agreement in a family medicine clerkship oral examination. METHODS: Our study analyzed oral examination scores awarded to a class of 98 junior medical students by three categories of examiner (the clerkship director, full-time faculty, and third-year residents). For each student, the difference among the three examiners' raw scores was compared using a Cronbach statistic. Mean scores for the year awarded by each of the three groups of examiners were compared using a correlation coefficient and paired t tests. For students with high inter-rater disagreement, examiner narrative comments and types of questions on which raters disagreed were analyzed. RESULTS: Overall inter-rater agreement was high (Cronbach's alpha = .875). Paired t tests were nonsignificant between residents and faculty but were significant between residents and the clerkship director and between faculty and the clerkship director. In the small subset of students with little inter-rater agreement, no clear trends were identified to explain reasons for evaluator disagreement. CONCLUSION: High levels of inter-rater agreement on a clerkship oral examination can occur when several measures to increase inter-rater agreement are used.