Observation of resident clinical skills: outcomes of a program of direct observation in the continuity clinic setting.

OBJECTIVE: To assess the feasibility of a new multi-institutional program of direct observation and report what faculty observed and the feedback they provided. METHODS: A program of direct observation of real patient encounters was implemented in 3 pediatric residency programs using a structured clinical observation (SCO) form to document what was observed and the feedback given. Outcome variables included the number of observations made, the nature of the feedback provided, resident attitudes about direct observation before and after implementation, and the response of the faculty. RESULTS: Seventy-nine preceptors and 145 residents participated; 320 SCO forms were completed. Faculty provided feedback in 4 areas: content, process of the encounter, patient-centered attitudes and behaviors, and interpersonal skills. Feedback was 85% specific and 41% corrective. Corrective feedback was most frequent for physical examination skills. After program implementation, residents reported an increase in feedback and a decrease in discomfort with direct observation; in addition, they agreed that direct observation was a valuable component of their education. Participation rates among faculty were high. CONCLUSIONS: Direct observation using SCOs results in timely and specific feedback to residents about behaviors rarely observed in traditional precepting models. Resident competency in these clinical skill domains is critical for assessing, diagnosing, and managing patients. The SCO methodology is a feasible way to provide formative feedback to residents about their clinical skills.

Decreasing variability in faculty ratings of student case presentations: a faculty development intervention focusing on reflective practice.

BACKGROUND: Faculty perceptions of medical student knowledge are often based on case presentations, but their assessment of student performance can vary significantly based on content, presentation skill, and interpersonal characteristics. PURPOSE: To determine if a brief faculty development intervention could significantly decrease (>or= 20%) faculty members' variability in rating student case presentations in the ambulatory setting. METHODS: During the 1998-99 academic year, we videotaped 10 third-year medical students during one well-child presentation. Seven general pediatric faculty rated the videotapes using a validated scoring instrument measuring content: History of Present Illness (HPI), Past Medical History, and Physical Exam (PE); and communication skill: Precision of Language, Fluency, and Economy. Baseline ratings were from June to September 1999 and then repeated in October to November 2000 following a workshop where faculty reflected on the rating process and decreasing rating variance. The change in mean faculty ratings of the student's overall performance and for each item and the change in the mean difference in standard deviation before and after the workshop were analyzed. RESULTS: Internal consistency of the scoring instrument using Cronbach's alpha was 0.88. The mean faculty rating for overall presentation performance and the PE did not significantly change after the workshop, whereas for all other items it did. The mean standard deviation of faculty ratings decreased significantly for the overall performance and all other items except HPI. CONCLUSION: Despite using a validated scoring instrument to rate student case presentations significant variability exists among faculty, with rating inflation likely. A brief faculty development intervention using "reflection-on-action" was associated with more congruent ratings in the short term.