Variation in faculty evaluations of clerkship students attributable to surgical service.

OBJECTIVE: The purpose of this study was to determine whether students' performance evaluations by faculty were influenced by the clinical service on which the student was evaluated. METHODS: Third-year medical students spent 8 weeks rotating on 3 (or 2) surgical services. Typically, students rotate on one 4-week general surgery service and two 2-week subspecialty services. Faculty members rated student performance on 5 characteristics and provided a numeric grade. Data were analyzed to determine whether any significant variations in evaluation patterns emerged. RESULTS: A total of 1033 evaluations were included in the analyses. Based on an analysis of variance, the numeric grade varied significantly by service (p < 0.001). The partial eta squared statistic was large (0.21). Ratings of students' performance on specific performance characteristics also varied significantly by service (p < 0.001). CONCLUSIONS: The assessment of a surgical student's clinical performance is influenced by the specific services on which he/she has rotated and may be related to the length of the rotation. Research is needed to determine whether the differences among services should be considered as a source of error in grading or considered to reflect the particular challenge of the service.

360-degree evaluations of plastic surgery resident accreditation council for graduate medical education competencies: experience using a short form.

BACKGROUND: The Accreditation Council for Graduate Medical Education has asked training programs to develop methods to evaluate resident performance, using competencies essential for outcomes. METHODS: A two-page form was completed by 12 surgeons and 28 nurses and clinical staff directly involved in plastic surgery patient care (n = 40), evaluating University of Kentucky plastic surgery residents at each level of training (n = 6). There were eight groups of health care professionals among the 40. Six Accreditation Council for Graduate Medical Education competencies were rated, with technical/operative skills added as a subset of patient care. Hierarchical cluster analysis was used to determine similarity of rating profiles of the rating groups; Kruskal-Wallis analysis of variance delineated the way in which the participants used the competencies to make their selections by asking them whether they would choose the resident for future surgical care. RESULTS: Rating profiles revealed two clusters of raters. In one cluster were nurses assigned to an ambulatory surgery center, faculty, residents, and an intern (the surgeons' cluster; n = 15); in the second cluster were other nurses and clinical staff (nurses' cluster; n = 25). The nurses' cluster was found to rate residents more positively, and the surgeons' cluster more often cited areas for improvement. Specific competencies deemed important to each group were identified. CONCLUSIONS: Resident performance is rated differently by health care professionals, in two distinct groups. Based on this clustered arrangement, the resident is able quarterly to enjoy two, independent, formative assessments, potentially over 6 years of integrated training.

Surgical resident peer evaluations-what have we learned.

PURPOSE: Residents spend most of their time in the hospital with their respective junior and senior colleagues. Therefore, residents have a unique and valuable insight into their peers' professional qualities. Peer evaluation of values and virtues of each group may bring a different or an even better aspect of evaluating some ACGME core competencies. The purpose of this study was to identify the characteristics of junior and senior residents who are perceived to be role models in contrast to those who are not and to determine among these characteristics which is the most influential factor(s). METHODS: Seventeen senior residents (4th and 5th) and 26 junior residents (1st, 2nd, and 3rd) evaluated each other in an anonymous manner 6 months apart on 3 separate occasions. Questionnaires for each group were designed by their evaluating group. Each questionnaire listed the qualities of a role-model resident anchored on a scale of 1-3. RESULTS: A 1-way analysis of variance indicated that most of the time, the senior residents are perceived to be role models (the mean role-model rating = 2.87). A partial eta-squared analysis was performed to determine how well the specific performance factors differentiate the cases in which senior residents were perceived to be role models and when they were not. Clearly, professionalism along with attitude/expectations was the most differentiating factor. Similar statistical analyses were performed on seniors evaluating juniors. The role-model mean ratings were different significantly from one another (p = 0.014), which indicated that the senior residents varied significantly in the degree to which juniors were perceived to be role models. CONCLUSION: In each group, both junior and senior residents value professionalism as the most important differentiating attribute of a role model. Seniors felt that knowledge was a less important attribute of a junior role model. Juniors felt teaching inside and outside of the operating room was a less important attribute of a senior role model.

Factors in faculty evaluation of medical students' performance.

CONTEXT: Faculty members often use global rating scales as a method of assessing various characteristics of medical students' clinical performance. The purpose of this study was to determine if some performance characteristics are more highly associated with the overall faculty grade than others. METHODS: The clinical performance of 211 surgery clerkship students was evaluated by 2 or 3 faculty preceptors. Faculty rated students on 10 specific performance characteristics, using a 5-point scale. Faculty then assigned a numerical grade summarising the faculty's view of the student's performance. Reliability of the ratings was estimated by the intraclass correlation, and 1-way (analysis of variance) anova was used to test for differences among the students' mean ratings. Logistic regression was employed to determine the accuracy of each performance measure in predicting students' grades (A or B). Stepwise logistic regression was used to determine if there was a combination of performance characteristics that best predicted students' grades. RESULTS: The inter-rater reliabilities were low (

What do faculty observe of medical students' clinical performance?

BACKGROUND: An earlier study of our faculty's evaluation of junior medical students indicated that performance ratings were unreliable and reflected 1 underlying dimension. Other researchers have obtained similar results. PURPOSE: The purpose of this study was to identify which aspects of students' clinical performance faculty actually observe. METHODS: We analyzed the responses of 9 faculty members to an open-ended questionnaire concerning which aspects of clinical performance attending faculty observe. We also reviewed and summarized the written comments of 331 faculty evaluations of third-year medical students. RESULTS: Analysis of the questionnaires and evaluations indicated that faculty members gauge medical knowledge, professionalism, and clinical reasoning skills from direct interaction with students. History-taking and physical examination skills are inferred from the quality of verbal presentations. Faculty have little basis for evaluating other important aspects of clinical performance. CONCLUSIONS: Faculty primarily observe medical students' cognitive skills and professionalism. Faculty have little basis for evaluating most other features of clinical performance.

Ongoing faculty evaluations: developmental gain or just more pain?

PURPOSE: We continue to increase the amount of evaluations to improve the outcomes of our residency programs. Although ongoing faculty evaluations clearly are an important part of faculty development, their value in terms of improving the program needs to be evaluated. The questions asked were as follows: (1) Do faculty evaluations continue to improve the faculty over the course of successive evaluation periods? (2) Are there groups of faculty who would benefit the most from faculty evaluation feedback? (3) Are there any specific objective categories within the evaluation that carry more value and may help to shorten this form? METHODS: Forty-two faculty members were evaluated by 40 surgical residents with an assessment form developed by surgical residents that assessed faculty members by 10 different criteria. The initial set of data was collected, and attending surgeons were given an intervention in the form of a letter detailing how they had been assessed in each of the 10 categories. The attending surgeons were evaluated again 6 months later and were given an intervention in the form of verbal feedback regarding their evaluations. The attending faculty members were then assessed 1 year after that. One way analyses of variance and Fisher Protected Least Significant Difference (PLSD) were used to analyze the resulting data to determine if there were significant differences in the faculty evaluations. A part-whole correlation was performed that correlated the 10 evaluation criteria against the mean score on each evaluation, and partial eta-squared analysis was used to determine which criterion had the largest effect on the overall means. RESULTS: The means for the 42 faculty members as a whole continued to improve from the first to the final evaluation period, with 30 faculty members increasing their mean score (18 significantly) and 12 decreasing their mean score (4 significantly). Seven of the 10 evaluation criteria's means improved sequentially by feedback session. These were (1) Didactic Teaching, (2) Teaching Rounds, (3) Attendance at Didactic Activities, (4) Allows [Resident] Autonomy to Make Independent Decisions, (5) Provides Feedback, (6) Stimulates Critical Thinking with Use of Literature, and (7) Encourages and Maintains an Atmosphere of Professional Mutual Respect for All Members of Health Care Team (Role Model). The faculty group with the lowest evaluations improved significantly more than those of both the middle and the role model group, with the middle and the role model groups improving, but not significantly differently from each other. Of the criteria that improved, only Provides Feedback improved significantly from the first to second and the second to third evaluation periods. The three criteria with the highest correlation coefficients were Role Model (0.76), Provides Feedback (0.75), and Stimulates Critical Thinking (0.74). The results from the partial eta-squared test showed that the criterion with the largest effect size was Provides Feedback (0.28). These analyses indicate that the criterion Provides Feedback was both highly correlated with the average score on a faculty member's evaluation and was more responsible than any other criteria for the overall improvement in the mean evaluation score of the faculty members. CONCLUSIONS: (1) Ongoing faculty evaluations indeed are a powerful tool to improve the faculty as a whole. (2) The faculty members with the lowest evaluations showed the largest amount of improvement. (3) Providing feedback to the residents seems to be the most valued factor by the residents for faculty evaluations and perhaps could become the basis of the evaluation for the most accomplished faculty.

Perioperative pain management education: a short structured regional anesthesia course compared with traditional teaching among medical students.

BACKGROUND AND OBJECTIVES: Previous research has demonstrated that a brief course on pain management improved knowledge and attitudes toward analgesic use among medical students. The purpose of this study is to compare a structured clinical instruction course on regional anesthesia techniques for perioperative pain management with traditional teaching given to senior medical students. METHODS: During a 1-month clerkship in anesthesiology, 40 fourth-year medical students were randomly and equally divided into 2 groups. The study group received a 2-hour structured course on regional anesthesia techniques for pain management, whereas the control group received a 1-hour lecture tutorial on regional anesthesia techniques for perioperative pain management and 1 hour of bedside teaching on acute pain management. Each student completed an objective structured clinical examination (OSCE) 2 weeks after completion of the course. RESULTS: The study group performed better on each of the 11 items of the OSCE and on the total performance scores (mean +/- SD of 36.2 +/- 7.3 for study group versus 14.8 +/- 8.4 for the control group; P < .05). All students rated the clinical course highly valuable (4.7 +/- 0.5). CONCLUSION: A structured clinical instructional course on regional techniques for perioperative pain management given to fourth-year medical students can significantly improve their understanding and knowledge compared with traditional teaching.

Surgical rotations: the good, the bad, and the ugly.

BACKGROUND: Every surgeon's career begins with a surgical residency. Each residency has dreaded rotations as well as those that seem ideal, but to date, no objective criteria exist to define such rotations. The aim of this study is to establish objective criteria that, when applied to a specific rotation, will define it as a "dream" or a "dreaded" rotation and, in the process, elucidate the good, the bad, and the ugly of surgical rotations in general. METHODS: All 39 residents from a multi-institutional residency program were asked to collaboratively identify the criteria they felt necessary for a dream rotation. Each resident evaluated 16 rotations at 4 hospitals using the identified criteria on a scale of "Inadequate," "Optimal," and "Ideal/Dream Rotation." Residents then gave an overall evaluation of each rotation as a dream rotation on a scale of "No," "For the Most Part," or "Yes." Finally, each resident was asked whether the individual rotations were following ACGME recommendations on duty hours. RESULTS: The best correlations with a dream rotation were: clinical experience - quality and quantity of cases (r = 0.83), operative experience (r = 0.83), patient management responsibilities (r = 0.78), and outpatient office experience (r = 0.77). All p-values were < 0.05. The following correlated less with a dream rotation: conferences (r = 0.56, p < 0.05), medical student experience (r = 0.56, p < 0.05), and hospital facilities (r = 0.28, not significant.). Scut work was the only category for which there was a negative correlation with a dream rotation (r = -0.53, p < 0.05). Eighty-six percent of residents reported compliance with ACGME work hour recommendations, with no apparent correlation with a dream rotation.

Faculty evaluations: diagnostic and therapeutic.

PURPOSE: An excellent teaching program is one that creates an atmosphere of mentorship, leadership, and professional development. The key to all of these may lie in the faculty evaluation. Currently, favorable faculty evaluations are often rewarded, whereas the more critical evaluations are downplayed and seldom used for faculty development. The purpose of this study is to investigate the efficacy of the faculty evaluation as a diagnostic tool to objectively identify individual strengths and weaknesses. Can such a tool then be used as a therapeutic modality to improve and enhance the faculty as a whole? METHODS: Fourth- and fifth-year surgical residents from 1 residency program were asked to collaboratively define 9 characteristics that make a surgical role model. They then anonymously evaluated each of the 44 teaching attendings on each of the 9 criteria. The results of the evaluations, which consisted of the overall mean scores and SD for each of the 9 criteria, the attending's score and SDs above or below the mean, as well as whether the residents considered them to be a Role Model, were sent to each faculty member as an intervention. Six months after the intervention, the residents evaluated the attendings using the same 9 criteria. Evaluations were collected and analyzed following the same procedure as 6 months prior. T-tests and p-values were calculated to determine any significant differences between the 2 data sets. RESULTS: Based on the results of the pre-intervention evaluations, we were able to delineate faculty members into 3 groups based on their Role Model score. Of the 10 faculty members with the lowest scores (defined as >1 SD below average), 7 improved after the intervention, 3 of them were statistically significant (2 with p < 0.05 and 1 with p < 0.10). In the middle group of 26 faculty members, 1 showed statistically significant improvement (p < 0.05). Interestingly, this was the faculty member with the lowest score in this group. Six faculty originally in the middle group improved to a score greater than 1 SD above average, defining them as a Role Model. The remaining 8 faculty members originally designated as Role Models showed no statistically significant changes in their scores after the intervention. After the intervention, the average score improved for all of the 9 criteria that make a faculty Role Model. Statistically significant improvements were seen in the categories of Provides Feedback (p < 0.05) and Didactic Teaching, Attendance at Didactic activities, and Stimulates Critical Thinking with the Use of Literature (all p < 0.10). CONCLUSIONS: (1) Faculty evaluations are a diagnostic tool with the ability to select out Role Models from the lower scoring groups. (2) After the intervention, the overall average score improved for all of the 9 criteria that make a Role Model, showing improvement of the faculty as a whole. (3) The therapeutic benefits of faculty evaluations were best seen in the faculty with the lowest scores.

Enhancing the clinical skills of surgical residents through structured cancer education.

OBJECTIVE: To assess the short and long-term educational value of a highly structured, interactive Breast Cancer Structured Clinical Instruction Module (BCSCIM). SUMMARY BACKGROUND DATA: Cancer education for surgical residents is generally unstructured, particularly when compared with surgical curricula like the Advanced Trauma Life Support (ATLS) course. METHODS: Forty-eight surgical residents were randomly assigned to 1 of 4 groups. Two of the groups received the BCSCIM and 2 served as controls. One of the BCSCIM groups and 1 of the control groups were administered an 11-problem Objective Structured Clinical Examination (OSCE) immediately after the workshop; the other 2 groups were tested with the same OSCE 8 months later. The course was an intensive multidisciplinary, multistation workshop where residents rotated in pairs from station to station interacting with expert faculty members and breast cancer patients. RESULTS: Residents who took the BCSCIM outperformed the residents in the control groups for each of the 7 performance measures at both the immediate and 8-month test times (P < 0.01). Although the residents who took the BCSCIM had higher competence ratings than the residents in the control groups, there was a decline in the faculty ratings of resident competence from the immediate test to the 8-month test (P < 0.004). CONCLUSIONS: This interactive patient-based workshop was associated with objective evidence of educational benefit as determined by a unique method of outcome assessment.

Are you a surgical role model?

PURPOSE: Multiple papers have examined the recent decline in the number of surgical residency applicants. Many have concluded that a lack of role models in surgery is at least partially responsible for this decline. However, to date, the definition of a surgical role model does not exist. This paper defines a surgical role model based on criteria determined by surgery residents. METHODS: Fourth- and fifth-year surgical residents from 1 residency program were asked to collaboratively define 9 characteristics that make a surgical role model. The 9 criteria as defined by the residents were didactic teaching, teaching rounds, attendance at didactic activities, demonstrates skills and decision making in the operating room with confidence and virtuosity, allows [resident] to do procedures according to ability, allows autonomy to make independent decisions, provides feedback, stimulates critical thinking with use of literature, and assists [resident] to find and complete research for publication. Each resident in the program was then given a questionnaire and asked to evaluate each of the 49 teaching attendings on each of the 9 criteria, on a 3-point scale. Finally, residents evaluated each attending based on professionalism and mutual respect on a scale of I Don't Want To Emulate Him/Her, OK, or Role Model. These categories were also assigned a number (1, 2, and 3, respectively). Pearson correlation and stepwise multiple regression were used to determine the relationship between "Role Model" and the 9 criteria. The unit of analysis was the mean rating given each attending on each of the 10 scales. RESULTS: A total of 847 questionnaires were analyzed. Each of the 9 criteria correlated significantly with the Role Model rating (all p < 0.01). The average correlation was 0.73 (range, 0.64 to 0.78). Of the 9 criteria, 4 correlated best with the Role Model, as shown in. The stepwise regression indicates that 3 of the 9 criteria are uniquely associated with the Role Model variable. These 3 criteria are stimulates critical thinking with use of literature, allows autonomy to make independent decisions, and attendance at didactic activities. CONCLUSIONS: All of the 9 criteria are important factors in residents' perception of the Role Model characteristics of attending faculty. A parsimonious, operational definition of the surgical attending role model is one who stimulates the resident to think, gives the resident the opportunity to think and act independently, and is available to the resident.

A valid method of laparoscopic simulation training and competence assessment.

BACKGROUND: The purpose of our study was to evaluate the construct validity of laparoscopic technical performance measures and the face validity of three laparoscopic simulations. MATERIALS AND METHODS: Subjects (N = 27) of varying levels of surgical experience performed three laparoscopic simulations, representing appendectomy (LA), cholecystectomy (LC), and inguinal hemiorrhaphy (LH). Five laparoscopic surgeons, blinded to the identity of the subjects, rated the subjects on procedural competence on a binary scale and in four skills categories on a 5-point scale: clinical judgment, dexterity, serial/simultaneous complexity, and spatial orientation. Using a task-specific checklist, non-clinical staff assessed the technical errors. The level of surgical experience was correlated with the ratings, the technical errors, and the time for each procedure. Subject responses to a survey regarding the utility of the inanimate models were evaluated. RESULTS: Years of experience directly correlated with the skills ratings (all P < 0.001) and with the competence ratings across the three procedures (P < 0.01). Experience inversely correlated with the time for each procedure (P < 0.01) and the technical error total across the three models (P < 0.05). Nearly all subjects agreed that the corresponding procedures were well represented by the simulations (LA 96%, LC 96%, LH 100%). CONCLUSION: The laparoscopic simulations demonstrated both face and construct validity. Regardless of the level of surgical experience, the subjects found the models to be suitable representations of actual laparoscopic procedures. Task speed improved with surgical experience. More importantly, the quality of performance increased with experience, as shown by the improvement in the skills assessments by expert laparoscopic surgeons.

Are multiple objective measures of student performance necessary?

BACKGROUND: This study examines the effect of using multiple modalities to evaluate medical students. METHODS: Thirty-four students were evaluated by a complex model utilizing National Board of Medical Examiners (NBME) shelf examination, Objective Structured Clinical Examination (OSCE), Computer Patient Simulation (CPS), faculty and peer evaluation. Results were compared with a traditional model based on NBME and faculty evaluation alone. RESULTS: Reliability (coefficient alpha) of the complex and traditional models were 0.72 and 0.47, respectively. Item correlations suggested that NBME was most discriminating (r = 0.75), followed by OSCE (r = 0.52), peer evaluation (r = 0.43), CPS (r = 0.39), and faculty evaluation (r = 0.32). Rank order correlation (Spearman's rho) between scores calculated using each model was 0.87. CONCLUSIONS: Although the complex model has improved reliability, both models rank students similarly. However, neither model fully captures and reflects the information provided by each of the specific evaluation methods.