Surgical Trainee Performance and Alignment With Surgical Program Director Expectations.

OBJECTIVE: To examine the alignment between graduating surgical trainee operative performance and a prior survey of surgical program director expectations. BACKGROUND: Surgical trainee operative training is expected to prepare residents to independently perform clinically important surgical procedures. METHODS: We conducted a cross-sectional observational study of US general surgery residents' rated operative performance for Core general surgery procedures. Residents' expected performance on those procedures at the time of graduation was compared to the current list of Core general surgery procedures ranked by their importance for clinical practice, as assessed via a previous national survey of general surgery program directors. We also examined the frequency of individual procedures logged by residents over the course of their training. RESULTS: Operative performance ratings for 29,885 procedures performed by 1861 surgical residents in 54 general surgery programs were analyzed. For each Core general surgery procedure, adjusted mean probability of a graduating resident being deemed practice-ready ranged from 0.59 to 0.99 (mean 0.90, standard deviation 0.08). There was weak correlation between the readiness of trainees to independently perform a procedure at the time of graduation and that procedure's historical importance to clinical practice ( p = 0.22, 95% confidence interval 0.01-0.41, P = 0.06). Residents also continue to have limited opportunities to learn many procedures that are important for clinical practice. CONCLUSION: The operative performance of graduating general surgery residents may not be well aligned with surgical program director expectations.

Using a Modified Delphi Method to Reach Consensus on Best Practices for Medical Education Abstract Submissions to Multiple Conferences.

PURPOSE: Medical education researchers are often uncertain whether they should submit abstracts to certain conferences. Therefore, we aimed to elicit consensus among medical education conference organizers to answer the question: what are best practices for research abstract submissions to multiple conferences? METHODS: Using a 44-question online survey, we conducted a modified Delphi process to identify best practices for abstract submissions to medical education conferences. Consistent with the Delphi process, we identified conference representatives from non-specialty medical education conferences and across four conference types (institutional, regional, national, and international) to serve as panelists. RESULTS: Eleven expert panelists, representing all four conference types-two institutional conferences, five regional conferences, two national conferences, and two international conferences-completed three rounds of the survey. After three rounds, panelists reached consensus on 39/44 survey items-26 items in round 1, 10 items in round 2, and three items in round 3. Panelists' consensus and rationale indicated that it is most appropriate to resubmit abstracts to conferences with a larger or different audience, but not to more homogeneous audiences. Among the four conference types, abstract resubmission from institutional conferences to other conference types was the most widely accepted. Panelists agreed that abstracts using data and results submitted or accepted for publication could be submitted to any conference type. CONCLUSION: The results of this study provide best practices for presenting scholarship to medical education conferences. We recommend that guidelines for medical education conference abstract submissions provide consistent, clear instructions regarding the appropriate life cycle of an abstract.

What faculty write versus what students see? Perspectives on multiple-choice questions using Bloom's taxonomy.

BACKGROUND: Using revised Bloom's taxonomy, some medical educators assume they can write multiple choice questions (MCQs) that specifically assess higher (analyze, apply) versus lower-order (recall) learning. The purpose of this study was to determine whether three key stakeholder groups (students, faculty, and education assessment experts) assign MCQs the same higher- or lower-order level. METHODS: In Phase 1, stakeholders' groups assigned 90 MCQs to Bloom's levels. In Phase 2, faculty wrote 25 MCQs specifically intended as higher- or lower-order. Then, 10 students assigned these questions to Bloom's levels. RESULTS: In Phase 1, there was low interrater reliability within the student group (Krippendorf's alpha = 0.37), the faculty group (alpha = 0.37), and among three groups (alpha = 0.34) when assigning questions as higher- or lower-order. The assessment team alone had high interrater reliability (alpha = 0.90). In Phase 2, 63% of students agreed with the faculty as to whether the MCQs were higher- or lower-order. There was low agreement between paired faculty and student ratings (Cohen's Kappa range .098-.448, mean .256). DISCUSSION: For many questions, faculty and students did not agree whether the questions were lower- or higher-order. While faculty may try to target specific levels of knowledge or clinical reasoning, students may approach the questions differently than intended.

Considerations for using race and ethnicity as quantitative variables in medical education research.

Throughout history, race and ethnicity have been used as key descriptors to categorize and label individuals. The use of these concepts as variables can impact resources, policy, and perceptions in medical education. Despite the pervasive use of race and ethnicity as quantitative variables, it is unclear whether researchers use them in their proper context. In this Eye Opener, we present the following seven considerations with corresponding recommendations, for using race and ethnicity as variables in medical education research: 1) Ensure race and ethnicity variables are used to address questions directly related to these concepts. 2) Use race and ethnicity to represent social experiences, not biological facts, to explain the phenomenon under study. 3) Allow study participants to define their preferred racial and ethnic identity. 4) Collect complete and accurate race and ethnicity data that maximizes data richness and minimizes opportunities for researchers' assumptions about participants' identity. 5) Follow evidence-based practices to describe and collapse individual-level race and ethnicity data into broader categories. 6) Align statistical analyses with the study's conceptualization and operationalization of race and ethnicity. 7) Provide thorough interpretation of results beyond simple reporting of statistical significance. By following these recommendations, medical education researchers can avoid major pitfalls associated with the use of race and ethnicity and make informed decisions around some of the most challenging race and ethnicity topics in medical education.

Number of Operative Performance Ratings Needed to Reliably Assess the Difficulty of Surgical Procedures.

OBJECTIVE: The profession of surgery is entering a new era of "big data," where analyses of longitudinal trainee assessment data will be used to inform ongoing efforts to improve surgical education. Given the high-stakes implications of these types of analyses, researchers must define the conditions under which estimates derived from these large datasets remain valid. With this study, we determine the number of assessments of residents' performances needed to reliably assess the difficulty of "Core" surgical procedures. DESIGN: Using the SIMPL smartphone application from the Procedural Learning and Safety Collaborative, 402 attending surgeons directly observed and provided workplace-based assessments for 488 categorical residents after 5259 performances of 87 Core surgical procedures performed at 14 institutions. We used these faculty ratings to construct a linear mixed model with resident performance as the outcome variable and multiple predictors including, most significantly, the operative procedure as a random effect. We interpreted the variance in performance ratings attributable to the procedure, after controlling for other variables, as the "difficulty" of performing the procedure. We conducted a generalizability analysis and decision study to estimate the number of SIMPL performance ratings needed to reliably estimate the difficulty of a typical Core procedure. RESULTS: Twenty-four faculty ratings of resident operative performance were necessary to reliably estimate the difficulty of a typical Core surgical procedure (mean dependability coefficient 0.80, 95% confidence interval 0.73-0.87). CONCLUSIONS: At least 24 operative performance ratings are required to reliably estimate the difficulty of a typical Core surgical procedure. Future research using performance ratings to establish procedure difficulty should include adequate numbers of ratings given the high-stakes implications of those results for curriculum design and policy.

Limited by our limitations.

Study limitations represent weaknesses within a research design that may influence outcomes and conclusions of the research. Researchers have an obligation to the academic community to present complete and honest limitations of a presented study. Too often, authors use generic descriptions to describe study limitations. Including redundant or irrelevant limitations is an ineffective use of the already limited word count. A meaningful presentation of study limitations should describe the potential limitation, explain the implication of the limitation, provide possible alternative approaches, and describe steps taken to mitigate the limitation. This includes placing research findings within their proper context to ensure readers do not overemphasize or minimize findings. A more complete presentation will enrich the readers' understanding of the study's limitations and support future investigation.

Does Performance on Evidence-Based Medicine and Urgent Clinical Scenarios Assessments Deteriorate During the Fourth Year of Medical School? Findings From One Institution.

PURPOSE: The fourth year of medical school (M4) should prepare students for residency yet remains generally unstructured, with ill-defined goals. The primary aim of this study was to determine whether there were performance changes in evidence-based medicine (EBM) and urgent clinical scenarios (UCS) assessments before and after M4 year. METHOD: University of Michigan Medical School graduates who matched into internship at Michigan Medicine completed identical assessments on EBM and UCS at the beginning of M4 year and 13 months later during postgraduate year 1 (PGY1) orientation. Individual scores on these assessments were compared using paired t test analysis. The associations of academic performance, residency specialty classification, and initial performance on knowledge changes were analyzed. RESULTS: During academic years 2014 and 2015, 76 students matched into a Michigan Medicine internship; 52 completed identical EBM stations and 53 completed UCS stations. Learners' performance on the EBM assessment decreased from M4 to PGY1 (mean 93% [SD = 7%] vs. mean 80% [SD = 13%], P < .01), while performance on UCS remained stable (mean 80% [SD = 9%] vs. mean 82% [SD = 8%], P = .22). High M4 performers experienced a greater rate of decline in knowledge level compared with low M4 performers for EBM (-20% vs. -4%, P = .01). Residency specialty and academic performance did not affect performance. CONCLUSIONS: This study demonstrated degradation of performance in EBM during the fourth year and adds to the growing literature that highlights the need for curricular reform during this year.

Employing a Root Cause Analysis Process to Improve Examination Quality.

PROBLEM: Multiple-choice question (MCQ) examinations represent a primary mode of assessment used by medical schools. It can be challenging for faculty to produce content-aligned, comprehensive, and psychometrically sound MCQs. Despite best efforts, sometimes there are unexpected issues with examinations. Assessment best practices lack a systematic way to address gaps when actual and expected outcomes do not align. APPROACH: The authors propose using root cause analysis (RCA) to systematically review unexpected educational outcomes. Using a real-life example of a class's unexpectedly low reproduction examination scores (University of Michigan Medical School, 2015), the authors describe their RCA process, which included a system flow diagram, a fishbone diagram, and an application of the 5 Whys to understand the contributors and reasons for the lower-than-expected performance. Using this RCA approach, the authors identified multiple contributing factors that potentially led to the low examination scores. These included lack of examination quality improvement (QI) for poorly constructed items, content-question and pedagogy-assessment misalignment, and other issues related to environment and people. OUTCOMES: As a result of the RCA, the authors worked with stakeholders to address these issues and develop strategies to prevent similar systematic issues from reoccurring. For example, a more robust examination QI process was developed. NEXT STEPS: Using an RCA approach in health care is grounded in practice and can be easily adapted for assessment. Because this is a novel use of RCA, there are opportunities to expand beyond the authors' initial approach for using RCA in assessment.

Does Reducing Clerkship Lengths by 25% Affect Medical Student Performance and Perceptions?

PURPOSE: Transforming a medical school curriculum wherein students enter clerkships earlier could result in two cohorts in clerkships simultaneously during the transition. To avoid overlapping cohorts at the University of Michigan Medical School, the length of all required clerkships was decreased by 25% during the 2016-2017 academic year, without instituting other systematic structural changes. The authors hypothe sized that the reduction in clerkship duration would result in decreases in student perfor mance and changes in student perceptions. METHOD: One-way analyses of variance and Tukey post hoc tests were used to compare the 2016-2017 shortened clerkship cohort with the preceding traditional clerkship cohorts (2014-2015 and 2015-2016) on the following student outcomes: National Board of Medical Examiners (NBME) subject exam scores, year-end clinical skills exam scores, evaluation of clerkships, perceived stress, resiliency, well-being, and perception of the learning environment. RESULTS: There were no significant differences in performance on NBME subject exams between the shortened clerkship cohort and the 2015-2016 traditional cohort, but scores declined significantly over the three years for one exam. Perceptions of clerkship quality improved for three shortened clerkships; there were no significant declines. Learning environment perceptions were not worse for the shortened clerkships. There were no significant differences in performance on the clinical skills exam or in perceived stress, resiliency, and well-being. CONCLUSIONS: The optimal clerkship duration is a matter of strong opinion, supported by few empirical data. These results provide some evidence that accelerating clinical education may, for the studied outcomes, be feasible.

Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter.

PURPOSE: Many factors influence the reliable assessment of medical students' competencies in the clerkships. The purpose of this study was to determine how many clerkship competency assessment scores were necessary to achieve an acceptable threshold of reliability. METHOD: Clerkship student assessment data were collected during the 2015-2016 academic year as part of the medical school assessment program at the University of Michigan Medical School. Faculty and residents assigned competency assessment scores for third-year core clerkship students. Generalizability (G) and decision (D) studies were conducted using balanced, stratified, and random samples to examine the extent to which overall assessment scores could reliably differentiate between students' competency levels both within and across clerkships. RESULTS: In the across-clerkship model, the residual error accounted for the largest proportion of variance (75%), whereas the variance attributed to the student and student-clerkship effects was much smaller (7% and 10.1%, respectively). D studies indicated that generalizability estimates for eight assessors within a clerkship varied across clerkships (G coefficients range = 0.000-0.795). Within clerkships, the number of assessors needed for optimal reliability varied from 4 to 17. CONCLUSIONS: Minimal reliability was found in competency assessment scores for half of clerkships. The variability in reliability estimates across clerkships may be attributable to differences in scoring processes and assessor training. Other medical schools face similar variation in assessments of clerkship students; therefore, the authors hope this study will serve as a model for other institutions that wish to examine the reliability of their clerkship assessment scores.

Pushing Critical Thinking Skills With Multiple-Choice Questions: Does Bloom's Taxonomy Work?

Medical school assessments should foster the development of higher-order thinking skills to support clinical reasoning and a solid foundation of knowledge. Multiple-choice questions (MCQs) are commonly used to assess student learning, and well-written MCQs can support learner engagement in higher levels of cognitive reasoning such as application or synthesis of knowledge. Bloom's taxonomy has been used to identify MCQs that assess students' critical thinking skills, with evidence suggesting that higher-order MCQs support a deeper conceptual understanding of scientific process skills. Similarly, clinical practice also requires learners to develop higher-order thinking skills that include all of Bloom's levels. Faculty question writers and examinees may approach the same material differently based on varying levels of knowledge and expertise, and these differences can influence the cognitive levels being measured by MCQs. Consequently, faculty question writers may perceive that certain MCQs require higher-order thinking skills to process the question, whereas examinees may only need to employ lower-order thinking skills to render a correct response. Likewise, seemingly lower-order questions may actually require higher-order thinking skills to respond correctly. In this Perspective, the authors describe some of the cognitive processes examinees use to respond to MCQs. The authors propose that various factors affect both the question writer and examinee's interaction with test material and subsequent cognitive processes necessary to answer a question.

The influence of MCAT and GPA preadmission academic metrics on interview scores.

Medical school admissions interviews are used to assess applicants' nonacademic characteristics as advocated by the Association of American Medical Colleges' Advancing Holistic Review Initiative. The objective of this study is to determine whether academic metrics continue to significantly influence interviewers' scores in holistic processes by blinding interviewers to applicants' undergraduate grade point averages (uGPA) and Medical College Admission Test (MCAT). This study examines academic and demographic predictors of interview scores for two applicant cohorts at the University of Michigan Medical School. In 2012, interviewers were provided applicants' uGPA and MCAT scores; in 2013, these academic metrics were withheld from interviewers' files. Hierarchical regression analysis was conducted to examine the influence of academic and demographic variables on overall cohort interview scores. When interviewers were provided uGPA and MCAT scores, academic metrics explained more variation in interview scores (7.9%) than when interviewers were blinded to these metrics (4.1%). Further analysis showed a statistically significant interaction between cohort and uGPA, indicating that the association between uGPA and interview scores was significantly stronger for the 2012 unblinded cohort compared to the 2013 blinded cohort (ß = .573, P < .05). By contrast, MCAT scores had no interactive effects on interviewer scores. While MCAT scores accounted for some variation in interview scores for both cohorts, only access to uGPA significantly influenced interviewers' scores when looking at interaction effects. Withholding academic metrics from interviewers' files may promote assessment of nonacademic characteristics independently from academic metrics.

Hidden item variance in multiple mini-interview scores.

The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by "hiding" the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores.

A Hybrid Interview Model for Medical School Interviews: Combining Traditional and Multisampling Formats.

PROBLEM: Most medical schools have either retained a traditional admissions interview or fully adopted an innovative, multisampling format (e.g., the multiple mini-interview) despite there being advantages and disadvantages associated with each format. APPROACH: The University of Michigan Medical School (UMMS) sought to maximize the strengths associated with both interview formats after recognizing that combining the two approaches had the potential to capture additional, unique information about an applicant. In September 2014, the UMMS implemented a hybrid interview model with six, 6-minute short-form interviews-highly structured scenario-based encounters-and two, 30-minute semistructured long-form interviews. Five core skills were assessed across both interview formats. OUTCOMES: Overall, applicants and admissions committee members reported favorable reactions to the hybrid model, supporting continued use of the model. The generalizability coefficients for the six-station short-form and the two-interview long-form formats were estimated to be 0.470 and 0.176, respectively. Different skills were more reliably assessed by different interview formats. Scores from each format seemed to be operating independently as evidenced through moderate to low correlations (r = 0.100-0.403) for the same skills measured across different interview formats; however, after correcting for attenuation, these correlations were much higher. NEXT STEPS: This hybrid model will be revised and optimized to capture the skills most reliably assessed by each format. Future analysis will examine validity by determining whether short-form and long-form interview scores accurately measure the skills intended to be assessed. Additionally, data collected from both formats will be used to establish baselines for entering students' competencies.