How Reliable are Single-Question Workplace-Based Assessments in Surgery?

OBJECTIVE: Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking. This study aimed to evaluate the inter-rater reliability of multiple operative WBA tools utilized in general surgery residency. DESIGN: General surgery residents and teaching faculty were recorded during 6 general surgery operations. Nine faculty raters each reviewed 6 videos and rated each resident on performance (using the Society for Improving Medical Professional Learning, or SIMPL, Performance Scale as well as the operative performance rating system (OPRS) Scale), entrustment (using the ten Cate Entrustment-Supervision Scale), and autonomy (using the Zwisch Scale). The ratings were reviewed for inter-rater reliability using percent agreement and intraclass correlations. PARTICIPANTS: Nine faculty members viewed the videos and assigned ratings for multiple WBAs. RESULTS: Absolute intraclass correlation coefficients for each scale ranged from 0.33 to 0.47. CONCLUSIONS: All single-item WBA scales had low to moderate inter-rater reliability. While rater training may improve inter-rater reliability for single observations, many observations by many raters are needed to reliably assess trainee performance in the workplace.

Characteristics and Practice Patterns of Non-certified Surgeons Treating Medicare Patients.

OBJECTIVE: The objective of this study is to explore the patient characteristics and practice patterns of non-certified surgeons who treat Medicare patients in the United States. SUMMARY BACKGROUND DATA: While most surgeons in the United States are board-certified, non-certified surgeons are permitted to practice in many locations. At the same time, surgical workforce shortages threaten access to surgical care for many patients. It is possible that non-certified surgeons may be able to help fill these access gaps. However, little is known about the practice patterns of non-certified surgeons. METHODS: A 100% sample of Medicare claims data from 2014-2019 were used to identify practicing general surgeons. Surgeons were categorized as certified or non-certified in general surgery​​ based on data from the American Board of Surgery. Surgeon practice patterns and patient characteristics were analyzed. RESULTS: A total of 2,097,206 patient cases were included in the study. These patients were treated by 16,076 surgeons, of which 6% were identified as non-certified surgeons. Compared to certified surgeons, non-certified surgeons were less frequently fellowship-trained (20.5% vs. 24.2%, P=0.008) and more likely to be a foreign medical graduate (14.5% vs. 9.2%, P<0.001). Non-certified surgeons were more frequently practicing in for-profit hospitals (21.2% vs. 14.2%, P<0.001) and critical access hospitals (2.2% vs. 1.3%, P<0.001), and were less likely to practice in a teaching hospital (63.2% vs. 72.4%, P<0.001). Compared to certified surgeons, non-certified surgeons treated more non-White patients (19.6% vs. 14%, P<0.001) as well as a higher percentage of patients in the two lowest socioeconomic status (SES) quintiles (36.2% vs. 29.2%, P<0.001). Operations related to emergency admissions were more common amongst non-certified surgeons (68.8% vs. 55.7%, P<0.001). There were no differences in gender or age of the patients treated by certified and non-certified surgeons. CONCLUSION: For Medicare patients, non-certified surgeons treated more patients who are non-White, of lower SES, and in more rural, critical-access hospitals.

Variation in Competence of Graduating General Surgery Trainees.

OBJECTIVE: To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training. DESIGN: Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021. RESULTS: The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy. CONCLUSIONS: There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Trainees' Perspectives on the Next Era of Assessment and Precision Education.

ABSTRACT: The next era of assessment in medical education promises new assessment systems, increased focus on ensuring high-quality equitable patient care, and precision education to drive learning and improvement. The potential benefits of using learning analytics and technology to augment medical training abound. To ensure that the ideals of this future for medical education are realized, educators should partner with trainees to build and implement new assessment systems. Coproduction of assessment systems by educators and trainees will help to ensure that new educational interventions are feasible and sustainable. In this paper, the authors provide a trainee perspective on 5 key areas that affect trainees in the next era of assessment: (1) precision education, (2) assessor education, (3) transparency in assessment development and implementation, (4) ongoing evaluation of the consequences of assessment, and (5) patient care data as sources of education outcomes.As precision education is developed, it is critical that trainees understand how their educational data are collected, stored, and ultimately utilized for educational outcomes. Since assessors play a key role in generating assessment data, it is important that they are prepared to give high-quality assessments and are continuously evaluated on their abilities. Transparency in the development and implementation of assessments requires communicating how assessments are created, the evidence behind them, and their intended uses. Furthermore, ongoing evaluation of the intended and unintended consequences that new assessments have on trainees should be conducted and communicated to trainees. Finally, trainees should participate in determining what patient care data are used to inform educational outcomes. The authors believe that trainee coproduction is critical to building stronger assessment systems that utilize evidence-based educational theories for improved learning and ultimately better patient care.

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps.

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.

Association of Gender and Operative Feedback Quality in Surgical Residents.

OBJECTIVE: Feedback is critical for learning, however, gender differences exist in the quality of feedback that trainees receive. For example, narrative feedback on surgical trainees' end-of-block rotations differs based on trainee-faculty gender dyads, with female faculty giving higher quality feedback and male trainees receiving higher quality feedback. Though this represents evidence of gender bias in global evaluations, there is limited understanding of how much bias might be present in operative workplace-based assessments (WBAs). In this study, we explore the quality of narrative feedback among trainee-faculty gender dyads in an operative WBA. DESIGN: A previously validated natural language processing model was used to examine instances of narrative feedback and assign a probability of being characterized as high quality feedback (defined as feedback which was relevant as well as corrective and/or specific). A linear mixed model was performed, with probability of high quality feedback as the outcome, and resident gender, faculty gender, PGY, case complexity, autonomy rating, and operative performance rating as explanatory variables. PARTICIPANTS: Analyses included 67,434 SIMPL operative performance evaluations (2,319 general surgery residents, 70 institutions) collected from September 2015 through September 2021. RESULTS: Of 36.3% evaluations included narrative feedback. Male faculty were more likely to provide narrative feedback compared to female faculty. Mean probabilities of receiving high quality feedback ranged from 81.6 (female faculty-male resident) to 84.7 (male faculty-female resident). Model-based results demonstrated that female residents were more likely to receive high quality feedback (p < 0.01), however, there was no significant difference in probability of high quality narrative feedback based on faculty-resident gender dyad (p = 0.77). CONCLUSIONS: Our study revealed resident gender differences in the probability of receiving high-quality narrative feedback following a general surgery operation. However, we found no significant differences based on faculty-resident gender dyad. Male faculty were more likely to provide narrative feedback compared to their female colleagues. Further research using general surgery resident-specific feedback quality models may be warranted.

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

OBJECTIVE: Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment. DESIGN: The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures. SETTING: This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative. PARTICIPANTS: A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded. RESULTS: The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures. CONCLUSIONS: Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.

The Long Shadow: A Historical Perspective on Racism in Medical Education.

To dismantle racism in U.S. medical education, people must understand how the history of Christian Europe, Enlightenment-era racial science, colonization, slavery, and racism shaped modern American medicine. Beginning with the coalescence of Christian European identity and empire, the authors trace European racial reasoning through the racial science of the Enlightenment into the White supremacist and anti-Black ideology behind Europe's global system of racialized colonization and enslavement. The authors then follow this racist ideology as it becomes an organizing principle of Euro-American medicine and examine how it manifests in medical education in the United States today. Within this historical context, the authors expose the histories of violence underlying contemporary terms such as implicit bias and microaggressions. Through this history, they also gain a deeper appreciation of why racism is so prevalent in medical education and how it affects admissions, assessments, faculty and trainee diversity, retention, racial climate, and the physical environment. The authors then recommend 6 historically informed steps for confronting racism in medical education: (1) incorporate the history of racism into medical education and unmask institutional histories of racism, (2) create centralized reporting mechanisms and implement systematic reviews of bias in educational and clinical activities, (3) adopt mastery-based assessment in medical education, (4) embrace holistic review and expand its possibilities in admissions, (5) increase faculty diversity by using holistic review principles in hiring and promotions, and (6) leverage accreditation to combat bias in medical education. These strategies will help academic medicine begin to acknowledge the harms propagated throughout the history of racism in medicine and start taking meaningful steps to address them. Although the authors have focused on racism in this paper, they recognize there are many forms of bias that impact medical education and intersect with racism, each with its particular history, that deserve their own telling and redress.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery.

Importance: Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education. Objective: To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees. Design, Setting, and Participants: This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities. Exposures: Longitudinal SIMPL ratings. Main Outcomes and Measures: Performance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year. Results: Using 63 248 SIMPL ratings, the association between prior and future performance was positive (ß, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated. Conclusions and Relevance: In this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination.

OBJECTIVE: While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education. DESIGN: Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: 67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback. RESULTS: Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44). CONCLUSIONS: The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

Reducing opioid misuse among adolescents through physician education.

Increased prescribing of opioids has been associated with an epidemic of nonmedical prescription opioid use in the United States; adolescents and young adults are particularly vulnerable to opioid misuse. The role of physicians as health care providers, educators, and confidants for their adolescent patients equips them to intervene in adolescent opioid misuse. The authors advocate for improving the education of physicians and residents regarding opioid use and misuse among adolescents. To achieve this, we can require residency education that includes opioid misuse and appropriate prescribing, widely disseminate existing resources on management of pain and opioid misuse, and develop pain management and addiction mentorship programs.

A previously unclassified variant of sternalis muscle.

PURPOSE: We propose an addition to the Snosek et al. classification to include a subtype variant of sternalis muscle: mixed type and triple subtype. METHODS: Dissection of the anterior thorax of a 96-year-old female cadaver revealed bilateral sternalis muscles with an undocumented variant of the right sternalis muscle. RESULTS: The left sternalis muscle presented as a simple type-left single using the Snosek et al. classification scheme. The right sternalis muscle revealed a previously undocumented classification type. It consisted of three bellies and two heads, with the lateral head formed by two converging bellies and the medial head formed from the superficial medial belly. CONCLUSIONS: The unique presentation of right sternalis muscle can be classified by expanding the Snosek et al. classification scheme to include triple-bellied subtypes. This presentation is classified as a mixed type-right triple, with single bicipital converging and single bicipital diverging. Documentation of sternalis muscle variations can prevent misdiagnoses within the anterior thorax.