How Reliable are Single-Question Workplace-Based Assessments in Surgery?

OBJECTIVE: Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking. This study aimed to evaluate the inter-rater reliability of multiple operative WBA tools utilized in general surgery residency. DESIGN: General surgery residents and teaching faculty were recorded during 6 general surgery operations. Nine faculty raters each reviewed 6 videos and rated each resident on performance (using the Society for Improving Medical Professional Learning, or SIMPL, Performance Scale as well as the operative performance rating system (OPRS) Scale), entrustment (using the ten Cate Entrustment-Supervision Scale), and autonomy (using the Zwisch Scale). The ratings were reviewed for inter-rater reliability using percent agreement and intraclass correlations. PARTICIPANTS: Nine faculty members viewed the videos and assigned ratings for multiple WBAs. RESULTS: Absolute intraclass correlation coefficients for each scale ranged from 0.33 to 0.47. CONCLUSIONS: All single-item WBA scales had low to moderate inter-rater reliability. While rater training may improve inter-rater reliability for single observations, many observations by many raters are needed to reliably assess trainee performance in the workplace.

Development and validation of a multicenter Cox regression model to predict all-cause mortality in patients with renal masses suspicious for renal cancer.

OBJECTIVE: Life expectancy models are useful tools to support clinical decision-making. Prior models have not been used widely in clinical practice for patients with renal masses. We sought to develop and validate a model to predict life expectancy following the detection of a localized renal mass suspicious for renal cell carcinoma. MATERIALS AND METHODS: Using retrospective data from 2 large centers, we identified patients diagnosed with clinically localized renal parenchymal masses from 1998 to 2018. After 2:1 random sampling into a derivation and validation cohort stratified by site, we used age, sex, log-transformed tumor size, simplified cardiovascular index and planned treatment to fit a Cox regression model to predict all-cause mortality from the time of diagnosis. The model's discrimination was evaluated using a C-statistic, and calibration was evaluated visually at 1, 5, and 10 years. RESULTS: We identified 2,667 patients (1,386 at Corewell Health and 1,281 at Johns Hopkins) with renal masses. Of these, 420 (16%) died with a median follow-up of 5.2 years (interquartile range 2.2-8.3). Statistically significant predictors in the multivariable Cox regression model were age (hazard ratio [HR] 1.04; 95% confidence interval [CI] 1.03-1.05); male sex (HR 1.40; 95% CI 1.08-1.81); log-transformed tumor size (HR 1.71; 95% CI 1.30-2.24); cardiovascular index (HR 1.48; 95% CI 1.32-1.67), and planned treatment (HR: 0.10, 95% CI: 0.06-0.18 for kidney-sparing intervention and HR: 0.20, 95% CI: 0.11-0.35 for radical nephrectomy vs. no intervention). The model achieved a C-statistic of 0.74 in the derivation cohort and 0.73 in the validation cohort. The model was well-calibrated at 1, 5, and 10 years of follow-up. CONCLUSIONS: For patients with localized renal masses, accurate determination of life expectancy is essential for decision-making regarding intervention vs. active surveillance as a primary treatment modality. We have made available a simple tool for this purpose.

Characteristics and Practice Patterns of Non-certified Surgeons Treating Medicare Patients.

OBJECTIVE: The objective of this study is to explore the patient characteristics and practice patterns of non-certified surgeons who treat Medicare patients in the United States. SUMMARY BACKGROUND DATA: While most surgeons in the United States are board-certified, non-certified surgeons are permitted to practice in many locations. At the same time, surgical workforce shortages threaten access to surgical care for many patients. It is possible that non-certified surgeons may be able to help fill these access gaps. However, little is known about the practice patterns of non-certified surgeons. METHODS: A 100% sample of Medicare claims data from 2014-2019 were used to identify practicing general surgeons. Surgeons were categorized as certified or non-certified in general surgery​​ based on data from the American Board of Surgery. Surgeon practice patterns and patient characteristics were analyzed. RESULTS: A total of 2,097,206 patient cases were included in the study. These patients were treated by 16,076 surgeons, of which 6% were identified as non-certified surgeons. Compared to certified surgeons, non-certified surgeons were less frequently fellowship-trained (20.5% vs. 24.2%, P=0.008) and more likely to be a foreign medical graduate (14.5% vs. 9.2%, P<0.001). Non-certified surgeons were more frequently practicing in for-profit hospitals (21.2% vs. 14.2%, P<0.001) and critical access hospitals (2.2% vs. 1.3%, P<0.001), and were less likely to practice in a teaching hospital (63.2% vs. 72.4%, P<0.001). Compared to certified surgeons, non-certified surgeons treated more non-White patients (19.6% vs. 14%, P<0.001) as well as a higher percentage of patients in the two lowest socioeconomic status (SES) quintiles (36.2% vs. 29.2%, P<0.001). Operations related to emergency admissions were more common amongst non-certified surgeons (68.8% vs. 55.7%, P<0.001). There were no differences in gender or age of the patients treated by certified and non-certified surgeons. CONCLUSION: For Medicare patients, non-certified surgeons treated more patients who are non-White, of lower SES, and in more rural, critical-access hospitals.

Simplified cardiovascular index may be the best comorbidity index for clinical use in prediction of mortality for renal cancer patients.

OBJECTIVE: Understanding the relationship between comorbidities and life expectancy is important in cancer patients who carry risks of cancer and noncancer-related mortality. Comorbidity indices (CI) are tools to provide an objective measure of competing risks of death. We sought to determine which CI might be best incorporated into clinical practice for patients with suspected renal cancer. MATERIALS AND METHODS: 1572 patients diagnosed with renal masses (stage I-IV) between 1998 and 2016 were analyzed for this study. Patient data were gathered from a community-based health center. Comorbidities were evaluated individually, and with 1 of 4 CI: Charlson (CCI), updated CCI (uCCI), age-adjusted CCI (aCCI), and simplified cardiovascular index (CVI). Cox-proportional hazard analysis of all-cause mortality was performed using the four CI, adjusting for the 4 CI, adjusting for age, gender, race, tumor size, and tumor stage. RESULTS: Univariable analyses revealed the four CI were significant predictors of mortality (P < 0.05), as were age, gender, tumor size, and stage. Comorbid conditions at diagnosis included hypertension (47.8%), diabetes mellitus (47.2%), coronary artery disease (41.1%), chronic kidney disease (31.8%), peripheral vascular disease (8.0%), congestive heart failure (5.7%), chronic obstructive pulmonary disease (5.7%), and cerebrovascular disease (2.0%). When analyzing the 4 CI in multivariable survival analyses accounting for factors available at diagnosis, and analyses incorporating pathologic and recurrence data, only CVI score and uCCI remained statistically significant (P < 0.05). Limitations of this work are the retrospective nature of data collection and data from a single institution, limiting the generalizability. CONCLUSION: Increasing comorbidity, age, tumor size, and cM stage are predictors of ACM for suspected renal cancer patients. CVI appears to provide comparable information to various iterations of CCI (uCCI, aCCI) while being the simplest to use. Utilization of CVI may assist clinicians and patients when considering between interventional and noninterventional approaches for suspected renal cancer.

How Many Operative Performance Ratings Does a Pediatric Surgery Fellow Need to Be Deemed Practice Ready?

PURPOSE: Identifying the number of cases required for a fellow to achieve competence has been challenging. Workplace-based assessment (WBA) systems make collecting performance data practical and create the opportunity to translate WBA ratings into probabilistic statements about a fellow's likelihood of performing to a given standard on a subsequent assessment opportunity. METHODS: We compared data from two pediatric surgery training programs that used the performance rating scale from the Society for Improving Medical Professional Learning (SIMPL). We used a Bayesian generalized linear mixed effects model to examine the relationship past and future performance for three procedures: Laparoscopic Inguinal Hernia Repair, Laparoscopic Gastrostomy Tube Placement, and Pyloromyotomy. RESULTS: For site one, 26 faculty assessed 9 fellows on 16 procedures yielding 1094 ratings, of which 778 (71%) earned practice-ready ratings. For site two, 25 faculty rated 3 fellows on 4 unique procedures yielding 234 ratings of which 151 (65%) were deemed practice-ready. We identified similar model-based future performance expectations, with prior practice-ready ratings having a similar average effect across both sites (Site one, B = 0.25; Site two, B = 0.25). Similar prior practice-ready ratings were needed for Laparoscopic G-Tube Placement (Site one = 13; Site two = 14), while greater differences were observed for Laparoscopic Inguinal Hernia Repair (Site one = 10; Site two = 15) and Pyloromyotomy (Site one = 10; Site two = 15). CONCLUSION: Our approach to modeling operative performance data is effective at determining future practice readiness of pediatric surgery fellows across multiple faculty and fellow groups. This method could be used to establish minimum case number requirements. TYPE OF STUDY: Original manuscript, Study of Diagnostic Test. LEVEL OF EVIDENCE: II.

Variation in Competence of Graduating General Surgery Trainees.

OBJECTIVE: To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training. DESIGN: Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021. RESULTS: The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy. CONCLUSIONS: There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Mortality and Severe Complications Among Newly Graduated Surgeons in the United States.

OBJECTIVE: To evaluate severe complications and mortality over years of independent practice among general surgeons. BACKGROUND: Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons. METHODS: We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery. Generalized additive mixed models were used to assess the association between surgeon years in practice and 30-day outcomes while adjusting for differences in patient, hospital, and surgeon characteristics. RESULTS: The cohort included 1,329,358 operations performed by 14,399 surgeons. In generalized mixed models, the relative risk (RR) of mortality was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [5.5% (95% CI: 4.1%-7.3%) vs 4.7% (95% CI: 3.5%-6.3%), RR: 1.17 (95% CI: 1.11-1.22)]. Similarly, the RR of severe complications was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [7.5% (95% CI: 6.6%-8.5%) versus 6.9% (95% CI: 6.1%-7.9%), RR: 1.08 (95% CI: 1.03-1.14)]. When stratified by individual operation, 21 operations had a significantly higher RR of mortality and all 26 operations had a significantly higher RR of severe complications in the first compared with the 15th year of practice. CONCLUSIONS: Among general surgeons performing common operations, rates of mortality and severe complications were higher among newly graduated surgeons compared with later career surgeons.

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps.

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.

Considering the Secondary Use of Clinical and Educational Data to Facilitate the Development of Artificial Intelligence Models.

ABSTRACT: Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives. More recently, however, these clinical and educational data are also increasingly being used to train artificial intelligence (AI) models. The implications of this relatively unique secondary use of data have not been well explored. These models can support the development of sophisticated AI products that can be commercialized. While these products have the potential to support and improve the educational system, there are challenges related to validity, patient and learner consent, and biased or discriminatory outputs. The authors consider the implications of developing AI models and products using educational and clinical data from learners, discuss the uses of these products within medical education, and outline considerations that should guide the appropriate use of data for this purpose. These issues are further explored by examining how they have been navigated in an educational collaborative.

Evaluating Educational Outcomes Using Patient Outcomes of New Surgeons Performing Partial Colectomy Compared to Cholecystectomy.

PURPOSE: Despite ongoing efforts to improve surgical education, surgical residents face gaps in their training. However, it is unknown if differences in the training of surgeons are reflected in the patient outcomes of those surgeons once they enter practice. This study aimed to compare the patient outcomes among new surgeons performing partial colectomy-a common procedure for which training is limited-and cholecystectomy-a common procedure for which training is robust. METHOD: The authors retrospectively analyzed all adult Medicare claims data for patients undergoing inpatient partial colectomy and inpatient cholecystectomy between 2007 and 2018. Generalized additive mixed models were used to investigate the associations between surgeon years in practice and risk-adjusted rates of 30-day serious complications and death for patients undergoing partial colectomy and cholecystectomy. RESULTS: A total of 14,449 surgeons at 4,011 hospitals performed 340,114 partial colectomy and 355,923 cholecystectomy inpatient operations during the study period. Patients undergoing a partial colectomy by a surgeon in their 1st vs 15th year of practice had higher rates of serious complications (5.22% [95% CI, 4.85%-5.60%] vs 4.37% [95% CI, 4.22%-4.52%]; P < .01) and death (3.05% [95% CI, 2.92%-3.17%] vs 2.83% [95% CI, 2.75%-2.91%]; P < .01). Patients undergoing a cholecystectomy by a surgeon in their 1st vs 15th year of practice had similar rates of 30-day serious complications (4.11% vs 3.89%; P = .11) and death (1.71% vs 1.70%; P = .93). CONCLUSIONS: Patients undergoing partial colectomy faced a higher risk of serious complications and death when the operation was performed by a new surgeon compared to an experienced surgeon. Conversely, patient outcomes following cholecystectomy were similar for new and experienced surgeons. More attention to partial colectomy during residency training may benefit patients.

Association of Gender and Operative Feedback Quality in Surgical Residents.

OBJECTIVE: Feedback is critical for learning, however, gender differences exist in the quality of feedback that trainees receive. For example, narrative feedback on surgical trainees' end-of-block rotations differs based on trainee-faculty gender dyads, with female faculty giving higher quality feedback and male trainees receiving higher quality feedback. Though this represents evidence of gender bias in global evaluations, there is limited understanding of how much bias might be present in operative workplace-based assessments (WBAs). In this study, we explore the quality of narrative feedback among trainee-faculty gender dyads in an operative WBA. DESIGN: A previously validated natural language processing model was used to examine instances of narrative feedback and assign a probability of being characterized as high quality feedback (defined as feedback which was relevant as well as corrective and/or specific). A linear mixed model was performed, with probability of high quality feedback as the outcome, and resident gender, faculty gender, PGY, case complexity, autonomy rating, and operative performance rating as explanatory variables. PARTICIPANTS: Analyses included 67,434 SIMPL operative performance evaluations (2,319 general surgery residents, 70 institutions) collected from September 2015 through September 2021. RESULTS: Of 36.3% evaluations included narrative feedback. Male faculty were more likely to provide narrative feedback compared to female faculty. Mean probabilities of receiving high quality feedback ranged from 81.6 (female faculty-male resident) to 84.7 (male faculty-female resident). Model-based results demonstrated that female residents were more likely to receive high quality feedback (p < 0.01), however, there was no significant difference in probability of high quality narrative feedback based on faculty-resident gender dyad (p = 0.77). CONCLUSIONS: Our study revealed resident gender differences in the probability of receiving high-quality narrative feedback following a general surgery operation. However, we found no significant differences based on faculty-resident gender dyad. Male faculty were more likely to provide narrative feedback compared to their female colleagues. Further research using general surgery resident-specific feedback quality models may be warranted.

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

OBJECTIVE: Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment. DESIGN: The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures. SETTING: This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative. PARTICIPANTS: A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded. RESULTS: The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures. CONCLUSIONS: Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.

Development and Validation of a Model to Predict Ureteral Stent Placement Following Ureteroscopy: Results From a Statewide Collaborative.

OBJECTIVE: To develop and validate a model to predict whether patients undergoing ureteroscopy (URS) will receive a stent. METHODS: Using registry data obtained from the Michigan Urological Surgery Improvement Collaborative Reducing Operative Complications from Kidney Stones initiative, we identified patients undergoing URS from 2016 to 2020. We used patients' age, sex, body mass index, size and location of the largest stone, current stent in place, history of any kidney stone procedure, procedure type, and acuity to fit a multivariable logistic regression model to a derivation cohort consisting of a random two-thirds of episodes. Model discrimination and calibration were evaluated in the validation cohort. A sensitivity analysis examined urologist variation using generalized mixed-effect models. RESULTS: We identified 15,048 URS procedures, of which 11,471 (76%) had ureteral stents placed. Older age, male sex, larger stone size, the largest stone being in the ureteropelvic junction, no prior stone surgery, no stent in place, a planned procedure type of laser lithotripsy, and urgent procedure were associated with a higher risk of stent placement. The model achieved an area under the receiver operating characteristic curve of 0.69 (95% CI 0.67, 0.71). Incorporating urologist-level variation improved the area under the receiver operating characteristic curve to 0.83 (95% CI 0.82, 0.84). CONCLUSION: Using a large clinical registry, we developed a multivariable regression model to predict ureteral stent placement following URS. Though well-calibrated, the model had modest discrimination due to heterogeneity in practice patterns in stent placement across urologists.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery.

Importance: Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education. Objective: To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees. Design, Setting, and Participants: This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities. Exposures: Longitudinal SIMPL ratings. Main Outcomes and Measures: Performance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year. Results: Using 63 248 SIMPL ratings, the association between prior and future performance was positive (ß, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated. Conclusions and Relevance: In this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

Predicting Resident Competence for Otolaryngology Key Indicator Procedures.

OBJECTIVE: Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs. METHODS: From November 2017 to September 2021, a smartphone application (SIMPL OR) was used by attendings at five otolaryngology training programs to rate resident operative performance after each case on a five-level scale. Bayesian mixed effects models were used to estimate the probability that postgraduate year (PGY) 3, 4, or 5 trainees would earn a "practice-ready" (PR) rating on a subsequent evaluation based on their previously earned PR ratings for each KIP. Probabilities of earning a subsequent PR rating were examined for interpretability, and cross-validation was used to assess predictive validity. RESULTS: A total of 842 assessments of KIPs were submitted by 72 attendings for 92 residents PGY 2-5. The predictive model had an average Area Under the Receiver Operating Curve of 0.77. The number of prior PR ratings that senior residents needed to attain a 95% probability of earning a PR rating on a subsequent evaluation was estimated for each KIP. For example, for mastoidectomies, PGY4 residents needed to earn 10 PR ratings whereas PGY5 residents needed 4 PR ratings on average to have a 95% probability of attaining a PR rating on a subsequent evaluation. CONCLUSION: Predictive modeling can inform assessment benchmarks for competency-based surgical education. LEVEL OF EVIDENCE: NA Laryngoscope, 133:3341-3345, 2023.

Association of Surgical Resident Competency Ratings With Patient Outcomes.

PURPOSE: Accurate assessment of clinical performance is essential to ensure graduating residents are competent for unsupervised practice. The Accreditation Council for Graduate Medical Education milestones framework is the most widely used competency-based framework in the United States. However, the relationship between residents' milestones competency ratings and their subsequent early career clinical outcomes has not been established. It is important to examine the association between milestones competency ratings of U.S. general surgical residents and those surgeons' patient outcomes in early career practice. METHOD: A retrospective, cross-sectional study was conducted using a sample of national Medicare claims for 23 common, high-risk inpatient general surgical procedures performed between July 1, 2015, and November 30, 2018 (n = 12,400 cases) by nonfellowship-trained U.S. general surgeons. Milestone ratings collected during those surgeons' last year of residency (n = 701 residents) were compared with their risk-adjusted rates of mortality, any complication, or severe complication within 30 days of index operation during their first 2 years of practice. RESULTS: There were no associations between mean milestone competency ratings of graduating general surgery residents and their subsequent early career patient outcomes, including any complication (23% proficient vs 22% not yet proficient; relative risk [RR], 0.97, [95% CI, 0.88-1.08]); severe complication (9% vs 9%, respectively; RR, 1.01, [95% CI, 0.86-1.19]); and mortality (5% vs 5%; RR, 1.07, [95% CI, 0.88-1.30]). Secondary analyses yielded no associations between patient outcomes and milestone ratings specific to technical performance, or between patient outcomes and composites of operative performance, professionalism, or leadership milestones ratings ( P ranged .32-.97). CONCLUSIONS: Milestone ratings of graduating general surgery residents were not associated with the patient outcomes of those surgeons when they performed common, higher-risk procedures in a Medicare population. Efforts to improve how milestones ratings are generated might strengthen their association with early career outcomes.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination.

OBJECTIVE: While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education. DESIGN: Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: 67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback. RESULTS: Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44). CONCLUSIONS: The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

Representativeness of Workplace-Based Operative Performance Assessments for Resident Operative Experience.

OBJECTIVE: Workplace-based assessment is increasingly prevalent in surgical education, especially for assessing operative skill. With current implementations, not all observed clinical performances are assessed, in part because trainees often have discretion about when they seek assessment. As a result, these samples of observed operative performances may not be representative of the full breadth of experience of surgical trainees. Therefore, analyses of these samples may be biased. We aimed to benchmark patterns of procedures logged in the SIMPL operative performance assessment system against records of trainee experience in Accreditation Council for Graduate Medical Education (ACGME) case logs. DESIGN: We analyzed SIMPL longitudinal intraoperative performance assessments from categorical trainees in US general surgery residency programs. We compared overall patterns of how procedures are logged in SIMPL and in ACGME case logs using a Pearson correlation, and we examined differences in how individual procedures are logged in each system using Fisher's exact test. RESULTS: Total procedure frequency from the SIMPL dataset was strongly correlated with total procedure frequency from ACGME case logs (r = 0.86, 95% CI 0.80-0.90). A subset of these procedures (10 of 116 procedures) was logged more frequently in the SIMPL dataset. These 10 procedures accounted for 56% of SIMPL observations and 30% of ACGME logged cases. Case complexity was comparable for assessments initiated by residents and faculty. CONCLUSIONS: Samples of intraoperative performance ratings gathered using the SIMPL application largely resemble ACGME case logs. There is no evidence to indicate that residents preferentially select fewer complex cases for assessment.

Feedback-Seeking Behavior and Practice Readiness for General Surgery.

OBJECTIVE: To analyze the relationship between feedback-seeking behavior, operationalized as the number of trainee-requested evaluations, with ratings of surgical trainees' operative autonomy and performance. DESIGN: We analyzed operative assessment data using the System for Improving and Measuring Procedural Learning's smartphone-based assessment app called Society for Improving Medical Professional Learning (SIMPL) OR. Using cross-classified mixed effects models, we analyzed the association between trainee-requested SIMPL OR app evaluations and both trainee performance and autonomy ratings. Models included covariates for requested evaluations, PGY-year, month of the academic year, and patient-related case complexity. Random effects for program, procedure, rater, and trainee were also included to account for correlations among evaluations. Only ratings for procedures deemed Core to general surgery were included. SETTING: Operative assessment data using the SIMPL OR app requested by categorical U.S. general surgery residents between September 2015 to April 2021. PARTICIPANTS: A total of 61 general surgery residency programs, encompassing 2190 categorical general surgery residents. RESULTS: A total of 58,104 SIMPL app operative assessments were analyzed. Autonomy scores were weakly but positively associated with number of trainee-requested evaluations (B = 0.002, p < 0.001). Trainee-requested evaluations were also statistically associated with operative performance scores ( B = 0.002, p < 0.001). CONCLUSIONS: The propensity of a resident to seek feedback using the SIMPL app was weakly associated with higher operative autonomy ratings and higher operative performance ratings. While regular feedback is important for monitoring performance over time, more direct approaches related to the quality of feedback that trainees receive may be needed to better assess the relationships between feedback-seeking behavior and operative autonomy as well as performance.

US general surgical trainee performance for representative global surgery procedures.

BACKGROUND: Many US general surgery residents are interested in global surgery, but their competence with key procedures is unknown. METHODS: Using a registry managed by the Society for Improving Medical Professional Learning (SIMPL), we extracted longitudinal operative performance ratings data for a national cohort of US general surgery residents. Operative performance at the time of graduation was estimated via a Bayesian generalized linear mixed model. RESULTS: Operative performance ratings for 12,976 procedures performed by 1584 residents in 52 general surgery programs were analyzed. These spanned 17 of 31 (55%) procedures deemed important for global surgical practice. For these procedures, the probability of a graduating resident being deemed competent to perform a procedure was 0.95 (95% confidence interval 0.86-1.00) but was less than 0.9 for 3 observed procedures. CONCLUSION: Our results highlight gaps in the preparedness of US general surgery trainees to perform procedures deemed most important for global surgery settings.