How Reliable are Single-Question Workplace-Based Assessments in Surgery?

OBJECTIVE: Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking. This study aimed to evaluate the inter-rater reliability of multiple operative WBA tools utilized in general surgery residency. DESIGN: General surgery residents and teaching faculty were recorded during 6 general surgery operations. Nine faculty raters each reviewed 6 videos and rated each resident on performance (using the Society for Improving Medical Professional Learning, or SIMPL, Performance Scale as well as the operative performance rating system (OPRS) Scale), entrustment (using the ten Cate Entrustment-Supervision Scale), and autonomy (using the Zwisch Scale). The ratings were reviewed for inter-rater reliability using percent agreement and intraclass correlations. PARTICIPANTS: Nine faculty members viewed the videos and assigned ratings for multiple WBAs. RESULTS: Absolute intraclass correlation coefficients for each scale ranged from 0.33 to 0.47. CONCLUSIONS: All single-item WBA scales had low to moderate inter-rater reliability. While rater training may improve inter-rater reliability for single observations, many observations by many raters are needed to reliably assess trainee performance in the workplace.

Characteristics and Practice Patterns of Non-certified Surgeons Treating Medicare Patients.

OBJECTIVE: The objective of this study is to explore the patient characteristics and practice patterns of non-certified surgeons who treat Medicare patients in the United States. SUMMARY BACKGROUND DATA: While most surgeons in the United States are board-certified, non-certified surgeons are permitted to practice in many locations. At the same time, surgical workforce shortages threaten access to surgical care for many patients. It is possible that non-certified surgeons may be able to help fill these access gaps. However, little is known about the practice patterns of non-certified surgeons. METHODS: A 100% sample of Medicare claims data from 2014-2019 were used to identify practicing general surgeons. Surgeons were categorized as certified or non-certified in general surgery​​ based on data from the American Board of Surgery. Surgeon practice patterns and patient characteristics were analyzed. RESULTS: A total of 2,097,206 patient cases were included in the study. These patients were treated by 16,076 surgeons, of which 6% were identified as non-certified surgeons. Compared to certified surgeons, non-certified surgeons were less frequently fellowship-trained (20.5% vs. 24.2%, P=0.008) and more likely to be a foreign medical graduate (14.5% vs. 9.2%, P<0.001). Non-certified surgeons were more frequently practicing in for-profit hospitals (21.2% vs. 14.2%, P<0.001) and critical access hospitals (2.2% vs. 1.3%, P<0.001), and were less likely to practice in a teaching hospital (63.2% vs. 72.4%, P<0.001). Compared to certified surgeons, non-certified surgeons treated more non-White patients (19.6% vs. 14%, P<0.001) as well as a higher percentage of patients in the two lowest socioeconomic status (SES) quintiles (36.2% vs. 29.2%, P<0.001). Operations related to emergency admissions were more common amongst non-certified surgeons (68.8% vs. 55.7%, P<0.001). There were no differences in gender or age of the patients treated by certified and non-certified surgeons. CONCLUSION: For Medicare patients, non-certified surgeons treated more patients who are non-White, of lower SES, and in more rural, critical-access hospitals.

Variation in Competence of Graduating General Surgery Trainees.

OBJECTIVE: To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training. DESIGN: Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021. RESULTS: The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy. CONCLUSIONS: There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Mortality and Severe Complications Among Newly Graduated Surgeons in the United States.

OBJECTIVE: To evaluate severe complications and mortality over years of independent practice among general surgeons. BACKGROUND: Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons. METHODS: We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery. Generalized additive mixed models were used to assess the association between surgeon years in practice and 30-day outcomes while adjusting for differences in patient, hospital, and surgeon characteristics. RESULTS: The cohort included 1,329,358 operations performed by 14,399 surgeons. In generalized mixed models, the relative risk (RR) of mortality was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [5.5% (95% CI: 4.1%-7.3%) vs 4.7% (95% CI: 3.5%-6.3%), RR: 1.17 (95% CI: 1.11-1.22)]. Similarly, the RR of severe complications was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [7.5% (95% CI: 6.6%-8.5%) versus 6.9% (95% CI: 6.1%-7.9%), RR: 1.08 (95% CI: 1.03-1.14)]. When stratified by individual operation, 21 operations had a significantly higher RR of mortality and all 26 operations had a significantly higher RR of severe complications in the first compared with the 15th year of practice. CONCLUSIONS: Among general surgeons performing common operations, rates of mortality and severe complications were higher among newly graduated surgeons compared with later career surgeons.

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps.

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program. Given the importance of assessment in CBME, it is critical that we build a better assessment system for measuring trainee competency. We propose that an ideal system of assessment is standardized, evidence-based, comprehensive, integrated, and continuously improving. In this article, we explore these characteristics and propose next steps to achieve such a system of assessment in general surgery.

Implementation of a Smartphone-Based Platform for Operative Feedback at Tenwek Hospital in Kenya: A Mixed Methods Study.

BACKGROUND: The SIMPL operative feedback tool is used in many U.S. surgical residency programs. However, the challenges of implementation and benefits of the web-based platform in low- and middle-income countries are unknown. The aim of this study was to evaluate implementation of SIMPL in a general surgery residency training program in Kenya. METHODS: SIMPL was pilot tested at Tenwek Hospital from January through December 2021. Participant perspectives of SIMPL were elicited through a survey and semi-structured interviews. Descriptive statistics were used to analyze survey data. Inductive qualitative content analysis of interview responses was performed by two independent researchers. RESULTS: Fourteen residents and six faculty (100% response rate) were included in the study and completed over 600 operative assessments. All respondents reported numerical evaluations and dictated feedback were useful. Respondents felt that SIMPL was easy to use, improved quality and frequency of feedback, helped refine surgical skills, and increased resident autonomy. Barriers to use included participants forgetting to complete evaluations, junior residents not submitting evaluations when minimally involved in cases, and technological challenges. Suggestions for improvement included expansion of SIMPL to surgical subspecialties and allowing senior residents to provide feedback to juniors. All respondents wanted to continue using SIMPL, and 90% recommended use at other programs. CONCLUSION: Residents and faculty at Tenwek Hospital believed SIMPL were a positive addition to their training program. There were a few barriers to use and suggestions for improvement specific to the training environment in Kenya, but this study demonstrates it is feasible to use SIMPL in settings outside the U.S. with the appropriate resources.

Thriving as a Surgical Intern: Three Tips From the Collaboration of Surgical Education Fellows (CoSEF).

We are the multi-institutional organization known as the Collaboration of Surgical Education Fellows (CoSEF). We've collectively reflected on our range of experiences across the country and identified 3 principles which promote a successful intern experience: (1) Own your patients; (2) Treat people like people; and (3) Take care of yourself.

Association of Gender and Operative Feedback Quality in Surgical Residents.

OBJECTIVE: Feedback is critical for learning, however, gender differences exist in the quality of feedback that trainees receive. For example, narrative feedback on surgical trainees' end-of-block rotations differs based on trainee-faculty gender dyads, with female faculty giving higher quality feedback and male trainees receiving higher quality feedback. Though this represents evidence of gender bias in global evaluations, there is limited understanding of how much bias might be present in operative workplace-based assessments (WBAs). In this study, we explore the quality of narrative feedback among trainee-faculty gender dyads in an operative WBA. DESIGN: A previously validated natural language processing model was used to examine instances of narrative feedback and assign a probability of being characterized as high quality feedback (defined as feedback which was relevant as well as corrective and/or specific). A linear mixed model was performed, with probability of high quality feedback as the outcome, and resident gender, faculty gender, PGY, case complexity, autonomy rating, and operative performance rating as explanatory variables. PARTICIPANTS: Analyses included 67,434 SIMPL operative performance evaluations (2,319 general surgery residents, 70 institutions) collected from September 2015 through September 2021. RESULTS: Of 36.3% evaluations included narrative feedback. Male faculty were more likely to provide narrative feedback compared to female faculty. Mean probabilities of receiving high quality feedback ranged from 81.6 (female faculty-male resident) to 84.7 (male faculty-female resident). Model-based results demonstrated that female residents were more likely to receive high quality feedback (p < 0.01), however, there was no significant difference in probability of high quality narrative feedback based on faculty-resident gender dyad (p = 0.77). CONCLUSIONS: Our study revealed resident gender differences in the probability of receiving high-quality narrative feedback following a general surgery operation. However, we found no significant differences based on faculty-resident gender dyad. Male faculty were more likely to provide narrative feedback compared to their female colleagues. Further research using general surgery resident-specific feedback quality models may be warranted.

The Demands of Surgery Residency: More Than Just Duty Hours?

INTRODUCTION: Efforts to improve surgical resident well-being could be accelerated with an improved understanding of resident job demands and resources. In this study, we sought to obtain a clearer picture of surgery resident job demands by assessing how residents distribute their time both inside and outside of the hospital. Furthermore, we aimed to elucidate residents' perceptions about current duty hour regulations. METHODS: A cross-sectional survey was sent to 1098 surgical residents at 27 US programs. Responses regarding work hours, demographics, well-being (utilizing the physician well-being index), and perceptions of duty hours in relation to education and rest, were collected. Data were evaluated using descriptive statistics and content analysis. RESULTS: A total of 163 residents (14.8% response rate) were included in the study. Residents reported a median total patient care hours per week of 78.0 h. Trainees spent 12.5 h on other professional activities. Greater than 40% of residents were "at risk" for depression and suicide based on physician well-being index scores. Four major themes associated with education and rest were identified: 1) duty hour definitions and reporting mechanisms do not completely reflect the amount of work residents perform, 2) quality patient care and educational opportunities do not fit neatly within the duty hour framework, 3) resident perceptions of duty hours are impacted the educational environment, and 4) long work hours and lack of adequate rest negatively affect well-being. CONCLUSIONS: The breadth and depth of trainee job demands are not accurately captured by current duty hour reporting mechanisms, and residents do not believe that their current work hours allow for adequate rest or even completion of other clinical or academic tasks outside of the hospital. Many residents are unwell. Duty hour policies and resident well-being may be improved with a more holistic accounting of resident job demands and greater attention to the resources that residents have to offset those demands.

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

OBJECTIVE: Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment. DESIGN: The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures. SETTING: This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative. PARTICIPANTS: A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded. RESULTS: The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures. CONCLUSIONS: Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.

Efficacy and Safety of Anti-Xa-Guided Versus Fixed Dosing of Low Molecular Weight Heparin for Prevention of Venous Thromboembolism in Trauma Patients: A Systematic Review and Meta-Analysis.

PURPOSE: Trauma patients are at high risk of venous thromboembolism (VTE). We summarize the comparative efficacy and safety of anti-Xa-guided versus fixed dosing for low molecular weight heparin (LMWH) for the prevention of VTE in adult trauma patients. METHODS: We searched Medline and Embase from inception through June 1, 2022. We included randomized controlled trials or observational studies comparing anti-Xa-guided versus fixed dosing of LMWH for thromboprophylaxis in adult trauma patients. We incorporated primary data from 2 large observational cohorts. We pooled effect estimates using a random-effects model. We assessed risk of bias using the ROBINS-I tool for observational studies and assessed certainty of findings using GRADE methodology. RESULTS: We included 15 observational studies involving 10,348 patients. No randomized controlled trials were identified. determined that, compared to fixed LMWH dosing, anti-Xa-guided dosing may reduce deep vein thrombosis [adjusted odds ratio (aOR); 0.52, 95% CI: 0.40-0.69], pulmonary embolism (aOR: 0.48, 95% CI: 0.30-0.78) or any VTE (aOR: 0.54, 95% CI: 0.42-0.69), though all estimates are based on low certainty evidence. There was an uncertain effect on mortality (aOR: 1.06, 95% CI: 0.85-1.32) and bleeding events (aOR: 0.84, 95% CI: 0.50-1.39), limited by serious imprecision. We used several sensitivity and subgroup analyses to confirm the validity of our assumptions. CONCLUSION: Anti-Xa-guided dosing may be more effective than fixed dosing for prevention of deep vein thrombosis, pulmonary embolism, and VTE for adult trauma patients. These promising findings justify the need for a high-quality randomized study with the potential to deliver practice changing results.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination.

OBJECTIVE: While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education. DESIGN: Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL. SETTING: This study was conducted using data from members of the SIMPL collaborative. PARTICIPANTS: 67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback. RESULTS: Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44). CONCLUSIONS: The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

Enoxaparin titrated by anti-Xa levels reduces venous thromboembolism in trauma patients.

BACKGROUND: Trauma is a major risk factor for the development of a venous thromboembolism (VTE). After observing higher than expected VTE rates within our center's Trauma Quality Improvement Program data, we instituted a change in our VTE prophylaxis protocol, moving to enoxaparin dosing titrated by anti-Xa levels. We hypothesized that this intervention would lower our symptomatic VTE rates. METHODS: Adult trauma patients at a single institution meeting National Trauma Data Standard criteria from April 2015 to September 2019 were examined with regards to VTE chemoprophylaxis regimen and VTE incidence. Two groups of patients were identified based on VTE protocol-those who received enoxaparin 30 mg twice daily without routine anti-Xa levels ("pre") versus those who received enoxaparin 40 mg twice daily with dose titrated by serial anti-Xa levels ("post"). Univariate and multivariate analyses were performed to define statistically significant differences in VTE incidence between the two cohorts. RESULTS: There were 1698 patients within the "pre" group and 1406 patients within the "post" group. The two groups were essentially the same in terms of demographics and risk factors for bleeding or thrombosis. There was a statistically significant reduction in VTE rate (p = 0.01) and deep vein thrombosis rate (p = 0.01) but no significant reduction in pulmonary embolism rate (p = 0.21) after implementation of the anti-Xa titration protocol. Risk-adjusted Trauma Quality Improvement Program data showed an improvement in rate of symptomatic pulmonary embolism from fifth decile to first decile. CONCLUSION: A protocol titrating prophylactic enoxaparin dose based on anti-Xa levels reduced VTE rates. Implementation of this type of protocol requires diligence from the physician and pharmacist team. Further research will investigate the impact of protocol compliance and time to appropriate anti-Xa level on incidence of VTE. LEVEL OF EVIDENCE: Therapeutic/care management, Level IV.

Job satisfaction and workplace stressors among surgical providers at a single institution.

OBJECTIVE: Job satisfaction and work stress are associated with provider health and patient outcomes. This study aimed to evaluate job satisfaction and workplace stressors in surgical providers (surgeons, physician assistants [PAs], and NPs). METHODS: A survey was distributed to providers within a single surgical department. Job satisfaction and workplace stressors were evaluated by sex, age, profession, career length, and work hours. RESULTS: Providers practicing for 11 to 15 years had greater job satisfaction than those practicing for more than 20 years, with no other differences by demographic group. Females cited supervisory support as a top workplace stressor more than did males (P = .01) and PAs and NPs cited supervisory support (P < .01) and compensation/finances more than surgeons (P = .05). Workplace stressors varied by practice years and work hours. CONCLUSIONS: Healthcare organizations should be aware of diversity in perceived workplace stressors. A "one size fits all" approach to provider well-being is likely to be ineffective.

Evaluating the Burnout-Thriving Index in a Multidisciplinary Cohort at a Large Academic Medical Center.

OBJECTIVES: There has been significant discussion about the quality of burnout research, especially with regard to abbreviated measurements of burnout and/or well-being. The purpose of this study was to compare a single-item, investigator-developed question measuring perceived well-being with validated multi-item measures of burnout and well-being. METHODS: Between 2016 and 2017, healthcare professionals and medical students at a large academic hospital system were sent an online survey measuring the risk of burnout (Maslach Burnout Inventory), well-being (Physician or Nurse Well-Being Self-Assessment Tool), and perception of personal well-being (Burnout-Thriving Index [BTI], an investigator-developed, single-item measure). Analyses included linear and multiple regression and Pearson correlations. RESULTS: The study sample included 1365 medical students, frontline nurses, resident physicians, supervising physicians or fellows, and advanced care practitioners. There were significant differences in all Maslach Burnout Inventory and Physician or Nurse Well-Being Self-Assessment Tool subscale scores based on BTI score (all P < 0.001). Adjusted R2 ranged from 0.066 (religiospiritual wellness) to 0.343 (emotional exhaustion). BTI had a stronger relation with personal accomplishment in medical students compared with nurses (P = 0.049) and a stronger relation with psychoemotional wellness in physicians and physicians-in-training compared with nurses (P < 0.05). A low BTI score demonstrated >80% sensitivity for high emotional exhaustion, depersonalization, and low personal accomplishment. CONCLUSIONS: The BTI may be used to screen for individuals who could benefit from completing a full burnout assessment and may be used to collect a quick "big picture" impression of burnout and well-being at a healthcare institution. Further research is needed to compare BTI score with known consequences of burnout and to explore differences in the relation between BTI score and psychoemotional wellness in different professional groups.

Gold amides as anticancer drugs: synthesis and activity studies.

Modular gold amide chemotherapeutics: Access to modern chemotherapeutics with robust and flexible synthetic routes that are amenable to extensive customisation is a key requirement in drug synthesis and discovery. A class of chiral gold amide complexes featuring amino acid derived ligands is reported herein. They all exhibit in vitro cytotoxicity against two slow growing breast cancer cell lines with limited toxicity towards normal epithelial cells.