The Surgical Learning Curve: Does Robotic Technical Skill Explain Differences in Operative Performance?

Background: Prior studies on technical skills use small collections of videos for assessment. However, there is likely heterogeneity of performance among surgeons and likely improvement after training. If technical skill explains these differences, then it should vary among practicing surgeons and improve over time. Materials and Methods: Sleeve gastrectomy cases (n = 162) between July 2018 and January 2021 at one health system were included. Global evaluative assessment of robotic skills (GEARS) scores were assigned by crowdsourced evaluators. Videos were manually annotated. Analysis of variance was used to compare continuous variables between surgeons. Tamhane's post hoc test was used to define differences between surgeons with the eta-squared value for effect size. Linear regression was used for temporal changes. A P value <.05 was considered significant. Results: Variations in operative time discriminated between individuals (e.g., between 2 surgeons, means were 91 and 112 minutes, Tamhane's = 0.001). Overall, GEARS scores did not vary significantly (e.g., between those 2 surgeons, means were 20.32 and 20.6, Tamhane's = 0.151). Operative time and total GEARS score did not change over time (R2 = 0.0001-0.096). Subcomponent scores showed idiosyncratic temporal changes, although force sensitivity increased among all (R2 = 0.172-0.243). For a novice surgeon, phase-adjusted operative time (R2 = 0.24), but not overall GEARS scores (R2 = 0.04), improved over time. Conclusions: GEARS scores showed less variability and did not improve with time for a novice surgeon. Improved technical skill does not explain the learning curve of a novice surgeon or variation among surgeons. More work could define valid surrogate metrics for performance analysis.

Outcome prediction in bariatric surgery through video-based assessment.

INTRODUCTION: The relationship between intraoperative surgical performance scores and patient outcomes has not been demonstrated at a single-case level. The GEARS score is a Likert-based scale that quantifies robotic surgical proficiency in 5 domains. Given that even highly skilled surgeons can have variability in their skill among their cases, we hypothesized that at a patient level, higher surgical skill as determined by the GEARS score will predict individual patient outcomes. METHODS: Patients undergoing robotic sleeve gastrectomy between July 2018 and January 2021 at a single-health care system were captured in a prospective database. Bivariate Pearson's correlation was used to compare continuous variables, one-way ANOVA for categorical variables compared with a continuous variable, and chi-square for two categorical variables. Significant variables in the univariable screen were included in a multivariable linear regression model. Two-tailed p-value < 0.05 was considered significant. RESULTS: Of 162 patients included, 9 patients (5.5%) experienced a serious morbidity within 30 days. The average excess weight loss (EWL) was 72 ± 12% at 6 months and 74 ± 15% at 12 months. GEARS score was not significantly correlated with EWL at 6 months (p = 0.349), 12 months (p = 0.468), or serious morbidity (p = 0.848) on unadjusted analysis. After adjusting, total GEARS score was not correlated with serious morbidity (p = 0.914); however, GEARS score did predict EWL at 6 (p < 0.001) and 12 months (p < 0.001). All GEARS subcomponent scores, bimanual dexterity, depth perception, efficiency, force sensitivity, and robotic control were predictive of EWL at 6 months (p < 0.001) and 12 months (p < 0.001) on multivariable analysis. CONCLUSION: For patients undergoing sleeve gastrectomy, surgical skill as assessed by the GEARS score was correlated with EWL, suggesting that better performance of a sleeve gastrectomy can result in improved postoperative weight loss.

Objective assessment of robotic surgical skills: review of literature and future directions.

BACKGROUND: Evaluation of robotic surgical skill has become increasingly important as robotic approaches to common surgeries become more widely utilized. However, evaluation of these currently lacks standardization. In this paper, we aimed to review the literature on robotic surgical skill evaluation. METHODS: A review of literature on robotic surgical skill evaluation was performed and representative literature presented over the past ten years. RESULTS: The study of reliability and validity in robotic surgical evaluation shows two main assessment categories: manual and automatic. Manual assessments have been shown to be valid but typically are time consuming and costly. Automatic evaluation and simulation are similarly valid and simpler to implement. Initial reports on evaluation of skill using artificial intelligence platforms show validity. Few data on evaluation methods of surgical skill connect directly to patient outcomes. CONCLUSION: As evaluation in surgery begins to incorporate robotic skills, a simultaneous shift from manual to automatic evaluation may occur given the ease of implementation of these technologies. Robotic platforms offer the unique benefit of providing more objective data streams including kinematic data which allows for precise instrument tracking in the operative field. Such data streams will likely incrementally be implemented in performance evaluations. Similarly, with advances in artificial intelligence, machine evaluation of human technical skill will likely form the next wave of surgical evaluation.

Blinded intraoperative skill evaluations avoid gender-based bias.

INTRODUCTION: Gender bias has been identified consistently in written performance evaluations. Qualitative tools may provide a standardized way to evaluate surgical skill and minimize gender bias. We hypothesized that there is no difference in operative time or GEARS scores in robotic hysterectomy for men vs women surgeons. METHODS: Patients undergoing robotic hysterectomies performed between June 2019 and March 2020 at 8 hospitals within the same hospital system were captured into a prospective database. GEARS scores were assigned by crowd-sourced evaluators by a third party blinded to any surgeon- or patient-identifying information. One-way ANOVA was used to compare the mean operative time and GEARS scores for each group, and significant variables were included in a one-way ANCOVA to control for confounders. Two-tailed p-value < 0.05 was considered significant. RESULTS: Seventeen women and 13 men performed a total of 188 hysterectomies; women performed 34 (18%) and men performed 153 (81%). Women surgeons had a higher mean operative time (133 ± 58 vs 86.3 ± 46 min, p = 0.024); after adjustment, there were no significant differences in operative time (p = 0.607). There was no significant difference between the genders in total GEARS score (20.0 ± 0.77 vs 20.2 ± 0.70, p = 0.415) or GEARS subcomponent scores: bimanual dexterity (3.98 ± 0.03 vs 4.00 ± 0.03, p = 0.705); depth perception (4.04 ± 0.04 vs 4.05 ± 0.02, p = 0.799); efficiency (3.79 ± 0.02 vs 3.82 ± 0.02, p = 0.437); force sensitivity (4.01 ± 0.04 vs 4.05 ± 0.05, p = 0.533); or robotic control (4.16 ± 0.03 vs 4.26 ± 0.01, p = 0.079). CONCLUSION: There was no difference in GEARS score between men vs women surgeons performing robotic hysterectomies. Video-based blinded assessment of skills may minimize gender biases when evaluating surgical skill for competency evaluation and credentialing.