Developing a comprehensive, proficiency-based training program for robotic surgery.

INTRODUCTION: Robotically assisted surgery has become very popular for numerous surgical disciplines, yet training practices remain variable with little to no validation. The purpose of this study was to develop a comprehensive, proficiency-based robotic training program. METHODS: A skill deconstruction list was generated by observation of robotic operations and interviews with experts. Available resources were used, and other components were developed as needed to develop a comprehensive, proficiency-based curriculum to teach all deconstructed skills. Preliminary construct and content validity and curriculum feasibility were evaluated. RESULTS: The skill deconstruction list contained 23 items. Curricular components included an online tutorial, a half-day interactive session, and 9 inanimate exercises with objective metrics. Novice (546 ± 26) and expert (923 ± 60) inanimate composite scores were different (P < .001), supporting construct validity, and substantial pre-test to post-test improvement was noted after successful training completion. All 23 deconstructed skills were rated as highly relevant (4.9 ± 0.5; 5-point scale), and no skills were absent from the curriculum, supporting content validity. CONCLUSION: These data suggest that this proficiency-based training curriculum comprehensively addresses the skills necessary to perform robotic operations with early construct and content validity and feasibility demonstrated. Further validation is encouraged.

Comprehensive proficiency-based inanimate training for robotic surgery: reliability, feasibility, and educational benefit.

BACKGROUND: We previously developed a comprehensive proficiency-based robotic training curriculum demonstrating construct, content, and face validity. This study aimed to assess reliability, feasibility, and educational benefit associated with curricular implementation. METHODS: Over an 11-month period, 55 residents, fellows, and faculty (robotic novices) from general surgery, urology, and gynecology were enrolled in a 2-month curriculum: online didactics, half-day hands-on tutorial, and self-practice using nine inanimate exercises. Each trainee completed a questionnaire and performed a single proctored repetition of each task before (pretest) and after (post-test) training. Tasks were scored for time and errors using modified FLS metrics. For inter-rater reliability (IRR), three trainees were scored by two raters and analyzed using intraclass correlation coefficients (ICC). Data from eight experts were analyzed using ICC and Cronbach's α to determine test-retest reliability and internal consistency, respectively. Educational benefit was assessed by comparing baseline (pretest) and final (post-test) trainee performance; comparisons used Wilcoxon signed-rank test. RESULTS: Of the 55 trainees that pretested, 53 (96 %) completed all curricular components in 9-17 h and reached proficiency after completing an average of 72 ± 28 repetitions over 5 ± 1 h. Trainees indicated minimal prior robotic experience and "poor comfort" with robotic skills at baseline (1.8 ± 0.9) compared to final testing (3.1 ± 0.8, p < 0.001). IRR data for the composite score revealed an ICC of 0.96 (p < 0.001). Test-retest reliability was 0.91 (p < 0.001) and internal consistency was 0.81. Performance improved significantly after training for all nine tasks and according to composite scores (548 ± 176 vs. 914 ± 81, p < 0.001), demonstrating educational benefit. CONCLUSION: This curriculum is associated with high reliability measures, demonstrated feasibility for a large cohort of trainees, and yielded significant educational benefit. Further studies and adoption of this curriculum are encouraged.

Proficiency-based training for robotic surgery: construct validity, workload, and expert levels for nine inanimate exercises.

BACKGROUND: We previously developed nine inanimate training exercises as part of a comprehensive, proficiency-based robotic training curriculum that addressed 23 unique skills identified via task deconstruction of robotic operations. The purpose of this study was to evaluate construct validity, workload, and expert levels for the nine exercises. METHODS: Expert robotic surgeons (n = 8, fellows and faculty) and novice trainees (n = 4, medical students) each performed three to five consecutive repetitions of nine previously reported exercises (five FLS models with or without modifications and four custom-made models). Each task was scored for time and accuracy using modified FLS metrics; task scores were normalized to a previously established (preliminary) proficiency level and a composite score equaled the sum of the nine normalized task scores. Questionnaires were administered regarding prior experience. After each exercise, participants completed a validated NASA-TLX Workload Scale to rate the mental, physical, temporal, performance, effort, and frustration levels of each task. RESULTS: Experts had performed 119 (range = 15-600) robotic operations; novices had observed ≤ 1 robotic operation. For all nine tasks and the composite score, experts achieved significantly better performance than novices (932 ± 67 vs. 618 ± 111, respectively; P < 0.001). No significant differences in workload between experts and novices were detected (32.9 ± 3.5 vs. 32.0 ± 9.1, respectively; n.s.). Importantly, frustration ratings were relatively low for both groups (4.0 ± 0.7 vs. 3.8 ± 1.6, n.s.). The mean performance of the eight experts was deemed suitable as a revised proficiency level for each task. CONCLUSION: Using objective performance metrics, all nine exercises demonstrated construct validity. Workload was similar between experts and novices and frustration was low for both groups. These data suggest that the nine structured exercises are suitable for proficiency-based robotic training.

Content and face validity of a comprehensive robotic skills training program for general surgery, urology, and gynecology.

BACKGROUND: The authors previously developed a comprehensive, proficiency-based robotic training curriculum that aimed to address 23 unique skills identified via task deconstruction of robotic operations. The purpose of this study was to determine the content and face validity of this curriculum. METHODS: Expert robotic surgeons (n = 12) rated each deconstructed skill regarding relevance to robotic operations, were oriented to the curricular components, performed 3 to 5 repetitions on the 9 exercises, and rated each exercise. RESULTS: In terms of content validity, experts rated all 23 deconstructed skills as highly relevant (4.5 on a 5-point scale). Ratings for the 9 inanimate exercises indicated moderate to thorough measurement of designated skills. For face validity, experts indicated that each exercise effectively measured relevant skills (100% agreement) and was highly effective for training and assessment (4.5 on a 5-point scale). CONCLUSIONS: These data indicate that the 23 deconstructed skills accurately represent the appropriate content for robotic skills training and strongly support content and face validity for this curriculum.

Construct and face validity of the American College of Surgeons/Association of Program Directors in Surgery laparoscopic troubleshooting team training exercise.

BACKGROUND: Our aim was to develop an objective scoring system and evaluate construct and face validity for a laparoscopic troubleshooting team training exercise. METHODS: Surgery and gynecology novices (n = 14) and experts (n = 10) participated. Assessments included the following: time-out, scenario decision making (SDM) score (based on essential treatments rendered and completion time), operating room communication assessment (investigator developed), line operations safety audits (teamwork), and National Aeronautics and Space Administration-Task Load Index (workload). RESULTS: Significant differences were detected for SDM scores for scenarios 1 (192 vs 278; P = .01) and 3 (129 vs 225; P = .004), operating room communication assessment (67 vs 91; P = .002), and line operations safety audits (58 vs 87; P = .001), but not for time-out (46 vs 51) or scenario 2 SDM score (301 vs 322). Workload was similar for both groups and face validity (8.8 on a 10-point scale) was strongly supported. CONCLUSIONS: Objective decision-making scoring for 2 of 3 scenarios and communication and teamwork ratings showed construct validity. Face validity and participant feedback were excellent.

Early surgeon impressions and technical difficulty associated with laparoendoscopic single-site surgery: a Society of American Gastrointestinal and Endoscopic Surgeons Learning Center study.

BACKGROUND: Interest in laparoendoscopic single-site surgery (LESS) is growing rapidly among surgeons. This study aimed to characterize current surgeon impressions about LESS and to determine the relative difficulty of performing a simulated LESS task using a multiport access device. METHODS: This study was conducted at the 2009 Society of Gastrointestinal Endoscopic Surgeons (SAGES) Learning Center. The 56 study participants were asked to complete pre- and post-test questionnaires regarding their level of training, prior clinical experience, and opinions about LESS. Technical skill performance was evaluated using the standardized fundamentals of laparoscopic surgery Peg Transfer task scored according to time and error metrics. The participants completed three repetitions: conventional laparoscopy (LAP), LESS with nonarticulated instruments (LESS Straight), and LESS with articulated instruments (LESS Articulating). RESULTS: Complete data were collected for 45 (80%) of the 56 participants, which included 27 practicing surgeons, nine minimally invasive surgery (MIS) fellows, seven residents, and two allied health professionals. Five surgeons (LESS experienced) had managed at least one LESS case in the preceding 6 months. Participants rated their comfort with LESS as 2.0 ± 1.2 (5-point scale, 1 = very uncomfortable). Compared with conventional laparoscopy, the participants indicated that LESS had 97% better cosmesis, 25% decreased postoperative pain, 18% faster recovery, 97% more demanding, 73% increased rate of complications, and 82% anticipated wide adoption. They all indicated a readiness to offer LESS to their patients if appropriately trained. Peg Transfer performance was significantly worse for LESS than for LAP (40-65% performance decline), and for LESS Articulating than for LESS Straight (44% performance decline). Construct validity for the LESS simulated tasks was supported because the LESS-experienced scores were significantly better than the LESS-nonexpert scores. CONCLUSION: Despite the increased technical difficulty associated with the LESS approach, surgeons are enthusiastic about offering these techniques and seeking additional training. Robust simulation-based methods that foster skill acquisition through repetitive practice and verification of proficiency are needed such that safe adoption may be fostered.

Two-year skill retention and certification exam performance after fundamentals of laparoscopic skills training and proficiency maintenance.

BACKGROUND: The purpose of this study was to determine 2-year performance retention and certification exam pass rate after completion of a proficiency-based fundamental laparoscopic skills (FLS) curriculum and subsequent interval training. METHODS: Surgery residents (postgraduate year [PGY]1-5, n = 91) were enrolled in an Institutional Review Board approved protocol. All participants initially underwent proficiency-based training on all 5 FLS tasks. Subsequently, available residents were enrolled every 6 months in an ongoing training curriculum that included retention tests on tasks 4 and 5, with mandatory retraining to proficiency if the proficiency levels were not achieved. The final retention test included the actual FLS certification examination for PGY4-5 trainees. RESULTS: A 96% participation rate was achieved for all curricular components during the 2-year study period (PGY3-5, n = 33). Skill retention at retention 1-4 was 83%, 94%, 98%, and 91% for task 4 and 85%, 95%, 96%, and 100% for task 5, respectively. All PGY4-5 (n = 20) residents passed the FLS certification examination, achieving 413 +/- 28 total score on the skills portion (passing score > or =270) and demonstrating 92% retention for all 5 tasks. CONCLUSION: Proficiency-based training with subsequent ongoing practice results in a very high level of skill retention after 2 years and uniformly allows trainees to pass the FLS certification examination.

Proficiency-based Fundamentals of Laparoscopic Surgery skills training results in durable performance improvement and a uniform certification pass rate.

BACKGROUND: The authors have previously documented a 100% certification pass rate immediately after a proficiency-based skills training curriculum for the Fundamentals of Laparoscopic Surgery (FLS) program. This study aimed to determine the durability of skills acquired after initial training. METHODS: For this study, 21 novice medical students were enrolled in institutional review board (IRB)-approved protocols at two institutions. As previously reported, all the participants successfully completed a structured proficiency-based training curriculum by practicing the five FLS tasks in a distributed fashion over a 2-month period. Pre- and posttesting was conducted, and standard testing metrics were used. The participants were recruited for repeat testing 6 months (retention 1) and 1 year (retention 2) after initial curriculum completion. Of the original 21 students, 15 (10 at University of Texas Southwestern and 5 at Uniformed Services University) were available and agreed to participate. The participants had no additional skills lab training and minimal clinical laparoscopic exposure. RESULTS: None of the 15 participants demonstrated proficiency at the initial pretest (mean score, 146 ± 65), and performance showed significant improvement (p < 0.001) at the posttest (469 ± 20). The participants retained a very high level of performance at retention 1 (437 ± 39; 93% retention of the posttest score) and retention 2 (444 ± 55; 95% retention of the posttest score). Their performance at both retention testing-intervals was sufficient for passing the certification exam (270 cutoff score for passing) with a comfortable margin. There were no significant differences in performance between the two institutions at any time points. CONCLUSION: The proficiency-based FLS skills curriculum reliably results in a high level of skill retention, even in the absence of ongoing simulator-based training or clinical experience. This curriculum is suitable for widespread implementation.

Maintaining proficiency after fundamentals of laparoscopic surgery training: a 1-year analysis of skill retention for surgery residents.

BACKGROUND: The purpose of this study was to determine performance retention after initial and ongoing Fundamentals of Laparoscopic Surgery (FLS) skills training. METHODS: Surgery residents (postgraduate year [PGY] 1-5; n = 91) initially underwent proficiency-based training during a 2-month period for all 5 FLS tasks. Subsequently, available residents (PGY 2-5; n = 44) were enrolled in a follow-up curriculum for Tasks 4 and 5, with retention testing at 6.5 +/- 1.0 months (Retention 1) and 12.5 +/- 1.3 months (Retention 2), followed by mandatory retraining if needed. RESULTS: Forty-two residents participated in the follow-up curriculum. For Task 4, retraining was required for 55% of trainees after Retention 1 and for 40% after Retention 2 testing. For Task 5, retraining was required for 86% of trainees after Retention 1 and for 48% after Retention 2 testing. Compared with posttest performance, skill retention was 86% and 96% for Task 4 and 87% and 96% for Task 5 at Retentions 1 and 2, respectively. CONCLUSION: A modest decrement in performance was detected at 6.5 months, but a small amount of additional structured practice reinforced skill acquisition and minimized skill loss at 12.5 months. In addition to clinical experience, trainees may benefit from ongoing FLS simulator training to optimally maintain proficiency.

Implementation, construct validity, and benefit of a proficiency-based knot-tying and suturing curriculum.

OBJECTIVES: The aim of this proficiency-based, open knot-tying and suturing study was to evaluate the feasibility of implementing this curriculum within a residency program, and to assess construct validity and educational benefit. METHODS: PGY1 residents (n = 37) were enrolled in an Institutional Review Board (IRB)-approved prospective study that was conducted over a 12-week period. Trainees viewed a video tutorial during orientation and as needed; they self-practiced to proficiency for 12 standardized knot-tying, practiced suturing tasks; performed 1 repetition of each task at baseline and posttesting; and completed questionnaires. RESULTS: Curriculum implementation required 376 person-hours, and material costs were $776. All trainees achieved proficiency within allotted 12 weeks. Overall, trainees completed 141 +/- 80 repetitions over 12.7 +/- 5.3 hours in addition to performing 13.4 +/- 12.4 operations. Baseline trainee and expert performance were significantly different for all 12 tasks and composite score (732 +/- 294 vs 1488 +/- 26, p < 0.001), which supported construct validity. Baseline trainees demonstrated significant improvement at posttesting according to composite scores (732 +/- 294 vs 1503 +/- 131, p < 0.001), which validates skill acquisition. CONCLUSIONS: Implementation of this proficiency-based curriculum within the constraints of a residency program is feasible. This curriculum is educationally beneficial and cost effective; our data support construct validity. Evaluation of transferability to the operating room and more widespread adoption of this curriculum are warranted.

Certification pass rate of 100% for fundamentals of laparoscopic surgery skills after proficiency-based training.

BACKGROUND: The fundamentals of laparoscopic surgery (FLS) program has been extensively validated for use as a high-stakes examination for certification purposes, but optimal methods for its use in skills training have not been described. This study aimed to investigate the feasibility of implementing a proficiency-based FLS skills training curriculum and to evaluate its effectiveness in preparing trainees for certification. METHODS: For this study, 21 novice medical students at two institutions viewed video tutorials, then performed one repetition of the five FLS tasks as a pretest. The pretests were scored using standard testing metrics. The trainees next practiced the tasks over a 2-month period until they achieved proficiency for all the tasks. A modified on-the-fly scoring system based on expert-derived performance was used. The trainees were posttested using the high-stakes examination format. RESULTS: No trainee passed the certification examination at pretesting. The trainees achieved proficiency for 96% of the five tasks during training, which required 9.7 +/- 2.4 h (range, 6-14 h) and 119 +/- 31 repetitions (range, 66-161 repetitions). The trainees rated the proficiency levels as "moderately difficult" (3.0 +/- 0.7 on a 5-point scale) and "highly appropriate" (4.7 +/- 0.1 on a 5-point scale). At posttesting, 100% of the trainees passed the certification examination and demonstrated significant improvement compared with pretesting for normalized score (468 +/- 24 vs 126 +/- 75; p < 0.001), self-rated laparoscopic comfort (89.4% vs 4.8%; p < 0.001), and skill level (3.6 +/- 0.9 vs 1.2 +/- 0.5; p < 0.001, 5-point scale). CONCLUSIONS: This proficiency-based curriculum is feasible for training novices and uniformly allows sufficient skill acquisition for FLS certification. Endorsed by the Society of American Gastrointestinal Endoscopic Surgeons (SAGES), this curriculum is available for use as an optimal method for FLS skills training. More widespread adoption of this curriculum is encouraged.

A cost-effective proficiency-based knot-tying and suturing curriculum for residency programs.

BACKGROUND: The purpose of this study was to develop a structured open skills curriculum for knot-tying and suturing using expert-derived performance goals and to examine its feasibility, cost-effectiveness, and construct validity. METHODS: Using commercially available bench models, 11 standardized tasks (ranging from 2-handed knot-tying to running subcuticular closure) were developed and scored using previously validated metrics based on time and errors. Expert performance was used to establish training endpoints and to create a video tutorial. PGY 1 residents (n = 4) were enrolled in a prospective Institutional Review Board-approved pilot study that included proctored orientation and baseline testing, self-training to proficiency, and proctored post-testing (conducted over a 4-wk period). Baseline trainee scores were compared with expert scores to evaluate construct validity. RESULTS: The 11 tasks proved relatively robust, and excellent feedback was obtained from the trainees regarding educational benefit. Overall, trainees performed 144 +/- 33 repetitions over 11 +/- 2 h. Trainees achieved proficiency for 4.6% of the 11 tasks at baseline, 91% during training, and 84% at post-testing. Trainees demonstrated significant improvement from baseline to post-testing, validating skill acquisition; baseline trainee and expert performance were significantly different, confirming construct validity. Curriculum development cost $1200 and required 72 man-hours. Incremental training cost less than $12 per participant and required 8 man-hours per rotation using the video-based self-practice curriculum. In response to participant feedback, two of the 11 tasks were modified and a twelfth task was added. CONCLUSIONS: This curriculum is cost-effective, feasible within the context of residency training, educationally beneficial, and demonstrates construct validity. More widespread adoption of standardized, validated skills curricula such as this by residency programs is warranted.

Multicenter construct validity for southwestern laparoscopic videotrainer stations.

BACKGROUND: The "Southwestern" videotrainer stations have demonstrated concurrent validity (transferability to the operating room). The purpose of this study was to evaluate the Southwestern stations for construct validity (the ability to discriminate between subjects at different levels of experience). MATERIALS AND METHODS: From two surgical training programs, Institutional Review Board approved protocol data were collected from 142 subjects, including novice (medical students and R1, n = 66), intermediate (R2-R4, n = 67), and advanced (R5 and expert surgeons, n = 9) groups. All participants performed three repetitions on each of five stations. Completion time was scored for each task. Laparoscopic experience was determined from residency case log databases and from expert surgeon personal case logs. Results for the three groups were compared using one-way ANOVA, including relevant pair-wise comparisons. Correlations between number of laparoscopic cases performed and task scores were determined by Pearson's and Spearman's rho-correlation coefficients. RESULTS: The mean number of laparoscopic cases performed prior to completing the five tasks was 0 for novices, 9 for intermediates, and 431 for the advanced group. Significant differences (P < 0.001) were noted between groups for all five tasks and composite score. Task scores and composite scores significantly correlated with laparoscopic experience (P < 0.01). CONCLUSION: These data suggest that differences in laparoscopic ability are detected by performance on the videotrainer; thus, construct validity is demonstrated. Moreover, scores accurately reflect laparoscopic experience. Further validation may allow such simulators to be used for testing and credentialing purposes.