Machine Learning for Surgical Phase Recognition: A Systematic Review.

OBJECTIVE: To provide an overview of ML models and data streams utilized for automated surgical phase recognition. BACKGROUND: Phase recognition identifies different steps and phases of an operation. ML is an evolving technology that allows analysis and interpretation of huge data sets. Automation of phase recognition based on data inputs is essential for optimization of workflow, surgical training, intraoperative assistance, patient safety, and efficiency. METHODS: A systematic review was performed according to the Cochrane recommendations and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement. PubMed, Web of Science, IEEExplore, GoogleScholar, and CiteSeerX were searched. Literature describing phase recognition based on ML models and the capture of intraoperative signals during general surgery procedures was included. RESULTS: A total of 2254 titles/abstracts were screened, and 35 full-texts were included. Most commonly used ML models were Hidden Markov Models and Artificial Neural Networks with a trend towards higher complexity over time. Most frequently used data types were feature learning from surgical videos and manual annotation of instrument use. Laparoscopic cholecystectomy was used most commonly, often achieving accuracy rates over 90%, though there was no consistent standardization of defined phases. CONCLUSIONS: ML for surgical phase recognition can be performed with high accuracy, depending on the model, data type, and complexity of surgery. Different intraoperative data inputs such as video and instrument type can successfully be used. Most ML models still require significant amounts of manual expert annotations for training. The ML models may drive surgical workflow towards standardization, efficiency, and objectiveness to improve patient outcome in the future. REGISTRATION PROSPERO: CRD42018108907.

Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying.

INTRODUCTION: The most common way of assessing surgical performance is by expert raters to view a surgical task and rate a trainee's performance. However, there is huge potential for automated skill assessment and workflow analysis using modern technology. The aim of the present study was to evaluate machine learning (ML) algorithms using the data of a Myo armband as a sensor device for skills level assessment and phase detection in laparoscopic training. MATERIALS AND METHODS: Participants of three experience levels in laparoscopy performed a suturing and knot tying task on silicon models. Experts rated performance using Objective Structured Assessment of Surgical Skills (OSATS). Participants wore Myo armbands (Thalmic Labs™, Ontario, Canada) to record acceleration, angular velocity, orientation, and Euler orientation. ML algorithms (decision forest, neural networks, boosted decision tree) were compared for skill level assessment and phase detection. RESULTS: 28 participants (8 beginner, 10 intermediate, 10 expert) were included, and 99 knots were available for analysis. A neural network regression model had the lowest mean absolute error in predicting OSATS score (3.7 ± 0.6 points, r2 = 0.03 ± 0.81; OSATS min.-max.: 4-37 points). An ensemble of binary-class neural networks yielded the highest accuracy in predicting skill level (beginners: 82.2% correctly identified, intermediate: 3.0%, experts: 79.5%) whereas standard statistical analysis failed to discriminate between skill levels. Phase detection on raw data showed the best results with a multi-class decision jungle (average 16% correctly identified), but improved to 43% average accuracy with two-class boosted decision trees after Dynamic time warping (DTW) application. CONCLUSION: Modern machine learning algorithms aid in interpreting complex surgical motion data, even when standard analysis fails. Dynamic time warping offers the potential to process and compare surgical motion data in order to allow automated surgical workflow detection. However, further research is needed to interpret and standardize available data and improve sensor accuracy.

The Heidelberg VR Score: development and validation of a composite score for laparoscopic virtual reality training.

INTRODUCTION: Virtual reality (VR-)trainers are well integrated in laparoscopic surgical training. However, objective feedback is often provided in the form of single parameters, e.g., time or number of movements, making comparisons and evaluation of trainees' overall performance difficult. Therefore, a new standard for reporting outcome data is highly needed. The aim of this study was to create a weighted, expert-based composite score, to offer simple and direct evaluation of laparoscopic performance on common VR-trainers. MATERIALS AND METHODS: An integrated analytic hierarchy process-Delphi survey was conducted with 14 international experts to achieve a consensus on the importance of different skill categories and parameters in evaluation of laparoscopic performance. A scoring algorithm was established to allow comparability between tasks and VR-trainers. A weighted composite score was calculated for basic skills tasks and peg transfer on the LapMentor™ II and III and validated for both VR-trainers. RESULTS: Five major skill categories (time, efficiency, safety, dexterity, and outcome) were identified and weighted in two Delphi rounds. Safety, with a weight of 67%, was determined the most important category, followed by efficiency with 17%. The LapMentor™-specific score was validated using 15 (14) novices and 9 experts; the score was able to differentiate between both groups for basic skills tasks and peg transfer (LapMentor™ II: Exp: 86.5 ± 12.7, Nov. 52.8 ± 18.3; p < 0.001; LapMentor™ III: Exp: 80.8 ± 7.1, Nov: 50.6 ± 16.9; p < 0.001). CONCLUSION: An effective and simple performance measurement was established to propose a new standard in analyzing and reporting VR outcome data-the Heidelberg virtual reality (VR) score. The scoring algorithm and the consensus results on the importance of different skill aspects in laparoscopic surgery are universally applicable and can be transferred to any simulator or task. By incorporating specific expert baseline data for the respective task, comparability between tasks, studies, and simulators can be achieved.

Computer tomographic analysis of organ motion caused by respiration and intraoperative pneumoperitoneum in a porcine model for navigated minimally invasive esophagectomy.

BACKGROUND: Navigation systems have the potential to facilitate intraoperative orientation and recognition of anatomical structures. Intraoperative accuracy of navigation in thoracoabdominal surgery depends on soft tissue deformation. We evaluated esophageal motion caused by respiration and pneumoperitoneum in a porcine model for minimally invasive esophagectomy. METHODS: In ten pigs (20-34 kg) under general anesthesia, gastroscopic hemoclips were applied to the cervical (CE), high (T1), middle (T2), and lower thoracic (T3) level, and to the gastroesophageal junction (GEJ) of the esophagus. Furthermore, skin markers were applied. Three-dimensional (3D) and four-dimensional (4D) computed tomography (CT) scans were acquired before and after creation of pneumoperitoneum. Marker positions and lung volumes were analyzed with open source image segmentation software. RESULTS: Respiratory motion of the esophagus was higher at T3 (7.0 ± 3.3 mm, mean ± SD) and GEJ (6.9 ± 2.8 mm) than on T2 (4.5 ± 1.8 mm), T1 (3.1 ± 1.8 mm), and CE (1.3 ± 1.1 mm). There was significant motion correlation in between the esophageal levels. T1 motion correlated with all other esophagus levels (r = 0.51, p = 0.003). Esophageal motion correlated with ventilation volume (419 ± 148 ml) on T1 (r = 0.29), T2 (r = 0.44), T3 (r = 0.54), and GEJ (r = 0.58) but not on CE (r = - 0.04). Motion correlation of the esophagus with skin markers was moderate to high for T1, T2, T3, GEJ, but not evident for CE. Pneumoperitoneum led to considerable displacement of the esophagus (8.2 ± 3.4 mm) and had a level-specific influence on respiratory motion. CONCLUSIONS: The position and motion of the esophagus was considerably influenced by respiration and creation of pneumoperitoneum. Esophageal motion correlated with respiration and skin motion. Possible compensation mechanisms for soft tissue deformation were successfully identified. The porcine model is similar to humans for respiratory esophageal motion and can thus help to develop navigation systems with compensation for soft tissue deformation.

Validation of the mobile serious game application Touch Surgery™ for cognitive training and assessment of laparoscopic cholecystectomy.

BACKGROUND: Touch Surgery™ (TS) is a serious gaming application for cognitive task simulation and rehearsal of key steps in surgical procedures. The aim was to establish face, content, and construct validity of TS for laparoscopic cholecystectomy (LC). Furthermore, learning curves with TS and a virtual reality (VR) trainer were compared in a randomized trial. METHODS: The performance of medical students and general surgeons was compared for all three modules of LC in TS to establish construct validity. Questionnaires assessed face and content validity. For analysis of learning curves, students were randomized to train on VR or TS first, and then switched to the other training modality. Performance data were recorded. RESULTS: 54 Surgeons and 51 medical students completed the validation study. Surgeons outperformed students with TS: patient preparation (students = 45.0 ± 19.1%; surgeons = 57.3 ± 15.2%; p < 0.001), access and laparoscopy (students = 70.2 ± 10.9%; surgeons = 75.9 ± 9.7%; p = 0.008) and LC (students = 69.8 ± 12.4%; surgeons = 77.7 ± 9.6%; p < 0.001). Both groups agreed that TS was a highly useful and realistic application. 46 students were randomized for learning curve analysis. It took them 2-4 attempts to reach a 100% score with TS. Training with TS first did not improve students' performance on the VR trainer; however, students who trained with VR first scored significantly higher in module 3 of TS. CONCLUSION: TS is an accepted serious gaming application for learning cognitive aspects of LC with established construct, face, and content validity. There appeared to be a synergy between TS and the VR trainer. Therefore, the two training modalities should accompany one another in a multimodal training approach to laparoscopy.

Study protocol for a randomized controlled trial on a multimodal training curriculum for laparoscopic cholecystectomy - LapTrain.

BACKGROUND: Although minimally invasive surgery (MIS) has replaced many open procedures in visceral surgery, technical and psychomotor obstacles remain a constant challenge for surgeons and trainees. However, there are various training curricula enabling surgeons to acquire the visuospatial and psychomotor abilities additionally required when performing MIS. Currently accepted training modalities include box-trainers, organ and animal models as well as completely simulated training environments, realized in virtual reality (VR) trainers. All of these methods facilitate an adequate training prior to patient contact, so patient safety can benefit as well. This study aims to evaluate the benefit of a structured multi-modality laparoscopy training curriculum. METHODS: Junior and senior surgical residents are included (n = 60). Groups are stratified with concern to previous experience and training of participants. The training curriculum consists of a standardized sequence of available modalities and exercises on box- and VR-trainers. Specific consideration applies to the training effect during the repeated performance of a laparoscopic cholecystectomy (LC) between intervention (training in between LCs) and control group (no training in between LCs). Analysis of training effects is performed using a cadaveric model for LC and objectified using the validated scoring system Global Operative Assessment of Laparoscopic Skills (GOALS). DISCUSSION: This study assesses the value of a multimodal training platform in medical education and postgraduate training and aims at illustrating possible guidelines when establishing such a curriculum. Possible factors of influence, such as varying backgrounds, learning motivation and -success among participants are explored in the data analysis and add beneficially to further evaluating the efficacy of such training to more heterogeneous participant groups like medical students and other professionals.

Development and validation of a sensor- and expert model-based training system for laparoscopic surgery: the iSurgeon.

INTRODUCTION: Training and assessment outside of the operating room is crucial for minimally invasive surgery due to steep learning curves. Thus, we have developed and validated the sensor- and expert model-based laparoscopic training system, the iSurgeon. MATERIALS: Participants of different experience levels (novice, intermediate, expert) performed four standardized laparoscopic knots. Instruments and surgeons' joint motions were tracked with an NDI Polaris camera and Microsoft Kinect v1. With frame-by-frame image analysis, the key steps of suturing and knot tying were identified and registered with motion data. Construct validity, concurrent validity, and test-retest reliability were analyzed. The Objective Structured Assessment of Technical Skills (OSATS) was used as the gold standard for concurrent validity. RESULTS: The system showed construct validity by discrimination between experience levels by parameters such as time (novice = 442.9 ± 238.5 s; intermediate = 190.1 ± 50.3 s; expert = 115.1 ± 29.1 s; p < 0.001), total path length (novice = 18,817 ± 10318 mm; intermediate = 9995 ± 3286 mm; expert = 7265 ± 2232 mm; p < 0.001), average speed (novice = 42.9 ± 8.3 mm/s; intermediate = 52.7 ± 11.2 mm/s; expert = 63.6 ± 12.9 mm/s; p < 0.001), angular path (novice = 20,573 ± 12,611°; intermediate = 8652 ± 2692°; expert = 5654 ± 1746°; p < 0.001), number of movements (novice = 2197 ± 1405; intermediate = 987 ± 367; expert = 743 ± 238; p < 0.001), number of movements per second (novice = 5.0 ± 1.4; intermediate = 5.2 ± 1.5; expert = 6.6 ± 1.6; p = 0.025), and joint angle range (for different axes and joints all p < 0.001). Concurrent validity of OSATS and iSurgeon parameters was established. Test-retest reliability was given for 7 out of 8 parameters. The key steps "wrapping the thread around the instrument" and "needle positioning" were most difficult to learn. CONCLUSION: Validity and reliability of the self-developed sensor-and expert model-based laparoscopic training system "iSurgeon" were established. Using multiple parameters proved more reliable than single metric parameters. Wrapping of the needle around the thread and needle positioning were identified as difficult key steps for laparoscopic suturing and knot tying. The iSurgeon could generate automated real-time feedback based on expert models which may result in shorter learning curves for laparoscopic tasks. Our next steps will be the implementation and evaluation of full procedural training in an experimental model.

Sequential learning of psychomotor and visuospatial skills for laparoscopic suturing and knot tying-a randomized controlled trial "The Shoebox Study" DRKS00008668.

PURPOSE: Learning curves for minimally invasive surgery are prolonged since psychomotor skills and visuospatial orientation differ from open surgery and must be learned. This study explored potential advantages of sequential learning of psychomotor and visuospatial skills for laparoscopic suturing and knot tying compared to simultaneous learning. METHODS: Laparoscopy-naïve medical students were randomized into a sequential learning group (SEQ) or a simultaneous learning group (SIM). SEQ (n = 28) trained on a shoebox with direct 3D view before proceeding on a box trainer with 2D laparoscopic view. SIM (n = 25) trained solely on a box trainer with 2D laparoscopic view. Training time and number of attempts needed were recorded until a clearly defined proficiency level was reached. RESULTS: Groups were not different in total training time (SEQ 5868.7 ± 2857.2 s; SIM 5647.1 ± 2244.8 s; p = 0.754) and number of attempts to achieve proficiency in their training (SEQ 44.0 ± 17.7; SIM 36.8 ± 15.6; p = 0.123). SEQ needed less training time on the box trainer with 2D laparoscopic view than did SIM (SEQ 4170.9 ± 2350.8 s; SIM 5647.1 ± 2244.8 s; p = 0.024), while the number of attempts here was not different (SEQ 29.9 ± 14.1; SIM 36.8 ± 15.6; p = 0.097). SEQ was faster in the first attempts on the shoebox (281.9 ± 113.1 s) and box trainer (270.4 ± 133.1 s) compared to the first attempt of SIM on the box trainer (579.4 ± 323.8 s) (p < 0.001). CONCLUSION: In the present study, SEQ was faster than SIM at the beginning of the learning curve. SEQ did not reduce the total training time needed to reach an ambitious proficiency level. However, SEQ needed less training on the box trainer; thus, laparoscopic experience can be gained to a certain extent with a simple shoebox.

Sequential learning of psychomotor and visuospatial skills for laparoscopic suturing and knot tying - study protocol for a randomized controlled trial "The shoebox study".

BACKGROUND: Laparoscopy training has become an integral part of surgical education. Suturing and knot tying is a basic, yet inherent part of many laparoscopic operations, and should be mastered prior to operating on patients. One common and standardized suturing technique is the C-loop technique. In the standard training setting, on a box trainer, the trainee learns the psychomotor movements of the task and the laparoscopic visuospatial orientation simultaneously. Learning the psychomotor and visuospatial skills separately and sequentially may offer a more time-efficient alternative to the current standard of training. METHODS: This is a monocentric, two-arm randomized controlled trial. The participants are medical students in their clinical years (third to sixth year) at Heidelberg University who have not previously partaken in a laparoscopic training course lasting more than 2 hours. A total of 54 students are randomized into one of two arms in a 1:1 ratio to sequential learning (group 1) or control (group 2). Both groups receive a standardized introduction to the training center, laparoscopic instruments, and C-loop technique. Group 1 learn the C-loop using a transparent shoebox, thus only learning the psychomotor skills. Once they reach proficiency, they then perform the same knot tying procedure on a box trainer with standard laparoscopic view, where they combine their psychomotor skills with the visuospatial orientation inherent to laparoscopy. Group 2 learn the C-loop using solely a box trainer with standard laparoscopic view until they reach proficiency. Trainees work in pairs and time is recorded for each attempt. The primary outcome is mean total training time for each group. Secondary endpoints include procedural and knot quality subscore differences. Tertiary endpoints include studying the influence of gender and video game experience on performance. DISCUSSION: This study addresses whether the learning of the psychomotor and visuospatial aspects of laparoscopic suturing and knot tying is optimal sequentially or simultaneously, by assessing total training time, procedural, and knot quality differences between the two groups. It will improve the efficiency of future laparoscopic suturing courses and may serve as an indicator for laparoscopic training in a broader context, i.e., not only for suturing and knot tying. TRIAL REGISTRATION: This trial was registered on 12 August 2015 with the trial registration number DRKS00008668 .