A surgical instrument motion measurement system for skill evaluation in practical laparoscopic surgery training.

This study developed and validated a surgical instrument motion measurement system for skill evaluation during practical laparoscopic surgery training. Owing to the various advantages of laparoscopic surgery including minimal invasiveness, this technique has been widely used. However, expert surgeons have insufficient time for providing training to beginners due to the shortage of surgeons and limited working hours. Skill transfer efficiency has to be improved for which there is an urgent need to develop objective surgical skill evaluation methods. Therefore, a simple motion capture-based surgical instrument motion measurement system that could be easily installed in an operating room for skill assessment during practical surgical training was developed. The tip positions and orientations of the instruments were calculated based on the marker positions attached to the root of the instrument. Because the patterns of these markers are individual, this system can track multiple instruments simultaneously and detect exchanges. However due to the many obstacles in the operating room, the measurement data included noise and outliers. In this study, the effect of this decrease in measurement accuracy on feature calculation was determined. Accuracy verification experiments were conducted during wet-lab training to demonstrate the capability of this system to measure the motion of surgical instruments with practical accuracy. A surgical training experiment on a cadaver was conducted, and the motions of six surgical instruments were measured in 36 cases of laparoscopic radical nephrectomy. Outlier removal and smoothing methods were also developed and applied to remove the noise and outliers in the obtained data. The questionnaire survey conducted during the experiment confirmed that the measurement system did not interfere with the surgical operation. Thus, the proposed system was capable of making reliable measurements with minimal impact on surgery. The system will facilitate surgical education by enabling the evaluation of skill transfer of surgical skills.

Automatic assessment of laparoscopic surgical skill competence based on motion metrics.

The purpose of this study was to characterize the motion features of surgical devices associated with laparoscopic surgical competency and build an automatic skill-credential system in porcine cadaver organ simulation training. Participants performed tissue dissection around the aorta, dividing vascular pedicles after applying Hem-o-lok (tissue dissection task) and parenchymal closure of the kidney (suturing task). Movements of surgical devices were tracked by a motion capture (Mocap) system, and Mocap-metrics were compared according to the level of surgical experience (experts: ≥50 laparoscopic surgeries, intermediates: 10-49, novices: 0-9), using the Kruskal-Wallis test and principal component analysis (PCA). Three machine-learning algorithms: support vector machine (SVM), PCA-SVM, and gradient boosting decision tree (GBDT), were utilized for discrimination of the surgical experience level. The accuracy of each model was evaluated by nested and repeated k-fold cross-validation. A total of 32 experts, 18 intermediates, and 20 novices participated in the present study. PCA revealed that efficiency-related metrics (e.g., path length) significantly contributed to PC 1 in both tasks. Regarding PC 2, speed-related metrics (e.g., velocity, acceleration, jerk) of right-hand devices largely contributed to the tissue dissection task, while those of left-hand devices did in the suturing task. Regarding the three-group discrimination, in the tissue dissection task, the GBDT method was superior to the other methods (median accuracy: 68.6%). In the suturing task, SVM and PCA-SVM methods were superior to the GBDT method (57.4 and 58.4%, respectively). Regarding the two-group discrimination (experts vs. intermediates/novices), the GBDT method resulted in a median accuracy of 72.9% in the tissue dissection task, and, in the suturing task, the PCA-SVM method resulted in a median accuracy of 69.2%. Overall, the mocap-based credential system using machine-learning classifiers provides a correct judgment rate of around 70% (two-group discrimination). Together with motion analysis and wet-lab training, simulation training could be a practical method for objectively assessing the surgical competence of trainees.

Objective evaluation of laparoscopic surgical skills in wet lab training based on motion analysis and machine learning.

BACKGROUND: Our aim was to build a skill assessment system, providing objective feedback to trainees based on the motion metrics of laparoscopic surgical instruments. METHODS: Participants performed tissue dissection around the aorta (tissue dissection task) and renal parenchymal closure (parenchymal-suturing task), using swine organs in a box trainer under a motion capture (Mocap) system. Two experts assessed the recorded movies, according to the formula of global operative assessment of laparoscopic skills (GOALS: score range, 5-25), and the mean scores were utilized as objective variables in the regression analyses. The correlations between mean GOALS scores and Mocap metrics were evaluated, and potential Mocap metrics with a Spearman's rank correlation coefficient value exceeding 0.4 were selected for each GOALS item estimation. Four regression algorithms, support vector regression (SVR), principal component analysis (PCA)-SVR, ridge regression, and partial least squares regression, were utilized for automatic GOALS estimation. Model validation was conducted by nested and repeated k-fold cross validation, and the mean absolute error (MAE) was calculated to evaluate the accuracy of each regression model. RESULTS: Forty-five urologic, 9 gastroenterological, and 3 gynecologic surgeons, 4 junior residents, and 9 medical students participated in the training. In both tasks, a positive correlation was observed between the speed-related parameters (e.g., velocity, velocity range, acceleration, jerk) and mean GOALS scores, with a negative correlation between the efficiency-related parameters (e.g., task time, path length, number of opening/closing operations) and mean GOALS scores. Among the 4 algorithms, SVR showed the highest accuracy in the tissue dissection task ([Formula: see text]), and PCA-SVR in the parenchymal-suturing task ([Formula: see text]), based on 100 iterations of the validation process of automatic GOALS estimation. CONCLUSION: We developed a machine learning-based GOALS scoring system in wet lab training, with an error of approximately 1-2 points for the total score, and motion metrics that were explainable to trainees. Our future challenges are the further improvement of onsite GOALS feedback, exploring the educational benefit of our model and building an efficient training program.

Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs.

BACKGROUND: Our aim was to characterize the motions of multiple laparoscopic surgical instruments among participants with different levels of surgical experience in a series of wet-lab training drills, in which participants need to perform a range of surgical procedures including grasping tissue, tissue traction and dissection, applying a Hem-o-lok clip, and suturing/knotting, and digitize the level of surgical competency. METHODS: Participants performed tissue dissection around the aorta, dividing encountered vessels after applying a Hem-o-lok (Task 1), and renal parenchymal closure (Task 2: suturing, Task 3: suturing and knot-tying), using swine cadaveric organs placed in a box trainer under a motion capture (Mocap) system. Motion-related metrics were compared according to participants' level of surgical experience (experts: 50 ≤ laparoscopic surgeries, intermediates: 10-49, novices: 0-9), using the Kruskal-Wallis test, and significant metrics were subjected to principal component analysis (PCA). RESULTS: A total of 15 experts, 12 intermediates, and 18 novices participated in the training. In Task 1, a shorter path length and faster velocity/acceleration/jerk were observed using both scissors and a Hem-o-lok applier in the experts, and Hem-o-lok-related metrics markedly contributed to the 1st principal component on PCA analysis, followed by scissors-related metrics. Higher-level skills including a shorter path length and faster velocity were observed in both hands of the experts also in tasks 2 and 3. Sub-analysis showed that, in experts with 100 ≤ cases, scissors moved more frequently in the "close zone (0 ≤ to < 2.0 cm from aorta)" than those with 50-99 cases. CONCLUSION: Our novel Mocap system recognized significant differences in several metrics in multiple instruments according to the level of surgical experience. "Applying a Hem-o-lok clip on a pedicle" strongly reflected the level of surgical experience, and zone-metrics may be a promising tool to assess surgical expertise. Our next challenge is to give completely objective feedback to trainees on-site in the wet-lab.