ABSTRACT
The value of kinematic data for skill assessment is being investigated. This is the first virtual reality simulator developed for liver surgery. This simulator was coded in C++ using PhysX and FleX with a novel cutting algorithm and used a patient data-derived model and two instruments functioning as ultrasonic shears. The simulator was evaluated by nine expert surgeons and nine surgical novices. Each participant performed a simulated metastasectomy after training. Kinematic data were collected for the instrument position. Each participant completed a survey. The expert participants had a mean age of 47 years and 9/9 were certified in surgery. Novices had a mean age of 30 years and 0/9 were certified surgeons. The mean path length (novice 0.76 ± 0.20 m vs. expert 0.46 ± 0.16 m, p = 0.008), movements (138 ± 45 vs. 84 ± 32, p = 0.043) and time (174 ± 44 s vs. 102 ± 42 s, p = 0.004) were significantly different for the two participant groups. There were no significant differences in activating the instrument (107 ± 25 vs. 109 ± 53). Participants considered the simulator realistic (6.5/7) (face validity), appropriate for education (5/7) (content validity) with an effective interface (6/7), consistent motion (5/7) and realistic soft tissue behavior (5/7). This study showed that the simulator differentiates between experts and novices. Simulation may be an effective way to obtain kinematic data.
ABSTRACT
PURPOSE: The manual generation of training data for the semantic segmentation of medical images using deep neural networks is a time-consuming and error-prone task. In this paper, we investigate the effect of different levels of realism on the training of deep neural networks for semantic segmentation of robotic instruments. An interactive virtual-reality environment was developed to generate synthetic images for robot-aided endoscopic surgery. In contrast with earlier works, we use physically based rendering for increased realism. METHODS: Using a virtual reality simulator that replicates our robotic setup, three synthetic image databases with an increasing level of realism were generated: flat, basic, and realistic (using the physically-based rendering). Each of those databases was used to train 20 instances of a UNet-based semantic-segmentation deep-learning model. The networks trained with only synthetic images were evaluated on the segmentation of 160 endoscopic images of a phantom. The networks were compared using the Dwass-Steel-Critchlow-Fligner nonparametric test. RESULTS: Our results show that the levels of realism increased the mean intersection-over-union (mIoU) of the networks on endoscopic images of a phantom ([Formula: see text]). The median mIoU values were 0.235 for the flat dataset, 0.458 for the basic, and 0.729 for the realistic. All the networks trained with synthetic images outperformed naive classifiers. Moreover, in an ablation study, we show that the mIoU of physically based rendering is superior to texture mapping ([Formula: see text]) of the instrument (0.606), the background (0.685), and the background and instruments combined (0.672). CONCLUSIONS: Using physical-based rendering to generate synthetic images is an effective approach to improve the training of neural networks for the semantic segmentation of surgical instruments in endoscopic images. Our results show that this strategy can be an essential step in the broad applicability of deep neural networks in semantic segmentation tasks and help bridge the domain gap in machine learning.
