ERegPose: An explicit regression based 6D pose estimation for snake-like wrist-type surgical instruments.

BACKGROUND: Accurately estimating the 6D pose of snake-like wrist-type surgical instruments is challenging due to their complex kinematics and flexible design. METHODS: We propose ERegPose, a comprehensive strategy for precise 6D pose estimation. The strategy consists of two components: ERegPoseNet, an original deep neural network model designed for explicit regression of the instrument's 6D pose, and an annotated in-house dataset of simulated surgical operations. To capture rotational features, we employ an Single Shot multibox Detector (SSD)-like detector to generate bounding boxes of the instrument tip. RESULTS: ERegPoseNet achieves an error of 1.056 mm in 3D translation, 0.073 rad in 3D rotation, and an average distance (ADD) metric of 3.974 mm, indicating an overall spatial transformation error. The necessity of the SSD-like detector and L1 loss is validated through experiments. CONCLUSIONS: ERegPose outperforms existing approaches, providing accurate 6D pose estimation for snake-like wrist-type surgical instruments. Its practical applications in various surgical tasks hold great promise.

Stereo attention-based all-in-one super-resolution for robot-assisted minimally invasive surgery.

Robot-assisted minimally invasive surgery (MIS) faces challenges in obtaining high-quality imaging results due to the limited spatial environment. In this paper, we present an all-in-one image super-resolution (SR) algorithm designed to tackle this challenge. By utilizing the stereo information from binocular images, we effectively convert low-resolution images into high-resolution ones. Our model architecture amalgamates the prowess of Convolutional Neural Networks (CNNs) and Transformers, capitalizing on the advantages of both methodologies. To achieve super-resolution across all scale factors, we employ a trainable upsampling module within our proposed network. We substantiate the effectiveness of our method through extensive quantitative and qualitative experiments. The results of our evaluations provide strong evidence supporting the superior performance of our approach in enhancing the quality of surgical images. Our method improves the resolution and thus the overall image quality, which allows the surgeon to perform precise operations conveniently. Simultaneously, it also facilitates the scaling of the region of interest (ROI) to achieve high-quality visualization during surgical procedures. Furthermore, it has the potential to enhance the image quality during telesurgery.

Dynamic surface reconstruction in robot-assisted minimally invasive surgery based on neural radiance fields.

PURPOSE: The purpose of this study was to improve surgical scene perception by addressing the challenge of reconstructing highly dynamic surgical scenes. We proposed a novel depth estimation network and a reconstruction framework that combines neural radiance fields to provide more accurate scene information for surgical task automation and AR navigation. METHODS: We added a spatial pyramid pooling module and a Swin-Transformer module to enhance the robustness of stereo depth estimation. We also improved depth accuracy by adding unique matching constraints from optimal transport. To avoid deformation distortion in highly dynamic scenes, we used neural radiance fields to implicitly represent scenes in the time dimension and optimized them with depth and color information in a learning-based manner. RESULTS: Our experiments on the KITTI and SCARED datasets show that the proposed depth estimation network performs close to the state-of-the-art method on natural images and surpasses the SOTA method on medical images with 1.12% in 3 px Error and 0.45 px in EPE. The proposed dynamic reconstruction framework successfully reconstructed the dynamic cardiac surface on a totally endoscopic coronary artery bypass video, achieving SOTA performance with 27.983 dB in PSNR, 0.812 in SSIM, and 0.189 in LPIPS. CONCLUSION: Our proposed depth estimation network and reconstruction framework provide a significant contribution to the field of surgical scene perception. The framework achieves better results than SOTA methods on medical datasets, reducing mismatches on depth maps and resulting in more accurate depth maps with clearer edges. The proposed ER framework is verified on a series of dynamic cardiac surgical images. Future efforts will focus on improving the training speed and solving the problem of limited field of view.

Surgical smoke removal via residual Swin transformer network.

PURPOSE: In robot-assisted minimally invasive surgery (RMIS), smoke produced by laser ablation and cauterization causes degradation in the visual quality of the operating field, increasing the difficulty and risk of surgery. Therefore, it is important and meaningful to remove fog or smoke from the endoscopic video to maintain a clear visual field. METHODS: In this paper, we propose a novel method for surgical smoke removal based on the Swin transformer. Our method firstly uses convolutional neural network to extract shallow features, then uses the Swin transformer block to further extract deep features and finally generates smoke-free images. RESULTS: We conduct quantitative and qualitative experiments on the proposed method, and we also validate the desmoking results in the surgical instrument segmentation task. Extensive experiments on synthetic and real dataset show that the proposed approach has good performance and outperforms the state-of-the-art surgical smoke removal methods. CONCLUSION: Our method effectively removes surgical smoke, improves image quality and reduces the risk of RMIS. It provides a clearer visual field for the surgeon, as well as for subsequent visual tasks, such as instrument segmentation, 3D scene reconstruction and surgery automation.

A parallel network utilizing local features and global representations for segmentation of surgical instruments.

PURPOSE: Automatic image segmentation of surgical instruments is a fundamental task in robot-assisted minimally invasive surgery, which greatly improves the context awareness of surgeons during the operation. A novel method based on Mask R-CNN is proposed in this paper to realize accurate instance segmentation of surgical instruments. METHODS: A novel feature extraction backbone is built, which could extract both local features through the convolutional neural network branch and global representations through the Swin-Transformer branch. Moreover, skip fusions are applied in the backbone to fuse both features and improve the generalization ability of the network. RESULTS: The proposed method is evaluated on the dataset of MICCAI 2017 EndoVis Challenge with three segmentation tasks and shows state-of-the-art performance with an mIoU of 0.5873 in type segmentation and 0.7408 in part segmentation. Furthermore, the results of ablation studies prove that the proposed novel backbone contributes to at least 17% improvement in mIoU. CONCLUSION: The promising results demonstrate that our method can effectively extract global representations as well as local features in the segmentation of surgical instruments and improve the accuracy of segmentation. With the proposed novel backbone, the network can segment the contours of surgical instruments' end tips more precisely. This method can provide more accurate data for localization and pose estimation of surgical instruments, and make a further contribution to the automation of robot-assisted minimally invasive surgery.

A variable baseline stereoscopic camera with fast deployable structure for natural orifice transluminal endoscopic surgery.

PURPOSE: Stereo vision can provide surgeons with 3D images and reduce the difficulty of operation in robot-assisted surgery. In natural orifice transluminal endoscopic surgery, distortions of the stereoscopic images could be induced at different observation depths. This would increase the risk of surgery. We proposed a novel camera to solve this problem. METHODS: This study integrated the camera calibration matrix and the geometric model of stereoscopic system to find the cause of distortion. It was found that image distortions were caused by inappropriate disparity, and this could be avoided by changing the camera baseline. We found the relationship between camera baseline and observation depth with the model. A variable baseline stereoscopic camera with deployable structure was designed to achieve this requirement. The baseline could be adjusted to provide appropriate disparity. RESULTS: Three controlled experiments were conducted to verify the stereo vision of the proposed camera at different observation depths. No significant difference was observed in the completion time. At the observation depths of 30 mm and 90 mm, the number of errors apparently decreased by 62.90% and 51.06%, respectively. CONCLUSIONS: The significant decrease in number of errors shows that the proposed camera has a better stereo vision than a regular camera at both small and large observation depths. It can produce more accurate stereoscopic images at any depth. This will further improve the safety of robot-assisted surgery.