Augmented-reality-based surgical navigation for endoscope retrograde cholangiopancreatography: A phantom study.

BACKGROUND: Endoscope retrograde cholangiopancreatography is a standard surgical treatment for gallbladder and pancreatic diseases. However, surgeons is at high risk and require sufficient surgical experience and skills. METHODS: (1) The simultaneous localisation and mapping technique to reconstruct the surgical environment. (2) The preoperative 3D model is transformed into the intraoperative video environment to implement the multi-modal fusion. (3) A framework for virtual-to-real projection based on hand-eye alignment. For the purpose of projecting the 3D model onto the imaging plane of the camera, it uses position data from electromagnetic sensors. RESULTS: Our AR-assisted navigation system can accurately guide physicians, which means a distance of registration error to be restricted to under 5 mm and a projection error of 5.76 ± 2.13, and the intubation procedure is done at 30 frames per second. CONCLUSIONS: Coupled with clinical validation and user studies, both the quantitative and qualitative results indicate that our navigation system has the potential to be highly useful in clinical practice.

Evaluation of Augmented Reality Surgical Navigation in Percutaneous Endoscopic Lumbar Discectomy: Clinical Study.

BACKGROUND: The puncture procedure in percutaneous endoscopic lumbar discectomy (PELD) is non-visual, and the learning curve for PELD is steep. METHODS: An augmented reality surgical navigation (ARSN) system was designed and utilized in PELD. The system possesses three core functionalities: augmented reality (AR) radiograph overlay, AR puncture needle real-time tracking, and AR navigation. We conducted a prospective randomized controlled trial to evaluate its feasibility and effectiveness. A total of 20 patients with lumbar disc herniation treated with PELD were analyzed. Of these, 10 patients were treated with the guidance of ARSN (ARSN group). The remaining 10 patients were treated using C-arm fluoroscopy guidance (control group). RESULTS: The AR radiographs and AR puncture needle were successfully superimposed on the intraoperative videos. The anteroposterior and lateral AR tracking distance errors were 1.55 ± 0.17 mm and 1.78 ± 0.21 mm. The ARSN group exhibited a significant reduction in both the number of puncture attempts (2.0 ± 0.4 vs. 6.9 ± 0.5, p = 0.000) and the number of fluoroscopies (10.6 ± 0.9 vs. 18.5 ± 1.6, p = 0.000) compared with the control group. Complications were not observed in either group. CONCLUSIONS: The results indicate that the clinical application of the ARSN system in PELD is effective and feasible.

Augmented Reality Surgical Navigation in Minimally Invasive Spine Surgery: A Preclinical Study.

BACKGROUND: In minimally invasive spine surgery (MISS), where the surgeon cannot directly see the patient's internal anatomical structure, the implementation of augmented reality (AR) technology may solve this problem. METHODS: We combined AR, artificial intelligence, and optical tracking to enhance the augmented reality minimally invasive spine surgery (AR-MISS) system. The system has three functions: AR radiograph superimposition, AR real-time puncture needle tracking, and AR intraoperative navigation. The three functions of the system were evaluated through beagle animal experiments. RESULTS: The AR radiographs were successfully superimposed on the real intraoperative videos. The anteroposterior (AP) and lateral errors of superimposed AR radiographs were 0.74 ± 0.21 mm and 1.13 ± 0.40 mm, respectively. The puncture needles could be tracked by the AR-MISS system in real time. The AP and lateral errors of the real-time AR needle tracking were 1.26 ± 0.20 mm and 1.22 ± 0.25 mm, respectively. With the help of AR radiographs and AR puncture needles, the puncture procedure could be guided visually by the system in real-time. The anteroposterior and lateral errors of AR-guided puncture were 2.47 ± 0.86 mm and 2.85 ± 1.17 mm, respectively. CONCLUSIONS: The results indicate that the AR-MISS system is accurate and applicable.

Highlight removal for endoscopic images based on accelerated adaptive non-convex RPCA decomposition.

OBJECTIVE: Highlights always occur in endoscopic images due to their special imaging environment. It not only increases the difficulty of observation and diagnosis from surgeons, but also influences the performance of Mixed/Augmented Reality techniques in surgery navigation. METHODS: In this paper, we propose a novel accelerated adaptive non-convex robust principal component analysis method (AANC-RPCA) to remove highlights in endoscopic images. We first detect absolute highlight pixels of the enhanced endoscopic images with adaptive threshold segmentation. The quasi-convex function is proposed to approximate a new non-convex objective function. With detected highlight pixels and quasi-convex function, it introduces gradient to shrink sparse matrix and obtains a faster speed of convergence. Then we divide the image into multiple blocks and perform the parallel computation to enhance the efficiency. Finally, we design a weighted template that decays outward with dilation and linear filtering to reconstruct the endoscopic images. Our approach is almost independent of hyper-parameters and can achieve adaptive decomposition. RESULTS: It has been verified on multiple types of endoscopic images through experiments and clinical blind tests. The results demonstrate that our method can obtain the best performance for the recovered images with more details in a shorter time (about 3-5 times). CONCLUSION: Coupled with the user study, both the quantitative and qualitative results indicate that our approach has the potential to be highly useful in endoscopy images. Compared with the existing highlight removal approaches, our method obtains the SOTA results and has the potential to be applied in the various medical processing processes.

Scene-graph-driven semantic feature matching for monocular digestive endoscopy.

BACKGROUND AND OBJECTIVE: Registration of the preoperative 3D model with the video of the digestive tract is the key task in endoscopy surgical navigation. Accurate 3D reconstruction of soft tissue surfaces is essential to complete registration. However, existing feature matching methods still fall short of desirable performance, due to the soft tissue deformation and smooth but less-textured surface. METHODS: In this paper, we present a new semantic description based on the scene graph to integrate contour features and SIFT features. Firstly, we construct the semantic feature descriptor using the SIFT features and dense points in the contour regions to obtain more dense point feature matching. Secondly, we design a clustering algorithm based on the proposed semantic feature descriptor. Finally, we apply the semantic description to the structure from motion (SfM) reconstruction framework. RESULTS: Our techniques are validated by the phantom tests and real surgery videos. We compare our approaches with other typical methods in contour extraction, feature matching, and SfM reconstruction. On average, the feature matching accuracy reaches 75.6% and improves 16.6% in pose estimation. In addition, 39.8% of sparse points are increased in SfM results, and 35.31% more valid points are obtained for the DenseDescriptorNet training in 3D reconstruction. CONCLUSIONS: The new semantic feature description has the potential to reveal more accurate and dense feature correspondence and provides local semantic information in feature matching. Our experiments on the clinical dataset demonstrate the effectiveness and robustness of the novel approach.

Multi-Modality guidance based surgical navigation for percutaneous endoscopic transforaminal discectomy.

OBJECTIVE: Fluoroscopic guidance is a critical step for the puncture procedure in percutaneous endoscopic transforaminal discectomy (PETD). However, two-dimensional observations of the three-dimensional anatomic structure suffer from the effects of projective simplification. To accurately assess the spatial relations between the patient vertebra tissues and puncture needle, a considerable number of fluoroscopic images from different orientations need to be acquired by the surgeons. This process significantly increases the radiation risk for both the patient and surgeons. METHODS: In this paper, we propose an augmented reality (AR) surgical navigation system for PETD based on multi-modality information, which contains fluoroscopy, optical tracking, and depth camera. To register the fluoroscopic image with the intraoperative video, we design a lightweight non-invasive fiducial with markers and detect the markers based on the deep learning method. It can display the intraoperative video fused with the registered fluoroscopic images. We also present a self-adaptive calibration and transformation method between a 6-DOF optical tracking device and a depth camera, which are in different coordinate systems. RESULTS: With the substantially reduced frequency of fluoroscopy imaging, the system can accurately track and superimpose the virtual puncture needle on fluoroscopy images in real-time. From operating theatre in vivo animal experiments, the results illustrate that the system average positioning accuracy can reach 1.98mm and the orientation accuracy can reach 1.19∘. From the clinical validation results, the system significantly lower the frequency of fluoroscopy imaging (42.7%) and reduce the radiation risk for both the patient and surgeons. CONCLUSION: Coupled with the user study, both the quantitative and qualitative results indicate that our navigation system has the potential to be highly useful in clinical practice. Compared with the existing navigation systems, which are usually equipped with a variety of large and high-cost medical equipments, such as O-arm, cone-beam CT, and robots, our navigation system does not need special equipment and can be implemented with common equipment in the operating room, such as C-arm, desktop, etc., even in small hospitals.

Accurate and robust feature description and dense point-wise matching based on feature fusion for endoscopic images.

Despite the rapid technical advancement of augmented reality (AR) and mixed reality (MR) in minimally invasive surgery (MIS) in recent years, monocular-based 2D/3D reconstruction still remains technically challenging in AR/MR guided surgery navigation nowadays. In principle, soft tissue surface is smooth and watery with sparse texture, specular reflection, and frequent deformation. As a result, we frequently obtain only sparse feature points that give rise to incorrect matching results with conventional image processing methods. To ameliorate, in this paper we enunciate an accurate and robust description and matching method for dense feature points in endoscopic videos. Our new method first extracts contours of the low-rank image sequences based on the adaptive robust principal component analysis (RPCA) decomposition. Then we propose a multi-scale dense geometric feature description approach, which simultaneously extracts dense feature descriptors of the contours in the original Euclidean coordinate space, the accompanying 3D color coordinate space, and the derived curvature-gradient coordinate space. Finally, we devise a new algorithm for both global and local point-wise matching based on feature fusion. For global matching, we employ the fast Fourier transform (FFT) to reduce the dimension of the dense feature descriptors. For local feature point matching, in order to enhance the robustness and accuracy of the matching, we cluster multiple contour points to form "super-point" based on dense feature descriptors and their spatio-temporal continuity. The comprehensive experimental results confirm that our novel approach can overcome the highlight influence, and robustly describe contours from image sequences of soft tissue surfaces. Compared with the state-of-the-art feature point description and matching methods, our analysis framework shows the key advantages of both robustness and accuracy in dense point-wise matching, even when the severe soft tissue deformation occurs. Our new approach is expected to have high potential in 2D/3D reconstruction in endoscopy.

Specular Reflections Removal for Endoscopic Image Sequences With Adaptive-RPCA Decomposition.

Specular reflections (i.e., highlight) always exist in endoscopic images, and they can severely disturb surgeons' observation and judgment. In an augmented reality (AR)-based surgery navigation system, the highlight may also lead to the failure of feature extraction or registration. In this paper, we propose an adaptive robust principal component analysis (Adaptive-RPCA) method to remove the specular reflections in endoscopic image sequences. It can iteratively optimize the sparse part parameter during RPCA decomposition. In this new approach, we first adaptively detect the highlight image based on pixels. With the proposed distance metric algorithm, it then automatically measures the similarity distance between the sparse result image and the detected highlight image. Finally, the low-rank and sparse results are obtained by enforcing the similarity distance between the two types of images to fall within a certain range. Our method has been verified by multiple different types of endoscopic image sequences in minimally invasive surgery (MIS). The experiments and clinical blind tests demonstrate that the new Adaptive-RPCA method can obtain the optimal sparse decomposition parameters directly and can generate robust highlight removal results. Compared with the state-of-the-art approaches, the proposed method not only achieves the better highlight removal results but also can adaptively process image sequences.

Aspirin insensitive thrombophilia: transcript profiling of blood identifies platelet abnormalities and HLA restriction.

Aspirin is the most widely used antiplatelet agent because it is safe, efficient, and inexpensive. However, a significant subset of patients does not exhibit a full inhibition of platelet aggregation, termed 'aspirin resistance' (AR). Several major studies have observed that AR patients have a 4-fold increased risk of myocardial infarction (MI), stroke, and other thrombotic events. Arachidonic acid-stimulated whole blood aggregation was tested in 132 adults at risk for ischemic events, and identified an inadequate response to aspirin therapy in 9 patients (6.8%). Expression profiling of blood RNA by microarray was used to generate new hypotheses about the etiology of AR. Among the differentially expressed genes, there were decreases in several known platelet transcripts, including clusterin (CLU), glycoproteins IIb/IIIa (ITGA2B/3), lipocalin (LCN2), lactoferrin (LTF), and the thrombopoetin receptor (MPL), but with increased mRNA for the T-cell Th1 chemokine CXCL10. There was a strong association of AR with expression of HLA-DRB4 and HLA-DQA1. Similar HLA changes have been linked to autoimmune disorders, particularly antiphospholipid syndrome (APS), in which autoantibodies to phospholipid/protein complexes can trigger platelet activation. Consistent with APS, AR patients exhibited a 30% reduction in platelet counts. Follow-up testing for autoimmune antibodies observed only borderline titers in AR patients. Overall, these results suggest that AR may be related to changes in platelet gene expression creating a hyperreactive platelet, despite antiplatelet therapy. Future studies will focus on determining the protein levels of these differential transcripts in platelets, and the possible involvement of HLA restriction as a contributing factor.