Calibration-free structured-light-based 3D scanning system in laparoscope for robotic surgery.

Accurate 3D shape measurement is crucial for surgical support and alignment in robotic surgery systems. Stereo cameras in laparoscopes offer a potential solution; however, their accuracy in stereo image matching diminishes when the target image has few textures. Although stereo matching with deep learning has gained significant attention, supervised learning requires a large dataset of images with depth annotations, which are scarce for laparoscopes. Thus, there is a strong demand to explore alternative methods for depth reconstruction or annotation for laparoscopes. Active stereo techniques are a promising approach for achieving 3D reconstruction without textures. In this study, a 3D shape reconstruction method is proposed using an ultra-small patterned projector attached to a laparoscopic arm to address these issues. The pattern projector emits a structured light with a grid-like pattern that features node-wise modulation for positional encoding. To scan the target object, multiple images are taken while the projector is in motion, and the relative poses of the projector and a camera are auto-calibrated using a differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using images obtained from a surgical robot and comparing the results with a ground-truth shape obtained from X-ray CT.

A method for chronic and semi-chronic microelectrode array implantation in deep brain structures using image guided neuronavigation.

BACKGROUND: Accurate targeting of brain structures for in-vivo electrophysiological recordings is essential for basic as well as clinical neuroscience research. Although methodologies for precise targeting and recording from the cortical surface are abundant, such protocols are scarce for deep brain structures. NEW METHOD: We have incorporated stable fiducial markers within a custom cranial cap for improved image-guided neuronavigation targeting of subcortical structures in macaque monkeys. Anchor bolt chambers allowed for a minimally invasive entrance into the brain for chronic recordings. A 3D-printed microdrive allowed for semi-chronic applications. RESULTS: We achieved an average Euclidean targeting error of 1.6 mm and a radial error of 1.2 mm over three implantations in two animals. Chronic and semi-chronic implantations allowed for recording of extracellular neuronal activity, with single-neuron activity examples shown from one macaque monkey. COMPARISON WITH EXISTING METHOD(S): Traditional stereotactic methods ignore individual anatomical variability. Our targeting approach allows for a flexible, subject-specific surgical plan with targeting errors lower than what is reported in humans, and equal to or lower than animal models using similar methods. Utilizing an anchor bolt as a chamber reduced the craniotomy size needed for electrode implantation, compared to conventional large access chambers which are prone to infection. Installation of an in-house, 3D-printed, screw-to-mount mechanical microdrive is in contrast to existing semi-chronic methods requiring fabrication, assembly, and installation of complex parts. CONCLUSIONS: Leveraging commercially available tools for implantation, our protocol decreases the risk of infection from open craniotomies, and improves the accuracy of chronic electrode implantations targeting deep brain structures in large animal models.

X-ray image decomposition for improved magnetic navigation.

PURPOSE: Existing field generators (FGs) for magnetic tracking cause severe image artifacts in X-ray images. While FG with radio-lucent components significantly reduces these imaging artifacts, traces of coils and electronic components may still be visible to trained professionals. In the context of X-ray-guided interventions using magnetic tracking, we introduce a learning-based approach to further reduce traces of field-generator components from X-ray images to improve visualization and image guidance. METHODS: An adversarial decomposition network was trained to separate the residual FG components (including fiducial points introduced for pose estimation), from the X-ray images. The main novelty of our approach lies in the proposed data synthesis method, which combines existing 2D patient chest X-ray and FG X-ray images to generate 20,000 synthetic images, along with ground truth (images without the FG) to effectively train the network. RESULTS: For 30 real images of a torso phantom, our enhanced X-ray image after image decomposition obtained an average local PSNR of 35.04 and local SSIM of 0.97, whereas the unenhanced X-ray images averaged a local PSNR of 31.16 and local SSIM of 0.96. CONCLUSION: In this study, we proposed an X-ray image decomposition method to enhance X-ray image for magnetic navigation by removing FG-induced artifacts, using a generative adversarial network. Experiments on both synthetic and real phantom data demonstrated the efficacy of our method.

3D US-CT/MRI registration for percutaneous focal liver tumor ablations.

PURPOSE: US-guided percutaneous focal liver tumor ablations have been considered promising curative treatment techniques. To address cases with invisible or poorly visible tumors, registration of 3D US with CT or MRI is a critical step. By taking advantage of deep learning techniques to efficiently detect representative features in both modalities, we aim to develop a 3D US-CT/MRI registration approach for liver tumor ablations. METHODS: Facilitated by our nnUNet-based 3D US vessel segmentation approach, we propose a coarse-to-fine 3D US-CT/MRI image registration pipeline based on the liver vessel surface and centerlines. Then, phantom, healthy volunteer and patient studies are performed to demonstrate the effectiveness of our proposed registration approach. RESULTS: Our nnUNet-based vessel segmentation model achieved a Dice score of 0.69. In healthy volunteer study, 11 out of 12 3D US-MRI image pairs were successfully registered with an overall centerline distance of 4.03±2.68 mm. Two patient cases achieved target registration errors (TRE) of 4.16 mm and 5.22 mm. CONCLUSION: We proposed a coarse-to-fine 3D US-CT/MRI registration pipeline based on nnUNet vessel segmentation models. Experiments based on healthy volunteers and patient trials demonstrated the effectiveness of our registration workflow. Our code and example data are publicly available in this r epository.

Automatic Plane of Minimal Hiatal Dimensions Extraction From 3D Female Pelvic Floor Ultrasound.

There is an increasing interest in the applications of 3D ultrasound imaging of the pelvic floor to improve the diagnosis, treatment, and surgical planning of female pelvic floor dysfunction (PFD). Pelvic floor biometrics are obtained on an oblique image plane known as the plane of minimal hiatal dimensions (PMHD). Identifying this plane requires the detection of two anatomical landmarks, the pubic symphysis and anorectal angle. The manual detection of the anatomical landmarks and the PMHD in 3D pelvic ultrasound requires expert knowledge of the pelvic floor anatomy, and is challenging, time-consuming, and subject to human error. These challenges have hindered the adoption of such quantitative analysis in the clinic. This work presents an automatic approach to identify the anatomical landmarks and extract the PMHD from 3D pelvic ultrasound volumes. To demonstrate clinical utility and a complete automated clinical task, an automatic segmentation of the levator-ani muscle on the extracted PMHD images was also performed. Experiments using 73 test images of patients during a pelvic muscle resting state showed that this algorithm has the capability to accurately identify the PMHD with an average Dice of 0.89 and an average mean boundary distance of 2.25mm. Further evaluation of the PMHD detection algorithm using 35 images of patients performing pelvic muscle contraction resulted in an average Dice of 0.88 and an average mean boundary distance of 2.75mm. This work had the potential to pave the way towards the adoption of ultrasound in the clinic and development of personalized treatment for PFD.

3D US-Based Evaluation and Optimization of Tumor Coverage for US-Guided Percutaneous Liver Thermal Ablation.

Complete tumor coverage by the thermal ablation zone and with a safety margin (5 or 10 mm) is required to achieve the entire tumor eradication in liver tumor ablation procedures. However, 2D ultrasound (US) imaging has limitations in evaluating the tumor coverage by imaging only one or multiple planes, particularly for cases with multiple inserted applicators or irregular tumor shapes. In this paper, we evaluate the intra-procedural tumor coverage using 3D US imaging and investigate whether it can provide clinically needed information. Using data from 14 cases, we employed surface- and volume-based evaluation metrics to provide information on any uncovered tumor region. For cases with incomplete tumor coverage or uneven ablation margin distribution, we also proposed a novel margin uniformity -based approach to provide quantitative applicator adjustment information for optimization of tumor coverage. Both the surface- and volume-based metrics showed that 5 of 14 cases had incomplete tumor coverage according to the estimated ablation zone. After applying our proposed applicator adjustment approach, the simulated results showed that 92.9% (13 of 14) cases achieved 100% tumor coverage and the remaining case can benefit by increasing the ablation time or power. Our proposed method can evaluate the intra-procedural tumor coverage and intuitively provide applicator adjustment information for the physician. Our 3D US-based method is compatible with the constraints of conventional US-guided ablation procedures and can be easily integrated into the clinical workflow.

3D localization of vena contracta using Doppler ICE imaging in tricuspid valve interventions.

PURPOSE: Tricuspid valve (TV) interventions face the challenge of imaging the anatomy and tools because of the 'TEE-unfriendly' nature of the TV. In edge-to-edge TV repair, a core step is to position the clip perpendicular to the coaptation gap. In this study, we provide a semi-automated method to localize the VC from Doppler intracardiac echo (ICE) imaging in a tracked 3D space, thus providing a pre-mapped location of the coaptation gap to assist device positioning. METHODS: A magnetically tracked ICE probe with Doppler imaging capabilities is employed in this study for imaging three patient-specific TVs placed in a pulsatile heart phantom. For each of the valves, the ICE probe is positioned to image the maximum regurgitant flow for five cardiac cycles. An algorithm then extracts the regurgitation imaging and computes the exact location of the vena contracta on the image. RESULTS: Across the three pathological, patient-specific valves, the average distance error between the detected VC and the ground truth model is [Formula: see text]mm. For each of the valves, one case represented the outlier where the algorithm misidentified the vena contracta to be near the annulus. In such cases, it is recommended to retake the five-second imaging data. CONCLUSION: This study presented a method for ultrasound-based localization of vena contracta in 3D space. Mapping such anatomical landmarks has the potential to assist with device positioning and to simplify tricuspid valve interventions by providing more contextual information to the interventionalists, thus enhancing their spatial awareness. Additionally, ICE can be used to provide live US and Doppler imaging of the complex TV anatomy throughout the procedure.

Toward Fluoro-Free Interventions: Using Radial Intracardiac Ultrasound for Vascular Navigation.

Transcatheter cardiovascular interventions have the advantage of patient safety, reduced surgery time and minimal trauma to the patient's body. Transcathether interventions, which are performed percutaneously, are limited by the lack of direct line of sight with the procedural tools and the patient anatomy. Therefore, such interventional procedures rely heavily on image guidance for navigating toward and delivering therapy at the target site. Vascular navigation via the inferior vena cava, from the groin to the heart, is an imperative part of most transcatheter cardiovascular interventions including heart valve repair surgeries and ablation therapy. Traditionally, the inferior vena cava is navigated using fluoroscopic techniques such as venography and computed tomography venography. These X-ray-based techniques can have detrimental effects on the patient as well as the surgical team, causing increased radiation exposure, leading to risk of cancer, fetal defects and eye cataracts. The use of a heavy lead apron has also been reported to cause back pain and spine issues, thus leading to interventionalist's disc disease. We propose the use of a catheter-based ultrasound augmented with electromagnetic tracking technology to generate a vascular roadmap in real time and perform navigation without harmful radiation. In this pilot study, we used spatially tracked intracardiac echocardiography to reconstruct a vessel from a phantom in a 3-D virtual environment. We illustrate how the proposed ultrasound-based navigation will appear in a virtual environment, by navigating a tracked guidewire within the vessels in the phantom without any radiation-based imaging. The geometric accuracy is assessed using a computed tomography scan of the phantom, with a Dice coefficient of 0.79. The average distance between the surfaces of the two models comes out to be 1.7 ± 1.12 mm.

Towards a First-Person Perspective Mixed Reality Guidance System for Needle Interventions.

While ultrasound (US) guidance has been used during central venous catheterization to reduce complications, including the puncturing of arteries, the rate of such problems remains non-negligible. To further reduce complication rates, mixed-reality systems have been proposed as part of the user interface for such procedures. We demonstrate the use of a surgical navigation system that renders a calibrated US image, and the needle and its trajectory, in a common frame of reference. We compare the effectiveness of this system, whereby images are rendered on a planar monitor and within a head-mounted display (HMD), to the standard-of-care US-only approach, via a phantom-based user study that recruited 31 expert clinicians and 20 medical students. These users performed needle-insertions into a phantom under the three modes of visualization. The success rates were significantly improved under HMD-guidance as compared to US-guidance, for both expert clinicians (94% vs. 70%) and medical students (70% vs. 25%). Users more consistently positioned their needle closer to the center of the vessel's lumen under HMD-guidance compared to US-guidance. The performance of the clinicians when interacting with this monitor system was comparable to using US-only guidance, with no significant difference being observed across any metrics. The results suggest that the use of an HMD to align the clinician's visual and motor fields promotes successful needle guidance, highlighting the importance of continued HMD-guidance research.

A Robust Edge-Preserving Stereo Matching Method for Laparoscopic Images.

Stereo matching has become an active area of research in the field of computer vision. In minimally invasive surgery, stereo matching provides depth information to surgeons, with the potential to increase the safety of surgical procedures, particularly those performed laparoscopically. Many stereo matching methods have been reported to perform well for natural images, but for images acquired during a laparoscopic procedure, they are limited by image characteristics including illumination differences, weak texture content, specular highlights, and occlusions. To overcome these limitations, we propose a robust edge-preserving stereo matching method for laparoscopic images, comprising an efficient sparse-dense feature matching step, left and right image illumination equalization, and refined disparity optimization. We validated the proposed method using both benchmark biological phantoms and surgical stereoscopic data. Experimental results illustrated that, in the presence of heavy illumination differences between image pairs, texture and textureless surfaces, specular highlights and occlusions, our proposed approach consistently obtains a more accurate estimate of the disparity map than state-of-the-art stereo matching methods in terms of robustness and boundary preservation.

Automatic segmentation of the carotid artery and internal jugular vein from 2D ultrasound images for 3D vascular reconstruction.

PURPOSE: In the context of analyzing neck vascular morphology, this work formulates and compares Mask R-CNN and U-Net-based algorithms to automatically segment the carotid artery (CA) and internal jugular vein (IJV) from transverse neck ultrasound (US). METHODS: US scans of the neck vasculature were collected to produce a dataset of 2439 images and their respective manual segmentations. Fourfold cross-validation was employed to train and evaluate Mask RCNN and U-Net models. The U-Net algorithm includes a post-processing step that selects the largest connected segmentation for each class. A Mask R-CNN-based vascular reconstruction pipeline was validated by performing a surface-to-surface distance comparison between US and CT reconstructions from the same patient. RESULTS: The average CA and IJV Dice scores produced by the Mask R-CNN across the evaluation data from all four sets were [Formula: see text] and [Formula: see text]. The average Dice scores produced by the post-processed U-Net were [Formula: see text] and [Formula: see text], for the CA and IJV, respectively. The reconstruction algorithm utilizing the Mask R-CNN was capable of producing accurate 3D reconstructions with majority of US reconstruction surface points being within 2 mm of the CT equivalent. CONCLUSIONS: On average, the Mask R-CNN produced more accurate vascular segmentations compared to U-Net. The Mask R-CNN models were used to produce 3D reconstructed vasculature with a similar accuracy to that of a manually segmented CT scan. This implementation of the Mask R-CNN network enables automatic analysis of the neck vasculature and facilitates 3D vascular reconstruction.

A simple, realistic walled phantom for intravascular and intracardiac applications.

PURPOSE: This work aims to develop a simple, anatomically and haptically realistic vascular phantom, compatible with intravascular and intracardiac ultrasound. The low-cost, dual-layered phantom bridges the gap between traditional wall-only and wall-less phantoms by showing both the vessel wall and surrounding tissue in ultrasound imaging. This phantom can better assist clinical tool training, testing of intravascular devices, blood flow studies, and validation of algorithms for intravascular and intracardiac surgical systems. METHODS: Polyvinyl alcohol cryogel (PVA-c) incorporating a scattering agent was used to obtain vessel and tissue-mimicking materials. Our specific design targeted the inferior vena cava and renal bifurcations which were modelled using CAD software. A custom mould and container were 3D-printed for shaping the desired vessel wall. Three phantoms were prepared by varying both the concentrations of scattering agent as well as the number of freeze-thaw cycles to which the phantom layers were subjected during the manufacturing process. Each phantom was evaluated using ultrasound imaging using the Foresight™ ICE probe. Geometrical validation was provided by comparing CAD design to a CT scan of the phantom. RESULTS: The desired vascular phantom was constructed using 2.5% and 0.05% scattering agent concentration in the vessel and tissue-mimicking layers, respectively. Imaging of the three phantoms showed that increasing the number of freeze-thaw cycles did not significantly enhance the image contrast. Comparison of the US images with their CT equivalents resulted in an average error of 0.9[Formula: see text] for the lumen diameter. CONCLUSION: The phantom is anatomically realistic when imaged with intracardiac ultrasound and provides a smooth lumen for the ultrasound probe and catheter to manoeuvre. The vascular phantom enables validation of intravascular and intracardiac image guidance systems. The simple construction technique also provides a workflow for designing complex, multi-layered arterial phantoms.

The Effects of Positioning on the Volume/Location of the Internal Jugular Vein Using 2-Dimensional Tracked Ultrasound.

OBJECTIVE: To investigate the effects of different positioning on the volume/location of the internal jugular vein (IJV) using 2-dimensional (2D) tracked ultrasound. DESIGN: This was a prospective, observational study. SETTING: Local research institute. PARTICIPANTS: Healthy volunteers. INTERVENTIONS: Twenty healthy volunteers were scanned in the following 6 positions: (1) supine with head neutral, rotated 15 and 30 degrees to the left and (2) 5-, 10-, and 15-degree Trendelenburg position with head neutral. In each position the volunteer's neck was scanned using a 2D ultrasound probe tracked with a magnetic tracker. These spatially tracked 2D images were collected and reconstructed into a 3D volume of the IJV and carotid artery. This 3D ultrasound volume then was segmented to obtain a 3D surface on which measurements and calculations were performed. MEASUREMENTS AND MAIN RESULTS: The measurements included average cross-section area (CSA), CSA along the length of IJV, and average overlap rate. CSA (mm2) in the supine and 5-, 10-, and 15-degree Trendelenburg positions were as follows: 86.7 ± 44.8, 104.3 ± 54.5, 119.1 ± 58.6, and 133.7 ± 53.3 (p < 0.0001). CSA enlarged with the increase of Trendelenburg degree. Neither Trendelenburg position nor head rotation showed a correlation with overlap rate. CONCLUSIONS: Trendelenburg position significantly increased the CSA of the IJV, thus facilitating IJV cannulation. This new 3D reconstruction method permits the creation of a 3D volume through a tracked 2D ultrasound scanning system with image acquisition and integration and may prove useful in providing the user with a "road map" of the vascular anatomy of a patient's neck or other anatomic structures.

Development and Evaluation of an Augmented Reality Ultrasound Guidance System for Spinal Anesthesia: Preliminary Results.

Applications of ultrasound guidance for epidural injections are hindered by poor needle and epidural space visualization. This work presents an augmented reality (AR) ultrasound guidance system that addresses challenges in both needle visualization during navigation and epidural space identification for needle positioning. In this system, (i) B-mode ultrasound and the needle are visualized in a 3-D AR environment for improved navigation, and (ii) A-mode ultrasound, obtained from a custom-made single-element transducer housed at the needle tip, is used to identify the epidural space for improved needle positioning. Performance of the system was evaluated against ultrasound-only guidance in a phantom study with novice operators and an expert anesthesiologist. The procedure success rate was higher with the AR system (100%) than ultrasound-only guidance (57%). The AR system has the potential to improve procedure outcomes in terms of success rate, time, needle path-length and usability.

Accuracy assessment for the co-registration between optical and VIVE head-mounted display tracking.

PURPOSE: We report on the development and accuracy assessment of a hybrid tracking system that integrates optical spatial tracking into a video pass-through head-mounted display. METHODS: The hybrid system uses a dual-tracked co-calibration apparatus to provide a co-registration between the origins of an optical dynamic reference frame and the VIVE Pro controller through a point-based registration. This registration provides the location of optically tracked tools with respect to the VIVE controller's origin and thus the VIVE's tracking system. RESULTS: The positional accuracy was assessed using a CNC machine to collect a grid of points with 25 samples per location. The positional trueness and precision for the hybrid tracking system were [Formula: see text] and [Formula: see text], respectively. The rotational accuracy was assessed through inserting a stylus tracked by all three systems into a hemispherical phantom with cylindrical openings at known angles and collecting 25 samples per cylinder for each system. The rotational trueness and precision for the hybrid tracking system were [Formula: see text] and [Formula: see text], respectively. The difference in position and rotational trueness between the OTS and the hybrid tracking system was [Formula: see text] and [Formula: see text], respectively. CONCLUSIONS: We developed a hybrid tracking system that allows the pose of optically tracked surgical instruments to be known within a first-person HMD visualization system, achieving submillimeter accuracy. This research validated the positional and rotational accuracy of the hybrid tracking system and subsequently the optical tracking and VIVE tracking systems. This work provides a method to determine the position of an optically tracked surgical tool with a surgically acceptable accuracy within a low-cost commercial-grade video pass-through HMD. The hybrid tracking system provides the foundation for the continued development of virtual reality or augmented virtuality surgical navigation systems for training or practicing surgical techniques.

Deep learning approach for automatic out-of-plane needle localisation for semi-automatic ultrasound probe calibration.

The authors present a deep learning algorithm for the automatic centroid localisation of out-of-plane US needle reflections to produce a semi-automatic ultrasound (US) probe calibration algorithm. A convolutional neural network was trained on a dataset of 3825 images at a 6 cm imaging depth to predict the position of the centroid of a needle reflection. Applying the automatic centroid localisation algorithm to a test set of 614 annotated images produced a root mean squared error of 0.62 and 0.74 mm (6.08 and 7.62 pixels) in the axial and lateral directions, respectively. The mean absolute errors associated with the test set were 0.50 ± 0.40 mm and 0.51 ± 0.54 mm (4.9 ± 3.96 pixels and 5.24 ± 5.52 pixels) for the axial and lateral directions, respectively. The trained model was able to produce visually validated US probe calibrations at imaging depths on the range of 4-8 cm, despite being solely trained at 6 cm. This work has automated the pixel localisation required for the guided-US calibration algorithm producing a semi-automatic implementation available open-source through 3D Slicer. The automatic needle centroid localisation improves the usability of the algorithm and has the potential to decrease the fiducial localisation and target registration errors associated with the guided-US calibration method.

Robust, Intrinsic Tracking of a Laparoscopic Ultrasound Probe for Ultrasound-Augmented Laparoscopy.

In situ visualization of laparoscopic ultrasound in both conventional and robot-assisted laparoscopic surgery requires robust and efficient computation of the pose of the laparoscopic ultrasound probe with respect to the laparoscopic camera. Image-based intrinsic methods of computing this relative pose need to overcome challenges due to irregular illumination, partial feature occlusion, and clutter that are unavoidable in practical laparoscopic surgery. In this paper, we propose an accurate image-based method that is robust to partial occlusion of the fiducials and outliers. The method is extended to multi-view imaging model with applications in stereoscopic laparoscopy and robot-assisted surgery. Rather than treating the model-to-image correspondence and pose computation as separate problems, we solve them jointly using the Kalman Filter-based framework that demonstrates video rate running time (~24fps). By keeping the optical tracking measurements as a reference, we demonstrate that the proposed methods result in clinically acceptable tracking accuracy, reaching target registration errors well below 1.5mm on average. In addition, our multi-view tracking method is compared to a conventional stereo triangulation-based pose estimation scheme that commercial optical tracking systems are based on, to experimentally demonstrate its superiority in terms of accuracy. Finally, we qualitatively demonstrate the suitability of our methods for practical laparoscopic applications by conducting a phantom-based experiment.