Virtual and Augmented Reality in Ventriculostomy: A Systematic Review.

BACKGROUND: Ventriculostomy, one of the most common neurosurgical procedures, involves inserting a draining catheter into the brain's ventricular system to alleviate excessive cerebrospinal fluid accumulation. Traditionally, this procedure has relied on freehand techniques guided by anatomical landmarks, which have shown a high rate of misplacement. Recent advancements in virtual reality (VR) and augmented reality (AR) technologies have opened up new possibilities in the field. This comprehensive review aims to analyze the existing literature, examine the diverse applications of VR and AR in ventriculostomy procedures, address their limitations, and propose potential future directions. METHODS: A systematic search was conducted in Web of Science and PubMed databases to identify studies employing VR and AR technologies in ventriculostomy procedures. Review papers, non-English records, studies unrelated to VR/AR technologies in ventriculostomy, and supplementary documents were excluded. In total 29 papers were included in the review. RESULTS: The development of various VR and AR systems aimed at enhancing the ventriculostomy procedure are categorized according to the Data, Visualization and View taxonomy. The study investigates the data utilized by these systems, the visualizations employed, and the virtual or augmented environments created. Furthermore, the surgical scenarios and applications of each method, as well as the validation and evaluation metrics used, are discussed. DISCUSSION: The review delves into the fundamental challenges encountered in the implementation of VR and AR systems in ventriculostomy. Additionally, potential future directions and areas for improvement are proposed, addressing the identified limitations and paving the way for further advancements in the field.

A usability analysis of augmented reality and haptics for surgical planning.

PURPOSE: Proper visualization and interaction with complex anatomical data can improve understanding, allowing for more intuitive surgical planning. The goal of our work was to study what the most intuitive yet practical platforms for interacting with 3D medical data are in the context of surgical planning. METHODS: We compared planning using a monitor and mouse, a monitor with a haptic device, and an augmented reality (AR) head-mounted display which uses a gesture-based interaction. To determine the most intuitive system, two user studies, one with novices and one with experts, were conducted. The studies involved planning of three scenarios: (1) heart valve repair, (2) hip tumor resection, and (3) pedicle screw placement. Task completion time, NASA Task Load Index and system-specific questionnaires were used for the evaluation. RESULTS: Both novices and experts preferred the AR system for pedicle screw placement. Novices preferred the haptic system for hip tumor planning, while experts preferred the mouse and keyboard. In the case of heart valve planning, novices preferred the AR system but there was no clear preference for experts. Both groups reported that AR provides the best spatial depth perception. CONCLUSION: The results of the user studies suggest that different surgical cases may benefit from varying interaction and visualization methods. For example, for planning surgeries with implants and instrumentations, mixed reality could provide better 3D spatial perception, whereas using landmarks for delineating specific targets may be more effective using a traditional 2D interface.

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room.

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

Breamy: An augmented reality mHealth prototype for surgical decision-making in breast cancer.

Breast cancer is one of the most prevalent forms of cancer, affecting approximately one in eight women during their lifetime. Deciding on breast cancer treatment, which includes the choice between surgical options, frequently demands prompt decision-making within an 8-week timeframe. However, many women lack the necessary knowledge and preparation for making informed decisions. Anxiety and unsatisfactory outcomes can result from inadequate decision-making processes, leading to decisional regret and revision surgeries. Shared decision-making and personalized decision aids have shown positive effects on patient satisfaction and treatment outcomes. Here, Breamy, a prototype mobile health application that utilizes augmented reality technology to assist breast cancer patients in making more informed decisions is introduced. Breamy provides 3D visualizations of different surgical procedures, aiming to improve confidence in surgical decision-making, reduce decisional regret, and enhance patient well-being after surgery. To determine the perception of the usefulness of Breamy, data was collected from 166 participants through an online survey. The results suggest that Breamy has the potential to reduce patients' anxiety levels and assist them in decision-making.

SCANED: Siamese collateral assessment network for evaluation of collaterals from ischemic damage.

This study conducts collateral evaluation from ischemic damage using a deep learning-based Siamese network, addressing the challenges associated with a small and imbalanced dataset. The collateral network provides an alternative oxygen and nutrient supply pathway in ischemic stroke cases, influencing treatment decisions. Research in this area focuses on automated collateral assessment using deep learning (DL) methods to expedite decision-making processes and enhance accuracy. Our study employed a 3D ResNet-based Siamese network, referred to as SCANED, to classify collaterals as good/intermediate or poor. Utilizing non-contrast computed tomography (NCCT) images, the network automates collateral identification and assessment by analyzing tissue degeneration around the ischemic site. Relevant features from the left/right hemispheres were extracted, and Euclidean Distance (ED) was employed for similarity measurement. Finally, dichotomized classification of good/intermediate or poor collateral is performed by SCANED using an optimal threshold derived from ROC analysis. SCANED provides a sensitivity of 0.88, a specificity of 0.63, and a weighted F1 score of 0.86 in the dichotomized classification.

Deep learning for collateral evaluation in ischemic stroke with imbalanced data.

PURPOSE: Collateral evaluation is typically done using visual inspection of cerebral images and thus suffers from intra- and inter-rater variability. Large open databases of ischemic stroke patients are rare, limiting the use of deep learning methods in treatment decision-making. METHODS: We adapted a pre-trained EfficientNet B0 network through transfer learning to improve collateral evaluation using slice-based and subject-level classification. Our method uses stacking and overlapping of 2D slices from a patient's 4D computed tomography angiography (CTA) and a majority voting scheme to determine a patient's final collateral grade based on all classified 2D MIPs. Class imbalance is handled in the evaluation process by using the focal loss with class weight to penalize the majority class. RESULTS: We evaluated our method using a nine-fold cross-validation performed with 83 subjects. Mean sensitivity of 0.71, specificity of 0.84, and a weighted F1 score of 0.71 in multi-class (good, intermediate, and poor) classification were obtained. Considering treatment effect, a dichotomized decision is also made for collateral scoring of a subject based on two classes (good/intermediate and poor) which achieves a sensitivity of 0.89 and specificity of 0.96 with a weighted F1 score of 0.95. CONCLUSION: An automatic and robust collateral assessment method that mitigates the issues with the small imbalanced dataset was developed. Computer-aided evaluation of collaterals can help decision-making of ischemic stroke treatment strategy in clinical settings.

Automatic collateral circulation scoring in ischemic stroke using 4D CT angiography with low-rank and sparse matrix decomposition.

PURPOSE: Sufficient collateral blood supply is crucial for favorable outcomes with endovascular treatment. The current practice of collateral scoring relies on visual inspection and thus can suffer from inter and intra-rater inconsistency. We present a robust and automatic method to score cerebral collateral blood supply to aid ischemic stroke treatment decision making. The developed method is based on 4D dynamic CT angiography (CTA) and the ASPECTS scoring protocol. METHODS: The proposed method, ACCESS (Automatic Collateral Circulation Evaluation in iSchemic Stroke), estimates a target patient's unfilled cerebrovasculature in contrast-enhanced CTA using the lack of contrast agent due to clotting. To do so, the fast robust matrix completion algorithm with in-face extended Frank-Wolfe optimization is applied on a cohort of healthy subjects and a target patient, to model the patient's unfilled vessels and the estimated full vasculature as sparse and low-rank components, respectively. The collateral score is computed as the ratio of the unfilled vessels to the full vasculature, mimicking existing clinical protocols. RESULTS: ACCESS was tested with 46 stroke patients and obtained an overall accuracy of 84.78%. The optimal threshold selection was evaluated using a receiver operating characteristics curve with the leave-one-out approach, and a mean area under the curve of 85.39% was obtained. CONCLUSION: ACCESS automates collateral scoring to mitigate the shortcomings of the standard clinical practice. It is a robust approach, which resembles how radiologists score clinical scans, and can be used to help radiologists in clinical decisions of stroke treatment.

MARIN: an open-source mobile augmented reality interactive neuronavigation system.

PURPOSE: Neuronavigation systems making use of augmented reality (AR) have been the focus of much research in the last couple of decades. In recent years, there has been considerable interest in using mobile devices for AR in the operating room (OR). We propose a complete system that performs real-time AR video augmentation on a mobile device in the context of image-guided neurosurgery. METHODS: MARIN (mobile augmented reality interactive neuronavigation system) improves upon the state of the art in terms of performance, allowing real-time augmentation, and interactivity by allowing users to interact with the displayed data. The system was tested in a user study with 17 subjects for qualitative and quantitative evaluation in the context of target localization and brought into the OR for preliminary feasibility tests, where qualitative feedback from surgeons was obtained. RESULTS: The results of the user study showed that MARIN performs significantly better in terms of both time ([Formula: see text]) and accuracy ([Formula: see text]) for the task of target localization in comparison with a traditional image-guided neurosurgery (IGNS) navigation system. Further, MARIN AR visualization was found to be more intuitive and allowed users to estimate target depth more easily. CONCLUSION: MARIN improves upon previously proposed mobile AR neuronavigation systems with its real-time performance, higher accuracy, full integration in the normal workflow and greater interactivity and customizability of the displayed information. The improvement in efficiency and usability over previous systems will facilitate bringing AR into the OR.

Brain Shift in Neuronavigation of Brain Tumors: An Updated Review of Intra-Operative Ultrasound Applications.

Neuronavigation using pre-operative imaging data for neurosurgical guidance is a ubiquitous tool for the planning and resection of oncologic brain disease. These systems are rendered unreliable when brain shift invalidates the patient-image registration. Our previous review in 2015, Brain shift in neuronavigation of brain tumours: A review offered a new taxonomy, classification system, and a historical perspective on the causes, measurement, and pre- and intra-operative compensation of this phenomenon. Here we present an updated review using the same taxonomy and framework, focused on the developments of intra-operative ultrasound-based brain shift research from 2015 to the present (2020). The review was performed using PubMed to identify articles since 2015 with the specific words and phrases: "Brain shift" AND "Ultrasound". Since 2015, the rate of publication of intra-operative ultrasound based articles in the context of brain shift has increased from 2-3 per year to 8-10 per year. This efficient and low-cost technology and increasing comfort among clinicians and researchers have allowed unique avenues of development. Since 2015, there has been a trend towards more mathematical advancements in the field which is often validated on publicly available datasets from early intra-operative ultrasound research, and may not give a just representation to the intra-operative imaging landscape in modern image-guided neurosurgery. Focus on vessel-based registration and virtual and augmented reality paradigms have seen traction, offering new perspectives to overcome some of the different pitfalls of ultrasound based technologies. Unfortunately, clinical adaptation and evaluation has not seen as significant of a publication boost. Brain shift continues to be a highly prevalent pitfall in maintaining accuracy throughout oncologic neurosurgical intervention and continues to be an area of active research. Intra-operative ultrasound continues to show promise as an effective, efficient, and low-cost solution for intra-operative accuracy management. A major drawback of the current research landscape is that mathematical tool validation based on retrospective data outpaces prospective clinical evaluations decreasing the strength of the evidence. The need for newer and more publicly available clinical datasets will be instrumental in more reliable validation of these methods that reflect the modern intra-operative imaging in these procedures.

Cognitive load associations when utilizing auditory display within image-guided neurosurgery.

PURPOSE: The combination of data visualization and auditory display (e.g., sonification) has been shown to increase accuracy, and reduce perceived difficulty, within 3D navigation tasks. While accuracy within such tasks can be measured in real time, subjective impressions about the difficulty of a task are more elusive to obtain. Prior work utilizing electrophysiology (EEG) has found robust support that cognitive load and working memory can be monitored in real time using EEG data. METHODS: In this study, we replicated a 3D navigation task (within the context of image-guided surgery) while recording data pertaining to participants' cognitive load through the use of EEG relative alpha-band weighting data. Specifically, 13 subjects navigated a tracked surgical tool to randomly placed 3D virtual locations on a CT cerebral angiography volume while being aided by visual, aural, or both visual and aural feedback. During the study EEG data were captured from the participants, and after the study a NASA TLX questionnaire was filled out by the subjects. In addition to replicating an existing experimental design on auditory display within image-guided neurosurgery, our primary aim sought to determine whether EEG-based markers of cognitive load mirrored subjective ratings of task difficulty RESULTS : Similar to existing literature, our study found evidence consistent with the hypothesis that auditory display can increase the accuracy of navigating to a specified target. We also found significant differences in cognitive working load across different feedback modalities, but none of which supported the experiments hypotheses. Finally, we found mixed results regarding the relationship between real-time measurements of cognitive workload and a posteriori subjective impressions of task difficulty. CONCLUSIONS: Although we did not find a significant correlation between the subjective and physiological measurements, differences in cognitive working load were found. As well, our study further supports the use of auditory display in image-guided surgery.

Augmented reality mastectomy surgical planning prototype using the HoloLens template for healthcare technology letters.

In breast reconstruction following a single mastectomy, the surgeon needs to choose between tens of available implants to find the one that can reproduce the symmetry of the patient's breasts. However, due to the lack of measurement tools this decision is made purely visually, which means the surgeon has to order multiple implants to confirm the size for every single patient. In this Letter, the authors present an augmented reality application, which enables surgeons to see the shape of the implants, as 3D holograms on the patient's body. They custom developed a two-chamber implant that can gain different shapes and be used to test the system. Furthermore, the system was tested in a user study with 13 subjects. The study showed that subjects were able to do a comparison between real and holographic implants and come to a decision about which should be used. This method can be quicker than the traditional way and eliminates sizer implants from the process. Further advantages of the method include the use of a more accurate, user-friendly device, which is easily extendable as new implants that are on the market can be easily added to the system dataset.

Interaction driven enhancement of depth perception in angiographic volumes.

User interaction has the potential to greatly facilitate the exploration and understanding of 3D medical images for diagnosis and treatment. However, in certain specialized environments such as in an operating room (OR), technical and physical constraints such as the need to enforce strict sterility rules, make interaction challenging. In this paper, we propose to facilitate the intraoperative exploration of angiographic volumes by leveraging the motion of a tracked surgical pointer, a tool that is already manipulated by the surgeon when using a navigation system in the OR. We designed and implemented three interactive rendering techniques based on this principle. The benefit of each of these techniques is compared to its non-interactive counterpart in a psychophysics experiment where 20 medical imaging experts were asked to perform a reaching/targeting task while visualizing a 3D volume of angiographic data. The study showed a significant improvement of the appreciation of local vascular structure when using dynamic techniques, while not having a negative impact on the appreciation of the global structure and only a marginal impact on the execution speed. A qualitative evaluation of the different techniques showed a preference for dynamic chroma-depth in accordance with the objective metrics but a discrepancy between objective and subjective measures for dynamic aerial perspective and shading.

Combining intraoperative ultrasound brain shift correction and augmented reality visualizations: a pilot study of eight cases.

We present our work investigating the feasibility of combining intraoperative ultrasound for brain shift correction and augmented reality (AR) visualization for intraoperative interpretation of patient-specific models in image-guided neurosurgery (IGNS) of brain tumors. We combine two imaging technologies for image-guided brain tumor neurosurgery. Throughout surgical interventions, AR was used to assess different surgical strategies using three-dimensional (3-D) patient-specific models of the patient's cortex, vasculature, and lesion. Ultrasound imaging was acquired intraoperatively, and preoperative images and models were registered to the intraoperative data. The quality and reliability of the AR views were evaluated with both qualitative and quantitative metrics. A pilot study of eight patients demonstrates the feasible combination of these two technologies and their complementary features. In each case, the AR visualizations enabled the surgeon to accurately visualize the anatomy and pathology of interest for an extended period of the intervention. Inaccuracies associated with misregistration, brain shift, and AR were improved in all cases. These results demonstrate the potential of combining ultrasound-based registration with AR to become a useful tool for neurosurgeons to improve intraoperative patient-specific planning by improving the understanding of complex 3-D medical imaging data and prolonging the reliable use of IGNS.

Gesture-based registration correction using a mobile augmented reality image-guided neurosurgery system.

In image-guided neurosurgery, a registration between the patient and their pre-operative images and the tracking of surgical tools enables GPS-like guidance to the surgeon. However, factors such as brainshift, image distortion, and registration error cause the patient-to-image alignment accuracy to degrade throughout the surgical procedure no longer providing accurate guidance. The authors present a gesture-based method for manual registration correction to extend the usage of augmented reality (AR) neuronavigation systems. The authors' method, which makes use of the touchscreen capabilities of a tablet on which the AR navigation view is presented, enables surgeons to compensate for the effects of brainshift, misregistration, or tracking errors. They tested their system in a laboratory user study with ten subjects and found that they were able to achieve a median registration RMS error of 3.51 mm on landmarks around the craniotomy of interest. This is comparable to the level of accuracy attainable with previously proposed methods and currently available commercial systems while being simpler and quicker to use. The method could enable surgeons to quickly and easily compensate for most of the observed shift. Further advantages of their method include its ease of use, its small impact on the surgical workflow and its small-time requirement.

Quantifying attention shifts in augmented reality image-guided neurosurgery.

Image-guided surgery (IGS) has allowed for more minimally invasive procedures, leading to better patient outcomes, reduced risk of infection, less pain, shorter hospital stays and faster recoveries. One drawback that has emerged with IGS is that the surgeon must shift their attention from the patient to the monitor for guidance. Yet both cognitive and motor tasks are negatively affected with attention shifts. Augmented reality (AR), which merges the realworld surgical scene with preoperative virtual patient images and plans, has been proposed as a solution to this drawback. In this work, we studied the impact of two different types of AR IGS set-ups (mobile AR and desktop AR) and traditional navigation on attention shifts for the specific task of craniotomy planning. We found a significant difference in terms of the time taken to perform the task and attention shifts between traditional navigation, but no significant difference between the different AR set-ups. With mobile AR, however, users felt that the system was easier to use and that their performance was better. These results suggest that regardless of where the AR visualisation is shown to the surgeon, AR may reduce attention shifts, leading to more streamlined and focused procedures.

Distance sonification in image-guided neurosurgery.

Image-guided neurosurgery, or neuronavigation, has been used to visualise the location of a surgical probe by mapping the probe location to pre-operative models of a patient's anatomy. One common limitation of this approach is that it requires the surgeon to divert their attention away from the patient and towards the neuronavigation system. In order to improve this type of application, the authors designed a system that sonifies (i.e. provides audible feedback of) distance information between a surgical probe and the location of the anatomy of interest. A user study (n = 15) was completed to determine the utility of sonified distance information within an existing neuronavigation platform (Intraoperative Brain Imaging System (IBIS) Neuronav). The authors' results were consistent with the idea that combining auditory distance cues with existing visual information from image-guided surgery systems may result in greater accuracy when locating specified points on a pre-operative scan, thereby potentially reducing the extent of the required surgical openings, as well as potentially increasing the precision of individual surgical tasks. Further, the authors' results were also consistent with the hypothesis that combining auditory and visual information reduces the perceived difficulty in locating a target location within a three-dimensional volume.

Brain shift in neuronavigation of brain tumors: A review.

PURPOSE: Neuronavigation based on preoperative imaging data is a ubiquitous tool for image guidance in neurosurgery. However, it is rendered unreliable when brain shift invalidates the patient-to-image registration. Many investigators have tried to explain, quantify, and compensate for this phenomenon to allow extended use of neuronavigation systems for the duration of surgery. The purpose of this paper is to present an overview of the work that has been done investigating brain shift. METHODS: A review of the literature dealing with the explanation, quantification and compensation of brain shift is presented. The review is based on a systematic search using relevant keywords and phrases in PubMed. The review is organized based on a developed taxonomy that classifies brain shift as occurring due to physical, surgical or biological factors. RESULTS: This paper gives an overview of the work investigating, quantifying, and compensating for brain shift in neuronavigation while describing the successes, setbacks, and additional needs in the field. An analysis of the literature demonstrates a high variability in the methods used to quantify brain shift as well as a wide range in the measured magnitude of the brain shift, depending on the specifics of the intervention. The analysis indicates the need for additional research to be done in quantifying independent effects of brain shift in order for some of the state of the art compensation methods to become useful. CONCLUSION: This review allows for a thorough understanding of the work investigating brain shift and introduces the needs for future avenues of investigation of the phenomenon.

IBIS: an OR ready open-source platform for image-guided neurosurgery.

PURPOSE: Navigation systems commonly used in neurosurgery suffer from two main drawbacks: (1) their accuracy degrades over the course of the operation and (2) they require the surgeon to mentally map images from the monitor to the patient. In this paper, we introduce the Intraoperative Brain Imaging System (IBIS), an open-source image-guided neurosurgery research platform that implements a novel workflow where navigation accuracy is improved using tracked intraoperative ultrasound (iUS) and the visualization of navigation information is facilitated through the use of augmented reality (AR). METHODS: The IBIS platform allows a surgeon to capture tracked iUS images and use them to automatically update preoperative patient models and plans through fast GPU-based reconstruction and registration methods. Navigation, resection and iUS-based brain shift correction can all be performed using an AR view. IBIS has an intuitive graphical user interface for the calibration of a US probe, a surgical pointer as well as video devices used for AR (e.g., a surgical microscope). RESULTS: The components of IBIS have been validated in the laboratory and evaluated in the operating room. Image-to-patient registration accuracy is on the order of [Formula: see text] and can be improved with iUS to a median target registration error of 2.54 mm. The accuracy of the US probe calibration is between 0.49 and 0.82 mm. The average reprojection error of the AR system is [Formula: see text]. The system has been used in the operating room for various types of surgery, including brain tumor resection, vascular neurosurgery, spine surgery and DBS electrode implantation. CONCLUSIONS: The IBIS platform is a validated system that allows researchers to quickly bring the results of their work into the operating room for evaluation. It is the first open-source navigation system to provide a complete solution for AR visualization.

Augmented reality in neurovascular surgery: feasibility and first uses in the operating room.

PURPOSE: The aim of this report is to present a prototype augmented reality (AR) intra-operative brain imaging system. We present our experience of using this new neuronavigation system in neurovascular surgery and discuss the feasibility of this technology for aneurysms, arteriovenous malformations (AVMs), and arteriovenous fistulae (AVFs). METHODS: We developed an augmented reality system that uses an external camera to capture the live view of the patient on the operating room table and to merge this view with pre-operative volume-rendered vessels. We have extensively tested the system in the laboratory and have used the system in four surgical cases: one aneurysm, two AVMs and one AVF case. RESULTS: The developed AR neuronavigation system allows for precise patient-to-image registration and calibration of the camera, resulting in a well-aligned augmented reality view. Initial results suggest that augmented reality is useful for tailoring craniotomies, localizing vessels of interest, and planning resection corridors. CONCLUSION: Augmented reality is a promising technology for neurovascular surgery. However, for more complex anomalies such as AVMs and AVFs, better visualization techniques that allow one to distinguish between arteries and veins and determine the absolute depth of a vessel of interest are needed.