ReMIND: The Brain Resection Multimodal Imaging Database.

The standard of care for brain tumors is maximal safe surgical resection. Neuronavigation augments the surgeon's ability to achieve this but loses validity as surgery progresses due to brain shift. Moreover, gliomas are often indistinguishable from surrounding healthy brain tissue. Intraoperative magnetic resonance imaging (iMRI) and ultrasound (iUS) help visualize the tumor and brain shift. iUS is faster and easier to incorporate into surgical workflows but offers a lower contrast between tumorous and healthy tissues than iMRI. With the success of data-hungry Artificial Intelligence algorithms in medical image analysis, the benefits of sharing well-curated data cannot be overstated. To this end, we provide the largest publicly available MRI and iUS database of surgically treated brain tumors, including gliomas (n = 92), metastases (n = 11), and others (n = 11). This collection contains 369 preoperative MRI series, 320 3D iUS series, 301 iMRI series, and 356 segmentations collected from 114 consecutive patients at a single institution. This database is expected to help brain shift and image analysis research and neurosurgical training in interpreting iUS and iMRI.

ReMIND: The Brain Resection Multimodal Imaging Database.

The standard of care for brain tumors is maximal safe surgical resection. Neuronavigation augments the surgeon's ability to achieve this but loses validity as surgery progresses due to brain shift. Moreover, gliomas are often indistinguishable from surrounding healthy brain tissue. Intraoperative magnetic resonance imaging (iMRI) and ultrasound (iUS) help visualize the tumor and brain shift. iUS is faster and easier to incorporate into surgical workflows but offers a lower contrast between tumorous and healthy tissues than iMRI. With the success of data-hungry Artificial Intelligence algorithms in medical image analysis, the benefits of sharing well-curated data cannot be overstated. To this end, we provide the largest publicly available MRI and iUS database of surgically treated brain tumors, including gliomas (n=92), metastases (n=11), and others (n=11). This collection contains 369 preoperative MRI series, 320 3D iUS series, 301 iMRI series, and 356 segmentations collected from 114 consecutive patients at a single institution. This database is expected to help brain shift and image analysis research and neurosurgical training in interpreting iUS and iMRI.

A Tool-free Neuronavigation Method based on Single-view Hand Tracking.

This work presents a novel tool-free neuronavigation method that can be used with a single RGB commodity camera. Compared with freehand craniotomy placement methods, the proposed system is more intuitive and less error prone. The proposed method also has several advantages over standard neuronavigation platforms. First, it has a much lower cost, since it doesn't require the use of an optical tracking camera or electromagnetic field generator, which are typically the most expensive parts of a neuronavigation system, making it much more accessible. Second, it requires minimal setup, meaning that it can be performed at the bedside and in circumstances where using a standard neuronavigation system is impractical. Our system relies on machine-learning-based hand pose estimation that acts as a proxy for optical tool tracking, enabling a 3D-3D pre-operative to intra-operative registration. Qualitative assessment from clinical users showed that the concept is clinically relevant. Quantitative assessment showed that on average a target registration error (TRE) of 1.3cm can be achieved. Furthermore, the system is framework-agnostic, meaning that future improvements to hand-tracking frameworks would directly translate to a higher accuracy.

Deep Learning for Detection and Localization of B-Lines in Lung Ultrasound.

Lung ultrasound (LUS) is an important imaging modality used by emergency physicians to assess pulmonary congestion at the patient bedside. B-line artifacts in LUS videos are key findings associated with pulmonary congestion. Not only can the interpretation of LUS be challenging for novice operators, but visual quantification of B-lines remains subject to observer variability. In this work, we investigate the strengths and weaknesses of multiple deep learning approaches for automated B-line detection and localization in LUS videos. We curate and publish, BEDLUS, a new ultrasound dataset comprising 1,419 videos from 113 patients with a total of 15,755 expert-annotated B-lines. Based on this dataset, we present a benchmark of established deep learning methods applied to the task of B-line detection. To pave the way for interpretable quantification of B-lines, we propose a novel "single-point" approach to B-line localization using only the point of origin. Our results show that (a) the area under the receiver operating characteristic curve ranges from 0.864 to 0.955 for the benchmarked detection methods, (b) within this range, the best performance is achieved by models that leverage multiple successive frames as input, and (c) the proposed single-point approach for B-line localization reaches an F 1-score of 0.65, performing on par with the inter-observer agreement. The dataset and developed methods can facilitate further biomedical research on automated interpretation of lung ultrasound with the potential to expand the clinical utility.

SlicerCBM: automatic framework for biomechanical analysis of the brain.

PURPOSE: Brain shift that occurs during neurosurgery disturbs the brain's anatomy. Prediction of the brain shift is essential for accurate localisation of the surgical target. Biomechanical models have been envisaged as a possible tool for such predictions. In this study, we created a framework to automate the workflow for predicting intra-operative brain deformations. METHODS: We created our framework by uniquely combining our meshless total Lagrangian explicit dynamics (MTLED) algorithm for computing soft tissue deformations, open-source software libraries and built-in functions within 3D Slicer, an open-source software package widely used for medical research. Our framework generates the biomechanical brain model from the pre-operative MRI, computes brain deformation using MTLED and outputs results in the form of predicted warped intra-operative MRI. RESULTS: Our framework is used to solve three different neurosurgical brain shift scenarios: craniotomy, tumour resection and electrode placement. We evaluated our framework using nine patients. The average time to construct a patient-specific brain biomechanical model was 3 min, and that to compute deformations ranged from 13 to 23 min. We performed a qualitative evaluation by comparing our predicted intra-operative MRI with the actual intra-operative MRI. For quantitative evaluation, we computed Hausdorff distances between predicted and actual intra-operative ventricle surfaces. For patients with craniotomy and tumour resection, approximately 95% of the nodes on the ventricle surfaces are within two times the original in-plane resolution of the actual surface determined from the intra-operative MRI. CONCLUSION: Our framework provides a broader application of existing solution methods not only in research but also in clinics. We successfully demonstrated the application of our framework by predicting intra-operative deformations in nine patients undergoing neurosurgical procedures.

Mapping Resection Progress by Tool-Tip Tracking during Brain Tumor Surgery for Real-Time Estimation of Residual Tumor.

Surgical resection continues to be the primary initial therapeutic strategy in the treatment of patients with brain tumors. Computerized cranial neuronavigation based on preoperative imaging offers precision guidance during craniotomy and early tumor resection but progressively loses validity with brain shift. Intraoperative MRI (iMRI) and intraoperative ultrasound (iUS) can update the imaging used for guidance and navigation but are limited in terms of temporal and spatial resolution, respectively. We present a system that uses time-stamped tool-tip positions of surgical instruments to generate a map of resection progress with high spatial and temporal accuracy. We evaluate this system and present results from 80 cranial tumor resections. Regions of the preoperative tumor segmentation that are covered by the resection map (True Positive Tracking) and regions of the preoperative tumor segmentation not covered by the resection map (True Negative Tracking) are determined for each case. We compare True Negative Tracking, which estimates the residual tumor, with the actual residual tumor identified using iMRI. We discuss factors that can cause False Positive Tracking and False Negative Tracking, which underestimate and overestimate the residual tumor, respectively. Our method provides good estimates of the residual tumor when there is minimal brain shift, and line-of-sight is maintained. When these conditions are not met, surgeons report that it is still useful for identifying regions of potential residual.

Investigative needle core biopsies for multi-omics in Glioblastoma.

Glioblastoma (GBM) is a primary brain cancer with an abysmal prognosis and few effective therapies. The ability to investigate the tumor microenvironment before and during treatment would greatly enhance both understanding of disease response and progression, as well as the delivery and impact of therapeutics. Stereotactic biopsies are a routine surgical procedure performed primarily for diagnostic histopathologic purposes. The role of investigative biopsies - tissue sampling for the purpose of understanding tumor microenvironmental responses to treatment using integrated multi-modal molecular analyses ('Multi-omics") has yet to be defined. Secondly, it is unknown whether comparatively small tissue samples from brain biopsies can yield sufficient information with such methods. Here we adapt stereotactic needle core biopsy tissue in two separate patients. In the first patient with recurrent GBM we performed highly resolved multi-omics analysis methods including single cell RNA sequencing, spatial-transcriptomics, metabolomics, proteomics, phosphoproteomics, T-cell clonotype analysis, and MHC Class I immunopeptidomics from biopsy tissue that was obtained from a single procedure. In a second patient we analyzed multi-regional core biopsies to decipher spatial and genomic variance. We also investigated the utility of stereotactic biopsies as a method for generating patient derived xenograft models in a separate patient cohort. Dataset integration across modalities showed good correspondence between spatial modalities, highlighted immune cell associated metabolic pathways and revealed poor correlation between RNA expression and the tumor MHC Class I immunopeptidome. In conclusion, stereotactic needle biopsy cores are of sufficient quality to generate multi-omics data, provide data rich insight into a patient's disease process and tumor immune microenvironment and can be of value in evaluating treatment responses. One sentence summary: Integrative multi-omics analysis of stereotactic needle core biopsies in glioblastoma.

Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations.

We introduce MHVAE, a deep hierarchical variational autoencoder (VAE) that synthesizes missing images from various modalities. Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation while having the flexibility to handle incomplete image sets as input. Moreover, adversarial learning is employed to generate sharper images. Extensive experiments are performed on the challenging problem of joint intra-operative ultrasound (iUS) and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for synthesizing missing images, demonstrating the advantage of using a hierarchical latent representation and a principled probabilistic fusion operation. Our code is publicly available.

Learning Expected Appearances for Intraoperative Registration during Neurosurgery.

We present a novel method for intraoperative patient-to-image registration by learning Expected Appearances. Our method uses preoperative imaging to synthesize patient-specific expected views through a surgical microscope for a predicted range of transformations. Our method estimates the camera pose by minimizing the dissimilarity between the intraoperative 2D view through the optical microscope and the synthesized expected texture. In contrast to conventional methods, our approach transfers the processing tasks to the preoperative stage, reducing thereby the impact of low-resolution, distorted, and noisy intraoperative images, that often degrade the registration accuracy. We applied our method in the context of neuronavigation during brain surgery. We evaluated our approach on synthetic data and on retrospective data from 6 clinical cases. Our method outperformed state-of-the-art methods and achieved accuracies that met current clinical standards.

SurfaceNets for Multi-Label Segmentations with Preservation of Sharp Boundaries.

We extend 3D SurfaceNets to generate surfaces of segmented 3D medical images composed of multiple materials represented as indexed labels. Our extension generates smooth, high-quality triangle meshes suitable for rendering and tetrahedralization, preserves topology and sharp boundaries between materials, guarantees a user-specified accuracy, and is fast enough that users can interactively explore the trade-off between accuracy and surface smoothness. We provide open-source code in the form of an extendable C++ library with a simple API, and a Qt and OpenGL-based application that allows users to import or randomly generate multi-label volumes to experiment with surface fairing parameters. In this paper, we describe the basic SurfaceNets algorithm, our extension to handle multiple materials, our method for preserving sharp boundaries between materials, and implementation details used to achieve efficient processing.

Using temporal and structural data to reconstruct 3D cerebral vasculature from a pair of 2D digital subtraction angiography sequences.

PURPOSE: The purpose of this work is to present a new method for reconstructing patient-specific three-dimensional (3D) vasculature of the brain from a pair of digital subtraction angiography (DSA) image sequences from different viewpoints, e.g., from bi-plane angiography. Our long-term goal is to provide high resolution visualization of 3D vasculature with dynamic flow of contrast agent from limited data that is readily available during surgical procedures. The proposed method is the second of a three-stage process composed of 1) augmenting vessel segmentation with vessel radii and timing of the arrival of a bolus of contrast agent, 2) reconstructing a volumetric representation of the augmented vessel data from the augmented 2D segmentations, and 3) generating a 3D model of vessels and flow of contrast agent from the volumetric reconstruction. Unlike previous methods, which are either limited to relatively simple vessel structures or rely on multiple views and/or prior models of the vasculature, our method requires only a single pair of 2D DSA sequences taken from different view directions. METHODS: We developed a new mathematical algorithm that augments vessel centerlines with vessel radii and bolus arrival times derived directly from the 2D DSA sequences to constrain the 3D reconstruction. We validated this method on digital phantoms derived from clinical data and from fractal models of branching tree structures. RESULTS: In standard reconstruction methods, reconstruction by projection of two views into 3D space results in 'ghosting' artifacts, i.e., false 3D structure that occurs where vessels or vessel segments overlap in the 2D images. For the complex vascular of the brain, this ghosting is severe and is a major hurdle for methods that attempt to generate 3D structure from 2D images. We show that our approach reduces ghosting by up to 99% in digital phantoms derived from clinical data. CONCLUSION: Our dramatic reduction in ghosting artifacts in 3D reconstructions from a pair of 2D image sequences is an important step towards generating high resolution 3D vasculature with dynamic flow information from a single DSA sequence acquired using bi-plane angiography.

NousNav: A low-cost neuronavigation system for deployment in lower-resource settings.

PURPOSE: NousNav is a complete low-cost neuronavigation system that aims to democratize access to higher-quality healthcare in lower-resource settings. NousNav's goal is to provide a model for local actors to be able to reproduce, build and operate a fully functional neuronavigation system at an affordable cost. METHODS: NousNav is entirely open source and relies on low-cost off-the-shelf components, which makes it easy to reproduce and deploy in any region. NousNav's software is also specifically devised with the low-resource setting in mind. RESULTS: It offers means for intuitive intraoperative control. The designed interface is also clean and simple. This allows for easy intraoperative use by either the practicing clinician or a nurse. It thus alleviates the need for a dedicated technician for operation. CONCLUSION: A prototype implementation of the design was built. Hardware and algorithms were designed for robustness, ruggedness, modularity, to be standalone and data-agnostic. The built prototype demonstrates feasibility of the objectives.

Automatic framework for patient-specific modelling of tumour resection-induced brain shift.

Our motivation is to enable non-biomechanical engineering specialists to use sophisticated biomechanical models in the clinic to predict tumour resection-induced brain shift, and subsequently know the location of the residual tumour and its boundary. To achieve this goal, we developed a framework for automatically generating and solving patient-specific biomechanical models of the brain. This framework automatically determines patient-specific brain geometry from MRI data, generates patient-specific computational grid, assigns material properties, defines boundary conditions, applies external loads to the anatomical structures, and solves differential equations of nonlinear elasticity using Meshless Total Lagrangian Explicit Dynamics (MTLED) algorithm. We demonstrated the effectiveness and appropriateness of our framework on real clinical cases of tumour resection-induced brain shift.

Double-Uncertainty Guided Spatial and Temporal Consistency Regularization Weighting for Learning-based Abdominal Registration.

In order to tackle the difficulty associated with the ill-posed nature of the image registration problem, regularization is often used to constrain the solution space. For most learning-based registration approaches, the regularization usually has a fixed weight and only constrains the spatial transformation. Such convention has two limitations: (i) Besides the laborious grid search for the optimal fixed weight, the regularization strength of a specific image pair should be associated with the content of the images, thus the "one value fits all" training scheme is not ideal; (ii) Only spatially regularizing the transformation may neglect some informative clues related to the ill-posedness. In this study, we propose a mean-teacher based registration framework, which incorporates an additional temporal consistency regularization term by encouraging the teacher model's prediction to be consistent with that of the student model. More importantly, instead of searching for a fixed weight, the teacher enables automatically adjusting the weights of the spatial regularization and the temporal consistency regularization by taking advantage of the transformation uncertainty and appearance uncertainty. Extensive experiments on the challenging abdominal CT-MRI registration show that our training strategy can promisingly advance the original learning-based method in terms of efficient hyperparameter tuning and a better tradeoff between accuracy and smoothness.

Pose Estimation and Non-Rigid Registration for Augmented Reality During Neurosurgery.

OBJECTIVE: A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. METHODS: Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a non-rigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. RESULTS: We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. CONCLUSION: The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. SIGNIFICANCE: This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.

Computer simulation of tumour resection-induced brain deformation by a meshless approach.

Tumour resection requires precise planning and navigation to maximise tumour removal while simultaneously protecting nearby healthy tissues. Neurosurgeons need to know the location of the remaining tumour after partial tumour removal before continuing with the resection. Our approach to the problem uses biomechanical modelling and computer simulation to compute the brain deformations after the tumour is resected. In this study, we use meshless Total Lagrangian explicit dynamics as the solver. The problem geometry is extracted from the patient-specific magnetic resonance imaging (MRI) data and includes the parenchyma, tumour, cerebrospinal fluid and skull. The appropriate non-linear material formulation is used. Loading is performed by imposing intra-operative conditions of gravity and reaction forces between the tumour and surrounding healthy parenchyma tissues. A finite frictionless sliding contact is enforced between the skull (rigid) and parenchyma. The meshless simulation results are compared to intra-operative MRI sections. We also calculate Hausdorff distances between the computed deformed surfaces (ventricles and tumour cavities) and surfaces observed intra-operatively. Over 80% of points on the ventricle surface and 95% of points on the tumour cavity surface were successfully registered (results within the limits of two times the original in-plane resolution of the intra-operative image). Computed results demonstrate the potential for our method in estimating the tissue deformation and tumour boundary during the resection.

Challenges and Opportunities of Intraoperative 3D Ultrasound With Neuronavigation in Relation to Intraoperative MRI.

INTRODUCTION: Neuronavigation greatly improves the surgeons ability to approach, assess and operate on brain tumors, but tends to lose its accuracy as the surgery progresses and substantial brain shift and deformation occurs. Intraoperative MRI (iMRI) can partially address this problem but is resource intensive and workflow disruptive. Intraoperative ultrasound (iUS) provides real-time information that can be used to update neuronavigation and provide real-time information regarding the resection progress. We describe the intraoperative use of 3D iUS in relation to iMRI, and discuss the challenges and opportunities in its use in neurosurgical practice. METHODS: We performed a retrospective evaluation of patients who underwent image-guided brain tumor resection in which both 3D iUS and iMRI were used. The study was conducted between June 2020 and December 2020 when an extension of a commercially available navigation software was introduced in our practice enabling 3D iUS volumes to be reconstructed from tracked 2D iUS images. For each patient, three or more 3D iUS images were acquired during the procedure, and one iMRI was acquired towards the end. The iUS images included an extradural ultrasound sweep acquired before dural incision (iUS-1), a post-dural opening iUS (iUS-2), and a third iUS acquired immediately before the iMRI acquisition (iUS-3). iUS-1 and preoperative MRI were compared to evaluate the ability of iUS to visualize tumor boundaries and critical anatomic landmarks; iUS-3 and iMRI were compared to evaluate the ability of iUS for predicting residual tumor. RESULTS: Twenty-three patients were included in this study. Fifteen patients had tumors located in eloquent or near eloquent brain regions, the majority of patients had low grade gliomas (11), gross total resection was achieved in 12 patients, postoperative temporary deficits were observed in five patients. In twenty-two iUS was able to define tumor location, tumor margins, and was able to indicate relevant landmarks for orientation and guidance. In sixteen cases, white matter fiber tracts computed from preoperative dMRI were overlaid on the iUS images. In nineteen patients, the EOR (GTR or STR) was predicted by iUS and confirmed by iMRI. The remaining four patients where iUS was not able to evaluate the presence or absence of residual tumor were recurrent cases with a previous surgical cavity that hindered good contact between the US probe and the brainsurface. CONCLUSION: This recent experience at our institution illustrates the practical benefits, challenges, and opportunities of 3D iUS in relation to iMRI.

Estimation of High Framerate Digital Subtraction Angiography Sequences at Low Radiation Dose.

Digital Subtraction Angiography (DSA) provides high resolution image sequences of blood flow through arteries and veins and is considered the gold standard for visualizing cerebrovascular anatomy for neurovascular interventions. However, acquisition frame rates are typically limited to 1-3 fps to reduce radiation exposure, and thus DSA sequences often suffer from stroboscopic effects. We present the first approach that permits generating high frame rate DSA sequences from low frame rate acquisitions eliminating these artifacts without increasing the patient's exposure to radiation. Our approach synthesizes new intermediate frames using a phase-aware Convolutional Neural Network. This network accounts for the non-linear blood flow progression due to vessel geometry and initial velocity of the contrast agent. Our approach out-performs existing methods and was tested on several low frame rate DSA sequences of the human brain resulting in sequences of up to 17 fps with smooth and continuous contrast flow, free of flickering artifacts.

SlicerDMRI: Diffusion MRI and Tractography Research Software for Brain Cancer Surgery Planning and Visualization.

PURPOSE: We present SlicerDMRI, an open-source software suite that enables research using diffusion magnetic resonance imaging (dMRI), the only modality that can map the white matter connections of the living human brain. SlicerDMRI enables analysis and visualization of dMRI data and is aimed at the needs of clinical research users. SlicerDMRI is built upon and deeply integrated with 3D Slicer, a National Institutes of Health-supported open-source platform for medical image informatics, image processing, and three-dimensional visualization. Integration with 3D Slicer provides many features of interest to cancer researchers, such as real-time integration with neuronavigation equipment, intraoperative imaging modalities, and multimodal data fusion. One key application of SlicerDMRI is in neurosurgery research, where brain mapping using dMRI can provide patient-specific maps of critical brain connections as well as insight into the tissue microstructure that surrounds brain tumors. PATIENTS AND METHODS: In this article, we focus on a demonstration of SlicerDMRI as an informatics tool to enable end-to-end dMRI analyses in two retrospective imaging data sets from patients with high-grade glioma. Analyses demonstrated here include conventional diffusion tensor analysis, advanced multifiber tractography, automated identification of critical fiber tracts, and integration of multimodal imagery with dMRI. RESULTS: We illustrate the ability of SlicerDMRI to perform both conventional and advanced dMRI analyses as well as to enable multimodal image analysis and visualization. We provide an overview of the clinical rationale for each analysis along with pointers to the SlicerDMRI tools used in each. CONCLUSION: SlicerDMRI provides open-source and clinician-accessible research software tools for dMRI analysis. SlicerDMRI is available for easy automated installation through the 3D Slicer Extension Manager.

Deformation Aware Augmented Reality for Craniotomy using 3D/2D Non-rigid Registration of Cortical Vessels.

Intra-operative brain shift is a well-known phenomenon that describes non-rigid deformation of brain tissues due to gravity and loss of cerebrospinal fluid among other phenomena. This has a negative influence on surgical outcome that is often based on pre-operative planning where the brain shift is not considered. We present a novel brain-shift aware Augmented Reality method to align pre-operative 3D data onto the deformed brain surface viewed through a surgical microscope. We formulate our non-rigid registration as a Shape-from-Template problem. A pre-operative 3D wire-like deformable model is registered onto a single 2D image of the cortical vessels, which is automatically segmented. This 3D/2D registration drives the underlying brain structures, such as tumors, and compensates for the brain shift in sub-cortical regions. We evaluated our approach on simulated and real data composed of 6 patients. It achieved good quantitative and qualitative results making it suitable for neurosurgical guidance.