Sensorless volumetric reconstruction of fetal brain freehand ultrasound scans with deep implicit representation.

Three-dimensional (3D) ultrasound imaging has contributed to our understanding of fetal developmental processes by providing rich contextual information of the inherently 3D anatomies. However, its use is limited in clinical settings, due to the high purchasing costs and limited diagnostic practicality. Freehand 2D ultrasound imaging, in contrast, is routinely used in standard obstetric exams, but inherently lacks a 3D representation of the anatomies, which limits its potential for more advanced assessment. Such full representations are challenging to recover even with external tracking devices due to internal fetal movement which is independent from the operator-led trajectory of the probe. Capitalizing on the flexibility offered by freehand 2D ultrasound acquisition, we propose ImplicitVol to reconstruct 3D volumes from non-sensor-tracked 2D ultrasound sweeps. Conventionally, reconstructions are performed on a discrete voxel grid. We, however, employ a deep neural network to represent, for the first time, the reconstructed volume as an implicit function. Specifically, ImplicitVol takes a set of 2D images as input, predicts their locations in 3D space, jointly refines the inferred locations, and learns a full volumetric reconstruction. When testing natively-acquired and volume-sampled 2D ultrasound video sequences collected from different manufacturers, the 3D volumes reconstructed by ImplicitVol show significantly better visual and semantic quality than the existing interpolation-based reconstruction approaches. The inherent continuity of implicit representation also enables ImplicitVol to reconstruct the volume to arbitrarily high resolutions. As formulated, ImplicitVol has the potential to integrate seamlessly into the clinical workflow, while providing richer information for diagnosis and evaluation of the developing brain.

Normative spatiotemporal fetal brain maturation with satisfactory development at 2 years.

Maturation of the human fetal brain should follow precisely scheduled structural growth and folding of the cerebral cortex for optimal postnatal function1. We present a normative digital atlas of fetal brain maturation based on a prospective international cohort of healthy pregnant women2, selected using World Health Organization recommendations for growth standards3. Their fetuses were accurately dated in the first trimester, with satisfactory growth and neurodevelopment from early pregnancy to 2 years of age4,5. The atlas was produced using 1,059 optimal quality, three-dimensional ultrasound brain volumes from 899 of the fetuses and an automated analysis pipeline6-8. The atlas corresponds structurally to published magnetic resonance images9, but with finer anatomical details in deep grey matter. The between-study site variability represented less than 8.0% of the total variance of all brain measures, supporting pooling data from the eight study sites to produce patterns of normative maturation. We have thereby generated an average representation of each cerebral hemisphere between 14 and 31 weeks' gestation with quantification of intracranial volume variability and growth patterns. Emergent asymmetries were detectable from as early as 14 weeks, with peak asymmetries in regions associated with language development and functional lateralization between 20 and 26 weeks' gestation. These patterns were validated in 1,487 three-dimensional brain volumes from 1,295 different fetuses in the same cohort. We provide a unique spatiotemporal benchmark of fetal brain maturation from a large cohort with normative postnatal growth and neurodevelopment.

Challenges for machine learning in clinical translation of big data imaging studies.

Combining deep learning image analysis methods and large-scale imaging datasets offers many opportunities to neuroscience imaging and epidemiology. However, despite these opportunities and the success of deep learning when applied to a range of neuroimaging tasks and domains, significant barriers continue to limit the impact of large-scale datasets and analysis tools. Here, we examine the main challenges and the approaches that have been explored to overcome them. We focus on issues relating to data availability, interpretability, evaluation, and logistical challenges and discuss the problems that still need to be tackled to enable the success of "big data" deep learning approaches beyond research.

STAMP: Simultaneous Training and Model Pruning for low data regimes in medical image segmentation.

Acquisition of high quality manual annotations is vital for the development of segmentation algorithms. However, to create them we require a substantial amount of expert time and knowledge. Large numbers of labels are required to train convolutional neural networks due to the vast number of parameters that must be learned in the optimisation process. Here, we develop the STAMP algorithm to allow the simultaneous training and pruning of a UNet architecture for medical image segmentation with targeted channelwise dropout to make the network robust to the pruning. We demonstrate the technique across segmentation tasks and imaging modalities. It is then shown that, through online pruning, we are able to train networks to have much higher performance than the equivalent standard UNet models while reducing their size by more than 85% in terms of parameters. This has the potential to allow networks to be directly trained on datasets where very low numbers of labels are available.

BEAN: Brain Extraction and Alignment Network for 3D Fetal Neurosonography.

Brain extraction (masking of extra-cerebral tissues) and alignment are fundamental first steps of most neuroimage analysis pipelines. The lack of automated solutions for 3D ultrasound (US) has therefore limited its potential as a neuroimaging modality for studying fetal brain development using routinely acquired scans. In this work, we propose a convolutional neural network (CNN) that accurately and consistently aligns and extracts the fetal brain from minimally pre-processed 3D US scans. Our multi-task CNN, Brain Extraction and Alignment Network (BEAN), consists of two independent branches: 1) a fully-convolutional encoder-decoder branch for brain extraction of unaligned scans, and 2) a two-step regression-based branch for similarity alignment of the brain to a common coordinate space. BEAN was tested on 356 fetal head 3D scans spanning the gestational range of 14 to 30 weeks, significantly outperforming all current alternatives for fetal brain extraction and alignment. BEAN achieved state-of-the-art performance for both tasks, with a mean Dice Similarity Coefficient (DSC) of 0.94 for the brain extraction masks, and a mean DSC of 0.93 for the alignment of the target brain masks. The presented experimental results show that brain structures such as the thalamus, choroid plexus, cavum septum pellucidum, and Sylvian fissure, are consistently aligned throughout the dataset and remain clearly visible when the scans are averaged together. The BEAN implementation and related code can be found under www.github.com/felipemoser/kelluwen.

The impact of transfer learning on 3D deep learning convolutional neural network segmentation of the hippocampus in mild cognitive impairment and Alzheimer disease subjects.

Research on segmentation of the hippocampus in magnetic resonance images through deep learning convolutional neural networks (CNNs) shows promising results, suggesting that these methods can identify small structural abnormalities of the hippocampus, which are among the earliest and most frequent brain changes associated with Alzheimer disease (AD). However, CNNs typically achieve the highest accuracy on datasets acquired from the same domain as the training dataset. Transfer learning allows domain adaptation through further training on a limited dataset. In this study, we applied transfer learning on a network called spatial warping network segmentation (SWANS), developed and trained in a previous study. We used MR images of patients with clinical diagnoses of mild cognitive impairment (MCI) and AD, segmented by two different raters. By using transfer learning techniques, we developed four new models, using different training methods. Testing was performed using 26% of the original dataset, which was excluded from training as a hold-out test set. In addition, 10% of the overall training dataset was used as a hold-out validation set. Results showed that all the new models achieved better hippocampal segmentation quality than the baseline SWANS model (ps < .001), with high similarity to the manual segmentations (mean dice [best model] = 0.878 ± 0.003). The best model was chosen based on visual assessment and volume percentage error (VPE). The increased precision in estimating hippocampal volumes allows the detection of small hippocampal abnormalities already present in the MCI phase (SD = [3.9 ± 0.6]%), which may be crucial for early diagnosis.

Subcortical segmentation of the fetal brain in 3D ultrasound using deep learning.

The quantification of subcortical volume development from 3D fetal ultrasound can provide important diagnostic information during pregnancy monitoring. However, manual segmentation of subcortical structures in ultrasound volumes is time-consuming and challenging due to low soft tissue contrast, speckle and shadowing artifacts. For this reason, we developed a convolutional neural network (CNN) for the automated segmentation of the choroid plexus (CP), lateral posterior ventricle horns (LPVH), cavum septum pellucidum et vergae (CSPV), and cerebellum (CB) from 3D ultrasound. As ground-truth labels are scarce and expensive to obtain, we applied few-shot learning, in which only a small number of manual annotations (n = 9) are used to train a CNN. We compared training a CNN with only a few individually annotated volumes versus many weakly labelled volumes obtained from atlas-based segmentations. This showed that segmentation performance close to intra-observer variability can be obtained with only a handful of manual annotations. Finally, the trained models were applied to a large number (n = 278) of ultrasound image volumes of a diverse, healthy population, obtaining novel US-specific growth curves of the respective structures during the second trimester of gestation.

Learning to map 2D ultrasound images into 3D space with minimal human annotation.

In fetal neurosonography, aligning two-dimensional (2D) ultrasound scans to their corresponding plane in the three-dimensional (3D) space remains a challenging task. In this paper, we propose a convolutional neural network that predicts the position of 2D ultrasound fetal brain scans in 3D atlas space. Instead of purely supervised learning that requires heavy annotations for each 2D scan, we train the model by sampling 2D slices from 3D fetal brain volumes, and target the model to predict the inverse of the sampling process, resembling the idea of self-supervised learning. We propose a model that takes a set of images as input, and learns to compare them in pairs. The pairwise comparison is weighted by the attention module based on its contribution to the prediction, which is learnt implicitly during training. The feature representation for each image is thus computed by incorporating the relative position information to all the other images in the set, and is later used for the final prediction. We benchmark our model on 2D slices sampled from 3D fetal brain volumes at 18-22 weeks' gestational age. Using three evaluation metrics, namely, Euclidean distance, plane angles and normalized cross correlation, which account for both the geometric and appearance discrepancy between the ground-truth and prediction, in all these metrics, our model outperforms a baseline model by as much as 23%, when the number of input images increases. We further demonstrate that our model generalizes to (i) real 2D standard transthalamic plane images, achieving comparable performance as human annotations, as well as (ii) video sequences of 2D freehand fetal brain scans.

Deep learning-based unlearning of dataset bias for MRI harmonisation and confound removal.

Increasingly large MRI neuroimaging datasets are becoming available, including many highly multi-site multi-scanner datasets. Combining the data from the different scanners is vital for increased statistical power; however, this leads to an increase in variance due to nonbiological factors such as the differences in acquisition protocols and hardware, which can mask signals of interest. We propose a deep learning based training scheme, inspired by domain adaptation techniques, which uses an iterative update approach to aim to create scanner-invariant features while simultaneously maintaining performance on the main task of interest, thus reducing the influence of scanner on network predictions. We demonstrate the framework for regression, classification and segmentation tasks with two different network architectures. We show that not only can the framework harmonise many-site datasets but it can also adapt to many data scenarios, including biased datasets and limited training labels. Finally, we show that the framework can be extended for the removal of other known confounds in addition to scanner. The overall framework is therefore flexible and should be applicable to a wide range of neuroimaging studies.

The association between flow and oxygenation and cortical development in fetuses with congenital heart defects using a brain-age prediction algorithm.

OBJECTIVES: Presumably, changes in fetal circulation contribute to the delay in maturation of the cortex in fetuses with congenital heart defect (CHD). The aim of the current study is to analyze fetal brain development based on hemodynamic differences, using novel brain-age prediction software. METHODS: We have performed detailed neurosonography, including acquiring 3D volumes, prospectively in cases with isolated CHD from 20 weeks onwards. An algorithm that assesses the degree of fetal brain-age automatically was used to compare CHD cases to controls. We stratified CHD cases according to flow and oxygenation profiles by lesion physiology and performed subgroup analyses. RESULTS: A total of 616 ultrasound volumes of 162 CHD cases and 75 controls were analyzed. Significant differences in maturation of the cortex were observed in cases with normal blood flow toward the brain (-3.8 days, 95%CI [-5.5; -2.0], P = <.001) and low (-4.0 days, 95% CI [-6.7; -1.2] P = <.05; hypoplastic left heart syndrome[HLHS]) and mixed (-4.4 days, 95%CI [-6.4; -2.5] p = <.001) oxygen saturation in the ascending aorta (TGA) and in cardiac mixing (eg, Fallot) cases. CONCLUSION: The current study shows significant delay in brain-age in TGA and Fallot cases as compared to control cases. However, the small differences found in this study questions the clinical relevance.

Learning patterns of the ageing brain in MRI using deep convolutional networks.

Both normal ageing and neurodegenerative diseases cause morphological changes to the brain. Age-related brain changes are subtle, nonlinear, and spatially and temporally heterogenous, both within a subject and across a population. Machine learning models are particularly suited to capture these patterns and can produce a model that is sensitive to changes of interest, despite the large variety in healthy brain appearance. In this paper, the power of convolutional neural networks (CNNs) and the rich UK Biobank dataset, the largest database currently available, are harnessed to address the problem of predicting brain age. We developed a 3D CNN architecture to predict chronological age, using a training dataset of 12,802 T1-weighted MRI images and a further 6,885 images for testing. The proposed method shows competitive performance on age prediction, but, most importantly, the CNN prediction errors ΔBrainAge=AgePredicted-AgeTrue correlated significantly with many clinical measurements from the UK Biobank in the female and male groups. In addition, having used images from only one imaging modality in this experiment, we examined the relationship between ΔBrainAge and the image-derived phenotypes (IDPs) from all other imaging modalities in the UK Biobank, showing correlations consistent with known patterns of ageing. Furthermore, we show that the use of nonlinearly registered images to train CNNs can lead to the network being driven by artefacts of the registration process and missing subtle indicators of ageing, limiting the clinical relevance. Due to the longitudinal aspect of the UK Biobank study, in the future it will be possible to explore whether the ΔBrainAge from models such as this network were predictive of any health outcomes.

Self-Supervised Ultrasound to MRI Fetal Brain Image Synthesis.

Fetal brain magnetic resonance imaging (MRI) offers exquisite images of the developing brain but is not suitable for second-trimester anomaly screening, for which ultrasound (US) is employed. Although expert sonographers are adept at reading US images, MR images which closely resemble anatomical images are much easier for non-experts to interpret. Thus in this article we propose to generate MR-like images directly from clinical US images. In medical image analysis such a capability is potentially useful as well, for instance for automatic US-MRI registration and fusion. The proposed model is end-to-end trainable and self-supervised without any external annotations. Specifically, based on an assumption that the US and MRI data share a similar anatomical latent space, we first utilise a network to extract the shared latent features, which are then used for MRI synthesis. Since paired data is unavailable for our study (and rare in practice), pixel-level constraints are infeasible to apply. We instead propose to enforce the distributions to be statistically indistinguishable, by adversarial learning in both the image domain and feature space. To regularise the anatomical structures between US and MRI during synthesis, we further propose an adversarial structural constraint. A new cross-modal attention technique is proposed to utilise non-local spatial information, by encouraging multi-modal knowledge fusion and propagation. We extend the approach to consider the case where 3D auxiliary information (e.g., 3D neighbours and a 3D location index) from volumetric data is also available, and show that this improves image synthesis. The proposed approach is evaluated quantitatively and qualitatively with comparison to real fetal MR images and other approaches to synthesis, demonstrating its feasibility of synthesising realistic MR images.

Cortical development in fetuses with congenital heart defects using an automated brain-age prediction algorithm.

INTRODUCTION: Congenital heart defects are associated with neurodevelopmental delay. It is hypothesized that fetuses affected by congenital heart defect have altered cerebral oxygen perfusion and are therefore prone to delay in cortical maturation. The aim of this study was to determine the difference in fetal brain age between consecutive congenital heart defect cases and controls in the second and third trimester using ultrasound. MATERIAL AND METHODS: Since 2014, we have included 90 isolated severe congenital heart defect cases in the Heart And Neurodevelopment (HAND)-study. Every 4 weeks, detailed neurosonography was performed in these fetuses, including the recording of a 3D volume of the fetal brain, from 20 weeks onwards. In all, 75 healthy fetuses underwent the same protocol to serve as a control group. The volumes were analyzed by automated age prediction software which determines gestational age by the assessment of cortical maturation. RESULTS: In total, 477 volumes were analyzed using the age prediction software (199 volumes of 90 congenital heart defect cases; 278 volumes of 75 controls). Of these, 16 (3.2%) volume recordings were excluded because of imaging quality. The age distribution was 19-33 weeks. Mixed model analysis showed that the age predicted by brain maturation was 3 days delayed compared with the control group (P = .002). CONCLUSIONS: This study shows that fetuses with isolated cases of congenital heart defects show some delay in cortical maturation as compared with healthy control cases. The clinical relevance of this small difference is debatable. This finding was consistent throughout pregnancy and did not progress during the third trimester.

Fully-automated alignment of 3D fetal brain ultrasound to a canonical reference space using multi-task learning.

Methods for aligning 3D fetal neurosonography images must be robust to (i) intensity variations, (ii) anatomical and age-specific differences within the fetal population, and (iii) the variations in fetal position. To this end, we propose a multi-task fully convolutional neural network (FCN) architecture to address the problem of 3D fetal brain localization, structural segmentation, and alignment to a referential coordinate system. Instead of treating these tasks as independent problems, we optimize the network by simultaneously learning features shared within the input data pertaining to the correlated tasks, and later branching out into task-specific output streams. Brain alignment is achieved by defining a parametric coordinate system based on skull boundaries, location of the eye sockets, and head pose, as predicted from intracranial structures. This information is used to estimate an affine transformation to align a volumetric image to the skull-based coordinate system. Co-alignment of 140 fetal ultrasound volumes (age range: 26.0 ±â€¯4.4 weeks) was achieved with high brain overlap and low eye localization error, regardless of gestational age or head size. The automatically co-aligned volumes show good structural correspondence between fetal anatomies.

Learning-based prediction of gestational age from ultrasound images of the fetal brain.

We propose an automated framework for predicting gestational age (GA) and neurodevelopmental maturation of a fetus based on 3D ultrasound (US) brain image appearance. Our method capitalizes on age-related sonographic image patterns in conjunction with clinical measurements to develop, for the first time, a predictive age model which improves on the GA-prediction potential of US images. The framework benefits from a manifold surface representation of the fetal head which delineates the inner skull boundary and serves as a common coordinate system based on cranial position. This allows for fast and efficient sampling of anatomically-corresponding brain regions to achieve like-for-like structural comparison of different developmental stages. We develop bespoke features which capture neurosonographic patterns in 3D images, and using a regression forest classifier, we characterize structural brain development both spatially and temporally to capture the natural variation existing in a healthy population (N=447) over an age range of active brain maturation (18-34weeks). On a routine clinical dataset (N=187) our age prediction results strongly correlate with true GA (r=0.98,accurate within±6.10days), confirming the link between maturational progression and neurosonographic activity observable across gestation. Our model also outperforms current clinical methods by ±4.57 days in the third trimester-a period complicated by biological variations in the fetal population. Through feature selection, the model successfully identified the most age-discriminating anatomies over this age range as being the Sylvian fissure, cingulate, and callosal sulci.

Data-driven shape parameterization for segmentation of the right ventricle from 3D+t echocardiography.

Model-based segmentation facilitates the accurate measurement of geometric properties of anatomy from ultrasound images. Regularization of the model surface is typically necessary due to the presence of noisy and incomplete boundaries. When simple regularizers are insufficient, linear basis shape models have been shown to be effective. However, for problems such as right ventricle (RV) segmentation from 3D+t echocardiography, where dense consistent landmarks and complete boundaries are absent, acquiring accurate training surfaces in dense correspondence is difficult. As a solution, this paper presents a framework which performs joint segmentation of multiple 3D+t sequences while simultaneously optimizing an underlying linear basis shape model. In particular, the RV is represented as an explicit continuous surface, and segmentation of all frames is formulated as a single continuous energy minimization problem. Shape information is automatically shared between frames, missing boundaries are implicitly handled, and only coarse surface initializations are necessary. The framework is demonstrated to successfully segment both multiple-view and multiple-subject collections of 3D+t echocardiography sequences, and the results confirm that the linear basis shape model is an effective model constraint. Furthermore, the framework is shown to achieve smaller segmentation errors than a state-of-art commercial semi-automatic RV segmentation package.

Predicting fetal neurodevelopmental age from ultrasound images.

We propose an automated framework for predicting age and neurodevelopmental maturation of a fetus based on 3D ultrasound (US) brain image appearance. A topology-preserving manifold representation of the fetal skull enabled design of bespoke scale-invariant image features. Our regression forest model used these features to learn a mapping from age-related sonographic image patterns to fetal age and development. The Sylvian Fissure was identified as a critical region for accurate age estimation, and restricting the search space to this anatomy improved prediction accuracy on a set of 130 healthy fetuses (error ± 3.8 days; r = 0.98 performing the best current clinical method. Our framework remained robust when applied to a routine clinical population.

Regional variations in fascicle curvatures within a muscle belly change during contraction.

During muscle contraction, the fascicles curve in response to changes in internal pressures within the muscle. Muscle modelling studies have predicted that fascicles curve to different extents in different regions of the muscle and, as such, curvature is expected to vary along and across the muscle belly. In the present study, the local variations in fascicle curvature within the muscle belly were investigated for a range of contractile conditions. B-mode ultrasound scans of the medial and lateral gastrocnemii muscles were collected at five ankle positions-ranging from dorsiflexion to plantarflexion. An automated algorithm was applied to the images in order to extract the local curvatures from the muscle belly regions. Significant variations in fascicle curvature were seen in the superficial-to-deep direction. Curvatures were positive in the superficial layer, negative in the deep layer, and had intermediate values close to zero in the central muscle region. This is indicative of the fascicles following an S-shaped trajectory across the muscle image. The relation between external pressure and curvature regionalization was also investigated by applying elastic compression bandages on the calf. The application of pressure was associated with greater negative curvatures in the distal and central regions of the middle layer, but appeared to have little effect on the superficial and deep layers. The results from this study showed that (1) fascicle curvature increases with contraction level, (2) there is curvature regionalization within the muscle belly, (3) curvature increases with pressure, and (4) fascicles follow an S-shaped trajectory across the muscle images.

Computational methods for quantifying in vivo muscle fascicle curvature from ultrasound images.

Muscle fascicles curve during contraction, and this has been seen using B-mode ultrasound. Curvature can vary along a fascicle, and amongst the fascicles within a muscle. The purpose of this study was to develop an automated method for quantifying curvature across the entirety of an imaged muscle, to test the accuracy of the method against synthetic images of known curvature and noise, and to test the sensitivity of the method to ultrasound probe placement. Both synthetic and ultrasound images were processed using multiscale vessel enhancement filtering to accentuate the muscle fascicles, wavelet-based methods were used to quantify fascicle orientations and curvature distribution grids were produced by quantifying local curvatures for each point within the image. Ultrasound images of ramped isometric contractions of the human medial gastrocnemius were acquired in a test-retest study. The methods enabled distinct curvatures to be determined in different regions of the muscle. The methods were sensitive to kernel sizes during image processing, noise within the image and the variability of probe placements during retesting. Across the physiological range of curvatures and noise, curvatures calculated from validation grids were quantified with a typical standard error of less than 0.026 m⁻¹, and this is about 1% of the maximum curvatures observed in fascicles of contracting muscle.