Multimodal Pediatric Lymphoma Detection using PET and MRI.

Lymphoma is one of the most common types of cancer for children (ages 0 to 19). Due to the reduced radiation exposure, PET/MR systems that allow simultaneous PET and MR imaging have become the standard of care for diagnosing cancers and monitoring tumor response to therapy in the pediatric population. In this work, we developed a multimodal deep learning algorithm for automatic pediatric lymphoma detection using PET and MRI. Through innovative designs such as standardized uptake value (SUV) guided tumor candidate generation, location aware classification model learning and weighted multimodal feature fusion, our algorithm can be effectively trained with limited data and achieved superior tumor detection performance over the state-of-the-art in our experiments.

Convolutional autoencoder based model HistoCAE for segmentation of viable tumor regions in liver whole-slide images.

Liver cancer is one of the leading causes of cancer deaths in Asia and Africa. It is caused by the Hepatocellular carcinoma (HCC) in almost 90% of all cases. HCC is a malignant tumor and the most common histological type of the primary liver cancers. The detection and evaluation of viable tumor regions in HCC present an important clinical significance since it is a key step to assess response of chemoradiotherapy and tumor cell proportion in genetic tests. Recent advances in computer vision, digital pathology and microscopy imaging enable automatic histopathology image analysis for cancer diagnosis. In this paper, we present a multi-resolution deep learning model HistoCAE for viable tumor segmentation in whole-slide liver histopathology images. We propose convolutional autoencoder (CAE) based framework with a customized reconstruction loss function for image reconstruction, followed by a classification module to classify each image patch as tumor versus non-tumor. The resulting patch-based prediction results are spatially combined to generate the final segmentation result for each WSI. Additionally, the spatially organized encoded feature map derived from small image patches is used to compress the gigapixel whole-slide images. Our proposed model presents superior performance to other benchmark models with extensive experiments, suggesting its efficacy for viable tumor area segmentation with liver whole-slide images.

Comparison of Chest Radiograph Interpretations by Artificial Intelligence Algorithm vs Radiology Residents.

Importance: Chest radiography is the most common diagnostic imaging examination performed in emergency departments (EDs). Augmenting clinicians with automated preliminary read assistants could help expedite their workflows, improve accuracy, and reduce the cost of care. Objective: To assess the performance of artificial intelligence (AI) algorithms in realistic radiology workflows by performing an objective comparative evaluation of the preliminary reads of anteroposterior (AP) frontal chest radiographs performed by an AI algorithm and radiology residents. Design, Setting, and Participants: This diagnostic study included a set of 72 findings assembled by clinical experts to constitute a full-fledged preliminary read of AP frontal chest radiographs. A novel deep learning architecture was designed for an AI algorithm to estimate the findings per image. The AI algorithm was trained using a multihospital training data set of 342 126 frontal chest radiographs captured in ED and urgent care settings. The training data were labeled from their associated reports. Image-based F1 score was chosen to optimize the operating point on the receiver operating characteristics (ROC) curve so as to minimize the number of missed findings and overcalls per image read. The performance of the model was compared with that of 5 radiology residents recruited from multiple institutions in the US in an objective study in which a separate data set of 1998 AP frontal chest radiographs was drawn from a hospital source representative of realistic preliminary reads in inpatient and ED settings. A triple consensus with adjudication process was used to derive the ground truth labels for the study data set. The performance of AI algorithm and radiology residents was assessed by comparing their reads with ground truth findings. All studies were conducted through a web-based clinical study application system. The triple consensus data set was collected between February and October 2018. The comparison study was preformed between January and October 2019. Data were analyzed from October to February 2020. After the first round of reviews, further analysis of the data was performed from March to July 2020. Main Outcomes and Measures: The learning performance of the AI algorithm was judged using the conventional ROC curve and the area under the curve (AUC) during training and field testing on the study data set. For the AI algorithm and radiology residents, the individual finding label performance was measured using the conventional measures of label-based sensitivity, specificity, and positive predictive value (PPV). In addition, the agreement with the ground truth on the assignment of findings to images was measured using the pooled κ statistic. The preliminary read performance was recorded for AI algorithm and radiology residents using new measures of mean image-based sensitivity, specificity, and PPV designed for recording the fraction of misses and overcalls on a per image basis. The 1-sided analysis of variance test was used to compare the means of each group (AI algorithm vs radiology residents) using the F distribution, and the null hypothesis was that the groups would have similar means. Results: The trained AI algorithm achieved a mean AUC across labels of 0.807 (weighted mean AUC, 0.841) after training. On the study data set, which had a different prevalence distribution, the mean AUC achieved was 0.772 (weighted mean AUC, 0.865). The interrater agreement with ground truth finding labels for AI algorithm predictions had pooled κ value of 0.544, and the pooled κ for radiology residents was 0.585. For the preliminary read performance, the analysis of variance test was used to compare the distributions of AI algorithm and radiology residents' mean image-based sensitivity, PPV, and specificity. The mean image-based sensitivity for AI algorithm was 0.716 (95% CI, 0.704-0.729) and for radiology residents was 0.720 (95% CI, 0.709-0.732) (P = .66), while the PPV was 0.730 (95% CI, 0.718-0.742) for the AI algorithm and 0.682 (95% CI, 0.670-0.694) for the radiology residents (P < .001), and specificity was 0.980 (95% CI, 0.980-0.981) for the AI algorithm and 0.973 (95% CI, 0.971-0.974) for the radiology residents (P < .001). Conclusions and Relevance: These findings suggest that it is possible to build AI algorithms that reach and exceed the mean level of performance of third-year radiology residents for full-fledged preliminary read of AP frontal chest radiographs. This diagnostic study also found that while the more complex findings would still benefit from expert overreads, the performance of AI algorithms was associated with the amount of data available for training rather than the level of difficulty of interpretation of the finding. Integrating such AI systems in radiology workflows for preliminary interpretations has the potential to expedite existing radiology workflows and address resource scarcity while improving overall accuracy and reducing the cost of care.

Combining Deep Learning and Knowledge-driven Reasoning for Chest X-Ray Findings Detection.

The application of deep learning algorithms in medical imaging analysis is a steadily growing research area. While deep learning methods are thriving in the medical domain, they seldom utilize the rich knowledge associated with connected radiology reports. The knowledge derived from these reports can be utilized to enhance the performance of deep learning models. In this work, we used a comprehensive chest X-ray findings vocabulary to automatically annotate an extensive collection of chest X-rays using associated radiology reports and a vocabulary-driven concept annotation algorithm. The annotated X-rays are used to train a deep neural network classifier for finding detection. Finally, we developed a knowledge-driven reasoning algorithm that leverages knowledge learned from X-ray reports to improve upon the deep learning module's performance on finding detection. Our results suggest that combining deep learning and knowledge from radiology reports in a hybrid framework can significantly enhance overall performance in the CXR finding detection.

Building medical image classifiers with very limited data using segmentation networks.

Deep learning has shown promising results in medical image analysis, however, the lack of very large annotated datasets confines its full potential. Although transfer learning with ImageNet pre-trained classification models can alleviate the problem, constrained image sizes and model complexities can lead to unnecessary increase in computational cost and decrease in performance. As many common morphological features are usually shared by different classification tasks of an organ, it is greatly beneficial if we can extract such features to improve classification with limited samples. Therefore, inspired by the idea of curriculum learning, we propose a strategy for building medical image classifiers using features from segmentation networks. By using a segmentation network pre-trained on similar data as the classification task, the machine can first learn the simpler shape and structural concepts before tackling the actual classification problem which usually involves more complicated concepts. Using our proposed framework on a 3D three-class brain tumor type classification problem, we achieved 82% accuracy on 191 testing samples with 91 training samples. When applying to a 2D nine-class cardiac semantic level classification problem, we achieved 86% accuracy on 263 testing samples with 108 training samples. Comparisons with ImageNet pre-trained classifiers and classifiers trained from scratch are presented.

A Meshfree Representation for Cardiac Medical Image Computing.

The prominent advantage of meshfree method, is the way to build the representation of computational domain, based on the nodal points without any explicit meshing connectivity. Therefore, meshfree method can conveniently process the numerical computation inside interested domains with large deformation or inhomogeneity. In this paper, we adopt the idea of meshfree representation into cardiac medical image analysis in order to overcome the difficulties caused by large deformation and inhomogeneous materials of the heart. In our implementation, as element-free Galerkin method can efficiently build a meshfree representation using its shape function with moving least square fitting, we apply this meshfree method to handle large deformation or inhomogeneity for solving cardiac segmentation and motion tracking problems. We evaluate the performance of meshfree representation on a synthetic heart data and an in-vivo cardiac MRI image sequence. Results showed that the error of our framework against the ground truth was 0.1189 ± 0.0672 while the error of the traditional FEM was 0.1793 ± 0.1166. The proposed framework has minimal consistency constraints, handling large deformation and material discontinuities are simple and efficient, and it provides a way to avoid the complicated meshing procedures while preserving the accuracy with a relatively small number of nodes.

Pancreatic Tumor Growth Prediction With Elastic-Growth Decomposition, Image-Derived Motion, and FDM-FEM Coupling.

Pancreatic neuroendocrine tumors are abnormal growths of hormone-producing cells in the pancreas. Unlike the brain which is protected by the skull, the pancreas can be significantly deformed by its surrounding organs. Consequently, the tumor shape differences observable from images at different time points arise from both tumor growth and pancreatic motion, and tumor growth model personalization may be compromised if such motion is ignored. Therefore, we incorporate pancreatic motion information derived from deformable image registration in model personalization. For more accurate mechanical interactions between tumor growth and pancreatic motion, elastic-growth decomposition is used with a hyperelastic constitutive law to model the mass effect, which allows growth modeling while conserving the mechanical properties. Furthermore, a way of coupling the finite difference method and the finite element method is proposed to greatly reduce the computation time. With both 2-[18F]-fluoro-2-deoxy-D-glucose positron emission tomographic and contrast-enhanced computed tomographic images, functional, structural, and motion data are combined for a patient-specific model. Experiments on synthetic and clinical data show the importance of image-derived motion on estimating pathophysiologically plausible mechanical properties and the promising performance of our framework. From seven patient data sets, the recall, precision, Dice coefficient, relative volume difference, and average surface distance between the personalized tumor growth simulations and the measurements were 83.2 ±8.8%, 86.9 ±8.3%, 84.4 ±4.0%, 13.9 ±9.8%, and 0.6 ±0.1 mm, respectively.

Regional infarction identification from cardiac CT images: a computer-aided biomechanical approach.

PURPOSE: Regional infarction identification is important for heart disease diagnosis and management, and myocardial deformation has been shown to be effective for this purpose. Although tagged and strain-encoded MR images can provide such measurements, they are uncommon in clinical routine. On the contrary, cardiac CT images are more available with lower costs, but they only provide motion of cardiac boundaries and additional constraints are required to obtain the myocardial strains. The goal of this study is to verify the potential of contrast-enhanced CT images on computer-aided regional infarction identification. METHODS: We propose a biomechanical approach combined with machine learning algorithms. A hyperelastic biomechanical model is used with deformable image registration to estimate 3D myocardial strains from CT images. The regional strains and CT image intensities are input to a classifier for regional infarction identification. Cross-validations on ten canine image sequences with artificially induced infarctions were used to study the performances of using different feature combinations and machine learning algorithms. RESULTS: Radial strain, circumferential strain, first principal strain, and image intensity were shown to be discriminative features. The highest identification accuracy ([Formula: see text] %) was achieved when combining radial strain with image intensity. Random forests gave better results than support vector machines on less discriminative features. Random forests also performed better when all strains were used together. CONCLUSION: Although CT images cannot directly measure myocardial deformation, with the use of a biomechanical model, the estimated strains can provide promising identification results especially when combined with CT image intensity.

Pancreatic Tumor Growth Prediction with Multiplicative Growth and Image-Derived Motion.

Pancreatic neuroendocrine tumors are abnormal growths of hormone-producing cells in the pancreas. Different from the brain in the skull, the pancreas in the abdomen can be largely deformed by the body posture and the surrounding organs. In consequence, both tumor growth and pancreatic motion attribute to the tumor shape difference observable from images. As images at different time points are used to personalize the tumor growth model, the prediction accuracy may be reduced if such motion is ignored. Therefore, we incorporate the image-derived pancreatic motion to tumor growth personalization. For realistic mechanical interactions, the multiplicative growth decomposition is used with a hyperelastic constitutive law to model tumor mass effect, which allows growth modeling without compromising the mechanical accuracy. With also the FDG-PET and contrast-enhanced CT images, the functional, structural, and motion data are combined for a more patient-specific model. Experiments on synthetic and clinical data show the importance of image-derived motion on estimating physiologically plausible mechanical properties and the promising performance of our framework. From six patient data sets, the recall, precision, Dice coefficient, relative volume difference, and average surface distance were 89.8 ± 3.5%, 85.6 ± 7.5%, 87.4 ± 3.6%, 9.7 ± 7.2%, and 0.6 ± 0.2 mm, respectively.

Tumor growth prediction with reaction-diffusion and hyperelastic biomechanical model by physiological data fusion.

The goal of tumor growth prediction is to model the tumor growth process, which can be achieved by physiological modeling and model personalization from clinical measurements. Although image-driven frameworks have been proposed with promising results, several issues such as infinitesimal strain assumptions, complicated personalization procedures, and the lack of functional information, may limit their prediction accuracy. In view of these issues, we propose a framework for pancreatic neuroendocrine tumor growth prediction, which comprises a FEM-based tumor growth model with coupled reaction-diffusion equation and nonlinear biomechanics. Physiological data fusion of structural and functional images is used to improve the subject-specificity of model personalization, and a derivative-free global optimization algorithm is adopted to facilitate the complicated model and accommodate flexible choices of objective functions. With this flexibility, we propose an objective function accounting for both the tumor volume difference and the root-mean-squared error of intracellular volume fractions. Experiments were performed on synthetic and clinical data to verify the parameter estimation capability and the prediction performance. Comparisons of using different biomechanical models and objective functions were also performed. From the experimental results of eight patient data sets, the average recall, precision, Dice coefficient, and relative volume difference between predicted and measured tumor volumes were 84.5 ± 6.9%, 85.8 ± 8.2%, 84.6 ± 1.7%, and 14.2 ± 8.4%, respectively.

Velocity-based cardiac contractility personalization from images using derivative-free optimization.

Model personalization is a key aspect for biophysical models to impact clinical practice, and cardiac contractility personalization from medical images is a major step in this direction. Existing gradient-based optimization approaches show promising results of identifying the maximum contractility from images, but the contraction and relaxation rates are not accounted for. A main reason is the limited choices of objective functions when their gradients are required. For complicated cardiac models, analytical evaluations of gradients are very difficult if not impossible, and finite difference approximations are computationally expensive and may introduce numerical difficulties. By removing such limitations with derivative-free optimization, we found that a velocity-based objective function can properly identify regional maximum contraction stresses, contraction rates, and relaxation rates simultaneously with intact model complexity. Experiments on synthetic data show that the parameters are better identified using the velocity-based objective function than its position-based counterpart, and the proposed framework is insensitive to initial parameters with the adopted derivative-free optimization algorithm. Experiments on clinical data show that the framework can provide personalized contractility parameters which are consistent with the underlying physiologies of the patients and healthy volunteers.

Tumor growth prediction with hyperelastic biomechanical model, physiological data fusion, and nonlinear optimization.

Tumor growth prediction is usually achieved by physiological modeling and model personalization from clinical measurements. Although image-based frameworks have been proposed with promising results, different issues such as infinitesimal strain assumption, complicated optimization procedures, and lack of functional information, may limit the prediction performance. Therefore, we propose a framework which comprises a hyperelastic biomechanical model for better physiological plausibility, gradient-free nonlinear optimization for more flexible choices of models and objective functions, and physiological data fusion of structural and functional images for better subject-specificity. Experiments were performed on synthetic and clinical data to verify parameter estimation capability and prediction performance of the framework. Comparisons of using different biomechanical models and objective functions were also performed. From the experimental results on eight patient data sets, the recall, precision, and relative volume difference (RVD) between predicted and measured tumor volumes are 84.85 ± 6.15%, 87.08 ± 7.83%, and 13.81 ± 6.64% respectively.

Transmural imaging of ventricular action potentials and post-infarction scars in swine hearts.

The problem of using surface data to reconstruct transmural electrophysiological (EP) signals is intrinsically ill-posed without a unique solution in its unconstrained form. Incorporating physiological spatiotemporal priors through probabilistic integration of dynamic EP models, we have previously developed a Bayesian approach to transmural electrophysiological imaging (TEPI) using body-surface electrocardiograms. In this study, we generalize TEPI to using electrical signals collected from heart surfaces, and we test its feasibility on two pre-clinical swine models provided through the STACOM 2011 EP simulation Challenge. Since this new application of TEPI does not require whole-body imaging, there may be more immediate potential in EP laboratories where it could utilize catheter mapping data and produce transmural information for therapy guidance. Another focus of this study is to investigate the consistency among three modalities in delineating scar after myocardial infarction: TEPI, electroanatomical voltage mapping (EAVM), and magnetic resonance imaging (MRI). Our preliminary data demonstrate that, compared to the low-voltage scar area in EAVM, the 3-D electrical scar volume detected by TEPI is more consistent with anatomical scar volume delineated in MRI. Furthermore, TEPI could complement anatomical imaging by providing EP functional features related to both scar and healthy tissue.

Strain-based regional nonlinear cardiac material properties estimation from medical images.

Model personalization is essential for model-based surgical planning and treatment assessment. As alteration in material elasticity is a fundamental cause to various cardiac pathologies, estimation of material properties is important to model personalization. Although the myocardium is heterogeneous, hyperelastic, and orthotropic, existing image-based estimation frameworks treat the tissue as either heterogeneous but linear, or hyperelastic but homogeneous. In view of these, we present a physiology-based framework for estimating regional, hyperelastic, and orthotropic material properties. A cardiac physiological model is adopted to describe the macroscopic cardiac physiology. By using a strain-based objective function which properly reflects the change of material constants, the regional material properties of a hyperelastic and orthotropic constitutive law are estimated using derivative-free optimization. Experiments were performed on synthetic and real data to show the characteristics of the framework.

Personalization of cardiac motion and contractility from images using variational data assimilation.

Personalization is a key aspect of biophysical models in order to impact clinical practice. In this paper, we propose a personalization method of electromechanical models of the heart from cine-MR images based on the adjoint method. After estimation of electrophysiological parameters, the cardiac motion is estimated based on a proactive electromechanical model. Then cardiac contractilities on two or three regions are estimated by minimizing the discrepancy between measured and simulation motion. Evaluation of the method on three patients with infarcted or dilated myocardium is provided.

A comparative study of physiological models on cardiac deformation recovery: effects of biomechanical constraints.

Cardiac deformation recovery is to recover quantitative subject-specific myocardial deformation from imaging data. In the last decade, cardiac physiological models derived from anatomy, biomechanics, and cardiac electrophysiology have become increasingly popular in constraining the recovery problems because of their physiological meaningfulness. Although physiological models with various electrical and biomechanical components have been adopted by different frameworks and have exhibited promising results, these models have not been systematically compared under the same recovery framework, input data, and experimental setups. As different models comprise varying physiological plausibilities and complexities, comparisons under the same settings can aid choosing the proper models for specific goals and available resources. In this paper, under a state-space filtering framework for statistically optimal couplings between models and image data, we compare the performances of six different cardiac physiological models with different biomechanical constraints. Experiments were performed on synthetic data for quantitative comparisons, and on clinical data for their capabilities in identifying pathological situations.

Physiological fusion of functional and structural images for cardiac deformation recovery.

The recent advances in meaningful constraining models have resulted in increasingly useful quantitative information recovered from cardiac images. Nevertheless, as most frameworks utilize either functional or structural images, the analyses cannot benefit from the complementary information provided by the other image sources. To better characterize subject-specific cardiac physiology and pathology, data fusion of multiple image sources is essential. Traditional image fusion strategies are performed by fusing information of commensurate images through various mathematical operators. Nevertheless, when image data are dissimilar in physical nature and spatiotemporal quantity, such approaches may not provide meaningful connections between different data. In fact, as different image sources provide partial measurements of the same cardiac system dynamics, it is more natural and suitable to utilize cardiac physiological models for the fusions. Therefore, we propose to use the cardiac physiome model as the central link to fuse functional and structural images for more subject-specific cardiac deformation recovery through state-space filtering. Experiments were performed on synthetic and real data for the characteristics and potential clinical applicability of our framework, and the results show an increase of the overall subject specificity of the recovered deformations.

Noninvasive computational imaging of cardiac electrophysiology for 3-D infarct.

Myocardial infarction (MI) creates electrophysiologically altered substrates that are responsible for ventricular arrhythmias, such as tachycardia and fibrillation. The presence, size, location, and composition of infarct scar bear significant prognostic and therapeutic implications for individual subjects. We have developed a statistical physiological model-constrained framework that uses noninvasive body-surface-potential data and tomographic images to estimate subject-specific transmembrane-potential (TMP) dynamics inside the 3-D myocardium. In this paper, we adapt this framework for the purpose of noninvasive imaging, detection, and quantification of 3-D scar mass for postMI patients: the framework requires no prior knowledge of MI and converges to final subject-specific TMP estimates after several passes of estimation with intermediate feedback; based on the primary features of the estimated spatiotemporal TMP dynamics, we provide 3-D imaging of scar tissue and quantitative evaluation of scar location and extent. Phantom experiments were performed on a computational model of realistic heart-torso geometry, considering 87 transmural infarct scars of different sizes and locations inside the myocardium, and 12 compact infarct scars (extent between 10% and 30%) at different transmural depths. Real-data experiments were carried out on BSP and magnetic resonance imaging (MRI) data from four postMI patients, validated by gold standards and existing results. This framework shows unique advantage of noninvasive, quantitative, computational imaging of subject-specific TMP dynamics and infarct mass of the 3-D myocardium, with the potential to reflect details in the spatial structure and tissue composition/heterogeneity of 3-D infarct scar.

Physiological fusion of functional and structural data for cardiac deformation recovery.

The advancement in meaningful constraining models has resulted in increasingly useful quantitative information recovered from cardiac images. Nevertheless, single-source data used by most of these algorithms have put certain limits on the clinical completeness and relevance of the analysis results, especially for pathological cases where data fusion of multiple complementary sources is essential. As traditional image fusion strategies are typically performed at pixel level by fusing commensurate information of registered images through various mathematical operators, such approaches are not necessarily based on meaningful biological bases, particularly when the data are dissimilar in physical nature and spatiotemporal quantity. In this work, we present a physiological fusion framework for integrating information from different yet complementary sources. Using a cardiac physiome model as the central link, structural and functional data are naturally fused together for a more complete subject-specific information recovery. Experiments were performed on synthetic and real data to show the benefits and potential clinical applicability of our framework.

Physiological-model-constrained noninvasive reconstruction of volumetric myocardial transmembrane potentials.

Personalized noninvasive imaging of subject-specific cardiac electrical activity can guide and improve preventive diagnosis and treatment of cardiac arrhythmia. Compared to body surface potential (BSP) recordings and electrophysiological information reconstructed on heart surfaces, volumetric myocardial transmembrane potential (TMP) dynamics is of greater clinical importance in exhibiting arrhythmic details and arrythmogenic substrates inside the myocardium. This paper presents a physiological-model-constrained statistical framework to reconstruct volumetric TMP dynamics inside the 3-D myocardium from noninvasive BSP recordings. General knowledge of volumetric TMP activity is incorporated through the modeling of cardiac electrophysiological system, and is used to constrain TMP reconstruction. This physiological system is reformulated into a stochastic state-space representation to take into account model and data uncertainties, and nonlinear data assimilation is developed to estimate volumetric myocardial TMP dynamics from personal BSP data. Robustness of the presented framework to practical model and data errors is evaluated. Comparison of epicardial potential reconstructions with classical regularization-based approaches is performed on computational phantom regarding right bundle branch blocks. Further, phantom experiments on intramural focal activities and an initial real-data study on postmyocardial infarction demonstrate the potential of the framework in reconstructing local arrhythmic details and identifying arrhythmogenic substrates inside the myocardium.