Merlin: A Vision Language Foundation Model for 3D Computed Tomography.

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.

GP-GAN: Brain tumor growth prediction using stacked 3D generative adversarial networks from longitudinal MR Images.

Brain tumors are one of the major common causes of cancer-related death, worldwide. Growth prediction of these tumors, particularly gliomas which are the most dominant type, can be quite useful to improve treatment planning, quantify tumor aggressiveness, and estimate patients' survival time towards precision medicine. Studying tumor growth prediction basically requires multiple time points of single or multimodal medical images of the same patient. Recent models are based on complex mathematical formulations that basically rely on a system of partial differential equations, e.g. reaction diffusion model, to capture the diffusion and proliferation of tumor cells in the surrounding tissue. However, these models usually have small number of parameters that are insufficient to capture different patterns and other characteristics of the tumors. In addition, such models consider tumor growth independently for each subject, not being able to get benefit from possible common growth patterns existed in the whole population under study. In this paper, we propose a novel data-driven method via stacked 3D generative adversarial networks (GANs), named GP-GAN, for growth prediction of glioma. Specifically, we use stacked conditional GANs with a novel objective function that includes both l1 and Dice losses. Moreover, we use segmented feature maps to guide the generator for better generated images. Our generator is designed based on a modified 3D U-Net architecture with skip connections to combine hierarchical features and thus have a better generated image. The proposed method is trained and tested on 18 subjects with 3 time points (9 subjects from collaborative hospital and 9 subjects from BRATS 2014 dataset). Results show that our proposed GP-GAN outperforms state-of-the-art methods for glioma growth prediction and attain average Jaccard index and Dice coefficient of 78.97% and 88.26%, respectively.

Breast Cancer Detection and Diagnosis Using Mammographic Data: Systematic Review.

BACKGROUND: Machine learning (ML) has become a vital part of medical imaging research. ML methods have evolved over the years from manual seeded inputs to automatic initializations. The advancements in the field of ML have led to more intelligent and self-reliant computer-aided diagnosis (CAD) systems, as the learning ability of ML methods has been constantly improving. More and more automated methods are emerging with deep feature learning and representations. Recent advancements of ML with deeper and extensive representation approaches, commonly known as deep learning (DL) approaches, have made a very significant impact on improving the diagnostics capabilities of the CAD systems. OBJECTIVE: This review aimed to survey both traditional ML and DL literature with particular application for breast cancer diagnosis. The review also provided a brief insight into some well-known DL networks. METHODS: In this paper, we present an overview of ML and DL techniques with particular application for breast cancer. Specifically, we search the PubMed, Google Scholar, MEDLINE, ScienceDirect, Springer, and Web of Science databases and retrieve the studies in DL for the past 5 years that have used multiview mammogram datasets. RESULTS: The analysis of traditional ML reveals the limited usage of the methods, whereas the DL methods have great potential for implementation in clinical analysis and improve the diagnostic capability of existing CAD systems. CONCLUSIONS: From the literature, it can be found that heterogeneous breast densities make masses more challenging to detect and classify compared with calcifications. The traditional ML methods present confined approaches limited to either particular density type or datasets. Although the DL methods show promising improvements in breast cancer diagnosis, there are still issues of data scarcity and computational cost, which have been overcome to a significant extent by applying data augmentation and improved computational power of DL algorithms.