Merlin: A Vision Language Foundation Model for 3D Computed Tomography.

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.

Identifying high school risk factors that forecast heavy drinking onset in understudied young adults.

Heavy alcohol drinking is a major, preventable problem that adversely impacts the physical and mental health of US young adults. Studies seeking drinking risk factors typically focus on young adults who enrolled in 4-year residential college programs (4YCP) even though most high school graduates join the workforce, military, or community colleges. We examined 106 of these understudied young adults (USYA) and 453 4YCPs from the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA) by longitudinally following their drinking patterns for 8 years from adolescence to young adulthood. All participants were no-to-low drinkers during high school. Whereas 4YCP individuals were more likely to initiate heavy drinking during college years, USYA participants did so later. Using mental health metrics recorded during high school, machine learning forecasted individual-level risk for initiating heavy drinking after leaving high school. The risk factors differed between demographically matched USYA and 4YCP individuals and between sexes. Predictors for USYA drinkers were sexual abuse, physical abuse for girls, and extraversion for boys, whereas 4YCP drinkers were predicted by the ability to recognize facial emotion and, for boys, greater openness. Thus, alcohol prevention programs need to give special consideration to those joining the workforce, military, or community colleges, who make up the majority of this age group.

Investigating pulse-echo sound speed estimation in breast ultrasound with deep learning.

Ultrasound is an adjunct tool to mammography that can quickly and safely aid physicians in diagnosing breast abnormalities. Clinical ultrasound often assumes a constant sound speed to form diagnostic B-mode images. However, the components of breast tissue, such as glandular tissue, fat, and lesions, differ in sound speed. Given a constant sound speed assumption, these differences can degrade the quality of reconstructed images via phase aberration. Sound speed images can be a powerful tool for improving image quality and identifying diseases if properly estimated. To this end, we propose a supervised deep-learning approach for sound speed estimation from analytic ultrasound signals. We develop a large-scale simulated ultrasound dataset that generates representative breast tissue samples by modeling breast gland, skin, and lesions with varying echogenicity and sound speed. We adopt a fully convolutional neural network architecture trained on a simulated dataset to produce an estimated sound speed map. The simulated tissue is interrogated with a plane wave transmit sequence, and the complex-value reconstructed images are used as input for the convolutional network. The network is trained on the sound speed distribution map of the simulated data, and the trained model can estimate sound speed given reconstructed pulse-echo signals. We further incorporate thermal noise augmentation during training to enhance model robustness to artifacts found in real ultrasound data. To highlight the ability of our model to provide accurate sound speed estimations, we evaluate it on simulated, phantom, and in-vivo breast ultrasound data.

Multimodal graph attention network for COVID-19 outcome prediction.

When dealing with a newly emerging disease such as COVID-19, the impact of patient- and disease-specific factors (e.g., body weight or known co-morbidities) on the immediate course of the disease is largely unknown. An accurate prediction of the most likely individual disease progression can improve the planning of limited resources and finding the optimal treatment for patients. In the case of COVID-19, the need for intensive care unit (ICU) admission of pneumonia patients can often only be determined on short notice by acute indicators such as vital signs (e.g., breathing rate, blood oxygen levels), whereas statistical analysis and decision support systems that integrate all of the available data could enable an earlier prognosis. To this end, we propose a holistic, multimodal graph-based approach combining imaging and non-imaging information. Specifically, we introduce a multimodal similarity metric to build a population graph that shows a clustering of patients. For each patient in the graph, we extract radiomic features from a segmentation network that also serves as a latent image feature encoder. Together with clinical patient data like vital signs, demographics, and lab results, these modalities are combined into a multimodal representation of each patient. This feature extraction is trained end-to-end with an image-based Graph Attention Network to process the population graph and predict the COVID-19 patient outcomes: admission to ICU, need for ventilation, and mortality. To combine multiple modalities, radiomic features are extracted from chest CTs using a segmentation neural network. Results on a dataset collected in Klinikum rechts der Isar in Munich, Germany and the publicly available iCTCF dataset show that our approach outperforms single modality and non-graph baselines. Moreover, our clustering and graph attention increases understanding of the patient relationships within the population graph and provides insight into the network's decision-making process.

Bridging the Gap between Deep Learning and Hypothesis-Driven Analysis via Permutation Testing.

A fundamental approach in neuroscience research is to test hypotheses based on neuropsychological and behavioral measures, i.e., whether certain factors (e.g., related to life events) are associated with an outcome (e.g., depression). In recent years, deep learning has become a potential alternative approach for conducting such analyses by predicting an outcome from a collection of factors and identifying the most "informative" ones driving the prediction. However, this approach has had limited impact as its findings are not linked to statistical significance of factors supporting hypotheses. In this article, we proposed a flexible and scalable approach based on the concept of permutation testing that integrates hypothesis testing into the data-driven deep learning analysis. We apply our approach to the yearly self-reported assessments of 621 adolescent participants of the National Consortium of Alcohol and Neurodevelopment in Adolescence (NCANDA) to predict negative valence, a symptom of major depressive disorder according to the NIMH Research Domain Criteria (RDoC). Our method successfully identifies categories of risk factors that further explain the symptom.

Detecting negative valence symptoms in adolescents based on longitudinal self-reports and behavioral assessments.

BACKGROUND: Given the high prevalence of depressive symptoms reported by adolescents and associated risk of experiencing psychiatric disorders as adults, differentiating the trajectories of the symptoms related to negative valence at an individual level could be crucial in gaining a better understanding of their effects later in life. METHODS: A longitudinal deep learning framework is presented, identifying self-reported and behavioral measurements that detect the depressive symptoms associated with the Negative Valence System domain of the NIMH Research Domain Criteria (RDoC). RESULTS: Applied to the annual records of 621 participants (age range: 12 to 17 years) of the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA), the deep learning framework identifies predictors of negative valence symptoms, which include lower extraversion, poorer sleep quality, impaired executive control function and factors related to substance use. LIMITATIONS: The results rely mainly on self-reported measures and do not provide information about the underlying neural correlates. Also, a larger sample is required to understand the role of sex and other demographics related to the risk of experiencing symptoms of negative valence. CONCLUSIONS: These results provide new information about predictors of negative valence symptoms in individuals during adolescence that could be critical in understanding the development of depression and identifying targets for intervention. Importantly, findings can inform preventive and treatment approaches for depression in adolescents, focusing on a unique predictor set of modifiable modulators to include factors such as sleep hygiene training, cognitive-emotional therapy enhancing coping and controllability experience and/or substance use interventions.