Merlin: A Vision Language Foundation Model for 3D Computed Tomography.

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.

Predicting cardiovascular disease risk using photoplethysmography and deep learning.

Cardiovascular diseases (CVDs) are responsible for a large proportion of premature deaths in low- and middle-income countries. Early CVD detection and intervention is critical in these populations, yet many existing CVD risk scores require a physical examination or lab measurements, which can be challenging in such health systems due to limited accessibility. We investigated the potential to use photoplethysmography (PPG), a sensing technology available on most smartphones that can potentially enable large-scale screening at low cost, for CVD risk prediction. We developed a deep learning PPG-based CVD risk score (DLS) to predict the probability of having major adverse cardiovascular events (MACE: non-fatal myocardial infarction, stroke, and cardiovascular death) within ten years, given only age, sex, smoking status and PPG as predictors. We compare the DLS with the office-based refit-WHO score, which adopts the shared predictors from WHO and Globorisk scores (age, sex, smoking status, height, weight and systolic blood pressure) but refitted on the UK Biobank (UKB) cohort. All models were trained on a development dataset (141,509 participants) and evaluated on a geographically separate test (54,856 participants) dataset, both from UKB. DLS's C-statistic (71.1%, 95% CI 69.9-72.4) is non-inferior to office-based refit-WHO score (70.9%, 95% CI 69.7-72.2; non-inferiority margin of 2.5%, p<0.01) in the test dataset. The calibration of the DLS is satisfactory, with a 1.8% mean absolute calibration error. Adding DLS features to the office-based score increases the C-statistic by 1.0% (95% CI 0.6-1.4). DLS predicts ten-year MACE risk comparable with the office-based refit-WHO score. Interpretability analyses suggest that the DLS-extracted features are related to PPG waveform morphology and are independent of heart rate. Our study provides a proof-of-concept and suggests the potential of a PPG-based approach strategies for community-based primary prevention in resource-limited regions.

An intentional approach to managing bias in general purpose embedding models.

Advances in machine learning for health care have brought concerns about bias from the research community; specifically, the introduction, perpetuation, or exacerbation of care disparities. Reinforcing these concerns is the finding that medical images often reveal signals about sensitive attributes in ways that are hard to pinpoint by both algorithms and people. This finding raises a question about how to best design general purpose pretrained embeddings (GPPEs, defined as embeddings meant to support a broad array of use cases) for building downstream models that are free from particular types of bias. The downstream model should be carefully evaluated for bias, and audited and improved as appropriate. However, in our view, well intentioned attempts to prevent the upstream components-GPPEs-from learning sensitive attributes can have unintended consequences on the downstream models. Despite producing a veneer of technical neutrality, the resultant end-to-end system might still be biased or poorly performing. We present reasons, by building on previously published data, to support the reasoning that GPPEs should ideally contain as much information as the original data contain, and highlight the perils of trying to remove sensitive attributes from a GPPE. We also emphasise that downstream prediction models trained for specific tasks and settings, whether developed using GPPEs or not, should be carefully designed and evaluated to avoid bias that makes models vulnerable to issues such as distributional shift. These evaluations should be done by a diverse team, including social scientists, on a diverse cohort representing the full breadth of the patient population for which the final model is intended.

Developing a Machine Learning Algorithm to Predict the Probability of Medical Staff Work Mode Using Human-Smartphone Interaction Patterns: Algorithm Development and Validation Study.

BACKGROUND: Traditional methods for investigating work hours rely on an employee's physical presence at the worksite. However, accurately identifying break times at the worksite and distinguishing remote work outside the worksite poses challenges in work hour estimations. Machine learning has the potential to differentiate between human-smartphone interactions at work and off work. OBJECTIVE: In this study, we aimed to develop a novel approach called "probability in work mode," which leverages human-smartphone interaction patterns and corresponding GPS location data to estimate work hours. METHODS: To capture human-smartphone interactions and GPS locations, we used the "Staff Hours" app, developed by our team, to passively and continuously record participants' screen events, including timestamps of notifications, screen on or off occurrences, and app usage patterns. Extreme gradient boosted trees were used to transform these interaction patterns into a probability, while 1-dimensional convolutional neural networks generated successive probabilities based on previous sequence probabilities. The resulting probability in work mode allowed us to discern periods of office work, off-work, breaks at the worksite, and remote work. RESULTS: Our study included 121 participants, contributing to a total of 5503 person-days (person-days represent the cumulative number of days across all participants on which data were collected and analyzed). The developed machine learning model exhibited an average prediction performance, measured by the area under the receiver operating characteristic curve, of 0.915 (SD 0.064). Work hours estimated using the probability in work mode (higher than 0.5) were significantly longer (mean 11.2, SD 2.8 hours per day) than the GPS-defined counterparts (mean 10.2, SD 2.3 hours per day; P<.001). This discrepancy was attributed to the higher remote work time of 111.6 (SD 106.4) minutes compared to the break time of 54.7 (SD 74.5) minutes. CONCLUSIONS: Our novel approach, the probability in work mode, harnessed human-smartphone interaction patterns and machine learning models to enhance the precision and accuracy of work hour investigation. By integrating human-smartphone interactions and GPS data, our method provides valuable insights into work patterns, including remote work and breaks, offering potential applications in optimizing work productivity and well-being.

Integrated biosensor platform based on graphene transistor arrays for real-time high-accuracy ion sensing.

Two-dimensional materials such as graphene have shown great promise as biosensors, but suffer from large device-to-device variation due to non-uniform material synthesis and device fabrication technologies. Here, we develop a robust bioelectronic sensing platform  composed of  more than 200 integrated sensing units, custom-built high-speed readout electronics, and machine learning inference that overcomes these challenges to achieve rapid, portable, and reliable measurements. The platform demonstrates reconfigurable multi-ion electrolyte sensing capability and provides highly sensitive, reversible, and real-time response for potassium, sodium, and calcium ions in complex solutions despite variations in device performance. A calibration method leveraging the sensor redundancy and device-to-device variation is also proposed, while a machine learning model trained with multi-dimensional information collected through the multiplexed sensor array is used to enhance the sensing system's functionality and accuracy in ion classification.

Explainable deep learning in healthcare: A methodological survey from an attribution view.

The increasing availability of large collections of electronic health record (EHR) data and unprecedented technical advances in deep learning (DL) have sparked a surge of research interest in developing DL based clinical decision support systems for diagnosis, prognosis, and treatment. Despite the recognition of the value of deep learning in healthcare, impediments to further adoption in real healthcare settings remain due to the black-box nature of DL. Therefore, there is an emerging need for interpretable DL, which allows end users to evaluate the model decision making to know whether to accept or reject predictions and recommendations before an action is taken. In this review, we focus on the interpretability of the DL models in healthcare. We start by introducing the methods for interpretability in depth and comprehensively as a methodological reference for future researchers or clinical practitioners in this field. Besides the methods' details, we also include a discussion of advantages and disadvantages of these methods and which scenarios each of them is suitable for, so that interested readers can know how to compare and choose among them for use. Moreover, we discuss how these methods, originally developed for solving general-domain problems, have been adapted and applied to healthcare problems and how they can help physicians better understand these data-driven technologies. Overall, we hope this survey can help researchers and practitioners in both artificial intelligence and clinical fields understand what methods we have for enhancing the interpretability of their DL models and choose the optimal one accordingly. This article is categorized under: Cancer > Computational Models.

Cost-effectiveness analysis of multimodal prognostication in cardiac arrest with EEG monitoring.

OBJECTIVE: To determine cost-effectiveness parameters for EEG monitoring in cardiac arrest prognostication. METHODS: We conducted a cost-effectiveness analysis to estimate the cost per quality-adjusted life-year (QALY) gained by adding continuous EEG monitoring to standard cardiac arrest prognostication using the American Academy of Neurology Practice Parameter (AANPP) decision algorithm: neurologic examination, somatosensory evoked potentials, and neuron-specific enolase. We explored lifetime cost-effectiveness in a closed system that incorporates revenue back into the medical system (return) from payers who survive a cardiac arrest with good outcome and contribute to the health system during the remaining years of life. Good outcome was defined as a Cerebral Performance Category (CPC) score of 1-2 and poor outcome as CPC of 3-5. RESULTS: An improvement in specificity for poor outcome prediction of 4.2% would be sufficient to make continuous EEG monitoring cost-effective (baseline AANPP specificity = 83.9%). In sensitivity analysis, the effect of increased sensitivity on the cost-effectiveness of EEG depends on the utility (u) assigned to a poor outcome. For patients who regard surviving with a poor outcome (CPC 3-4) worse than death (u = -0.34), an increased sensitivity for poor outcome prediction of 13.8% would make AANPP + EEG monitoring cost-effective (baseline AANPP sensitivity = 76.3%). In the closed system, an improvement in sensitivity of 1.8% together with an improvement in specificity of 3% was sufficient to make AANPP + EEG monitoring cost-effective, assuming lifetime return of 50% (USD $70,687). CONCLUSION: Incorporating continuous EEG monitoring into cardiac arrest prognostication is cost-effective if relatively small improvements in sensitivity and specificity are achieved.

Determining Follow-Up Imaging Study Using Radiology Reports.

Radiology reports often contain follow-up imaging recommendations. Failure to comply with these recommendations in a timely manner can lead to delayed treatment, poor patient outcomes, complications, unnecessary testing, lost revenue, and legal liability. The objective of this study was to develop a scalable approach to automatically identify the completion of a follow-up imaging study recommended by a radiologist in a preceding report. We selected imaging-reports containing 559 follow-up imaging recommendations and all subsequent reports from a multi-hospital academic practice. Three radiologists identified appropriate follow-up examinations among the subsequent reports for the same patient, if any, to establish a ground-truth dataset. We then trained an Extremely Randomized Trees that uses recommendation attributes, study meta-data and text similarity of the radiology reports to determine the most likely follow-up examination for a preceding recommendation. Pairwise inter-annotator F-score ranged from 0.853 to 0.868; the corresponding F-score of the classifier in identifying follow-up exams was 0.807. Our study describes a methodology to automatically determine the most likely follow-up exam after a follow-up imaging recommendation. The accuracy of the algorithm suggests that automated methods can be integrated into a follow-up management application to improve adherence to follow-up imaging recommendations. Radiology administrators could use such a system to monitor follow-up compliance rates and proactively send reminders to primary care providers and/or patients to improve adherence.

Medical subdomain classification of clinical notes using a machine learning-based natural language processing approach.

BACKGROUND: The medical subdomain of a clinical note, such as cardiology or neurology, is useful content-derived metadata for developing machine learning downstream applications. To classify the medical subdomain of a note accurately, we have constructed a machine learning-based natural language processing (NLP) pipeline and developed medical subdomain classifiers based on the content of the note. METHODS: We constructed the pipeline using the clinical NLP system, clinical Text Analysis and Knowledge Extraction System (cTAKES), the Unified Medical Language System (UMLS) Metathesaurus, Semantic Network, and learning algorithms to extract features from two datasets - clinical notes from Integrating Data for Analysis, Anonymization, and Sharing (iDASH) data repository (n = 431) and Massachusetts General Hospital (MGH) (n = 91,237), and built medical subdomain classifiers with different combinations of data representation methods and supervised learning algorithms. We evaluated the performance of classifiers and their portability across the two datasets. RESULTS: The convolutional recurrent neural network with neural word embeddings trained-medical subdomain classifier yielded the best performance measurement on iDASH and MGH datasets with area under receiver operating characteristic curve (AUC) of 0.975 and 0.991, and F1 scores of 0.845 and 0.870, respectively. Considering better clinical interpretability, linear support vector machine-trained medical subdomain classifier using hybrid bag-of-words and clinically relevant UMLS concepts as the feature representation, with term frequency-inverse document frequency (tf-idf)-weighting, outperformed other shallow learning classifiers on iDASH and MGH datasets with AUC of 0.957 and 0.964, and F1 scores of 0.932 and 0.934 respectively. We trained classifiers on one dataset, applied to the other dataset and yielded the threshold of F1 score of 0.7 in classifiers for half of the medical subdomains we studied. CONCLUSION: Our study shows that a supervised learning-based NLP approach is useful to develop medical subdomain classifiers. The deep learning algorithm with distributed word representation yields better performance yet shallow learning algorithms with the word and concept representation achieves comparable performance with better clinical interpretability. Portable classifiers may also be used across datasets from different institutions.

Differentiating intratumoral melanocytes from Langerhans cells in nonmelanocytic pigmented skin tumors in vivo by label-free third-harmonic generation microscopy.

Morphology and distribution of melanocytes are critical imaging information for the diagnosis of melanocytic lesions. However, how to image intratumoral melanocytes noninvasively in pigmented skin tumors is seldom investigated. Third-harmonic generation (THG) is shown to be enhanced by melanin, whereas high accuracy has been demonstrated using THG microscopy for in vivo differential diagnosis of nonmelanocytic pigmented skin tumors. It is thus desirable to investigate if label-free THG microscopy was capable to in vivo identify intratumoral melanocytes. In this study, histopathological correlations of label-free THG images with the immunohistochemical images stained with human melanoma black (HMB)-45 and cluster of differentiation 1a (CD1a) were made. The correlation results indicated that the intratumoral THG-bright dendritic-cell-like signals were endogenously derived from melanocytes rather than Langerhans cells (LCs). The consistency between THG-bright dendritic-cell-like signals and HMB-45 melanocyte staining showed a kappa coefficient of 0.807, 84.6% sensitivity, and 95% specificity. In contrast, a kappa coefficient of −0.37, 21.7% sensitivity, and 30% specificity were noted between the THG-bright dendritic-cell-like signals and CD1a staining for LCs. Our study indicates the capability of noninvasive label-free THG microscopy to differentiate intratumoral melanocytes from LCs, which is not feasible in previous in vivo label-free clinical-imaging modalities.

The process of coping with stress by Taiwanese medical interns: a qualitative study.

BACKGROUND: Internship, the transition period from medical student to junior doctor, is highly stressful for interns in the West; however, little is known about the experience of interns in coping with stress in Taiwan. This study aimed to develop a model for coping with stress among Taiwanese interns and to examine the relationship between stress and learning outcomes. METHODS: For this qualitative study, we used grounded theory methodology with theoretical sampling. We collected data through in-depth interviews and participant observations. We employed the constant comparative method to analyse the data until data saturation was achieved. RESULTS: The study population was 124 medical interns in a teaching hospital in northern Taiwan; 21 interns (12 males) participated. Data analysis revealed that the interns encountered stressors (such as sense of responsibility, coping with uncertainty, and interpersonal relationships) resulting from their role transition from observer to practitioner. The participants used self-directed learning and avoidance as strategies to deal with their stress. CONCLUSIONS: A self-directed learning strategy can be beneficial for an intern's motivation to learn as well as for patient welfare. However, avoiding stressors can result in less motivation to learn and hinder the quality of care. Understanding how interns experience and cope with stress and its related outcomes can help medical educators and policy makers improve the quality of medical education by encouraging interns' self-directed learning strategy and discouraging the avoidance of stressors.

Gender differences in cardiac autonomic modulation during medical internship.

Medical internship is known to be a time of high stress and long working hours, which increases the risk of depression and cardiovascular disease. Gender differences in medical interns' cardiovascular risk have not been reported previously. Thirty-eight medical interns (29 males) were repeatedly tested for depressive symptoms using the Hospital Anxiety and Depression Scale and 5-min spectral analysis of heart rate variability (HRV) at 3-month intervals during their internship. Among the male interns, the variance of the heart rate decreased at 6, 9, 12 months, and a reduced high frequency, which suggests reduced cardiac parasympathetic modulation, was found at 9 and 12 months into their internship. Increased depressive symptoms were also identified at 12 months in the male group. No significant differences in depression or any of the HRV indices were identified among the female interns during their internship.

DNA vaccines for cervical cancer.

Human papillomavirus (HPV), particularly type 16, has been associated with more than 99% of cervical cancers. There are two HPV oncogenic proteins, E6 and E7, which play a major role in the induction and maintenance of cellular transformation. Thus, immunotherapy targeting these proteins may be employed for the control of HPV-associated cervical lesions. Although the commercially available preventive HPV vaccines are highly efficient in preventing new HPV infection, they do not have therapeutic effects against established HPV infection or HPV-associated lesions. Since T cell-mediated immunity is important for treating established HPV infections and HPV-associated lesions, therapeutic HPV vaccine should aim at generating potent E6 and E7-specific T cell-mediated immune responses. DNA vaccines have now developed into a promising approach for antigen-specific T cell-mediated immunotherapy to combat infection and cancer. Because dendritic cells are the most potent professional antigen-presenting cells, and are highly effective in priming antigen-specific T cells, several DNA vaccines have employed innovative strategies to modify the properties of dendritic cells (DCs) for the enhancement of the DNA vaccine potency. These studies have revealed impressive pre-clinical data that has led to several ongoing HPV DNA vaccine clinical trials.