Predicting multiple sclerosis severity with multimodal deep neural networks.

Multiple Sclerosis (MS) is a chronic disease developed in the human brain and spinal cord, which can cause permanent damage or deterioration of the nerves. The severity of MS disease is monitored by the Expanded Disability Status Scale, composed of several functional sub-scores. Early and accurate classification of MS disease severity is critical for slowing down or preventing disease progression via applying early therapeutic intervention strategies. Recent advances in deep learning and the wide use of Electronic Health Records (EHR) create opportunities to apply data-driven and predictive modeling tools for this goal. Previous studies focusing on using single-modal machine learning and deep learning algorithms were limited in terms of prediction accuracy due to data insufficiency or model simplicity. In this paper, we proposed the idea of using patients' multimodal longitudinal and longitudinal EHR data to predict multiple sclerosis disease severity in the future. Our contribution has two main facets. First, we describe a pioneering effort to integrate structured EHR data, neuroimaging data and clinical notes to build a multi-modal deep learning framework to predict patient's MS severity. The proposed pipeline demonstrates up to 19% increase in terms of the area under the Area Under the Receiver Operating Characteristic curve (AUROC) compared to models using single-modal data. Second, the study also provides valuable insights regarding the amount useful signal embedded in each data modality with respect to MS disease prediction, which may improve data collection processes.

aBnormal motION capture In aCute Stroke (BIONICS): A Low-Cost Tele-Evaluation Tool for Automated Assessment of Upper Extremity Function in Stroke Patients.

BACKGROUND: The incidence of stroke and stroke-related hemiparesis has been steadily increasing and is projected to become a serious social, financial, and physical burden on the aging population. Limited access to outpatient rehabilitation for these stroke survivors further deepens the healthcare issue and estranges the stroke patient demographic in rural areas. However, new advances in motion detection deep learning enable the use of handheld smartphone cameras for body tracking, offering unparalleled levels of accessibility. METHODS: In this study we want to develop an automated method for evaluation of a shortened variant of the Fugl-Meyer assessment, the standard stroke rehabilitation scale describing upper extremity motor function. We pair this technology with a series of machine learning models, including different neural network structures and an eXtreme Gradient Boosting model, to score 16 of 33 (49%) Fugl-Meyer item activities. RESULTS: In this observational study, 45 acute stroke patients completed at least 1 recorded Fugl-Meyer assessment for the training of the auto-scorers, which yielded average accuracies ranging from 78.1% to 82.7% item-wise. CONCLUSION: In this study, an automated method was developed for the evaluation of a shortened variant of the Fugl-Meyer assessment, the standard stroke rehabilitation scale describing upper extremity motor function. This novel method is demonstrated with potential to conduct telehealth rehabilitation evaluations and assessments with accuracy and availability.

Non-invasive arterial blood pressure measurement and SpO2 estimation using PPG signal: a deep learning framework.

BACKGROUND: Monitoring blood pressure and peripheral capillary oxygen saturation plays a crucial role in healthcare management for patients with chronic diseases, especially hypertension and vascular disease. However, current blood pressure measurement methods have intrinsic limitations; for instance, arterial blood pressure is measured by inserting a catheter in the artery causing discomfort and infection. METHOD: Photoplethysmogram (PPG) signals can be collected via non-invasive devices, and therefore have stimulated researchers' interest in exploring blood pressure estimation using machine learning and PPG signals as a non-invasive alternative. In this paper, we propose a Transformer-based deep learning architecture that utilizes PPG signals to conduct a personalized estimation of arterial systolic blood pressure, arterial diastolic blood pressure, and oxygen saturation. RESULTS: The proposed method was evaluated with a subset of 1,732 subjects from the publicly available ICU dataset MIMIC III. The mean absolute error is 2.52 ± 2.43 mmHg for systolic blood pressure, 1.37 ± 1.89 mmHg for diastolic blood pressure, and 0.58 ± 0.79% for oxygen saturation, which satisfies the requirements of the Association of Advancement of Medical Instrumentation standard and achieve grades A for the British Hypertension Society standard. CONCLUSIONS: The results indicate that our model meets clinical standards and could potentially boost the accuracy of blood pressure and oxygen saturation measurement to deliver high-quality healthcare.

Dental CLAIRES: Contrastive LAnguage Image REtrieval Search for Dental Research.

Learning about diagnostic features and related clinical information from dental radiographs is important for dental research. However, the lack of expert-annotated data and convenient search tools poses challenges. Our primary objective is to design a search tool that uses a user's query for oral-related research. The proposed framework, Contrastive LAnguage Image REtrieval Search for dental research, Dental CLAIRES, utilizes periapical radiographs and associated clinical details such as periodontal diagnosis, demographic information to retrieve the best-matched images based on the text query. We applied a contrastive representation learning method to find images described by the user's text by maximizing the similarity score of positive pairs (true pairs) and minimizing the score of negative pairs (random pairs). Our model achieved a hit@3 ratio of 96% and a Mean Reciprocal Rank (MRR) of 0.82. We also designed a graphical user interface that allows researchers to verify the model's performance with interactions.

A comprehensive artificial intelligence framework for dental diagnosis and charting.

BACKGROUND: The aim of this study was to develop artificial intelligence (AI) guided framework to recognize tooth numbers in panoramic and intraoral radiographs (periapical and bitewing) without prior domain knowledge and arrange the intraoral radiographs into a full mouth series (FMS) arrangement template. This model can be integrated with different diseases diagnosis models, such as periodontitis or caries, to facilitate clinical examinations and diagnoses. METHODS: The framework utilized image segmentation models to generate the masks of bone area, tooth, and cementoenamel junction (CEJ) lines from intraoral radiographs. These masks were used to detect and extract teeth bounding boxes utilizing several image analysis methods. Then, individual teeth were matched with a patient's panoramic images (if available) or tooth repositories for assigning tooth numbers using the multi-scale matching strategy. This framework was tested on 1240 intraoral radiographs different from the training and internal validation cohort to avoid data snooping. Besides, a web interface was designed to generate a report for different dental abnormalities with tooth numbers to evaluate this framework's practicality in clinical settings. RESULTS: The proposed method achieved the following precision and recall via panoramic view: 0.96 and 0.96 (via panoramic view) and 0.87 and 0.87 (via repository match) by handling tooth shape variation and outperforming other state-of-the-art methods. Additionally, the proposed framework could accurately arrange a set of intraoral radiographs into an FMS arrangement template based on positions and tooth numbers with an accuracy of 95% for periapical images and 90% for bitewing images. The accuracy of this framework was also 94% in the images with missing teeth and 89% with restorations. CONCLUSIONS: The proposed tooth numbering model is robust and self-contained and can also be integrated with other dental diagnosis modules, such as alveolar bone assessment and caries detection. This artificial intelligence-based tooth detection and tooth number assignment in dental radiographs will help dentists with enhanced communication, documentation, and treatment planning accurately. In addition, the proposed framework can correctly specify detailed diagnostic information associated with a single tooth without human intervention.

Advances in non-invasive biosensing measures to monitor wound healing progression.

Impaired wound healing is a significant financial and medical burden. The synthesis and deposition of extracellular matrix (ECM) in a new wound is a dynamic process that is constantly changing and adapting to the biochemical and biomechanical signaling from the extracellular microenvironments of the wound. This drives either a regenerative or fibrotic and scar-forming healing outcome. Disruptions in ECM deposition, structure, and composition lead to impaired healing in diseased states, such as in diabetes. Valid measures of the principal determinants of successful ECM deposition and wound healing include lack of bacterial contamination, good tissue perfusion, and reduced mechanical injury and strain. These measures are used by wound-care providers to intervene upon the healing wound to steer healing toward a more functional phenotype with improved structural integrity and healing outcomes and to prevent adverse wound developments. In this review, we discuss bioengineering advances in 1) non-invasive detection of biologic and physiologic factors of the healing wound, 2) visualizing and modeling the ECM, and 3) computational tools that efficiently evaluate the complex data acquired from the wounds based on basic science, preclinical, translational and clinical studies, that would allow us to prognosticate healing outcomes and intervene effectively. We focus on bioelectronics and biologic interfaces of the sensors and actuators for real time biosensing and actuation of the tissues. We also discuss high-resolution, advanced imaging techniques, which go beyond traditional confocal and fluorescence microscopy to visualize microscopic details of the composition of the wound matrix, linearity of collagen, and live tracking of components within the wound microenvironment. Computational modeling of the wound matrix, including partial differential equation datasets as well as machine learning models that can serve as powerful tools for physicians to guide their decision-making process are discussed.

Secure human action recognition by encrypted neural network inference.

Advanced computer vision technology can provide near real-time home monitoring to support "aging in place" by detecting falls and symptoms related to seizures and stroke. Affordable webcams, together with cloud computing services (to run machine learning algorithms), can potentially bring significant social benefits. However, it has not been deployed in practice because of privacy concerns. In this paper, we propose a strategy that uses homomorphic encryption to resolve this dilemma, which guarantees information confidentiality while retaining action detection. Our protocol for secure inference can distinguish falls from activities of daily living with 86.21% sensitivity and 99.14% specificity, with an average inference latency of 1.2 seconds and 2.4 seconds on real-world test datasets using small and large neural nets, respectively. We show that our method enables a 613x speedup over the latency-optimized LoLa and achieves an average of 3.1x throughput increase in secure inference compared to the throughput-optimized nGraph-HE2.

A multitask deep learning approach for pulmonary embolism detection and identification.

Pulmonary embolism (PE) is a blood clot traveling to the lungs and is associated with substantial morbidity and mortality. Therefore, rapid diagnoses and treatments are essential. Chest computed tomographic pulmonary angiogram (CTPA) is the gold standard for PE diagnoses. Deep learning can enhance the radiologists'workflow by identifying PE using CTPA, which helps to prioritize important cases and hasten the diagnoses for at-risk patients. In this study, we propose a two-phase multitask learning method that can recognize the presence of PE and its properties such as the position, whether acute or chronic, and the corresponding right-to-left ventricle diameter (RV/LV) ratio, thereby reducing false-negative diagnoses. Trained on the RSNA-STR Pulmonary Embolism CT Dataset, our model demonstrates promising PE detection performances on the hold-out test set with the window-level AUROC achieving 0.93 and the sensitivity being 0.86 with a specificity of 0.85, which is competitive with the radiologists'sensitivities ranging from 0.67 to 0.87 with specificities of 0.89-0.99. In addition, our model provides interpretability through attention weight heatmaps and gradient-weighted class activation mapping (Grad-CAM). Our proposed deep learning model could predict PE existence and other properties of existing cases, which could be applied to practical assistance for PE diagnosis.

Metabolic Imaging Using Hyperpolarization for Assessment of Premalignancy.

There is an unmet need for noninvasive surrogate markers that can help identify premalignant lesions across different tumor types. Here we describe the methodology and technical details of protocols employed for in vivo 13C pyruvate metabolic imaging experiments. The goal of the method described is to identify and understand metabolic changes, to enable detection of pancreatic premalignant lesions, as a proof of concept of the high sensitivity of this imaging modality.

Machine learning to predict sports-related concussion recovery using clinical data.

OBJECTIVES: Sport-related concussions (SRCs) are a concern for high school athletes. Understanding factors contributing to SRC recovery time may improve clinical management. However, the complexity of the many clinical measures of concussion data precludes many traditional methods. This study aimed to answer the question, what is the utility of modeling clinical concussion data using machine-learning algorithms for predicting SRC recovery time and protracted recovery? METHODS: This was a retrospective case series of participants aged 8 to 18 years with a diagnosis of SRC. A 6-part measure was administered to assess pre-injury risk factors, initial injury severity, and post-concussion symptoms, including the Vestibular Ocular Motor Screening (VOMS) measure, King-Devick Test and C3 Logix Trails Test data. These measures were used to predict recovery time (days from injury to full medical clearance) and binary protracted recovery (recovery time > 21 days) according to several sex-stratified machine-learning models. The ability of the models to discriminate protracted recovery was compared to a human-driven model according to the area under the receiver operating characteristic curve (AUC). RESULTS: For 293 males (mean age 14.0 years) and 362 females (mean age 13.7 years), the median (interquartile range) time to recover from an SRC was 26 (18-39) and 21 (14-31) days, respectively. Among 9 machine-learning models trained, the gradient boosting on decision-tree algorithms achieved the best performance to predict recovery time and protracted recovery in males and females. The models' performance improved when VOMS data were used in conjunction with the King-Devick Test and C3 Logix Trails Test data. For males and females, the AUC was 0.84 and 0.78 versus 0.74 and 0.73, respectively, for statistical models for predicting protracted recovery. CONCLUSIONS: Machine-learning models were able to manage the complexity of the vestibular-ocular motor system data. These results demonstrate the clinical utility of machine-learning models to inform prognostic evaluation for SRC recovery time and protracted recovery.

Use of the deep learning approach to measure alveolar bone level.

AIM: The goal was to use a deep convolutional neural network to measure the radiographic alveolar bone level to aid periodontal diagnosis. MATERIALS AND METHODS: A deep learning (DL) model was developed by integrating three segmentation networks (bone area, tooth, cemento-enamel junction) and image analysis to measure the radiographic bone level and assign radiographic bone loss (RBL) stages. The percentage of RBL was calculated to determine the stage of RBL for each tooth. A provisional periodontal diagnosis was assigned using the 2018 periodontitis classification. RBL percentage, staging, and presumptive diagnosis were compared with the measurements and diagnoses made by the independent examiners. RESULTS: The average Dice Similarity Coefficient (DSC) for segmentation was over 0.91. There was no significant difference in the RBL percentage measurements determined by DL and examiners ( p=.65 ). The area under the receiver operating characteristics curve of RBL stage assignment for stages I, II, and III was 0.89, 0.90, and 0.90, respectively. The accuracy of the case diagnosis was 0.85. CONCLUSIONS: The proposed DL model provides reliable RBL measurements and image-based periodontal diagnosis using periapical radiographic images. However, this model has to be further optimized and validated by a larger number of images to facilitate its application.

Hyperpolarized Magnetic Resonance and Artificial Intelligence: Frontiers of Imaging in Pancreatic Cancer.

BACKGROUND: There is an unmet need for noninvasive imaging markers that can help identify the aggressive subtype(s) of pancreatic ductal adenocarcinoma (PDAC) at diagnosis and at an earlier time point, and evaluate the efficacy of therapy prior to tumor reduction. In the past few years, there have been two major developments with potential for a significant impact in establishing imaging biomarkers for PDAC and pancreatic cancer premalignancy: (1) hyperpolarized metabolic (HP)-magnetic resonance (MR), which increases the sensitivity of conventional MR by over 10,000-fold, enabling real-time metabolic measurements; and (2) applications of artificial intelligence (AI). OBJECTIVE: Our objective of this review was to discuss these two exciting but independent developments (HP-MR and AI) in the realm of PDAC imaging and detection from the available literature to date. METHODS: A systematic review following the PRISMA extension for Scoping Reviews (PRISMA-ScR) guidelines was performed. Studies addressing the utilization of HP-MR and/or AI for early detection, assessment of aggressiveness, and interrogating the early efficacy of therapy in patients with PDAC cited in recent clinical guidelines were extracted from the PubMed and Google Scholar databases. The studies were reviewed following predefined exclusion and inclusion criteria, and grouped based on the utilization of HP-MR and/or AI in PDAC diagnosis. RESULTS: Part of the goal of this review was to highlight the knowledge gap of early detection in pancreatic cancer by any imaging modality, and to emphasize how AI and HP-MR can address this critical gap. We reviewed every paper published on HP-MR applications in PDAC, including six preclinical studies and one clinical trial. We also reviewed several HP-MR-related articles describing new probes with many functional applications in PDAC. On the AI side, we reviewed all existing papers that met our inclusion criteria on AI applications for evaluating computed tomography (CT) and MR images in PDAC. With the emergence of AI and its unique capability to learn across multimodal data, along with sensitive metabolic imaging using HP-MR, this knowledge gap in PDAC can be adequately addressed. CT is an accessible and widespread imaging modality worldwide as it is affordable; because of this reason alone, most of the data discussed are based on CT imaging datasets. Although there were relatively few MR-related papers included in this review, we believe that with rapid adoption of MR imaging and HP-MR, more clinical data on pancreatic cancer imaging will be available in the near future. CONCLUSIONS: Integration of AI, HP-MR, and multimodal imaging information in pancreatic cancer may lead to the development of real-time biomarkers of early detection, assessing aggressiveness, and interrogating early efficacy of therapy in PDAC.

Population stratification enables modeling effects of reopening policies on mortality and hospitalization rates.

OBJECTIVE: Study the impact of local policies on near-future hospitalization and mortality rates. MATERIALS AND METHODS: We introduce a novel risk-stratified SIR-HCD model that introduces new variables to model the dynamics of low-contact (e.g., work from home) and high-contact (e.g., work on-site) subpopulations while sharing parameters to control their respective R0(t) over time. We test our model on data of daily reported hospitalizations and cumulative mortality of COVID-19 in Harris County, Texas, from May 1, 2020, until October 4, 2020, collected from multiple sources (USA FACTS, U.S. Bureau of Labor Statistics, Southeast Texas Regional Advisory Council COVID-19 report, TMC daily news, and Johns Hopkins University county-level mortality reporting). RESULTS: We evaluated our model's forecasting accuracy in Harris County, TX (the most populated county in the Greater Houston area) during Phase-I and Phase-II reopening. Not only does our model outperform other competing models, but it also supports counterfactual analysis to simulate the impact of future policies in a local setting, which is unique among existing approaches. DISCUSSION: Mortality and hospitalization rates are significantly impacted by local quarantine and reopening policies. Existing models do not directly account for the effect of these policies on infection, hospitalization, and death rates in an explicit and explainable manner. Our work is an attempt to improve prediction of these trends by incorporating this information into the model, thus supporting decision-making. CONCLUSION: Our work is a timely effort to attempt to model the dynamics of pandemics under the influence of local policies.

Early Detection of Pancreatic Intraepithelial Neoplasias (PanINs) in Transgenic Mouse Model by Hyperpolarized 13C Metabolic Magnetic Resonance Spectroscopy.

While pancreatic cancer (PC) survival rates have recently shown modest improvement, the disease remains largely incurable. Early detection of pancreatic cancer may result in improved outcomes and therefore, methods for early detection of cancer, even premalignant lesions, may provide more favorable outcomes. Pancreatic intraepithelial neoplasias (PanINs) have been identified as premalignant precursor lesions to pancreatic cancer. However, conventional imaging methods used for screening high-risk populations do not have the sensitivity to detect PanINs. Here, we have employed hyperpolarized metabolic imaging in vivo and nuclear magnetic resonance (1H-NMR) metabolomics ex vivo to identify and understand metabolic changes, towards enabling detection of early PanINs and progression to advanced PanINs lesions that precede pancreatic cancer formation. Progression of disease from tissue containing predominantly low-grade PanINs to tissue with high-grade PanINs showed a decreasing alanine/lactate ratio from high-resolution NMR metabolomics ex vivo. Hyperpolarized magnetic resonance spectroscopy (HP-MRS) allows over 10,000-fold sensitivity enhancement relative to conventional magnetic resonance. Real-time HP-MRS was employed to measure non-invasively changes of alanine and lactate metabolites with disease progression and in control mice in vivo, following injection of hyperpolarized [1-13C] pyruvate. The alanine-to-lactate signal intensity ratio was found to decrease as the disease progressed from low-grade PanINs to high-grade PanINs. The biochemical changes of alanine transaminase (ALT) and lactate dehydrogenase (LDH) enzyme activity were assessed. These results demonstrate that there are significant alterations of ALT and LDH activities during the transformation from early to advanced PanINs lesions. Furthermore, we demonstrate that real-time conversion kinetic rate constants (kPA and kPL) can be used as metabolic imaging biomarkers of pancreatic premalignant lesions. Findings from this emerging HP-MRS technique can be translated to the clinic for detection of pancreatic premalignant lesion in high-risk populations.