Consensus modeling: Safer transfer learning for small health systems.

Predictive modeling is becoming an essential tool for clinical decision support, but health systems with smaller sample sizes may construct suboptimal or overly specific models. Models become over-specific when beside true physiological effects, they also incorporate potentially volatile site-specific artifacts. These artifacts can change suddenly and can render the model unsafe. To obtain safer models, health systems with inadequate sample sizes may adopt one of the following options. First, they can use a generic model, such as one purchased from a vendor, but often such a model is not sufficiently specific to the patient population and is thus suboptimal. Second, they can participate in a research network. Paradoxically though, sites with smaller datasets contribute correspondingly less to the joint model, again rendering the final model suboptimal. Lastly, they can use transfer learning, starting from a model trained on a large data set and updating this model to the local population. This strategy can also result in a model that is over-specific. In this paper we present the consensus modeling paradigm, which uses the help of a large site (source) to reach a consensus model at the small site (target). We evaluate the approach on predicting postoperative complications at two health systems with 9,044 and 38,045 patients (rare outcomes at about 1% positive rate), and conduct a simulation study to understand the performance of consensus modeling relative to the other three approaches as a function of the available training sample size at the target site. We found that consensus modeling exhibited the least over-specificity at either the source or target site and achieved the highest combined predictive performance.

Robust Methods for Quantifying the Effect of a Continuous Exposure From Observational Data.

A cornerstone of clinical medicine is intervening on a continuous exposure, such as titrating the dosage of a pharmaceutical or controlling a laboratory result. In clinical trials, continuous exposures are dichotomized into narrow ranges, excluding large portions of the realistic treatment scenarios. The existing computational methods for estimating the effect of continuous exposure rely on a set of strict assumptions. We introduce new methods that are more robust towards violations of these assumptions. Our methods are based on the key observation that changes of exposure in the clinical setting are often achieved gradually, so effect estimates must be "locally" robust in narrower exposure ranges. We compared our methods with several existing methods on three simulated studies with increasing complexity. We also applied the methods to data from 14 k sepsis patients at M Health Fairview to estimate the effect of antibiotic administration latency on prolonged hospital stay. The proposed methods achieve good performance in all simulation studies. When the assumptions were violated, the proposed methods had estimation errors of one half to one fifth of the state-of-the-art methods. Applying our methods to the sepsis cohort resulted in effect estimates consistent with clinical knowledge.

Causal structure discovery identifies risk factors and early brain markers related to evolution of white matter hyperintensities.

Our goal was to understand the complex relationship between age, sex, midlife risk factors, and early white matter changes measured by diffusion tensor imaging (DTI) and their role in the evolution of longitudinal white matter hyperintensities (WMH). We identified 1564 participants (1396 cognitively unimpaired, 151 mild cognitive impairment and 17 dementia participants) with age ranges of 30-90 years from the population-based sample of Mayo Clinic Study of Aging. We used computational causal structure discovery and regression analyses to evaluate the predictors of WMH and DTI, and to ascertain the mediating effect of DTI on WMH. We further derived causal graphs to understand the complex interrelationships between midlife protective factors, vascular risk factors, diffusion changes, and WMH. Older age, female sex, and hypertension were associated with higher baseline and progression of WMH as well as DTI measures (P ≤ 0.003). The effects of hypertension and sex on WMH were partially mediated by microstructural changes measured on DTI. Higher midlife physical activity was predictive of lower WMH through a direct impact on better white matter tract integrity as well as an indirect effect through reducing the risk of hypertension by lowering BMI. This study identified key risks factors, early brain changes, and pathways that may lead to the evolution of WMH.

Accelerating Surgical Site Infection Abstraction With a Semi-automated Machine-learning Approach.

OBJECTIVE: To demonstrate that a semi-automated approach to health data abstraction provides significant efficiencies and high accuracy. BACKGROUND: Surgical outcome abstraction remains laborious and a barrier to the sustainment of quality improvement registries like ACS-NSQIP. A supervised machine learning algorithm developed for detecting SSi using structured and unstructured electronic health record data was tested to perform semi-automated SSI abstraction. METHODS: A Lasso-penalized logistic regression model with 2011-3 data was trained (baseline performance measured with 10-fold cross-validation). A cutoff probability score from the training data was established, dividing the subsequent evaluation dataset into "negative" and "possible" SSI groups, with manual data abstraction only performed on the "possible" group. We evaluated performance on data from 2014, 2015, and both years. RESULTS: Overall, 6188 patients were in the 2011-3 training dataset and 5132 patients in the 2014-5 evaluation dataset. With use of the semi-automated approach, applying the cut-off score decreased the amount of manual abstraction by >90%, resulting in < 1% false negatives in the "negative" group and a sensitivity of 82%. A blinded review of 10% of the "possible" group, considering only the features selected by the algorithm, resulted in high agreement with the gold standard based on full chart abstraction, pointing towards additional efficiency in the abstraction process by making it possible for abstractors to review limited, salient portions of the chart. CONCLUSION: Semi-automated machine learning-aided SSI abstraction greatly accelerates the abstraction process and achieves very good performance. This could be translated to other post-operative outcomes and reduce cost barriers for wider ACS-NSQIP adoption.

Disparity in Access to Care Impacts Liver Transplant Mortality.

This study aims to analyze how access to care influences patient mortality rates after liver transplants in adults by analyzing the relationships between insurance coverage, income, geographic location, and mortality rates post-transplantation. It was hypothesized that a sociodemographic variable, such as insurance type, geographical location, and income level would impact mortality rates post-liver transplant. Results showed that unknown insurance coverage increased the likelihood of mortality post-transplant, income level was not found to be a significant indicator, and patients living in the Northeast region of the United States were more likely to die post-liver transplant.

A novel method for causal structure discovery from EHR data and its application to type-2 diabetes mellitus.

Modern AI-based clinical decision support models owe their success in part to the very large number of predictors they use. Safe and robust decision support, especially for intervention planning, requires causal, not associative, relationships. Traditional methods of causal discovery, clinical trials and extracting biochemical pathways, are resource intensive and may not scale up to the number and complexity of relationships sufficient for precision treatment planning. Computational causal structure discovery (CSD) from electronic health records (EHR) data can represent a solution, however, current CSD methods fall short on EHR data. This paper presents a CSD method tailored to the EHR data. The application of the proposed methodology was demonstrated on type-2 diabetes mellitus. A large EHR dataset from Mayo Clinic was used as development cohort, and another large dataset from an independent health system, M Health Fairview, as external validation cohort. The proposed method achieved very high recall (.95) and substantially higher precision than the general-purpose methods (.84 versus .29, and .55). The causal relationships extracted from the development and external validation cohorts had a high (81%) overlap. Due to the adaptations to EHR data, the proposed method is more suitable for use in clinical decision support than the general-purpose methods.

A Computational Method for Learning Disease Trajectories From Partially Observable EHR Data.

Diseases can show different courses of progression even when patients share the same risk factors. Recent studies have revealed that the use of trajectories, the order in which diseases manifest throughout life, can be predictive of the course of progression. In this study, we propose a novel computational method for learning disease trajectories from EHR data. The proposed method consists of three parts: first, we propose an algorithm for extracting trajectories from EHR data; second, three criteria for filtering trajectories; and third, a likelihood function for assessing the risk of developing a set of outcomes given a trajectory set. We applied our methods to extract a set of disease trajectories from Mayo Clinic EHR data and evaluated it internally based on log-likelihood, which can be interpreted as the trajectories' ability to explain the observed (partial) disease progressions. We then externally evaluated the trajectories on EHR data from an independent health system, M Health Fairview. The proposed algorithm extracted a comprehensive set of disease trajectories that can explain the observed outcomes substantially better than competing methods and the proposed filtering criteria selected a small subset of disease trajectories that are highly interpretable and suffered only a minimal (relative 5%) loss of the ability to explain disease progression in both the internal and external validation.

Applying Machine Learning Across Sites: External Validation of a Surgical Site Infection Detection Algorithm.

BACKGROUND: Surgical complications have tremendous consequences and costs. Complication detection is important for quality improvement, but traditional manual chart review is burdensome. Automated mechanisms are needed to make this more efficient. To understand the generalizability of a machine learning algorithm between sites, automated surgical site infection (SSI) detection algorithms developed at one center were tested at another distinct center. STUDY DESIGN: NSQIP patients had electronic health record (EHR) data extracted at one center (University of Minnesota Medical Center, Site A) over a 4-year period for model development and internal validation, and at a second center (University of California San Francisco, Site B) over a subsequent 2-year period for external validation. Models for automated NSQIP SSI detection of superficial, organ space, and total SSI within 30 days postoperatively were validated using area under the curve (AUC) scores and corresponding 95% confidence intervals. RESULTS: For the 8,883 patients (Site A) and 1,473 patients (Site B), AUC scores were not statistically different for any outcome including superficial (external 0.804, internal [0.784, 0.874] AUC); organ/space (external 0.905, internal [0.867, 0.941] AUC); and total (external 0.855, internal [0.854, 0.908] AUC) SSI. False negative rates decreased with increasing case review volume and would be amenable to a strategy in which cases with low predicted probabilities of SSI could be excluded from chart review. CONCLUSIONS: Our findings demonstrated that SSI detection machine learning algorithms developed at 1 site were generalizable to another institution. SSI detection models are practically applicable to accelerate and focus chart review.

Predicting diabetes clinical outcomes using longitudinal risk factor trajectories.

BACKGROUND: The ubiquity of electronic health records (EHR) offers an opportunity to observe trajectories of laboratory results and vital signs over long periods of time. This study assessed the value of risk factor trajectories available in the electronic health record to predict incident type 2 diabetes. STUDY DESIGN AND METHODS: Analysis was based on a large 13-year retrospective cohort of 71,545 adult, non-diabetic patients with baseline in 2005 and median follow-up time of 8 years. The trajectories of fasting plasma glucose, lipids, BMI and blood pressure were computed over three time frames (2000-2001, 2002-2003, 2004) before baseline. A novel method, Cumulative Exposure (CE), was developed and evaluated using Cox proportional hazards regression to assess risk of incident type 2 diabetes. We used the Framingham Diabetes Risk Scoring (FDRS) Model as control. RESULTS: The new model outperformed the FDRS Model (.802 vs .660; p-values <2e-16). Cumulative exposure measured over different periods showed that even short episodes of hyperglycemia increase the risk of developing diabetes. Returning to normoglycemia moderates the risk, but does not fully eliminate it. The longer an individual maintains glycemic control after a hyperglycemic episode, the lower the subsequent risk of diabetes. CONCLUSION: Incorporating risk factor trajectories substantially increases the ability of clinical decision support risk models to predict onset of type 2 diabetes and provides information about how risk changes over time.

The Value of Aggregated High-Resolution Intraoperative Data for Predicting Post-Surgical Infectious Complications at Two Independent Sites.

Surgical procedures carry the risk of postoperative infectious complications, which can be severe, expensive, and morbid. A growing body of evidence indicates that high-resolution intraoperative data can be predictive of these complications. However, these studies are often contradictory in their findings as well as difficult to replicate, suggesting that these predictive models may be capturing institutional artifacts. In this work, data and models from two independent institutions, Mayo Clinic and University of Minnesota-affiliated Fairview Health Services, were directly compared using a common set of definitions for the variables and outcomes. We built perioperative risk models for seven infectious post-surgical complications at each site to assess the value of intraoperative variables. Models were internally validated. We found that including intraoperative variables significantly improved the models' predictive performance at both sites for five out of seven complications. We also found that significant intraoperative variables were similar between the two sites for four of the seven complications. Our results suggest that intraoperative variables can be related to the underlying physiology for some infectious complications.

A Framework for Visualizing Data Quality for Predictive Models and Clinical Quality Measures.

The ability to assess data quality is essential for secondary use of EHR data and an automated Healthcare Data Quality Framework (HDQF) can be used as a tool to support a healthcare organization's data quality initiatives. Use of a general purpose HDQF provides a method to assess and visualize data quality to quickly identify areas for improvement. The value of the approach is illustrated for two analytics use cases: 1) predictive models and 2) clinical quality measures. The results show that data quality issues can be efficiently identified and visualized. The automated HDQF is much less time consuming than a manual approach to data quality and the framework can be rerun repeatedly on additional datasets without much effort.

Towards more Accessible Precision Medicine: Building a more Transferable Machine Learning Model to Support Prognostic Decisions for Micro- and Macrovascular Complications of Type 2 Diabetes Mellitus.

Although machine learning models are increasingly being developed for clinical decision support for patients with type 2 diabetes, the adoption of these models into clinical practice remains limited. Currently, machine learning (ML) models are being constructed on local healthcare systems and are validated internally with no expectation that they would validate externally and thus, are rarely transferrable to a different healthcare system. In this work, we aim to demonstrate that (1) even a complex ML model built on a national cohort can be transferred to two local healthcare systems, (2) while a model constructed on a local healthcare system's cohort is difficult to transfer; (3) we examine the impact of training cohort size on the transferability; and (4) we discuss criteria for external validity. We built a model using our previously published Multi-Task Learning-based methodology on a national cohort extracted from OptumLabs® Data Warehouse and transferred the model to two local healthcare systems (i.e., University of Minnesota Medical Center and Mayo Clinic) for external evaluation. The model remained valid when applied to the local patient populations and performed as well as locally constructed models (concordance: .73-.92), demonstrating transferability. The performance of the locally constructed models reduced substantially when applied to each other's healthcare system (concordance: .62-.90). We believe that our modeling approach, in which a model is learned from a national cohort and is externally validated, produces a transferable model, allowing patients at smaller healthcare systems to benefit from precision medicine.

Multi-Task Learning to Identify Outcome-Specific Risk Factors that Distinguish Individual Micro and Macrovascular Complications of Type 2 Diabetes.

Because deterioration in overall metabolic health underlies multiple complications of Type 2 Diabetes Mellitus, a substantial overlap among risk factors for the complications exists, and this makes the outcomes difficult to distinguish. We hypothesized each risk factor had two roles: describing the extent of deteriorating overall metabolic health and signaling a particular complication the patient is progressing towards. We aimed to examine feasibility of our proposed methodology that separates these two roles, thereby, improving interpretation of predictions and helping prioritize which complication to target first. To separate these two roles, we built models for six complications utilizing Multi-Task Learning-a machine learning technique for modeling multiple related outcomes by exploiting their commonality-in 80% of EHR data (N=9,793) from a university hospital and validated them in remaining 20% of the data. Additionally, we externally validated the models in claims and EHR data from the OptumLabs™ Data Warehouse (N=72,720). Our methodology successfully separated the two roles, revealing distinguishing outcome-specific risk factors without compromising predictive performance. We believe that our methodology has a great potential to generate more understandable thus actionable clinical information to make a more accurate and timely prognosis for the patients.

A Holistic Clustering Methodology for Liver Transplantation Survival.

BACKGROUND: Liver transplants account for a high number of procedures with major investments from all stakeholders involved; however, limited studies address liver transplant population heterogeneity pretransplant predictive of posttransplant survival. OBJECTIVE: The aim of the study was to identify novel and meaningful patient clusters predictive of mortality that explains the heterogeneity of liver transplant population, taking a holistic approach. METHODS: A retrospective cohort study of 344 adult patients who underwent liver transplantation between 2008 through 2014. Predictors were summarized severity scores for comorbidities and other suboptimal health states grouped into 11 body systems, the primary reason for transplantation, demographics/environmental factors, and Model for End Liver Disease score. Logistic regression was used to compute the severity scores, hierarchical clustering with weighted Euclidean distance for clustering, Lasso-penalized regression for characterizing the clusters, and Kaplan-Meier analysis to compare survival across the clusters. RESULTS: Cluster 1 included patients with more severe circulatory problems. Cluster 2 represented older patients with more severe primary disease, whereas Cluster 3 contained healthiest patients. Clusters 4 and 5 represented patients with musculoskeletal (e.g., pain) and endocrine problems (e.g., malnutrition), respectively. There was a statistically significant difference for mortality between clusters (p < .001). CONCLUSIONS: This study developed a novel methodology to address heterogeneous and high-dimensional liver transplant population characteristics in a single study predictive of survival. A holistic approach for data modeling and additional psychosocial risk factors has the potential to address holistically nursing challenges on liver transplant care and research.

Providers' Response to Clinical Decision Support for QT Prolonging Drugs.

Commonly used drugs in hospital setting can cause QT prolongation and trigger life-threatening arrhythmias. We evaluate changes in prescribing behavior after the implementation of a clinical decision support system to prevent the use of QT prolonging medications in the hospital setting. We conducted a quasi-experimental study, before and after the implementation of a clinical decision support system integrated in the electronic medical record (QT-alert system). This system detects patients at risk of significant QT prolongation (QTc>500ms) and alerts providers ordering QT prolonging drugs. We reviewed the electronic health record to assess the provider's responses which were classified as "action taken" (QT drug avoided, QT drug changed, other QT drug(s) avoided, ECG monitoring, electrolytes monitoring, QT issue acknowledged, other actions) or "no action taken". Approximately, 15.5% (95/612) of the alerts were followed by a provider's action in the pre-intervention phase compared with 21% (228/1085) in the post-intervention phase (p=0.006). The most common type of actions taken during pre-intervention phase compared to post-intervention phase were ECG monitoring (8% vs. 13%, p=0.002) and QT issue acknowledgment (2.1% vs. 4.1%, p=0.03). Notably, there was no significant difference for other actions including QT drug avoided (p=0.8), QT drug changed (p=0.06) and other QT drug(s) avoided (p=0.3). Our study demonstrated that the QT alert system prompted a higher proportion of providers to take action on patients at risk of complications. However, the overall impact was modest underscoring the need for educating providers and optimizing clinical decision support to further reduce drug-induced QT prolongation.

Strategies for handling missing clinical data for automated surgical site infection detection from the electronic health record.

Proper handling of missing data is important for many secondary uses of electronic health record (EHR) data. Data imputation methods can be used to handle missing data, but their use for analyzing EHR data is limited and specific efficacy for postoperative complication detection is unclear. Several data imputation methods were used to develop data models for automated detection of three types (i.e., superficial, deep, and organ space) of surgical site infection (SSI) and overall SSI using American College of Surgeons National Surgical Quality Improvement Project (NSQIP) Registry 30-day SSI occurrence data as a reference standard. Overall, models with missing data imputation almost always outperformed reference models without imputation that included only cases with complete data for detection of SSI overall achieving very good average area under the curve values. Missing data imputation appears to be an effective means for improving postoperative SSI detection using EHR clinical data.

Causal Phenotyping for Susceptibility to Cardiotoxicity from Antineoplastic Breast Cancer Medications.

Cardiotoxicity is a relatively common and particularly important adverse event caused by chemotherapy for breast cancer patients. Typical associative phenotypes, such as risk factors associated with diabetes, can often be detected solely based on the data elements existing in electronic health records; however, causal phenotypes, such as risk factors causing cardiotoxicity, require establishing causation between chemotherapy and determining new heart disease, and cannot be directly observedfrom EHR. We propose three phenotyping algorithms to assess breast cancer patients' susceptibility to cardiotoxicity caused by five first-line antineoplastic drugs: (1) causal phenotype model to predict the patients' risk of cardiotoxicity as the difference between the heart disease risks with exposure and nonexposure to the drugs; (2) regular predictive model; (3) combined predictive model of the above two models. Concordances for three methods were 0.60, 0.62, and 0.68. When considering all exposed patients, concordances were 0.66, 0.58 and 0.65 at 280 days after treatment. The study demonstrates the potential utility of causal phenotyping.

Estimating Disease Onset Time by Modeling Lab Result Trajectories via Bayes Networks.

The true onset time of a disease, particularly slow-onset diseases like Type 2 diabetes mellitus (T2DM), is rarely observable in electronic health records (EHRs). However, it is critical for analysis of time to events and for studying sequences of diseases. The aim of this study is to demonstrate a method for estimating the onset time of such diseases from intermittently observable laboratory results in the specific context of T2DM. A retrospective observational study design is used. A cohort of 5,874 non-diabetic patients from a large healthcare system in the Upper Midwest United States was constructed with a three-year follow-up period. The HbA1c level of each patient was collected from earliest and the latest follow-up. We modeled the patients' HbA1c trajectories through Bayesian networks to estimate the onset time of diabetes. Due to non-random censoring and interventions unobservable from EHR data (such as lifestyle changes), naïve modeling of HbA1c through linear regression or modeling time-to-event through proportional hazard model leads to a clinically infeasible model with no or limited ability to predict the onset time of diabetes. Our model is consistent with clinical knowledge and estimated the onset of diabetes with less than a six-month error for almost half the patients for whom the onset time could be clinically ascertained. To our knowledge, this is the first study of modeling long-term HbA1c progression in non-diabetic patients and estimating the onset time of diabetes.

Characterizing Surgical Site Infection Signals in Clinical Notes.

Surgical site infections (SSIs) are the most common and costly of hospital acquired infections. An important step in reducing SSIs is accurate SSI detection, which enables measurement quality improvement, but currently remains expensive through manual chart review. Building off of previous work for automated and semi-automated SSI detection using expert-derived "strong features" from clinical notes, we hypothesized that additional SSI phrases may be contained in clinical notes. We systematically characterized phrases and expressions associated with SSIs. While 83% of expert-derived original terms overlapped with new terms and modifiers, an additional 362 modifiers associated with both positive and negative SSI signals were identified and 62 new base observations and actions were identified. Clinical note queries with the most common base terms revealed another 49 modifiers. Clinical notes contain a wide variety of expressions describing infections occurring among surgical specialties which may provide value in improving the performance of SSI detection algorithms.

Predictors of Liver Transplant Patient Survival.

OBJECTIVE: Liver transplantation is a costly and risky procedure, representing 25 050 procedures worldwide in 2013, with 6729 procedures performed in the United States in 2014. Considering the scarcity of organs and uncertainty regarding prognosis, limited studies address the variety of risk factors before transplantation that might contribute to predicting patient's survival and therefore developing better models that address a holistic view of transplant patients. This critical review aimed to identify predictors of liver transplant patient survival included in large-scale studies and assess the gap in risk factors from a holistic approach using the Wellbeing Model and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement. DATA SOURCE: Search of the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Medline, and PubMed from the 1980s to July 2014. STUDY SELECTION: Original longitudinal large-scale studies, of 500 or more subjects, published in English, Spanish, or Portuguese, which described predictors of patient survival after deceased donor liver transplantation. DATA EXTRACTION: Predictors were extracted from 26 studies that met the inclusion criteria. DATA SYNTHESIS: Each article was reviewed and predictors were categorized using a holistic framework, the Wellbeing Model (health, community, environment, relationship, purpose, and security dimensions). CONCLUSIONS: The majority (69.7%) of the predictors represented the Wellbeing Model Health dimension. There were no predictors representing the Wellbeing Dimensions for purpose and relationship nor emotional, mental, and spiritual health. This review showed that there is rigorously conducted research of predictors of liver transplant survival; however, the reported significant results were inconsistent across studies, and further research is needed to examine liver transplantation from a whole-person perspective.