Designing a Clinical Decision Support Tool That Leverages Machine Learning for Suicide Risk Prediction: Development Study in Partnership With Native American Care Providers.

BACKGROUND: Machine learning algorithms for suicide risk prediction have been developed with notable improvements in accuracy. Implementing these algorithms to enhance clinical care and reduce suicide has not been well studied. OBJECTIVE: This study aims to design a clinical decision support tool and appropriate care pathways for community-based suicide surveillance and case management systems operating on Native American reservations. METHODS: Participants included Native American case managers and supervisors (N=9) who worked on suicide surveillance and case management programs on 2 Native American reservations. We used in-depth interviews to understand how case managers think about and respond to suicide risk. The results from interviews informed a draft clinical decision support tool, which was then reviewed with supervisors and combined with appropriate care pathways. RESULTS: Case managers reported acceptance of risk flags based on a predictive algorithm in their surveillance system tools, particularly if the information was available in a timely manner and used in conjunction with their clinical judgment. Implementation of risk flags needed to be programmed on a dichotomous basis, so the algorithm could produce output indicating high versus low risk. To dichotomize the continuous predicted probabilities, we developed a cutoff point that favored specificity, with the understanding that case managers' clinical judgment would help increase sensitivity. CONCLUSIONS: Suicide risk prediction algorithms show promise, but implementation to guide clinical care remains relatively elusive. Our study demonstrates the utility of working with partners to develop and guide the operationalization of risk prediction algorithms to enhance clinical care in a community setting.

Prognostic models will be victims of their own success, unless .

Predictive analytics have begun to change the workflows of healthcare by giving insight into our future health. Deploying prognostic models into clinical workflows should change behavior and motivate interventions that affect outcomes. As users respond to model predictions, downstream characteristics of the data, including the distribution of the outcome, may change. The ever-changing nature of healthcare necessitates maintenance of prognostic models to ensure their longevity. The more effective a model and intervention(s) are at improving outcomes, the faster a model will appear to degrade. Improving outcomes can disrupt the association between the model's predictors and the outcome. Model refitting may not always be the most effective response to these challenges. These problems will need to be mitigated by systematically incorporating interventions into prognostic models and by maintaining robust performance surveillance of models in clinical use. Holistically modeling the outcome and intervention(s) can lead to resilience to future compromises in performance.

A method for analyzing inpatient care variability through physicians' orders.

OBJECTIVE: Administrators assess care variability through chart review or cost variability to inform care standardization efforts. Chart review is costly and cost variability is imprecise. This study explores the potential of physician orders as an alternative measure of care variability. MATERIALS & METHODS: The authors constructed an order variability metric from adult Vanderbilt University Hospital patients treated between 2013 and 2016. The study compared how well a cost variability model predicts variability in the length of stay compared to an order variability model. Both models adjusted for covariates such as severity of illness, comorbidities, and hospital transfers. RESULTS: The order variability model significantly minimized the Akaike information criterion (superior outcome) compared to the cost variability model. This result also held when excluding patients who received intensive care. CONCLUSION: Order variability can potentially typify care variability better than cost variability. Order variability is a scalable metric, calculable during the course of care.

Outpatient Engagement and Predicted Risk of Suicide Attempts in Fibromyalgia.

OBJECTIVE: Patients with fibromyalgia (FM) are 10 times more likely to die by suicide than the general population. The purpose of this study was to externally validate published models predicting suicidal ideation and suicide attempts in patients with FM and to identify interpretable risk and protective factors for suicidality unique to FM. METHODS: This was a case-control study of large-scale electronic health record data collected from 1998 to 2017, identifying FM cases with validated Phenotype KnowledgeBase criteria. Model performance was measured through discrimination, including the receiver operating area under the curve (AUC), sensitivity, and specificity, and through calibration, including calibration plots. Risk factors were selected by L1 penalized regression with bootstrapping for both outcomes. Secondary utilization analyses converted time-based billing codes to equivalent minutes to estimate face-to-face provider contact. RESULTS: We identified 8,879 patients with FM, with 34 known suicide attempts and 96 documented cases of suicidal ideation. External validity was good for both suicidal ideation (AUC 0.80) and attempts (AUC 0.82) with excellent calibration. Risk factors specific to suicidal ideation included polysomatic symptoms such as fatigue (odds ratio [OR] 1.29 [95% confidence interval (95% CI) 1.25-1.32]), dizziness (OR 1.25 [95% CI 1.22-1.28]), and weakness (OR 1.17 [95% CI 1.15-1.19]). Risk factors specific to suicide attempt included obesity (OR 1.18 [95% CI 1.10-1.27]) and drug dependence (OR 1.15 [95% CI 1.12-1.18]). Per utilization analyses, those patients with FM and no suicidal ideation spent 3.5 times more time in follow-up annually, and those without documented suicide attempts spent more than 40 times more time face-to-face with providers annually. CONCLUSION: This is the first study to successfully apply machine learning to reliably detect suicidality in patients with FM, identifying novel risk factors for suicidality and highlighting outpatient engagement as a protective factor against suicide.

Discovering hidden knowledge through auditing clinical diagnostic knowledge bases.

OBJECTIVE: Evaluate potential for data mining auditing techniques to identify hidden concepts in diagnostic knowledge bases (KB). Improving completeness enhances KB applications such as differential diagnosis and patient case simulation. MATERIALS AND METHODS: Authors used unsupervised (Pearson's correlation - PC, Kendall's correlation - KC, and a heuristic algorithm - HA) methods to identify existing and discover new finding-finding interrelationships ("properties") in the INTERNIST-1/QMR KB. Authors estimated KB maintenance efficiency gains (effort reduction) of the approaches. RESULTS: The methods discovered new properties at 95% CI rates of [0.1%, 5.4%] (PC), [2.8%, 12.5%] (KC), and [5.6%, 18.8%] (HA). Estimated manual effort reduction for HA-assisted determination of new properties was approximately 50-fold. CONCLUSION: Data mining can provide an efficient supplement to ensuring the completeness of finding-finding interdependencies in diagnostic knowledge bases. Authors' findings should be applicable to other diagnostic systems that record finding frequencies within diseases (e.g., DXplain, ISABEL).

Balancing Performance and Interpretability: Selecting Features with Bootstrapped Ridge Regression.

Informctticists sometimes attempt to predict chronic healthcare events that are not fully understood. The resulting models often incorporate copious numbers of predictors derived across diverse datasets. This approach may yield desirable performance characteristics, but it sacrifices interpretability and portability. The Bootstrapped Ridge Selector (BoRidge) offers a tool to balance performance with interpretability. Compared to two modern feature selection methods, Bootstrapped LASSO regression (BoLASSO) and a minimal-redundancy-maximal-relevance selector (mRMR), the BoRidge bested them for binary classification on artificially generated data (sensitivity: 0.83, specificity:0.72) versus BoLASSO (sensitivity: 0.1, specificity:1) and mRMR (sensitivity: 0.69, specificity: 0.69). On a dataset used to validate a published suicide risk prediction model, the BoRidge selected an equally precise model to the publication, with far fewer predictors (114 versus the 1,538 used in the published model). The BoRidge has the potential to simplify classification models for complex problems, making them easier to translate and act upon.

X Marks the Spot: Mapping Similarity Between Clinical Trial Cohorts and US Counties.

When patients and doctors collaborate to make healthcare decisions, they rely on clinical trial results to guide discussions. Trials are designed to recruit diverse participants. The question remains - how well do trial results apply to me or to people who live in our area? This study compared one complete clinical trial dataset (SPRINT) and one published study (ACCORD) to the Community Health Status Indicators dataset to assess the similarity of the trial populations to US county populations. Counties up to 495 miles to the closest SPRINT trial site and up to 712 miles to the closest ACCORD trial site had populations that were significantly more similar to the study cohort than counties farther away. The investigators detail a generalizable method for both assessing recruitment gaps in large multicenter trials and creating maps for clinicians to provide intuition on trial applicability in their area.