Leveraging Natural Language Processing to Improve Electronic Health Record Suicide Risk Prediction for Veterans Health Administration Users.

Background: Suicide risk prediction models frequently rely on structured electronic health record (EHR) data, including patient demographics and health care usage variables. Unstructured EHR data, such as clinical notes, may improve predictive accuracy by allowing access to detailed information that does not exist in structured data fields. To assess comparative benefits of including unstructured data, we developed a large case-control dataset matched on a state-of-the-art structured EHR suicide risk algorithm, utilized natural language processing (NLP) to derive a clinical note predictive model, and evaluated to what extent this model provided predictive accuracy over and above existing predictive thresholds.Methods: We developed a matched case-control sample of Veterans Health Administration (VHA) patients in 2017 and 2018. Each case (all patients that died by suicide in that interval, n = 4,584) was matched with 5 controls (patients who remained alive during treatment year) who shared the same suicide risk percentile. All sample EHR notes were selected and abstracted using NLP methods. We applied machine-learning classification algorithms to NLP output to develop predictive models. We calculated area under the curve (AUC) and suicide risk concentration to evaluate predictive accuracy overall and for high-risk patients.Results: The best performing NLP-derived models provided 19% overall additional predictive accuracy (AUC = 0.69; 95% CI, 0.67, 0.72) and 6-fold additional risk concentration for patients at the highest risk tier (top 0.1%), relative to the structured EHR model.Conclusions: The NLP-supplemented predictive models provided considerable benefit when compared to conventional structured EHR models. Results support future structured and unstructured EHR risk model integrations.

Dynamic suicide topic modelling: Deriving population-specific, psychosocial and time-sensitive suicide risk variables from Electronic Health Record psychotherapy notes.

In the machine learning subfield of natural language processing, a topic model is a type of unsupervised method that is used to uncover abstract topics within a corpus of text. Dynamic topic modelling (DTM) is used for capturing change in these topics over time. The study deploys DTM on corpus of electronic health record psychotherapy notes. This retrospective study examines whether DTM helps distinguish closely matched patients that did and did not die by suicide. Cohort consists of United States Department of Veterans Affairs (VA) patients diagnosed with Posttraumatic Stress Disorder (PTSD) between 2004 and 2013. Each case (those who died by suicide during the year following diagnosis) was matched with five controls (those who remained alive) that shared psychotherapists and had similar suicide risk based on VA's suicide prediction algorithm. Cohort was restricted to patients who received psychotherapy for 9+ months after initial PTSD diagnoses (cases = 77; controls = 362). For cases, psychotherapy notes from diagnosis until death were examined. For controls, psychotherapy notes from diagnosis until matched case's death date were examined. A Python-based DTM algorithm was utilized. Derived topics identified population-specific themes, including PTSD, psychotherapy, medication, communication and relationships. Control topics changed significantly more over time than case topics. Topic differences highlighted engagement, expressivity and therapeutic alliance. This study strengthens groundwork for deriving population-specific, psychosocial and time-sensitive suicide risk variables.

Tools to Detect Risk of Death by Suicide: A Systematic Review and Meta-Analysis.

Objective: There is limited knowledge about the ability of instruments to detect risk of suicide in a range of settings. Prior reviews have not considered whether the utility of instruments depends on prior probability of risk. We performed a systematic review to determine the diagnostic accuracy of instruments to detect risk of suicide in adults using likelihood ratio analysis. This method aids evaluation of prior probabilities of risk.Data Sources: We searched MEDLINE, Cochrane Database of Systematic Reviews, PsycINFO, EMBASE, and Scopus from inception through January 19, 2021.Study Selection: We included clinical trials, observational studies, and quasi-experimental studies assessing the diagnostic accuracy of instruments to detect risk of suicide in adults. There were no language restrictions.Data Extraction: Three reviewers in duplicate assessed full texts to determine eligibility and extracted data from included studies. Positive (LR+) and negative likelihood ratio (LR-) and 95% CIs were calculated for each instrument.Results: Thirty studies met inclusion criteria. Most instruments showed minimal utility to detect or rule out risk of suicide, with LR+ ≤ 2.0 and LR- ≥ 0.5. A few instruments had a high utility for improving risk detection in emergency department, inpatient mental health, and prison settings when patients scored above the cutoff (LR+ > 10). For example, among patients discharged from an emergency department, the Columbia Suicide Severity Rating Scale-Clinical Practice Screener had a LR+ of 10.3 (95% CI, 6.3-16.8) at 3-month follow-up. The clinical utility of the instruments depends on the pretest probability of suicide in the setting. Because studies spanned over 6 decades, the findings are at risk for secular trends.Discussion: We identified several instruments that may hold promise for detecting risk of suicide in emergency department, inpatient mental health, or prison settings. The utility of the instrument hinges, in part, on baseline suicide risk.Registration: PROSPERO: CRD42021285528.

Leveraging unstructured electronic medical record notes to derive population-specific suicide risk models.

Electronic medical record (EMR)-based suicide risk prediction methods typically rely on analysis of structured variables such as demographics, visit history, and prescription data. Leveraging unstructured EMR notes may improve predictive accuracy by allowing access to nuanced clinical information. We utilized natural language processing (NLP) to analyze a large EMR note corpus to develop a data-driven suicide risk prediction model. We developed a matched case-control sample of U.S. Department of Veterans Affairs (VA) patients in 2015 and 2016. We randomly matched each case (all patients that died by suicide in that interval, n = 5029) with five controls (patients that remained alive). We processed note corpus using NLP methods and applied machine-learning classification algorithms to output. We calculated area under the curve (AUC) and risk tiers to determine predictive accuracy. NLP-derived models demonstrated strong predictive accuracy. Patients that scored within top 10% of risk model accounted for up to 29% of suicide decedents. NLP-derived model compares positively to other leading prediction methods. Our approach is highly implementable, only requiring access to text data and open-source software. Additional studies should evaluate ensemble models incorporating NLP-derived information alongside more typical structured variables.

National cross-sectional cohort study of the relationship between quality of mental healthcare and death by suicide.

BACKGROUND: Patient safety-based interventions aimed at lethal means restriction are effective at reducing death by suicide in inpatient mental health settings but are more challenging in the outpatient arena. As an alternative approach, we examined the association between quality of mental healthcare and suicide in a national healthcare system. METHODS: We calculated regional suicide rates for Department of Veterans Affairs (VA) Healthcare users from 2013 to 2017. To control for underlying variation in suicide risk in each of our 115 mental health referral regions (MHRRs), we calculated standardised rate ratios (SRRs) for VA users compared with the general population. We calculated quality metrics for outpatient mental healthcare in each MHRR using individual metrics as well as an Overall Quality Index. We assessed the correlation between quality metrics and suicide rates. RESULTS: Among the 115 VA MHRRs, the age-adjusted, sex-adjusted and race-adjusted annual suicide rates varied from 6.8 to 92.9 per 100 000 VA users, and the SRRs varied between 0.7 and 5.7. Mean regional-level adherence to each of our quality metrics ranged from a low of 7.7% for subspecialty care access to a high of 58.9% for care transitions. While there was substantial regional variation in quality, there was no correlation between an overall index of mental healthcare quality and SRR. CONCLUSION: There was no correlation between overall quality of outpatient mental healthcare and rates of suicide in a national healthcare system. Although it is possible that quality was not high enough anywhere to prevent suicide at the population level or that we were unable to adequately measure quality, this examination of core mental health services in a well-resourced system raises doubts that a quality-based approach alone can lower population-level suicide rates.

Is It Time for the Mental Health Field to Consider Unplanned Discharge a Key Metric of Patient Safety?

ABSTRACT: Mental health lacks robust measures to assess patient safety. Unplanned discharge is common in mental health populations and associated with poor outcomes. Clarifying whether unplanned discharge varies across settings may highlight the need to develop measures to reduce harms associated with this event. Unplanned discharge rates were compared across the Department of Veterans Affairs' acute inpatient and residential mental health treatment settings from 2009 to 2019. Logistic regression was used to create facility-level, adjusted unplanned discharge rates stratified by setting. Results were described using central tendency. Among 847,661 acute inpatient discharges, the mean unplanned discharge rate was 3.3% (range, 0%-18%). Among 358,117 residential discharges, the mean unplanned discharge rate was 17.9% (range, 1%-48.3%). Unplanned discharge is a marked problem in mental health, with large variation across treatment settings. Unplanned discharge should be measured as part of patient safety efforts.

An implementation and effectiveness study evaluating Conflict Analysis in VA residential substance abuse services: Whole Health informed self-guided online care.

CONTEXT: Whole Health is an emerging healthcare framework that emphasizes wellbeing in place of illness. Conflict Analysis (CA), an online self-guided assessment, leverages innovative diagnostic and therapeutic resources that shares Whole Health objectives, including helping users explore their identity and develop a personalized health plan and helping users develop resources to optimize their health. OBJECTIVES: Paper presents CA implementation-effectiveness study in a Veteran Affairs inpatient substance recovery care. DESIGN: Patients were randomized to CA or mindfulness control. Patients completed Whole Health outcomes measures at baseline, completion (post), and three-week follow-up. Interventions took 2.5 h. Attending psychologist assessed CA protocols and completed outcome evaluation. Due to Coronavirus, recruitment and follow-up were curtailed. SETTING: Study took place in a rural northern New England Veteran Affairs inpatient substance recovery unit. OUTCOME MEASURES: Measures include The Personal Growth Initiative Scale, The Beck Cognitive Insight Scale, Perceived Stress Scale, The Patient Health Questionnaire, Perceived Psychological Wellbeing, and Perceived Therapeutic and Diagnostic Benefit. RESULTS: 12 patients were randomized, 11 completed post measures (CA=5; Mindfulness = 6), and 7 completed follow-up measures (CA=3; Mindfulness=4). CA offered significant Whole Health benefits when compared to control. Additionally, participant and clinician evaluations indicated that CA can be personally relevant, meaningful, and motivate therapeutic growth. Implications include extending CA research and expanding Whole Health related interventions. Although initial results suggest implementation feasibility and Whole Health benefit, more research is necessary to establish CA's utility within inpatient substance recovery care in particular and psychiatric rehabilitation in general.

Improvements to PTSD quality metrics with natural language processing.

RATIONALE AIMS AND OBJECTIVES: As quality measurement becomes increasingly reliant on the availability of structured electronic medical record (EMR) data, clinicians are asked to perform documentation using tools that facilitate data capture. These tools may not be available, feasible, or acceptable in all clinical scenarios. Alternative methods of assessment, including natural language processing (NLP) of clinical notes, may improve the completeness of quality measurement in real-world practice. Our objective was to measure the quality of care for a set of evidence-based practices using structured EMR data alone, and then supplement those measures with additional data derived from NLP. METHOD: As a case example, we studied the quality of care for posttraumatic stress disorder (PTSD) in the United States Department of Veterans Affairs (VA) over a 20-year period. We measured two aspects of PTSD care, including delivery of evidence-based psychotherapy (EBP) and associated use of measurement-based care (MBC), using structured EMR data. We then recalculated these measures using additional data derived from NLP of clinical note text. RESULTS: There were 2 098 389 VA patients with a diagnosis of PTSD between 2000 and 2019, 72% (n = 1 515 345) of whom had not previously received EBP for PTSD and were treated after a 2015 mandate to document EBP using templates that generate structured EMR data. Using structured EMR data, we determined that 3.2% (n = 48 004) of those patients met our EBP for PTSD quality standard between 2015 and 2019, and 48.1% (n = 23 088) received associated MBC. With the addition of NLP-derived data, estimates increased to 4.1% (n = 62 789) and 58.0% (n = 36 435), respectively. CONCLUSION: Healthcare quality data can be significantly improved by supplementing structured EMR data with NLP-derived data. By using NLP, health systems may be able to fill the gaps in documentation when structured tools are not yet available or there are barriers to using them in clinical practice.

The role of race in rural-urban suicide disparities.

PURPOSE: To assess the role that race-ethnicity plays in modifying the observed rural-urban disparity in suicide among Veteran Health Administration (VHA) users. METHODS: We performed a retrospective cohort study of 10,737,864 VHA users between 2003 and 2017, using cross-linked VHA medical records and National Death Index mortality data to assess longitudinal race-stratified rural-urban differences in age- and sex-adjusted annual suicide rates. We used Poisson regression and generated incident rate ratios (IRRs) to formally assess the impact of race on the rural-urban suicide disparity. Given evidence of effect modification, we performed additional race-stratified Poisson regression models. FINDINGS: Rurality is significantly associated with a higher risk of suicide in models which do not control for race (IRR = 1.14, 95% CI: 1.10-1.17). However, when race is added to the model, rural residence is no longer significant (0.98, CI: 0.95-1.01). Stratified models demonstrate that rural residence is significantly associated with a higher suicide risk among Hispanic VHA users (1.41, CI: 1.11-1.79), but it is not substantially associated with suicide among White (0.97, CI: 0.94-1.00) and Black (1.03, CI: 0.86-1.23) VHA users. White VHA users have considerably higher suicide rates than Black and Hispanic VHA users, though the suicide rate among Hispanic VHA users, particularly those in rural settings, increased markedly over the period of observation. CONCLUSIONS: Race significantly modifies the relationship between rural residence and suicide risk. Studies seeking to assess suicide disparity between rural and urban VHA user populations must include adjustment or stratification by race.

Irregular hospital discharge from acute inpatient and residential mental health treatment settings in a large integrated healthcare system.

OBJECTIVE: Irregular discharge is a concern among mental health populations and associated with poor outcomes. Little is known about the relationship between irregular discharge and treatment setting. Because care processes differ between acute inpatient and residential settings, it is important to evaluate irregular discharge in these settings. METHOD: A retrospective study was conducted in patients with mental health conditions admitted to acute inpatient or residential mental health settings in the Department of Veterans Affairs, 2003-2019. Logistic regression and multivariate Cox proportional hazards were used to evaluate factors associated with irregular discharge risk in the first 90- days of admission. RESULTS: Among 1.8 million discharges, 7.4% had an irregular discharge within 90- days of admission. Younger age was a central predictor of risk. Irregular discharge rates were four-fold higher in residential versus acute settings. When accounting for length of stay (LOS) across settings, there was a modest higher risk of irregular discharge from acute versus residential settings (HR = 1.06, 95% Confidence Interval 1.04-1.07). CONCLUSIONS: Patients are at high risk for irregular discharge from acute and residential settings when they are young. LOS is an important determinant of irregular discharge risk.. Interventions are needed to address drivers of irregular discharge.

Adaptation of an NLP system to a new healthcare environment to identify social determinants of health.

Social determinants of health (SDoH) are increasingly important factors for population health, healthcare outcomes, and care delivery. However, many of these factors are not reliably captured within structured electronic health record (EHR) data. In this work, we evaluated and adapted a previously published NLP tool to include additional social risk factors for deployment at Vanderbilt University Medical Center in an Acute Myocardial Infarction cohort. We developed a transformation of the SDoH outputs of the tool into the OMOP common data model (CDM) for re-use across many potential use cases, yielding performance measures across 8 SDoH classes of precision 0.83 recall 0.74 and F-measure of 0.78.

Natural language processing of clinical mental health notes may add predictive value to existing suicide risk models.

BACKGROUND: This study evaluated whether natural language processing (NLP) of psychotherapy note text provides additional accuracy over and above currently used suicide prediction models. METHODS: We used a cohort of Veterans Health Administration (VHA) users diagnosed with post-traumatic stress disorder (PTSD) between 2004-2013. Using a case-control design, cases (those that died by suicide during the year following diagnosis) were matched to controls (those that remained alive). After selecting conditional matches based on having shared mental health providers, we chose controls using a 5:1 nearest-neighbor propensity match based on the VHA's structured Electronic Medical Records (EMR)-based suicide prediction model. For cases, psychotherapist notes were collected from diagnosis until death. For controls, psychotherapist notes were collected from diagnosis until matched case's date of death. After ensuring similar numbers of notes, the final sample included 246 cases and 986 controls. Notes were analyzed using Sentiment Analysis and Cognition Engine, a Python-based NLP package. The output was evaluated using machine-learning algorithms. The area under the curve (AUC) was calculated to determine models' predictive accuracy. RESULTS: NLP derived variables offered small but significant predictive improvement (AUC = 0.58) for patients that had longer treatment duration. A small sample size limited predictive accuracy. CONCLUSIONS: Study identifies a novel method for measuring suicide risk over time and potentially categorizing patient subgroups with distinct risk sensitivities. Findings suggest leveraging NLP derived variables from psychotherapy notes offers an additional predictive value over and above the VHA's state-of-the-art structured EMR-based suicide prediction model. Replication with a larger non-PTSD specific sample is required.

Contextual assessment: evaluating a novel self-guided online therapeutic assessment.

BACKGROUND: Paper introduces Conflict Analysis (CA), an online self-guided therapeutic assessment. CA combines a diagnostic self-report scale with narrative exercises and self-analytical tasks. CA automatically generates detailed diagnostic records and frameworks for changes. OBJECTIVE: To evaluate therapeutic and diagnostic benefits associated with CA over time. METHODS: This online study compared CA over 2 weeks on outcome measures predicting psychotherapy outcome. Novel scale measuring perceived diagnostic benefit and perceived therapeutic benefit was delivered at post and follow-up. Cohort (n = 59, average age = 35, 50% female) was either in therapy or interested to start therapy in near future. RESULTS: Repeated-measure ANOVAs suggest that scores significantly changed on measures predicting negative affect, depression, performance and appearance self-esteem, insight, and growth initiative. Agreement rates on items measuring perceived diagnostic and therapeutic benefits were at least 74.5% for both post and follow-up. CONCLUSIONS: Evidence supports further exploration of CA as a self-guided diagnostic and therapeutic resource.Key pointsResults demonstrate feasibility and utility of online self-guided therapeutic assessment.Described model is associated with increased perceived diagnostic and therapeutic benefits.Described model illustrates therapeutic benefits over time.Results demonstrate that even self-guided assessment can have therapeutic implications.