Sepsis Prediction at Emergency Department Triage Using Natural Language Processing: Retrospective Cohort Study.

BACKGROUND: Despite its high lethality, sepsis can be difficult to detect on initial presentation to the emergency department (ED). Machine learning-based tools may provide avenues for earlier detection and lifesaving intervention. OBJECTIVE: The study aimed to predict sepsis at the time of ED triage using natural language processing of nursing triage notes and available clinical data. METHODS: We constructed a retrospective cohort of all 1,234,434 consecutive ED encounters in 2015-2021 from 4 separate clinically heterogeneous academically affiliated EDs. After exclusion criteria were applied, the final cohort included 1,059,386 adult ED encounters. The primary outcome criteria for sepsis were presumed severe infection and acute organ dysfunction. After vectorization and dimensional reduction of triage notes and clinical data available at triage, a decision tree-based ensemble (time-of-triage) model was trained to predict sepsis using the training subset (n=950,921). A separate (comprehensive) model was trained using these data and laboratory data, as it became available at 1-hour intervals, after triage. Model performances were evaluated using the test (n=108,465) subset. RESULTS: Sepsis occurred in 35,318 encounters (incidence 3.45%). For sepsis prediction at the time of patient triage, using the primary definition, the area under the receiver operating characteristic curve (AUC) and macro F1-score for sepsis were 0.94 and 0.61, respectively. Sensitivity, specificity, and false positive rate were 0.87, 0.85, and 0.15, respectively. The time-of-triage model accurately predicted sepsis in 76% (1635/2150) of sepsis cases where sepsis screening was not initiated at triage and 97.5% (1630/1671) of cases where sepsis screening was initiated at triage. Positive and negative predictive values were 0.18 and 0.99, respectively. For sepsis prediction using laboratory data available each hour after ED arrival, the AUC peaked to 0.97 at 12 hours. Similar results were obtained when stratifying by hospital and when Centers for Disease Control and Prevention hospital toolkit for adult sepsis surveillance criteria were used to define sepsis. Among septic cases, sepsis was predicted in 36.1% (1375/3814), 49.9% (1902/3814), and 68.3% (2604/3814) of encounters, respectively, at 3, 2, and 1 hours prior to the first intravenous antibiotic order or where antibiotics where not ordered within the first 12 hours. CONCLUSIONS: Sepsis can accurately be predicted at ED presentation using nursing triage notes and clinical information available at the time of triage. This indicates that machine learning can facilitate timely and reliable alerting for intervention. Free-text data can improve the performance of predictive modeling at the time of triage and throughout the ED course.

The Use of a Self-triage Tool to Predict COVID-19 Cases and Hospitalizations in the State of Georgia.

INTRODUCTION: The coronavirus 2019 (COVID-19) pandemic has created significant burden on healthcare systems throughout the world. Syndromic surveillance, which collects real-time data based on a range of symptoms rather than laboratory diagnoses, can help provide timely information in emergency response. We examined the effectiveness of a web-based COVID-19 symptom checking tool (C19Check) in the state of Georgia (GA) in predicting COVID-19 cases and hospitalizations. METHODS: We analyzed C19Check use data, COVID-19 cases, and hospitalizations from April 22-November 28, 2020. Cases and hospitalizations in GA were extracted from the Georgia Department of Public Health data repository. We used the Granger causality test to assess whether including C19Check data can improve predictions compared to using previous COVID-19 cases and hospitalizations data alone. Vector autoregression (VAR) models were fitted to forecast cases and hospitalizations from November 29 - December 12, 2020. We calculated mean absolute percentage error to estimate the errors in forecast of cases and hospitalizations. RESULTS: There were 25,861 C19Check uses in GA from April 22-November 28, 2020. Time-lags tested in Granger causality test for cases (6-8 days) and hospitalizations (10-12 days) were significant (P= <0.05); the mean absolute percentage error of fitted VAR models were 39.63% and 15.86%, respectively. CONCLUSION: The C19Check tool was able to help predict COVID-19 cases and related hospitalizations in GA. In settings where laboratory tests are limited, a real-time, symptom-based assessment tool can provide timely and inexpensive data for syndromic surveillance to guide pandemic response. Findings from this study demonstrate that online symptom-checking tools can be a source of data for syndromic surveillance, and the data may help improve predictions of cases and hospitalizations.

Development and Usability Testing of a Web-based COVID-19 Self-triage Platform.

INTRODUCTION: The development and deployment of a web-based, self-triage tool for severe respiratory syndrome coronavirus 2 (COVID-19 disease) aimed at preventing surges in healthcare utilization could provide easily understandable health guidance with the goal of mitigating unnecessary emergency department (ED) and healthcare visits. We describe the iterative development and usability testing of such a tool. We hypothesized that adult users could understand and recall the recommendations provided by a COVID-19 web-based, self-triage tool. METHODS: We convened a multidisciplinary panel of medical experts at two academic medical schools in an iterative redesign process of a previously validated web-based, epidemic screening tool for the current COVID-19 pandemic. We then conducted a cross-sectional usability study over a 24-hour period among faculty, staff, and students at the two participating universities. Participants were randomly assigned a pre-written health script to enter into the self-triage website for testing. The primary outcome was immediate recall of website recommendations. Secondary outcomes included usability measures. We stratified outcomes by demographic characteristics. RESULTS: A final sample of 877 participants (mean age, 32 years [range, 19-84 years]; 65.3% female) was used in the analysis. We found that 79.4% of the participants accurately recalled the recommendations provided by the website. Almost all participants (96.9%) found the website easy to use and navigate. CONCLUSION: Adult users of a COVID-19 self-triage website, recruited from an academic setting, were able to successfully recall self-care instructions from the website and found it user-friendly. This website appears to be a feasible way to provide evidence-based health guidance to adult patients during a pandemic. Website guidance could be used to reduce unnecessary ED and healthcare visits.

Prediction of emergency department resource requirements during triage: An application of current natural language processing techniques.

OBJECTIVE: Accurate triage in the emergency department (ED) is critical for medical safety and operational efficiency. We aimed to predict the number of future required ED resources, as defined by the Emergency Severity Index (ESI) triage protocol, using natural language processing of nursing triage notes. METHODS: We constructed a retrospective cohort of all 265,572 consecutive ED encounters from 2015 to 2016 from 3 separate clinically heterogeneous academically affiliated EDs. We excluded encounters missing relevant information, leaving 226,317 encounters. We calculated the number of resources used by patients in the ED retrospectively and based outcome categories on criteria defined in the ESI algorithm: 0 (30,604 encounters), 1 (49,315 encounters), and 2 or more (146,398 encounters). A neural network model was trained on a training subset to predict the number of resources using triage notes and clinical variables at triage. Model performance was evaluated using the test subset and was compared with human ratings. RESULTS: Overall model accuracy and macro F1 score for number of resources were 66.5% and 0.601, respectively. The model had similar macro F1 (0.589 vs 0.592) and overall accuracy (65.9% vs 69.0%) compared to human raters. Model predictions had slightly higher F1 scores and accuracy for 0 resources and were less accurate for 2 or more resources. CONCLUSIONS: Machine learning of nursing triage notes, combined with clinical data available at ED presentation, can be used to predict the number of required future ED resources. These findings suggest that machine learning may be a valuable adjunct tool in the initial triage of ED patients.

Prediction of emergency department patient disposition based on natural language processing of triage notes.

BACKGROUND: Nursing triage documentation is the first free-form text data created at the start of an emergency department (ED) visit. These 1-3 unstructured sentences reflect the clinical impression of an experienced nurse and are key in gauging a patient's illness. We aimed to predict final ED disposition using three commonly-employed natural language processing (NLP) techniques of nursing triage notes in isolation from other data. METHODS: We constructed a retrospective cohort of all 260,842 consecutive ED encounters in 2015-16, from three clinically heterogeneous academically-affiliated EDs. After exclusion of 3964 encounters based on completeness of triage, and disposition data, we included 256,878 encounters. We defined the outcome as: 1) admission, transfer, or in-ED death [68,092 encounters] vs. 2) discharge, "left without being seen," and "left against medical advice" [188,786 encounters]. The dataset was divided into training and testing subsets. Neural network regression models were trained using bag-of-words, paragraph vectors, and topic distributions to predict disposition and were evaluated using the testing dataset. RESULTS: Area under the curve for disposition using triage notes as bag-of-words, paragraph vectors, and topic distributions were 0.737 (95% CI: 0.734 - 0.740), 0.785 (95% CI: 0.782 - 0.788), and 0.687 (95% CI: 0.684 - 0.690), respectively. CONCLUSIONS: Nursing triage notes can be used to predict final ED patient disposition, even when used separately from other clinical information. These findings have substantial implications for future studies, suggesting that free text from medical records may be considered as a critical predictor in research of patient outcomes.

Racial and Ethnic Differences in Diagnostic Imaging Utilization During Adult Emergency Department Visits in the United States, 2005 to 2014.

OBJECTIVE: To compare the use of medical imaging (x-ray [XR], CT, ultrasound, and MRI) in the emergency department (ED) for adult patients of different racial and ethnic groups in the United States from 2005 to 2014. METHODS: We performed a multilevel stratified regression analysis of the National Hospital Ambulatory Medical Care Survey ED Subfile, a nationally representative database of hospital-based ED visits. We examined race (white, black, Asian, other) and ethnicity (Hispanic versus non-Hispanic) as the primary exposures for the outcomes of ED medical imaging use (XR, CT, ultrasound, MRI, and any imaging). We controlled for other potential patient-level and facility-level determinants of ED imaging use. RESULTS: Approximately half (48.8%) of the 225,037 adult patient ED visits underwent imaging; 36.1% underwent XR, 16.4% CT, 4.1% ultrasound, and 0.8% MRI. White patients received imaging during 51.3% of their encounters, black patients received imaging during 43.6% of their encounters, Asians received imaging during 50.8% of their encounters, and other races received imaging during 46% of their encounters. As compared with white patients, black patients had decreased adjusted odds of receiving imaging in the ED (odds ratio [OR] = 0.86, 95% confidence interval [CI]: 0.84-0.89). Comparatively, black patients had a lower odds of CT scan (OR = 0.80, 95% CI: 0.77-0.83) or MRI (OR = 0.74, 95% CI: 0.65-0.85). Hispanic patients and Asian patients had a higher odds of receiving ultrasound (OR = 1.36, 95% CI: 1.27-1.44 and OR = 1.25, 95% CI: 1.10-1.42), respectively. IMPLICATIONS: We observed significant racial and ethnic differences in medical imaging use in the ED even after controlling for patient- and facility-level factors.

Advanced diagnostic imaging utilization during emergency department visits in the United States: A predictive modeling study for emergency department triage.

BACKGROUND: Emergency department (ED) crowding is associated with negative health outcomes, patient dissatisfaction, and longer length of stay (LOS). The addition of advanced diagnostic imaging (ADI), namely CT, ultrasound (U/S), and MRI to ED encounter work up is a predictor of longer length of stay. Earlier and improved prediction of patients' need for advanced imaging may improve overall ED efficiency. The aim of the study was to detect the association between ADI utilization and the structured and unstructured information immediately available during ED triage, and to develop and validate models to predict utilization of ADI during an ED encounter. METHODS: We used the United States National Hospital Ambulatory Medical Care Survey data from 2009 to 2014 to examine which sociodemographic and clinical factors immediately available at ED triage were associated with the utilization of CT, U/S, MRI, and multiple ADI during a patient's ED stay. We used natural language processing (NLP) topic modeling to incorporate free-text reason for visit data available at time of ED triage in addition to other structured patient data to predict the use of ADI using multivariable logistic regression models. RESULTS: Among the 139,150 adult ED visits from a national probability sample of hospitals across the U.S, 21.9% resulted in ADI use, including 16.8% who had a CT, 3.6% who had an ultrasound, 0.4% who had an MRI, and 1.2% of the population who had multiple types of ADI. The c-statistic of the predictive models was greater than or equal to 0.78 for all imaging outcomes, and the addition of text-based reason for visit information improved the accuracy of all predictive models. CONCLUSIONS: Patient information immediately available during ED triage can accurately predict the eventual use of advanced diagnostic imaging during an ED visit. Such models have the potential to be incorporated into the ED triage workflow in order to more rapidly identify patients who may require advanced imaging during their ED stay and assist with medical decision-making.

Emergency department use among kidney transplant recipients in the United States.

Patients with end-stage renal disease use the emergency department (ED) at a 6-fold higher rate than do other US adults. No national studies have described ED use rates among kidney transplant (KTx) recipients, and the factors associated with higher ED use. We examined a cohort of 132 725 adult KTx recipients in the United States Renal Data System (2005-2013). Data on ED visits, hospitalization, and outpatient nephrology visits were obtained from Medicare claims databases. Nearly half (46.1%) of KTx recipients had at least one ED visit (1.61 ED visits/patient-year [PY]), and 39.7% of ED visits resulted in hospitalization in the first year posttransplantation. ED visit rate was high in the first 30 days (5.26 visits/PY) but declined substantially thereafter (1.81 visits/PY in months 1-3; 1.13 visits/PY in months 3-12 posttransplantation). ED visit rates were higher in the first 30 days versus rates for dialysis patients but less than half the rate thereafter. Female sex, public insurance, medical comorbidities, longer pretransplantation dialysis vintage, and delayed graft function were associated with higher ED use in the first year post-KTx. Policies and strategies addressing potentially preventable ED visits should be promoted to help improve patient care and increase efficient use of ED resources.

Prediction of Emergency Department Hospital Admission Based on Natural Language Processing and Neural Networks.

OBJECTIVE: To describe and compare logistic regression and neural network modeling strategies to predict hospital admission or transfer following initial presentation to Emergency Department (ED) triage with and without the addition of natural language processing elements. METHODS: Using data from the National Hospital Ambulatory Medical Care Survey (NHAMCS), a cross-sectional probability sample of United States EDs from 2012 and 2013 survey years, we developed several predictive models with the outcome being admission to the hospital or transfer vs. discharge home. We included patient characteristics immediately available after the patient has presented to the ED and undergone a triage process. We used this information to construct logistic regression (LR) and multilayer neural network models (MLNN) which included natural language processing (NLP) and principal component analysis from the patient's reason for visit. Ten-fold cross validation was used to test the predictive capacity of each model and receiver operating curves (AUC) were then calculated for each model. RESULTS: Of the 47,200 ED visits from 642 hospitals, 6,335 (13.42%) resulted in hospital admission (or transfer). A total of 48 principal components were extracted by NLP from the reason for visit fields, which explained 75% of the overall variance for hospitalization. In the model including only structured variables, the AUC was 0.824 (95% CI 0.818-0.830) for logistic regression and 0.823 (95% CI 0.817-0.829) for MLNN. Models including only free-text information generated AUC of 0.742 (95% CI 0.731- 0.753) for logistic regression and 0.753 (95% CI 0.742-0.764) for MLNN. When both structured variables and free text variables were included, the AUC reached 0.846 (95% CI 0.839-0.853) for logistic regression and 0.844 (95% CI 0.836-0.852) for MLNN. CONCLUSIONS: The predictive accuracy of hospital admission or transfer for patients who presented to ED triage overall was good, and was improved with the inclusion of free text data from a patient's reason for visit regardless of modeling approach. Natural language processing and neural networks that incorporate patient-reported outcome free text may increase predictive accuracy for hospital admission.

Pilot Study of the Importance of Factors Affecting Emergency Department Opioid Analgesic Prescribing Decisions.

INTRODUCTION: Little is known about the factors driving decision-making among emergency department (ED) providers when prescribing opioid analgesics (OA). The aim of this pilot study was to identify the importance of factors influencing OA-prescribing decisions and to determine how this varied among different types of providers. METHODS: This was an observational cross-sectional survey study of 203 ED providers. The importance of decisional factors was rated on a 5-point Likert scale. Differences between provider groups were tested using Chi-squared or ANOVA tests where applicable. RESULTS: Overall, 142/203 (69.9 %) potential respondents participated in the study. The five highest-rated factors were (mean ± SD) patient's opioid prescription history (4.4 ± 0.8), patient's history of substance abuse or dependence (4.4 ± 0.7), diagnosis thought to be the cause of patient's pain (4.2 ± 0.8), clinical gestalt (4.2 ± 0.7), and provider's concern about unsafe use of the medication (4.0 ± 0.9). The importance of 6 of 21 decisional factors varied significantly between different groups of providers. CONCLUSION: In this pilot study of ED providers, the self-reported importance of several factors influencing OA-prescribing decisions were significantly different among attending physicians, resident physicians, and advanced practice providers. Further investigation into how ED providers make OA-prescribing decisions is needed to help guide interventions aimed at improving appropriate pain management.

Emergency department resource use by supervised residents vs attending physicians alone.

IMPORTANCE: Few studies have evaluated the common assumption that graduate medical education is associated with increased resource use. OBJECTIVE: To compare resources used in supervised vs attending-only visits in a nationally representative sample of patient visits to US emergency departments (EDs). DESIGN, SETTING, AND PARTICIPANTS: Cross-sectional study of the National Hospital Ambulatory Medical Care Survey (2010), a probability sample of US EDs and ED visits. EXPOSURES: Supervised visits, defined as visits involving both resident and attending physicians. Three ED teaching types were defined by the proportion of sampled visits that were supervised visits: nonteaching ED, minor teaching ED (half or fewer supervised visits), and major teaching ED (more than half supervised visits). MAIN OUTCOMES AND MEASURES: Association of supervised visits with hospital admission, advanced imaging (computed tomography, ultrasound, or magnetic resonance imaging), any blood test, and ED length of stay, adjusted for visit acuity, demographic characteristics, payer type, and geographic region. RESULTS: Of 29,182 ED visits to the 336 nonpediatric EDs in the sample, 3374 visits were supervised visits. Compared with the 25,808 attending-only visits, supervised visits were significantly associated with more frequent hospital admission (21% vs 14%; adjusted odds ratio [aOR], 1.42; 95% CI, 1.09-1.85), advanced imaging (28% vs 21%; aOR, 1.27; 95% CI, 1.06-1.51), and a longer median ED stay (226 vs 153 minutes; adjusted geometric mean ratio, 1.32; 95% CI, 1.19-1.45), but not with blood testing (53% vs 45%; aOR, 1.18; 95% CI, 0.96-1.46). Of visits to the sample of 121 minor teaching EDs, a weighted estimate of 9% were supervised visits, compared with 82% of visits to the 34 major teaching EDs. Supervised visits in major teaching EDs compared with attending-only visits were not associated with hospital admission (aOR, 1.15; 95% CI, 0.83-1.58), advanced imaging (aOR, 1.21; 95% CI, 0.96-1.53), or any blood test (aOR, 1.02; 95% CI, 0.79-1.33), but had longer ED stays (adjusted geometric mean ratio, 1.32; 95% CI, 1.14-1.53). CONCLUSIONS AND RELEVANCE: In a sample of US EDs, supervised visits were associated with a greater likelihood of hospital admission and use of advanced imaging and with longer ED stays. Whether these associations are different in EDs in which more than half of visits are seen by residents requires further investigation.

Does stage of change predict improved intimate partner violence outcomes following an emergency department intervention?

OBJECTIVES: The objective was to assess the effect of an emergency department (ED)-based computer screening and referral intervention on the safety-seeking behaviors of female intimate partner violence (IPV) victims at differing stages of change. The study also aimed to determine which personal and behavioral characteristics were associated with a positive change in safety-seeking behavior. The hypothesis was that women who were in contemplation or action stages of change would be more likely to endorse safety behaviors during follow-up. METHODS: This was a prospective cohort study of female IPV victims at three urban EDs, using a computer kiosk to deliver targeted education about IPV to provide referrals to local resources. All noncritically ill adult English-speaking women triaged to the ED waiting room during study hours were eligible to participate. Women were screened for IPV using the validated Universal Violence Prevention Screening Protocol (UVPSP), and all IPV-positive women further responded to validated questionnaires for alcohol and drug abuse, depression, and IPV severity. The women were assigned a baseline stage of change using the University of Rhode Island Change Assessment (URICA) scale for readiness to change their IPV behaviors. Study participants were contacted at 1 week and 3 months to assess a variety of predetermined safety behaviors to prevent further IPV during that period. Descriptive analyses were performed to determine if stage of change at enrollment and a variety of specific sociodemographic characteristics were associated with taking protective action during follow-up. RESULTS: A total of 1,474 women were screened for IPV; 154 (10.4%) disclosed IPV and completed the full survey. Approximately half (47.4%) of the IPV victims were in the precontemplation stage of change, and 50.0% were in the contemplation stage. A total of 110 women returned at 1 week of follow-up (71.4%), and 63 (40.9%) women returned for the 3-month follow-up. Fifty-five percent of those who returned at 1 week and 73% of those who returned at 3 months took protective action against further IPV. Stage of change at enrollment was not significantly associated with taking protective action during follow-up. There was no association between demographic characteristics and taking protective action at 1 week or 3 months. CONCLUSIONS: Emergency department-based kiosk screening and health information delivery is a feasible method of health information dissemination for women experiencing IPV and was associated with a high proportion of study participants taking protective action. Stage of change was not associated with actual IPV protective measures.