Multisite Pragmatic Cluster-Randomized Controlled Trial of the CONCERN Early Warning System.

Importance: Late predictions of hospitalized patient deterioration, resulting from early warning systems (EWS) with limited data sources and/or a care team's lack of shared situational awareness, contribute to delays in clinical interventions. The COmmunicating Narrative Concerns Entered by RNs (CONCERN) Early Warning System (EWS) uses real-time nursing surveillance documentation patterns in its machine learning algorithm to identify patients' deterioration risk up to 42 hours earlier than other EWSs. Objective: To test our a priori hypothesis that patients with care teams informed by the CONCERN EWS intervention have a lower mortality rate and shorter length of stay (LOS) than the patients with teams not informed by CONCERN EWS. Design: One-year multisite, pragmatic controlled clinical trial with cluster-randomization of acute and intensive care units to intervention or usual-care groups. Setting: Two large U.S. health systems. Participants: Adult patients admitted to acute and intensive care units, excluding those on hospice/palliative/comfort care, or with Do Not Resuscitate/Do Not Intubate orders. Intervention: The CONCERN EWS intervention calculates patient deterioration risk based on nurses' concern levels measured by surveillance documentation patterns, and it displays the categorical risk score (low, increased, high) in the electronic health record (EHR) for care team members. Main Outcomes and Measures: Primary outcomes: in-hospital mortality, LOS; survival analysis was used. Secondary outcomes: cardiopulmonary arrest, sepsis, unanticipated ICU transfers, 30-day hospital readmission. Results: A total of 60 893 hospital encounters (33 024 intervention and 27 869 usual-care) were included. Both groups had similar patient age, race, ethnicity, and illness severity distributions. Patients in the intervention group had a 35.6% decreased risk of death (adjusted hazard ratio [HR], 0.644; 95% confidence interval [CI], 0.532-0.778; P<.0001), 11.2% decreased LOS (adjusted incidence rate ratio, 0.914; 95% CI, 0.902-0.926; P<.0001), 7.5% decreased risk of sepsis (adjusted HR, 0.925; 95% CI, 0.861-0.993; P=.0317), and 24.9% increased risk of unanticipated ICU transfer (adjusted HR, 1.249; 95% CI, 1.093-1.426; P=.0011) compared with patients in the usual-care group. Conclusions and Relevance: A hospital-wide EWS based on nursing surveillance patterns decreased in-hospital mortality, sepsis, and LOS when integrated into the care team's EHR workflow. Trial Registration: ClinicalTrials.gov Identifier: NCT03911687.

Quantifying emergency department nursing workload at the task level using NASA-TLX: An exploratory descriptive study.

BACKGROUND: Emergency departments (ED) nurses experience high mental workloads because of unpredictable work environments; however, research evaluating ED nursing workload using a tool incorporating nurses' perception is lacking. Quantify ED nursing subjective workload and explore the impact of work experience on perceived workload. METHODS: Thirty-two ED nurses at a tertiary academic hospital in the Republic of Korea were surveyed to assess their subjective workload for ED procedures using the National Aeronautics and Space Administration Task Load Index (NASA-TLX). Nonparametric statistical analysis was performed to describe the data, and linear regression analysis was conducted to estimate the impact of work experience on perceived workload. RESULTS: Cardiopulmonary resuscitation (CPR) had the highest median workload, followed by interruption from a patient and their family members. Although inexperienced nurses perceived the 'special care' procedures (CPR and defibrillation) as more challenging compared with other categories, analysis revealed that nurses with more than 107 months of experience reported a significantly higher workload than those with less than 36 months of experience. CONCLUSION: Addressing interruptions and customizing training can alleviate ED nursing workload. Quantified perceived workload is useful for identifying acceptable thresholds to maintain optimal workload, which ultimately contributes to predicting nursing staffing needs and ED crowding.

Improved accuracy and efficiency of primary care fall risk screening of older adults using a machine learning approach.

BACKGROUND: While many falls are preventable, they remain a leading cause of injury and death in older adults. Primary care clinics largely rely on screening questionnaires to identify people at risk of falls. Limitations of standard fall risk screening questionnaires include suboptimal accuracy, missing data, and non-standard formats, which hinder early identification of risk and prevention of fall injury. We used machine learning methods to develop and evaluate electronic health record (EHR)-based tools to identify older adults at risk of fall-related injuries in a primary care population and compared this approach to standard fall screening questionnaires. METHODS: Using patient-level clinical data from an integrated healthcare system consisting of 16-member institutions, we conducted a case-control study to develop and evaluate prediction models for fall-related injuries in older adults. Questionnaire-derived prediction with three questions from a commonly used fall risk screening tool was evaluated. We then developed four temporal machine learning models using routinely available longitudinal EHR data to predict the future risk of fall injury. We also developed a fall injury-prevention clinical decision support (CDS) implementation prototype to link preventative interventions to patient-specific fall injury risk factors. RESULTS: Questionnaire-based risk screening achieved area under the receiver operating characteristic curve (AUC) up to 0.59 with 23% to 33% similarity for each pair of three fall injury screening questions. EHR-based machine learning risk screening showed significantly improved performance (best AUROC = 0.76), with similar prediction performance between 6-month and one-year prediction models. CONCLUSIONS: The current method of questionnaire-based fall risk screening of older adults is suboptimal with redundant items, inadequate precision, and no linkage to prevention. A machine learning fall injury prediction method can accurately predict risk with superior sensitivity while freeing up clinical time for initiating personalized fall prevention interventions. The developed algorithm and data science pipeline can impact routine primary care fall prevention practice.

Predicting hospitalization of COVID-19 positive patients using clinician-guided machine learning methods.

OBJECTIVES: The coronavirus disease 2019 (COVID-19) is a resource-intensive global pandemic. It is important for healthcare systems to identify high-risk COVID-19-positive patients who need timely health care. This study was conducted to predict the hospitalization of older adults who have tested positive for COVID-19. METHODS: We screened all patients with COVID test records from 11 Mass General Brigham hospitals to identify the study population. A total of 1495 patients with age 65 and above from the outpatient setting were included in the final cohort, among which 459 patients were hospitalized. We conducted a clinician-guided, 3-stage feature selection, and phenotyping process using iterative combinations of literature review, clinician expert opinion, and electronic healthcare record data exploration. A list of 44 features, including temporal features, was generated from this process and used for model training. Four machine learning prediction models were developed, including regularized logistic regression, support vector machine, random forest, and neural network. RESULTS: All 4 models achieved area under the receiver operating characteristic curve (AUC) greater than 0.80. Random forest achieved the best predictive performance (AUC = 0.83). Albumin, an index for nutritional status, was found to have the strongest association with hospitalization among COVID positive older adults. CONCLUSIONS: In this study, we developed 4 machine learning models for predicting general hospitalization among COVID positive older adults. We identified important clinical factors associated with hospitalization and observed temporal patterns in our study cohort. Our modeling pipeline and algorithm could potentially be used to facilitate more accurate and efficient decision support for triaging COVID positive patients.

Detecting Language Associated With Home Healthcare Patient's Risk for Hospitalization and Emergency Department Visit.

BACKGROUND: About one in five patients receiving home healthcare (HHC) services are hospitalized or visit an emergency department (ED) during a home care episode. Early identification of at-risk patients can prevent these negative outcomes. However, risk indicators, including language in clinical notes that indicate a concern about a patient, are often hidden in narrative documentation throughout their HHC episode. OBJECTIVE: The aim of the study was to develop an automated natural language processing (NLP) algorithm to identify concerning language indicative of HHC patients' risk of hospitalizations or ED visits. METHODS: This study used the Omaha System-a standardized nursing terminology that describes problems/signs/symptoms that can occur in the community setting. First, five HHC experts iteratively reviewed the Omaha System and identified concerning concepts indicative of HHC patients' risk of hospitalizations or ED visits. Next, we developed and tested an NLP algorithm to identify these concerning concepts in HHC clinical notes automatically. The resulting NLP algorithm was applied on a large subset of narrative notes (2.3 million notes) documented for 66,317 unique patients ( n = 87,966 HHC episodes) admitted to one large HHC agency in the Northeast United States between 2015 and 2017. RESULTS: A total of 160 Omaha System signs/symptoms were identified as concerning concepts for hospitalizations or ED visits in HHC. These signs/symptoms belong to 31 of the 42 available Omaha System problems. Overall, the NLP algorithm showed good performance in identifying concerning concepts in clinical notes. More than 18% of clinical notes were detected as having at least one concerning concept, and more than 90% of HHC episodes included at least one Omaha System problem. The most frequently documented concerning concepts were pain, followed by issues related to neuromusculoskeletal function, circulation, mental health, and communicable/infectious conditions. CONCLUSION: Our findings suggest that concerning problems or symptoms that could increase the risk of hospitalization or ED visit were frequently documented in narrative clinical notes. NLP can automatically extract information from narrative clinical notes to improve our understanding of care needs in HHC. Next steps are to evaluate which concerning concepts identified in clinical notes predict hospitalization or ED visit.

The Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) Clinical Decision Support Early Warning System: Protocol for a Cluster Randomized Pragmatic Clinical Trial.

BACKGROUND: Every year, hundreds of thousands of inpatients die from cardiac arrest and sepsis, which could be avoided if those patients' risk for deterioration were detected and timely interventions were initiated. Thus, a system is needed to convert real-time, raw patient data into consumable information that clinicians can utilize to identify patients at risk of deterioration and thus prevent mortality and improve patient health outcomes. The overarching goal of the COmmunicating Narrative Concerns Entered by Registered Nurses (CONCERN) study is to implement and evaluate an early warning score system that provides clinical decision support (CDS) in electronic health record systems. With a combination of machine learning and natural language processing, the CONCERN CDS utilizes nursing documentation patterns as indicators of nurses' increased surveillance to predict when patients are at the risk of clinical deterioration. OBJECTIVE: The objective of this cluster randomized pragmatic clinical trial is to evaluate the effectiveness and usability of the CONCERN CDS system at 2 different study sites. The specific aim is to decrease hospitalized patients' negative health outcomes (in-hospital mortality, length of stay, cardiac arrest, unanticipated intensive care unit transfers, and 30-day hospital readmission rates). METHODS: A multiple time-series intervention consisting of 3 phases will be performed through a 1-year period during the cluster randomized pragmatic clinical trial. Phase 1 evaluates the adoption of our algorithm through pilot and trial testing, phase 2 activates optimized versions of the CONCERN CDS based on experience from phase 1, and phase 3 will be a silent release mode where no CDS is viewable to the end user. The intervention deals with a series of processes from system release to evaluation. The system release includes CONCERN CDS implementation and user training. Then, a mixed methods approach will be used with end users to assess the system and clinician perspectives. RESULTS: Data collection and analysis are expected to conclude by August 2022. Based on our previous work on CONCERN, we expect the system to have a positive impact on the mortality rate and length of stay. CONCLUSIONS: The CONCERN CDS will increase team-based situational awareness and shared understanding of patients predicted to be at risk for clinical deterioration in need of intervention to prevent mortality and associated harm. TRIAL REGISTRATION: ClinicalTrials.gov NCT03911687; https://clinicaltrials.gov/ct2/show/NCT03911687. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/30238.

Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events.

OBJECTIVE: To propose an algorithm that utilizes only timestamps of longitudinal electronic health record data to classify clinical deterioration events. MATERIALS AND METHODS: This retrospective study explores the efficacy of machine learning algorithms in classifying clinical deterioration events among patients in intensive care units using sequences of timestamps of vital sign measurements, flowsheets comments, order entries, and nursing notes. We design a data pipeline to partition events into discrete, regular time bins that we refer to as timesteps. Logistic regressions, random forest classifiers, and recurrent neural networks are trained on datasets of different length of timesteps, respectively, against a composite outcome of death, cardiac arrest, and Rapid Response Team calls. Then these models are validated on a holdout dataset. RESULTS: A total of 6720 intensive care unit encounters meet the criteria and the final dataset includes 830 578 timestamps. The gated recurrent unit model utilizes timestamps of vital signs, order entries, flowsheet comments, and nursing notes to achieve the best performance on the time-to-outcome dataset, with an area under the precision-recall curve of 0.101 (0.06, 0.137), a sensitivity of 0.443, and a positive predictive value of 0. 092 at the threshold of 0.6. DISCUSSION AND CONCLUSION: This study demonstrates that our recurrent neural network models using only timestamps of longitudinal electronic health record data that reflect healthcare processes achieve well-performing discriminative power.

Identifying nursing documentation patterns associated with patient deterioration and recovery from deterioration in critical and acute care settings.

OBJECTIVES: Nursing documentation behavior within electronic health records may reflect a nurse's concern about a patient and can be used to predict patient deterioration. Our study objectives were to quantify variations in nursing documentation patterns, confirm those patterns and variations with clinicians, and identify which patterns indicate patient deterioration and recovery from clinical deterioration events in the critical and acute care settings. METHODS: We collected patient data from electronic health records and conducted a regression analysis to identify different nursing documentation patterns associated with patient outcomes resulting from clinical deterioration events in the intensive care unit (ICU) and acute care unit (ACU). The primary outcome measures were whether patients were discharged alive from the hospital or expired during their hospital encounter. Secondary outcome measures were clinical deterioration events. RESULTS: In the ICU, the increased documentation of heart rate, body temperature, and withheld medication administrations were significantly associated with inpatient mortality. In the ACU, the documentation of blood pressure, respiratory rate with comments, singular vital signs, and withheld medications were significantly related to inpatient mortality. In contrast, the documentation of heart rate and "as needed" medication administrations were significantly associated with patient survival to discharge in the ACU. CONCLUSION: We successfully identified and confirmed the clinical relevancy of the nursing documentation patterns indicative of patient deterioration and recovery from clinical deterioration events in both the ICU and ACU.

Nursing Documentation Variation Across Different Medical Facilities Within an Integrated Healthcare System.

The purpose of this study was to demonstrate nursing documentation variation based on electronic health record design and its relationship with different levels of care by reviewing how various flowsheet measures, within the same electronic health record across an integrated healthcare system, are documented in different types of medical facilities. Flowsheet data with information on patients who were admitted to academic medical centers, community hospitals, and rehabilitation centers were used to calculate the frequency of flowsheet entries documented. We then compared the distinct flowsheet measures documented in five flowsheet templates across the different facilities. We observed that each type of healthcare facility appeared to have distinct clinical care foci and flowsheet measures documented differed within the same template based on facility type. Designing flowsheets tailored to study settings can meet the needs of end users and increase documentation efficiency by reducing time spent on unrelated flowsheet measures. Furthermore, this process can save nurses time for direct patient care.

Embedding, aligning and reconstructing clinical notes to explore sepsis.

OBJECTIVE: Our goal was to research and develop exploratory analysis tools for clinical notes, which now are underrepresented to limit the diversity of data insights on medically relevant applications. RESULTS: We characterize how exploratory analysis can affect representation learning on clinical narratives and present several self-developed tools to explore sepsis. Our experiments focus on patients with sepsis in the MIMIC-III Clinical Database or in our institution's research patient data repository. We found that global embeddings assist in learning local representations of clinical notes. Second, aligning at any specific time facilitates the use of learning models by pooling more available clinical notes to form a training set. Furthermore, reconstruction of the timeline enhances downstream-processing techniques by emphasizing temporal expressions and temporal relationships in clinical documentation. We demonstrate that clustering helps plot various types of clinical notes against a scale, which conveys a sense of the range or spread of the data and is useful for understanding data correlations. Appropriate exploratory analysis tools provide keen insights into preprocessing clinical notes, thereby further enhancing downstream analysis capabilities, making data driven medicine possible. Our examples can help generate better data representation of clinical documentation for models with improved performance and interpretability.

Healthcare Process Modeling to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals): Development and evaluation of a conceptual framework.

OBJECTIVE: There are signals of clinicians' expert and knowledge-driven behaviors within clinical information systems (CIS) that can be exploited to support clinical prediction. Describe development of the Healthcare Process Modeling Framework to Phenotype Clinician Behaviors for Exploiting the Signal Gain of Clinical Expertise (HPM-ExpertSignals). MATERIALS AND METHODS: We employed an iterative framework development approach that combined data-driven modeling and simulation testing to define and refine a process for phenotyping clinician behaviors. Our framework was developed and evaluated based on the Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) predictive model to detect and leverage signals of clinician expertise for prediction of patient trajectories. RESULTS: Seven themes-identified during development and simulation testing of the CONCERN model-informed framework development. The HPM-ExpertSignals conceptual framework includes a 3-step modeling technique: (1) identify patterns of clinical behaviors from user interaction with CIS; (2) interpret patterns as proxies of an individual's decisions, knowledge, and expertise; and (3) use patterns in predictive models for associations with outcomes. The CONCERN model differentiated at risk patients earlier than other early warning scores, lending confidence to the HPM-ExpertSignals framework. DISCUSSION: The HPM-ExpertSignals framework moves beyond transactional data analytics to model clinical knowledge, decision making, and CIS interactions, which can support predictive modeling with a focus on the rapid and frequent patient surveillance cycle. CONCLUSIONS: We propose this framework as an approach to embed clinicians' knowledge-driven behaviors in predictions and inferences to facilitate capture of healthcare processes that are activated independently, and sometimes well before, physiological changes are apparent.

Predicting pressure injury using nursing assessment phenotypes and machine learning methods.

OBJECTIVE: Pressure injuries are common and serious complications for hospitalized patients. The pressure injury rate is an important patient safety metric and an indicator of the quality of nursing care. Timely and accurate prediction of pressure injury risk can significantly facilitate early prevention and treatment and avoid adverse outcomes. While many pressure injury risk assessment tools exist, most were developed before there was access to large clinical datasets and advanced statistical methods, limiting their accuracy. In this paper, we describe the development of machine learning-based predictive models, using phenotypes derived from nurse-entered direct patient assessment data. METHODS: We utilized rich electronic health record data, including full assessment records entered by nurses, from 5 different hospitals affiliated with a large integrated healthcare organization to develop machine learning-based prediction models for pressure injury. Five-fold cross-validation was conducted to evaluate model performance. RESULTS: Two pressure injury phenotypes were defined for model development: nonhospital acquired pressure injury (N = 4398) and hospital acquired pressure injury (N = 1767), representing 2 distinct clinical scenarios. A total of 28 clinical features were extracted and multiple machine learning predictive models were developed for both pressure injury phenotypes. The random forest model performed best and achieved an AUC of 0.92 and 0.94 in 2 test sets, respectively. The Glasgow coma scale, a nurse-entered level of consciousness measurement, was the most important feature for both groups. CONCLUSIONS: This model accurately predicts pressure injury development and, if validated externally, may be helpful in widespread pressure injury prevention.

Estimating Time to Progression of Chronic Obstructive Pulmonary Disease With Tolerance.

We defined tolerance range as the distance of observing similar disease conditions or functional status from the upper to the lower boundaries of a specified time interval. A tolerance range was identified for linear regression and support vector machines to optimize the improvement rate (defined as IR) on accuracy in predicting mortality risk in patients with chronic obstructive pulmonary disease using clinical notes. The corpus includes pulmonary, cardiology, and radiology reports of 15,500 patients who died between 2011 and 2017. Their performance was compared against a long short-term memory recurrent neural network. The results demonstrate an overall improvement by those basic machine learning approaches after considering an optimal tolerance range: the average IR of linear regression was 90.1% and the maximum IR of support vector machines was 66.2%. There was a similitude between the time segments produced by our tolerance algorithms and those produced by the long short-term memory.

Developing hierarchical standardized home care nursing statements using nursing standard terminologies.

OBJECTIVES: Home care nursing requires high quality documentation to facilitate communication between health providers. Despite the awareness surrounding the importance of documentation, the home care nursing environment lends itself to incomplete and inaccurate documentation. Our study aims to develop a hierarchical standardized home care nursing statement (S-HCNS) structure that can increase documentation quality and completeness. METHODS: We collected a year of home care nursing notes from a hospital-based home care nursing agency in South Korea. Two nursing terminology standards, the International Classification for Nursing Practice (ICNP) and the Clinical Care Classification (CCC), were used to develop the hierarchical S-HCNS structure. RESULTS: A total of 1,230 S-HCNSs were derived by mapping 61,061 home care nursing notes to the ICNP. Among the total statements, 82.8 % were completely mapped and 0.3 % were not mapped. A total of 99.2 % of the S-HCNSs were classified within the CCC system to build a hierarchical S-HCNS structure. CONCLUSIONS: The ICNP and CCC showed high mapping rates when building the hierarchical S-HCNSs. The unmapped concepts did not exist in the CCC or ICNP but existed in other terminology systems such as SNOMED CT. The differences in granularity between the two terminology standards enabled the development of a hierarchical structure, which enabled the representation of the appropriate level of detail found within clinical documentation. We expect this structure will increase documentation quality and completeness.

Development and validation of early warning score system: A systematic literature review.

OBJECTIVES: This review aims to: 1) evaluate the quality of model reporting, 2) provide an overview of methodology for developing and validating Early Warning Score Systems (EWSs) for adult patients in acute care settings, and 3) highlight the strengths and limitations of the methodologies, as well as identify future directions for EWS derivation and validation studies. METHODOLOGY: A systematic search was conducted in PubMed, Cochrane Library, and CINAHL. Only peer reviewed articles and clinical guidelines regarding developing and validating EWSs for adult patients in acute care settings were included. 615 articles were extracted and reviewed by five of the authors. Selected studies were evaluated based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist. The studies were analyzed according to their study design, predictor selection, outcome measurement, methodology of modeling, and validation strategy. RESULTS: A total of 29 articles were included in the final analysis. Twenty-six articles reported on the development and validation of a new EWS, while three reported on validation and model modification. Only eight studies met more than 75% of the items in the TRIPOD checklist. Three major techniques were utilized among the studies to inform their predictive algorithms: 1) clinical-consensus models (n = 6), 2) regression models (n = 15), and 3) tree models (n = 5). The number of predictors included in the EWSs varied from 3 to 72 with a median of seven. Twenty-eight models included vital signs, while 11 included lab data. Pulse oximetry, mental status, and other variables extracted from electronic health records (EHRs) were among other frequently used predictors. In-hospital mortality, unplanned transfer to the intensive care unit (ICU), and cardiac arrest were commonly used clinical outcomes. Twenty-eight studies conducted a form of model validation either within the study or against other widely-used EWSs. Only three studies validated their model using an external database separate from the derived database. CONCLUSION: This literature review demonstrates that the characteristics of the cohort, predictors, and outcome selection, as well as the metrics for model validation, vary greatly across EWS studies. There is no consensus on the optimal strategy for developing such algorithms since data-driven models with acceptable predictive accuracy are often site-specific. A standardized checklist for clinical prediction model reporting exists, but few studies have included reporting aligned with it in their publications. Data-driven models are subjected to biases in the use of EHR data, thus it is particularly important to provide detailed study protocols and acknowledge, leverage, or reduce potential biases of the data used for EWS development to improve transparency and generalizability.

Mining clinical phrases from nursing notes to discover risk factors of patient deterioration.

OBJECTIVE: Early identification and treatment of patient deterioration is crucial to improving clinical outcomes. To act, hospital rapid response (RR) teams often rely on nurses' clinical judgement typically documented narratively in the electronic health record (EHR). We developed a data-driven, unsupervised method to discover potential risk factors of RR events from nursing notes. METHODS: We applied multiple natural language processing methods, including language modelling, word embeddings, and two phrase mining methods (TextRank and NC-Value), to identify quality phrases that represent clinical entities from unannotated nursing notes. TextRank was used to determine the important word-sequences in each note. NC-Value was then used to globally rank the locally-important sequences across the whole corpus. We evaluated our method both on its accuracy compared to human judgement and on the ability of the mined phrases to predict a clinical outcome, RR event hazard. RESULTS: When applied to 61,740 hospital encounters with 1,067 RR events and 778,955 notes, our method achieved an average precision of 0.590 to 0.764 (when excluding numeric tokens). Time-dependent covariates Cox model using the phrases achieved a concordance index of 0.739. Clustering the phrases revealed clinical concepts significantly associated with RR event hazard. DISCUSSION: Our findings demonstrate that our minimal-annotation, unsurprised method can rapidly mine quality phrases from a large amount of nursing notes, and these identified phrases are useful for downstream tasks, such as clinical outcome predication and risk factor identification.

Identifying nurses' concern concepts about patient deterioration using a standard nursing terminology.

OBJECTIVES: Nurse concerns documented in nursing notes are important predictors of patient risk of deterioration. Using a standard nursing terminology and inputs from subject-matter experts (SMEs), we aimed to identify and define nurse concern concepts and terms about patient deterioration, which can be used to support subsequent automated tasks, such as natural language processing and risk predication. METHODS: Group consensus meetings with nurse SMEs were held to identify nursing concerns by grading Clinical Care Classification (CCC) system concepts based on clinical knowledge. Next, a fundamental lexicon was built placing selected CCC concepts into a framework of entities and seed terms to extend CCC granularity. RESULTS: A total of 29 CCC concepts were selected as reflecting nurse concerns. From these, 111 entities and 586 seed terms were generated into a fundamental lexicon. Nursing concern concepts differed across settings (intensive care units versus non-intensive care units) and unit types (medicine versus surgery units). CONCLUSIONS: The CCC concepts were useful for representing nursing concern as they encompass a nursing-centric conceptual framework and are practical in lexicon construction. It enabled the codification of nursing concerns for deteriorating patients at a standardized conceptual level. The boundary of selected CCC concepts and lexicons were determined by the SMEs. The fundamental lexicon offers more granular terms that can be identified and processed in an automated fashion.

A temporal visualization of chronic obstructive pulmonary disease progression using deep learning and unstructured clinical notes.

BACKGROUND: Chronic obstructive pulmonary disease (COPD) is a progressive lung disease that is classified into stages based on disease severity. We aimed to characterize the time to progression prior to death in patients with COPD and to generate a temporal visualization that describes signs and symptoms during different stages of COPD progression. METHODS: We present a two-step approach for visualizing COPD progression at the level of unstructured clinical notes. We included 15,500 COPD patients who both received care within Partners Healthcare's network and died between 2011 and 2017. We first propose a four-layer deep learning model that utilizes a specially configured recurrent neural network to capture irregular time lapse segments. Using those irregular time lapse segments, we created a temporal visualization (the COPD atlas) to demonstrate COPD progression, which consisted of representative sentences at each time window prior to death based on a fraction of theme words produced by a latent Dirichlet allocation model. We evaluated our approach on an annotated corpus of COPD patients' unstructured pulmonary, radiology, and cardiology notes. RESULTS: Experiments compared to the baselines showed that our proposed approach improved interpretability as well as the accuracy of estimating COPD progression. CONCLUSIONS: Our experiments demonstrated that the proposed deep-learning approach to handling temporal variation in COPD progression is feasible and can be used to generate a graphical representation of disease progression using information extracted from clinical notes.

Unsupervised Machine Learning of Topics Documented by Nurses about Hospitalized Patients Prior to a Rapid-Response Event.

BACKGROUND: In the hospital setting, it is crucial to identify patients at risk for deterioration before it fully develops, so providers can respond rapidly to reverse the deterioration. Rapid response (RR) activation criteria include a subjective component ("worried about the patient") that is often documented in nurses' notes and is hard to capture and quantify, hindering active screening for deteriorating patients. OBJECTIVES: We used unsupervised machine learning to automatically discover RR event risk/protective factors from unstructured nursing notes. METHODS: In this retrospective cohort study, we obtained nursing notes of hospitalized, nonintensive care unit patients, documented from 2015 through 2018 from Partners HealthCare databases. We applied topic modeling to those notes to reveal topics (clusters of associated words) documented by nurses. Two nursing experts named each topic with a representative Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT) concept. We used the concepts along with vital signs and demographics in a time-dependent covariates extended Cox model to identify risk/protective factors for RR event risk. RESULTS: From a total of 776,849 notes of 45,299 patients, we generated 95 stable topics, of which 80 were mapped to 72 distinct SNOMED CT concepts. Compared with a model containing only demographics and vital signs, the latent topics improved the model's predictive ability from a concordance index of 0.657 to 0.720. Thirty topics were found significantly associated with RR event risk at a 0.05 level, and 11 remained significant after Bonferroni correction of the significance level to 6.94E-04, including physical examination (hazard ratio [HR] = 1.07, 95% confidence interval [CI], 1.03-1.12), informing doctor (HR = 1.05, 95% CI, 1.03-1.08), and seizure precautions (HR = 1.08, 95% CI, 1.04-1.12). CONCLUSION: Unsupervised machine learning methods can automatically reveal interpretable and informative signals from free-text and may support early identification of patients at risk for RR events.

Leveraging Clinical Expertise as a Feature - not an Outcome - of Predictive Models: Evaluation of an Early Warning System Use Case.

Identifying patients at risk of deterioration in the hospital and intervening more quickly to prevent adverse events is a top patient safety priority. Early warning scores (EWS) identify at risk patients, but there is much opportunity for improvement particularly related to increasing lead time - the time from an alert trigger to adverse event (e.g., cardiac arrest, death). Our team develops healthcare process models of clinical concern (HPM-CC) and in this work has identified documentation signals that are proxies of nurses concern and can be used to predict patient risk earlier than current EWS systems that rely only on physiological data. We compared the performance of a validated EWS - the MEWS - to our novel model (MEWS-CC) comprised of MEWS criteria plus 3 proxy variables of nursing concern. MEWS-CC performed similarly to MEWS, with the added benefit of increased the time from EWS trigger to event by 5-26 hours.