Automating detection of diagnostic error of infectious diseases using machine learning.

Diagnostic error, a cause of substantial morbidity and mortality, is largely discovered and evaluated through self-report and manual review, which is costly and not suitable to real-time intervention. Opportunities exist to leverage electronic health record data for automated detection of potential misdiagnosis, executed at scale and generalized across diseases. We propose a novel automated approach to identifying diagnostic divergence considering both diagnosis and risk of mortality. Our objective was to identify cases of emergency department infectious disease misdiagnoses by measuring the deviation between predicted diagnosis and documented diagnosis, weighted by mortality. Two machine learning models were trained for prediction of infectious disease and mortality using the first 24h of data. Charts were manually reviewed by clinicians to determine whether there could have been a more correct or timely diagnosis. The proposed approach was validated against manual reviews and compared using the Spearman rank correlation. We analyzed 6.5 million ED visits and over 700 million associated clinical features from over one hundred emergency departments. The testing set performances of the infectious disease (Macro F1 = 86.7, AUROC 90.6 to 94.7) and mortality model (Macro F1 = 97.6, AUROC 89.1 to 89.1) were in expected ranges. Human reviews and the proposed automated metric demonstrated positive correlations ranging from 0.231 to 0.358. The proposed approach for diagnostic deviation shows promise as a potential tool for clinicians to find diagnostic errors. Given the vast number of clinical features used in this analysis, further improvements likely need to either take greater account of data structure (what occurs before when) or involve natural language processing. Further work is needed to explain the potential reasons for divergence and to refine and validate the approach for implementation in real-world settings.

Pneumonia diagnosis performance in the emergency department: a mixed-methods study about clinicians' experiences and exploration of individual differences and response to diagnostic performance feedback.

OBJECTIVES: We sought to (1) characterize the process of diagnosing pneumonia in an emergency department (ED) and (2) examine clinician reactions to a clinician-facing diagnostic discordance feedback tool. MATERIALS AND METHODS: We designed a diagnostic feedback tool, using electronic health record data from ED clinicians' patients to establish concordance or discordance between ED diagnosis, radiology reports, and hospital discharge diagnosis for pneumonia. We conducted semistructured interviews with 11 ED clinicians about pneumonia diagnosis and reactions to the feedback tool. We administered surveys measuring individual differences in mindset beliefs, comfort with feedback, and feedback tool usability. We qualitatively analyzed interview transcripts and descriptively analyzed survey data. RESULTS: Thematic results revealed: (1) the diagnostic process for pneumonia in the ED is characterized by diagnostic uncertainty and may be secondary to goals to treat and dispose the patient; (2) clinician diagnostic self-evaluation is a fragmented, inconsistent process of case review and follow-up that a feedback tool could fill; (3) the feedback tool was described favorably, with task and normative feedback harnessing clinician values of high-quality patient care and personal excellence; and (4) strong reactions to diagnostic feedback varied from implicit trust to profound skepticism about the validity of the concordance metric. Survey results suggested a relationship between clinicians' individual differences in learning and failure beliefs, feedback experience, and usability ratings. DISCUSSION AND CONCLUSION: Clinicians value feedback on pneumonia diagnoses. Our results highlight the importance of feedback about diagnostic performance and suggest directions for considering individual differences in feedback tool design and implementation.

Temporary Financial Assistance Reduced The Probability Of Unstable Housing Among Veterans For More Than 1 Year.

The Department of Veterans Affairs (VA) aims to reduce homelessness among veterans through programs such as Supportive Services for Veteran Families (SSVF). An important component of SSVF is temporary financial assistance. Previous research has demonstrated the effectiveness of temporary financial assistance in reducing short-term housing instability, but studies have not examined its long-term effect on housing outcomes. Using data from the VA's electronic health record system, we analyzed the effect of temporary financial assistance on veterans' housing instability for three years after entry into SSVF. We extracted housing outcomes from clinical notes, using natural language processing, and compared the probability of unstable housing among veterans who did and did not receive temporary financial assistance. We found that temporary financial assistance rapidly reduced the probability of unstable housing, but the effect attenuated after forty-five days. Our findings suggest that to maintain long-term housing stability for veterans who have exited SSVF, additional interventions may be needed.

A Linguistic Analysis Examining the Impact of COVID-19 on Pneumonia Diagnosis and Disease Models.

Written clinical language embodies and reflects the clinician's mental models of disease. Prior to the COVID-19 pandemic, pneumonia was shifting away from concern for healthcare-associated pneumonia and toward recognition of heterogeneity of pathogens and host response. How these models are reflected in clinical language or whether they were impacted by the pandemic has not been studied. We aimed to assess changes in the language used to describe pneumonia following the COVID-19 pandemic.

Assessing longitudinal housing status using Electronic Health Record data: a comparison of natural language processing, structured data, and patient-reported history.

Introduction: Measuring long-term housing outcomes is important for evaluating the impacts of services for individuals with homeless experience. However, assessing long-term housing status using traditional methods is challenging. The Veterans Affairs (VA) Electronic Health Record (EHR) provides detailed data for a large population of patients with homeless experiences and contains several indicators of housing instability, including structured data elements (e.g., diagnosis codes) and free-text clinical narratives. However, the validity of each of these data elements for measuring housing stability over time is not well-studied. Methods: We compared VA EHR indicators of housing instability, including information extracted from clinical notes using natural language processing (NLP), with patient-reported housing outcomes in a cohort of homeless-experienced Veterans. Results: NLP achieved higher sensitivity and specificity than standard diagnosis codes for detecting episodes of unstable housing. Other structured data elements in the VA EHR showed promising performance, particularly when combined with NLP. Discussion: Evaluation efforts and research studies assessing longitudinal housing outcomes should incorporate multiple data sources of documentation to achieve optimal performance.

Hospital admission decisions for older Veterans with community-onset pneumonia: An analysis of 118 U.S. Veterans Affairs Medical Centers.

OBJECTIVES: Age is important for prognosis in community-onset pneumonia, but how it influences admission decisions in the emergency department (ED) is not well characterized. Using clinical data from the electronic health record in a national cohort, we examined pneumonia hospitalization patterns, variation, and relationships with mortality among older versus younger Veterans. METHODS: In a retrospective cohort of patients ≥ 18 years presenting to EDs with a diagnosis of pneumonia at 118 VA Medical Centers January 1, 2006, to December 31, 2016, we compared observed, predicted, and residual hospitalization risk for Veterans < 70, 70-79, and ≥ 80 years of age using generalized estimating equations and machine learning models with 71 patient factors. We examined facility variation in residual hospitalization across facilities and explored whether facility differences in hospitalization risk correlated with differences in 30-day mortality. RESULTS: Among 297,498 encounters, 165,003 (55%) were for Veterans < 70 years, 61,076 (21%) 70-80, and 71,419 (24%) ≥ 80. Hospitalization rates were 52%, 67%, and 76%, respectively. After other patient factors were adjusting for, age 70-79 had an odds ratio (OR) of 1.39 (95% confidence interval [CI] 1.34-1.44) and ≥ 80 had an OR of 2.1 (95% CI 2.0-2.2) compared to age < 70. There was substantial variation in hospitalization across facilities among Veterans < 70 (<35% hospitalization at the lowest decile of facilities vs. > 66% at the highest decile) that was similar but with higher risk for patients 70-79 years (54% vs. 82%) and ≥ 80 years (59% vs. 85%) and remained after accounting for patient factors, with no consistently positive or negative associations with facility-level 30-day mortality. CONCLUSIONS: Older Veterans with community-onset pneumonia experience high risk of hospitalization, with widespread facility variation that has no clear relationship to short-term mortality.

Predictors of homeless service utilization and stable housing status among Veterans receiving services from a nationwide homelessness prevention and rapid rehousing program.

Homelessness prevention and rapid rehousing (RRH) programs are increasingly important components of the homeless assistance system in the United States. Yet, there are key gaps in knowledge about the dynamics of the utilization of these programs, with scant attention paid to examining the duration of homelessness prevention and RRH service episodes or to patterns of repeated use of these programs over time. To address these gaps, we use data from the U.S. Department of Veterans Affairs' (VA) Supportive Services for Veteran Families (SSVF) program-the largest program in the country providing homelessness prevention and RRH services-to assess the relationship between individual and program-level factors and exits to stable housing, length of service episodes, and patterns of repeated service use over time. We analyze data for a primary cohort of 570,798 of Veterans who received SSVF services during Fiscal Years (FY) 2012-2021, and for separate cohorts of Veterans who received SSVF prevention and RRH services, respectively, during FY 2016-2021. We find that participants' income, indicators of their health status, their use of other VA homeless programs, and rurality are consistent predictors of our outcomes. These findings have implications for how to allocate homelessness prevention and RRH resources in the most efficient manner to help households maintain or obtain stable housing.

Using natural language processing to study homelessness longitudinally with electronic health record data subject to irregular observations.

The Electronic Health Record (EHR) contains information about social determinants of health (SDoH) such as homelessness. Much of this information is contained in clinical notes and can be extracted using natural language processing (NLP). This data can provide valuable information for researchers and policymakers studying long-term housing outcomes for individuals with a history of homelessness. However, studying homelessness longitudinally in the EHR is challenging due to irregular observation times. In this work, we applied an NLP system to extract housing status for a cohort of patients in the US Department of Veterans Affairs (VA) over a three-year period. We then applied inverse intensity weighting to adjust for the irregularity of observations, which was used generalized estimating equations to estimate the probability of unstable housing each day after entering a VA housing assistance program. Our methods generate unique insights into the long-term outcomes of individuals with a history of homelessness and demonstrate the potential for using EHR data for research and policymaking.

Correlated gene modules uncovered by high-precision single-cell transcriptomics.

Correlations in gene expression are used to infer functional and regulatory relationships between genes. However, correlations are often calculated across different cell types or perturbations, causing genes with unrelated functions to be correlated. Here, we demonstrate that correlated modules can be better captured by measuring correlations of steady-state gene expression fluctuations in single cells. We report a high-precision single-cell RNA-seq method called MALBAC-DT to measure the correlation between any pair of genes in a homogenous cell population. Using this method, we were able to identify numerous cell-type specific and functionally enriched correlated gene modules. We confirmed through knockdown that a module enriched for p53 signaling predicted p53 regulatory targets more accurately than a consensus of ChIP-seq studies and that steady-state correlations were predictive of transcriptome-wide response patterns to perturbations. This approach provides a powerful way to advance our functional understanding of the genome.

A flexible framework for visualizing and exploring patient misdiagnosis over time.

Diagnosis is a complex and ambiguous process and yet, it is the critical hinge point for all subsequent clinical reasoning and decision-making. Tracking the quality of the patient diagnostic process has the potential to provide valuable insights in improving the diagnostic accuracy and to reduce downstream errors but needs to be informative, timely, and efficient at scale. However, due to the rate at which healthcare data are captured on a daily basis, manually reviewing the diagnostic history of each patient would be a severely taxing process without efficient data reduction and representation. Application of data visualization and visual analytics to healthcare data is one promising approach for addressing these challenges. This paper presents a novel flexible visualization and analysis framework for exploring the patient diagnostic process over time (i.e., patient diagnosis paths). Our framework allows users to select a specific set of patients, events and/or conditions, filter data based on different attributes, and view further details on the selected patient cohort while providing an interactive view of the resulting patient diagnosis paths. A practical demonstration of our system is presented with a case study exploring infection-based patient diagnosis paths.

Analysis of a national response to a White House directive for ending veteran suicide.

OBJECTIVE: Analyze responses to a national request for information (RFI) to uncover gaps in policy, practice, and understanding of veteran suicide to inform federal research strategy. DATA SOURCE: An RFI with 21 open-ended questions generated from Presidential Executive Order #1386, administered nationally from July 3 to August 5, 2019. STUDY DESIGN: Semi-structured, open-ended responses analyzed using a collaborative qualitative and text-mining data process. DATA EXTRACTION METHODS: We aligned traditional qualitative methods with natural language processing (NLP) text-mining techniques to analyze 9040 open-ended question responses from 722 respondents to provide results within 3 months. Narrative inquiry and the medical explanatory model guided the data extraction and analytic process. RESULTS: Five major themes were identified: risk factors, risk assessment, prevention and intervention, barriers to care, and data/research. Individuals and organizations mentioned different concepts within the same themes. In responses about risk factors, individuals frequently mentioned generic terms like "illness" while organizations mentioned specific terms like "traumatic brain injury." Organizations and individuals described unique barriers to care and emphasized ways to integrate data and research to improve points of care. Organizations often identified lack of funding as barriers while individuals often identified key moments for prevention such as military transitions and ensuring care providers have military cultural understanding. CONCLUSIONS: This study provides an example of a rapid, adaptive analysis of a large body of qualitative, public response data about veteran suicide to support a federal strategy for an important public health topic. Combining qualitative and text-mining methods allowed a representation of voices and perspectives including the lived experiences of individuals who described stories of military transition, treatments that worked or did not, and the perspective of organizations treating veterans for suicide. The results supported the development of a national strategy to reduce suicide risks for veterans as well as civilians.

Development and evaluation of an interoperable natural language processing system for identifying pneumonia across clinical settings of care and institutions.

Objective: To evaluate the feasibility, accuracy, and interoperability of a natural language processing (NLP) system that extracts diagnostic assertions of pneumonia in different clinical notes and institutions. Materials and Methods: A rule-based NLP system was designed to identify assertions of pneumonia in 3 types of clinical notes from electronic health records (EHRs): emergency department notes, radiology reports, and discharge summaries. The lexicon and classification logic were tailored for each note type. The system was first developed and evaluated using annotated notes from the Department of Veterans Affairs (VA). Interoperability was assessed using data from the University of Utah (UU). Results: The NLP system was comprised of 782 rules and achieved moderate-to-high performance in all 3 note types in VA (precision/recall/f1: emergency = 88.1/86.0/87.1; radiology = 71.4/96.2/82.0; discharge = 88.3/93.0/90.1). When applied to UU data, performance was maintained in emergency and radiology but decreased in discharge summaries (emergency = 84.7/94.3/89.3; radiology = 79.7/100.0/87.9; discharge = 65.5/92.7/76.8). Customization with 34 additional rules increased performance for all note types (emergency = 89.3/94.3/91.7; radiology = 87.0/100.0/93.1; discharge = 75.0/95.1/83.4). Conclusion: NLP can be used to accurately identify the diagnosis of pneumonia across different clinical settings and institutions. A limited amount of customization to account for differences in lexicon, clinical definition of pneumonia, and EHR structure can achieve high accuracy without substantial modification.

ReHouSED: A novel measurement of Veteran housing stability using natural language processing.

Housing stability is an important determinant of health. The US Department of Veterans Affairs (VA) administers several programs to assist Veterans experiencing unstable housing. Measuring long-term housing stability of Veterans who receive assistance from VA is difficult due to a lack of standardized structured documentation in the Electronic Health Record (EHR). However, the text of clinical notes often contains detailed information about Veterans' housing situations that may be extracted using natural language processing (NLP). We present a novel NLP-based measurement of Veteran housing stability: Relative Housing Stability in Electronic Documentation (ReHouSED). We first develop and evaluate a system for classifying documents containing information about Veterans' housing situations. Next, we aggregate information from multiple documents to derive a patient-level measurement of housing stability. Finally, we demonstrate this method's ability to differentiate between Veterans who are stably and unstably housed. Thus, ReHouSED provides an important methodological framework for the study of long-term housing stability among Veterans receiving housing assistance.

Automated Travel History Extraction From Clinical Notes for Informing the Detection of Emergent Infectious Disease Events: Algorithm Development and Validation.

BACKGROUND: Patient travel history can be crucial in evaluating evolving infectious disease events. Such information can be challenging to acquire in electronic health records, as it is often available only in unstructured text. OBJECTIVE: This study aims to assess the feasibility of annotating and automatically extracting travel history mentions from unstructured clinical documents in the Department of Veterans Affairs across disparate health care facilities and among millions of patients. Information about travel exposure augments existing surveillance applications for increased preparedness in responding quickly to public health threats. METHODS: Clinical documents related to arboviral disease were annotated following selection using a semiautomated bootstrapping process. Using annotated instances as training data, models were developed to extract from unstructured clinical text any mention of affirmed travel locations outside of the continental United States. Automated text processing models were evaluated, involving machine learning and neural language models for extraction accuracy. RESULTS: Among 4584 annotated instances, 2659 (58%) contained an affirmed mention of travel history, while 347 (7.6%) were negated. Interannotator agreement resulted in a document-level Cohen kappa of 0.776. Automated text processing accuracy (F1 85.6, 95% CI 82.5-87.9) and computational burden were acceptable such that the system can provide a rapid screen for public health events. CONCLUSIONS: Automated extraction of patient travel history from clinical documents is feasible for enhanced passive surveillance public health systems. Without such a system, it would usually be necessary to manually review charts to identify recent travel or lack of travel, use an electronic health record that enforces travel history documentation, or ignore this potential source of information altogether. The development of this tool was initially motivated by emergent arboviral diseases. More recently, this system was used in the early phases of response to COVID-19 in the United States, although its utility was limited to a relatively brief window due to the rapid domestic spread of the virus. Such systems may aid future efforts to prevent and contain the spread of infectious diseases.

Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python.

Despite impressive success of machine learning algorithms in clinical natural language processing (cNLP), rule-based approaches still have a prominent role. In this paper, we introduce medspaCy, an extensible, open-source cNLP library based on spaCy framework that allows flexible integration of rule-based and machine learning-based algorithms adapted to clinical text. MedspaCy includes a variety of components that meet common cNLP needs such as context analysis and mapping to standard terminologies. By utilizing spaCy's clear and easy-to-use conventions, medspaCy enables development of custom pipelines that integrate easily with other spaCy-based modules. Our toolkit includes several core components and facilitates rapid development of pipelines for clinical text.

Detecting Adverse Drug Events with Rapidly Trained Classification Models.

INTRODUCTION: Identifying occurrences of medication side effects and adverse drug events (ADEs) is an important and challenging task because they are frequently only mentioned in clinical narrative and are not formally reported. METHODS: We developed a natural language processing (NLP) system that aims to identify mentions of symptoms and drugs in clinical notes and label the relationship between the mentions as indications or ADEs. The system leverages an existing word embeddings model with induced word clusters for dimensionality reduction. It employs a conditional random field (CRF) model for named entity recognition (NER) and a random forest model for relation extraction (RE). RESULTS: Final performance of each model was evaluated separately and then combined on a manually annotated evaluation set. The micro-averaged F1 score was 80.9% for NER, 88.1% for RE, and 61.2% for the integrated systems. Outputs from our systems were submitted to the NLP Challenges for Detecting Medication and Adverse Drug Events from Electronic Health Records (MADE 1.0) competition (Yu et al. in http://bio-nlp.org/index.php/projects/39-nlp-challenges , 2018). System performance was evaluated in three tasks (NER, RE, and complete system) with multiple teams submitting output from their systems for each task. Our RE system placed first in Task 2 of the challenge and our integrated system achieved third place in Task 3. CONCLUSION: Adding to the growing number of publications that utilize NLP to detect occurrences of ADEs, our study illustrates the benefits of employing innovative feature engineering.

Detecting Evidence of Intra-abdominal Surgical Site Infections from Radiology Reports Using Natural Language Processing.

Free-text reports in electronic health records (EHRs) contain medically significant information - signs, symptoms, findings, diagnoses - recorded by clinicians during patient encounters. These reports contain rich clinical information which can be leveraged for surveillance of disease and occurrence of adverse events. In order to gain meaningful knowledge from these text reports to support surveillance efforts, information must first be converted into a structured, computable format. Traditional methods rely on manual review of charts, which can be costly and inefficient. Natural language processing (NLP) methods offer an efficient, alternative approach to extracting the information and can achieve a similar level of accuracy. We developed an NLP system to automatically identify mentions of surgical site infections in radiology reports and classify reports containing evidence of surgical site infections leveraging these mentions. We evaluated our system using a reference standard of reports annotated by domain experts, administrative data generated for each patient encounter, and a machine learning-based approach.

Single-Cell Whole-Genome Amplification and Sequencing: Methodology and Applications.

We present a survey of single-cell whole-genome amplification (WGA) methods, including degenerate oligonucleotide-primed polymerase chain reaction (DOP-PCR), multiple displacement amplification (MDA), and multiple annealing and looping-based amplification cycles (MALBAC). The key parameters to characterize the performance of these methods are defined, including genome coverage, uniformity, reproducibility, unmappable rates, chimera rates, allele dropout rates, false positive rates for calling single-nucleotide variations, and ability to call copy-number variations. Using these parameters, we compare five commercial WGA kits by performing deep sequencing of multiple single cells. We also discuss several major applications of single-cell genomics, including studies of whole-genome de novo mutation rates, the early evolution of cancer genomes, circulating tumor cells (CTCs), meiotic recombination of germ cells, preimplantation genetic diagnosis (PGD), and preimplantation genomic screening (PGS) for in vitro-fertilized embryos.

Single cell transcriptome amplification with MALBAC.

Recently, Multiple Annealing and Looping-Based Amplification Cycles (MALBAC) has been developed for whole genome amplification of an individual cell, relying on quasilinear instead of exponential amplification to achieve high coverage. Here we adapt MALBAC for single-cell transcriptome amplification, which gives consistently high detection efficiency, accuracy and reproducibility. With this newly developed technique, we successfully amplified and sequenced single cells from 3 germ layers from mouse embryos in the early gastrulation stage, and examined the epithelial-mesenchymal transition (EMT) program among cells in the mesoderm layer on a single-cell level.

Spatial organization of RNA polymerase II inside a mammalian cell nucleus revealed by reflected light-sheet superresolution microscopy.

Superresolution microscopy based on single-molecule centroid determination has been widely applied to cellular imaging in recent years. However, quantitative imaging of the mammalian nucleus has been challenging due to the lack of 3D optical sectioning methods for normal-sized cells, as well as the inability to accurately count the absolute copy numbers of biomolecules in highly dense structures. Here we report a reflected light-sheet superresolution microscopy method capable of imaging inside the mammalian nucleus with superior signal-to-background ratio as well as molecular counting with single-copy accuracy. Using reflected light-sheet superresolution microscopy, we probed the spatial organization of transcription by RNA polymerase II (RNAP II) molecules and quantified their global extent of clustering inside the mammalian nucleus. Spatiotemporal clustering analysis that leverages on the blinking photophysics of specific organic dyes showed that the majority (>70%) of the transcription foci originate from single RNAP II molecules, and no significant clustering between RNAP II molecules was detected within the length scale of the reported diameter of "transcription factories." Colocalization measurements of RNAP II molecules equally labeled by two spectrally distinct dyes confirmed the primarily unclustered distribution, arguing against a prevalent existence of transcription factories in the mammalian nucleus as previously proposed. The methods developed in our study pave the way for quantitative mapping and stoichiometric characterization of key biomolecular species deep inside mammalian cells.