Automating detection of diagnostic error of infectious diseases using machine learning.

Diagnostic error, a cause of substantial morbidity and mortality, is largely discovered and evaluated through self-report and manual review, which is costly and not suitable to real-time intervention. Opportunities exist to leverage electronic health record data for automated detection of potential misdiagnosis, executed at scale and generalized across diseases. We propose a novel automated approach to identifying diagnostic divergence considering both diagnosis and risk of mortality. Our objective was to identify cases of emergency department infectious disease misdiagnoses by measuring the deviation between predicted diagnosis and documented diagnosis, weighted by mortality. Two machine learning models were trained for prediction of infectious disease and mortality using the first 24h of data. Charts were manually reviewed by clinicians to determine whether there could have been a more correct or timely diagnosis. The proposed approach was validated against manual reviews and compared using the Spearman rank correlation. We analyzed 6.5 million ED visits and over 700 million associated clinical features from over one hundred emergency departments. The testing set performances of the infectious disease (Macro F1 = 86.7, AUROC 90.6 to 94.7) and mortality model (Macro F1 = 97.6, AUROC 89.1 to 89.1) were in expected ranges. Human reviews and the proposed automated metric demonstrated positive correlations ranging from 0.231 to 0.358. The proposed approach for diagnostic deviation shows promise as a potential tool for clinicians to find diagnostic errors. Given the vast number of clinical features used in this analysis, further improvements likely need to either take greater account of data structure (what occurs before when) or involve natural language processing. Further work is needed to explain the potential reasons for divergence and to refine and validate the approach for implementation in real-world settings.

Pneumonia diagnosis performance in the emergency department: a mixed-methods study about clinicians' experiences and exploration of individual differences and response to diagnostic performance feedback.

OBJECTIVES: We sought to (1) characterize the process of diagnosing pneumonia in an emergency department (ED) and (2) examine clinician reactions to a clinician-facing diagnostic discordance feedback tool. MATERIALS AND METHODS: We designed a diagnostic feedback tool, using electronic health record data from ED clinicians' patients to establish concordance or discordance between ED diagnosis, radiology reports, and hospital discharge diagnosis for pneumonia. We conducted semistructured interviews with 11 ED clinicians about pneumonia diagnosis and reactions to the feedback tool. We administered surveys measuring individual differences in mindset beliefs, comfort with feedback, and feedback tool usability. We qualitatively analyzed interview transcripts and descriptively analyzed survey data. RESULTS: Thematic results revealed: (1) the diagnostic process for pneumonia in the ED is characterized by diagnostic uncertainty and may be secondary to goals to treat and dispose the patient; (2) clinician diagnostic self-evaluation is a fragmented, inconsistent process of case review and follow-up that a feedback tool could fill; (3) the feedback tool was described favorably, with task and normative feedback harnessing clinician values of high-quality patient care and personal excellence; and (4) strong reactions to diagnostic feedback varied from implicit trust to profound skepticism about the validity of the concordance metric. Survey results suggested a relationship between clinicians' individual differences in learning and failure beliefs, feedback experience, and usability ratings. DISCUSSION AND CONCLUSION: Clinicians value feedback on pneumonia diagnoses. Our results highlight the importance of feedback about diagnostic performance and suggest directions for considering individual differences in feedback tool design and implementation.

Building the future of ID: A call to action for quality improvement research and measurement.

Quality is central to value-based care and measurement is essential for assessing performance and understanding improvement over time. Both value-based care and methods for quality measurement are evolving. Infectious Diseases has been less engaged than other specialties in quality measure development, and Infectious Diseases providers must seize the opportunity to engage with quality measure development and research. Antimicrobial stewardship programs are an ideal starting point for Infectious Diseases-related quality measure development; antimicrobial stewardship program interventions and best practices are Infectious Diseases-specific, measurable, and impactful, yet grossly undercompensated. Herein, we provide a scheme for prioritizing research focused on development of Infectious Diseases-specific quality measures. Maturation of quality measurement research in Infectious Diseases, beginning with an initial focus on stewardship-related conditions then expanding to non-stewardship topics, will allow Infectious Diseases to take control of its future in value-based care, and promote the growth of Infectious Diseases through greater recognition of its value.

Patient Safety and Artificial Intelligence in Clinical Care.

This Viewpoint offers 3 recommendations for health care organizations and other stakeholders to consider as part of the Health and Human Services' artificial intelligence safety program.

A Linguistic Analysis Examining the Impact of COVID-19 on Pneumonia Diagnosis and Disease Models.

Written clinical language embodies and reflects the clinician's mental models of disease. Prior to the COVID-19 pandemic, pneumonia was shifting away from concern for healthcare-associated pneumonia and toward recognition of heterogeneity of pathogens and host response. How these models are reflected in clinical language or whether they were impacted by the pandemic has not been studied. We aimed to assess changes in the language used to describe pneumonia following the COVID-19 pandemic.

Improving Sepsis Outcomes in the Era of Pay-for-Performance and Electronic Quality Measures: A Joint IDSA/ACEP/PIDS/SHEA/SHM/SIDP Position Paper.

The Centers for Medicare & Medicaid Services (CMS) introduced the Severe Sepsis/Septic Shock Management Bundle (SEP-1) as a pay-for-reporting measure in 2015 and is now planning to make it a pay-for-performance measure by incorporating it into the Hospital Value-Based Purchasing Program. This joint IDSA/ACEP/PIDS/SHEA/SHM/SIPD position paper highlights concerns with this change. Multiple studies indicate that SEP-1 implementation was associated with increased broad-spectrum antibiotic use, lactate measurements, and aggressive fluid resuscitation for patients with suspected sepsis but not with decreased mortality rates. Increased focus on SEP-1 risks further diverting attention and resources from more effective measures and comprehensive sepsis care. We recommend retiring SEP-1 rather than using it in a payment model and shifting instead to new sepsis metrics that focus on patient outcomes. CMS is developing a community-onset sepsis 30-day mortality electronic clinical quality measure (eCQM) that is an important step in this direction. The eCQM preliminarily identifies sepsis using systemic inflammatory response syndrome (SIRS) criteria, antibiotic administrations or diagnosis codes for infection or sepsis, and clinical indicators of acute organ dysfunction. We support the eCQM but recommend removing SIRS criteria and diagnosis codes to streamline implementation, decrease variability between hospitals, maintain vigilance for patients with sepsis but without SIRS, and avoid promoting antibiotic use in uninfected patients with SIRS. We further advocate for CMS to harmonize the eCQM with the Centers for Disease Control and Prevention's (CDC) Adult Sepsis Event surveillance metric to promote unity in federal measures, decrease reporting burden for hospitals, and facilitate shared prevention initiatives. These steps will result in a more robust measure that will encourage hospitals to pay more attention to the full breadth of sepsis care, stimulate new innovations in diagnosis and treatment, and ultimately bring us closer to our shared goal of improving outcomes for patients.

Healthcare-associated infections and conditions in the era of digital measurement.

As the third edition of the Compendium of Strategies to Prevent Healthcare-Associated Infections in Acute Care Hospitals is released with the latest recommendations for the prevention and management of healthcare-associated infections (HAIs), a new approach to reporting HAIs is just beginning to unfold. This next generation of HAI reporting will be fully electronic and based largely on existing data in electronic health record (EHR) systems and other electronic data sources. It will be a significant change in how hospitals report HAIs and how the Centers for Disease Control and Prevention (CDC) and other agencies receive this information. This paper outlines what that future electronic reporting system will look like and how it will impact HAI reporting.

How Safe are Outpatient Electronic Health Records? An Evaluation of Medication-Related Decision Support using the Ambulatory Electronic Health Record Evaluation Tool.

BACKGROUND: The purpose of the Ambulatory Electronic Health Record (EHR) Evaluation Tool is to provide outpatient clinics with an assessment that they can use to measure the ability of the EHR system to detect and prevent common prescriber errors. The tool consists of a medication safety test and a medication reconciliation module. OBJECTIVES: The goal of this study was to perform a broad evaluation of outpatient medication-related decision support using the Ambulatory EHR Evaluation Tool. METHODS: We performed a cross-sectional study with 10 outpatient clinics using the Ambulatory EHR Evaluation Tool. For the medication safety test, clinics were provided test patients and associated medication test orders to enter in their EHR, where they recorded any advice or information they received. Once finished, clinics received an overall percentage score of unsafe orders detected and individual order category scores. For the medication reconciliation module, clinics were asked to electronically reconcile two medication lists, where modifications were made by adding and removing medications and changing the dosage of select medications. RESULTS: For the medication safety test, the mean overall score was 57%, with the highest score being 70%, and the lowest score being 40%. Clinics performed well in the drug allergy (100%), drug dose daily (85%), and inappropriate medication combinations (74%) order categories. Order categories with the lowest performance were drug laboratory (10%) and drug monitoring (3%). Most clinics (90%) scored a 0% in at least one order category. For the medication reconciliation module, only one clinic (10%) could reconcile medication lists electronically; however, there was no clinical decision support available that checked for drug interactions. CONCLUSION: We evaluated a sample of ambulatory practices around their medication-related decision support and found that advanced capabilities within these systems have yet to be widely implemented. The tool was practical to use and identified substantial opportunities for improvement in outpatient medication safety.

How can artificial intelligence decrease cognitive and work burden for front line practitioners?

Artificial intelligence (AI) has tremendous potential to improve the cognitive and work burden of clinicians across a range of clinical activities, which could lead to reduced burnout and better clinical care. The recent explosion of generative AI nicely illustrates this potential. Developers and organizations deploying AI have a responsibility to ensure AI is designed and implemented with end-user input, has mechanisms to identify and potentially reduce bias, and that the impact on cognitive and work burden is measured, monitored, and improved. This article focuses specifically on the role AI can play in reducing cognitive and work burden, outlines the critical issues associated with the use of AI, and serves as a call to action for vendors and users to work together to develop functionality that addresses these challenges.

Inpatient EHR User Experience and Hospital EHR Safety Performance.

IMPORTANCE: Despite the broad adoption and optimization of electronic health record (EHR) systems across the continuum of care, serious usability and safety problems persist. OBJECTIVE: To assess whether EHR safety performance is associated with EHR frontline user experience in a national sample of hospitals. DESIGN, SETTING, AND PARTICIPANTS: This cross-sectional study included all US adult hospitals that used the National Quality Forum Leapfrog Health IT Safety Measure and also used the ARCH Collaborative EHR User experience survey from January 1, 2017, to January 1, 2019. Data analysis was performed from September 2020 to November 2022. MAIN OUTCOMES AND MEASURES: The primary outcomes were hospital performance on the Leapfrog Health IT Safety measure (overall and 10 subcomponents) and the ARCH collaborative frontline user experience scores (overall and 8 subcomponents). Ordinary least squares models with survey responses clustered by hospital were used to assess associations between the overall measures and their subcomponents. RESULTS: There were 112 hospitals and 5689 frontline user surveys included in the study. Hospitals scored a mean of 0.673 (range, 0.297-0.973) on the Leapfrog Health IT safety measure; the mean ARCH EHR user experience score was 3.377 (range, 1 [best] to 5 [worst]). The adjusted ß coefficient between the overall safety score and overall user experience score was 0.011 (95% CI, 0.006-0.016). The ARCH overall score was also significantly associated with 10 subcategory scores of the Leapfrog Health IT safety score, and the overall Leapfrog score was associated with the 8 subcategory scores of the ARCH user experience score. CONCLUSIONS AND RELEVANCE: This cross-sectional study found a positive association between frontline user-rated EHR usability and EHR safety performance. This finding suggests that improving EHR usability, which is a current well-known pain point for EHR users, could have direct benefits in terms of improved EHR safety.

Risk Factors for Severe COVID-19 Among Health Care Workers.

OBJECTIVE: Evaluate potential risk factors for severe coronavirus disease 2019 (COVID-19) among health care workers (HCWs) at the University of Virginia Medical Center in Charlottesville, Virginia. METHODS: We conducted a retrospective manual chart review of data from HCWs who were diagnosed with COVID-19 from March 2020 to March 2021. Using data from patient medical histories, we ascertained risk factors for COVID-19-related emergency department encounter, hospitalization, or death. RESULTS: We had 634 patients in total, and 9.8% had a severe COVID-19-related outcome. A history of deep vein thrombosis/pulmonary embolism/stroke (odds ratio, 19.6; 95% confidence interval, 5.11 to 94.7), as well as asthma, chronic lung disease, diabetes, or current immunocompromised status, was associated with increased adjusted odds of COVID-19-related emergency department encounter/hospitalization/death. CONCLUSIONS: A preexisting history of deep vein thrombosis/pulmonary embolism/stroke is a novel risk factor for poor COVID-19 outcomes among a cohort of HCWs.

Examining medication ordering errors using AHRQ network of patient safety databases.

BACKGROUND: Studies examining the effects of computerized order entry (CPOE) on medication ordering errors demonstrate that CPOE does not consistently prevent these errors as intended. We used the Agency for Healthcare Research and Quality (AHRQ) Network of Patient Safety Databases (NPSD) to investigate the frequency and degree of harm of reported events that occurred at the ordering stage, characterized by error type. MATERIALS AND METHODS: This was a retrospective observational study of safety events reported by healthcare systems in participating patient safety organizations from 6/2010 through 12/2020. All medication and other substance ordering errors reported to NPSD via common format v1.2 between 6/2010 through 12/2020 were analyzed. We aggregated and categorized the frequency of reported medication ordering errors by error type, degree of harm, and demographic characteristics. RESULTS: A total of 12 830 errors were reported during the study period. Incorrect dose accounted for 3812 errors (29.7%), followed by incorrect medication 2086 (16.3%), and incorrect duration 765 (6.0%). Of 5282 events that reached the patient and had a known level of severity, 12 resulted in death, 4 resulted in severe harm, 45 resulted in moderate harm, 341 resulted in mild harm, and 4880 resulted in no harm. CONCLUSION: Incorrect dose and incorrect drug orders were the most commonly reported and harmful types of medication ordering errors. Future studies should aim to develop and test interventions focused on CPOE to prevent medication ordering errors, prioritizing wrong-dose and wrong-drug errors.

Electronic Health Record Legal Settlements in the US Since the 2009 Health Information Technology for Economic and Clinical Health Act.

This cross-sectional study describes allegations of misconduct by electronic health record vendors.

A flexible framework for visualizing and exploring patient misdiagnosis over time.

Diagnosis is a complex and ambiguous process and yet, it is the critical hinge point for all subsequent clinical reasoning and decision-making. Tracking the quality of the patient diagnostic process has the potential to provide valuable insights in improving the diagnostic accuracy and to reduce downstream errors but needs to be informative, timely, and efficient at scale. However, due to the rate at which healthcare data are captured on a daily basis, manually reviewing the diagnostic history of each patient would be a severely taxing process without efficient data reduction and representation. Application of data visualization and visual analytics to healthcare data is one promising approach for addressing these challenges. This paper presents a novel flexible visualization and analysis framework for exploring the patient diagnostic process over time (i.e., patient diagnosis paths). Our framework allows users to select a specific set of patients, events and/or conditions, filter data based on different attributes, and view further details on the selected patient cohort while providing an interactive view of the resulting patient diagnosis paths. A practical demonstration of our system is presented with a case study exploring infection-based patient diagnosis paths.

Patient Perceptions of Hospital Experiences: Implications for Innovations in Patient Safety.

OBJECTIVE: The purpose of this study was to qualitatively examine safety experiences of hospitalized patients and families. METHODS: We conducted 5 focus groups at 2 sites with patients and family members of patients who had been hospitalized at least once within the preceding 2 years. Using a semistructured focus group script, participants were asked to describe hospital experiences, including any safety risks or problems, and to discuss trust in the hospital care team or members of the care team. All focus groups were audiorecorded and transcribed, and transcriptions were qualitatively analyzed using thematic analysis by experienced qualitative analysts and experts in patient safety. RESULTS: We collected rich descriptions of safety problems in the hospital. We identified 4 main themes from our focus group data. (1) Experiences with safety problems were not unusual among participants, (2) patients and families develop a structured "care story" about their hospital experiences, (3) there is a spectrum of trust between patients and the hospital care team members that can be diminished or enhanced by experiences, and (4) patients believed having someone who could advocate for them during their hospitalization was important. CONCLUSIONS: Our results suggest that acknowledgment of safety problems, clear communication, building trust, and a role for advocacy are impactful pathways health care providers and health care systems can improve patient experiences. Information technology such as patient- and clinician-facing displays can support each of these actions.

Engaging Patients in the Use of Real-Time Electronic Clinical Data to Improve the Safety and Reliability of Their Own Care.

OBJECTIVES: There is considerable evidence that providing patients with access to their health information is beneficial, but there is limited evidence regarding the effect of providing real-time patient safety-related information on health outcomes. The aim of this study was to evaluate the association between use of an electronic patient safety dashboard (Safety Advisor) and health outcomes. METHODS: The Safety Advisor was implemented in 6 adult medicine units at one hospital in the United States. Study participants were asked to use the Safety Advisor, which provides real-time patient safety-related information through a Web-based portal. The primary outcome was the association between the application usage and health outcomes (readmission rate and mortality rate) per 3 different usage groups, and the secondary outcome was the association of Patient Activation Measure (PAM) scores with use. RESULTS: One hundred eighty-one participants were included for the data analysis. Approximately 90% of users accessed the application during the first 4 days of enrollment: 51.6% of users only accessed it on 1 day, whereas 5.8% used it more than 3 days. Patients who used the application more had lower 30-day readmission rates (P = 0.01) compared with the lower-usage group. The PAM scores for users of Safety Advisor (71.8) were higher than the nonpatient portal users (60.8, P < 0.0001). CONCLUSIONS: We found an association between the use of Safety Advisor and health outcomes. Differences in PAM scores between groups were statistically significant. A larger-scale randomized control trial is warranted to evaluate the impact on patient outcomes among a high-risk patient population.

Development and evaluation of an interoperable natural language processing system for identifying pneumonia across clinical settings of care and institutions.

Objective: To evaluate the feasibility, accuracy, and interoperability of a natural language processing (NLP) system that extracts diagnostic assertions of pneumonia in different clinical notes and institutions. Materials and Methods: A rule-based NLP system was designed to identify assertions of pneumonia in 3 types of clinical notes from electronic health records (EHRs): emergency department notes, radiology reports, and discharge summaries. The lexicon and classification logic were tailored for each note type. The system was first developed and evaluated using annotated notes from the Department of Veterans Affairs (VA). Interoperability was assessed using data from the University of Utah (UU). Results: The NLP system was comprised of 782 rules and achieved moderate-to-high performance in all 3 note types in VA (precision/recall/f1: emergency = 88.1/86.0/87.1; radiology = 71.4/96.2/82.0; discharge = 88.3/93.0/90.1). When applied to UU data, performance was maintained in emergency and radiology but decreased in discharge summaries (emergency = 84.7/94.3/89.3; radiology = 79.7/100.0/87.9; discharge = 65.5/92.7/76.8). Customization with 34 additional rules increased performance for all note types (emergency = 89.3/94.3/91.7; radiology = 87.0/100.0/93.1; discharge = 75.0/95.1/83.4). Conclusion: NLP can be used to accurately identify the diagnosis of pneumonia across different clinical settings and institutions. A limited amount of customization to account for differences in lexicon, clinical definition of pneumonia, and EHR structure can achieve high accuracy without substantial modification.

Assessing hospital electronic health record vendor performance across publicly reported quality measures.

OBJECTIVE: Little is known regarding variation among electronic health record (EHR) vendors in quality performance. This issue is compounded by selection effects in which high-quality hospitals coalesce to a subset of market leading vendors. We measured hospital performance, stratified by EHR vendor, across 4 quality metrics. MATERIALS AND METHODS: We used data on 1272 hospitals in 2018 across 4 quality measures: Leapfrog Computerized Provider Order Entry/EHR Evaluation, Centers for Medicare and Medicaid Services Hospital Compare Star Ratings, Hospital-Acquired Condition (HAC) score, and Hospital Readmission Reduction Program (HRRP) ratio. We examined score distributions and used multivariable regression to evaluate the association between vendor and score, recovering partial R2 to assess the proportion of quality variation explained by vendor. RESULTS: We found significant variation across and within EHR vendors. The largest vendor, vendor A, had the highest mean score on the Leapfrog Computerized Provider Order Entry/EHR Evaluation and HRRP ratio, vendor G had the highest Hospital Compare score, and vendor F had the highest HAC score. In adjusted models, no vendor was significantly associated with higher performance on more than 2 measures. EHR vendor explained between 1.2% (HAC) and 7.6 (HRRP) of the variation in quality performance. DISCUSSION: No EHR vendor was associated with higher quality across all measures, and the 2 largest vendors were not associated with the highest scores. Only a small fraction of quality variation was explained by EHR vendor choice. CONCLUSIONS: Top performance on quality measures can be achieved with any EHR vendor; much of quality performance is driven by the hospital and how it uses the EHR.

The Development and Piloting of the Ambulatory Electronic Health Record Evaluation Tool: Lessons Learned.

BACKGROUND: Substantial research has been performed about the impact of computerized physician order entry on medication safety in the inpatient setting; however, relatively little has been done in ambulatory care, where most medications are prescribed. OBJECTIVE: To outline the development and piloting process of the Ambulatory Electronic Health Record (EHR) Evaluation Tool and to report the quantitative and qualitative results from the pilot. METHODS: The Ambulatory EHR Evaluation Tool closely mirrors the inpatient version of the tool, which is administered by The Leapfrog Group. The tool was piloted with seven clinics in the United States, each using a different EHR. The tool consists of a medication safety test and a medication reconciliation module. For the medication test, clinics entered test patients and associated test orders into their EHR and recorded any decision support they received. An overall percentage score of unsafe orders detected, and order category scores were provided to clinics. For the medication reconciliation module, clinics demonstrated how their EHR electronically detected discrepancies between two medication lists. RESULTS: For the medication safety test, the clinics correctly alerted on 54.6% of unsafe medication orders. Clinics scored highest in the drug allergy (100%) and drug-drug interaction (89.3%) categories. Lower scoring categories included drug age (39.3%) and therapeutic duplication (39.3%). None of the clinics alerted for the drug laboratory or drug monitoring orders. In the medication reconciliation module, three (42.8%) clinics had an EHR-based medication reconciliation function; however, only one of those clinics could demonstrate it during the pilot. CONCLUSION: Clinics struggled in areas of advanced decision support such as drug age, drug laboratory, and drub monitoring. Most clinics did not have an EHR-based medication reconciliation function and this process was dependent on accessing patients' medication lists. Wider use of this tool could improve outpatient medication safety and can inform vendors about areas of improvement.