Identifying signs and symptoms of urinary tract infection from emergency department clinical notes using large language models.

BACKGROUND: Natural language processing (NLP) tools including recently developed large language models (LLMs) have myriad potential applications in medical care and research, including the efficient labeling and classification of unstructured text such as electronic health record (EHR) notes. This opens the door to large-scale projects that rely on variables that are not typically recorded in a structured form, such as patient signs and symptoms. OBJECTIVES: This study is designed to acquaint the emergency medicine research community with the foundational elements of NLP, highlighting essential terminology, annotation methodologies, and the intricacies involved in training and evaluating NLP models. Symptom characterization is critical to urinary tract infection (UTI) diagnosis, but identification of symptoms from the EHR has historically been challenging, limiting large-scale research, public health surveillance, and EHR-based clinical decision support. We therefore developed and compared two NLP models to identify UTI symptoms from unstructured emergency department (ED) notes. METHODS: The study population consisted of patients aged ≥ 18 who presented to an ED in a northeastern U.S. health system between June 2013 and August 2021 and had a urinalysis performed. We annotated a random subset of 1250 ED clinician notes from these visits for a list of 17 UTI symptoms. We then developed two task-specific LLMs to perform the task of named entity recognition: a convolutional neural network-based model (SpaCy) and a transformer-based model designed to process longer documents (Clinical Longformer). Models were trained on 1000 notes and tested on a holdout set of 250 notes. We compared model performance (precision, recall, F1 measure) at identifying the presence or absence of UTI symptoms at the note level. RESULTS: A total of 8135 entities were identified in 1250 notes; 83.6% of notes included at least one entity. Overall F1 measure for note-level symptom identification weighted by entity frequency was 0.84 for the SpaCy model and 0.88 for the Longformer model. F1 measure for identifying presence or absence of any UTI symptom in a clinical note was 0.96 (232/250 correctly classified) for the SpaCy model and 0.98 (240/250 correctly classified) for the Longformer model. CONCLUSIONS: The study demonstrated the utility of LLMs and transformer-based models in particular for extracting UTI symptoms from unstructured ED clinical notes; models were highly accurate for detecting the presence or absence of any UTI symptom on the note level, with variable performance for individual symptoms.

Automated HEART score determination via ChatGPT: Honing a framework for iterative prompt development.

Objectives: This study presents a design framework to enhance the accuracy by which large language models (LLMs), like ChatGPT can extract insights from clinical notes. We highlight this framework via prompt refinement for the automated determination of HEART (History, ECG, Age, Risk factors, Troponin risk algorithm) scores in chest pain evaluation. Methods: We developed a pipeline for LLM prompt testing, employing stochastic repeat testing and quantifying response errors relative to physician assessment. We evaluated the pipeline for automated HEART score determination across a limited set of 24 synthetic clinical notes representing four simulated patients. To assess whether iterative prompt design could improve the LLMs' ability to extract complex clinical concepts and apply rule-based logic to translate them to HEART subscores, we monitored diagnostic performance during prompt iteration. Results: Validation included three iterative rounds of prompt improvement for three HEART subscores with 25 repeat trials totaling 1200 queries each for GPT-3.5 and GPT-4. For both LLM models, from initial to final prompt design, there was a decrease in the rate of responses with erroneous, non-numerical subscore answers. Accuracy of numerical responses for HEART subscores (discrete 0-2 point scale) improved for GPT-4 from the initial to final prompt iteration, decreasing from a mean error of 0.16-0.10 (95% confidence interval: 0.07-0.14) points. Conclusion: We established a framework for iterative prompt design in the clinical space. Although the results indicate potential for integrating LLMs in structured clinical note analysis, translation to real, large-scale clinical data with appropriate data privacy safeguards is needed.

PROSER: A Web-Based Peripheral Blood Smear Interpretation Support Tool Utilizing Electronic Health Record Data.

OBJECTIVES: Peripheral blood smear (PBS) interpretation represents a cornerstone of pathology practice and resident training but has remained largely static for decades. Here, we describe a novel PBS interpretation support tool. METHODS: In a mixed-methods quality improvement study, a web-based clinical decision support (CDS) tool to assist pathologists in PBS interpretation, PROSER, was deployed in an academic hospital over a 2-month period in 2022. PROSER interfaced with the hospital system's electronic health record and data warehouse to obtain and display relevant demographic, laboratory, and medication information for patients with pending PBS consults. PROSER used these data along with morphologic findings entered by the pathologist to draft a PBS interpretation using rule-based logic. We evaluated users' perceptions of PROSER with a Likert-type survey. RESULTS: PROSER displayed 46 laboratory values with corresponding reference ranges and abnormal flags, allowed for entry of 14 microscopy findings, and computed 2 calculations based on laboratory values; it composed automated PBS reports using a library of 92 prewritten phrases. Overall, PROSER was well received by residents. CONCLUSIONS: In this quality improvement study, we successfully deployed a web-based CDS tool for PBS interpretation. Future work is needed to quantitatively evaluate this intervention's effects on clinical outcomes and resident training.

Development and Implementation of a Standard Format for Clinical Laboratory Test Results.

OBJECTIVES: Surprisingly, laboratory results, the principal output of clinical laboratories, are not standardized. Thus, laboratories frequently report results with identical meaning in different formats. For example, laboratories report a positive pregnancy test as "+," "P," or "Positive." To assess the feasibility of a widespread implementation of a result standard, we (1) developed a standard result format for common laboratory tests and (2) implemented a feedback system for clinical laboratories to view their unstandardized results. METHODS: In the largest integrated health care system in America, 130 facilities had the opportunity to collaboratively develop the standard. For 15 weeks, clinical laboratories received a weekly report of their unstandardized results. At the study's conclusion, laboratories were compared with themselves and their peers by metrics that reflected their unstandardized results. RESULTS: We rereviewed 156 million test results and observed a 51% decline in the rate of unstandardized results. The number of facilities with fewer than 23 unstandardized results per 100,000 (Six Sigma σ > 5) increased by 58% (52 to 82 facilities; ß = 1.79; P < .001). CONCLUSIONS: This study demonstrated significant improvement in the standardization of clinical laboratory results in a relatively short time. The laboratory community should create and promulgate a standardized result format.

Cluster Analysis of Primary Care Physician Phenotypes for Electronic Health Record Use: Retrospective Cohort Study.

BACKGROUND: Electronic health records (EHRs) have become ubiquitous in US office-based physician practices. However, the different ways in which users engage with EHRs remain poorly characterized. OBJECTIVE: The aim of this study is to explore EHR use phenotypes among ambulatory care physicians. METHODS: In this retrospective cohort analysis, we applied affinity propagation, an unsupervised clustering machine learning technique, to identify EHR user types among primary care physicians. RESULTS: We identified 4 distinct phenotype clusters generalized across internal medicine, family medicine, and pediatrics specialties. Total EHR use varied for physicians in 2 clusters with above-average ratios of work outside of scheduled hours. This finding suggested that one cluster of physicians may have worked outside of scheduled hours out of necessity, whereas the other preferred ad hoc work hours. The two remaining clusters represented physicians with below-average EHR time and physicians who spend the largest proportion of their EHR time on documentation. CONCLUSIONS: These findings demonstrate the utility of cluster analysis for exploring EHR use phenotypes and may offer opportunities for interventions to improve interface design to better support users' needs.

Teaching medical students about cost-effectiveness.

BACKGROUND: Rising and burdensome health care costs have driven interest in the practice of high-value care (HVC) and have inspired calls for increased HVC training across all levels of medical education, including among undergraduate medical students. CONTEXT: Classroom-based HVC curricula targeted to medical students have not been previously described in the medical literature. INNOVATION: We developed and evaluated a workshop comprising a lecture, a small-group exercise and a group discussion to instruct medical students on interpreting cost-effectiveness analyses (CEA), applying CEA to patient care and discussing the cost of care with patients. From January 2014 to September 2015 the workshop was administered to five cohorts, 120 students in total, in the internal medicine clerkships at two US medical schools. Pre- and post-intervention confidence in various domains was assessed with a Likert-type scale ranging from 1 to 4. The overall response rate was 87.9 per cent. The proportion of students reporting high confidence scores (3 or 4) rose significantly (p < 0.01) in each domain: from 16.2 to 76.9 per cent for calculating an incremental cost-effectiveness ratio (ICER); from 16.0 to 79.6 per cent for interpreting quality-adjusted life-years (QALYs); from 8.7 to 71.3 per cent for using CEA in patient management; and from 15.3 to 71.4 per cent for discussing costs with patients. Students rated the overall quality of the course as 3.82 out of 5. Rising and burdensome health care costs have driven interest in the practice of high-value care IMPLICATIONS: Our experience of developing, evaluating and refining an HVC course targeted at medical students taught us that such a course is needed, can be educational and can be well-received. Future research is needed to assess the effects of curricula on clinical practice.

The Student Curriculum Review Team: How we catalyze curricular changes through a student-centered approach.

Student feedback is a valuable asset in curriculum evaluation and improvement, but many institutions have faced challenges implementing it in a meaningful way. In this article, we report the rationale, process and impact of the Student Curriculum Review Team (SCRT), a student-led and faculty-supported organization at the Johns Hopkins University School of Medicine. SCRT's evaluation of each pre-clinical course is composed of a comprehensive three-step process: a review of course evaluation data, a Town Hall Meeting and online survey to generate and assess potential solutions, and a thoughtful discussion with course directors. Over the past two years, SCRT has demonstrated the strength of its approach by playing a substantial role in improving medical education, as reported by students and faculty. Furthermore, SCRT's uniquely student-centered, collaborative model has strengthened relationships between students and faculty and is one that could be readily adapted to other medical schools or academic institutions.

HCV and HBV testing acceptability and knowledge among urban emergency department patients and pharmacy clients.

OBJECTIVE: Hepatitis C and hepatitis B are public health problems in the United States and remain largely undiagnosed. In response to the availability of rapid, point of care hepatitis tests, we assessed hepatitis knowledge and acceptability of hepatitis testing during an emergency department (ED) or pharmacy visit. METHODS: From June 2010 to May 2011, an anonymous prospective survey was administered to a convenience sample of New York City ED patients and pharmacy clients. RESULTS: The study population (N=2078) was 54% female, 36% Hispanic and 41% black. Mean age was 39, SD ± 15 years. The majority (72%;1480/2,2060) of the participants responded that they would get tested if free testing were offered, and 67% (1272/1912) of those responded that they would test for hepatitis B/C in conjunction with HIV. Participants who had previously tested for hepatitis had higher mean knowledge scores than those who had never tested. Pharmacy clients, those of black race, and those with higher mean knowledge scores would be more willing to accept hepatitis B/C testing if offered. CONCLUSIONS: Urban ED patients and pharmacy clients were receptive to hepatitis testing. Most individuals would elect to be tested for hepatitis with HIV, which raises the possibility of integrated testing.