Automated HEART score determination via ChatGPT: Honing a framework for iterative prompt development.

Objectives: This study presents a design framework to enhance the accuracy by which large language models (LLMs), like ChatGPT can extract insights from clinical notes. We highlight this framework via prompt refinement for the automated determination of HEART (History, ECG, Age, Risk factors, Troponin risk algorithm) scores in chest pain evaluation. Methods: We developed a pipeline for LLM prompt testing, employing stochastic repeat testing and quantifying response errors relative to physician assessment. We evaluated the pipeline for automated HEART score determination across a limited set of 24 synthetic clinical notes representing four simulated patients. To assess whether iterative prompt design could improve the LLMs' ability to extract complex clinical concepts and apply rule-based logic to translate them to HEART subscores, we monitored diagnostic performance during prompt iteration. Results: Validation included three iterative rounds of prompt improvement for three HEART subscores with 25 repeat trials totaling 1200 queries each for GPT-3.5 and GPT-4. For both LLM models, from initial to final prompt design, there was a decrease in the rate of responses with erroneous, non-numerical subscore answers. Accuracy of numerical responses for HEART subscores (discrete 0-2 point scale) improved for GPT-4 from the initial to final prompt iteration, decreasing from a mean error of 0.16-0.10 (95% confidence interval: 0.07-0.14) points. Conclusion: We established a framework for iterative prompt design in the clinical space. Although the results indicate potential for integrating LLMs in structured clinical note analysis, translation to real, large-scale clinical data with appropriate data privacy safeguards is needed.

Correction: How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment.

[This corrects the article DOI: 10.2196/45312.].

The Impact of Four Smoking Metrics on Survival After Diagnosis with HPV+ Oropharyngeal Cancer.

OBJECTIVE: While tobacco use is understood to negatively impact HPV+ oropharyngeal squamous cell carcinoma (OPSCC) outcomes, debate remains as to how this impact differs between cohorts. Multiple smoking metrics have been identified as having the greatest prognostic significance, and some recent works have found smoking to have no significant impact. Herein, we show through an analysis of four common smoking metrics that while smoking impacts overall survival (OS), it has a limited impact on recurrence-free survival (RFS) in our cohort. METHODS: We conducted a retrospective review of patients treated for HPV+ OPSCC in our health system from 2012 to 2019. Patients with metastatic disease or concurrent second primaries were excluded. Four metrics of tobacco use were assessed: current/former/never smokers, ever/never smokers, and smokers with >10 or >20 pack-year (PY) smoking histories. Our main outcomes were 3-year RFS and OS. RESULTS: Three hundred and sixty-seven patients met inclusion criteria. 37.3% of patients (137/367) were never-smokers; 13.8% of patients (51/367) were currently smoking at diagnosis and 48.8% of patients (179/367) were former smokers. No tobacco-use metric significantly impacted 3-year RFS. On univariate analysis, all smoking metrics yielded inferior OS. On multivariate analysis, current and ever smoking status significantly impacted 3-year OS. CONCLUSION: The impact of tobacco use on HPV+ OPSCC outcomes is not universal, but may instead be modulated by other cohort-specific factors. The impact of smoking may decrease as rates of tobacco use decline. LEVEL OF EVIDENCE: 3 (Cohort and case-control studies) Laryngoscope, 134:3158-3164, 2024.

Automated AJCC Restaging for Oropharyngeal Cancer Research.

With the American Joint Committee on Cancer (AJCC) 8th edition staging guidelines update, human papillomavirus-positive (HPV+) oropharyngeal squamous cell carcinoma (OPSCC) is now staged separately from its HPV-negative counterpart, preventing meaningful comparison of cases staged with the 7th versus 8th edition criteria. Manual restaging is time-consuming and error-prone, hindering multiyear analyses for HPV+ OPSCC. We developed an automated computational tool for re-classifying HPV+ OPSCC pathological and clinical tumor staging from AJCC 7th to 8th edition. The tool is designed to handle large data sets, ensuring comprehensive and accurate analysis of historic HPV+ OPSCC data. Validated against institutional and National Cancer Database data sets, the algorithm achieved accuracies of 100% (95% confidence interval [CI] 98.8%-100%) and 93.4% (95% CI 93.1%-93.7%), successfully restaging 326/326 and 26,505/28,374 cases, respectively. With its open-source design, this computational tool can enhance future HPV+ OPSCC research and inspire similar tools for other cancer types and subsequent AJCC editions.

The Role of Large Language Models in Medical Education: Applications and Implications.

Large language models (LLMs) such as ChatGPT have sparked extensive discourse within the medical education community, spurring both excitement and apprehension. Written from the perspective of medical students, this editorial offers insights gleaned through immersive interactions with ChatGPT, contextualized by ongoing research into the imminent role of LLMs in health care. Three distinct positive use cases for ChatGPT were identified: facilitating differential diagnosis brainstorming, providing interactive practice cases, and aiding in multiple-choice question review. These use cases can effectively help students learn foundational medical knowledge during the preclinical curriculum while reinforcing the learning of core Entrustable Professional Activities. Simultaneously, we highlight key limitations of LLMs in medical education, including their insufficient ability to teach the integration of contextual and external information, comprehend sensory and nonverbal cues, cultivate rapport and interpersonal interaction, and align with overarching medical education and patient care goals. Through interacting with LLMs to augment learning during medical school, students can gain an understanding of their strengths and weaknesses. This understanding will be pivotal as we navigate a health care landscape increasingly intertwined with LLMs and artificial intelligence.

How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment.

BACKGROUND: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. OBJECTIVE: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. METHODS: We used 2 sets of multiple-choice questions to evaluate ChatGPT's performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT's performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. RESULTS: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT's answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. CONCLUSIONS: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT's capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.

A computer modeling method to analyze rideshare data for the surveillance of novel strains of SARS-CoV-2.

PURPOSE: No method is available to systematically study SARS-CoV-2 transmission dynamics using the data that rideshare companies share with government agencies. We developed a proof-of-concept method for the analysis of SARS-CoV-2 transmissions between rideshare passengers and drivers. METHOD: To assess whether this method could enable hypothesis testing about SARS-CoV-2, we repeated ten 200-day agent-based simulations of SARS-CoV-2 propagation within the Los Angeles County rideshare network. Assuming data access for 25% of infections, we estimated an epidemiologist's ability to analyze the observable infection patterns to correctly identify a baseline viral variant A, as opposed to viral variant A with mask use (50% reduction in viral particle exchange), or a more infectious viral variant B (300% higher cumulative viral load). RESULTS: Simulations had an average of 190,387 potentially infectious rideshare interactions, resulting in 409 average diagnosed infections. Comparison of the number of observed and expected passenger-to-driver infections under each hypothesis demonstrated our method's ability to consistently discern large infectivity differences (viral variant A vs. viral variant B) given partial data from one large city, and to discern smaller infectivity differences (viral variant A vs. viral variant A with masks) given partial data aggregated across multiple cities. CONCLUSIONS: This novel statistical method suggests that, for the present and subsequent pandemics, government-facilitated analysis of rideshare data combined with diagnosis records may augment efforts to better understand viral transmission dynamics and to measure changes in infectivity associated with nonpharmaceutical interventions and emergent viral strains.

Visualizing Opioid-Use Variation in a Pediatric Perioperative Dashboard.

BACKGROUND: Anesthesiologists integrate numerous variables to determine an opioid dose that manages patient nociception and pain while minimizing adverse effects. Clinical dashboards that enable physicians to compare themselves to their peers can reduce unnecessary variation in patient care and improve outcomes. However, due to the complexity of anesthetic dosing decisions, comparative visualizations of opioid-use patterns are complicated by case-mix differences between providers. OBJECTIVES: This single-institution case study describes the development of a pediatric anesthesia dashboard and demonstrates how advanced computational techniques can facilitate nuanced normalization techniques, enabling meaningful comparisons of complex clinical data. METHODS: We engaged perioperative-care stakeholders at a tertiary care pediatric hospital to determine patient and surgical variables relevant to anesthesia decision-making and to identify end-user requirements for an opioid-use visualization tool. Case data were extracted, aggregated, and standardized. We performed multivariable machine learning to identify and understand key variables. We integrated interview findings and computational algorithms into an interactive dashboard with normalized comparisons, followed by an iterative process of improvement and implementation. RESULTS: The dashboard design process identified two mechanisms-interactive data filtration and machine-learning-based normalization-that enable rigorous monitoring of opioid utilization with meaningful case-mix adjustment. When deployed with real data encompassing 24,332 surgical cases, our dashboard identified both high and low opioid-use outliers with associated clinical outcomes data. CONCLUSION: A tool that gives anesthesiologists timely data on their practice patterns while adjusting for case-mix differences empowers physicians to track changes and variation in opioid administration over time. Such a tool can successfully trigger conversation amongst stakeholders in support of continuous improvement efforts. Clinical analytics dashboards can enable physicians to better understand their practice and provide motivation to change behavior, ultimately addressing unnecessary variation in high impact medication use and minimizing adverse effects.

Injectable hyaluronic acid based microrods provide local micromechanical and biochemical cues to attenuate cardiac fibrosis after myocardial infarction.

Repairing cardiac tissue after myocardial infarction (MI) is one of the most challenging goals in tissue engineering. Following ischemic injury, significant matrix remodeling and the formation of avascular scar tissue significantly impairs cell engraftment and survival in the damaged myocardium. This limits the efficacy of cell replacement therapies, demanding strategies that reduce pathological scarring to create a suitable microenvironment for healthy tissue regeneration. Here, we demonstrate the successful fabrication of discrete hyaluronic acid (HA)-based microrods to provide local biochemical and biomechanical signals to reprogram cells and attenuate cardiac fibrosis. HA microrods were produced in a range of physiological stiffness and shown to degrade in the presence of hyaluronidase. Additionally, we show that fibroblasts interact with these microrods in vitro, leading to significant changes in proliferation, collagen expression and other markers of a myofibroblast phenotype. When injected into the myocardium of an adult rat MI model, HA microrods prevented left ventricular wall thinning and improved cardiac function at 6 weeks post infarct.