Integrating clinical research into electronic health record workflows to support a learning health system.

Objective: Integrating clinical research into routine clinical care workflows within electronic health record systems (EHRs) can be challenging, expensive, and labor-intensive. This case study presents a large-scale clinical research project conducted entirely within a commercial EHR during the COVID-19 pandemic. Case Report: The UCSD and UCSDH COVID-19 NeutraliZing Antibody Project (ZAP) aimed to evaluate antibody levels to SARS-CoV-2 virus in a large population at an academic medical center and examine the association between antibody levels and subsequent infection diagnosis. Results: The project rapidly and successfully enrolled and consented over 2000 participants, integrating the research trial with standing COVID-19 testing operations, staff, lab, and mobile applications. EHR-integration increased enrollment, ease of scheduling, survey distribution, and return of research results at a low cost by utilizing existing resources. Conclusion: The case study highlights the potential benefits of EHR-integrated clinical research, expanding their reach across multiple health systems and facilitating rapid learning during a global health crisis.

Ransomware Cyberattack Associated With Cardiac Arrest Incidence and Outcomes at Untargeted, Adjacent Hospitals.

OBJECTIVES: Healthcare ransomware cyberattacks have been associated with major regional hospital disruptions, but data reporting patient-oriented outcomes in critical conditions such as cardiac arrest (CA) are limited. This study examined the CA incidence and outcomes of untargeted hospitals adjacent to a ransomware-infected healthcare delivery organization (HDO). DESIGN SETTING AND PATIENTS: This cohort study compared the CA incidence and outcomes of two untargeted academic hospitals adjacent to an HDO under a ransomware cyberattack during the pre-attack (April 3-30, 2021), attack (May 1-28, 2021), and post-attack (May 29, 2021-June 25, 2021) phases. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Emergency department and hospital mean daily census, number of CAs, mean daily CA incidence per 1,000 admissions, return of spontaneous circulation, survival to discharge, and survival with favorable neurologic outcome were measured. The study evaluated 78 total CAs: 44 out-of-hospital CAs (OHCAs) and 34 in-hospital CAs. The number of total CAs increased from the pre-attack to attack phase (21 vs. 38; p = 0.03), followed by a decrease in the post-attack phase (38 vs. 19; p = 0.01). The number of total CAs exceeded the cyberattack month forecast (May 2021: 41 observed vs. 27 forecasted cases; 95% CI, 17.0-37.4). OHCA cases also exceeded the forecast (May 2021: 24 observed vs. 12 forecasted cases; 95% CI, 6.0-18.8). Survival with favorable neurologic outcome rates for all CAs decreased, driven by increases in OHCA mortality: survival with favorable neurologic rates for OHCAs decreased from the pre-attack phase to attack phase (40.0% vs. 4.5%; p = 0.02) followed by an increase in the post-attack phase (4.5% vs. 41.2%; p = 0.01). CONCLUSIONS: Untargeted hospitals adjacent to ransomware-infected HDOs may see worse outcomes for patients suffering from OHCA. These findings highlight the critical need for cybersecurity disaster planning and resiliency.

AI-Generated Draft Replies Integrated Into Health Records and Physicians' Electronic Communication.

Importance: Timely tests are warranted to assess the association between generative artificial intelligence (GenAI) use and physicians' work efforts. Objective: To investigate the association between GenAI-drafted replies for patient messages and physician time spent on answering messages and the length of replies. Design, Setting, and Participants: Randomized waiting list quality improvement (QI) study from June to August 2023 in an academic health system. Primary care physicians were randomized to an immediate activation group and a delayed activation group. Data were analyzed from August to November 2023. Exposure: Access to GenAI-drafted replies for patient messages. Main Outcomes and Measures: Time spent (1) reading messages, (2) replying to messages, (3) length of replies, and (4) physician likelihood to recommend GenAI drafts. The a priori hypothesis was that GenAI drafts would be associated with less physician time spent reading and replying to messages. A mixed-effects model was used. Results: Fifty-two physicians participated in this QI study, with 25 randomized to the immediate activation group and 27 randomized to the delayed activation group. A contemporary control group included 70 physicians. There were 18 female participants (72.0%) in the immediate group and 17 female participants (63.0%) in the delayed group; the median age range was 35-44 years in the immediate group and 45-54 years in the delayed group. The median (IQR) time spent reading messages in the immediate group was 26 (11-69) seconds at baseline, 31 (15-70) seconds 3 weeks after entry to the intervention, and 31 (14-70) seconds 6 weeks after entry. The delayed group's median (IQR) read time was 25 (10-67) seconds at baseline, 29 (11-77) seconds during the 3-week waiting period, and 32 (15-72) seconds 3 weeks after entry to the intervention. The contemporary control group's median (IQR) read times were 21 (9-54), 22 (9-63), and 23 (9-60) seconds in corresponding periods. The estimated association of GenAI was a 21.8% increase in read time (95% CI, 5.2% to 41.0%; P = .008), a -5.9% change in reply time (95% CI, -16.6% to 6.2%; P = .33), and a 17.9% increase in reply length (95% CI, 10.1% to 26.2%; P < .001). Participants recognized GenAI's value and suggested areas for improvement. Conclusions and Relevance: In this QI study, GenAI-drafted replies were associated with significantly increased read time, no change in reply time, significantly increased reply length, and some perceived benefits. Rigorous empirical tests are necessary to further examine GenAI's performance. Future studies should examine patient experience and compare multiple GenAIs, including those with medical training.

Generative artificial intelligence responses to patient messages in the electronic health record: early lessons learned.

Background: Electronic health record (EHR)-based patient messages can contribute to burnout. Messages with a negative tone are particularly challenging to address. In this perspective, we describe our initial evaluation of large language model (LLM)-generated responses to negative EHR patient messages and contend that using LLMs to generate initial drafts may be feasible, although refinement will be needed. Methods: A retrospective sample (n = 50) of negative patient messages was extracted from a health system EHR, de-identified, and inputted into an LLM (ChatGPT). Qualitative analyses were conducted to compare LLM responses to actual care team responses. Results: Some LLM-generated draft responses varied from human responses in relational connection, informational content, and recommendations for next steps. Occasionally, the LLM draft responses could have potentially escalated emotionally charged conversations. Conclusion: Further work is needed to optimize the use of LLMs for responding to negative patient messages in the EHR.

Call me Ishmael: addressing the white whale of team communication in the operating room with labelled surgical caps at an academic medical centre.

INTRODUCTION: Effective communication in the operating room (OR) is crucial. Addressing a colleague by their name is respectful, humanising, entrusting and associated with improved clinical outcomes. We aimed to enhance team communication in the perioperative environment by offering personalised surgical caps labelled with name and provider role to all OR team members at a large academic medical centre. MATERIALS AND METHODS: This was a quasi-experimental, uncontrolled, before-and-after quality improvement study. A survey regarding perceptions of team communication, knowledge of names and roles, communication barriers, and culture was administered before and after cap delivery. Survey results were measured on a 5-point Likert Scale; descriptive statistics and mean scores were compared. All cause National Surgical Quality Improvement Project (NSQIP) morbidity and mortality outcomes for surgical specialties were examined. RESULTS: 1420 caps were delivered across the institution. Mean survey scores increased for knowing the names and roles of providers around the OR, feeling that people know my name and feeling comfortable communicating without barriers across disciplines. The mean score for team communication around the OR is excellent was unchanged. The highest score both before and after was knowing the name of an interdisciplinary team member is important for patient care. A total of 383 and 212 providers participated in the study before and after cap delivery, respectively. Participants agreed or strongly agreed that labelled surgical caps made it easier to talk to colleagues (64.9%) while improving communication (66.0%), team culture (60.5%) and patient care (56.8%). No significant differences were noted in NSQIP outcomes. CONCLUSIONS: Personalised labelled surgical caps are a simple, inexpensive tool that demonstrates promise in improving perioperative team communication. Creating highly reliable surgical teams with optimal communication channels requires a multifaceted approach with engaged leadership, empowered front-line providers and an institutional commitment to continuous process improvement.

Implementation of an electronic health record-integrated instant messaging system in an academic health system.

OBJECTIVES: Effective communication amongst healthcare workers simultaneously promotes optimal patient outcomes when present and is deleterious to outcomes when absent. The advent of electronic health record (EHR)-embedded secure instantaneous messaging systems has provided a new conduit for provider communication. This manuscript describes the experience of one academic medical center with deployment of one such system (Secure Chat). METHODS: Data were collected on Secure Chat message volume from June 2017 to April 2023. Significant perideployment events were reviewed chronologically. RESULTS: After the first coronavirus disease 2019 lockdown in March 2020, messaging use increased by over 25 000 messages per month, with 1.2 million messages sent monthly by April 2023. Comparative features of current communication modalities in healthcare were summarized, highlighting the many advantages of Secure Chat. CONCLUSIONS: While EHR-embedded secure instantaneous messaging systems represent a novel and potentially valuable communication medium in healthcare, generally agreed-upon best practices for their implementation are, as of yet, undetermined.

An Automated System for Physician Trainee Procedure Logging via Electronic Health Records.

Importance: Procedural proficiency is a core competency for graduate medical education; however, procedural reporting often relies on manual workflows that are duplicative and generate data whose validity and accuracy are difficult to assess. Failure to accurately gather these data can impede learner progression, delay procedures, and negatively impact patient safety. Objective: To examine accuracy and procedure logging completeness of a system that extracts procedural data from an electronic health record system and uploads these data securely to an application used by many residency programs for accreditation. Design, Setting, and Participants: This quality improvement study of all emergency medicine resident physicians at University of California, San Diego Health was performed from May 23, 2023, to June 25, 2023. Exposures: Automated system for procedure data extraction and upload to a residency management software application. Main Outcomes and Measures: The number of procedures captured by the automated system when running silently compared with manually logged procedures in the same timeframe, as well as accuracy of the data upload. Results: Forty-seven residents participated in the initial silent assessment of the extraction component of the system. During a 1-year period (May 23, 2022, to May 7, 2023), 4291 procedures were manually logged by residents, compared with 7617 procedures captured by the automated system during the same period, representing a 78% increase. During assessment of the upload component of the system (May 8, 2023, to June 25, 2023), a total of 1353 procedures and patient encounters were evaluated, with the system operating with a sensitivity of 97.4%, specificity of 100%, and overall accuracy of 99.5%. Conclusions and Relevance: In this quality improvement study of emergency medicine resident physicians, an automated system demonstrated that reliance on self-reported procedure logging resulted in significant procedural underreporting compared with the use of data obtained at the point of performance. Additionally, this system afforded a degree of reliability and validity heretofore absent from the usual after-the-fact procedure logging workflows while using a novel application programming interface-based approach. To our knowledge, this system constitutes the first generalizable implementation of an automated solution to a problem that has existed in graduate medical education for decades.

Integrated Genomic and Social Network Analyses of SARS-CoV-2 Transmission in the Healthcare Setting.

BACKGROUND: Infection prevention (IP) measures are designed to mitigate the transmission of pathogens in healthcare. Using large-scale viral genomic and social network analyses, we determined if IP measures used during the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic were adequate in protecting healthcare workers (HCWs) and patients from acquiring SARS-CoV-2. METHODS: We performed retrospective cross-sectional analyses of viral genomics from all available SARS-CoV-2 viral samples collected at UC San Diego Health and social network analysis using the electronic medical record to derive temporospatial overlap of infections among related viromes and supplemented with contact tracing data. The outcome measure was any instance of healthcare transmission, defined as cases with closely related viral genomes and epidemiological connection within the healthcare setting during the infection window. Between November 2020 through January 2022, 12 933 viral genomes were obtained from 35 666 patients and HCWs. RESULTS: Among 5112 SARS-CoV-2 viral samples sequenced from the second and third waves of SARS-CoV-2 (pre-Omicron), 291 pairs were derived from persons with a plausible healthcare overlap. Of these, 34 pairs (12%) were phylogenetically linked: 19 attributable to household and 14 to healthcare transmission. During the Omicron wave, 2106 contact pairs among 7821 sequences resulted in 120 (6%) related pairs among 32 clusters, of which 10 were consistent with healthcare transmission. Transmission was more likely to occur in shared spaces in the older hospital compared with the newer hospital (2.54 vs 0.63 transmission events per 1000 admissions, P < .001). CONCLUSIONS: IP strategies were effective at identifying and preventing healthcare SARS-CoV-2 transmission.

Impact of a deep learning sepsis prediction model on quality of care and survival.

Sepsis remains a major cause of mortality and morbidity worldwide. Algorithms that assist with the early recognition of sepsis may improve outcomes, but relatively few studies have examined their impact on real-world patient outcomes. Our objective was to assess the impact of a deep-learning model (COMPOSER) for the early prediction of sepsis on patient outcomes. We completed a before-and-after quasi-experimental study at two distinct Emergency Departments (EDs) within the UC San Diego Health System. We included 6217 adult septic patients from 1/1/2021 through 4/30/2023. The exposure tested was a nurse-facing Best Practice Advisory (BPA) triggered by COMPOSER. In-hospital mortality, sepsis bundle compliance, 72-h change in sequential organ failure assessment (SOFA) score following sepsis onset, ICU-free days, and the number of ICU encounters were evaluated in the pre-intervention period (705 days) and the post-intervention period (145 days). The causal impact analysis was performed using a Bayesian structural time-series approach with confounder adjustments to assess the significance of the exposure at the 95% confidence level. The deployment of COMPOSER was significantly associated with a 1.9% absolute reduction (17% relative decrease) in in-hospital sepsis mortality (95% CI, 0.3%-3.5%), a 5.0% absolute increase (10% relative increase) in sepsis bundle compliance (95% CI, 2.4%-8.0%), and a 4% (95% CI, 1.1%-7.1%) reduction in 72-h SOFA change after sepsis onset in causal inference analysis. This study suggests that the deployment of COMPOSER for early prediction of sepsis was associated with a significant reduction in mortality and a significant increase in sepsis bundle compliance.

Inpatient EHR User Experience and Hospital EHR Safety Performance.

IMPORTANCE: Despite the broad adoption and optimization of electronic health record (EHR) systems across the continuum of care, serious usability and safety problems persist. OBJECTIVE: To assess whether EHR safety performance is associated with EHR frontline user experience in a national sample of hospitals. DESIGN, SETTING, AND PARTICIPANTS: This cross-sectional study included all US adult hospitals that used the National Quality Forum Leapfrog Health IT Safety Measure and also used the ARCH Collaborative EHR User experience survey from January 1, 2017, to January 1, 2019. Data analysis was performed from September 2020 to November 2022. MAIN OUTCOMES AND MEASURES: The primary outcomes were hospital performance on the Leapfrog Health IT Safety measure (overall and 10 subcomponents) and the ARCH collaborative frontline user experience scores (overall and 8 subcomponents). Ordinary least squares models with survey responses clustered by hospital were used to assess associations between the overall measures and their subcomponents. RESULTS: There were 112 hospitals and 5689 frontline user surveys included in the study. Hospitals scored a mean of 0.673 (range, 0.297-0.973) on the Leapfrog Health IT safety measure; the mean ARCH EHR user experience score was 3.377 (range, 1 [best] to 5 [worst]). The adjusted ß coefficient between the overall safety score and overall user experience score was 0.011 (95% CI, 0.006-0.016). The ARCH overall score was also significantly associated with 10 subcategory scores of the Leapfrog Health IT safety score, and the overall Leapfrog score was associated with the 8 subcategory scores of the ARCH user experience score. CONCLUSIONS AND RELEVANCE: This cross-sectional study found a positive association between frontline user-rated EHR usability and EHR safety performance. This finding suggests that improving EHR usability, which is a current well-known pain point for EHR users, could have direct benefits in terms of improved EHR safety.

Surgery and Anesthesia Preoperative "Virtual Huddle": A Pilot Trial to Enhance Communication across the Drape.

OBJECTIVES: Effective communication between surgeons and anesthesiologists is critical for high-quality, safe, and efficient perioperative patient care. Despite widespread implementation of surgical safety checklists and time-outs, ineffective team communication remains a leading cause of patient safety events in the operating room. To promote effective communication, we conducted a pilot trial of a "virtual huddle" between anesthesiologists and surgeons. METHODS: Attending anesthesiologists and surgeons at an academic medical center were recruited by email to participate in this feasibility trial. An electronic health record-based smartphone application was utilized to create secure group chats among trial participants the day before a surgery. Text notifications connected a surgeon/anesthesiologist pair in order to introduce colleagues, facilitate a preoperative virtual huddle, and enable open-ended, text message-based communication. A 5-point Likert scale-based survey with a free-text component was used to evaluate the utility of the virtual huddle and usability of the electronic platform. RESULTS: A total of 51 unique virtual huddles occurred between 16 surgeons and 12 anesthesiologists over 99 operations. All postintervention survey questions received a positive rating (range: 3.50/5.00-4.53/5.00) and the virtual huddle was considered to be easy to use (4.47/5.00), improve attending-to-attending communication (4.29/5.00), and improve patient care (4.22/5.00). There were no statistically significant differences in the ratings between surgery and anesthesia. In thematic analysis of qualitative survey results, Participants indicated the intervention was particularly useful in interdisciplinary relationship-building and reducing room turnover. The huddle was less useful for simple, routine cases or when participation was one sided. CONCLUSION: A preoperative virtual huddle may be a simple and effective intervention to improve communication and teamwork in the operating room. Further study and consideration of broader implementation is warranted.

Exposure notification system activity as a leading indicator for SARS-COV-2 caseload forecasting.

PURPOSE: Digital methods to augment traditional contact tracing approaches were developed and deployed globally during the COVID-19 pandemic. These "Exposure Notification (EN)" systems present new opportunities to support public health interventions. To date, there have been attempts to model the impact of such systems, yet no reports have explored the value of real-time system data for predictive epidemiological modeling. METHODS: We investigated the potential to short-term forecast COVID-19 caseloads using data from California's implementation of the Google Apple Exposure Notification (GAEN) platform, branded as CA Notify. CA Notify is a digital public health intervention leveraging resident's smartphones for anonymous EN. We extended a published statistical model that uses prior case counts to investigate the possibility of predicting short-term future case counts and then added EN activity to test for improved forecast performance. Additional predictive value was assessed by comparing the pandemic forecasting models with and without EN activity to the actual reported caseloads from 1-7 days in the future. RESULTS: Observation of time series presents noticeable evidence for temporal association of system activity and caseloads. Incorporating earlier ENs in our model improved prediction of the caseload counts. Using Bayesian inference, we found nonzero influence of EN terms with probability one. Furthermore, we found a reduction in both the mean absolute percentage error and the mean squared prediction error, the latter of at least 5% and up to 32% when using ENs over the model without. CONCLUSIONS: This preliminary investigation suggests smartphone based ENs can significantly improve the accuracy of short-term forecasting. These predictive models can be readily deployed as local early warning systems to triage resources and interventions.

Association of physician burnout with perceived EHR work stress and potentially actionable factors.

OBJECTIVE: Physicians of all specialties experienced unprecedented stressors during the COVID-19 pandemic, exacerbating preexisting burnout. We examine burnout's association with perceived and actionable electronic health record (EHR) workload factors and personal, professional, and organizational characteristics with the goal of identifying levers that can be targeted to address burnout. MATERIALS AND METHODS: Survey of physicians of all specialties in an academic health center, using a standard measure of burnout, self-reported EHR work stress, and EHR-based work assessed by the number of messages regarding prescription reauthorization and use of a staff pool to triage messages. Descriptive and multivariable regression analyses examined the relationship among burnout, perceived EHR work stress, and actionable EHR work factors. RESULTS: Of 1038 eligible physicians, 627 responded (60% response rate), 49.8% reported burnout symptoms. Logistic regression analysis suggests that higher odds of burnout are associated with physicians feeling higher level of EHR stress (odds ratio [OR], 1.15; 95% confidence interval [CI], 1.07-1.25), having more prescription reauthorization messages (OR, 1.23; 95% CI, 1.04-1.47), not feeling valued (OR, 3.38; 95% CI, 1.69-7.22) or aligned in values with clinic leaders (OR, 2.81; 95% CI, 1.87-4.27), in medical practice for ≤15 years (OR, 2.57; 95% CI, 1.63-4.12), and sleeping for <6 h/night (OR, 1.73; 95% CI, 1.12-2.67). DISCUSSION: Perceived EHR stress and prescription reauthorization messages are significantly associated with burnout, as are non-EHR factors such as not feeling valued or aligned in values with clinic leaders. Younger physicians need more support. CONCLUSION: A multipronged approach targeting actionable levers and supporting young physicians is needed to implement sustainable improvements in physician well-being.

Developing a highly-reliable learning health system.

Multiple independent frameworks to support continuous improvement have been proposed to guide healthcare organizations. Two of the most visible are High-reliability Health care, (Chassin et al., 2013) which is emphasized by The Joint Commission, and Learning Health Systems, (Institute of Medicine, 2011) highlighted by the National Academy of Medicine. We propose that organizations consider tightly linking these two models, creating a "Highly-reliable Learning Health System." We describe several efforts at our organization that has resulted from this combined model and have helped our organization weather the COVID-19 pandemic. The organizational changes created using this framework will enable our health system to support a culture of quality across our teams and better fulfill our tripartite mission of high-quality care, effective education of trainees, and dissemination of important innovations.

Ransomware Attack Associated With Disruptions at Adjacent Emergency Departments in the US.

Importance: Cyberattacks on health care delivery organizations are increasing in frequency and sophistication. Ransomware infections have been associated with significant operational disruption, but data describing regional associations of these cyberattacks with neighboring hospitals have not been previously reported, to our knowledge. Objective: To examine an institution's emergency department (ED) patient volume and stroke care metrics during a month-long ransomware attack on a geographically proximal but separate health care delivery organization. Design, Setting, and Participants: This before and after cohort study compares adult and pediatric patient volume and stroke care metrics of 2 US urban academic EDs in the 4 weeks prior to the ransomware attack on May 1, 2021 (April 3-30, 2021), as well as during the attack and recovery (May 1-28, 2021) and 4 weeks after the attack and recovery (May 29 to June 25, 2021). The 2 EDs had a combined mean annual census of more than 70 000 care encounters and 11% of San Diego County's total acute inpatient discharges. The health care delivery organization targeted by the ransomware constitutes approximately 25% of the regional inpatient discharges. Exposure: A month-long ransomware cyberattack on 4 adjacent hospitals. Main Outcomes and Measures: Emergency department encounter volumes (census), temporal throughput, regional diversion of emergency medical services (EMS), and stroke care metrics. Results: This study evaluated 19 857 ED visits at the unaffected ED: 6114 (mean [SD] age, 49.6 [19.3] years; 2931 [47.9%] female patients; 1663 [27.2%] Hispanic, 677 [11.1%] non-Hispanic Black, and 2678 [43.8%] non-Hispanic White patients) in the preattack phase, 7039 (mean [SD] age, 49.8 [19.5] years; 3377 [48.0%] female patients; 1840 [26.1%] Hispanic, 778 [11.1%] non-Hispanic Black, and 3168 [45.0%] non-Hispanic White patients) in the attack and recovery phase, and 6704 (mean [SD] age, 48.8 [19.6] years; 3326 [49.5%] female patients; 1753 [26.1%] Hispanic, 725 [10.8%] non-Hispanic Black, and 3012 [44.9%] non-Hispanic White patients) in the postattack phase. Compared with the preattack phase, during the attack phase, there were significant associated increases in the daily mean (SD) ED census (218.4 [18.9] vs 251.4 [35.2]; P < .001), EMS arrivals (1741 [28.8] vs 2354 [33.7]; P < .001), admissions (1614 [26.4] vs 1722 [24.5]; P = .01), patients leaving without being seen (158 [2.6] vs 360 [5.1]; P < .001), and patients leaving against medical advice (107 [1.8] vs 161 [2.3]; P = .03). There were also significant associated increases during the attack phase compared with the preattack phase in median waiting room times (21 minutes [IQR, 7-62 minutes] vs 31 minutes [IQR, 9-89 minutes]; P < .001) and total ED length of stay for admitted patients (614 minutes [IQR, 424-1093 minutes] vs 822 minutes [IQR, 497-1524 minutes]; P < .001). There was also a significant increase in stroke code activations during the attack phase compared with the preattack phase (59 vs 102; P = .01) as well as confirmed strokes (22 vs 47; P = .02). Conclusions and Relevance: This study found that hospitals adjacent to health care delivery organizations affected by ransomware attacks may see increases in patient census and may experience resource constraints affecting time-sensitive care for conditions such as acute stroke. These findings suggest that targeted hospital cyberattacks may be associated with disruptions of health care delivery at nontargeted hospitals within a community and should be considered a regional disaster.

Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.

Importance: The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. Artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians. Objective: To evaluate the ability of an AI chatbot assistant (ChatGPT), released in November 2022, to provide quality and empathetic responses to patient questions. Design, Setting, and Participants: In this cross-sectional study, a public and nonidentifiable database of questions from a public social media forum (Reddit's r/AskDocs) was used to randomly draw 195 exchanges from October 2022 where a verified physician responded to a public question. Chatbot responses were generated by entering the original question into a fresh session (without prior questions having been asked in the session) on December 22 and 23, 2022. The original question along with anonymized and randomly ordered physician and chatbot responses were evaluated in triplicate by a team of licensed health care professionals. Evaluators chose "which response was better" and judged both "the quality of information provided" (very poor, poor, acceptable, good, or very good) and "the empathy or bedside manner provided" (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic). Mean outcomes were ordered on a 1 to 5 scale and compared between chatbot and physicians. Results: Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% (95% CI, 75.0%-81.8%) of the 585 evaluations. Mean (IQR) physician responses were significantly shorter than chatbot responses (52 [17-62] words vs 211 [168-245] words; t = 25.4; P < .001). Chatbot responses were rated of significantly higher quality than physician responses (t = 13.3; P < .001). The proportion of responses rated as good or very good quality (≥ 4), for instance, was higher for chatbot than physicians (chatbot: 78.5%, 95% CI, 72.3%-84.1%; physicians: 22.1%, 95% CI, 16.4%-28.2%;). This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses (t = 18.9; P < .001). The proportion of responses rated empathetic or very empathetic (≥4) was higher for chatbot than for physicians (physicians: 4.6%, 95% CI, 2.1%-7.7%; chatbot: 45.1%, 95% CI, 38.5%-51.8%; physicians: 4.6%, 95% CI, 2.1%-7.7%). This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot. Conclusions: In this cross-sectional study, a chatbot generated quality and empathetic responses to patient questions posed in an online forum. Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.

Optimizing the Implementation of Clinical Predictive Models to Minimize National Costs: Sepsis Case Study.

BACKGROUND: Sepsis costs and incidence vary dramatically across diagnostic categories, warranting a customized approach for implementing predictive models. OBJECTIVE: The aim of this study was to optimize the parameters of a sepsis prediction model within distinct patient groups to minimize the excess cost of sepsis care and analyze the potential effect of factors contributing to end-user response to sepsis alerts on overall model utility. METHODS: We calculated the excess costs of sepsis to the Centers for Medicare and Medicaid Services (CMS) by comparing patients with and without a secondary sepsis diagnosis but with the same primary diagnosis and baseline comorbidities. We optimized the parameters of a sepsis prediction algorithm across different diagnostic categories to minimize these excess costs. At the optima, we evaluated diagnostic odds ratios and analyzed the impact of compliance factors such as noncompliance, treatment efficacy, and tolerance for false alarms on the net benefit of triggering sepsis alerts. RESULTS: Compliance factors significantly contributed to the net benefit of triggering a sepsis alert. However, a customized deployment policy can achieve a significantly higher diagnostic odds ratio and reduced costs of sepsis care. Implementing our optimization routine with powerful predictive models could result in US $4.6 billion in excess cost savings for CMS. CONCLUSIONS: We designed a framework for customizing sepsis alert protocols within different diagnostic categories to minimize excess costs and analyzed model performance as a function of false alarm tolerance and compliance with model recommendations. We provide a framework that CMS policymakers could use to recommend minimum adherence rates to the early recognition and appropriate care of sepsis that is sensitive to hospital department-level incidence rates and national excess costs. Customizing the implementation of clinical predictive models by accounting for various behavioral and economic factors may improve the practical benefit of predictive models.

Tele-Untethered: Telemedicine Without Waiting Rooms.

BACKGROUND AND OBJECTIVES: Telemedicine bridges the gap between care needs and provider availability. The value of telemedicine can be eclipsed by long wait times, especially if patients are stuck in virtual waiting rooms. UCSD Tele-Untethered allows patients to join visits without waiting in virtual waiting rooms. Tele-Untethered uses a text-to-video link to improve clinic flow, decrease virtual waiting room reliance, improve throughput, and potentially improve satisfaction. METHODS: This institutional review board (IRB)-approved quality improvement pilot (IRB #210364QI) included patients seen in a single vascular neurology clinic, within the pilot period, if they had a smartphone/cell phone, and agreed to participate in a flexible approach to telehealth visits. Standard work was disseminated (patient instructions, scripting, and workflows). Patients provided a cell phone number to receive a text link when the provider was ready to see them. Metrics included demographics, volumes, visit rates, percentage seen early/late, time savings, and satisfaction surveys. RESULTS: Over 2.5 months, 22 patients were scheduled. Of those arriving, 76% were "Tele-Untethered" and 24% were "Standard Telemedicine." Text-for-video link was used for 94% of Tele-Untethered. Fifty-five percent were seen early. There was a 55-minute-per-session time savings. CONCLUSION: This UCSD Tele-Untethered pilot benefitted patients by allowing scheduling flexibility while not being tied to a "virtual waiting room." It benefited providers as it allowed them to see patients in order/not tied to exact times, improved throughput, and saved time. Even modest time savings for busy providers, coupled with Lean workflows, can provide critical value. High Tele-Untethered uptake and use of verbal check-in highlight that patients expect flexibility and ease of use. As our initial UCSD Tele-Untethered successes included patient flexibility and time savings for patients and providers, it can serve as a model as enterprises strive for optimal care and improved satisfaction. Expansion to other clinic settings is underway with a mantra of "UCSD Tele-Untethered: Your provider can see you now."