Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI.

Generative Artificial Intelligence is set to revolutionize healthcare delivery by transforming traditional patient care into a more personalized, efficient, and proactive process. Chatbots, serving as interactive conversational models, will probably drive this patient-centered transformation in healthcare. Through the provision of various services, including diagnosis, personalized lifestyle recommendations, dynamic scheduling of follow-ups, and mental health support, the objective is to substantially augment patient health outcomes, all the while mitigating the workload burden on healthcare providers. The life-critical nature of healthcare applications necessitates establishing a unified and comprehensive set of evaluation metrics for conversational models. Existing evaluation metrics proposed for various generic large language models (LLMs) demonstrate a lack of comprehension regarding medical and health concepts and their significance in promoting patients' well-being. Moreover, these metrics neglect pivotal user-centered aspects, including trust-building, ethics, personalization, empathy, user comprehension, and emotional support. The purpose of this paper is to explore state-of-the-art LLM-based evaluation metrics that are specifically applicable to the assessment of interactive conversational models in healthcare. Subsequently, we present a comprehensive set of evaluation metrics designed to thoroughly assess the performance of healthcare chatbots from an end-user perspective. These metrics encompass an evaluation of language processing abilities, impact on real-world clinical tasks, and effectiveness in user-interactive conversations. Finally, we engage in a discussion concerning the challenges associated with defining and implementing these metrics, with particular emphasis on confounding factors such as the target audience, evaluation methods, and prompt techniques involved in the evaluation process.

Voices of Musicians: Virtual Live Bedside Music Concerts in Inpatient Care.

The COVID-19 pandemic presented unprecedented challenges to patients, family members, and healthcare staff that resulted in increased stress and isolation and decreased quality of life. We evaluate the impact of a novel virtual concert program, the Vital Sounds Initiative (VSI) of Project: Music Heals Us (PMHU), which began at the beginning of the pandemic to combat patient isolation and provide employment to professional musicians. Using a qualitative analysis of VSI data, we examined post-concert written responses by musicians. These responses were coded by independent coders via inductive coding and thematic analysis. Between 7 April 2020 and 20 July 2022, 192 musicians played 2203 h of music for 11,222 audience members in 39 care facilities nationwide. A total of 114 musicians submitted a total of 658 responses. Three main themes (with corresponding subthemes) arose: (1) Patient Experience; (2) Musician Experience; (3) Caregiver (family or staff) Experience. The responses offered valuable insight into the overwhelmingly positive aspects of the virtual concerts. Overall, we found that VSI favorably impacts individuals at every level, including the patients, musician, and caregivers. These findings provide preliminary evidence for the benefits of virtual music concerts. Upscaling similar virtual music interventions/programs should be considered.

Acoustic resonance therapy is safe and effective for the treatment of nasal congestion in rhinitis: A randomized sham-controlled trial.

BACKGROUND: Acoustic resonance therapy (ART) is a novel vibrational treatment that delivers patient-specific resonant frequency acoustic energy to the sinonasal cavities. In a pilot study, ART was effective for the acute treatment of nasal congestion. We conducted a sham-controlled randomized trial to validate the efficacy of ART when administered daily for 2 weeks. METHODS: A total of 52 adult patients were enrolled in a multi-center, randomized, double-blinded, sham-controlled, interventional study evaluating ART administered by a vibrational headband. Patients received either active treatment or a non-therapeutic sham treatment twice daily over 2 weeks. Clinical endpoints were the average change in nasal congestion sub-score of the Total Nasal Symptom Score (TNSS) and the average change in composite TNSS. RESULTS: ART resulted in a significantly greater mean change in the nasal congestion sub-score compared to sham (-0.87 [95% confidence interval [CI] -1.11, -0.62] vs. -0.44 [95% CI -0.64, -0.23], p = 0.008). ART also resulted in a significantly greater reduction in the composite TNSS versus sham, (-2.85 [95% CI -3.85, -1.85], vs. -1.32 [95% CI -2.27, -0.36], p = 0.027). The response rate, determined by a nasal congestion sub-score minimal clinically important difference of 0.23, was 80.8% for ART and 46.2% for sham, with an adjusted risk ratio of 1.95 (95% CI 1.26, 3.02, p = 0.003) in favor of ART. Safety endpoints showed no adverse events. CONCLUSION: ART is a safe and effective non-pharmacologic alternative for the treatment of nasal congestion.

Design and development of a digital shared decision-making tool for stroke prevention in atrial fibrillation.

Background: Shared decision-making (SDM) is an approach in which patients and clinicians act as partners in making medical decisions. Patients receive the information needed to decide and are encouraged to balance risks, benefits, and preferences. Informative materials are vital to SDM. Atrial fibrillation (AF) is the most common cardiac arrhythmia and responsible for 10% of ischemic strokes, however 1/3 of patients are not on appropriate anticoagulation. Decision sharing may facilitate treatment acceptance, improving outcomes. Aims: To develop a framework of the components needed to create novel SDM tools and to provide practical examples through a case-study of stroke prevention in AF. Methods: We analyze the design values of a web-based SDM tool created to better inform AF patients about anticoagulation. The tool was developed in partnership with patient advocates, multi-disciplinary investigators, and private design firms. It was refined through iterative, recursive testing in patients with AF. Its effectiveness is being evaluated in a multisite clinical trial led by Stanford University and sponsored by the American Heart Association. Findings: The main components considered when creating the Stanford AFib tool included: design and software; content identification; information delivery; inclusive communication, user engagement; patient feedback; clinician experience; and anticipation of implementation and dissemination. We also highlight the ethical principles underlying SDM; matters of diversity and inclusion, linguistic variety, accessibility, and health literacy. The Stanford AFib Guide patient tool is available at: https://afibguide.com and the clinician tool at https://afibguide.com/clinician. Conclusion: Attention to a range of vital development and design factors can facilitate tool adoption and information acquisition by diverse cultural, educational, and socioeconomic subpopulations. With thoughtful design, digital tools may decrease decision regret and improve treatment outcomes across many decision-making situations in healthcare.

Randomized Clinical Trial to Evaluate an Atrial Fibrillation Stroke Prevention Shared Decision-Making Pathway.

Background Oral anticoagulation reduces stroke and disability in atrial fibrillation (AF) but is underused. We evaluated the effects of a novel patient-clinician shared decision-making (SDM) tool in reducing oral anticoagulation patient's decisional conflict as compared with usual care. Methods and Results We designed and evaluated a new digital decision aid in a multicenter, randomized, comparative effectiveness trial, ENHANCE-AF (Engaging Patients to Help Achieve Increased Patient Choice and Engagement for AF Stroke Prevention). The digital AF shared decision-making toolkit was developed using patient-centered design with clear health communication principles (eg, meaningful images, limited text). Available in English and Spanish, the toolkit included the following: (1) a brief animated video; (2) interactive questions with answers; (3) a quiz to check on understanding; (4) a worksheet to be used by the patient during the encounter; and (5) an online guide for clinicians. The study population included English or Spanish speakers with nonvalvular AF and a CHA2DS2-VASc stroke score ≥1 for men or ≥2 for women. Participants were randomized in a 1:1 ratio to either usual care or the shared decision-making toolkit. The primary end point was the validated 16-item Decision Conflict Scale at 1 month. Secondary outcomes included Decision Conflict Scale at 6 months and the 10-item Decision Regret Scale at 1 and 6 months as well as a weighted average of Mann-Whitney U-statistics for both the Decision Conflict Scale and the Decision Regret Scale. A total of 1001 participants were enrolled and followed at 5 different sites in the United States between December 18, 2019, and August 17, 2022. The mean patient age was 69±10 years (40% women, 16.9% Black, 4.5% Hispanic, 3.6% Asian), and 50% of participants had CHA2DS2-VASc scores ≥3 (men) or ≥4 (women). The primary end point at 1 month showed a clinically meaningful reduction in decisional conflict: a 7-point difference in median scores between the 2 arms (16.4 versus 9.4; Mann-Whitney U-statistics=0.550; P=0.007). For the secondary end point of 1-month Decision Regret Scale, the difference in median scores between arms was 5 points in the direction of less decisional regret (P=0.078). The treatment effects lessened over time: at 6 months the difference in medians was 4.7 points for Decision Conflict Scale (P=0.060) and 0 points for Decision Regret Scale (P=0.35). Conclusions Implementation of a novel shared decision-making toolkit (afibguide.com; afibguide.com/clinician) achieved significantly lower decisional conflict compared with usual care in patients with AF. Registration URL: https://www.clinicaltrials.gov; Unique identifier: NCT04096781.

Diagnostic journeys: characterization of patients and diagnostic outcomes from an academic second opinion clinic.

OBJECTIVES: Diagnostic programs and second opinion clinics have grown and evolved in the recent years to help patients with rare, puzzling, and complex conditions who often suffer prolonged diagnostic journeys, but there is a paucity of literature on the clinical characteristics of these patients and the efficacy of these diagnostic programs. This study aims to characterize the diagnostic journey, case features, and diagnostic outcomes of patients referred to a team-based second opinion clinic at Stanford. METHODS: Retrospective chart review was performed for 237 patients evaluated for diagnostic second opinion in the Stanford Consultative Medicine Clinic over a 5 year period. Descriptive case features and diagnostic outcomes were assessed, and correlation between the two was analyzed. RESULTS: Sixty-three percent of our patients were women. 49% of patients had a potential precipitating event within about a month prior to the start of their illness, such as medication change, infection, or medical procedure. A single clear diagnosis was determined in 33% of cases, whereas the remaining cases were assessed to have multifactorial contributors/diagnoses (20%) or remained unclear despite extensive evaluation (47%). Shorter duration of illness, fewer prior specialties seen, and single chief symptom were associated with higher likelihood of achieving a single clear diagnosis. CONCLUSIONS: A single-site academic consultative service can offer additional diagnostic insights for about half of all patients evaluated for puzzling conditions. Better understanding of the clinical patterns and patient experiences gained from this study helps inform strategies to shorten their diagnostic odysseys.

Training Internal Medicine Residents in Difficult Diagnosis: A Novel Diagnostic Second Opinion Clinic Experience.

Background: In primary care clinics, time constraints and lack of exposure to highly complex cases may limit the breadth and depth of learning for internal medicine residents. To address these issues, we piloted a novel experience for residents to evaluate patients with puzzling symptoms referred by another clinician. Objective: To increase internal medicine residents' exposure to patients with perplexing presentations and foster a team-based approach to solving diagnostically challenging cases. Methods: During the academic year 2020-2021, residents participating in their 2-week primary care "block" rotation were given protected time to evaluate 1-2 patients from the Stanford Consultative Medicine clinic, an internist-led diagnostic second opinion service, and present their patients at the case conference. We assessed the educational value of the program with resident surveys including 5-point Lickert scale and open-ended questions. Results: 21 residents participated in the pilot with a survey response rate of 66.6% (14/21). Both the educational value and overall quality of the experience were rated as 4.8 out of 5 (SD 0.4, range 4-5; 1:"very poor"; 5:"excellent"). Residents learned about new diagnostic tools as well as how to approach complex presentations and diagnostic dilemmas. Residents valued the increased time devoted to patient care, the team-based approach to tackling difficult cases, and the intellectual challenge of these cases. Barriers to implementation include patient case volume, time, and faculty engagement. Conclusions: Evaluation of diagnostically challenging cases in a structured format is a highly valuable experience that offers a framework to enhance outpatient training in internal medicine.

The ENHANCE-AF clinical trial to evaluate an atrial fibrillation shared decision-making pathway: Rationale and study design.

INTRODUCTION: Shared decision making (SDM) may result in treatment plans that best reflect the goals and wishes of patients, increasing patient satisfaction with the decision-making process. There is a knowledge gap to support the use of decision aids in SDM for anticoagulation therapy in patients with atrial fibrillation (AF). We describe the development and testing of a new decision aid, including a multicenter, randomized, controlled, 2-arm, open-label ENHANCE-AF clinical trial (Engaging Patients to Help Achieve Increased Patient Choice and Engagement for AF Stroke Prevention) to evaluate its effectiveness in 1,200 participants. METHODS: Participants will be randomized to either usual care or to a SDM pathway incorporating a digital tool designed to simplify the complex concepts surrounding AF in conjunction with a clinician tool and a non-clinician navigator to guide the participants through each step of the tool. The participant-determined primary outcome for this study is the Decisional Conflict Scale, measured at 1 month after the index visit during which a decision was made regarding anticoagulation use. Secondary outcomes at both 1 and 6 months will include other decision making related scales as well as participant and clinician satisfaction, oral anticoagulation adherence, and a composite rate of major bleeding, death, stroke, or transient ischemic attack. The study will be conducted at four sites selected for their ability to enroll participants of varying racial and ethnic backgrounds, health literacy, and language skills. Participants will be followed in the study for 6 months. CONCLUSIONS: The results of the ENHANCE-AF trial will determine whether a decision aid facilitates high quality shared decision making in anticoagulation discussions for stroke reduction in AF. An improved shared decision-making experience may allow patients to make decisions better aligned with their personal values and preferences, while improving overall AF care.

Disaggregated mortality from gastrointestinal cancers in Asian Americans: Analysis of United States death records.

Asian Americans (AAs) are heterogeneous, and aggregation of diverse AA populations in national reporting may mask high-risk groups. Gastrointestinal (GI) cancers constitute one-third of global cancer mortality, and an improved understanding of GI cancer mortality by disaggregated AA subgroups may inform future primary and secondary prevention strategies. Using national mortality records from the United States from 2003 to 2017, we report age-standardized mortality rates, standardized mortality ratios and annual percent change trends from GI cancers (esophageal, gastric, colorectal, liver and pancreatic) for the six largest AA subgroups (Asian Indians, Chinese, Filipinos, Japanese, Koreans and Vietnamese). Non-Hispanic Whites (NHWs) are used as the reference population. We found that mortality from GI cancers demonstrated nearly 3-fold difference between the highest (Koreans, 61 per 100 000 person-years) and lowest (Asian Indians, 21 per 100 000 person-years) subgroups. The distribution of GI cancer mortality demonstrates high variability between subgroups, with Korean Americans demonstrating high mortality from gastric cancer (16 per 100 000), and Vietnamese Americans demonstrating high mortality from liver cancer (19 per 100 000). Divergent temporal trends emerged, such as increasing liver cancer burden in Vietnamese Americans, which exacerbated existing mortality differences. There exist striking differences in the mortality burden of GI cancers by disaggregated AA subgroups. These data highlight the need for disaggregated data reporting, and the importance of race-specific and personalized strategies of screening and prevention.

Achieving Equity in Asian American Healthcare: Critical Issues and Solutions.

Equity is a fundamental goal of the US health care system. Asians comprise 6% of the US population, and 60% of the world's population. Less than 1% of National Institutes of Health funding is directed toward Asian health. Asian health outcomes are often worse than non-Hispanic Whites (NHWS) in America. Increasing federal and foundation resources and funding toward Asian health research could illuminate these risks and develop precision interventions to reduce known health disparities. When disaggregated by race/ethnicity, Asian health risks are even more apparent. Here, we discuss critical health outcome differences between the six major Asian American subgroups (Asian Indian, Chinese, Filipino, Japanese, Korean, and Vietnamese) comprising 80% of the US Asian population, highlighting demographic, pharmacologic, disease prevalence, and mortality outcomes. We then outline seven critical issues contributing to Asian American health disparities, including aggregated Asian health data, undersampling, invalid extrapolations, underrepresentation in clinical trials, lack of funding and awareness of disparities, and the model minority myth. Building on the successes of national public health initiatives, we propose nine leverage points to improve Asian American health including the following: obtaining disaggregated Asian health data, improved Asian health research (oversampling Asians, improving clinical trial participation, and increasing research funding), stakeholder collaboration (national and with Asian nations), community engagement, providing culturally precise health care, and expansion of the Asian American research ecosystem. Achieving health equity takes deliberate practice and does not occur by accident. By addressing critical issues that perpetuate Asian health disparities, we grow closer to understanding how to effectively improve Asian health and build a nationally unified mindset toward action that emphasizes equitable care for all.

Machine Learning Clustering for Blood Pressure Variability Applied to Systolic Blood Pressure Intervention Trial (SPRINT) and the Hong Kong Community Cohort.

Visit-to-visit blood pressure variability (BPV) has been shown to be a predictor of cardiovascular disease. We aimed to classify the BPV levels using different machine learning algorithms. Visit-to-visit blood pressure readings were extracted from the SPRINT study in the United States and eHealth cohort in Hong Kong (HK cohort). Patients were clustered into low, medium, and high BPV levels with the traditional quantile clustering and 5 machine learning algorithms including K-means. Clustering methods were assessed by Stability Index. Similarities were assessed by Davies-Bouldin Index and Silhouette Index. Cox proportional hazard regression models were fitted to compare the risk of myocardial infarction, stroke, and heart failure. A total of 8133 participants had average blood pressure measurement 14.7 times in 3.28 years in SPRINT and 1094 participants who had average blood pressure measurement 165.4 times in 1.37 years in HK cohort. Quantile clustering assigned one-third participants as high BPV level, but machine learning methods only assigned 10% to 27%. Quantile clustering is the most stable method (stability index: 0.982 in the SPRINT and 0.948 in the HK cohort) with some levels of clustering similarities (Davies-Bouldin Index: 0.752 and 0.764, respectively). K-means clustering is the most stable across the machine learning algorithms (stability index: 0.975 and 0.911, respectively) with the lowest clustering similarities (Davies-Bouldin Index: 0.653 and 0.680, respectively). One out of 7 in the population was classified with high BPV level, who showed to have higher risk of stroke and heart failure. Machine learning methods can improve BPV classification for better prediction of cardiovascular diseases.

Pharmacogenomics in Asian Subpopulations and Impacts on Commonly Prescribed Medications.

Asians as a group comprise > 60% the world's population. There is an incredible amount of diversity in Asian and admixed populations that has not been addressed in a pharmacogenetic context. The known pharmacogenetic differences in Asian subgroups generally represent previously known variants that are present at much lower or higher frequencies in Asians compared with other populations. In this review we summarize the main drugs and known genes that appear to have differences in their pharmacogenetic properties in certain Asian populations. Evidence-based guidelines and summary statistics from the US Food and Drug Administration and the Clinical Pharmacogenetics Implementation Consortium were analyzed for ethnic differences in outcomes. Implicated drugs included commonly prescribed drugs such as warfarin, clopidogrel, carbamazepine, and allopurinol. The majority of these associations are due to Asians more commonly being poor metabolizers of cytochrome P450 (CYP) 2C19 and carriers of the human leukocyte antigen (HLA)-B*15:02 allele. The relative risk increase was shown to vary between genes and drugs, but could be > 100-fold higher in Asians. Specifically, there was a 172-fold increased risk of Stevens-Johnson syndrome and toxic epidermal necrolysis with carbamazepine use among HLA-B*15:02 carriers. The effects ranged from relatively benign reactions such as reduced drug efficacy to severe cutaneous skin reactions. These reactions are severe and prevalent enough to warrant pharmacogenetic testing and appropriate changes in dose and medication choice for at-risk populations. Further studies should be done on Asian cohorts to more fully understand pharmacogenetic variants in these populations and to clarify how such differences may influence drug response.

Cryosurgical posterior nasal tissue ablation for the treatment of rhinitis.

BACKGROUND: Endoscopic posterior nasal nerve (PNN) resection has been described as an efficacious surgical treatment of allergic and nonallergic rhinitis, but the requirement for surgery under general anesthesia has limited its acceptance. We report the first series of patients treated for chronic rhinitis using a novel device designed for office-based cryosurgical ablation of the PNN. METHODS: Twenty-seven patients with chronic rhinorrhea and/or nasal congestion for >3 months were recruited (allergic or nonallergic rhinitis), with minimum rhinorrhea and/or congestion subscores of 2 as part of the Total Nasal Symptom Score [TNSS]). Under local anesthesia, the cryotherapy device was applied endoscopically to the posterior middle meatus and was used to freeze the PNN region bilaterally. Patients were followed up after 7, 30, 90, 180, and 365 days to assess TNSS. RESULTS: The procedure was successfully completed in 100% of patients, with no complications; 74% reported no or mild discomfort by the first postprocedure day. TNSS was reduced significantly at 30 days (mean ± standard deviation: 6.2 ± 0.5 at baseline, 2.6 ± 0.3 at 30 days, n = 27, p < 0.001), with continued reduction at 90 (2.7 ± 0.4, n = 24, p < 0.001), 180 (2.3 ± 0.5, n = 21, p < 0.001), and 365 days (1.9 ± 0.3, n = 15, p < 0.001). Both rhinorrhea and congestion subscores decreased significantly at 30, 90, 180, and 365 days compared to baseline (p < 0.001). Allergic and nonallergic subcohorts both appeared to benefit from treatment. CONCLUSION: Office-based cryotherapy of the PNN region is safe and well tolerated. Symptom scores were significantly decreased by 7 days postprocedure and remained lower at 30, 90, 180, and 365 days.