ABSTRACT
OBJECTIVES: Understanding the strengths and weaknesses of chatbots as a source of patient information is critical for providers in the rising artificial intelligence landscape. This study is the first to quantitatively analyze and compare four of the most used chatbots available regarding treatments of common pathologies in rhinology. METHODS: The treatment of epistaxis, chronic sinusitis, sinus infection, allergic rhinitis, allergies, and nasal polyps was asked to chatbots ChatGPT, ChatGPT Plus, Google Bard, and Microsoft Bing in May 2023. Individual responses were analyzed by reviewers for readability, quality, understandability, and actionability using validated scoring metrics. Accuracy and comprehensiveness were evaluated for each response by two experts in rhinology. RESULTS: ChatGPT, Plus, Bard, and Bing had FRE readability scores of 33.17, 35.93, 46.50, and 46.32, respectively, indicating higher readability for Bard and Bing compared to ChatGPT (p = 0.003, p = 0.008) and Plus (p = 0.025, p = 0.048). ChatGPT, Plus, and Bard had mean DISCERN quality scores of 20.42, 20.89, and 20.61, respectively, which was higher than the score for Bing of 16.97 (p < 0.001). For understandability, ChatGPT and Bing had PEMAT scores of 76.67 and 66.61, respectively, which were lower than both Plus at 92.00 (p < 0.001, p < 0.001) and Bard at 92.67 (p < 0.001, p < 0.001). ChatGPT Plus had an accuracy score of 4.39 which was higher than ChatGPT (3.97, p = 0.118), Bard (3.72, p = 0.002), and Bing (3.19, p < 0.001). CONCLUSION: On aggregate of the tested domains, our results suggest ChatGPT Plus and Google Bard are currently the most patient-friendly chatbots for the treatment of common pathologies in rhinology. LEVEL OF EVIDENCE: N/A Laryngoscope, 2024.
ABSTRACT
OBJECTIVE: To quantitatively compare online patient education materials found using traditional search engines (Google) versus conversational Artificial Intelligence (AI) models (ChatGPT) for benign paroxysmal positional vertigo (BPPV). STUDY DESIGN: The top 30 Google search results for "benign paroxysmal positional vertigo" were compared to the OpenAI conversational AI language model, ChatGPT, responses for 5 common patient questions posed about BPPV in February 2023. Metrics included readability, quality, understandability, and actionability. SETTING: Online information. METHODS: Validated online information metrics including Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE), DISCERN instrument score, and Patient Education Materials Assessment Tool for Printed Materials were analyzed and scored by reviewers. RESULTS: Mean readability scores, FKGL and FRE, for the Google webpages were 10.7 ± 2.6 and 46.5 ± 14.3, respectively. ChatGPT responses had a higher FKGL score of 13.9 ± 2.5 (P < .001) and a lower FRE score of 34.9 ± 11.2 (P = .005), both corresponding to lower readability. The Google webpages had a DISCERN part 2 score of 25.4 ± 7.5 compared to the individual ChatGPT responses with a score of 17.5 ± 3.9 (P = .001), and the combined ChatGPT responses with a score of 25.0 ± 0.9 (P = .928). The average scores of the reviewers for all ChatGPT responses for accuracy were 4.19 ± 0.82 and 4.31 ± 0.67 for currency. CONCLUSION: The results of this study suggest that the information on ChatGPT is more difficult to read, of lower quality, and more difficult to comprehend compared to information on Google searches.
ABSTRACT
OBJECTIVE: To analyze tweets associated with Ménière's disease (MD), including type of users who engage, change in usage patterns, and temporal associations, and to compare the perceptions of the general public with healthcare providers. METHODS: An R-program code, academictwitterR API, was used to query Twitter. All tweets mentioning MD from 2007 to 2021 were retrieved and analyzed. Valence Aware Dictionary and Sentiment Reasoning was used as a model to assess sentiment of tweets. Two reviewers assessed 1,007 tweets for qualitative analysis, identifying the source and the topic of the tweet. RESULTS: A total of 37,402 tweets were analyzed. The number of tweets per user ranged from 1 to 563 (M = 33.7, SD = 91.1). Quantitative analysis showed no temporal or seasonal association; however, tweeting increased when celebrities were diagnosed with MD. Of the 1007 representative tweets analyzed, 60.6% of tweets came from the general public and were largely of negative sentiment focusing on quality of life and support, whereas healthcare providers accounted for 23% of all tweets and focused on treatment/prevention. Tweets by news sources accounted for the remaining 13% of all tweets and were primarily positive in sentiment and focused on awareness. CONCLUSIONS: MD is commonly tweeted about by the general public, with limited input regarding the disease from healthcare providers. Healthcare providers must provide accurate information and awareness regarding MD, especially when awareness is highest, such as when celebrities are diagnosed. LEVEL OF EVIDENCE: Level IV.Indicate IRB or IACUCNot applicable.
Subject(s)
Meniere Disease , Social Media , Humans , Public Opinion , Quality of LifeABSTRACT
OBJECTIVES: To assess the quality, readability, and understandability of posterior glottic stenosis (PGS) information available to patients online. METHODS: The top 50 Google search results for "posterior glottic stenosis" were categorized based on website affiliation and target audience (patient or provider). Readability was assessed using the Flesch-Kincaid Grade Level (FKGL) and the Flesch Reading Ease (FRE) scores. The DISCERN tool was used to assess quality and the Patient Education Assessment Tool for Printed Materials (PEMAT-P) was used to assess understandability and actionability. Simple descriptive statistics were used to analyze the data. RESULTS: 36 of the top 50 results were eligible for scoring. 17% (6 of 36) were classified as patient-focused while 83% (30 of 36) were provider-focused. Patient-focused materials had a higher mean FRE score (36.9) than provider-focused materials (15.5) (P < 0.001). Patient-focused materials had an average reading level of 12.5 compared to 15.8 for provider-focused materials (P < 0.001). There was a significant correlation between overall PEMAT-P and DISCERN (r = 0.63, P < 0.001), PEMAT-P understandability and DISCERN (r = 0.63, P < 0.001) and FRES and FKGL (r = -0.67, P < 0.001). From this, we can infer that higher quality sites are easier to understand but not necessarily tailored to a certain reading level. CONCLUSIONS: Shared decision making in PGS management is crucial as patients must be aware of how treatment modalities affect airway, voice, and swallowing. However, this study shows that patient targeted PGS information is limited, and the readability, quality, and understandability is generally low. We suggest the development of web pages with PGS information tailored for patient education and search optimization to make this information appear earlier in Google search results. Furthermore, future studies should seek to characterize the link between online health information and socioeconomic-based health disparities.
