A Comparison of Google and ChatGPT for Automatic Generation of Health-related Multiple-choice Questions.

Critical to producing accessible content is an understanding of what characteristics affect understanding and comprehension. To answer this question, we are producing a large corpus of health-related texts with associated questions that can be read or listened to by study participants to measure the difficulty of the underlying content, which can later be used to better understand text difficulty and user comprehension. In this paper, we examine methods for automatically generating multiple-choice questions using Google's related questions and ChatGPT. Overall, we find both algorithms generate reasonable questions that are complementary; ChatGPT questions are more similar to the snippet while Google related-search questions have more lexical variation.

Crowdsourcing with Enhanced Data Quality Assurance: An Efficient Approach to Mitigate Resource Scarcity Challenges in Training Large Language Models for Healthcare.

Large Language Models (LLMs) have demonstrated immense potential in artificial intelligence across various domains, including healthcare. However, their efficacy is hindered by the need for high-quality labeled data, which is often expensive and time-consuming to create, particularly in low-resource domains like healthcare. To address these challenges, we propose a crowdsourcing (CS) framework enriched with quality control measures at the pre-, real-time-, and post-data gathering stages. Our study evaluated the effectiveness of enhancing data quality through its impact on LLMs (Bio-BERT) for predicting autism-related symptoms. The results show that real-time quality control improves data quality by 19% compared to pre-quality control. Fine-tuning Bio-BERT using crowdsourced data generally increased recall compared to the Bio-BERT baseline but lowered precision. Our findings highlighted the potential of crowdsourcing and quality control in resource-constrained environments and offered insights into optimizing healthcare LLMs for informed decision-making and improved patient care.

Utilizing Large Language Models to Generate Synthetic Data to Increase the Performance of BERT-Based Neural Networks.

An important problem impacting healthcare is the lack of available experts. Machine learning (ML) models may help resolve this by aiding in screening and diagnosing patients. However, creating large, representative datasets to train models is expensive. We evaluated large language models (LLMs) for data creation. Using Autism Spectrum Disorders (ASD), we prompted GPT-3.5 and GPT-4 to generate 4,200 synthetic examples of behaviors to augment existing medical observations. Our goal is to label behaviors corresponding to autism criteria and improve model accuracy with synthetic training data. We used a BERT classifier pretrained on biomedical literature to assess differences in performance between models. A random sample (N=140) from the LLM-generated data was also evaluated by a clinician and found to contain 83% correct behavioral example-label pairs. Augmenting the dataset increased recall by 13% but decreased precision by 16%. Future work will investigate how different synthetic data characteristics affect ML outcomes.

Text and Audio Simplification: Human vs. ChatGPT.

Text and audio simplification to increase information comprehension are important in healthcare. With the introduction of ChatGPT, evaluation of its simplification performance is needed. We provide a systematic comparison of human and ChatGPT simplified texts using fourteen metrics indicative of text difficulty. We briefly introduce our online editor where these simplification tools, including ChatGPT, are available. We scored twelve corpora using our metrics: six text, one audio, and five ChatGPT simplified corpora (using five different prompts). We then compare these corpora with texts simplified and verified in a prior user study. Finally, a medical domain expert evaluated the user study texts and five, new ChatGPT simplified versions. We found that simple corpora show higher similarity with the human simplified texts. ChatGPT simplification moves metrics in the right direction. The medical domain expert's evaluation showed a preference for the ChatGPT style, but the text itself was rated lower for content retention.

Effects of Added Emphasis and Pause in Audio Delivery of Health Information.

Health literacy is crucial to supporting good health and is a major national goal. Audio delivery of information is becoming more popular for informing oneself. In this study, we evaluate the effect of audio enhancements in the form of information emphasis and pauses with health texts of varying difficulty and we measure health information comprehension and retention. We produced audio snippets from difficult and easy text and conducted the study on Amazon Mechanical Turk (AMT). Our findings suggest that emphasis matters for both information comprehension and retention. When there is no added pause, emphasizing significant information can lower the perceived difficulty for difficult and easy texts. Comprehension is higher (54%) with correctly placed emphasis for the difficult texts compared to not adding emphasis (50%). Adding a pause lowers perceived difficulty and can improve retention but adversely affects information comprehension.

Transparent deep learning to identify autism spectrum disorders (ASD) in EHR using clinical notes.

OBJECTIVE: Machine learning (ML) is increasingly employed to diagnose medical conditions, with algorithms trained to assign a single label using a black-box approach. We created an ML approach using deep learning that generates outcomes that are transparent and in line with clinical, diagnostic rules. We demonstrate our approach for autism spectrum disorders (ASD), a neurodevelopmental condition with increasing prevalence. METHODS: We use unstructured data from the Centers for Disease Control and Prevention (CDC) surveillance records labeled by a CDC-trained clinician with ASD A1-3 and B1-4 criterion labels per sentence and with ASD cases labels per record using Diagnostic and Statistical Manual of Mental Disorders (DSM5) rules. One rule-based and three deep ML algorithms and six ensembles were compared and evaluated using a test set with 6773 sentences (N = 35 cases) set aside in advance. Criterion and case labeling were evaluated for each ML algorithm and ensemble. Case labeling outcomes were compared also with seven traditional tests. RESULTS: Performance for criterion labeling was highest for the hybrid BiLSTM ML model. The best case labeling was achieved by an ensemble of two BiLSTM ML models using a majority vote. It achieved 100% precision (or PPV), 83% recall (or sensitivity), 100% specificity, 91% accuracy, and 0.91 F-measure. A comparison with existing diagnostic tests shows that our best ensemble was more accurate overall. CONCLUSIONS: Transparent ML is achievable even with small datasets. By focusing on intermediate steps, deep ML can provide transparent decisions. By leveraging data redundancies, ML errors at the intermediate level have a low impact on final outcomes.

Retrieval augmentation of large language models for lay language generation.

The complex linguistic structures and specialized terminology of expert-authored content limit the accessibility of biomedical literature to the general public. Automated methods have the potential to render this literature more interpretable to readers with different educational backgrounds. Prior work has framed such lay language generation as a summarization or simplification task. However, adapting biomedical text for the lay public includes the additional and distinct task of background explanation: adding external content in the form of definitions, motivation, or examples to enhance comprehensibility. This task is especially challenging because the source document may not include the required background knowledge. Furthermore, background explanation capabilities have yet to be formally evaluated, and little is known about how best to enhance them. To address this problem, we introduce Retrieval-Augmented Lay Language (RALL) generation, which intuitively fits the need for external knowledge beyond that in expert-authored source documents. In addition, we introduce CELLS, the largest (63k pairs) and broadest-ranging (12 journals) parallel corpus for lay language generation. To evaluate RALL, we augmented state-of-the-art text generation models with information retrieval of either term definitions from the UMLS and Wikipedia, or embeddings of explanations from Wikipedia documents. Of these, embedding-based RALL models improved summary quality and simplicity while maintaining factual correctness, suggesting that Wikipedia is a helpful source for background explanation in this context. We also evaluated the ability of both an open-source Large Language Model (Llama 2) and a closed-source Large Language Model (GPT-4) in background explanation, with and without retrieval augmentation. Results indicate that these LLMs can generate simplified content, but that the summary quality is not ideal. Taken together, this work presents the first comprehensive study of background explanation for lay language generation, paving the path for disseminating scientific knowledge to a broader audience. Our code and data are publicly available at: https://github.com/LinguisticAnomalies/pls_retrieval.

Audio delivery of health information: An NLP study of information difficulty and bias in listeners.

Health literacy is the ability to understand, process, and obtain health information and make suitable decisions about health care [3]. Traditionally, text has been the main medium for delivering health information. However, virtual assistants are gaining popularity in this digital era; and people increasingly rely on audio and smart speakers for health information. We aim to identify audio/text features that contribute to the difficulty of the information delivered over audio. We are creating a health-related audio corpus. We selected text snippets and calculated seven text features. Then, we converted the text snippets to audio snippets. In a pilot study with Amazon Mechanical Turk (AMT) workers, we measured the perceived and actual difficulty of the audio using the response of multiple choice and free recall questions. We collected demographic information as well as bias about doctors' gender, task preference, and health information preference. Thirteen workers completed thirty audio snippets and related questions. We found a strong correlation between text features lexical chain, and the dependent variables, and multiple choice response, percentage of matching word, percentage of similar word, cosine similarity, and time taken (in seconds). In addition, doctors were generally perceived to be more competent than warm. How warm workers perceive male doctors correlated significantly with perceived difficulty.

Psycholinguistic Markers of COVID-19 Conspiracy Tweets and Predictors of Tweet Dissemination.

The adoption of conspiracy theories about COVID-19 has been fairly widespread among the general public and associated with the rejection of self-protective behaviors. Despite their significance, however, a gap remains in our understanding of the underlying characteristics of messages used to disseminate COVID-19 conspiracies. We used the construct of resonance as a framework to examine a sample of more than 1.8 million posts to Twitter about COVID-19 made between April and June 2020. Our analyses focused on the psycholinguistic properties that distinguish conspiracy theory tweets from other COVID-19 topics and predict their spread. COVID-19 conspiracy tweets were distinct and most likely to resonate when they provided explanations and expressed negative emotions. The results highlight the sensemaking functions served by conspiracy tweets in response to the profound upheaval caused by the pandemic.

Improving the Quality of Suggestions for Medical Text Simplification Tools.

Text continues to be an important medium for communicating health-related information. We have built a text simplification tool that gives concrete suggestions on how to simplify health and medical texts. An important component of the tool identifies difficult words and suggests simpler synonyms based on pre-existing resources (WordNet and UMLS). These candidate substitutions are not always appropriate in all contexts. In this paper, we introduce a filtering algorithm that utilizes semantic similarity based on word embeddings to determine if the candidate substitution is appropriate in the context of the text. We provide an analysis of our approach on a new dataset of 788 labeled substitution examples. The filtering algorithm is particularly helpful at removing obvious examples and can improve the precision by 3% at a recall level of 95%.

Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty.

Objective: Simplifying healthcare text to improve understanding is difficult but critical to improve health literacy. Unfortunately, few tools exist that have been shown objectively to improve text and understanding. We developed an online editor that integrates simplification algorithms that suggest concrete simplifications, all of which have been shown individually to affect text difficulty. Materials and Methods: The editor was used by a health educator at a local community health center to simplify 4 texts. A controlled experiment was conducted with community center members to measure perceived and actual difficulty of the original and simplified texts. Perceived difficulty was measured using a Likert scale; actual difficulty with multiple-choice questions and with free recall of information evaluated by the educator and 2 sets of automated metrics. Results: The results show that perceived difficulty improved with simplification. Several multiple-choice questions, measuring actual difficulty, were answered more correctly with the simplified text. Free recall of information showed no improvement based on the educator evaluation but was better for simplified texts when measured with automated metrics. Two follow-up analyses showed that self-reported education level and the amount of English spoken at home positively correlated with question accuracy for original texts and the effect disappears with simplified text. Discussion: Simplifying text is difficult and the results are subtle. However, using a variety of different metrics helps quantify the effects of changes. Conclusion: Text simplification can be supported by algorithmic tools. Without requiring tool training or linguistic knowledge, our simplification editor helped simplify healthcare related texts.

Cognitive and Cultural Factors That Affect General Vaccination and COVID-19 Vaccination Attitudes.

The development of COVID-19 vaccines is a major scientific accomplishment that has armed communities worldwide with powerful epidemic control tools. Yet, COVID-19 vaccination efforts in the US have been marred by persistent vaccine hesitancy. We used survey methodology to explore the impact of different cognitive and cultural factors on the public's general vaccination attitudes, attitudes towards COVID-19 vaccines, and COVID-19 vaccination status. The factors include information literacy, science literacy, attitudes towards science, interpersonal trust, public health trust, political ideology, and religiosity. The analysis suggests that attitudes towards vaccination are influenced by a multitude of factors that operate in a complex manner. General vaccination attitude was most affected by attitudes towards science and public health trust and to a lesser degree by information literacy, science literacy, and religiosity. Attitudes towards COVID-19 vaccines were most affected by public health trust and to a lesser extent by general trust, ideology and attitudes towards science. Vaccination status was most influenced by public health trust. Possible mediating effects of correlated variables in the model need to be further explored. The study underscores the importance of understanding the relationship between public health trust, literacies, and sociocultural factors.

Factors Influencing Willingness to Share Health Misinformation Videos on the Internet: Web-Based Survey.

BACKGROUND: The rapidly evolving digital environment of the social media era has increased the reach of both quality health information and misinformation. Platforms such as YouTube enable easy sharing of attractive, if not always evidence-based, videos with large personal networks and the public. Although much research has focused on characterizing health misinformation on the internet, it has not sufficiently focused on describing and measuring individuals' information competencies that build resilience. OBJECTIVE: This study aims to assess individuals' willingness to share a non-evidence-based YouTube video about strengthening the immune system; to describe types of evidence that individuals view as supportive of the claim by the video; and to relate information-sharing behavior to several information competencies, namely, information literacy, science literacy, knowledge of the immune system, interpersonal trust, and trust in health authority. METHODS: A web-based survey methodology with 150 individuals across the United States was used. Participants were asked to watch a YouTube excerpt from a morning TV show featuring a wellness pharmacy representative promoting an immunity-boosting dietary supplement produced by his company; answer questions about the video and report whether they would share it with a cousin who was frequently sick; and complete instruments pertaining to the information competencies outlined in the objectives. RESULTS: Most participants (105/150, 70%) said that they would share the video with their cousins. Their confidence in the supplement would be further boosted by a friend's recommendations, positive reviews on a crowdsourcing website, and statements of uncited effectiveness studies on the producer's website. Although all information literacy competencies analyzed in this study had a statistically significant relationship with the outcome, each competency was also highly correlated with the others. Information literacy and interpersonal trust independently predicted the largest amount of variance in the intention to share the video (17% and 16%, respectively). Interpersonal trust was negatively related to the willingness to share the video. Science literacy explained 7% of the variance. CONCLUSIONS: People are vulnerable to web-based misinformation and are likely to propagate it on the internet. Information literacy and science literacy are associated with less vulnerability to misinformation and a lower propensity to spread it. Of the two, information literacy holds a greater promise as an intervention target. Understanding the role of different kinds of trust in information sharing merits further research.

Comparison of women and men in biomedical informatics scientific dissemination: retrospective observational case study of the AMIA Annual Symposium: 2017-2020.

OBJECTIVE: Although the representation of women in science has improved, women remain underrepresented in scientific publications. This study compares women and men in scholarly dissemination through the AMIA Annual Symposium. MATERIALS AND METHODS: Through a retrospective observational study, we analyzed 2017-2020 AMIA submissions for differences in panels, papers, podium abstracts, posters, workshops, and awards for men compared with women. We assigned a label of woman or man to authors and reviewers using Genderize.io, and then compared submission and acceptance rates, performed regression analyses to evaluate the impact of the assumed gender, and performed sentiment analysis of reviewer comments. RESULTS: Of the 4687 submissions for which Genderize.io could predict man or woman based on first name, 40% were led by women and 60% were led by men. The acceptance rate was smilar. Although submission and acceptance rates for women increased over the 4 years, women-led podium abstracts, panels, and workshops were underrepresented. Men reviewers increased the odds of rejection. Men provided longer reviews and lower reviewer scores, but women provided reviews that had more positive words. DISCUSSION: Overall, our findings reflect significant gains for women in the 4 years of conference data analyzed. However, there remain opportunities to improve representation of women in workshop submissions, panel and podium abstract speakers, and balanced peer reviews. Future analyses could be strengthened by collecting gender directly from authors, including diverse genders such as non-binary. CONCLUSION: We found little evidence of major bias against women in submission, acceptance, and awards associated with the AMIA Annual Symposium from 2017 to 2020. Our study is unique because of the analysis of both authors and reviewers. The encouraging findings raise awareness of progress and remaining opportunities in biomedical informatics scientific dissemination.

Clinician Practice Patterns That Result in the Diagnosis of Coccidioidomycosis Before or During Hospitalization.

BACKGROUND: Coccidioidomycosis (CM) is common and important within endemic regions, requiring specific testing for diagnosis. Long delays in diagnosis have been ascribed to ambulatory clinicians. However, how their testing practices have impacted patient care has not been systematically unexplored. METHODS: We analyzed practice patterns for CM diagnoses over 3 years within a large Arizona healthcare system, including diagnosis location, patient characteristics, and care-seeking patterns associated with missed diagnosis. RESULTS: For 2043 CM diagnoses, 72.9% were made during hospital admission, 21.7% in ambulatory clinics, 3.2% in emergency units, and only 0.5% in urgent care units. A 40.6% subgroup of hospitalized patients required neither intensive care unit or hospital-requiring procedures, had a median length of stay of only 3 days, but still incurred both substantial costs ($27.0 million) and unnecessary antibiotic administrations. Prior to hospital diagnosis (median of 32 days), 45.1% of patients had 1 or more visits with symptoms consistent with CM. During those visits, 71.3% were not tested for CM. Diagnoses were delayed a median of 27 days. CONCLUSIONS: Lack of testing for CM in ambulatory care settings within a region endemic for CM resulted in a large number of hospital admissions, attendant costs, and unneeded antibacterial drug use, much of which would otherwise be unnecessary. Improving this practice is challenging since many clinicians did not train where CM is common, resulting in significant inertia to change. Determining the best way to retrain clinicians to diagnose CM earlier is an opportunity to explore which strategies might be the most effective.

Incidence and Impact of Missing Functional Elements on Information Comprehension using Audio and Text.

Audio is increasingly used to communicate health information. Initial evaluations have shown it to be an effective means with many features that can be optimized. This study focuses on missing functional elements: words that relate concepts in a sentence but are often excluded for brevity. They are not easily recognizable without linguistics expertise but can be detected algorithmically. Two studies showed that they are common and affect comprehension. A corpus statistics study with medical (Cochrane sentences, N=44,488) and general text (English and Simple English Wikipedia sentences, N=318,056 each) showed that functional elements were missing in 20-30% of sentences. A user study with Cochrane (N=50) and Wikipedia (N=50) paragraphs in text and audio format showed that more missing functional elements increased perceived difficulty of reading text, with the effect less pronounced with audio, and increased actual difficulty of both written and audio information with less information recalled with more missing elements.

Informatics Approaches for Recognition, Management, and Prevention of Occupational Respiratory Disease.

Computer and information systems can improve occupational respiratory disease prevention and surveillance by providing efficient resources for patients, workers, clinicians, and public health practitioners. Advances include interlinking electronic health records, autocoding surveillance data, clinical decision support systems, and social media applications for acquiring and disseminating information. Obstacles to advances include inflexible hierarchical coding schemes, inadequate occupational health electronic health record systems, and inadequate public focus on occupational respiratory disease. Potentially transformative approaches include machine learning, natural language processing, and improved ontologies.

A comparison of text versus audio for information comprehension with future uses for smart speakers.

OBJECTIVE: Audio is increasingly used to access information on the Internet through virtual assistants and smart speakers. Our objective is to evaluate the distribution of health information through audio. MATERIALS AND METHODS: We conducted 2 studies to compare comprehension after reading or listening to information using a new corpus containing short text snippets from Cochrane (N = 50) and Wikipedia (N = 50). In study 1, the snippets were first presented as audio or text followed by a multiple-choice question. Then, the same information was presented as text and the question was repeated in addition to questions about perceived difficulty, severity and the likelihood of encountering the disease. In study 2, the first multiple-choice question was replaced with a free recall question. RESULTS: Study 1 showed that information comprehension is very similar in both presentation modes (53% accuracy for text and 55% for audio). Study 2 showed that information retention is higher with text, but similar comprehension. Both studies show improvement in performance with repeated information presentation. DISCUSSION: Audio presentation of information is effective and the format novel. Performance was slightly lower with audio when asked to repeat information, but comparable to text for answering questions. Additional studies are needed with different types of information and presentation combinations. CONCLUSION: The use of audio to provide health information is a promising field and will become increasingly important with the popularity of smart speakers and virtual assistants, particularly for consumers who do not use computers, for example minority groups, or those with limited sight or motor control.

Algorithmic Generation of Grammar Simplification Rules Using Large Corpora.

There is often a discontinuity between patients' literacy level and educational materials. In response, we are developing an online medical text simplification editor. In this paper, we describe generating grammar simplification rules from a large parallel corpus (N=141,500) containing original sentences and their simplified variants. We algorithmically identified grammatical transformations between sentences (N=26,600) and used distributional characteristics in two corpora to select transformations with the broadest application and the least ambiguity. This resulted in a top set of 146 rules. Two experts evaluated 20 representative rules reflecting 4 characteristics (long/short and weak/strong) each with 5 example sentences. Generally, we found that the rules are helpful for guiding simplification. Using a 5-point Likert scale (5=best), stronger rules scored higher for ease of applying (4.11), overall helpfulness (4.40) and usefulness of examples (4.05). Rule length did not affect the expert scores. The grammar simplification rules are being integrated in our text editor.

Insights from Twitter About Public Perceptions of Asthma, COPD, and Exposures.

OBJECTIVE: The aim of this study was to analyze tweets concerning asthma and chronic obstructive pulmonary disease (COPD). METHODS: Approximately 40,000 tweets containing asthma or COPD were analyzed. Lexical analysis ranked terms and domains of interest, compared COPD with asthma tweets, evaluated co-occurrence of terms within tweets, and assessed differences by source (personal, institutional, or retweet). The frequency of indicator terms relevant to occupational health was determined. RESULTS: Many tweets address community pollution and effects on children, but there is much less interest in work-related factors and occupational regulatory agencies. Environment is considered much more relevant for asthma than COPD. CONCLUSION: Although epidemiologic studies demonstrate a major burden of occupational factors upon both diseases, significantly improved outreach is needed to overcome inadequate public interest. Social media represent a valuable resource for assessing perceptions about work-related disease and potentially discovering new associations.