A Comparison of ChatGPT and Fine-Tuned Open Pre-Trained Transformers (OPT) Against Widely Used Sentiment Analysis Tools: Sentiment Analysis of COVID-19 Survey Data.

BACKGROUND: Health care providers and health-related researchers face significant challenges when applying sentiment analysis tools to health-related free-text survey data. Most state-of-the-art applications were developed in domains such as social media, and their performance in the health care context remains relatively unknown. Moreover, existing studies indicate that these tools often lack accuracy and produce inconsistent results. OBJECTIVE: This study aims to address the lack of comparative analysis on sentiment analysis tools applied to health-related free-text survey data in the context of COVID-19. The objective was to automatically predict sentence sentiment for 2 independent COVID-19 survey data sets from the National Institutes of Health and Stanford University. METHODS: Gold standard labels were created for a subset of each data set using a panel of human raters. We compared 8 state-of-the-art sentiment analysis tools on both data sets to evaluate variability and disagreement across tools. In addition, few-shot learning was explored by fine-tuning Open Pre-Trained Transformers (OPT; a large language model [LLM] with publicly available weights) using a small annotated subset and zero-shot learning using ChatGPT (an LLM without available weights). RESULTS: The comparison of sentiment analysis tools revealed high variability and disagreement across the evaluated tools when applied to health-related survey data. OPT and ChatGPT demonstrated superior performance, outperforming all other sentiment analysis tools. Moreover, ChatGPT outperformed OPT, exhibited higher accuracy by 6% and higher F-measure by 4% to 7%. CONCLUSIONS: This study demonstrates the effectiveness of LLMs, particularly the few-shot learning and zero-shot learning approaches, in the sentiment analysis of health-related survey data. These results have implications for saving human labor and improving efficiency in sentiment analysis tasks, contributing to advancements in the field of automated sentiment analysis.

Trends in Language Use During the COVID-19 Pandemic and Relationship Between Language Use and Mental Health: Text Analysis Based on Free Responses From a Longitudinal Study.

BACKGROUND: The COVID-19 pandemic and its associated restrictions have been a major stressor that has exacerbated mental health worldwide. Qualitative data play a unique role in documenting mental states through both language features and content. Text analysis methods can provide insights into the associations between language use and mental health and reveal relevant themes that emerge organically in open-ended responses. OBJECTIVE: The aim of this web-based longitudinal study on mental health during the early COVID-19 pandemic was to use text analysis methods to analyze free responses to the question, "Is there anything else you would like to tell us that might be important that we did not ask about?" Our goals were to determine whether individuals who responded to the item differed from nonresponders, to determine whether there were associations between language use and psychological status, and to characterize the content of responses and how responses changed over time. METHODS: A total of 3655 individuals enrolled in the study were asked to complete self-reported measures of mental health and COVID-19 pandemic-related questions every 2 weeks for 6 months. Of these 3655 participants, 2497 (68.32%) provided at least 1 free response (9741 total responses). We used various text analysis methods to measure the links between language use and mental health and to characterize response themes over the first year of the pandemic. RESULTS: Response likelihood was influenced by demographic factors and health status: those who were male, Asian, Black, or Hispanic were less likely to respond, and the odds of responding increased with age and education as well as with a history of physical health conditions. Although mental health treatment history did not influence the overall likelihood of responding, it was associated with more negative sentiment, negative word use, and higher use of first-person singular pronouns. Responses were dynamically influenced by psychological status such that distress and loneliness were positively associated with an individual's likelihood to respond at a given time point and were associated with more negativity. Finally, the responses were negative in valence overall and exhibited fluctuations linked with external events. The responses covered a variety of topics, with the most common being mental health and emotion, social or physical distancing, and policy and government. CONCLUSIONS: Our results identify trends in language use during the first year of the pandemic and suggest that both the content of responses and overall sentiments are linked to mental health.

Test-Retest Reliability of an Adaptive Thermal Pain Calibration Procedure in Healthy Volunteers.

Quantitative sensory testing (QST) allows researchers to evaluate associations between noxious stimuli and acute pain in clinical populations and healthy participants. Despite its widespread use, our understanding of QST's reliability is limited, as reliability studies have used small samples and restricted time windows. We examined the reliability of pain ratings in response to noxious thermal stimulation in 171 healthy volunteers (n = 99 female, n = 72 male) who completed QST on multiple visits ranging from 1 day to 952 days between visits. On each visit, participants underwent an adaptive pain calibration in which they experienced 24 heat trials and rated pain intensity after stimulus offset on a 0 to 10 Visual Analog Scale. We used linear regression to determine pain threshold, pain tolerance, and the correlation between temperature and pain for each session and examined the reliability of these measures. Threshold and tolerance were moderately reliable (Intra-class correlation = .66 and .67, respectively; P < .001), whereas temperature-pain correlations had low reliability (Intra-class correlation = .23). In addition, pain tolerance was significantly more reliable in female participants than male participants, and we observed similar trends for other pain sensitive measures. Our findings indicate that threshold and tolerance are largely consistent across visits, whereas sensitivity to changes in temperature vary over time and may be influenced by contextual factors. PERSPECTIVE: This article assesses the reliability of an adaptive thermal pain calibration procedure. We find that pain threshold and tolerance are moderately reliable whereas the correlation between pain rating and stimulus temperature has low reliability. Female participants were more reliable than male participants on all pain sensitivity measures.

High-beta/low-gamma frequency activity reflects top-down predictive coding during a spatial working memory test.

Numerous mental health disorders are characterized by cognitive impairments that result in poor vocational and social outcomes. Among the cognitive domains commonly affected, working memory deficits have been noted in patients with attention-deficit/hyperactivity disorder (Martinussen et al. in J Am Acad Child Adolesc Psychiatry 44:377-384, 2005), post-traumatic stress disorder (Honzel et al. in Cogn Affect Behav Neurosci 14:792-804, 2014), and consistently with schizophrenia patients (Callicott et al. in Cereb Cortex 10:1078-1092, 2000; Lewis et al. in Front Hum Neurosci 10:85, 2005; Amann et al. in Brain Res Bull 83:147-161, 2010; Limongi et al. in Schizophr Res 197:386-391, 2018). Oscillations in neural activity from electroencephalogram (EEG) recordings are decomposed by frequency, and band-specific decreases in gamma power (> 30 Hz) have been correlated with working memory ability. This study examined within-subject changes in power of frequency-specific bands during sample versus choice trials during a spatial working memory paradigm (T-maze). EEG was recorded using a relatively novel wireless EEG telemetry system fully implanted within the mouse, enabling uninhibited movement during behavioral tasks. No significant differences were found between sample and correct choice phases in the alpha, theta or gamma frequency ranges. Evoked power was significantly higher during the choice phase than the sample phase in the high-beta/low-gamma frequency range. This frequency range has been implicated in the propagation of cortical predictions to lower levels of stimuli encoding in a top-down hierarchical manner. Results suggest there is an increase in brain activity during correct trials when the mouse enters the opposite arm during the choice phase compared to the sample phase, likely due to prediction error resulting from a discrepancy between present and prior experience. Future studies should identify specific cortical networks involved and investigate neural activity at the neuronal level.