ChatGPT and Bard exhibit spontaneous citation fabrication during psychiatry literature search.

ChatGPT (Generative Pre-Trained Transformer) is a large language model (LLM), which comprises a neural network that has learned information and patterns of language use from large amounts of text on the internet. ChatGPT, introduced by OpenAI, responds to human queries in a conversational manner. Here, we aimed to assess whether ChatGPT could reliably produce accurate references to supplement the literature search process. We describe our March 2023 exchange with ChatGPT, which generated thirty-five citations, two of which were real. 12 citations were similar to actual manuscripts (e.g., titles with incorrect author lists, journals, or publication years) and the remaining 21, while plausible, were in fact a pastiche of multiple existent manuscripts. In June 2023, we re-tested ChatGPT's performance and compared it to that of Google's GPT counterpart, Bard 2.0. We investigated performance in English, as well as in Spanish and Italian. Fabrications made by LLMs, including erroneous citations, have been called "hallucinations"; we discuss reasons for which this is a misnomer. Furthermore, we describe potential explanations for citation fabrication by GPTs, as well as measures being taken to remedy this issue, including reinforcement learning. Our results underscore that output from conversational LLMs should be verified.

Differential Expression of Anomalous Self-Experiences in Spontaneous Speech in Clinical High-Risk and Early-Course Psychosis Quantified by Natural Language Processing.

BACKGROUND: Basic self-disturbance, or anomalous self-experiences (ASEs), is a core feature of the schizophrenia spectrum. We propose a novel method of natural language processing to quantify ASEs in spoken language by direct comparison to an inventory of self-disturbance, the Inventory of Psychotic-Like Anomalous Self-Experiences (IPASE). We hypothesized that there would be increased similarity in open-ended speech to the IPASE items in individuals with early-course psychosis (PSY) compared with healthy individuals, with clinical high-risk (CHR) individuals intermediate in similarity. METHODS: Open-ended interviews were obtained from 170 healthy control participants, 167 CHR participants, and 89 PSY participants. We calculated the semantic similarity between IPASE items and "I" sentences from transcribed speech samples using S-BERT (Sentence Bidirectional Encoder Representation from Text). Kolmogorov-Smirnov tests were used to compare distributions across groups. A nonnegative matrix factorization of cosine similarity was performed to rank IPASE items. RESULTS: Spoken language of CHR individuals had the greatest semantic similarity to IPASE items when compared to both healthy control (s = 0.44, p < 10-14) and PSY (s = 0.36, p < 10-6) individuals, while IPASE scores were higher among PSY than CHR group participants. In addition, the nonnegative matrix factorization approach produced a data-driven domain that differentiated the CHR group from the others. CONCLUSIONS: We found that open-ended interviews elicited language with increased semantic similarity to the IPASE by participants in the CHR group compared with patients with psychosis. This demonstrates the utility of these methods for differentiating patients from healthy control participants. This complementary approach has the capacity to scale to large studies investigating phenomenological features of schizophrenia and potentially other clinical populations.

Linguistic correlates of suicidal ideation in youth at clinical high-risk for psychosis.

Suicidal ideation (SI) is prevalent among individuals at clinical high-risk for psychosis (CHR). Natural language processing (NLP) provides an efficient method to identify linguistic markers of suicidality. Prior work has demonstrated that an increased use of "I", as well as words with semantic similarity to "anger", "sadness", "stress" and "lonely", are correlated with SI in other cohorts. The current project analyzes data collected in an SI supplement to an NIH R01 study of thought disorder and social cognition in CHR. This study is the first to use NLP analyses of spoken language to identify linguistic correlates of recent suicidal ideation among CHR individuals. The sample included 43 CHR individuals, 10 with recent suicidal ideation and 33 without, as measured by the Columbia-Suicide Severity Rating Scale, as well as 14 healthy volunteers without SI. NLP methods include part-of-speech (POS) tagging, a GoEmotions-trained BERT Model, and Zero-Shot Learning. As hypothesized, individuals at CHR for psychosis who endorsed recent SI utilized more words with semantic similarity to "anger" compared to those who did not. Words with semantic similarity to "stress", "loneliness", and "sadness" were not significantly different between the two CHR groups. Contrary to our hypotheses, CHR individuals with recent SI did not use the word "I" more than those without recent SI. As anger is not characteristic of CHR, findings have implications for the consideration of subthreshold anger-related sentiment in suicidal risk assessment. As NLP is scalable, findings suggest that language markers may improve suicide screening and prediction in this population.

Construct validity for computational linguistic metrics in individuals at clinical risk for psychosis: Associations with clinical ratings.

Language deficits are prevalent in psychotic illness, including its risk states, and are related to marked impairment in functioning. It is therefore important to characterize language impairment in the psychosis spectrum in order to develop potential preventive interventions. Natural language processing (NLP) metrics of semantic coherence and syntactic complexity have been used to discriminate schizophrenia patients from healthy controls (HC) and predict psychosis onset in individuals at clinical high-risk (CHR) for psychosis. To date, no studies have yet examined the construct validity of key NLP features with respect to clinical ratings of thought disorder in a CHR cohort. Herein we test the association of key NLP metrics of coherence and complexity with ratings of positive and negative thought disorder, respectively, in 60 CHR individuals, using Andreasen's Scale of Assessment of Thought, Language and Communication (TLC) Scale to measure of positive and negative thought disorder. As hypothesized, in CHR individuals, the NLP metric of semantic coherence was significantly correlated with positive thought disorder severity and the NLP metrics of complexity (sentence length and determiner use) were correlated with negative thought disorder severity. The finding of construct validity supports the premise that NLP analytics, at least in respect to core features of reduction of coherence and complexity, are capturing clinically relevant language disturbances in risk states for psychosis. Further psychometric study is required, in respect to reliability and other forms of validity.

A qualitative study on identity in individuals at clinical high risk for psychosis: " Why does it have to be one thing?".

OBJECTIVE: Qualitative research can shed light on the subjective experiences of individuals at clinical high risk (CHR) for psychosis, complement quantitative research, broaden our understanding of experiencing CHR, and inform intervention development. The aim of this study was to explore life experiences of individuals at CHR through qualitative research. METHOD: Participants were 37 individuals at CHR (20 male, 17 female) aged 16-34 (Mage = 23.32 ± 5.26), and 16 healthy controls (HCs; 7 male, 9 female) aged 18-34 (Mage = 25.37 ± 4.05). Qualitative data were obtained through open-ended interviews (30-45 min). No a priori hypotheses were made, and thematic analyses were used to develop themes. RESULTS: Four major themes and one subtheme related to identity were identified through the iterative thematic analysis: defining a self-concept (with a subtheme of creativity), identity development/formation, feeling different from others, and change from a former self. Over 80% of the CHR cohort spontaneously discussed topics related to their identity, compared to 38% of HCs. HCs only reported content within the defining a self-concept theme, while the CHR group reported content within all themes. CONCLUSIONS AND IMPLICATIONS FOR PRACTICE: The present study demonstrates that identity formation is a major process for youth in general and that psychosis experiences can make this process more challenging. CHR participants spontaneously brought up multiple themes related to identity in open-ended interviews, suggesting the relevance of this topic in this population. Clinicians should continue to probe identity-related concerns on an individual basis and research should focus on integrating this framework into the conceptualization and treatment of CHR. (PsycInfo Database Record (c) 2022 APA, all rights reserved).