Mental Health Counseling From Conversational Content With Transformer-Based Machine Learning.

Importance: Use of asynchronous text-based counseling is rapidly growing as an easy-to-access approach to behavioral health care. Similar to in-person treatment, it is challenging to reliably assess as measures of process and content do not scale. Objective: To use machine learning to evaluate clinical content and client-reported outcomes in a large sample of text-based counseling episodes of care. Design, Setting, and Participants: In this quality improvement study, participants received text-based counseling between 2014 and 2019; data analysis was conducted from September 22, 2022, to November 28, 2023. The deidentified content of messages was retained as a part of ongoing quality assurance. Treatment was asynchronous text-based counseling via an online and mobile therapy app (Talkspace). Therapists were licensed to provide mental health treatment and were either independent contractors or employees of the product company. Participants were self-referred via online sign-up and received services via their insurance or self-pay and were assigned a diagnosis from their health care professional. Exposure: All clients received counseling services from a licensed mental health clinician. Main Outcomes and Measures: The primary outcomes were client engagement in counseling (number of weeks), treatment satisfaction, and changes in client symptoms, measured via the 8-item version of Patient Health Questionnaire (PHQ-8). A previously trained, transformer-based, deep learning model automatically categorized messages into types of therapist interventions and summaries of clinical content. Results: The total sample included 166 644 clients treated by 4973 therapists (20 600 274 messages). Participating clients were predominantly female (75.23%), aged 26 to 35 years (55.4%), single (37.88%), earned a bachelor's degree (59.13%), and were White (61.8%). There was substantial variability in intervention use and treatment content across therapists. A series of mixed-effects regressions indicated that collectively, interventions and clinical content were associated with key outcomes: engagement (multiple R = 0.43), satisfaction (multiple R = 0.46), and change in PHQ-8 score (multiple R = 0.13). Conclusions and Relevance: This quality improvement study found associations between therapist interventions, clinical content, and client-reported outcomes. Consistent with traditional forms of counseling, higher amounts of supportive counseling were associated with improved outcomes. These findings suggest that machine learning-based evaluations of content may increase the scale and specificity of psychotherapy research.

How do you feel? Using natural language processing to automatically rate emotion in psychotherapy.

Emotional distress is a common reason for seeking psychotherapy, and sharing emotional material is central to the process of psychotherapy. However, systematic research examining patterns of emotional exchange that occur during psychotherapy sessions is often limited in scale. Traditional methods for identifying emotion in psychotherapy rely on labor-intensive observer ratings, client or therapist ratings obtained before or after sessions, or involve manually extracting ratings of emotion from session transcripts using dictionaries of positive and negative words that do not take the context of a sentence into account. However, recent advances in technology in the area of machine learning algorithms, in particular natural language processing, have made it possible for mental health researchers to identify sentiment, or emotion, in therapist-client interactions on a large scale that would be unattainable with more traditional methods. As an attempt to extend prior findings from Tanana et al. (2016), we compared their previous sentiment model with a common dictionary-based psychotherapy model, LIWC, and a new NLP model, BERT. We used the human ratings from a database of 97,497 utterances from psychotherapy to train the BERT model. Our findings revealed that the unigram sentiment model (kappa = 0.31) outperformed LIWC (kappa = 0.25), and ultimately BERT outperformed both models (kappa = 0.48).

Determining the therapist's contribution in therapist multicultural competence process and outcome.

Efforts to help therapists improve their multicultural competence (MCC) rely on measures that can distinguish between different levels of competence. MCC is often assessed by asking clients to rate their experiences with their therapists. However, differences in client ratings of therapist MCC do not necessarily provide information about the relative performance of therapists and can be influenced by other factors including the client's own characteristics. In this study, we used a repeated measures design of 8,497 observations from 1,458 clients across 35 therapists to clarify the proportion of variability in MCC ratings attributed to the therapist versus the client and better understand the extent that an MCC measure detects therapist differences. Overall, we found that a small amount of variability in MCC ratings was attributed to the therapist (2%) and substantial amount attributed to the client (70%). These findings suggest that our measure of MCC primarily detected differences at the client level versus therapist level, indicating that therapist MCC scores were largely dependent on the client. Clinical implications and recommendations for future MCC research and measurement are discussed. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Machine learning and natural language processing in psychotherapy research: Alliance as example use case.

Artificial intelligence generally and machine learning specifically have become deeply woven into the lives and technologies of modern life. Machine learning is dramatically changing scientific research and industry and may also hold promise for addressing limitations encountered in mental health care and psychotherapy. The current paper introduces machine learning and natural language processing as related methodologies that may prove valuable for automating the assessment of meaningful aspects of treatment. Prediction of therapeutic alliance from session recordings is used as a case in point. Recordings from 1,235 sessions of 386 clients seen by 40 therapists at a university counseling center were processed using automatic speech recognition software. Machine learning algorithms learned associations between client ratings of therapeutic alliance exclusively from session linguistic content. Using a portion of the data to train the model, machine learning algorithms modestly predicted alliance ratings from session content in an independent test set (Spearman's ρ = .15, p < .001). These results highlight the potential to harness natural language processing and machine learning to predict a key psychotherapy process variable that is relatively distal from linguistic content. Six practical suggestions for conducting psychotherapy research using machine learning are presented along with several directions for future research. Questions of dissemination and implementation may be particularly important to explore as machine learning improves in its ability to automate assessment of psychotherapy process and outcome. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

The experimental effects of psilocybin on symptoms of anxiety and depression: A meta-analysis.

The current meta-analysis examined the effects of psilocybin in combination with behavioral interventions on anxiety and depression in samples with elevated symptoms. Across four studies (one uncontrolled; three randomized, placebo-controlled; N = 117), within-group pre-post and pre-follow-up effects on anxiety and depression were large (Hedges' gs=1.16 to 1.47) and statistically significant. Across three placebo-controlled studies, pre-post placebo-controlled effects were also large (gs = 0.82 to 0.83) and statistically significant. No serious adverse events were reported. Limitations include the small number of studies and risk for bias within studies. Results tentatively support future research on psilocybin for the treatment of anxiety and depression.

Design feasibility of an automated, machine-learning based feedback system for motivational interviewing.

Direct observation of psychotherapy and providing performance-based feedback is the gold-standard approach for training psychotherapists. At present, this requires experts and training human coding teams, which is slow, expensive, and labor intensive. Machine learning and speech signal processing technologies provide a way to scale up feedback in psychotherapy. We evaluated an initial proof of concept automated feedback system that generates motivational interviewing quality metrics and provides easy access to other session data (e.g., transcripts). The system automatically provides a report of session-level metrics (e.g., therapist empathy) and therapist behavior codes at the talk-turn level (e.g., reflections). We assessed usability, therapist satisfaction, perceived accuracy, and intentions to adopt. A sample of 21 novice (n = 10) or experienced (n = 11) therapists each completed a 10-min session with a standardized patient. The system received the audio from the session as input and then automatically generated feedback that therapists accessed via a web portal. All participants found the system easy to use and were satisfied with their feedback, 83% found the feedback consistent with their own perceptions of their clinical performance, and 90% reported they were likely to use the feedback in their practice. We discuss the implications of applying new technologies to evaluation of psychotherapy. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Variability in motivational interviewing adherence across sessions, providers, sites, and research contexts.

Providers' adherence in the delivery of behavioral interventions for substance use disorders is not fixed, but instead can vary across sessions, providers, and intervention sites. This variability can substantially impact the quality of intervention that clients receive. However, there has been limited work to systematically evaluate the extent to which substance use intervention adherence varies from session-to-session, provider-to-provider, and site-to-site. The present study quantifies the extent to which adherence to Motivational Interviewing (MI) for alcohol and drug use varies across sessions, providers, and intervention sites and compares the extent of this variability across three common MI research contexts that evaluate MI efficacy, MI effectiveness, and MI training. Independent raters coded intervention adherence to MI from 1275 sessions delivered by 216 providers at 15 intervention sites. Multilevel models indicated that 57%-94% of the variance in MI adherence was attributable to variability between sessions (i.e., within providers), while smaller proportions of variance were attributable to variability between providers (3%-26%) and between intervention sites (0.1%-28%). MI adherence was typically lowest and most variable within contexts evaluating MI training (i.e., where MI was not protocol-guided and delivered by community treatment providers) and, conversely, adherence was typically highest and least variable in contexts evaluating MI efficacy and effectiveness (i.e., where MI was highly protocolized and delivered by trained therapists). These results suggest that MI adherence in efficacy and effectiveness trials may be substantially different from that obtained in community treatment settings, where adherence is likely to be far more heterogeneous.

A multivariate meta-analysis of motivational interviewing process and outcome.

Motivational interviewing (MI) theory proposes a process whereby a set of therapist behaviors has direct effects on client outcomes and indirect effects through in-session processes (e.g., client change talk). Despite clear empirical support for the efficacy of MI across settings, the results of studies evaluating proposed links between MI process and outcome have been less clear. In the present study, we used a series of multivariate meta-analyses to test whether there are differential relationships between specific MI-consistent and MI-inconsistent therapist behaviors, MI therapist global ratings, client change language, and clinical outcomes. Based on 19 primary studies (N = 2,614), we found a significant relationship between MI-consistent therapist behaviors and greater client change talk, as well as greater client sustain talk. Higher therapist global ratings (empathy and MI spirit) were significantly related to increased MI-consistent behaviors, decreased MI-inconsistent behaviors, increased client change talk, yet also increased client sustain talk. Therapist global ratings were not significantly related to clinical outcomes. Client sustain talk was a significant predictor of worse clinical outcomes, while client change talk was unrelated to outcome. Variability within the correlations indicated that MI-consistent and MI-inconsistent therapist behaviors were differentially related to therapist global ratings of empathy and MI spirit. Similar to past research, present findings provide equivocal support for hypothesized MI process outcome relationships. Clinical implications and future areas of MI mechanism research are discussed. (PsycINFO Database Record

In pursuit of truth: A critical examination of meta-analyses of cognitive behavior therapy.

OBJECTIVE: Three recent meta-analyses have made the claim, albeit with some caveats, that cognitive-behavioral treatments (CBT) are superior to other psychotherapies, in general or for specific disorders (e.g., social phobia). METHOD: The purpose of the present article was to examine four issues in meta-analysis that mitigate claims of CBT superiority: (a) effect size, power, and statistical significance, (b) focusing on disorder-specific symptom measures and ignoring other important indicators of psychological functioning, (c) problems inherent in classifying treatments provided in primary studies into classes of treatments, and (d) the inclusion of problematic trials, which biases the results, and the exclusion of trials that fail to find differences among treatments. RESULTS: When these issues are examined, the effects demonstrating the superiority of CBT are small, nonsignificant for the most part, limited to targeted symptoms, or are due to flawed primary studies. CONCLUSION: Meta-analytic evidence for the superiority of CBT in the three meta-analysis are nonexistent or weak.

A meta-analysis of multicultural competencies and psychotherapy process and outcome.

For decades, psychologists have emphasized the provision of multiculturally competent psychotherapy to reduce racial and ethnic disparities in mental health treatment. However, the relationship between multicultural competencies (MC) and other measures of clinical process and treatment outcome has shown heterogeneity in effect sizes. This meta-analysis tested the association of client ratings of therapist MC with measures of therapeutic processes and outcome, including: (a) working alliance, (b) client satisfaction, (c) general counseling competence, (d) session impact, and (e) symptom improvement. Among 18 studies (20 independent samples) included in the analysis, the correlation between therapist MC and outcome (r = .29) was much smaller than the association with process measures (r = .75), but there were no significant differences in correlations across different types of MC or clinical process measures. Providing some evidence of publication bias, effect sizes from published studies (r = .67) were larger than those from unpublished dissertations (r = .28). Moderator analyses indicated that client age, gender, the representation of racial-ethnic minority (R-EM) clients, and clinical setting were not associated with effect size variability. Based on these findings, we discuss implications and recommendations for future research that might lead to a better understanding of the effects of therapist MC on treatment process and outcome. Primary needs in future research include the development and evaluation of observer ratings of therapist MC and the implementation of longitudinal research designs.

The enduring effects of psychodynamic treatments vis-à-vis alternative treatments: A multilevel longitudinal meta-analysis.

Although evidence suggests that the benefits of psychodynamic treatments are sustained over time, presently it is unclear whether these sustained benefits are superior to non-psychodynamic treatments. Additionally, the extant literature comparing the sustained benefits of psychodynamic treatments compared to alternative treatments is limited with methodological shortcomings. The purpose of the current study was to conduct a rigorous test of the growth of the benefits of psychodynamic treatments relative to alternative treatments across distinct domains of change (i.e., all outcome measures, targeted outcome measures, non-targeted outcome measures, and personality outcome measures). To do so, the study employed strict inclusion criteria to identify randomized clinical trials that directly compared at least one bona fide psychodynamic treatment and one bona fide non-psychodynamic treatment. Hierarchical linear modeling (Raudenbush, Bryk, Cheong, Congdon, & du Toit, 2011) was used to longitudinally model the impact of psychodynamic treatments compared to non-psychodynamic treatments at post-treatment and to compare the growth (i.e., slope) of effects beyond treatment completion. Findings from the present meta-analysis indicated that psychodynamic treatments and non-psychodynamic treatments were equally efficacious at post-treatment and at follow-up for combined outcomes (k=20), targeted outcomes (k=19), non-targeted outcomes (k=17), and personality outcomes (k=6). Clinical implications, directions for future research, and limitations are discussed.

Combat Exposure and Risk for Suicidal Thoughts and Behaviors Among Military Personnel and Veterans: A Systematic Review and Meta-Analysis.

Due to seemingly mixed empirical results, questions persist about the possible role of deployments and combat exposure. We conducted a narrative review and meta-analysis of 22 published studies to integrate findings regarding the relationship of deployment-related predictors (i.e., deployment, deployment to a combat zone, combat experience, and exposure to specific combat events) with suicide-related outcomes (i.e., suicide ideation, attempt, and death). Across all predictors and outcomes, the combined effect was small and positive, r = .08 [0.04, 0.13], and marked by significant heterogeneity, I2  = 99.9%, Q(21)=4880.16, p < .0001, corresponding to a 25% increased risk for suicide-related outcomes among those who have deployed. Studies examining the relationship between exposure to killing and atrocities (k = 5) showed the largest combined effect, r = .12 [0.08, 0.17], and less heterogeneity, I2  = 84.4%, Q(4)=34.96, p < .0001, corresponding to a 43% increased risk for suicide-related outcomes among those exposed to killing or atrocity. Implications for theory, research, and clinical practice are discussed.

Cognitive-behavioral therapy versus other therapies: redux.

Despite the evidence suggesting that all treatments intended to be therapeutic are equally efficacious, the conjecture that one form of treatment, namely cognitive-behavioral therapy (CBT), is superior to all other treatment persists. The purpose of the current study was to (a) reanalyze the clinical trials from an earlier meta-analysis that compared CBT to 'other therapies' for depression and anxiety (viz., Tolin, 2010) and (b) conduct a methodologically rigorous and comprehensive meta-analysis to determine the relative efficacy of CBT and bona fide non-CBT treatments for adult anxiety disorders. Although the reanalysis was consistent with the earlier meta-analysis' findings of small to medium effect sizes for disorder-specific symptom measures, the reanalysis revealed no evidence for the superiority of CBT for depression and anxiety for outcomes that were not disorder-specific. Following the reanalysis, a comprehensive anxiety meta-analysis that utilized a survey of 91 CBT experts from the Association of Behavioral and Cognitive Therapists (ABCT) to consensually identify CBT treatments was conducted. Thirteen clinical trials met the inclusion criteria. There were no differences between CBT treatments and bona fide non-CBT treatments across disorder-specific and non-disorder specific symptom measures. These analyses, in combination with previous meta-analytic findings, fail to provide corroborative evidence for the conjecture that CBT is superior to bona fide non-CBT treatments.