Pragmatic fidelity measurement in youth service settings.

Background: Fidelity measurement is critical for developing, evaluating, and implementing evidence-based treatments (EBTs). However, traditional fidelity measurement tools are often not feasible for community-based settings. We developed a short fidelity rating form for the Collaborative Problem Solving (CPS) approach from an existing manualized coding system that requires extensive training. We examined the reliability and accuracy of this short form when completed by trained observers, untrained observers, and self-reporting providers to evaluate multiple options for reducing barriers to fidelity measurement in community-based settings. Methods: Community-based treatment providers submitted recordings of youth service sessions in which they did, or did not, use CPS. For 60 recordings, we compared short-form fidelity ratings assigned by trained observers and untrained observers to those provided by trained observers on the manualized coding system. For 141 recordings, we compared providers' self-reported fidelity on the short form to ratings provided by trained observers on the manualized coding system and examined providers' accuracy as a function of their global fidelity. Results & Conclusions: The short form was reliable and accurate for trained observers. An assigned global integrity score and a calculated average of component scores on the short form, but not component scores themselves, were reliable and accurate for observers who had CPS expertise but no specific training on rating CPS fidelity. When providers self-reported fidelity on the short form, their global integrity score was a reliable estimate of their CPS integrity; however, providers with better CPS fidelity were most accurate in their self-reports. We discuss the costs and benefits of these more pragmatic fidelity measurement options in community-based settings.

Developing brief, easy-to-use, and reliable tools to measure how well providers deliver evidence-based treatments (EBTs) in community clinical settings is critical to ensure the benefits of EBTs. However, reliable tools are often too time-consuming and not feasible to use in community settings because they require independent observers to receive intensive training on a coding system and to observe live or recorded treatment sessions for reliable and accurate evaluation. This paper describes steps we took to develop a more practical measure of how well providers deliver one EBT, Collaborative Problem Solving (CPS), based on a previously validated measure, to explore whether the quality of the measure can be maintained while reducing the need for training independent observers and the need for recording treatment sessions. This work contributes to the growing efforts of developing more pragmatic fidelity measures and introduces a new tool, the CPS Practice Integrity Form (CPS-PIF), as a promising measure for community-based clinical settings using CPS.

An open trial of Collaborative Problem Solving in a naturalistic outpatient setting.

Collaborative Problem Solving (CPS) is an intervention for reducing children's challenging behaviors. The aim of the present study was to evaluate the effects of family therapy using CPS in an outpatient clinic that specializes in treating children with challenging behaviors. One hundred and twenty families presented for treatment. Diagnoses at intake were varied, and 100 children (83%) had symptoms that were in the clinical range at intake. Parents reported significant change in their understanding of challenging behavior and prediction of children's behavioral symptoms 3 months into treatment. Furthermore, children's improvement was predicted by their parents' increased understanding that cognitive skill deficits are responsible for challenging behavior. These results suggest that using CPS in community-based, outpatient family treatment is effective for helping children who exhibit a range of clinical symptoms. Results provide insights for clinical practice and research on CPS.

The value of implementation frameworks: Using the active implementation frameworks to guide system-wide implementation of Collaborative Problem Solving.

In the last decade, many implementation frameworks have emerged that consolidate the research on implementation science, guiding purveyors and service agencies in improving implementation of evidence-based practices (EBPs). In this paper, we describe how the purveyor of one EBP utilized the active implementation frameworks (AIFs) to define and standardize strategies for site-wide implementation. We illustrate what implementation looked like before and after using AIFs to understand implementation, as well as some ways in which using the AIFs helped the purveyor identify, and then overcome, barriers to implementation. This paper provides a model for others who seek to use AIFs to guide their implementation practices, or more broadly, an illustration of how to use any implementation framework to ensure best practices in implementation.

Reliability and validity of the Thinking Skills Inventory, a screening tool for cross-diagnostic skill deficits underlying youth behavioral challenges.

Deficits in a range of skill domains (including executive functioning, emotion regulation, social cognition and language/communication) are associated with disrupted youth behavior and functioning across mental health diagnoses. The identification of skill deficits are important for effective treatment planning, particularly for personalized interventions. While there are multiple ways to assess these skills, parent/caregiver reports represent an important information source. To date, no single, brief measure has been developed that gathers parent/caregiver ratings across this range of constructs. We have developed a short caregiver-report questionnaire (the Thinking Skills Inventory; TSI), to screen for skill deficits. Here, we examine the reliability and validity of this rating scale in 384 youth who were consecutively referred for neuropsychiatric evaluation. A primary caregiver completed the TSI as well as other established measures. Exploratory and confirmatory factor analyses support five subscales on the TSI: Attention and Working Memory, Language and Communication, Emotion Regulation, Cognitive Flexibility, and Social Thinking Skills. The subscales showed moderate to high internal consistency (Cronbach's alphas range from 0.84 to 0.91). Correlations with established caregiver-report measures confirm their convergent and discriminant validity, and associations with multiple clinical diagnoses and cross-diagnostic aggressive behavior further support the utility of the scale for our intended purpose. In sum, this free, brief measure is a valid and reliable way to identify variation in skill domains relevant to a range of psychopathology. The TSI may be useful in youth mental health settings to assist with treatment planning and to inform referral for further evaluation.

The Collaborative Problem Solving approach: outcomes across settings.

In the last decade, Collaborative Problem Solving (CPS) has become a popular approach to managing the challenging behaviors of children and adolescents, and has established a growing evidence base for reducing oppositional behavior and related outcomes. In contrast with standard behavioral methods that provide incentives for meeting adult expectations, CPS focuses on identifying and treating lagging cognitive skills that interfere with children's ability to meet these expectations. Since the majority of CPS outcomes have been evaluated in clinical and educational settings as part of internal quality-improvement efforts, only a small proportion of these findings has been published in peer-reviewed academic journals. Here, we describe the CPS approach and provide a summary of all known published and unpublished findings related to its implementation in outpatient, inpatient, residential, juvenile justice, and educational settings. Finally, we provide specific recommendations for future research on the model.

Can two psychotherapy process measures be dependably rated simultaneously? A generalizability study.

Observer ratings in psychotherapy are a common way of collecting information in psychotherapy research. However, human observers are imperfect instruments, and their ratings may be subject to variability from several sources. One source of variability can be raters' assessing more than 1 instrument at a time. The purpose of this research is to investigate whether it is possible to have raters assess 2 different psychotherapy process measures simultaneously and still produce dependable scores. Two studies were designed. The first compared scores assessed by raters who rated either 1 instrument or 2 instruments simultaneously. The second compared scores of raters who assessed 2 instruments simultaneously and scores of expert raters who assessed 1 instrument. The results show that variability in scores is largely due to differences across the object of measurement (e.g., patients). Small variability was reported for raters, raters' interaction with patients, and whether the rater assessed 1 instrument or 2 instruments simultaneously. The results are promising for the quality of observer ratings of psychotherapy process and for the feasibility of future psychotherapy process research. (PsycINFO Database Record (c) 2012 APA, all rights reserved).

The role of the couch in psychoanalysis: proposed research designs and some preliminary data.

Ever since Freud, the couch has been viewed as an important--some would argue essential--component of psychoanalysis. Although many theoretical papers and case reports have addressed the use of the couch in psychoanalysis, no empirical study has investigated its effect on psychoanalytic process or outcome. After a review of the literature, a number of research designs are proposed that might be used in such an investigation. Finally, preliminary empirical data are presented from archived audiotapes of two psychoanalyses: one in which the patient switched from lying down to sitting up, and one in which the opposite occurred. The aim is to stimulate research-oriented psychoanalysts to undertake empirical investigations of the theoretical concepts underlying use of the couch and, more generally, to present a specific example of research as a paradigm for a broader research agenda for empirical investigation of the key theoretical ideas underlying psychoanalysis.

Patient and practitioner influences on the placebo effect in irritable bowel syndrome.

OBJECTIVE: To determine whether placebo responses can be explained by characteristics of the patient, the practitioner, or their interpersonal interaction. METHODS: We performed an analysis of videotape and psychometric data from a clinical trial of patients with irritable bowel syndrome who were treated with placebo acupuncture in either a warm empathic interaction (Augmented, n = 96), a neutral interaction (Limited, n = 97), or a waitlist control (Waitlist, n = 96). We examined the relationships between the placebo response and a) patient personality and demographics; b) treating practitioner; and c) the patient-practitioner interaction as captured on videotape and rated by the Psychotherapy Process Q-Set. RESULTS: Patient extraversion, agreeableness, openness to experience, and female gender were associated with placebo response, but these effects held only in the augmented group. Regression analyses controlling for all other independent variables suggest that only extraversion is an independent predictor of placebo response. There were significant differences between practitioners in outcomes; this effect was twice as large as the effect attributable to treatment group assignment. Videotape analysis indicated that the augmented group fostered a treatment relationship similar to a prototype of an ideal healthcare interaction. CONCLUSIONS: Personality and gender influenced the placebo response, but only in the warm, empathic, augmented group. This suggests that, to the degree a placebo effect is evoked by the patient-practitioner relationship, personality characteristics of the patient will be associated with the placebo response. In addition, practitioners differed markedly in effectiveness, despite standardized interactions. We propose that the quality of the patient-practitioner interaction accounts for the significant difference between the groups in placebo response.

Psychoanalysis with avoidant personality disorder: A systematic case study.

This case study of process and outcome is based upon data obtained during a 5-year psychoanalysis of an adult male with avoidant personality disorder (AVPD). To date, no known systematic case studies, effectiveness studies, or randomized control trials exist for psychoanalysis in the treatment of AVPD. In this study, self-reported symptoms and observer-rated personality disorder (PD), global functioning, object relations, and psychological health were gathered at intake, after each year of treatment, and at 1-year follow-up. Process data was gathered to determine the extent to which the treatment adhered to prototypes of psychodynamic, cognitive-behavioral, and interpersonal therapy. Results indicated that the patient achieved clinically significant reductions in PD, symptom severity, and relational pathology. Gains were maintained at 1-year follow-up. The treatment significantly adhered to psychodynamic principles throughout, with some use of cognitive-behavioral and interpersonal principles in the third year of treatment. These findings warrant further investigation of psychoanalysis for AVPD and demonstrate the usefulness of assessing multiple domains of patient functioning and treatment process. (PsycINFO Database Record (c) 2010 APA, all rights reserved).

Measuring psychotherapy process with the Adolescent Psychotherapy Q-set (APQ): Development and applications for training.

This study presents the initial findings from the development of an adolescent version of the Child Psychotherapy Q-set (CPQ, Schneider, 2003). The CPQ is a psychotherapy process measure that utilizes Q-methodology to define psychotherapy process in an empirical form suitable for quantitative analysis. This new instrument, the Adolescent Psychotherapy Process Q-set (APQ), was developed to be pantheoretical in assessing the process of psychotherapy with adolescents. Panels of "expert" psychodynamic (PD) and cognitive-behavioral (CB) theoreticians and practitioners (N = 22) rated the 100 APQ items, and these ratings were factor analyzed to develop general prototypes of ideally conducted therapy for each respective orientation. Agreement between and amongst the experts was examined; in order to determine the reliability of, and shared variance between, each prototype. Coefficient alpha reliabilities were high for both PD (.93) and CB (.94) orientations. A significant correlation between the factor scores of each prototype (r = .31; p ≤ .01) was found. In a second study, a case example is provided to demonstrate the applications of the APQ and the prototypes in psychotherapy training and supervision. (PsycINFO Database Record (c) 2010 APA, all rights reserved).

Innovations: child & adolescent psychiatry: use of collaborative problem solving to reduce seclusion and restraint in child and adolescent inpatient units.

The authors describe "collaborative problem solving," a cognitive-behavioral approach for working with aggressive children and adolescents. The model conceptualizes aggressive behavior as the byproduct of lagging cognitive skills in the domains of flexibility, frustration tolerance, and problem solving. The goal is to train staff to assess specific cognitive skills that may be contributing to challenging behavior and to teach children new skills through collaborative problem solving. The authors present results from an inpatient unit that dramatically reduced rates of seclusion and restraint.

Beyond brand names of psychotherapy: Identifying empirically supported change processes.

There is considerable debate about which empirical research methods best advance clinical outcomes in psychotherapy. The prevailing tendency has been to test treatment packages using randomized, controlled clinical trials. Recently, focus has shifted to considering how studying the process of change in naturalistic treatments can be a useful complement to controlled trials. Clinicians self-identifying as psychodynamic treated 17 panic disorder patients in naturalistic psychotherapy for an average of 21 sessions. Patients achieved statistically significant reductions in symptoms across all domains. Rates of remission and clinically significant change as well as effect sizes were commensurate with those of empirically supported therapies for panic disorder. Treatment gains were maintained at 6-month follow-up. Intensive analysis of the process of the treatments revealed that integrative elements characterized the treatments: Adherence to cognitive-behavioral process was most characteristic, adherence to interpersonal and psychodynamic process, however, was most predictive of positive outcome. Specific process predictors of outcome were identified using the Psychotherapy Process Q-Set. These findings demonstrate how process research can be used to empirically validate change processes in naturalistic treatments as opposed to treatment packages in controlled trials. (PsycINFO Database Record (c) 2010 APA, all rights reserved).

The relationship among patient contemplation, early alliance, and continuation in psychotherapy.

The present study examined the relationship among contemplation stage of readiness to change, formation of an early therapeutic alliance, and psychological distress following the first session of psychotherapy. Significant correlations between the contemplation scores and the therapeutic alliance were found for patients in the contemplation stage. Although contemplation scores were not a factor in return for a second session of psychotherapy, the bond subscale of the alliance inventory did significantly contribute to whether patients returned for therapy. Patient psychological distress was not a significant factor in predicting the early alliance. Results indicate a need for further focus on contemplation with its inherent ambivalence, its relationship to alliance, and continuation in early psychotherapy. (PsycINFO Database Record (c) 2010 APA, all rights reserved).

On analytic process.

An innovative methodology is presented for identifying and assessing change process in psychoanalytic treatments. Using the Psychotherapy Process Q-set (PQS), a panel of experienced psychoanalysts developed a prototype of an ideal psychoanalytic hour. This prototype was then applied to verbatim transcripts of three archived treatment samples: psychoanalyses, long-term analytic therapies, and brief psychodynamic therapies. The degree to which these treatments fostered an analytic process as represented by the prototype was measured quantitatively. Analytic process was significantly more present in psychoanalyses than in the long-term analytic therapies, which, in turn fostered significantly more analytic process than did brief psychodynamic therapies. The study demonstrates that, given descriptive language that does not represent a particular theoretical perspective, analysts can agree on a definition of analytic process, and that analytic process can be operationalized and quantitatively assessed. A second study demonstrates that despite consensus on its definition, there is not just one proper analytic process; rather, there are change processes unique to each dyad. Two quantitative case studies illustrate how each analytic pair has a unique interaction pattern linked to treatment progress. These dyad-unique "interaction structures" are recurrent, mutually influencing patterns of interaction, the experience, recognition, and comprehension of which appear to be a fundamental component of therapeutic action. A bipersonal model is described that attempts to bridge theories of therapeutic action that focus on insight and self-understanding and those that emphasize the patient's experience of the therapist.

Effectiveness of collaborative problem solving in affectively dysregulated children with oppositional-defiant disorder: initial findings.

Oppositional-defiant disorder (ODD) refers to a recurrent pattern of negativistic, defiant, disobedient, and hostile behavior toward authority figures. Research has shown that children with ODD and comorbid mood disorders may be at particular risk for long-term adverse outcomes, including conduct disorder. In this study, the authors examined the effectiveness of a cognitive-behavioral model of intervention--called collaborative problem solving (CPS)--in comparison with parent training (PT) in 47 affectively dysregulated children with ODD. Results indicate that CPS produced significant improvements across multiple domains of functioning at posttreatment and at 4-month follow-up. These improvements were in all instances equivalent, and in many instances superior, to the improvements produced by PT. Implications of these findings for further research on and treatment selection in children with ODD are discussed.

Psychotherapy process: the missing link: comment on Westen, Novotny, and Thompson-Brenner (2004).

In this comment, J. S. Ablon and C. Marci argue that focusing on the empirical validation of manualized treatment packages misses important information about what is efficacious about a given treatment. Psychotherapy process has demonstrated that treatments may promote change in ways other than their underlying theories claim. Manualized therapies may appear distinct despite important similarities in dyadic interaction. These functional similarities in the emergent transactional process between therapist and patient may help explain the difficulty demonstrating differential outcomes across brands of brief therapy. Rather than focus on treatment packages targeting patient symptomatology, the authors recommend a shift in focus to the empirical validation of change processes coconstructed by therapist and patient in naturalistic settings.

A transactional model of oppositional behavior: underpinnings of the Collaborative Problem Solving approach.

Oppositional defiant disorder (ODD) refers to a recurrent pattern of developmentally inappropriate levels of negativistic, defiant, disobedient, and hostile behavior toward authority figures. ODD is one of the most common (and debilitating) comorbid disorders within Tourette's disorder (TD). Diverse psychosocial treatment approaches have been applied to children's ODD-related behaviors. In this paper, the authors articulate a transactional developmental conceptualization of oppositional behavior and describe a cognitive-behavioral model of intervention-called collaborative problem solving (CPS)-emanating from this conceptualization. The specific goals of the CPS approach are to help adults (1). understand the specific adult and child characteristics contributing to the development of a child's oppositional behavior; (2). become cognizant of three basic strategies for handling unmet expectations, including (a). imposition of adult will, (b). CPS, and (c). removing the expectation; (3). recognize the impact of each of these three approaches on parent-child interactions; and (4). become proficient, along with their children, at CPS as a means of resolving disagreements and defusing potentially conflictual situations so as to reduce oppositional episodes and improve parent-child compatibility. Summary data from an initial study documenting the effectiveness of the CPS approach (in comparison to the standard of care) are also presented.