A Randomized Trial to Identify Accurate Measurement Methods for Adherence to Cognitive-Behavioral Therapy.

Clinician fidelity to cognitive behavioral therapy (CBT) is an important mechanism by which desired clinical outcomes are achieved and is an indicator of care quality. Despite its importance, there are few fidelity measurement methods that are efficient and have demonstrated reliability and validity. Using a randomized trial design, we compared three methods of assessing CBT adherence-a core component of fidelity-to direct observation, the gold standard. Clinicians recruited from 27 community mental health agencies (n = 126; M age = 37.69 years, SD = 12.84; 75.7% female) were randomized 1:1:1 to one of three fidelity conditions: self-report (n = 41), chart-stimulated recall (semistructured interviews with the chart available; n = 42), or behavioral rehearsal (simulated role-plays; n = 43). All participating clinicians completed fidelity assessments for up to three sessions with three different clients that were recruited from clinicians' caseloads (n = 288; M age = 13.39 years SD = 3.89; 41.7% female); sessions were also audio-recorded and coded for comparison to determine the most accurate method. All fidelity measures had parallel scales that yielded an adherence maximum score (i.e., the highest-rated intervention in a session), a mean of techniques observed, and a count total of observed techniques. Results of three-level mixed effects regression models indicated that behavioral rehearsal produced comparable scores to observation for all adherence scores (all ps > .01), indicating no difference between behavioral rehearsal and observation. Self-report and chart-stimulated recall overestimated adherence compared to observation (ps < .01). Overall, findings suggested that behavioral rehearsal indexed CBT adherence comparably to direct observation, the gold-standard, in pediatric populations. Behavioral rehearsal may at times be able to replace the need for resource-intensive direct observation in implementation research and practice.

Performance of a Supervisor Observational Coding System and an Audit and Feedback Intervention.

Workplace-based clinical supervision is common in community based mental health care for youth and families and could be a leveraged to scale and improve the implementation of evidence-based treatment (EBTs). Accurate methods are needed to measure, monitor, and support supervisor performance with limited disruption to workflow. Audit and Feedback (A&F) interventions may offer some promise in this regard. The study-a randomized controlled trial with 60 clinical supervisors measured longitudinally for 7 months-had two parts: (1) psychometric evaluation of an observational coding system for measuring adherence and competence of EBT supervision and (2) evaluation of an experimental Supervisor Audit and Feedback (SAF) intervention on outcomes of supervisor adherence and competence. All supervisors recorded and uploaded weekly supervision sessions for 7 months, and those in the experimental condition were provided a single, monthly web-based feedback report. Psychometric performance was evaluated using measurement models based in Item Response Theory, and the effect of the SAF intervention was evaluated using mixed-effects regression models. The observational instrument performed well across psychometric indicators of dimensionality, rating scale functionality, and item fit; however, coder reliability was lower for competence than for adherence. Statistically significant A&F effects were largely in the expected directions and consistent with hypotheses. The observational coding system performed well, and a monthly electronic feedback report showed promise in maintaining or improving community-based clinical supervisors' adherence and, to a lesser extent, competence. Limitations discussed include unknown generalizability to the supervision of other EBTs.

Stakeholder intention to engage in fidelity measurement methods in community mental health settings: A mixed methods study.

Background: The current gold standard for measuring fidelity (specifically, adherence) to cognitive behavioral therapy (CBT) is direct observation, a costly, resource-intensive practice that is not feasible for many community organizations to implement regularly. Recent research indicates that behavioral rehearsal (i.e., role-play between clinician and individual with regard to session delivery) and chart-stimulated recall (i.e., brief structured interview between clinician and individual about what they did in session; clinicians use the client chart to prompt memory) may provide accurate and affordable alternatives for measuring adherence to CBT in such settings, with behavioral rehearsal yielding greater correspondence with direct observation. Methods: Drawing on established causal theories from social psychology and leading implementation science frameworks, this study evaluates stakeholders' intention to use behavioral rehearsal and chart-stimulated recall. Specifically, we measured attitudes, self-efficacy, and subjective norms toward using each, and compared these factors across the two methods. We also examined the relationship between attitudes, self-efficacy, subjective norms, and intention to use each method. Finally, using an integrated approach we asked stakeholders to discuss their perception of contextual factors that may influence beliefs about using each method. These data were collected from community-based supervisors (n = 17) and clinicians (n = 66). Results: Quantitative analyses suggest moderately strong intention to use both methods across stakeholders. There were no differences in supervisors' or clinicians' attitudes, self-efficacy, subjective norms, or intention across methods. More positive attitudes and greater reported subjective norms were associated with greater reported intention to use either measure. Qualitative analyses identified participants' specific beliefs about using each fidelity measure in their organization, and results were organized using the Consolidated Framework for Implementation Research. Conclusions: Strategies are warranted to overcome or minimize potential barriers to using fidelity measurement methods and to further increase the strength of intention to use them.Plain Language Summary: The best way to measure fidelity, or how closely a clinician follows the protocol, to Cognitive Behavioral Therapy (CBT) is watching the session. This is an expensive practice that is not feasible for many community organizations to do regularly. Recent research indicates that behavioral rehearsal, or a role-play between the clinician and individual with regard to session delivery, and chart-stimulated recall, or a brief discussion between an individual and the clinician about what they did in session with the clinician having access to the chart to help them remember, may provide accurate and affordable alternatives for measuring fidelity to CBT. We just completed a study demonstrating that both methods are promising, with behavioral rehearsal offering scores that are the most similar to watching the session. Drawing on established theories from social psychology and leading implementation science frameworks, this study evaluates future supervisor and clinician motivation to use these fidelity measurement methods. Specifically, we measured supervisor (n = 17) and clinician (n = 66) attitudes, norms, self-efficacy, intentions, and anticipated barriers and facilitators to using each of these fidelity measurement tools. Quantitative and qualitative analyses suggest similar intention to use both methods, and concerns about barriers to using each method. Further research is warranted to minimize the burden associated with implementing fidelity measurement methods and deploying strategies to increase use.

Improving the feasibility of fidelity measurement for community-based quality assurance: Partial- versus full-session observations of supervisor adherence and competence.

Background: Clinical supervision is a common quality assurance method for supporting the implementation and sustainment of evidence-based interventions (EBIs) in community mental health settings. However, assessing and supporting supervisor fidelity requires efficient and effective measurement methods. This study evaluated two observational coding approaches that are potentially more efficient than coding full sessions: a randomly selected 15-min segment and the first case discussion of the session. Method: Data were leveraged from a randomized trial of an Audit and Feedback (A&F) intervention for supervisor Adherence and Competence. Supervisors (N = 57) recorded and uploaded weekly group supervision sessions for 7 months, with one session observationally coded each month (N = 374). Of the coded sessions, one was randomly selected for each supervisor, and a random 15-min segment was coded. Additionally, the first case discussion was coded for the full sample of sessions. Results: Across all models (and controlling for the proportion of the session covered by the partial observation), Adherence and Competence scores from partial observations were positively and significantly associated with scores from full sessions. In all cases, partial observations were most accurate when the level of Adherence and Competence was moderate. At lower levels, partial observations were underestimates, and at higher levels, they were overestimates. Conclusions: The results suggest that efficient observational measurement can be achieved while retaining a general level of measurement effectiveness. Practically, first-case discussions are easier to implement, whereas 15-min segments have fewer potential threats to validity. Evaluation of resource requirements is needed, along with determining whether A&F effects are retained if feedback is based on partial observations. Nevertheless, more efficient observational coding could increase the feasibility of routine fidelity monitoring and quality assurance strategies, including A&F, which ultimately could support the implementation and sustainment of effective supervision practices and EBIs in community practice settings.Plain Language Summary: When delivering evidence-based mental health interventions in community-based practice settings, a common quality assurance method is clinical supervision. To support supervisors, assessment methods are needed, and those methods need to be both efficient and effective. Ideally, supervision sessions would be recorded, and trained coders would rate the supervisor's use of specific strategies. In most settings, though, this requires too many resources. The present study evaluated a more efficient approach. The data came from an existing randomized trial of an Audit and Feedback intervention for enhancing supervisor Adherence and Competence. This included 57 supervisors and 374 sessions across seven months of monitoring. Instead of rating full supervision sessions, a more efficient approach was to have coders rate partial sessions. Two types of partial observations were considered: a randomly selected 15-minute segment of the session and the first case discussion of the session. The aim was to see if partial observations and full observations led to similar conclusions about Adherence and Competence. In all cases, they did. The scores were most similar for sessions with moderate levels of Adherence and Competence. If Adherence and Competence were low, partial observations were underestimates, but if they were high, partial observations were overestimates. Observing partial sessions is more efficient, but in terms of accuracy, the benefits and limitations should be evaluated in light of how the scores will be used. Additionally, future research should consider whether Audit and Feedback interventions have the same effect if feedback is based on observations of partial sessions.

The TPOCS-self-reported Therapist Intervention Fidelity for Youth (TPOCS-SeRTIFY): A case study of pragmatic measure development.

Background: Developing pragmatic assessment tools to measure clinician use of evidence-based practices is critical to advancing implementation of evidence-based practices in mental health. This case study details our community-partnered process of developing the Therapy Process Observation Coding Scale-Self-Reported Therapist Intervention Fidelity for Youth (TPOCS-SeRTIFY), a pragmatic, clinician-report instrument to measure cognitive behavioral therapy (CBT) delivery. Approach: We describe a five-step community-partnered development process. Initial goals were to create a self-report instrument that paralleled an existing direct observation measure of clinician delivery of CBT use to facilitate later assessment of measure performance. Cognitive interviews with community clinicians (n = 6) and consultation with CBT experts (n = 6) were used to enhance interpretability and usability as part of an iterative refinement process. The instrument was administered to 247 community clinicians along with an established self-reported measure of clinician delivery of CBT and other treatments to assess preliminary psychometric performance. Preliminary psychometrics were promising. Conclusion: Our community-partnered development process showed promising success and can guide future development of pragmatic implementation measures both to facilitate measurement of ongoing implementation efforts and future research aimed at building learning mental health systems. Plain language summary: Developing brief, user-friendly, and accurate tools to measure how therapists deliver cognitive behavioral therapy (CBT) in routine practice is important for advancing the reach of CBT into community settings. To date, developing such "pragmatic" measures has been difficult. There is little known about how researchers can best develop these types of assessment tools so that they (1) are easy for clinicians in practice to use and (2) provide valid and useful information about implementation outcomes. As a result, there are few well-validated measures in existence that measure therapist use of CBT that are feasible for use in community practice. This paper contributes to the literature by describing our community-partnered process for developing a measure of therapist use of CBT (Therapy Process Observation Coding Scale -Self-Reported Therapist Intervention Fidelity for Youth; TPOCS-SeRTIFY). This descriptive case study outlines the community-partnered approach we took to develop this measure. This case study will contribute to future research by serving as a guide to others aiming to develop pragmatic implementation measures. In addition, the TPOCS-SeRTIFY is a pragmatic measure of clinician use of CBT that holds promise for its use by both researchers and clinicians to measure the success of CBT implementation efforts.

Festschrift for Leonard Bickman: Introduction to The Future of Children's Mental Health Services Special Issue.

This introductory article describes the genesis of the Festschrift for Leonard Bickman and of this Festschrift special issue entitled, The Future of Children's Mental Health Services. The special issue includes a collection of 11 original children's mental health services research articles, broadly organized in accordance with three themes (i.e., Improving Precision and Use of Service Data to Guide Policy and Practice, Implementation and Dissemination, and Preparing for Innovation), followed by an interview-style article with Bickman. Then follows a featured manuscript by Bickman himself, three invited commentaries, and a compilation of letters and notes in which colleagues reflect on his career and on their experiences of him. The introduction concludes with a few thoughts about the future of children's mental health services portended by the extraordinary scholarly contributions of Bickman and those who have been inspired by him.

Sustainment of Trauma-Focused and Evidence-Based Practices Following Learning Collaborative Implementation.

Given the need to develop and validate effective implementation models that lead to sustainable improvements, we prospectively examined changes in attitudes, behaviors, and perceived organizational support during and after statewide Community-Based Learning Collaboratives (CBLCs) promoting trauma-focused evidence-based practices (EBPs). Participants (N = 857; i.e., 492 clinicians, 218 brokers, and 139 senior leaders) from 10 CBLCs completed surveys pre- and post-CBLC; a subsample (n = 146) completed a follow-up survey approximately two years post-CBLC. Results indicated (a) medium, sustained increases in clinician-reported use of trauma-focused EBPs, (b) medium to large, sustained increases in perceived organizational support for trauma-focused EBPs, and (c) trivial to small, sustained increases in perceived organizational support for EBPs broadly. In contrast, clinician-reported overall attitudes towards EBPs decreased to a trivial degree pre- to post-CBLC, but then increased to a small, statistically significant degree from post-CBLC to follow-up. Notably, the degree of perceived improvements in organizational support for general and trauma-focused EBPs varied by professional role. Findings suggest the CBLC implementation strategies may both increase and sustain provider practices and organizational support towards EBPs, particularly those EBPs a CBLC explicitly targets.

Editors' welcome to Implementation Research and Practice.

This editorial introduces the new, online-only, open-access, international journal, Implementation Research and Practice (IRP), jointly published by SAGE and the Society for Implementation Research Collaboration (SIRC). IRP provides rapid publication of interdisciplinary research that advances the implementation in diverse contexts of effective approaches to assess, prevent, and treat mental health, substance use, or other addictive behaviors, in the general population or among those at risk or suffering from these disorders. IRP welcomes a variety of paper types designed to publish empirical, methodological, conceptual, and practical advances, as well as syntheses and perspectives intended to inspire new directions for future research. Consistent with the title of the journal, IRP welcomes manuscripts that stimulate, curate, and align research, policy, and practice.

Correction to: A repeated cross-sectional study of clinicians' use of psychotherapy techniques during 5 years of a system-wide effort to implement evidence-based practices in Philadelphia.

.

A repeated cross-sectional study of clinicians' use of psychotherapy techniques during 5 years of a system-wide effort to implement evidence-based practices in Philadelphia.

BACKGROUND: Little work investigates the effect of behavioral health system efforts to increase use of evidence-based practices or how organizational characteristics moderate the effect of these efforts. The objective of this study was to investigate clinician practice change in a system encouraging implementation of evidence-based practices over 5 years and how organizational characteristics moderate this effect. We hypothesized that evidence-based techniques would increase over time, whereas use of non-evidence-based techniques would remain static. METHOD: Using a repeated cross-sectional design, data were collected three times from 2013 to 2017 in Philadelphia's public behavioral health system. Clinicians from 20 behavioral health outpatient clinics serving youth were surveyed three times over 5 years (n = 340; overall response rate = 60%). All organizations and clinicians were exposed to system-level support provided by the Evidence-based Practice Innovation Center from 2013 to 2017. Additionally, approximately half of the clinicians participated in city-funded evidence-based practice training initiatives. The main outcome included clinician self-reported use of cognitive-behavioral and psychodynamic techniques measured by the Therapy Procedures Checklist-Family Revised. RESULTS: Clinicians were 80% female and averaged 37.52 years of age (SD = 11.40); there were no significant differences in clinician characteristics across waves (all ps > .05). Controlling for organizational and clinician covariates, average use of CBT techniques increased by 6% from wave 1 (M = 3.18) to wave 3 (M = 3.37, p = .021, d = .29), compared to no change in psychodynamic techniques (p = .570). Each evidence-based practice training initiative in which clinicians participated predicted a 3% increase in CBT use (p = .019) but no change in psychodynamic technique use (p = .709). In organizations with more proficient cultures at baseline, clinicians exhibited greater increases in CBT use compared to organizations with less proficient cultures (8% increase vs. 2% decrease, p = .048). CONCLUSIONS: System implementation of evidence-based practices is associated with modest changes in clinician practice; these effects are moderated by organizational characteristics. Findings identify preliminary targets to improve implementation.

Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study.

BACKGROUND: Identifying scalable strategies for assessing fidelity is a key challenge in implementation science. However, for psychosocial interventions, the existing, reliable ways to test treatment fidelity quality are often labor intensive, and less burdensome strategies may not reflect actual clinical practice. Cognitive behavioral therapies (CBTs) provide clinicians with a set of effective core elements to help treat a multitude of disorders, which, evidence suggests, need to be delivered with fidelity to maximize potential client impact. The current "gold standard" for rating CBTs is rating recordings of therapy sessions, which is extremely time-consuming and requires a substantial amount of initial training. Although CBTs can vary based on the target disorder, one common element employed in most CBTs is the use of worksheets to identify specific behaviors and thoughts that affect a client's ability to recover. The present study will develop and evaluate an innovative new approach to rate CBT fidelity, by developing a universal CBT scoring system based on worksheets completed in therapy sessions. METHODS: To develop a scoring system for CBT worksheets, we will compile common CBT elements from a variety of CBT worksheets for a range of psychiatric disorders and create adherence and competence measures. We will collect archival worksheets from past studies to test the scoring system and assess test-retest reliability. To evaluate whether CBT worksheet scoring accurately reflects clinician fidelity, we will recruit clinicians who are engaged in a CBT for depression, anxiety, and/or posttraumatic stress disorder. Clinicians and clients will transmit routine therapy materials produced in session (e.g., worksheets, clinical notes, session recordings) to the study team after each session. We will compare observer-rated fidelity, clinical notes, and fidelity-rated worksheets to identify the most effective and efficient method to assess clinician fidelity. Clients will also be randomly assigned to either complete the CBT worksheets on paper forms or on a mobile application (app) to learn if worksheet format influences clinician and client experience or differs in terms of reflecting fidelity. DISCUSSION: Scoring fidelity using CBT worksheets may allow clinics to test fidelity in a short and effective manner, enhancing continuous quality improvement in the workplace. Clinicians and clinics can use such data to improve clinician fidelity in real time, leading to improved patient outcomes. TRIAL REGISTRATION: ClinicalTrials.gov NCT03479398 . Retrospectively registered March 20, 2018.

Community-Based Learning Collaboratives and Participant Reports of Interprofessional Collaboration, Barriers to, and Utilization of Child Trauma Services.

Given the high prevalence and severe consequences of child trauma, effective implementation strategies are needed to increase the availability and utilization of evidence-based child trauma services. One promising strategy, the Community-Based Learning Collaborative (CBLC), augments traditional Learning Collaborative activities with a novel set of community-focused strategies. This prospective, observational study examined pre-to post-changes in CBLC participant reports of interprofessional collaboration (IPC), barriers to, and utilization of evidence-based child trauma treatment in their communities. Participants of five CBLCs from a statewide dissemination initiative, comprising 572 child abuse professionals (296 clinicians, 168 brokers, and 108 senior leaders), were surveyed pre-and post-CBLC participation. Results suggested that CBLCs significantly decreased barriers to child trauma treatment and significantly increased IPC and perceived utilization of evidence-based child trauma treatment. Further, changes in barriers partially mediated this relationship. Finally, small to medium differences in participants' reports were detected, such that senior leaders perceived significantly greater IPC than clinicians and brokers did, while brokers perceived significantly greater barriers to child trauma treatment than clinicians and senior leaders did. Collectively, these preliminary findings suggest the CBLC implementation model-which augments traditional Learning Collaborative models with a focus on fostering IPC-can reduce barriers and increase the utilization of evidence-based mental health treatment services.

A randomized trial to identify accurate and cost-effective fidelity measurement methods for cognitive-behavioral therapy: project FACTS study protocol.

BACKGROUND: This randomized trial will compare three methods of assessing fidelity to cognitive-behavioral therapy (CBT) for youth to identify the most accurate and cost-effective method. The three methods include self-report (i.e., therapist completes a self-report measure on the CBT interventions used in session while circumventing some of the typical barriers to self-report), chart-stimulated recall (i.e., therapist reports on the CBT interventions used in session via an interview with a trained rater, and with the chart to assist him/her) and behavioral rehearsal (i.e., therapist demonstrates the CBT interventions used in session via a role-play with a trained rater). Direct observation will be used as the gold-standard comparison for each of the three methods. METHODS/DESIGN: This trial will recruit 135 therapists in approximately 12 community agencies in the City of Philadelphia. Therapists will be randomized to one of the three conditions. Each therapist will provide data from three unique sessions, for a total of 405 sessions. All sessions will be audio-recorded and coded using the Therapy Process Observational Coding System for Child Psychotherapy-Revised Strategies scale. This will enable comparison of each measurement approach to direct observation of therapist session behavior to determine which most accurately assesses fidelity. Cost data associated with each method will be gathered. To gather stakeholder perspectives of each measurement method, we will use purposive sampling to recruit 12 therapists from each condition (total of 36 therapists) and 12 supervisors to participate in semi-structured qualitative interviews. DISCUSSION: Results will provide needed information on how to accurately and cost-effectively measure therapist fidelity to CBT for youth, as well as important information about stakeholder perspectives with regard to each measurement method. Findings will inform fidelity measurement practices in future implementation studies as well as in clinical practice. TRIAL REGISTRATION: NCT02820623 , June 3rd, 2016.

Testing the Community-Based Learning Collaborative (CBLC) implementation model: a study protocol.

BACKGROUND: High rates of youth exposure to violence, either through direct victimization or witnessing, result in significant health/mental health consequences and high associated lifetime costs. Evidence-based treatments (EBTs), such as Trauma-Focused Cognitive Behavioral Therapy (TF-CBT), can prevent and/or reduce these negative effects, yet these treatments are not standard practice for therapists working with children identified by child welfare or mental health systems as needing services. While research indicates that collaboration among child welfare and mental health services sectors improves availability and sustainment of EBTs for children, few implementation strategies designed specifically to promote and sustain inter-professional collaboration (IC) and inter-organizational relationships (IOR) have undergone empirical investigation. A potential candidate for evaluation is the Community-Based Learning Collaborative (CBLC) implementation model, an adaptation of the Learning Collaborative which includes strategies designed to develop and strengthen inter-professional relationships between brokers and providers of mental health services to promote IC and IOR and achieve sustained implementation of EBTs for children within a community. METHODS/DESIGN: This non-experimental, mixed methods study involves two phases: (1) analysis of existing prospective quantitative and qualitative quality improvement and project evaluation data collected pre and post, weekly, and monthly from 998 participants in one of seven CBLCs conducted as part of a statewide initiative; and (2) Phase 2 collection of new quantitative and qualitative (key informant interviews) data during the funded study period to evaluate changes in relations among IC, IOR, social networks and the penetration and sustainment of TF-CBT in targeted communities. Recruitment for Phase 2 is from the pool of 998 CBLC participants to achieve a targeted enrollment of n = 150. Study aims include: (1) Use existing quality improvement (weekly/monthly online surveys; pre-post surveys; interviews) and newly collected quantitative (monthly surveys) and qualitative (key informant interviews) data and social network analysis to test whether CBLC strategies are associated with penetration and sustainment of TF-CBT; and (2) Use existing quantitative quality improvement (weekly/monthly on-line surveys; pre/post surveys) and newly collected qualitative (key informant interviews) data and social network analysis to test whether CBLC strategies are associated with increased IOR and IC intensity. DISCUSSION: The proposed research leverages an on-going, statewide implementation initiative to generate evidence about implementation strategies needed to make trauma-focused EBTs more accessible to children. This study also provides feasibility data to inform an effectiveness trial that will utilize a time-series design to rigorously evaluate the CBLC model as a mechanism to improve access and sustained use of EBTs for children.

Redesigning community mental health services for urban children: Supporting schooling to promote mental health.

OBJECTIVE: This study examined a school- and home-based mental health service model, Links to Learning, focused on empirical predictors of learning as primary goals for services in high-poverty urban communities. METHOD: Teacher key opinion leaders were identified through sociometric surveys and trained, with mental health providers and parent advocates, on evidence-based practices to enhance children's learning. Teacher key opinion leaders and mental health providers cofacilitated professional development sessions for classroom teachers to disseminate 2 universal (Good Behavior Game, peer-assisted learning) and 2 targeted (Good News Notes, Daily Report Card) interventions. Group-based and home-based family education and support were delivered by mental health providers and parent advocates for children in kindergarten through 4th grade diagnosed with 1 or more disruptive behavior disorders. Services were Medicaid-funded through 4 social service agencies (N = 17 providers) in 7 schools (N = 136 teachers, 171 children) in a 2 (Links to Learning vs. services as usual) × 6 (pre- and posttests for 3 years) longitudinal design with random assignment of schools to conditions. Services as usual consisted of supported referral to a nearby social service agency. RESULTS: Mixed effects regression models indicated significant positive effects of Links to Learning on mental health service use, classroom observations of academic engagement, teacher report of academic competence and social skills, and parent report of social skills. Nonsignificant between-groups effects were found on teacher and parent report of problem behaviors, daily hassles, and curriculum-based measures. Effects were strongest for young children, girls, and children with fewer symptoms. CONCLUSION: Community mental health services targeting empirical predictors of learning can improve school and home behavior for children living in high-poverty urban communities.

Predictors of community therapists' use of therapy techniques in a large public mental health system.

IMPORTANCE: Few studies have examined the effects of individual and organizational characteristics on the use of evidence-based practices in mental health care. Improved understanding of these factors could guide future implementation efforts to ensure effective adoption, implementation, and sustainment of evidence-based practices. OBJECTIVE: To estimate the relative contribution of individual and organizational factors on therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques within the context of a large-scale effort to increase use of evidence-based practices in an urban public mental health system serving youth and families. DESIGN, SETTING, AND PARTICIPANTS: In this observational, cross-sectional study of 23 organizations, data were collected from March 1 through July 25, 2013. We used purposive sampling to recruit the 29 largest child-serving agencies, which together serve approximately 80% of youth receiving publically funded mental health care. The final sample included 19 agencies with 23 sites, 130 therapists, 36 supervisors, and 22 executive administrators. MAIN OUTCOMES AND MEASURES: Therapist self-reported use of cognitive-behavioral, family, and psychodynamic therapy techniques, as measured by the Therapist Procedures Checklist-Family Revised. RESULTS: Individual factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 16%; family therapy techniques, 7%; and psychodynamic therapy techniques, 20%. Organizational factors accounted for the following percentages of the overall variation: cognitive-behavioral therapy techniques, 23%; family therapy techniques, 19%; and psychodynamic therapy techniques, 7%. Older therapists and therapists with more open attitudes were more likely to endorse use of cognitive-behavioral therapy techniques, as were those in organizations that had spent fewer years participating in evidence-based practice initiatives, had more resistant cultures, and had more functional climates. Women were more likely to endorse use of family therapy techniques, as were those in organizations employing more fee-for-service staff and with more stressful climates. Therapists with more divergent attitudes and less knowledge about evidence-based practices were more likely to use psychodynamic therapy techniques. CONCLUSIONS AND RELEVANCE: This study suggests that individual and organizational factors are important in explaining therapist behavior and use of evidence-based practices, but the relative importance varies by therapeutic technique.

From Practice to Evidence in Child Welfare: Model Specification and Fidelity Measurement of Team Decisionmaking.

Fidelity measurement methods have traditionally been used to develop and evaluate the effects of psychosocial treatments and, more recently, their implementation in practice. The fidelity measurement process can also be used to operationally define and specify components of emerging but untested practices outside the realm of conventional treatment. Achieving optimal fidelity measurement effectiveness (scientific validity and reliability) and efficiency (feasibility and relevance in routine care contexts) is challenging. The purpose of this paper is to identify strategies to address these challenges in child welfare system practices. To illustrate the challenges, and operational steps to address them, we present a case example using the "Team Decisionmaking" (TDM; Annie E. Casey Foundation) intervention. This intervention has potential utility for decreasing initial entry into and time spent in foster care and increasing rates of reunification and relative care. While promising, the model requires rigorous research to refine knowledge regarding the relationship between intervention components and outcomes-research that requires fidelity measurement. The intent of this paper is to illustrate how potentially generalizable steps for developing effective and efficient fidelity measurement methods can be used to more clearly define and test the effects of child welfare system practices.

Long-term outcomes for the Child STEPs randomized effectiveness trial: a comparison of modular and standard treatment designs with usual care.

OBJECTIVE: This article reports outcomes from the Child STEPs randomized effectiveness trial conducted over a 2-year period to gauge the longer term impact of protocol design on the effectiveness of evidence-based treatment procedures. METHOD: An ethnoracially diverse sample of 174 youths ages 7- 13 (N = 121 boys) whose primary clinical concerns involved diagnoses or clinical elevations related to anxiety, depression, or disruptive behavior were treated by community therapists randomly assigned to 1 of 3 conditions: (a) standard, which involved the use of 1 or more of 3 manualized evidence-based treatments, (b) modular, which involved a single modular protocol (Modular Approach to Treatment of Children With Anxiety, Depression, or Conduct Problems; MATCH) having clinical procedures similar to the standard condition but flexibly selected and sequenced using a guiding clinical algorithm, and (c) usual care. RESULTS: As measured with combined Child Behavior Checklist and Youth Self-Report Total Problems, Internalizing, and Externalizing scales, the rate of improvement for youths in the modular condition was significantly better than for those in usual care. On a measure of functional impairment (Brief Impairment Scale), no significant differences were found among the 3 conditions. Analysis of service utilization also showed no significant differences among conditions, with almost half of youths receiving some additional services in the 1st year after beginning treatment, and roughly one third of youths in the 2nd year. CONCLUSIONS: Overall, these results extend prior findings, supporting incremental benefits of MATCH over usual care over a 2-year period.

Policy to implementation: evidence-based practice in community mental health--study protocol.

BACKGROUND: Evidence-based treatments (EBTs) are not widely available in community mental health settings. In response to the call for implementation of evidence-based treatments in the United States, states and counties have mandated behavioral health reform through policies and other initiatives. Evaluations of the impact of these policies on implementation are rare. A systems transformation about to occur in Philadelphia, Pennsylvania, offers an important opportunity to prospectively study implementation in response to a policy mandate. METHODS/DESIGN: Using a prospective sequential mixed-methods design, with observations at multiple points in time, we will investigate the responses of staff from 30 community mental health clinics to a policy from the Department of Behavioral Health encouraging and incentivizing providers to implement evidence-based treatments to treat youth with mental health problems. Study participants will be 30 executive directors, 30 clinical directors, and 240 therapists. Data will be collected prior to the policy implementation, and then at two and four years following policy implementation. Quantitative data will include measures of intervention implementation and potential moderators of implementation (i.e., organizational- and leader-level variables) and will be collected from executive directors, clinical directors, and therapists. Measures include self-reported therapist fidelity to evidence-based treatment techniques as measured by the Therapist Procedures Checklist-Revised, organizational variables as measured by the Organizational Social Context Measurement System and the Implementation Climate Assessment, leader variables as measured by the Multifactor Leadership Questionnaire, attitudes towards EBTs as measured by the Evidence-Based Practice Attitude Scale, and knowledge of EBTs as measured by the Knowledge of Evidence- Based Services Questionnaire. Qualitative data will include semi-structured interviews with a subset of the sample to assess the implementation experience of high-, average-, and low-performing agencies. Mixed methods will be integrated through comparing and contrasting results from the two methods for each of the primary hypotheses in this study. DISCUSSION: Findings from the proposed research will inform both future policy mandates around implementation and the support required for the success of these policies, with the ultimate goal of improving the quality of treatment provided to youth in the public sector.