Understanding pathways from implementation to sustainment: a longitudinal, mixed methods analysis of promising practices implemented in the Veterans Health Administration.

BACKGROUND: The Veterans Health Administration (VHA) is the United States largest learning health system. The Diffusion of Excellence (DoE) program is a large-scale model of diffusion that identifies and diffuses evidence-informed practices across VHA. During the period of 2016-2021, 57 evidence-informed practices were implemented across 82 VHA facilities. This setting provides a unique opportunity to understand sustainment determinants and pathways. Our objective was to characterize the longitudinal pathways of practices as they transition from initial implementation to long-term sustainment at each facility. METHODS: A longitudinal, mixed-methods evaluation of 82 VHA facilities. Eighty-two facility representatives, chosen by leadership as points-of-contact for 57 DoE practices, were eligible for post-implementation interviews and annual sustainment surveys. Primary outcomes (implementation, sustainment), and secondary outcomes (institutionalization, effectiveness, anticipated sustainment) at four time-points were collected. We performed descriptive statistics and directed content analysis using Hailemariam et al.'s factors influencing sustainment. RESULTS: After approximately five years post-implementation (e.g., 2021 sustainment outcomes), of the 82 facilities, about one-third fully sustained their practice compared to one-third that did not fully sustain their practice because it was in a "liminal" stage (neither sustained nor discontinued) or permanently discontinued. The remaining one-third of facilities had missing 2021 sustainment outcomes. A higher percentage of facilities (70%) had inconsistent primary outcomes (changing over time) compared to facilities (30%) with consistent primary outcomes (same over time). Thirty-four percent of facilities with sustained practices reported resilience since they overcame implementation and sustainment barriers. Facilities with sustained practices reported more positive secondary outcomes compared to those that did not sustain their practice. Key factors facilitating practice sustainment included: demonstrating practice effectiveness/benefit, sufficient organizational leadership, sufficient workforce, and adaptation/alignment with local context. Key factors hindering practice sustainment included: insufficient workforce, not able to maintain practice fidelity/integrity, critical incidents related to the COVID-19 pandemic, organizational leadership did not support sustainment of practice, and no ongoing support. CONCLUSIONS: We identified diverse pathways from implementation to sustainment, and our data underscore that initial implementation outcomes may not determine long-term sustainment outcomes. This longitudinal evaluation contributes to understanding impacts of the DoE program, including return on investment, achieving learning health system goals, and insights into achieving high-quality healthcare in VHA.

Technical Assistance for Systemic Change: Lessons Learned From a National Technical Assistance Center.

Despite the millions of dollars awarded annually by the United States Department of Education to build implementation capacity through technical assistance (TA), data on TA effectiveness are severely lacking. Foundational to the operationalization and consistent research on TA effectiveness is the development and use of standardized TA core competencies, practices, and structures. Despite advances toward a consistent definition of TA, a gap still exists in understanding how these competencies are used within an operationalized set of TA practices to produce targeted outcomes at both individual and organizational levels to facilitate implementation of evidence-based practices. The current article describes key insights derived from the evaluation of an operationalized set of TA practices used by a nationally funded TA center, the State Implementation & Scaling Up of Evidence Based Practices (SISEP) Center. The TA provided by the Center supports the uptake of evidence-based practices in K-12 education for students with disabilities. Lessons learned include: (1) the need to understand the complexities and dependencies of operationalizing TA both longitudinally and at multiple levels of the system (state, regional, local); (2) the relative importance of building general and innovation-specific capacity for implementation success; (3) the value of using a co-design and participatory approach for effective TA delivery; (4) the need to develop TA providers' educational and implementation fluency across areas and levels of the system receiving TA; and (5) the need to ensure coordination and alignment of TA providers from different centers. Gaining an understanding into optimal TA practices will not only provide clarity of definition fundamental to TA research, but it will also inform the conceptual framing and practice of TA.

Supporting Implementation Through Online Learning Communities: Lessons Learned From a National Training and Technical Assistance Network.

Recent implementation science frameworks highlight the role of training and technical assistance (TTA) in building workforce capacity to implement evidence-based practices (EBPs). However, evaluation of TTA is limited. We describe three case examples that highlight TTA by three regional centers in the national Mental Health Technology Transfer Center (MHTTC) network. Each MHTTC formed Learning Communities (LCs) to facilitate connections among behavioral health professionals with the goals of sharing implementation strategies, discussing best-practices, and developing problem solving techniques. Data on outcomes were collected through a combination of self-report surveys and qualitative interviews. LC participants reported strong connectedness, gains in knowledge and skills, improvements in implementation capacity, and intentions to advocate for organizational and systems-level change. Furthermore, across the case examples, we identified LC characteristics that are associated with participant perceptions of outcomes, including tailoring LC content to workforce needs, providing culturally relevant information, engaging leaders, forming connections among participants and trainers, and challenging participants' current workplace practices. These findings are interpreted through the lens of the Interactive Systems Framework, which focuses on how TTA, such as LCs, can facilitate connections between the theoretical and empirical foundations of interventions and the practices of implementing interventions in real-world settings to advance workforce capacity.

The Role of Cooperative Extension in Delivering Training and Technical Assistance to Support Evidence-Based Behavioral Health Practices in Rural Communities.

The opioid epidemic in the United States continues to disproportionately affect those in rural, compared to urban, areas due to a variety of treatment and recovery barriers. One mechanism to increase capacity of rural-serving providers is through delivery of training and technical assistance (TTA) for evidence-based programs by leveraging the Cooperative Extension System. Guided by the Interactive Systems Framework, the current study evaluates TTA delivered by the Northwest Rural Opioid Technical Assistance Collabroative to opioid prevention, treatment, and recovery providers on short- (satisfaction, anticipated benefit), medium-, (behavioral intention to change current practice), and long-term goals (changes toward adoption of evidence-based practices). We also evaluated differences in short- and medium-term goals by intensity of TTA event and rurality of provider. Surveys of 351 providers who received TTA indicated high levels of satisfaction with TTA events attended, expressed strong agreement that they would benefit from the event, intended to make a professional practice change, and preparation toward implementing changes. Compared to urban-based providers, rural providers reported higher intention to use TTA information to change current practice. We conclude with a review of remaining gaps in the research to practice pipeline and recommendations for moving forward.

Strengthening the Science and Practice of Implementation Support: Evaluating the Effectiveness of Training and Technical Assistance Centers.

Hundreds of millions of dollars are spent each year by U.S. federal agencies for training and technical assistance (TTA) to be delivered by training and technical assistance centers (TTACs) to "delivery system organizations" (e.g., federally qualified health centers, state departments of health, substance abuse treatment centers, schools, and healthcare organizations). TTACs are often requested to help delivery system organizations implement evidence-based interventions. Yet, counterintuitively, TTACs are rarely required to use evidence-based approaches when supporting delivery systems (in the use of evidence-based programs). In fact, evaluations of TTAC activities tend to be minimal; evaluation of technical assistance (if conducted at all) often emphasizes outputs (number of encounters), satisfaction, and self-reports of knowledge gained-more substantive outcomes are not evaluated. The gap between (a) the volume of TTA services being funded and provided and (b) the evaluation of those services is immense and has the potential to be costly. The basic question to be answered is: how effective are TTA services? This article introduces the special issue on Strengthening the Science and Practice of Implementation Support: Evaluating the Effectiveness of Training and Technical Assistance Centers. The special issue promotes 1) knowledge of the state of the art of evaluation of TTACs and 2) advances in what to evaluate in TTA. A major goal of the issue is to improve the science and practice of implementation support, particularly in the areas of TTA.

Inter-organizational alignment and implementation outcomes in integrated mental healthcare for children and adolescents: a cross-sectional observational study.

BACKGROUND: Integrated care involves care provided by a team of professionals, often in non-traditional settings. A common example worldwide is integrated school-based mental health (SBMH), which involves externally employed clinicians providing care at schools. Integrated mental healthcare can improve the accessibility and efficiency of evidence-based practices (EBPs) for vulnerable populations suffering from fragmented traditional care. However, integration can complicate EBP implementation due to overlapping organizational contexts, diminishing the public health impact. Emerging literature suggests that EBP implementation may benefit from the similarities in the implementation context factors between the different organizations in integrated care, which we termed inter-organizational alignment (IOA). This study quantitatively explored whether and how IOA in general and implementation context factors are associated with implementation outcomes in integrated SBMH. METHODS: SBMH clinicians from community-based organizations (CBOs; nclinician = 27) and their proximal student-support school staff (nschool = 99) rated their schools and CBOs (clinician only) regarding general (organizational culture and molar climate) and implementation context factors (Implementation Climate and Leadership), and nine common implementation outcomes (e.g., treatment integrity, service access, acceptability). The levels of IOA were estimated by intra-class correlations (ICCs). We fitted multilevel models to estimate the standalone effects of context factors from CBOs and schools on implementation outcomes. We also estimated the 2-way interaction effects between CBO and school context factors (i.e., between-setting interdependence) on implementation outcomes. RESULTS: The IOA in general context factors exceeded those of implementation context factors. The standalone effects of implementation context factors on most implementation outcomes were larger than those of general context factors. Similarly, implementation context factors between CBOs and schools showed larger 2-way interaction effects on implementation outcomes than general context factors. CONCLUSIONS: This study preliminarily supported the importance of IOA in context factors for integrated SBMH. The findings shed light on how IOA in implementation and general context factors may be differentially associated with implementation outcomes across a broad array of integrated mental healthcare settings.

How do the Substance Abuse and Mental Health Services Administration's Technology Transfer Centers Decide What Evidence-Based Practices to Disseminate and Determine How to Do So? A Cross-Sectional Study of a National Network.

It is important to use evidence-based programs and practices (EBPs) to address major public health issues. However, those who use EBPs in real-world settings often require support in bridging the research-to-practice gap. In the US, one of the largest systems that provides such support is the Substance Abuse and Mental Health Services Administration's (SAMHSA's) Technology Transfer Center (TTC) Network. As part of a large external evaluation of the Network, this study examined how TTCs determine which EBPs to promote and how to promote them. Using semi-structured interviews and pre-testing, we developed a "Determinants of Technology Transfer" survey that was completed by 100% of TTCs in the Network. Because the study period overlapped with the onset of the COVID-19 pandemic, we also conducted a retrospective pre/post-pandemic comparison of determinants. TTCs reported relying on a broad group of factors when selecting EBPs to disseminate and the methods to do so. Stakeholder and target audience input and needs were consistently the most important determinant (both before and during COVID-19), while some other determinants fluctuated around the pandemic (e.g., public health mandates, instructions in the funding opportunity announcements). We discuss implications of the findings for technology transfer and frame the analyses in terms of the Interactive Systems Framework for Dissemination and Implementation.

How can a behavioral economics lens contribute to implementation science?

BACKGROUND: Implementation science in health is an interdisciplinary field with an emphasis on supporting behavior change required when clinicians and other actors implement evidence-based practices within organizational constraints. Behavioral economics has emerged in parallel and works towards developing realistic models of how humans behave and categorizes a wide range of features of choices that can influence behavior. We argue that implementation science can be enhanced by the incorporation of approaches from behavioral economics. Main body First, we provide a general overview of implementation science and ways in which implementation science has been limited to date. Second, we review principles of behavioral economics and describe how concepts from BE have been successfully applied to healthcare including nudges deployed in the electronic health record. For example, de-implementation of low-value prescribing has been supported by changing the default in the electronic health record. We then describe what a behavioral economics lens offers to existing implementation science theories, models and frameworks, including rich and realistic models of human behavior, additional research methods such as pre-mortems and behavioral design, and low-cost and scalable implementation strategies. We argue that insights from behavioral economics can guide the design of implementation strategies and the interpretation of implementation studies. Key objections to incorporating behavioral economics are addressed, including concerns about sustainment and at what level the strategies work. CONCLUSION: Scholars should consider augmenting implementation science theories, models, and frameworks with relevant insights from behavioral economics. By drawing on these additional insights, implementation scientists have the potential to boost efforts to expand the provision and availability of high quality care.

Practice-Oriented Research: An Introduction to New Developments and Future Directions.

Aimed at understanding and improving psychological therapies as they are conducted in clinical routine, practice-oriented research (POR) is now a well-established approach to the scientific foundations of mental health care services. Resting on the accumulation of a wide range of practice-based evidence related to treatment outcome and process, as well as factors associated with the participants of psychotherapy and its context, POR is ripe for new developments - regarding what to investigate and how to investigate it. This paper is the introduction of a series devoted to recent advances and future directions of POR as their pertained to routine outcome monitoring, technologies and artificial intelligence, the integration of constructs and methods from program evaluation and implementation science, and the investigation of populations with limited financial resources across various regions of the world. The series also includes commentaries from two leaders of POR.

A Mixed-methods Examination of Culturally Responsive Adaptation to an Evidence-based Parent-mediated Intervention Implemented for Autistic Children.

Parent-mediated interventions (PMIs) are considered an evidence-based practice for fostering social communication skills in young autistic children and for promoting parent responsivity and empowerment, yet barriers to caregiver engagement are evident when PMIs are implemented within historically underserved community settings. Issues of caregiver engagement can reflect a lack of fit between PMIs and the needs of diverse families. We used a mixed methods approach to examine barriers to participating in an evidence-based PMI, Project ImPACT (Ingersoll & Dvortcsak, 2019), within an outpatient setting, as well as strategies that clinicians reported using to deliver and adapt Project ImPACT for minoritized families. Participants included 134 caregivers of a child 13 to 48 months with autism or other social communication differences and six clinicians delivering Project ImPACT. Findings suggest that caregivers experience barriers to participating in Project ImPACT and that these barriers are associated with caregivers' ability to complete the program. Although quantitative findings indicate that adaptation to Project ImPACT did not differ by caregiver and child background, qualitative findings highlighted that clinicians attempt to deliver Project ImPACT to respond to the needs of families from minoritized backgrounds by actively considering the family's culture, psychosocial experiences, goals, and specific barriers. Further, both qualitative and quantitative findings suggest that culturally responsive care and adaptations may support caregiver engagement, including rapport, trust, buy-in, and attendance. Approaches to center cultural alongside contextual/psychosocial considerations within family-centered care in the implementation of PMIs are also highlighted.

Using Participatory Implementation Science to Advance Health Equity.

Participatory approaches to implementation science (IS) offer an inclusive, collaborative, and iterative perspective on implementing and sustaining evidence-based interventions (EBIs) to advance health equity. This review provides guidance on the principles and practice of participatory IS, which enables academic researchers, community members, implementers, and other actors to collaboratively integrate practice-, community-, and research-based evidence into public health and health care services. With a foundational focus on supporting academics in coproducing knowledge and action, participatory IS seeks to improve health, reduce inequity, and create transformational change. The three main sections of this review provide (a) a rationale for participatory approaches to research in implementation science, (b) a framework for integrating participatory approaches in research utilizing IS theory and methods, and (c) critical considerations for optimizing the practice and impact of participatory IS. Ultimately, participatory approaches can move IS activities beyond efforts to make EBIs work within harmful systems toward transformative solutions that reshape these systems to center equity.

Testing a Theory of Implementation Leadership and Climate Across Autism Evidence-Based Interventions of Varying Complexity.

Prominent theories within the field of implementation science contend that organizational leaders can improve providers' fidelity to evidence-based practices (EBPs) by using focused implementation leadership behaviors that create an organizational climate for EBP implementation. However, this work has been criticized for overreliance on nonspecific, self-report fidelity measures and poor articulation of the boundary conditions that may attenuate leadership and climate's influence. This study tests the predictions of EBP implementation leadership and climate theory on observed fidelity to three school-based EBPs for autism that vary in complexity: pivotal response training (PRT), discrete trial training (DTT), and visual schedules (VS). Educators in kindergarten to third-grade autism support classrooms in 65 schools assessed their principals' EBP implementation leadership and school EBP implementation climate prior to the school year. Mid-school year, trained observers rated educator fidelity to all three interventions. Expert raters confirmed PRT was significantly more complex than DTT or VS using the Intervention Complexity Assessment Tool for Systematic Reviews. Linear regression analyses at the school level indicated principals' increased frequency of EBP implementation leadership predicted a higher school EBP implementation climate, which in turn predicted higher educator fidelity to PRT-however, there was no evidence of a relationship between implementation climate and fidelity to DTT or VS. Comparing principals whose EBP implementation leadership was ±1 SD from the mean, there was a significant indirect association of EBP implementation leadership with PRT fidelity through EBP implementation climate (d = 0.49, 95% CI [0.04, 0.93]). Strategies that target EBP implementation leadership and climate may support fidelity to complex behavioral interventions.

"Change Doesn't Happen by Itself": A Thematic Analysis of First-Level Leaders' Experiences Participating in the Leadership and Organizational Change for Implementation (LOCI) Strategy.

The Leadership and Organizational Change for Implementation (LOCI) strategy is a multifaceted implementation strategy that aims to support successful evidence-based practice (EBP) implementation by fostering effective general leadership, implementation leadership, and implementation climate. How implementation strategies are experienced by participants is important for their utilization and effectiveness in supporting EBP implementation. The current study is the first in-depth qualitative study exploring first-level leaders' experiences of participating in the LOCI strategy. Data were collected as part of a trial where Norwegian child and adult mental health outpatient clinics implemented EBPs for posttraumatic stress disorder (PTSD). Eleven first-level leaders from adult and child clinics participated in semi-structured interviews after completing the LOCI strategy. Data were analyzed through reflexive thematic analysis. The analysis generated four themes related to leaders' experiences of participating in the LOCI strategy: (1) structuring the EBP implementation, (2) taking responsibility for the EBP implementation, (3) interacting with others about the EBP implementation, and (4) becoming aware of EBP implementation and their own leadership. Most participants experienced the LOCI strategy as beneficial for implementing EBPs for PTSD in their clinic. The strategy succeeded in raising awareness of leadership for EBP implementation, and simultaneously provided participants with tools and support for leading the implementation in their clinic. Two participants experienced LOCI as less beneficial than the others. Our results support the strategy's potential to engage and empower first-level leaders to get involved in implementation processes and point to important challenges for future research on implementation strategies.

Mass Critical Care Surge Response During COVID-19: Implementation of Contingency Strategies - A Preliminary Report of Findings From the Task Force for Mass Critical Care.

BACKGROUND: After the publication of a 2014 consensus statement regarding mass critical care during public health emergencies, much has been learned about surge responses and the care of overwhelming numbers of patients during the COVID-19 pandemic. Gaps in prior pandemic planning were identified and require modification in the midst of severe ongoing surges throughout the world. RESEARCH QUESTION: A subcommittee from The Task Force for Mass Critical Care (TFMCC) investigated the most recent COVID-19 publications coupled with TFMCC members anecdotal experience in order to formulate operational strategies to optimize contingency level care, and prevent crisis care circumstances associated with increased mortality. STUDY DESIGN AND METHODS: TFMCC adopted a modified version of established rapid guideline methodologies from the World Health Organization and the Guidelines International Network-McMaster Guideline Development Checklist. With a consensus development process incorporating expert opinion to define important questions and extract evidence, the TFMCC developed relevant pandemic surge suggestions in a structured manner, incorporating peer-reviewed literature, "gray" evidence from lay media sources, and anecdotal experiential evidence. RESULTS: Ten suggestions were identified regarding staffing, load-balancing, communication, and technology. Staffing models are suggested with resilience strategies to support critical care staff. ICU surge strategies and strain indicators are suggested to enhance ICU prioritization tactics to maintain contingency level care and to avoid crisis triage, with early transfer strategies to further load-balance care. We suggest that intensivists and hospitalists be engaged with the incident command structure to ensure two-way communication, situational awareness, and the use of technology to support critical care delivery and families of patients in ICUs. INTERPRETATION: A subcommittee from the TFMCC offers interim evidence-informed operational strategies to assist hospitals and communities to plan for and respond to surge capacity demands resulting from COVID-19.

Cohort Studies.

Editor's note: This is the seventh article in a series on clinical research by nurses. The series is designed to give nurses the knowledge and skills they need to participate in research, step by step. Each column will present the concepts that underpin evidence-based practice-from research design to data interpretation. The articles will be accompanied by a podcast offering more insight and context from the authors. To see all the articles in the series, go to http://links.lww.com/AJN/A204.

The Fuld Institute for EBP Community Core: Supporting the Patient Perspective in EBP for Optimal Health and Wellness.

Health information and communication fall within patient preferences in evidence-based practice. Now more than ever, patients and families in the community have free access to "evidence" and healthcare information on the internet. However, is that information trustworthy, and how can we encourage people to use evidence to promote their optimal health and wellness? The recent rise of global spread of mis- and disinformation through social media outlets has affected public health. There is growing recognition that social media platforms provide magnified podiums leading to unfortunate outcomes. While much work has been done during the COVID-19 pandemic to address health misinformation, there is still much more work to do. We must respond to the widespread misinformation as a collective healthcare community to prevent poor healthcare decisions. Urging the public to be alert to information spread, assess the quality of health information (and whether it is evidence-based), and use shared decision-making tools is a path we can travel together.

Comparison of Teleintegrated Care and Telereferral Care for Treating Complex Psychiatric Disorders in Primary Care: A Pragmatic Randomized Comparative Effectiveness Trial.

Importance: Only one-third of patients with complex psychiatric disorders engage in specialty mental health care, and only one-tenth receive adequate treatment in primary care. Scalable approaches are critically needed to improve access to effective mental health treatments in underserved primary care settings. Objective: To compare 2 clinic-to-clinic interactive video approaches to delivering evidence-based mental health treatments to patients in primary care clinics. Design, Setting, and Participants: This pragmatic comparative effectiveness trial used a sequential, multiple-assignment, randomized trial (SMART) design with patient-level randomization. Adult patients treated at 24 primary care clinics without on-site psychiatrists or psychologists from 12 federally qualified health centers in 3 states who screened positive for posttraumatic stress disorder and/or bipolar disorder and who were not already receiving pharmacotherapy from a mental health specialist were recruited from November 16, 2016, to June 30, 2019, and observed for 12 months. Interventions: Two approaches were compared: (1) telepsychiatry/telepsychology-enhanced referral (TER), where telepsychiatrists and telepsychologists assumed responsibility for treatment, and (2) telepsychiatry collaborative care (TCC), where telepsychiatrists provided consultation to the primary care team. TER included an adaptive intervention (phone-enhanced referral [PER]) for patients not engaging in treatment, which involved telephone outreach and motivational interviewing. Main Outcomes and Measures: Survey questions assessed patient-reported outcomes. The Veterans RAND 12-item Health Survey Mental Component Summary (MCS) score was the primary outcome (range, 0-100). Secondary outcomes included posttraumatic stress disorder symptoms, manic symptoms, depressive symptoms, anxiety symptoms, recovery, and adverse effects. Results: Of 1004 included participants, 701 of 1000 (70.1%) were female, 660 of 994 (66.4%) were White, and the mean (SD) age was 39.4 (12.9) years. Baseline MCS scores were 2 SDs below the US mean; the mean (SD) MCS scores were 39.7 (14.1) and 41.2 (14.2) in the TCC and TER groups, respectively. There was no significant difference in 12-month MCS score between those receiving TCC and TER (ß = 1.0; 95% CI, -0.8 to 2.8; P = .28). Patients in both groups experienced large and clinically meaningful improvements from baseline to 12 months (TCC: Cohen d = 0.81; 95% CI, 0.67 to 0.95; TER: Cohen d = 0.90; 95% CI, 0.76 to 1.04). For patients not engaging in TER at 6 months, there was no significant difference in 12-month MCS score between those receiving PER and TER (ß = 2.0; 95% CI, -1.7 to 5.7; P = .29). Conclusions and Relevance: In this comparative effectiveness trial of patients with complex psychiatric disorders randomized to receive TCC or TER, significantly and substantially improved outcomes were observed in both groups. From a health care system perspective, clinical leadership should implement whichever approach is most sustainable. Trial Registration: ClinicalTrials.gov Identifier: NCT02738944.

Long-Term Outcomes of a Collaborative Regional Evidence-Based Practice Fellowship.

Academic-practice partnerships provide a model for sharing resources, increasing professional knowledge and skills, improving patient outcomes, and strengthening organizational cultures of quality and safety. This article describes the long-term outcomes of a regional collaborative evidence-based practice fellowship. Results reveal the fellowship had a measurable positive impact on fellows' evidence-based practice knowledge and practice, project outcomes, professional growth, and the culture of excellence within partner organizations.

The Elements of Engagement Framework to Improve Organizational Uptake of Evidence-Based Practice.

Considerable time, money, and training efforts in organizations have been spent advancing evidence-based practice (EBP). Adding science to clinical decision making is profound, yet organizational strategies to ensure mainstream use of EBP as a return on the training investment is sparse. The Elements of Engagement Framework addresses organizational dynamics: emotion, engagement, energy, expectations, and execution to normalize implementation of EBP within the organizational culture. [J Contin Educ Nurs. 2021;52(8):355-358.].