Describing implementation outcomes for a virtual community of practice: The ECHO Ontario Mental Health experience.

BACKGROUND: Project ECHO is a virtual education model aimed at building capacity among healthcare providers to support optimal management for a range of health conditions. The expansion of the ECHO model, further amplified by the pandemic, has demonstrated an increased need to evaluate implementation success to ensure that interventions are implemented as planned. This study describes how Proctor et al.'s implementation outcomes (acceptability, adoption, appropriateness, costs, feasibility, fidelity, penetration, and sustainability) were adapted and used to assess the implementation of ECHO Ontario Mental Health (ECHO-ONMH), a mental health-focused capacity-building programme. METHODS: Using Proctor et al.'s implementation outcomes, the authors developed an implementation outcomes framework for ECHO-ONMH more generally. Using this, outcome measures and success thresholds were identified for each outcome for the ECHO-ONMH context, and then applied to evaluate the implementation of ECHO-ONMH using data from the first 4 years of the programme. RESULTS: An ECHO-ONMH implementation outcomes framework was developed using Proctor's implementation outcomes. ECHO-ONMH adapted implementation outcomes suggest that ECHO-ONMH was implemented successfully in all domains except for penetration, which only had participation from 13/14 regions. Acceptability, appropriateness and adoption success thresholds were surpassed for all 4 years, showing strong signs of sustainability. The programme was deemed feasible all 4 years and was found to be more cost-effective. ECHO-ONMH also showed high rates of fidelity to the ECHO model, and high rates of penetration. CONCLUSIONS: This is the first study to use Proctor et al.'s implementation outcomes to describe implementation success for a virtual capacity-building model. The proposed ECHO implementation outcomes framework provides a base for similar interventions to evaluate implementation success, which is an important precursor to understanding learning, service or health outcomes related to the model. Additionally, these findings can act as a benchmark for other international ECHOs and educational programmes.

Building Complex Care Capacity in Primary Care: An Educational Evaluation of Project ECHO Ontario Integrated Mental and Physical Health.

BACKGROUND: Project Extension for Community Healthcare Outcomes (ECHO) is a virtual training and capacity building model that uses videoconferencing to link expert interdisciplinary teams with primary care clinicians in local communities. In this study, we evaluated ECHO Ontario Integrated Mental and Physical Health (ECHO-IMPH). This is the first consultation-liaison psychiatrist-led Project ECHO explicitly designed to support health care providers (HCPs) within primary care in delivering better care for patients with co-occurring mental and physical health needs. We assessed the impact of ECHO-IMPH on HCP engagement, learning, and practice change. METHODS: Using Moore's Evaluation Framework, we used attendance logs and weekly surveys to investigate HCP engagement and satisfaction with ECHO-IMPH, as well as questionnaires to assess impact on their learning, self-efficacy, and practice change with respect to patient care. A pre-post design was used to assess change in the latter. RESULTS: A total of 322 HCPs participated in ECHO-IMPH across five cycles. High mean ratings were observed for satisfaction across all five cycles (4.35 ± 0.59). Precycle and postcycle questionnaires were available for 145 participants, allowing for paired comparison. Mean self-efficacy scores were significantly higher after the cycle than that before (64.26 ± 15.63 to 78.15 ± 11.44; t(144) = 11.61, P < 0.001, d = 1.03). Over 80% of participants reported changes in their professional practice post ECHO-IMPH. CONCLUSIONS: This is the first study to describe the impact of a Project ECHO led by consultation-liaison psychiatrists focused on integrated mental and physical health care and to demonstrate that this can be effective in changing HCP professional practice and self-efficacy, with high engagement and satisfaction.

Virtual Education Program to Support Providers Caring for People With Intellectual and Developmental Disabilities During the COVID-19 Pandemic: Rapid Development and Evaluation Study.

BACKGROUND: People with intellectual and developmental disabilities are at increased health-related risk due to the COVID-19 pandemic. Virtual training programs that support providers in caring for the physical and mental health needs of this population, as well provide psychological support to the providers themselves, are needed during the pandemic. OBJECTIVE: This paper describes the design, delivery, and evaluation of a virtual educational COVID-19-focused Extension for Community Healthcare Outcomes program to support providers during the COVID-19 pandemic in caring for the mental health of people with intellectual and developmental disabilities. METHODS: A rapid design thinking approach was used to develop a 6-session program that incorporates mindfulness practice, a wellness check, COVID-19-related research and policy updates, a didactic presentation on a combination mental health and COVID-19 related topic, and a case-based discussion to encourage practical learning. We used the first 5 outcome levels of Moore's evaluation framework-focusing on participation, satisfaction, learning, self-efficacy, and change in practice-which were rated (out of 5) by care providers from health and disability service sectors, as well as additional reflection measures about innovations to the program. Qualitative feedback from open-text responses from participants were analyzed using modified manifest content analysis. RESULTS: A total of 104 care providers from health and disability service sectors participated in the program. High levels of engagement (81 participants per session on average) and satisfaction (overall satisfaction score: mean 4.31, SD 0.17) were observed. Self-efficacy (score improvement: 19.8%), support, and coping improved. Participants also rated the newly developed COVID-19 program and its innovative components highly. Open text feedback showed participants felt that the Extension for Community Healthcare Outcomes program expanded their knowledge and competency and created a sense of being part of a community of practice; provided value for the COVID-19 innovations; supported resource-sharing within and beyond program participants; and facilitated changes to participants' approaches to client care in practice and increased participants' confidence in supporting clients and families. CONCLUSIONS: The Extension for Community Healthcare Outcomes program is an effective model for capacity-building programs with a shared-learning approach. Future iterations should include targeted evaluation of long-term outcomes such as staff burnout.

The Role of Tele-Education in Advancing Mental Health Quality of Care: A Content Analysis of Project ECHO Recommendations.

Introduction: Project Extension for Community Healthcare Outcomes (Project ECHO®) is a global-guided practice initiative aimed at building primary care capacity and improving health care quality for underserved populations. This tele-education model brings together primary care providers and subject-matter specialists in online communities of practice to share knowledge, discuss complexities in patient care, and collaborate to reduce health disparities. Methods: Using co-generated clinical care recommendations from ECHO Ontario Mental Health, a mental health focused ECHO program, we explored alignment of recommendations across the Institute of Medicine's (IOM) six domains of health care quality to characterize its impact. A total of 417 recommendations, made for 32 patient cases, were analyzed using a modified directed content analysis method. Each recommendation was coded with one or multiple codes, representing each of the six IOM domains. Key examples of recommendations within each domain are described. Results: An average of 13 recommendations were generated per patient case. The effective domain occurred at least once in each complete set of patient care recommendations. The next highest occurring domain was safe (71.9%), followed by patient-centered (68.8%), efficient (40.6%), equitable (18.8%), and timely (12.5%). Recommendation distribution across the entire data set was effective (97.8%), safe (15.6%), patient-centered (12.0%), efficient (3.6%), equitable (1.9%), and timely (1.4%). Discussion: As the first study to characterize ECHO's impact using health care quality domains, the study highlights ECHO's significant focus on effective, safe, and patient-centered care. These findings can inform ways for ECHO to target quality improvement and measure impact in additional health care quality domains, such as efficient, equitable, and timely.

Responding to Health Care Professionals' Mental Health Needs During COVID-19 Through the Rapid Implementation of Project ECHO.

INTRODUCTION: The COVID-19 pandemic can cause significant mental health distress among health care professionals (HCPs). We describe the psychological needs of HCPs during COVID-19 and the implementation of Project Extension for Community Healthcare Outcomes (ECHO) Coping with COVID (ECHO-CWC) to help HCPs manage COVID-19 distress. METHODS: We used an established rapid implementation approach to accelerate the development and delivery of ECHO-CWC to address the emerging needs of HCPs. Participants' needs were identified using a 10-question survey of participants' perceived risk of COVID-19 and a five-item self-efficacy measure. Implementation outcomes consisted of participant engagement and session satisfaction scores using a five-point Likert scale. RESULTS: A total of 426 participants registered for ECHO-CWC. Needs assessment data (n = 129) showed that most participants reported feeling increased stress at work (84.5%), fear of infecting others (75.2%), and fear of falling ill (70.5%) from COVID-19, yet most participants accepted the risk associated with work during this time (59.7%). Participants were highly satisfied with the initial five sessions (mean = 4.26). DISCUSSION: HCPs reported the greatest concern with fears of infection and infecting others during the acute phase of the pandemic. Using an iterative curriculum design approach and existing implementation frameworks, the ECHO tele-education model can be rapidly mobilized to address HCPs' mental health needs during the COVID-19 pandemic.