Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale.

BACKGROUND: Enhancing the sustainability of evidence-based prevention programs for mental and behavioral health requires tools for measuring both sustainability determinants and sustainment outcomes. The aim of this study was to develop the Sustainment Measurement System Scale (SMSS) and to assess its reliability and construct validity for measuring both determinants and outcomes of efforts to sustain prevention programs and initiatives. METHODS: A 42-item scale comprised of items identified from qualitative data collected from 45 representatives of 10 programs and 8 SAMHSA program officers was administered to 186 representatives of 145 programs funded by 7 SAMHSA prevention grant initiatives. Cronbach's alphas were used to determine inter-item reliability. Convergent validity was assessed by comparisons of a global measure of sustainment with current SAMHSA-funding status and continued operation in the same form. Discriminant validity was assessed by comparisons of sustainability determinants with whether or not the program had undergone adaptations. RESULTS: Confirmatory factor analysis provided support for a 35-item model fit to the data. Cronbach's alpha was .84 for the sustainment outcome construct and ranged from .70 to .93 for the sustainability determinant constructs. All of the determinant constructs were significantly associated with sustainment outcome individual and global measures for the entire sample (p < 0.01 to 0.001) and for community-based programs and programs with a substance abuse focus (p < 0.05 to 0.001). Convergent validity was supported by significant associations between the global sustainment measure and current SAMHSA funding status and continued operation in the same form (p < 0.001). Four of the sustainability determinant constructs (responsive to community needs; coalitions, partnerships, and networks; organizational staff capability; and evaluation, feedback, and program outcomes) were also significantly associated with current SAMHSA funding status (p < 0.5 to 0.01). With the exception of organizational staff capability, all sustainability determinants were unrelated to program adaptation as predicted. CONCLUSIONS: The SMSS demonstrated good reliability and convergent and discriminant validity in assessing likelihood of sustainment of SAMHSA funded prevention programs and initiatives. The measure demonstrates potential in identifying predictors of program sustainment and as a tool for enhancing the likelihood of successful sustainment through ongoing evaluation and feedback.

Conceptualizing and measuring sustainability of prevention programs, policies, and practices.

A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. We interviewed 45 representatives of 10 grantees and 9 program officers within 4 Substance Abuse and Mental Health Services Administration prevention grant initiatives to identify experiences with implementation and sustainability barriers and facilitators; what "sustainability" means and what it will take to sustain their programs; and which Consolidated Framework for Implementation Research (CFIR) elements are important for sustainability. Lists of sustainability determinants and outcomes were then compiled from each data set and compared with one another. Analysis of themes from interviews and free lists revealed considerable overlap between sustainability determinants and outcomes. Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations, and networks and partnerships; infrastructure and capacity to support sustainability; community need for program; and ongoing evaluation of performance and outcomes), and 11 elements were identified by two of three data sets (availability of funding; consistency with organizational culture; evidence of positive outcomes; development of a plan for implementation and sustainment; presence of a champion; institutionalization and integration of program; institutional support and commitment; community buy-in and support; program continuity; supportive leadership; and opportunities for staff training). All but one of the CFIR domain elements (pressure from other states, tribes, or communities) were endorsed as important to sustainability by 50% or more of participants. It may be more important to implement a standardized process of eliciting determinants and outcomes of sustainability than to implement a single standardized instrument.

Innovations in Mixed Methods Evaluations.

Mixed methods research-i.e., research that draws on both qualitative and quantitative methods in varying configurations-is well suited to address the increasing complexity of public health problems and their solutions. This review focuses specifically on innovations in mixed methods evaluations of intervention, program or policy (i.e., practice) effectiveness, and implementation. The article begins with an overview of the structure, function, and process of different mixed methods designs and then provides illustrations of their use in effectiveness studies, implementation studies, and combined effectiveness-implementation hybrid studies. The article then examines four specific innovations: procedures for transforming (or "quantitizing") qualitative data, application of rapid assessment and analysis procedures in the context of mixed methods studies, development of measures to assess implementation outcomes, and strategies for conducting both random and purposive sampling, particularly in implementation-focused evaluation research. The article concludes with an assessment of challenges to integrating qualitative and quantitative data in evaluation research.

Measuring sustainment of prevention programs and initiatives: a study protocol.

BACKGROUND: Sustaining prevention efforts directed at substance use and mental health problems is one of the greatest, yet least understood, challenges in the field of implementation science. A large knowledge gap exists regarding the meaning of the term "sustainment" and what factors predict or even measure sustainability of effective prevention programs and support systems. METHODS/DESIGN: The U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) supports a diverse portfolio of prevention and treatment grant programs that aim to improve population and individual level behavioral health. This study focuses on four SAMHSA prevention grant programs, two of which target substance abuse prevention at the state or single community level, one targets suicide prevention, and one targets prevention of aggressive/disruptive behavior in elementary schools. An examination of all four grant programs simultaneously provides an opportunity to determine what is meant by the term sustainment and identify and support both the unique requirements for improving sustainability for each program as well as for developing a generalizable framework comprised of core components of sustainment across diverse prevention approaches. Based on an analysis of qualitative and quantitative data of 10 grantees supported by these four programs, we will develop a flexible measurement system, with both general and specific components, that can bring precision to monitoring sustainment of infrastructure, activities, and outcomes for each prevention approach. We will then transform this system for use in evaluating and improving the likelihood of achieving prevention effort sustainment. To achieve these goals, we will (1) identify core components of sustainment of prevention programs and their support infrastructures; (2) design a measurement system for monitoring and providing feedback regarding sustainment within the four SAMHSA's prevention-related grant programs; and (3) pilot test the predictability of this multilevel measurement system across these programs and the feasibility and acceptability of a measurement system to evaluate and improve the likelihood of sustainment. DISCUSSION: This project is intended to improve sustainment of the supporting prevention infrastructure, activities, and outcomes that are funded by federal, state, community, and foundation sources.

Client Views of Engagement in the RAISE Connection Program for Early Psychosis Recovery.

OBJECTIVE: This study assessed factors that facilitated or impeded clients' engagement in services offered by the Recovery After an Initial Schizophrenia Episode (RAISE) Connection Program for youths and young adults experiencing early psychosis. It was part of the larger RAISE Implementation and Engagement Study. METHODS: Thematic qualitative analyses of data from in-person, semistructured interviews with 32 clients were used to examine experiences of program services, staff practices, clients' engagement behaviors, and related factors, such as expectations, family involvement, illness, and setting. Eighteen clients were well engaged with services, and 14 were not. Thirteen were interviewed early in their program involvement (two to nine months after enrollment) and 18 others later (12 to 24 months after enrollment). RESULTS: Four domains of factors influenced engagement: individualized care, program attributes, family member engagement, and personal attributes. A central factor was the program's focus on clients' life goals. For many interviewees, engagement hinged substantially on receiving what could be considered nonclinical services, such as supported education and employment. Other key factors were individualized services and staff interactions that were respectful, warm, and flexible; engagement of family members; and a focus on shared decision making. CONCLUSIONS: The findings help explain the Connection Program's effectiveness regarding client engagement and deepen understanding of treatment engagement for youths and young adults experiencing early psychosis. The individualized, flexible, recovery-focused, and assertive model of services and client-staff interaction, incorporating shared decision making and a focus on client life goals, should be implemented and sustained in services for this population.

Implementing Coordinated Specialty Care for Early Psychosis: The RAISE Connection Program.

OBJECTIVE: The RAISE (Recovery After an Initial Schizophrenia Episode) Connection Program Implementation and Evaluation Study developed tools necessary to implement and disseminate an innovative team-based intervention designed to promote engagement and treatment participation, foster recovery, and minimize disability among individuals experiencing early psychosis. This article describes the treatment model and reports on service utilization and outcomes. It was hypothesized that individuals' symptoms and functioning would improve over time. METHODS: A total of 65 individuals in RAISE Connection Program treatment across two sites (Baltimore and New York City) were enrolled and received services for up to two years. Primary outcomes, including social and occupational functioning and symptoms, were evaluated. Trajectories for individuals' outcomes over time were examined with linear and quadratic mixed-effects models with repeated measures. RESULTS: Measures of occupational and social functioning improved significantly over time, symptoms declined, and rates of remission improved. Visits were most frequent during the first three months, with a mean±SD of 23.2±11.5 unduplicated staff encounters per quarter. Such encounters decreased to 8.8±5.2 in the final quarter of year 2. CONCLUSIONS: The overall project was successful in that the treatment program was delivered and tools useful to other clinical settings were produced. The strengths of this study lie in the demonstrated feasibility of delivering the coordinated specialty care model and the associated high rates of engagement among individuals who are typically difficult to engage in treatment. Notwithstanding the lack of a built-in comparison group, participant outcomes were promising, with improvements comparable to those seen with other successful interventions.

Practical Monitoring of Treatment Fidelity: Examples From a Team-Based Intervention for People With Early Psychosis.

Mental health programs can address many components of fidelity with routinely available data. Information from client interviews can be used to corroborate these administrative data. This column describes a practical approach to measuring fidelity that used both data sources. The approach was used in the Recovery After an Initial Schizophrenia Episode (RAISE) Connection Program, a team-based intervention designed to implement evidence-based practices for people experiencing early psychosis suggestive of schizophrenia. Data indicated that the intervention was implemented as intended, including program elements related to shared decision making and a range of evidence-based clinical interventions.