Getting cozy with causality: Advances to the causal pathway diagramming method to enhance implementation precision.

Background: Implementation strategies are theorized to work well when carefully matched to implementation determinants and when factors-preconditions, moderators, etc.-that influence strategy effectiveness are prospectively identified and addressed. Existing methods for strategy selection are either imprecise or require significant technical expertise and resources, undermining their utility. This article outlines refinements to causal pathway diagrams (CPDs), a method for articulating the causal process through which implementation strategies work and offers illustrations of their use. Method: CPDs are a visualization tool to represent an implementation strategy, its mechanism(s) (i.e., the processes through which a strategy is thought to operate), determinants it is intended to address, factors that may impede or facilitate its effectiveness, and the series of outcomes that should be expected if the strategy is operating as intended. We offer principles for constructing CPDs and describe their key functions. Results: Applications of the CPD method by study teams from two National Institute of Health-funded Implementation Science Centers and a research grant are presented. These include the use of CPDs to (a) match implementation strategies to determinants, (b) understand the conditions under which an implementation strategy works, and (c) develop causal theories of implementation strategies. Conclusions: CPDs offer a novel method for implementers to select, understand, and improve the effectiveness of implementation strategies. They make explicit theoretical assumptions about strategy operation while supporting practical planning. Early applications have led to method refinements and guidance for the field.

Advances to the Causal Pathway Diagramming Method to Enhance Implementation Precision Plain Language Summary Implementation strategies often fail to produce meaningful improvements in the outcomes we hope to impact. Better tools for choosing, designing, and evaluating implementation strategies may improve their performance. We developed a tool, causal pathway diagrams (CPD), to visualize and describe how implementation strategies are expected to work. In this article, we describe refinements to the CPD tool and accompanying approach. We use real illustrations to show how CPDs can be used to improve how to match strategies to barriers, understand the conditions in which those strategies work best, and develop generalizable theories describing how implementation strategies work. CPDs can serve as both a practical and scientific tool to improve the planning, deployment, and evaluation of implementation strategies. We demonstrate the range of ways that CPDs are being used, from a highly practical tool to improve implementation practice to a scientific approach to advance testing and theorizing about implementation strategies.

Sitting Time Reduction and Blood Pressure in Older Adults: A Randomized Clinical Trial.

Importance: Practical health promotion strategies for improving cardiometabolic health in older adults are needed. Objective: To examine the efficacy of a sedentary behavior reduction intervention for reducing sitting time and improving blood pressure in older adults. Design, Setting, and Participants: This parallel-group randomized clinical trial was conducted in adults aged 60 to 89 years with high sitting time and body mass index of 30 to 50 from January 1, 2019, to November 31, 2022, at a health care system in Washington State. Intervention: Participants were randomized 1:1 to the sitting reduction intervention or a healthy living attention control condition for 6 months. Intervention participants received 10 health coaching contacts, sitting reduction goals, and a standing desk and fitness tracker to prompt sitting breaks. The attention control group received 10 health coaching contacts to set general healthy living goals, excluding physical activity or sedentary behavior. Main Outcomes and Measures: The primary outcome, measured at baseline, 3 months, and 6 months, was sitting time assessed using accelerometers worn for 7 days at each time point. Coprimary outcomes were systolic and diastolic blood pressure measured at baseline and 6 months. Results: A total of 283 participants (140 intervention and 143 control) were randomized (baseline mean [SD] age, 68.8 [6.2] years; 186 [65.7%] female; mean [SD] body mass index, 34.9 [4.7]). At baseline, 147 (51.9%) had a hypertension diagnosis and 97 (69.3%) took at least 1 antihypertensive medication. Sitting time was reduced, favoring the intervention arm, with a difference in the mean change of -31.44 min/d at 3 months (95% CI, -48.69 to -14.19 min/d; P < .001) and -31.85 min/d at 6 months (95% CI, -52.91 to -10.79 min/d; P = .003). Systolic blood pressure change was lower by 3.48 mm Hg, favoring the intervention arm at 6 months (95% CI, -6.68 to -0.28 mm Hg; P = .03). There were 6 serious adverse events in each arm and none were study related. Conclusions and Relevance: In this study of a 6-month sitting reduction intervention, older adults in the intervention reduced sedentary time by more than 30 min/d and reduced systolic blood pressure. Sitting reduction could be a promising approach to improve health in older adults. Trial Registration: ClinicalTrials.gov Identifier: NCT03739762.

Study protocol: Novel Methods for Implementing Measurement-Based Care with youth in Low-Resource Environments (NIMBLE).

BACKGROUND: For youth receiving care in community mental health centers, comorbidities are the rule rather than the exception. Using measurement-based care (MBC), or the routine evaluation of symptoms to inform care decisions, as the foundation of treatment for youth with comorbid problems significantly improves the impact of psychotherapy by focusing care and building engagement and alliance. MBC increases the rate of symptom improvement, detects clients who would otherwise deteriorate, and alerts clinicians to non-responders. Despite its demonstrated utility, MBC is rarely implemented with fidelity; less than 15% of providers report using MBC per recommendations. Previous efforts to support MBC implementation have yielded suboptimal outcomes, in part, due to organizations' challenges with identifying and prioritizing barriers and selecting and developing strategies to overcome them. New methods are needed for identifying and prioritizing barriers, and matching strategies to barriers to optimize MBC implementation and treatment quality to improve youth mental health outcomes in community settings. METHODS: Pragmatic implementation methods will be piloted in four diverse community mental health centers. Methods include (a) rapid evidence synthesis; (b) rapid ethnography; (c) design kits (e.g., kits with disposable cameras, journals, maps); (d) barrier prioritization, and (e) causal pathway diagramming. These activities will generate actionable barriers; subsequently, we will use facilitated group processes to prioritize barriers and develop causal pathway diagrams to match strategies to barriers to create implementation plans that optimize MBC fidelity (Aim 1). We will track strategy deployment for 6 months, then compare MBC fidelity for another 6 months post-implementation with data from 2 years of historical controls (Aim 2). Finally, we will co-design a toolkit for design kit methods with youth and the practice and scientific communities (Aim 3). DISCUSSION: Optimizing MBC implementation in community mental health centers could transform youth mental health care by ensuring the most pressing symptoms are targeted early in treatment. The discussion section highlights expected challenges and limits to using the five methods, including recruitment and engagement given the high pressure on community mental health settings. TRIAL REGISTRATION: Clinicaltrials.gov. NCT05644756 . Registered on 18 November 2022. This trial was retrospectively registered.

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review.

BACKGROUND: Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings. METHODS: Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed. RESULTS: A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants. CONCLUSIONS: This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field. TRIAL REGISTRATION: This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

Exploring Differences in Older Adult Accelerometer-Measured Sedentary Behavior and Resting Blood Pressure Before and During the COVID-19 Pandemic.

Older adults have higher sedentary behavior (SB), lower physical activity, and are particularly susceptible to negative impacts from the COVID-19 pandemic and associated public health restrictions. Pandemic impacts to SB and health, particularly via objective assessment, are not well documented in the literature. Here we described differences in SB, physical activity, and blood pressure (BP) for older adults before and during the pandemic. Baseline thigh-worn activPAL accelerometer and BP measurements from 95 participants enrolled in a SB intervention trial pre-pandemic were compared to 60 enrolled post-pandemic. We used linear regression models adjusted for demographic and health factors to estimate differences in sample means of SB measures and BP. The post-COVID sample was older (age 67 vs. 70), more female (60% vs. 72%), and included more individuals of color (21% vs. 32%). In fully adjusted models, systolic BP was statistically significantly higher in the post-COVID group (6.8, 95% CI: [0.3,13.3]). After adjustment, activPAL-measured and self-reported activity were non-significant but trended towards greater total sitting (0.4 hours [-0.3, 1.1]), fewer daily steps (-270 [-1078, 538]), and greater self-reported TV time (0.4 hours, [-0.3, 1.1]) post-COVID. Future analyses are warranted to better quantify these impacts and guide clinical care and future interventions.

Mechanisms of implementation: An appraisal of causal pathways presented at the 5th biennial Society for Implementation Research Collaboration (SIRC) conference.

Background Implementation mechanisms are defined as processes or events through which implementation strategies operate to affect one or more implementation outcomes. Understanding the mechanisms through which implementation strategies work is critical to understanding how and why implementation efforts are successful, and to matching, tailoring, and optimizing implementation strategies. This study examined the content of abstracts included in the program for the 2019 Society for Implementation Research Collaboration (SIRC) conference to characterize the presence of data related to implementation strategy mechanisms and their larger causal pathways. Methods Trained coders reviewed all 205 accepted abstracts and extracted information regarding discrete implementation strategies, determinants of implementation, implementation mechanisms, service outcomes, and implementation outcomes. Theoretical articles were omitted from further analyses due to their inability to offer data related to implementation mechanisms. Results Of the 151 empirical studies included, only 11 (7.28%) reported studying mechanisms. Mechanisms were examined in projects utilizing 14 different implementation strategies. We were able to construct implementation causal pathways for just two implementation strategies, "assess for readiness and identify determinants," representing information pulled from four different abstracts, and "create a learning collaborative," with data pulled from just one abstract. Conclusions These findings indicate that, at least based on SIRC conference abstracts, the empirical investigation of implementation mechanisms remains understudied, highlighting the need for focused research on the study of mechanisms. Plain Language Summary Understanding the mechanisms through which implementation strategies work is critical to understanding how and why implementation efforts are successful. The study of implementation mechanisms may be used to optimize implementation strategy decisions. Investigations to date have not established causal pathways linking implementation strategies, mechanisms, barriers, and outcomes. This study examined abstracts included in the program for the 2019 Society for Implementation Research Collaboration (SIRC) conference. Trained coders reviewed all 205 accepted abstracts and extracted information to characterize the presence of data related to implementation strategy causal pathways. A minority of abstracts reported studying mechanisms. We were able to construct implementation causal pathways for just two implementation strategies, representing information pulled from five different abstracts all together. This highlights the need for focused research on the study of mechanisms.

Protocol for a randomized controlled trial of sitting reduction to improve cardiometabolic health in older adults.

Older adults with obesity spend the majority of their waking hours sedentary. Given substantial barriers to regular physical activity in this population, approaches to reduce sedentary time could be an effective health promotion strategy. We present the protocol of a randomized controlled trial to reduce sitting time in older adults with a body mass index of 30 kg/m2 or above. Participants (N = 284) will be randomized to receive a sitting reduction intervention (termed I-STAND) or a healthy living focused attention control condition. I-STAND includes 10 contacts with a health coach (10 sessions total) and participants receive a wrist-worn prompting device and portable standing desk. The healthy living condition includes 10 sessions with a health coach to set goals around various topics relating to healthy aging. Participants receive their assigned intervention for 6 months. After 6 months, those receiving the I-STAND condition are re-randomized to receive five booster health coaching sessions by 'phone or no further contact; healthy living participants receive no further contact and those in both conditions are followed for an additional 6 months. Measurements initially included wearing an activPAL device and completing several biometric tests (e.g., blood pressure, HbA1c), at baseline, 3 months, 6 months, and 12 months; however, during the COVID-19 pandemic we shifted to remote assessments and were unable to collect all of these measures. The primary outcomes remained activPAL-assessed sitting time and blood pressure. Recruitment is anticipated to be completed in 2022.

A roadmap to operationalize and evaluate impact in a learning health system.

BACKGROUND: Many health systems invest in initiatives to accelerate translation of knowledge into practice. However, organizations lack guidance on how to develop and operationalize such Learning Health System (LHS) programs and evaluate their impact. Kaiser Permanente Washington (KPWA) launched our LHS program in June 2017 and developed a logic model as a foundation to evaluate the program's impact. OBJECTIVE: To develop a roadmap for organizations that want to establish an LHS program, understand how LHS core components relate to one another when operationalized in practice, and evaluate and improve their progress. METHODS: We conducted a narrative review on LHS models, key model components, and measurement approaches. RESULTS: The KPWA LHS Logic Model provides a broad set of constructs relevant to LHS programs, depicts their relationship to LHS operations, harmonizes terms across models, and offers measurable operationalizations of each construct to guide other health systems. The model identifies essential LHS inputs, provides transparency into LHS activities, and defines key outcomes to evaluate LHS processes and impact. We provide reflections on the most helpful components of the model and identify areas that need further improvement using illustrative examples from deployment of the LHS model during the COVID-19 pandemic. CONCLUSION: The KPWA LHS Logic Model is a starting point for future LHS implementation research and a practical guide for healthcare organizations that are building, operationalizing, and evaluating LHS initiatives.

"They're Going to Zoom It": A Qualitative Investigation of Impacts and Coping Strategies During the COVID-19 Pandemic Among Older Adults.

Introduction: Older adults, who already have higher levels of social isolation, loneliness, and sedentary behavior, are particularly susceptible to negative impacts from social distancing mandates meant to control the spread of COVID-19. We sought to explore the physical, mental, and social health impacts of the pandemic on older adults and their coping techniques. Materials and Methods: We conducted 25 semi-structured interviews with a sub-sample of participants in an ongoing sedentary behavior reduction intervention. Interviews were recorded and transcribed, and iterative coding was used to extract key themes. Results: Most participants reported an increase in sedentary behavior due to limitations on leaving their home and increased free time to pursue seated hobbies (e.g., reading, knitting, tv). However, many participants also reported increased levels of intentional physical activity and exercise, particularly outdoors or online. Participants also reported high levels of stress and a large decrease in in-person social connection. Virtual connection with others through phone and video was commonly used to stay connected with friends and family, engage in community groups and activities, and cope with stress and social isolation. Maintenance of a positive attitude and perspective gained from past hardships was also an important coping strategy for many participants. Discussion: The COVID-19 pandemic and associated social distancing measures have impacted older adults' perceived levels of activity, stress, and social isolation, but many leveraged technology and prior life experiences to cope. These themes could inform future interventions for older adults dealing with chronic stress and isolation.

Effect of new Medicare enrollment on health, healthcare utilization, and cost: A scoping review.

BACKGROUND: More than three million Americans turn 65 each year and newly enroll in Medicare, making this one of the most common insurance transitions. Non-Medicare insurance transitions are associated with changes in health, healthcare utilization and costs. In addition, older Americans have higher morbidity, mortality, healthcare utilization, and healthcare costs than the general population. However, the effect of new Medicare enrollment on these outcomes is unclear. DESIGN: We conducted a scoping review to rigorously identify the scope of evidence on the association between new Medicare enrollment and health, healthcare utilization and costs. SETTING: We included English-language, peer-reviewed, studies cataloged in Medline (PubMed) and EconLit from 1998 to 2018. PARTICIPANTS: Individuals newly enrolling in Medicare. MEASUREMENTS: We measured health (e.g., self-reported health), healthcare utilization (e.g., provider visits, preventive care, and hospitalizations) and costs (e.g., patient out-of-pocket and health plan spending). RESULTS: We screened 5265 articles and included 20 articles. New Medicare enrollment was found to increase self-reported health and healthcare utilization overall, as well as reduce disparities across racial and socioeconomic strata. Provider visits, preventive care and hospitalizations all increased. However, patient out-of-pocket spending decreased, and health plan spending also decreased, when Medicare's lower prices were accounted for. Few studies compared outcomes among new Medicare Advantage enrollees with new Medicare fee-for-service enrollees. None of the studies specifically evaluated the effect of new Medicare enrollment on adults with multiple chronic conditions. CONCLUSION: New Medicare enrollment improves access overall and reduces access disparities. However, the impact of new Medicare enrollment among subgroups defined by insurance coverage type and number of chronic conditions is less clear. Future work should also evaluate the mechanism for increases in hospitalizations.

Self-reported Access to Firearms Among Patients Receiving Care for Mental Health and Substance Use.

Importance: Firearms are the most common method of suicide, one of the "diseases of despair" driving increased mortality in the US over the past decade. However, routine standardized questions about firearm access are uncommon, particularly among adult populations, who are more often asked at the discretion of health care clinicians. Because standard questions are rare, patterns of patient-reported access are unknown. Objective: To evaluate whether and how patients self-report firearm access information on a routine mental health monitoring questionnaire and additionally to examine sociodemographic and clinical associations of reported access. Design Setting and Participants: Cross-sectional study of patients receiving care for mental health and/or substance use in primary care or outpatient mental health specialty clinics of Kaiser Permanente Washington, an integrated health insurance provider and care delivery system. Main Outcomes and Measures: Electronic health records were used to identify patients who completed a standardized self-reported mental health monitoring questionnaire after a single question about firearm access was added from January 1, 2016, through December 31, 2019. Primary analyses evaluated response (answered vs not answered) and reported access (yes vs no) among those who answered, separately for patients seen in primary care and mental health. These analyses also evaluated associations between patient characteristics and reported firearm access. Data analysis took place from February 2020 through May 2021. Results: Among patients (n = 128 802) who completed a mental health monitoring questionnaire during the study period, 74.4% (n = 95 875) saw a primary care clinician and 39.3% (n = 50 631) saw a mental health specialty clinician. The primary care and mental health samples were predominantly female (63.1% and 64.9%, respectively) and White (75.7% and 77.0%), with a mean age of 42.8 and 51.1 years. In primary care, 83.4% of patients answered the question about firearm access, and 20.9% of patients who responded to the firearm question reported having access. In mental health, 91.8% of patients answered the question, and 15.3% reported having access. Conclusions and Relevance: In this cross-sectional study of adult patients receiving care for mental health and substance use, most patients answered a question about firearm access on a standardized mental health questionnaire. These findings provide a critical foundation to help advance understanding of the utility of standardized firearm access assessment and to inform development of practice guidelines and recommendations. Responses to standard firearm access questions used in combination with dialogue and decision-making resources about firearm access and storage may improve suicide prevention practices and outcomes.

Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS).

The use of reliable, valid measures in implementation practice will remain limited without pragmatic measures. Previous research identified the need for pragmatic measures, though the characteristic identification used only expert opinion and literature review. Our team completed four studies to develop a stakeholder-driven pragmatic rating criteria for implementation measures. We published Studies 1 (identifying dimensions of the pragmatic construct) and 2 (clarifying the internal structure) that engaged stakeholders-participants in mental health provider and implementation settings-to identify 17 terms/phrases across four categories: Useful, Compatible, Acceptable, and Easy. This paper presents Studies 3 and 4: a Delphi to ascertain stakeholder-prioritized dimensions within a mental health context, and a pilot study applying the rating criteria. Stakeholders (N = 26) participated in a Delphi and rated the relevance of 17 terms/phrases to the pragmatic construct. The investigator team further defined and shortened the list, which were piloted with 60 implementation measures. The Delphi confirmed the importance of all pragmatic criteria, but provided little guidance on relative importance. The investigators removed or combined terms/phrases to obtain 11 criteria. The 6-point rating system assigned to each criterion demonstrated sufficient variability across items. The grey literature did not add critical information. This work produced the first stakeholder-driven rating criteria to assess whether measures are pragmatic. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) combines the pragmatic criteria with psychometric rating criteria, from previous work. Use of PAPERS can inform development of implementation measures and to assess the quality of existing measures.

The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation.

To rigorously measure the implementation of evidence-based interventions, implementation science requires measures that have evidence of reliability and validity across different contexts and populations. Measures that can detect change over time and impact on outcomes of interest are most useful to implementers. Moreover, measures that fit the practical needs of implementers could be used to guide implementation outside of the research context. To address this need, our team developed a rating scale for implementation science measures that considers their psychometric and pragmatic properties and the evidence available. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) can be used in systematic reviews of measures, in measure development, and to select measures. PAPERS may move the field toward measures that inform robust research evaluations and practical implementation efforts.

Transitioning patients from outpatient mental health services to primary care: A rapid literature review.

Background: A lack of access to mental health services is a critical barrier to obtaining evidence-based care. One strategy to improve access is to transition stable patients out of mental health specialty services and into primary care, thus opening availability for new patients and those with acute mental health needs. To support these transitions, organizations might explore a range of new practices and implementation strategies. Methods: We conducted a rapid literature review to summarize descriptions from the research literature about practices for transitioning stable patients from outpatient mental health services to primary care, as well as implementation strategies to enhance the adoption and sustainment of these practices. We searched PsycINFO and Cumulated Index to Nursing and Allied Health Literature (CINAHL) for articles published between January 2000 and August 2019. For articles meeting inclusion criteria, we abstracted data on study characteristics, transition practices, and implementation strategies. Results: We included 11 articles representing diverse study designs, settings, and health care organizations. Across these articles, we identified six categories of commonly described transition practices, with patient engagement appearing the most frequently (10 articles), followed by shared treatment planning (eight articles), assessment of recovery and stability, care coordination, follow up and support, and medication management (seven articles each). Less frequently, articles included descriptions of implementation strategies, with five articles describing efforts to train and educate stakeholders and four articles describing the use of evaluative and iterative strategies. Conclusions: We identified descriptions of several common practices to help patients transition from mental health specialty services to primary care, but there are opportunities for an increased focus on implementation strategies to enhance the adoption and sustainment of these transition practices. More research is needed to better understand the effectiveness of specific transition interventions and the feasibility of deploying these interventions in heterogeneous health care settings.

A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence.

Background: Measurement is a critical component for any field. Systematic reviews are a way to locate measures and uncover gaps in current measurement practices. The present study identified measures used in behavioral health settings that assessed all constructs within the Process domain and two constructs from the Inner setting domain as defined by the Consolidated Framework for Implementation Research (CFIR). While previous conceptual work has established the importance social networks and key stakeholders play throughout the implementation process, measurement studies have not focused on investigating the quality of how these activities are being carried out. Methods: The review occurred in three phases: Phase I, data collection included (1) search string generation, (2) title and abstract screening, (3) full text review, (4) mapping to CFIR-constructs, and (5) "cited-by" searches. Phase II, data extraction, consisted of coding information relevant to the nine psychometric properties included in the Psychometric And Pragmatic Rating Scale (PAPERS). In Phase III, data analysis was completed. Results: Measures were identified in only seven constructs: Structural characteristics (n = 13), Networks and communication (n = 29), Engaging (n = 1), Opinion leaders (n = 5), Champions (n = 5), Planning (n = 5), and Reflecting and evaluating (n = 5). No quantitative assessment measures of Formally appointed implementation leaders, External change agents, or Executing were identified. Internal consistency and norms were reported on most often, whereas no studies reported on discriminant validity or responsiveness. Not one measure in the sample reported all nine psychometric properties evaluated by the PAPERS. Scores in the identified sample of measures ranged from "-2" to "10" out of a total of "36." Conclusions: Overall measures demonstrated minimal to adequate evidence and available psychometric information was limited. The majority were study specific, limiting their generalizability. Future work should focus on more rigorous measure development and testing of currently existing measures, while moving away from creating new, single use measures. Plain Language Summary: How we measure the processes and players involved for implementing evidence-based interventions is crucial to understanding what factors are helping or hurting the intervention's use in practice and how to take the intervention to scale. Unfortunately, measures of these factors-stakeholders, their networks and communication, and their implementation activities-have received little attention. This study sought to identify and evaluate the quality of these types of measures. Our review focused on collecting measures used for identifying influential staff members, known as opinion leaders and champions, and investigating how they plan, execute, engage, and evaluate the hard work of implementation. Upon identifying these measures, we collected all published information about their uses to evaluate the quality of their evidence with respect to their ability to produce consistent results across items within each use (i.e., reliable) and if they assess what they are intending to measure (i.e., valid). Our searches located over 40 measures deployed in behavioral health settings for evaluation. We observed a dearth of evidence for reliability and validity and when evidence existed the quality was low. These findings tell us that more measurement work is needed to better understand how to optimize players and processes for the purposes of successful implementation.

Determining the influence of intervention characteristics on implementation success requires reliable and valid measures: Results from a systematic review.

Background: Despite their inclusion in Rogers' seminal diffusion of innovations theory, few implementation studies empirically evaluate the role of intervention characteristics. Now, with growing evidence on the role of adaptation in implementation, high-quality measures of characteristics such as adaptability, trialability, and complexity are needed. Only two systematic reviews of implementation measures captured those related to the intervention or innovation and their assessment of psychometric properties was limited. This manuscript reports on the results of eight systematic reviews of measures of intervention characteristics with nuanced data regarding a broad range of psychometric properties. Methods: The systematic review proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and citation searches. Phase II, data extraction, involved coding psychometric information. Phase III, data analysis, involved two trained specialists independently rating each measure using PAPERS (Psychometric And Pragmatic Evidence Rating Scales). Results: Searches identified 16 measures or scales: zero for intervention source, one for evidence strength and quality, nine for relative advantage, five for adaptability, six for trialability, nine for complexity, and two for design quality and packaging. Information about internal consistency and norms was available for most measures, whereas information about other psychometric properties was most often not available. Ratings for psychometric properties fell in the range of "poor" to "good." Conclusion: The results of this review confirm that few implementation scholars are examining the role of intervention characteristics in behavioral health studies. Significant work is needed to both develop new measures (e.g., for intervention source) and build psychometric evidence for existing measures in this forgotten domain. Plain Language Summary: Intervention characteristics have long been perceived as critical factors that directly influence the rate of adopting an innovation. It remains unclear the extent to which intervention characteristics including relative advantage, complexity, trialability, intervention source, design quality and packaging, evidence strength and quality, adaptability, and cost impact implementation of evidence-based practices in behavioral health settings. To unpack the differential influence of these factors, high quality measures are needed. Systematic reviews can identify measures and synthesize the data regarding their quality to identify gaps in the field and inform measure development and testing efforts. Two previous reviews identified measures of intervention characteristics, but they did not provide information about the extent of the existing evidence nor did they evaluate the host of evidence available for identified measures. This manuscript summarizes the results of nine systematic reviews (i.e., one for each of the factors listed above) for which 16 unique measures or scales were identified. The nuanced findings will help direct measure development work in this forgotten domain.

Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review.

Background: Organizational culture, organizational climate, and implementation climate are key organizational constructs that influence the implementation of evidence-based practices. However, there has been little systematic investigation of the availability of psychometrically strong measures that can be used to assess these constructs in behavioral health. This systematic review identified and assessed the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs as defined by the Consolidated Framework for Implementation Research (CFIR) and Ehrhart and colleagues. Methods: Data collection involved search string generation, title and abstract screening, full-text review, construct assignment, and citation searches for all known empirical uses. Data relevant to nine psychometric criteria from the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) were extracted: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. Extracted data for each criterion were rated on a scale from -1 ("poor") to 4 ("excellent"), and each measure was assigned a total score (highest possible score = 36) that formed the basis for head-to-head comparisons of measures for each focal construct. Results: We identified full measures or relevant subscales of broader measures for organizational culture (n = 21), organizational climate (n = 36), implementation climate (n = 2), tension for change (n = 2), compatibility (n = 6), relative priority (n = 2), organizational incentives and rewards (n = 3), goals and feedback (n = 3), and learning climate (n = 2). Psychometric evidence was most frequently available for internal consistency and norms. Information about other psychometric properties was less available. Median ratings for psychometric properties across categories of measures ranged from "poor" to "good." There was limited evidence of responsiveness or predictive validity. Conclusion: While several promising measures were identified, the overall state of measurement related to these constructs is poor. To enhance understanding of how these constructs influence implementation research and practice, measures that are sensitive to change and predictive of key implementation and clinical outcomes are required. There is a need for further testing of the most promising measures, and ample opportunity to develop additional psychometrically strong measures of these important constructs. Plain Language Summary: Organizational culture, organizational climate, and implementation climate can play a critical role in facilitating or impeding the successful implementation and sustainment of evidence-based practices. Advancing our understanding of how these contextual factors independently or collectively influence implementation and clinical outcomes requires measures that are reliable and valid. Previous systematic reviews identified measures of organizational factors that influence implementation, but none focused explicitly on behavioral health; focused solely on organizational culture, organizational climate, and implementation climate; or assessed the evidence base of all known uses of a measure within a given area, such as behavioral health-focused implementation efforts. The purpose of this study was to identify and assess the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs that have been used in behavioral health-focused implementation research. We identified 21 measures of organizational culture, 36 measures of organizational climate, 2 measures of implementation climate, 2 measures of tension for change, 6 measures of compatibility, 2 measures of relative priority, 3 measures of organizational incentives and rewards, 3 measures of goals and feedback, and 2 measures of learning climate. Some promising measures were identified; however, the overall state of measurement across these constructs is poor. This review highlights specific areas for improvement and suggests the need to rigorously evaluate existing measures and develop new measures.

Measuring characteristics of individuals: An updated systematic review of instruments' psychometric properties.

Background: Identification of psychometrically strong implementation measures could (1) advance researchers' understanding of how individual characteristics impact implementation processes and outcomes, and (2) promote the success of real-world implementation efforts. The current study advances the work that our team published in 2015 by providing an updated and enhanced systematic review that identifies and evaluates the psychometric properties of implementation measures that assess individual characteristics. Methods: A full description of our systematic review methodology, which included three phases, is described in a previously published protocol paper. Phase I focused on data collection and involved search string generation, title and abstract screening, full-text review, construct assignment, and measure forward searches. During Phase II, we completed data extraction (i.e., coding psychometric information). Phase III involved data analysis, where two trained specialists independently rated each measurement tool using our psychometric rating criteria. Results: Our team identified 124 measures of individual characteristics used in mental or behavioral health research, and 123 of those measures were deemed suitable for rating using Psychometric and Pragmatic Evidence Rating Scale. We identified measures of knowledge and beliefs about the intervention (n = 76), self-efficacy (n = 24), individual stage of change (n = 2), individual identification with organization (n = 7), and other personal attributes (n = 15). While psychometric information was unavailable and/or unreported for many measures, information about internal consistency and norms were the most commonly identified psychometric data across all individual characteristics' constructs. Ratings for all psychometric properties predominantly ranged from "poor" to "good." Conclusion: The majority of research that develops, uses, or examines implementation measures that evaluate individual characteristics does not include the psychometric properties of those measures. The development and use of psychometric reporting standards could advance the use of valid and reliable tools within implementation research and practice, thereby enhancing the successful implementation and sustainment of evidence-based practice in community care. Plain Language Summary: Measurement is the foundation for advancing practice in health care and other industries. In the field of implementation science, the state of measurement is only recently being targeted as an area for improvement, given that high-quality measures need to be identified and utilized in implementation work to avoid developing another research to practice gap. For the current study, we utilized the Consolidated Framework for Implementation Research to identify measures related to individual characteristics' constructs, such as knowledge and beliefs about the intervention, self-efficacy, individual identification with the organization, individual stage of change, and other personal attributes. Our review showed that many measures exist for certain constructs (e.g., measures related to assessing providers' attitudes and perceptions about evidence-based practice interventions), while others have very few (e.g., an individual's stage of change). Also, we rated measures for their psychometric strength utilizing an anchored rating system and found that most measures assessing individual characteristics are in need of more research to establish their evidence of quality. It was also clear from our results that frequency of use/citations does not equate to high quality, psychometric strength. Ultimately, the state of the literature has demonstrated that assessing individual characteristics of implementation stakeholders is an area of strong interest in implementation work. It will be important for future research to focus on clearly delineating the psychometric properties of existing measures for saturated constructs, while for the others the emphasis should be on developing new, high-quality measures and make these available to stakeholders.

A pilot study comparing tools for tracking implementation strategies and treatment adaptations.

Background: Tailoring implementation strategies and adapting treatments to better fit the local context may improve their effectiveness. However, there is a dearth of valid, reliable, pragmatic measures that allow for the prospective tracking of strategies and adaptations according to reporting recommendations. This study describes the development and pilot testing of three tools to be designed to serve this purpose. Methods: Measure development was informed by two systematic reviews of the literature (implementation strategies and treatment adaptation). The three resulting tools vary with respect to the degree of structure (brainstorming log = low, activity log = moderate, detailed tracking log = high). To prospectively track treatment adaptations and implementation strategies, three stakeholder groups (treatment developer, implementation practitioners, and mental health providers) were randomly assigned one tool per week through an anonymous web-based survey for 12 weeks and incentivized to participate. Three established implementation outcome measures, the Acceptability of Intervention Measure, Intervention Appropriateness Measure, and Feasibility of Intervention Measure, were used to assess the tools. Semi-structured interviews were conducted to gather more nuanced information from stakeholders regarding their perceptions of the tools and the tracking process. Results: The three tracking tools demonstrated moderate to good acceptability, appropriateness, and feasibility; the activity log was deemed the most feasible of the three tools. Implementation practitioners rated the tools the highest of the three stakeholder groups. The tools took an average of 15 min or less to complete. Conclusion: This study sought to fill methodological gaps that prevent stakeholders and researchers from discerning which strategies are most important to deploy for promoting implementation and sustainment of evidence-based practices. These tools would allow researchers and practitioners to track whether activities were treatment adaptations or implementation strategies and what barrier(s) each targets. These tools could inform prospective tailoring of implementation strategies and treatment adaptations, which would promote scale out and spread. Plain Language Summary: Strategies to support the implementation of evidence-based practices may be more successful if they are carefully customized based on local factors. Evidence-based practices themselves may be thoughtfully changed to better meet the needs of the settings and recipients. This study reports on a pilot study that aimed to create various types of tools to help individuals involved in implementation efforts track the actions they take to modify and implement interventions. These tools allow individuals to track the types of activities they are involved in, when the activities occurred, who was involved in the implementation efforts, and the reasons or rationale for the actions. The three tools in this study used a combination of open-ended and forced-response questions to test how the type of data recorded changed. Participants generally found the tools quick and easy to use and helpful in planning the delivery of an evidence-based practice. Most participants wanted more training in implementation science terminology and how to complete the tracking tools. Participating mental health providers would have liked more opportunities to review the data collected from the tools with their supervisors to use the data to improve the delivery of the evidence-based practice. These tools can help researchers, providers, and staff involved in implementation efforts to better understand what actions are needed to improve implementation success. Future research should address gaps identified in this study, such as the need to involve more participants in the tool development process.

Quantitative measures of health policy implementation determinants and outcomes: a systematic review.

BACKGROUND: Public policy has tremendous impacts on population health. While policy development has been extensively studied, policy implementation research is newer and relies largely on qualitative methods. Quantitative measures are needed to disentangle differential impacts of policy implementation determinants (i.e., barriers and facilitators) and outcomes to ensure intended benefits are realized. Implementation outcomes include acceptability, adoption, appropriateness, compliance/fidelity, feasibility, penetration, sustainability, and costs. This systematic review identified quantitative measures that are used to assess health policy implementation determinants and outcomes and evaluated the quality of these measures. METHODS: Three frameworks guided the review: Implementation Outcomes Framework (Proctor et al.), Consolidated Framework for Implementation Research (Damschroder et al.), and Policy Implementation Determinants Framework (Bullock et al.). Six databases were searched: Medline, CINAHL Plus, PsycInfo, PAIS, ERIC, and Worldwide Political. Searches were limited to English language, peer-reviewed journal articles published January 1995 to April 2019. Search terms addressed four levels: health, public policy, implementation, and measurement. Empirical studies of public policies addressing physical or behavioral health with quantitative self-report or archival measures of policy implementation with at least two items assessing implementation outcomes or determinants were included. Consensus scoring of the Psychometric and Pragmatic Evidence Rating Scale assessed the quality of measures. RESULTS: Database searches yielded 8417 non-duplicate studies, with 870 (10.3%) undergoing full-text screening, yielding 66 studies. From the included studies, 70 unique measures were identified to quantitatively assess implementation outcomes and/or determinants. Acceptability, feasibility, appropriateness, and compliance were the most commonly measured implementation outcomes. Common determinants in the identified measures were organizational culture, implementation climate, and readiness for implementation, each aspects of the internal setting. Pragmatic quality ranged from adequate to good, with most measures freely available, brief, and at high school reading level. Few psychometric properties were reported. CONCLUSIONS: Well-tested quantitative measures of implementation internal settings were under-utilized in policy studies. Further development and testing of external context measures are warranted. This review is intended to stimulate measure development and high-quality assessment of health policy implementation outcomes and determinants to help practitioners and researchers spread evidence-informed policies to improve population health. REGISTRATION: Not registered.