Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review.

Background: Organizational culture, organizational climate, and implementation climate are key organizational constructs that influence the implementation of evidence-based practices. However, there has been little systematic investigation of the availability of psychometrically strong measures that can be used to assess these constructs in behavioral health. This systematic review identified and assessed the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs as defined by the Consolidated Framework for Implementation Research (CFIR) and Ehrhart and colleagues. Methods: Data collection involved search string generation, title and abstract screening, full-text review, construct assignment, and citation searches for all known empirical uses. Data relevant to nine psychometric criteria from the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) were extracted: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. Extracted data for each criterion were rated on a scale from -1 ("poor") to 4 ("excellent"), and each measure was assigned a total score (highest possible score = 36) that formed the basis for head-to-head comparisons of measures for each focal construct. Results: We identified full measures or relevant subscales of broader measures for organizational culture (n = 21), organizational climate (n = 36), implementation climate (n = 2), tension for change (n = 2), compatibility (n = 6), relative priority (n = 2), organizational incentives and rewards (n = 3), goals and feedback (n = 3), and learning climate (n = 2). Psychometric evidence was most frequently available for internal consistency and norms. Information about other psychometric properties was less available. Median ratings for psychometric properties across categories of measures ranged from "poor" to "good." There was limited evidence of responsiveness or predictive validity. Conclusion: While several promising measures were identified, the overall state of measurement related to these constructs is poor. To enhance understanding of how these constructs influence implementation research and practice, measures that are sensitive to change and predictive of key implementation and clinical outcomes are required. There is a need for further testing of the most promising measures, and ample opportunity to develop additional psychometrically strong measures of these important constructs. Plain Language Summary: Organizational culture, organizational climate, and implementation climate can play a critical role in facilitating or impeding the successful implementation and sustainment of evidence-based practices. Advancing our understanding of how these contextual factors independently or collectively influence implementation and clinical outcomes requires measures that are reliable and valid. Previous systematic reviews identified measures of organizational factors that influence implementation, but none focused explicitly on behavioral health; focused solely on organizational culture, organizational climate, and implementation climate; or assessed the evidence base of all known uses of a measure within a given area, such as behavioral health-focused implementation efforts. The purpose of this study was to identify and assess the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs that have been used in behavioral health-focused implementation research. We identified 21 measures of organizational culture, 36 measures of organizational climate, 2 measures of implementation climate, 2 measures of tension for change, 6 measures of compatibility, 2 measures of relative priority, 3 measures of organizational incentives and rewards, 3 measures of goals and feedback, and 2 measures of learning climate. Some promising measures were identified; however, the overall state of measurement across these constructs is poor. This review highlights specific areas for improvement and suggests the need to rigorously evaluate existing measures and develop new measures.

A cluster randomized clinical trial to evaluate the effectiveness of the Implementation of Infant Pain Practice Change (ImPaC) Resource to improve pain practices in hospitalized infants: a study protocol.

BACKGROUND: Hospitalized infants undergo multiple painful procedures daily. Despite the significant evidence, procedural pain assessment and management continues to be suboptimal. Repetitive and untreated pain at this vital developmental juncture is associated with negative behavioral and neurodevelopmental consequences. To address this knowledge to practice gap, we developed the web-based Implementation of Infant Pain Practice Change (ImPaC) Resource to guide change in healthcare professionals' pain practice behaviors. This protocol describes the evaluation of the intervention effectiveness and implementation of the Resource and how organizational context influences outcomes. METHODS: An effectiveness-implementation hybrid type 1 design, blending a cluster randomized clinical trial and a mixed-methods implementation study will be used. Eighteen Neonatal Intensive Care Units (NICUs) across Canada will be randomized to intervention (INT) or standard practice (SP) groups. NICUs in the INT group will receive the Resource for six months; those in the SP group will continue with practice as usual and will be offered the Resource after a six-month waiting period. Data analysts will be blinded to group allocation. To address the intervention effectiveness, the INT and SP groups will be compared on clinical outcomes including the proportion of infants who have procedural pain assessed and managed, and the frequency and nature of painful procedures. Data will be collected at baseline (before randomization) and at completion of the intervention (six months). Implementation outcomes (feasibility, fidelity, implementation cost, and reach) will be measured at completion of the intervention. Sustainability will be assessed at six and 12 months following the intervention. Organizational context will be assessed to examine its influence on intervention and implementation outcomes. DISCUSSION: This mixed-methods study aims to determine the effectiveness and the implementation of a multifaceted online strategy for changing healthcare professionals' pain practices for hospitalized infants. Implementation strategies that are easily and effectively implemented are important for sustained change. The results will inform healthcare professionals and decision-makers on how to address the challenges of implementing the Resource within various organizational contexts. TRIAL REGISTRATION: ClinicalTrials.gov, NCT03825822. Registered 31 January 2019.

What Systematic Reviews Exist for the Effectiveness of Orthopaedic Interventions.

INTRODUCTION: Orthopaedics procedures are frequent and expensive, but highly cost effective in improving the quality of life. The purpose of this study was to determine the number and topics of systematic overviews on the effectiveness of orthopaedic interventions. METHODS: We performed a review of PubMed, Ovid Embase, Scopus, OrthoEvidence, and the Cochrane Library for dates of publication from January 1, 2006, to February 3, 2017, to identify systematic overviews of randomized clinical trials for the effectiveness of therapeutic interventions involving orthopaedic surgeons. Abstracts were excluded based on the following sequentially applied criteria: (1) the systematic review did not include an intervention for an orthopaedic condition; the intervention was not therapeutic; the intervention was not likely to be applied or influenced by an orthopaedic surgeon; (2) the study was not a systematic review or the study was a single randomized controlled trial, and/or it included nonhuman studies; (3) the systematic review included nonrandomized studies; and (4) the systematic review did not state moderate or strong evidence in support of the study conclusion(s). RESULTS: Of the 6,864 abstracts found in the searches, 6,145 were excluded yielding 719 systematic overviews. Contrary to conventional wisdom, this study identified 719 reviews of randomized controlled trials of therapeutic orthopaedic interventions. The interventions were classified as surgical in 383 (55%), medication in 245 (34%), and rehabilitation in 42 (6%), and other nonsurgical interventions in 39 (5%). DISCUSSION: This study identified many systematic overviews of orthopaedic interventions. The findings of this study could both influence clinical practice and, given the frequency of orthopaedic procedures, have a major public health impact.

Toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping.

BACKGROUND: Advancing implementation research and practice requires valid and reliable measures of implementation determinants, mechanisms, processes, strategies, and outcomes. However, researchers and implementation stakeholders are unlikely to use measures if they are not also pragmatic. The purpose of this study was to establish a stakeholder-driven conceptualization of the domains that comprise the pragmatic measure construct. It built upon a systematic review of the literature and semi-structured stakeholder interviews that generated 47 criteria for pragmatic measures, and aimed to further refine that set of criteria by identifying conceptually distinct categories of the pragmatic measure construct and providing quantitative ratings of the criteria's clarity and importance. METHODS: Twenty-four stakeholders with expertise in implementation practice completed a concept mapping activity wherein they organized the initial list of 47 criteria into conceptually distinct categories and rated their clarity and importance. Multidimensional scaling, hierarchical cluster analysis, and descriptive statistics were used to analyze the data. FINDINGS: The 47 criteria were meaningfully grouped into four distinct categories: (1) acceptable, (2) compatible, (3) easy, and (4) useful. Average ratings of clarity and importance at the category and individual criteria level will be presented. CONCLUSIONS: This study advances the field of implementation science and practice by providing clear and conceptually distinct domains of the pragmatic measure construct. Next steps will include a Delphi process to develop consensus on the most important criteria and the development of quantifiable pragmatic rating criteria that can be used to assess measures.

Training and Maintaining System-Wide Reliability in Outcome Management.

The Child and Adolescent Functional Assessment Scale (CAFAS) is widely used for outcome management, for providing real time client and program level data, and the monitoring of evidence-based practices. Methods of reliability training and the assessment of rater drift are critical for service decision-making within organizations and systems of care. We assessed two approaches for CAFAS training: external technical assistance and internal technical assistance. To this end, we sampled 315 practitioners trained by external technical assistance approach from 2,344 Ontario practitioners who had achieved reliability on the CAFAS. To assess the internal technical assistance approach as a reliable alternative training method, 140 practitioners trained internally were selected from the same pool of certified raters. Reliabilities were high for both practitioners trained by external technical assistance and internal technical assistance approaches (.909-.995, .915-.997, respectively). 1 and 3-year estimates showed some drift on several scales. High and consistent reliabilities over time and training method has implications for CAFAS training of behavioral health care practitioners, and the maintenance of CAFAS as a global outcome management tool in systems of care.

Knowledge translation efforts in child and youth mental health: a systematic review.

The availability of knowledge translation strategies that have been empirically studied and proven useful is a critical prerequisite to narrowing the research-to-practice gap in child and youth mental health. Through this review the authors sought to determine the current state of scientific knowledge of the effectiveness of knowledge translation approaches in child and youth mental health by conducting a systematic review of the research evidence. The findings and quality of the 12 included studies are discussed. Future work of high methodological quality that explores a broader range of knowledge translation strategies and practitioners to which they are applied and that also attends to implementation process is recommended.

Getting to uptake: do communities of practice support the implementation of evidence-based practice?

INTRODUCTION: Practitioners are increasingly encouraged to adopt evidence-based practices (EBP) leading to a need for new knowledge translation strategies to support implementation and practice change. This study examined the benefits of a community of practice in the context of Ontario's children's mental health sector where organizations are mandated to adopt a standardized outcome measure to monitor client response to treatment. METHOD: Readiness for change, practice change, content knowledge, and satisfaction with and use of implementation supports were examined among practitioners newly trained on the measure who were randomly assigned to a community of practice (CoP) or a practice as usual (PaU) group. CoP practitioners attended 6 sessions over 12 months; PaU practitioners had access to usual implementation supports. RESULTS: Groups did not differ on readiness for change or reported practice change, although CoP participants demonstrated greater use of the tool in practice, better content knowledge and were more satisfied with implementation supports than PaU participants. CONCLUSION: CoPs present a promising model for translating EBP knowledge and promoting practice change in children's mental health that requires further study.

Research utilization among children's mental health providers.

BACKGROUND: Children with emotional and behavioural disorders should be able to count on receiving care that meets their needs and is based on the best scientific evidence available, however, many do not receive these services. Implementation of evidence-based practice (EBP) relies, in part, on the research utilization practices of mental health care providers. This study reports on a survey of research utilization practices among 80 children's mental health (CMH) service provider organizations in Ontario, Canada. METHODS: A web-based survey was distributed to 80 CMH service provider organizations, to which 51 executive directors and 483 children's mental health practitioners responded. Research utilization was assessed using questions with Likert-type responses based on the Canadian Health Services Research Foundation's Four-A's approach: access, assess, adapt, apply. RESULTS: There was general agreement among executive directors and practitioners regarding the capacity of their organizations to use - access, assess, adapt, and apply - research evidence. Overall, both groups rated their organizations as using research information 'somewhat well.' The low response rate to the practitioner survey should be noted. CONCLUSION: These findings provide a useful benchmark from which changes in reported research utilization in the Ontario CMH sector can be tracked over time, as a function of EBP training and implementation initiatives, for instance. The need to improve access to research evidence should be addressed because it relates to the eventual implementation and uptake of evidence-based practices. Communities of practice are recommended as a strategy that would enable practitioners to build capacity in their adaptation and application of research evidence.