Implementation of large, multi-site hospital interventions: a realist evaluation of strategies for developing capability.

BACKGROUND: This study presents guidelines for implementation distilled from the findings of a realist evaluation. The setting was local health districts in New South Wales, Australia that implemented three clinical improvement initiatives as part of a state-wide program. We focussed on implementation strategies designed to develop health professionals' capability to deliver value-based care initiatives for multisite programs. Capability, which increases implementers' ability to cope with unexpected scenarios is key to managing change. METHODS: We used a mixed methods realist evaluation which tested and refined program theories elucidating the complex dynamic between context (C), mechanism (M) and outcome (O) to determine what works, for whom, under what circumstances. Data was drawn from program documents, a realist synthesis, informal discussions with implementation designers, and interviews with 10 key informants (out of 37 identified) from seven sites. Data analysis employed a retroductive approach to interrogate the causal factors identified as contributors to outcomes. RESULTS: CMO statements were refined for four initial program theories: Making it Relevant- where participation in activities was increased when targeted to the needs of the staff; Investment in Quality Improvement- where engagement in capability development was enhanced when it was valued by all levels of the organisation; Turnover and Capability Loss- where the effects of staff turnover were mitigated; and Community-Wide Priority- where there was a strategy of spanning sites. From these data five guiding principles for implementers were distilled: (1) Involve all levels of the health system to effectively implement large-scale capability development, (2) Design capability development activities in a way that supports a learning culture, (3) Plan capability development activities with staff turnover in mind, (4) Increased capability should be distributed across teams to avoid bottlenecks in workflows and the risk of losing key staff, (5) Foster cross-site collaboration to focus effort, reduce variation in practice and promote greater cohesion in patient care. CONCLUSIONS: A key implementation strategy for interventions to standardise high quality practice is development of clinical capability. We illustrate how leadership support, attention to staff turnover patterns, and making activities relevant to current issues, can lead to an emergent learning culture.

The Mental Wellbeing of Child and Adolescent Mental Health Service (CAMHS) Workers in England: A Cross-Sectional Descriptive Study Reporting Levels of Burnout, Wellbeing and Job Satisfaction.

In the UK, there has been a notable increase in referrals to specialist children's mental health services. This, coupled with shortages of qualified staff, has raised concerns about the escalating occupational stress experienced by staff in this sector. In this brief report, we present cross-sectional quantitative data from 97 staff members working in one Child and Adolescent Mental Health Service (CAMHS) in the UK during spring 2023, reporting on their wellbeing, job satisfaction, and burnout. Our findings reveal that over a third of CAMHS staff experienced moderate or high levels of work-related burnout; 39% reported moderate or high levels of personal burnout, but levels of client-related burnout were much lower (13%). Both work- and client-related burnout showed a robust negative relationship with job satisfaction, with higher burnout predicting lower levels of job satisfaction. Only a small proportion of respondents reported high levels of wellbeing, with about a quarter experiencing levels of wellbeing that can be considered indicative of mild or clinical depressive symptoms. Whilst these results are from a small sample in one area of the UK, they present an important snapshot of CAMHS staff wellbeing and are discussed in the context of similar trends reported in the wider NHS sector.

Four System Enablers of Large-System Transformation in Health Care: A Mixed Methods Realist Evaluation.

Policy Points The implementation of large-scale health care interventions relies on a shared vision, commitment to change, coordination across sites, and a spanning of siloed knowledge. Enablers of the system should include building an authorizing environment; providing relevant, meaningful, transparent, and timely data; designating and distributing leadership and decision making; and fostering the emergence of a learning culture. Attention to these four enablers can set up a positive feedback loop to foster positive change that can protect against the loss of key staff, the presence of lone disruptors, and the enervating effects of uncertainty. CONTEXT: Large-scale transformative initiatives have the potential to improve the quality, efficiency, and safety of health care. However, change is expensive, complex, and difficult to implement and sustain. This paper advances system enablers, which will help to guide large-scale transformation in health care systems. METHODS: A realist study of the implementation of a value-based health care program between 2017 and 2021 was undertaken in every public hospital (n = 221) in New South Wales (NSW), Australia. Four data sources were used to elucidate initial program theories beginning with a set of literature reviews, a program document review, and informal discussions with key stakeholders. Semistructured interviews were then conducted with 56 stakeholders to confirm, refute, or refine the theories. A retroductive analysis produced a series of context-mechanism-outcome (CMO) statements. Next, the CMOs were validated with three health care quality expert panels (n = 51). Synthesized data were interrogated to distill the overarching system enablers. FINDINGS: Forty-two CMO statements from the eight initial program theory areas were developed, refined, and validated. Four system enablers were identified: (1) build an authorizing environment; (2) provide relevant, authentic, timely, and meaningful data; (3) designate and distribute leadership and decision making; and (4) support the emergence of a learning culture. The system enablers provide a nuanced understanding of large-system transformation that illustrates when, for whom, and in what circumstances large-system transformation worked well or worked poorly. CONCLUSIONS: System enablers offer nuanced guidance for the implementation of large-scale health care interventions. The four enablers may be portable to similar contexts and provide the empirical basis for an implementation model of large-system value-based health care initiatives. With concerted application, these findings can pave the way not just for a better understanding of greater or lesser success in intervening in health care settings but ultimately to contribute higher quality, higher value, and safer care.

Audit and feedback to reduce unwarranted clinical variation at scale: a realist study of implementation strategy mechanisms.

BACKGROUND: Unwarranted clinical variation in hospital care includes the underuse, overuse, or misuse of services. Audit and feedback is a common strategy to reduce unwarranted variation, but its effectiveness varies widely across contexts. We aimed to identify implementation strategies, mechanisms, and contextual circumstances contributing to the impact of audit and feedback on unwarranted clinical variation. METHODS: Realist study examining a state-wide value-based healthcare program implemented between 2017 and 2021 in New South Wales, Australia. Three initiatives within the program included audit and feedback to reduce unwarranted variation in inpatient care for different conditions. Multiple data sources were used to formulate the initial audit and feedback program theory: a systematic review, realist review, program document review, and informal discussions with key program stakeholders. Semi-structured interviews were then conducted with 56 participants to refute, refine, or confirm the initial program theories. Data were analysed retroductively using a context-mechanism-outcome framework for 11 transcripts which were coded into the audit and feedback program theory. The program theory was validated with three expert panels: senior health leaders (n = 19), Agency for Clinical Innovation (n = 11), and Ministry of Health (n = 21) staff. RESULTS: The program's audit and feedback implementation strategy operated through eight mechanistic processes. The strategy worked well when clinicians (1) felt ownership and buy-in, (2) could make sense of the information provided, (3) were motivated by social influence, and (4) accepted responsibility and accountability for proposed changes. The success of the strategy was constrained when the audit process led to (5) rationalising current practice instead of creating a learning opportunity, (6) perceptions of unfairness and concerns about data integrity, 7) development of improvement plans that were not followed, and (8) perceived intrusions on professional autonomy. CONCLUSIONS: Audit and feedback strategies may help reduce unwarranted clinical variation in care where there is engagement between auditors and local clinicians, meaningful audit indicators, clear improvement plans, and respect for clinical expertise. We contribute theoretical development for audit and feedback by proposing a Model for Audit and Feedback Implementation at Scale. Recommendations include limiting the number of audit indicators, involving clinical staff and local leaders in feedback, and providing opportunities for reflection.

Guiding principles for effective collaborative implementation strategies for multisite hospital improvement initiatives: a mixed-method realist evaluation of collaborative strategies used in four multisite initiatives at public hospitals in New South Wales, Australia.

OBJECTIVE: Large-scale, multisite hospital improvement initiatives can advance high-quality care for patients. Implementation support is key to adoption of change in this context. Strategies that foster collaboration within local teams, across sites and between initiative developers and users are important. However not all implementation strategies are successful in all settings, sometimes realising poor or unintended outcomes. Our objective here is to develop guiding principles for effective collaborative implementation strategies for multi-site hospital initiatives. DESIGN: Mixed-method realist evaluation. Realist studies aim to examine the underlying theories that explain differing outcomes, identifying mechanisms and contextual factors that may trigger them. SETTING: We report on collaborative strategies used in four multi-site initiatives conducted in all public hospitals in New South Wales, Australia (n>100). PARTICIPANTS: Using an iterative process, information was gathered on collaborative implementation strategies used, then initial programme theories hypothesised to underlie the strategies' outcomes were surfaced using a realist dialogic approach. A realist interview schedule was developed to elicit evidence for the posited initial programme theories. Fourteen participants from 20 key informants invited participated. Interviews were conducted via Zoom, transcribed and analysed. From these data, guiding principles of fostering collaboration were developed. RESULTS: Six guiding principles were distilled: (1) structure opportunities for collaboration across sites; (2) facilitate meetings to foster learning and problem-solving across sites; (3) broker useful long-term relationships; (4) enable support agencies to assist implementers by giving legitimacy to their efforts in the eyes of senior management; (5) consider investment in collaboration as effective well beyond the current projects; (6) promote a shared vision and build momentum for change by ensuring inclusive networks where everyone has a voice. CONCLUSION: Structuring and supporting collaboration in large-scale initiatives is a powerful implementation strategy if contexts described in the guiding principles are present.

Making implementation science more real.

Implementation science in healthcare aims to understand how to get evidence into practice. Once this is achieved in one setting, it becomes increasingly difficult to replicate elsewhere. The problem is often attributed to differences in context that influence how and whether implementation strategies work. We argue that realist research paradigms provide a useful framework to express the effect of contextual factors within implementation strategy causal processes. Realist studies are theory-driven evaluations that focus on understanding how and why interventions work under different circumstances. They consider the interaction between contextual circumstances, theoretical mechanisms of change and the outcomes they produce, to arrive at explanations of conditional causality (i.e., what tends to work, for whom, under what circumstances). This Commentary provides example applications using preliminary findings from a large realist implementation study of system-wide value-based healthcare initiatives in New South Wales, Australia. If applied judiciously, realist implementation studies may represent a sound approach to help optimise delivery of the right care in the right setting and at the right time.

Conceptualising contexts, mechanisms and outcomes for implementing large-scale, multisite hospital improvement initiatives: a realist synthesis.

DESIGN: Realist synthesis. STUDY BACKGROUND: Large-scale hospital improvement initiatives can standardise healthcare across multiple sites but results are contingent on the implementation strategies that complement them. The benefits of these implemented interventions are rarely able to be replicated in different contexts. Realist studies explore this phenomenon in depth by identifying underlying context-mechanism-outcome interactions. OBJECTIVES: To review implementation strategies used in large-scale hospital initiatives and hypothesise initial programme theories for how they worked across different contexts. METHODS: An iterative, four-step process was applied. Step 1 explored the concepts inherent in large-scale interventions using database searches and snowballing. Step 2 identified strategies used in their implementation. Step 3 identified potential initial programme theories that may explain strategies' mechanisms. Step 4 focused on one strategy-theory pairing to develop and test context-mechanism-outcome hypotheses. Data was drawn from searches (March-May 2020) of MEDLINE, Embase, PubMed and CINAHL, snowballed from key papers, implementation support websites and the expertise of the research team and experts. INCLUSION CRITERIA: reported implementation of a large-scale, multisite hospital intervention. RAMESES reporting standards were followed. RESULTS: Concepts were identified from 51 of 381 articles. Large-scale hospital interventions were characterised by a top-down approach, external and internal support and use of evidence-based interventions. We found 302 reports of 28 different implementation strategies from 31 reviews (from a total of 585). Formal theories proposed for the implementation strategies included Diffusion of Innovation, and Organisational Readiness Theory. Twenty-three context-mechanism-outcome statements for implementation strategies associated with planning and assessment activities were proposed. Evidence from the published literature supported the hypothesised programme theories and were consistent with Organisational Readiness Theory's tenets. CONCLUSION: This paper adds to the literature exploring why large-scale hospital interventions are not always successfully implemented and suggests 24 causative mechanisms and contextual factors that may drive outcomes in the planning and assessment stage.

Combining Realist approaches and Normalization Process Theory to understand implementation: a systematic review.

BACKGROUND: Realist approaches and Normalization Process Theory (NPT) have both gained significant traction in implementation research over the past 10 years. The aim of this study was therefore to explore how the approaches are combined to understand problems of implementation, to determine the degree of complementarity of the two approaches and to provide practical approaches for using them together. METHODS: Systematic review of research studies combining Realist and NPT approaches. Realist methodology is concerned with understanding and explaining causation, that is, how and why policies, programmes and interventions achieve their effects. NPT is a theory of implementation that explains how practices become normalised. Databases searched (January 2020) were ASSIA, CINAHL, Health Research Premium Collection via Proquest (Family Health Database, Health & Medical Collection, Health Management Database, MEDLINE, Nursing & Allied Health Database, Psychology Database, Public Health Database) and PsycARTICLES. Studies were included if the author(s) stated they used both approaches: a scientific Realist perspective applying the principles of Pawson and Tilley's Realist Evaluation or Pawson's Realist Synthesis and Normalization Process Theory either solely or in addition to other theories. Two authors screened records; discrepancies were reviewed by a third screener. Data was extracted by three members of the team and a narrative synthesis was undertaken. RESULTS: Of 245 total records identified, 223 unique records were screened and 39 full-text papers were reviewed, identifying twelve papers for inclusion in the review. These papers represented eight different studies. Extent and methods of integration of the approaches varied. In most studies (6/8), Realist approaches were the main driver. NPT was mostly used to enhance the explanatory power of Realist analyses, informing development of elements of Contexts, Mechanisms and Outcomes (a common heuristic in realist work). Authors' reflections on the integration of NPT and Realist approaches were limited. CONCLUSIONS: Using Realist and NPT approaches in combination can add explanatory power for understanding the implementation of interventions and programmes. Attention to detailed reporting on methods and analytical process when combining approaches, and appraisal of theoretical and practical utility is advised for advancing knowledge of applying these approaches in research. SYSTEMATIC REVIEW REGISTRATION: Not registered.

Using realist approaches in nutrition and dietetics research.

AIM: The aim of this study was to explore the use and future potential of realist approaches to research in nutrition and dietetics. METHODS: A targeted literature review was used to search key journals (n = 7) in nutrition and dietetics to identify existing research using a realist approach. A narrative synthesis was conducted to explore findings in relation to the research aim. RESULTS: Nine research papers (four realist evaluations, five realist reviews) describing seven nutrition interventions were found, which revealed the application of realist research in nutrition and dietetics has focused on public health interventions. Realist research provided a deeper, more nuanced understanding of varied outcomes including the role of context, and contributed to the development of theory about how and why interventions work. As a theory-driven research method, realist research was able to assist in overcoming methodological shortcomings to contribute to meaningful, transferable findings. CONCLUSION: The results highlight the potential contribution of the realist research in nutrition and dietetics to evaluate interventions and inform future practice.

Implementing large-system, value-based healthcare initiatives: a realist study protocol for seven natural experiments.

INTRODUCTION: Value-based healthcare delivery models have emerged to address the unprecedented pressure on long-term health system performance and sustainability and to respond to the changing needs and expectations of patients. Implementing and scaling the benefits from these care delivery models to achieve large-system transformation are challenging and require consideration of complexity and context. Realist studies enable researchers to explore factors beyond 'what works' towards more nuanced understanding of 'what tends to work for whom under which circumstances'. This research proposes a realist study of the implementation approach for seven large-system, value-based healthcare initiatives in New South Wales, Australia, to elucidate how different implementation strategies and processes stimulate the uptake, adoption, fidelity and adherence of initiatives to achieve sustainable impacts across a variety of contexts. METHODS AND ANALYSIS: This exploratory, sequential, mixed methods realist study followed RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) reporting standards for realist studies. Stage 1 will formulate initial programme theories from review of existing literature, analysis of programme documents and qualitative interviews with programme designers, implementation support staff and evaluators. Stage 2 envisages testing and refining these hypothesised programme theories through qualitative interviews with local hospital network staff running initiatives, and analyses of quantitative data from the programme evaluation, hospital administrative systems and an implementation outcome survey. Stage 3 proposes to produce generalisable middle-range theories by synthesising data from context-mechanism-outcome configurations across initiatives. Qualitative data will be analysed retroductively and quantitative data will be analysed to identify relationships between the implementation strategies and processes, and implementation and programme outcomes. Mixed methods triangulation will be performed. ETHICS AND DISSEMINATION: Ethical approval has been granted by Macquarie University (Project ID 23816) and Hunter New England (Project ID 2020/ETH02186) Human Research Ethics Committees. The findings will be published in peer-reviewed journals. Results will be fed back to partner organisations and roundtable discussions with other health jurisdictions will be held, to share learnings.

Realism and resources: Towards more explanatory economic evaluation.

To be successfully and sustainably adopted, policy-makers, service managers and practitioners want public programmes to be affordable and cost-effective, as well as effective. While the realist evaluation question is often summarised as what works for whom, under what circumstances, we believe the approach can be as salient to answering questions about resource use, costs and cost-effectiveness - the traditional domain of economic evaluation methods. This paper first describes the key similarities and differences between economic evaluation and realist evaluation. It summarises what health economists see as the challenges of evaluating complex interventions, and their suggested solutions. We then use examples of programme theory from a recent realist review of shared care for chronic conditions to illustrate two ways in which realist evaluations might better capture the resource requirements and resource consequences of programmes, and thereby produce explanations of how they are linked to outcomes (i.e. explanations of cost-effectiveness).

How do third sector organisations use research and other knowledge? A systematic scoping review.

BACKGROUND: Third sector organisations (TSOs) are a well-established component of health care provision in the UK's NHS and other health systems, but little is known about how they use research and other forms of knowledge in their work. There is an emerging body of evidence exploring these issues but there is no review of this literature. This scoping review summarises what is known about how health and social care TSOs use research and other forms of knowledge in their work. METHODS: A systematic search of electronic databases was carried out with initial exploratory searching of knowledge mobilisation websites, contacting authors, and hand searching of journals. The literature was narratively summarised to describe how TSOs use knowledge in decision making. RESULTS: Ten qualitative and mixed methods studies were retrieved. They show that TSOs wish to be "evidence-informed" in their decision making, and organisational context influences the kinds of research and knowledge they prefer, as well as how they use it. Barriers to research use include time, staff skill, resources and the acontextual nature of some academic research. Appropriate approaches to knowledge mobilisation may include using research intermediaries, involving TSOs in research, and better description of interventions and contexts in academic publications to aid applying it in the multi-disciplinary contexts of TSOs. TSOs identified specific benefits of using research, such as confidence that services were good quality, ability to negotiate with stakeholders and funders, and saving time and resources through implementing interventions shown to be effective. The small number of included studies means the findings need further confirmation through primary research. CONCLUSIONS: As the contribution of health and social care TSOs to service delivery is growing, the need to understand how they mobilise research and other forms of knowledge will continue. The research community could 1) develop relationships with TSOs to support the design and development of research projects, 2) use a range of methods to evaluate interventions to facilitate TSOs applying them to their organisational contexts and 3) improve our understanding of how TSOs use knowledge, through the use of complementary research methods, such as a realist review or ethnography.

Exploring reasons for variation in ordering thyroid function tests in primary care: a qualitative study.

BACKGROUND: The ordering of thyroid function tests (TFTs) is increasing but there is not a similar increase in thyroid disorders in the general population, leading some to query whether inappropriate testing is taking place. Inconsistent clinical practice is thought to be a cause of this, but there is little evidence of the views of general practitioners, practice nurses or practice managers on the reasons for variation in the ordering of TFTs. AIM: To find out the reasons for variation in ordering of TFTs from the perspective of primary healthcare professionals Methods: Fifteen semi-structured interviews were carried out with primary healthcare professionals (general practitioners, practice nurses, practice managers) that used one laboratory of a general hospital in South West England for TFTs. Framework Analysis was used to analyse views on test ordering variation at the societal, practice, individual practitioner and patient level. RESULTS: A number of reasons for variation in ordering across practices were suggested. These related to: primary healthcare professionals awareness of and adherence to national policy changes; practices having different protocols on TFTs ordering; the set-up and use of computer systems in practices; the range of practice healthcare professionals able to order TFTs; greater risk-aversion amongst general practitioners and changes in their training and finally how primary healthcare staff responded to patients who were perceived to seek help more readily than in the past. CONCLUSION: The reasons for variation in TFTs ordering are complex and interdependent. Interventions to reduce variation in TFTs ordering need to consider multiple behavioural and contextual factors to be most effective.

The effectiveness and cost-effectiveness of shared care: protocol for a realist review.

BACKGROUND: Shared care (an enhanced information exchange over and above routine outpatient letters) is commonly used to improve care coordination and communication between a specialist and primary care services for people with long-term conditions. Evidence of the effectiveness and cost-effectiveness of shared care is mixed. Informed decision-making for targeting shared care requires a greater understanding of how it works, for whom it works, in what contexts and why. This protocol outlines how realist review methods can be used to synthesise evidence on shared care for long-term conditions.A further aim of the review is to explore economic evaluations of shared care. Economic evaluations are difficult to synthesise due to problems in accounting for contextual differences that impact on resource use and opportunity costs. Realist review methods have been suggested as a way to overcome some of these issues, so this review will also assess whether realist review methods are amenable to synthesising economic evidence. METHODS/DESIGN: Database and web searching will be carried out in order to find relevant evidence to develop and test programme theories about how shared care works. The review will have two phases. Phase 1 will concentrate on the contextual conditions and mechanisms that influence how shared care works, in order to develop programme theories, which partially explain how it works. Phase 2 will focus on testing these programme theories. A Project Reference Group made up of health service professionals and people with actual experience of long-term conditions will be used to ground the study in real-life experience. Review findings will be disseminated through local and sub-national networks for integrated care and long-term conditions. DISCUSSION: This realist review will explore why and for whom shared care works, in order to support decision-makers working to improve the effectiveness of care for people outside hospital. The development of realist review methods to take into account cost and cost-effectiveness evidence is particularly innovative and challenging, and if successful will offer a new approach to synthesising economic evidence. This systematic review protocol is registered on the PROSPERO database (registration number: CRD42012002842).