Audit and feedback to reduce unwarranted clinical variation at scale: a realist study of implementation strategy mechanisms.

BACKGROUND: Unwarranted clinical variation in hospital care includes the underuse, overuse, or misuse of services. Audit and feedback is a common strategy to reduce unwarranted variation, but its effectiveness varies widely across contexts. We aimed to identify implementation strategies, mechanisms, and contextual circumstances contributing to the impact of audit and feedback on unwarranted clinical variation. METHODS: Realist study examining a state-wide value-based healthcare program implemented between 2017 and 2021 in New South Wales, Australia. Three initiatives within the program included audit and feedback to reduce unwarranted variation in inpatient care for different conditions. Multiple data sources were used to formulate the initial audit and feedback program theory: a systematic review, realist review, program document review, and informal discussions with key program stakeholders. Semi-structured interviews were then conducted with 56 participants to refute, refine, or confirm the initial program theories. Data were analysed retroductively using a context-mechanism-outcome framework for 11 transcripts which were coded into the audit and feedback program theory. The program theory was validated with three expert panels: senior health leaders (n = 19), Agency for Clinical Innovation (n = 11), and Ministry of Health (n = 21) staff. RESULTS: The program's audit and feedback implementation strategy operated through eight mechanistic processes. The strategy worked well when clinicians (1) felt ownership and buy-in, (2) could make sense of the information provided, (3) were motivated by social influence, and (4) accepted responsibility and accountability for proposed changes. The success of the strategy was constrained when the audit process led to (5) rationalising current practice instead of creating a learning opportunity, (6) perceptions of unfairness and concerns about data integrity, 7) development of improvement plans that were not followed, and (8) perceived intrusions on professional autonomy. CONCLUSIONS: Audit and feedback strategies may help reduce unwarranted clinical variation in care where there is engagement between auditors and local clinicians, meaningful audit indicators, clear improvement plans, and respect for clinical expertise. We contribute theoretical development for audit and feedback by proposing a Model for Audit and Feedback Implementation at Scale. Recommendations include limiting the number of audit indicators, involving clinical staff and local leaders in feedback, and providing opportunities for reflection.

Horizon scanning, rapid reviews and living evidence to support decision-making: lessons from the work of the Critical Intelligence Unit in New South Wales, Australia during the COVID-19 pandemic.

The COVID-19 pandemic has seen an increase in rapidly disseminated scientific evidence and highlighted that traditional evidence synthesis methods, such as time and resource intensive systematic reviews, may not be successful in responding to rapidly evolving policy and practice needs. In New South Wales (NSW) Australia, the Critical Intelligence Unit (CIU) was established early in the pandemic and acted as an intermediary organisation. It brought together clinical, analytical, research, organisational and policy experts to provide timely and considered advice to decision-makers. This paper provides an overview of the functions, challenges and future implications of the CIU, particularly the Evidence Integration Team. Outputs from the Evidence Integration Team included a daily evidence digest, rapid evidence checks and living evidence tables. These products have been widely disseminated and used to inform policy decisions in NSW, making valuable impacts. Changes and innovations to evidence generation, synthesis and dissemination in response to the COVID-19 pandemic provide an opportunity to shift the way evidence is used in future. The experience and methods of the CIU have potential to be adapted and applied to the broader health system nationally and internationally.

Virtual care in end of life and palliative care: A rapid evidence check.

The purpose of this rapid evidence check was to identify virtual care modalities that are safe and effective in the delivery of end of life and palliative care. Thirty-three peer reviewed articles which were either review articles or interventional/evaluative studies presenting comparative data were identified through PubMed, Google, and Google Scholar searches. Extracted data was synthesized narratively and outcomes were categorised separately for patients, healthcare providers, caregivers and health system. Included studies reported on a wide range of virtual care modalities, including video consultation, mobile apps, videos, websites, telephone support, email and alert messages. Generally, studies reported similar or favourable quality of life outcomes to face-to-face palliative care, especially when virtual care was used as a supplement rather than a substitute for face-to-face care. Positive attitudes for perceived usefulness and helpfulness were reported by patients, caregivers and healthcare providers. Challenges identified related to technology limitations, trust, ethical concerns, administrative burden and evidence gaps. Overall, most studies found virtual care modalities to be safe and effective in end of life and palliative care with no detrimental adverse outcomes, when used as a supplement to face-to-face care.

Indirect Impacts of COVID-19: A Case Study of Evidence, Advice and Representation From Consumer and Community Members in New South Wales Australia.

The COVID-19 pandemic continues to affect health care systems globally, and there is widespread concern about the indirect impacts of COVID-19. Indirect impacts are caused by missed or delayed health care-not as a direct consequence of COVID-19 infections. This study gathered experiences of, and perspectives on, the indirect impacts of COVID-19 for health consumers, patients, their families and carers, and the broader community in New South Wales, Australia. A series of semi-structured virtual group discussions were conducted with 33 health consumers and community members between August 24 and August 31, 2020. Data were analyzed using an inductive thematic analysis approach. The analysis identified 3 main themes: poor health outcomes for individuals; problems with how health care is designed and delivered; and increasing health inequality. This case study provides insight into the indirect impacts of COVID-19. Health systems can draw on the insights learned as a source of experiential evidence to help identify, monitor and respond to the indirect impacts of COVID-19.

Impact of COVID-19 on healthcare activity in NSW, Australia.

OBJECTIVE: COVID-19 has left no healthcare system untouched. In New South Wales (NSW), the most populous state in Australia, COVID-19 case numbers have to date been relatively low. However, that does not mean the state's healthcare system has been unaffected. Preparations to create sufficient capacity to accommodate a COVID-19 surge resulted in widespread deferment of scheduled medical procedures and appointments. Patterns of healthcare-seeking behaviour changed, with a reluctance to visit healthcare settings. The aim of this study was to quantify the changes in activity seen in the NSW health system in the first half of 2020. METHODS: Healthcare data were drawn from multiple sources, including primary care, ambulance, emergency departments and inpatient settings. Volumes of healthcare activity in 2020, overall, by urgency of cases and by reasons for care were compared with the figures for the same period in 2019. Changes in the modality of care provision were also examined. RESULTS: In March to June 2020, compared with the same period in 2019, primary care face-to-face consultations decreased by 22.1%, breast screening activity by 51.5%, ambulance incidents by 7.2%, emergency department visits by 13.9%, public hospital inpatient episodes by 14.3%, and public hospital planned surgical activity by 32.6%. CONCLUSION: There were substantial declines in a wide range of healthcare activities across the NSW health system between March and June 2020 due to the impact of the COVID-19 pandemic. Although activity levels were recovering by September 2020, they had not yet returned to 'normal'. The implications of these changes - and the indirect impact of COVID-19 - require further study.

Implementing large-system, value-based healthcare initiatives: a realist study protocol for seven natural experiments.

INTRODUCTION: Value-based healthcare delivery models have emerged to address the unprecedented pressure on long-term health system performance and sustainability and to respond to the changing needs and expectations of patients. Implementing and scaling the benefits from these care delivery models to achieve large-system transformation are challenging and require consideration of complexity and context. Realist studies enable researchers to explore factors beyond 'what works' towards more nuanced understanding of 'what tends to work for whom under which circumstances'. This research proposes a realist study of the implementation approach for seven large-system, value-based healthcare initiatives in New South Wales, Australia, to elucidate how different implementation strategies and processes stimulate the uptake, adoption, fidelity and adherence of initiatives to achieve sustainable impacts across a variety of contexts. METHODS AND ANALYSIS: This exploratory, sequential, mixed methods realist study followed RAMESES II (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) reporting standards for realist studies. Stage 1 will formulate initial programme theories from review of existing literature, analysis of programme documents and qualitative interviews with programme designers, implementation support staff and evaluators. Stage 2 envisages testing and refining these hypothesised programme theories through qualitative interviews with local hospital network staff running initiatives, and analyses of quantitative data from the programme evaluation, hospital administrative systems and an implementation outcome survey. Stage 3 proposes to produce generalisable middle-range theories by synthesising data from context-mechanism-outcome configurations across initiatives. Qualitative data will be analysed retroductively and quantitative data will be analysed to identify relationships between the implementation strategies and processes, and implementation and programme outcomes. Mixed methods triangulation will be performed. ETHICS AND DISSEMINATION: Ethical approval has been granted by Macquarie University (Project ID 23816) and Hunter New England (Project ID 2020/ETH02186) Human Research Ethics Committees. The findings will be published in peer-reviewed journals. Results will be fed back to partner organisations and roundtable discussions with other health jurisdictions will be held, to share learnings.

Envisioning the future of clinical analytics: a modified Delphi process in New South Wales, Australia.

BACKGROUND: Clinical analytics is a rapidly developing area of informatics and knowledge mobilisation which has huge potential to improve healthcare in the future. It is widely acknowledged to be a powerful mediator of clinical decision making, patient-centred care and organisational learning. As a result, healthcare systems require a strategic foundation for clinical analytics that is sufficiently directional to support meaningful change while flexible enough to allow for iteration and responsiveness to context as change occurs. METHODS: In New South Wales, the most populous state in Australia, the Clinical Analytics Working Group was charged with developing a five-year vision for the public health system. A modified Delphi process was undertaken to elicit expert views and to reach a consensus. The process included a combination of face-to-face workshops, traditional Delphi voting via email, and innovative, real-time iteration between text re-formulation and voting until consensus was reached. The six stage process engaged 35 experts - practising clinicians, patients and consumers, managers, policymakers, data scientists and academics. RESULTS: The process resulted in the production of 135 ideas that were subsequently synthesised into 23 agreed statements and encapsulated in a single page (456 word) narrative. CONCLUSION: The visioning process highlighted three key perspectives (clinicians, patients and managers) and the need for synchronous (during the clinical encounter) and asynchronous (outside the clinical encounter) clinical decision support and reflective practice tools; the use of new and multiple data sources and communication formats; and the role of research and education.

Assessing Performance in Health Care Using International Surveys: Are Patient and Clinician Perspectives Complementary or Substitutive.

BACKGROUND: Over the last decade, international surveys of patients and clinicians have been used to compare health care across countries. Findings from these surveys have been extensively used to create aggregate scores and rankings. OBJECTIVE: To assess the concordance of survey responses provided by patients and clinicians. METHODS: Analysis of 16 pairs of questions that focused on coordination, organizational factors, and patient-centered competencies from the Commonwealth Fund International Health Policy Survey of older adults (2014) and of primary care physicians (2015). Concordance was assessed by comparing absolute rates and relative rankings. RESULTS: In absolute terms, patients and clinicians gave differing responses for questions about coordination of care (patients were more positive) and provision of after-hours care (patients were less positive). In relative terms, country rankings were positively correlated for 5 of 16 question pairs (Spearman ρ > .6 and P < .05). CONCLUSION: Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients' and clinicians' perspectives before combining them to create aggregate assessments of performance.

Combining patient, clinical and system perspectives in assessing performance in healthcare: an integrated measurement framework.

BACKGROUND: The science of measuring and reporting on the performance of healthcare systems is rapidly evolving. In the past decade, across many jurisdictions, organisations tasked with monitoring progress towards reform targets have broadened their purview to take a more system-functioning approach. Their aim is to bring clarity to performance assessment, using relevant and robust concepts - and avoiding reductionist measures - to build a whole-of-system view of performance. Existing performance frameworks are not fully aligned with these developments. METHODS: An eight stage process to develop a conceptual framework incorporated literature review, mapping, categorisation, integration, synthesis and validation of performance constructs that have been used by organisations and researchers in order to assess, reflect and report on healthcare performance. RESULTS: A total of 19 performance frameworks were identified and included in the review. Existing frameworks mostly adopted either a logic model (inputs, outputs and outcomes), a functional, or a goal-achievement approach. The mapping process identified 110 performance terms and concepts. These were integrated, synthesised and resynthesised to produce a framework that features 12 derived constructs reflecting combinations of patients' needs and expectations; healthcare resources and structures; receipt and experience of healthcare services; healthcare processes, functions and context; and healthcare outcomes. The 12 constructs gauge performance in terms of coverage, accessibility, appropriateness, effectiveness, safety, productivity, efficiency, impact, sustainability, resilience, adaptability and equity. They reflect four performance perspectives (patient, population, delivery organisation and system). CONCLUSIONS: Internationally, healthcare systems and researchers have used a variety of terms to categorise indicators of healthcare performance, however few frameworks are based on a theoretically-based conceptual underpinning. The proposed framework incorporates a manageable number of performance domains that together provide a comprehensive assessment, as well as conceptual and operational clarity and coherence that support multifaceted measurement systems for healthcare.

Unwarranted clinical variation in health care: Definitions and proposal of an analytic framework.

RATIONALE, AIMS, AND OBJECTIVES: Unwarranted clinical variation is a topic of heightened interest in health care systems around the world. While there are many publications and reports on clinical variation, few studies are conceptually grounded in a theoretical model. This study describes the empirical foundations of the field and proposes an analytic framework. METHOD: Structured construct mapping of published empirical studies which explicitly address unwarranted clinical variation. RESULTS: A total of 190 studies were classified in terms of three key dimensions: perspective (assessing variation across geographical areas or across providers); criteria for assessment (measuring absolute variation against a standard, or relative variation within a comparator group); and object of analysis (using process, structure/resource, or outcome metrics). CONCLUSION: Consideration of the results of the mapping exercise-together with a review of adjustment, explanatory and stratification variables, and the factors associated with residual variation-informed the development of an analytic framework. This framework highlights the role that agency and motivation, evidence and judgement, and personal and organizational capacity play in clinical decision making and reveals key facets that distinguish warranted from unwarranted clinical variation. From a measurement perspective, it underlines the need for careful consideration of attribution, aggregation, models of care, and temporality in any assessment.

Is the weekend effect really ubiquitous? A retrospective clinical cohort analysis of 30-day mortality by day of week and time of day using linked population data from New South Wales, Australia.

OBJECTIVE: To examine the associations between day of week and time of admission and 30-day mortality for six clinical conditions: ischaemic and haemorrhagic stroke, acute myocardial infarction, pneumonia, chronic obstructive pulmonary disease and congestive heart failure. DESIGN: Retrospective population-based cohort analyses. Hospitalisation records were linked to emergency department and deaths data. Random-effect logistic regression models were used, adjusting for casemix and taking into account clustering within hospitals. SETTING: All hospitals in New South Wales, Australia, from July 2009 to June 2012. PARTICIPANTS: Patients admitted to hospital with a primary diagnosis for one of the six clinical conditions examined. OUTCOME MEASURES: Adjusted ORs for all-cause mortality within 30 days of admission, by day of week and time of day. RESULTS: A total of 148 722 patients were included in the study, with 17 721 deaths within 30 days of admission. Day of week of admission was not associated with significantly higher likelihood of death for five of the six conditions after adjusting for casemix. There was significant variation in mortality for chronic obstructive pulmonary disease by day of week; however, this was not consistent with a strict weekend effect (Thursday: OR 1.29, 95% CI 1.12 to 1.48; Friday: OR 1.25, 95% CI 1.08 to 1.44; Saturday: OR 1.18, 95% CI 1.02 to 1.37; Sunday OR 1.05, 95% CI 0.90 to 1.22; compared with Monday). There was evidence for a night effect for patients admitted for stroke (ischaemic: OR 1.30, 95% CI 1.17 to 1.45; haemorrhagic: OR 1.58, 95% CI 1.40 to 1.78). CONCLUSIONS: Mortality outcomes for these conditions, adjusted for casemix, do not vary in accordance with the weekend effect hypothesis. Our findings support a growing body of evidence that questions the ubiquity of the weekend effect.

What role does performance information play in securing improvement in healthcare? a conceptual framework for levers of change.

OBJECTIVE: Across healthcare systems, there is consensus on the need for independent and impartial assessment of performance. There is less agreement about how measurement and reporting performance improves healthcare. This paper draws on academic theories to develop a conceptual framework-one that classifies in an integrated manner the ways in which change can be leveraged by healthcare performance information. METHODS: A synthesis of published frameworks. RESULTS: The framework identifies eight levers for change enabled by performance information, spanning internal and external drivers, and emergent and planned processes: (1) cognitive levers provide awareness and understanding; (2) mimetic levers inform about the performance of others to encourage emulation; (3) supportive levers provide facilitation, implementation tools or models of care to actively support change; (4) formative levers develop capabilities and skills through teaching, mentoring and feedback; (5) normative levers set performance against guidelines, standards, certification and accreditation processes; (6) coercive levers use policies, regulations incentives and disincentives to force change; (7) structural levers modify the physical environment or professional cultures and routines; (8) competitive levers attract patients or funders. CONCLUSION: This framework highlights how performance measurement and reporting can contribute to eight different levers for change. It provides guidance into how to align performance measurement and reporting into quality improvement programme.

Barriers to accessing primary health care: comparing Australian experiences internationally.

Most highly developed economies have embarked on a process of primary health care (PHC) transformation. To provide evidence on how nations vary in terms of accessing PHC, the aim of this study is to describe the extent to which barriers to access were experienced by adults in Australia compared with other countries. Communities participating in an international research project on PHC access interventions were engaged to prioritise questions from the 2013 Commonwealth Fund International Health Policy Survey within a framework that conceptualises access across dimensions of approachability, acceptability, availability, affordability and appropriateness. Logistic regression models, with barriers to access as outcomes, found measures of availability to be a problematic dimension in Australia; 27% of adults experienced difficulties with out-of-hours access, which was higher than 5 of 10 comparator countries. Although less prevalent, affordability was also perceived as a substantial barrier; 16% of Australians said they had forgone health care due to cost in the previous year. After adjusting for age and health status, this barrier was more common in Australia than 7 of 10 countries. Findings of this integrated assessment of barriers to access offer insights for policymakers and researchers on Australia's international performance in this crucial PHC domain.

Chartbook: shining a light on the quality of healthcare in Canada: what can be learned to catalyze improvements in healthcare quality?

This paper provides a reflection on the findings of Canada's first-ever chartbook on the quality of healthcare in Canada. Quality of Healthcare in Canada: A Chartbook was published in 2010 by the Canadian Health Services Research Foundation in partnership with the Canadian Institute for Health Information and the Canadian Patient Safety Institute, and with support from Statistics Canada. This paper, by the chartbook authors (Sutherland and Leatherman) and colleagues (Law, Verma and Petersen), presents selected key findings and lessons from the chartbook and aims to serve as a catalyst for ideas and discussion in the papers that follow. The chartbook identified a lack of common language and indicators on quality across Canada's provinces and territories, underscoring the need to create and coordinate core measures. The Canadian chartbook and this issue of Healthcare Papers provide an update on the existing quality measures and the state of healthcare quality in Canada, and create the opportunity for jurisdictions to learn from one another and to contemplate the steps required to improve quality across the country.