Evaluation of a pharmacist-led actionable audit and feedback intervention for improving medication safety in UK primary care: An interrupted time series analysis.

BACKGROUND: We evaluated the impact of the pharmacist-led Safety Medication dASHboard (SMASH) intervention on medication safety in primary care. METHODS AND FINDINGS: SMASH comprised (1) training of clinical pharmacists to deliver the intervention; (2) a web-based dashboard providing actionable, patient-level feedback; and (3) pharmacists reviewing individual at-risk patients, and initiating remedial actions or advising general practitioners on doing so. It was implemented in 43 general practices covering a population of 235,595 people in Salford (Greater Manchester), UK. All practices started receiving the intervention between 18 April 2016 and 26 September 2017. We used an interrupted time series analysis of rates (prevalence) of potentially hazardous prescribing and inadequate blood-test monitoring, comparing observed rates post-intervention to extrapolations from a 24-month pre-intervention trend. The number of people registered to participating practices and having 1 or more risk factors for being exposed to hazardous prescribing or inadequate blood-test monitoring at the start of the intervention was 47,413 (males: 23,073 [48.7%]; mean age: 60 years [standard deviation: 21]). At baseline, 95% of practices had rates of potentially hazardous prescribing (composite of 10 indicators) between 0.88% and 6.19%. The prevalence of potentially hazardous prescribing reduced by 27.9% (95% CI 20.3% to 36.8%, p < 0.001) at 24 weeks and by 40.7% (95% CI 29.1% to 54.2%, p < 0.001) at 12 months after introduction of SMASH. The rate of inadequate blood-test monitoring (composite of 2 indicators) reduced by 22.0% (95% CI 0.2% to 50.7%, p = 0.046) at 24 weeks; the change at 12 months (23.5%) was no longer significant (95% CI -4.5% to 61.6%, p = 0.127). After 12 months, 95% of practices had rates of potentially hazardous prescribing between 0.74% and 3.02%. Study limitations include the fact that practices were not randomised, and therefore unmeasured confounding may have influenced our findings. CONCLUSIONS: The SMASH intervention was associated with reduced rates of potentially hazardous prescribing and inadequate blood-test monitoring in general practices. This reduction was sustained over 12 months after the start of the intervention for prescribing but not for monitoring of medication. There was a marked reduction in the variation in rates of hazardous prescribing between practices.

Understanding the utilisation of a novel interactive electronic medication safety dashboard in general practice: a mixed methods study.

BACKGROUND: Improving medication safety is a major concern in primary care settings worldwide. The Salford Medication safety dASHboard (SMASH) intervention provided general practices in Salford (Greater Manchester, UK) with feedback on their safe prescribing and monitoring of medications through an online dashboard, and input from practice-based trained clinical pharmacists. In this study we explored how staff working in general practices used the SMASH dashboard to improve medication safety, through interactions with the dashboard to identify potential medication safety hazards and their workflow to resolve identified hazards. METHODS: We used a mixed-methods study design involving quantitative data from dashboard user interaction logs from 43 general practices during the first year of receiving the SMASH intervention, and qualitative data from semi-structured interviews with 22 pharmacists and physicians from 18 practices in Salford. RESULTS: Practices interacted with the dashboard a median of 12.0 (interquartile range, 5.0-15.2) times per month during the first quarter of use to identify and resolve potential medication safety hazards, typically starting with the most prevalent hazards or those they perceived to be most serious. Having observed a potential hazard, pharmacists and practice staff worked together to resolve that in a sequence of steps (1) verifying the dashboard information, (2) reviewing the patient's clinical records, and (3) deciding potential changes to the patient's medicines. Over time, dashboard use transitioned towards regular but less frequent (median of 5.5 [3.5-7.9] times per month) checks to identify and resolve new cases. The frequency of dashboard use was higher in practices with a larger number of at-risk patients. In 24 (56%) practices only pharmacists used the dashboard; in 12 (28%) use by other practice staff increased as pharmacist use declined after the initial intervention period; and in 7 (16%) there was mixed use by both pharmacists and practice staff over time. CONCLUSIONS: An online medication safety dashboard enabled pharmacists to identify patients at risk of potentially hazardous prescribing. They subsequently worked with GPs to resolve risks on a case-by-case basis, but there were marked variations in processes between some practices. Workload diminished over time as it shifted towards resolving new cases of hazardous prescribing.

Facilitating action planning within audit and feedback interventions: a mixed-methods process evaluation of an action implementation toolbox in intensive care.

BACKGROUND: Audit and feedback (A&F) is more effective if it facilitates action planning, but little is known about how best to do this. We developed an electronic A&F intervention with an action implementation toolbox to improve pain management in intensive care units (ICUs); the toolbox contained suggested actions for improvement. A head-to-head randomised trial demonstrated that the toolbox moderately increased the intervention's effectiveness when compared with A&F only. OBJECTIVE: To understand the mechanisms through which A&F with action implementation toolbox facilitates action planning by ICUs to increase A&F effectiveness. METHODS: We extracted all individual actions from action plans developed by ICUs that received A&F with (n = 10) and without (n = 11) toolbox for 6 months and classified them using Clinical Performance Feedback Intervention Theory. We held semi-structured interviews with participants during the trial. We compared the number and type of planned and completed actions between study groups and explored barriers and facilitators to effective action planning. RESULTS: ICUs with toolbox planned more actions directly aimed at improving practice (p = 0.037) and targeted a wider range of practice determinants compared to ICUs without toolbox. ICUs with toolbox also completed more actions during the study period, but not significantly (p = 0.142). ICUs without toolbox reported more difficulties in identifying what actions they could take. Regardless of the toolbox, all ICUs still experienced barriers relating to the feedback (low controllability, accuracy) and organisational context (competing priorities, resources, cost). CONCLUSIONS: The toolbox helped health professionals to broaden their mindset about actions they could take to change clinical practice. Without the toolbox, professionals tended to focus more on feedback verification and exploring solutions without developing intentions for actual change. All feedback recipients experienced organisational barriers that inhibited eventual completion of actions. TRIAL REGISTRATION: ClinicalTrials.gov, NCT02922101 . Registered on 26 September 2016.

Control Theory to Design and Evaluate Audit and Feedback Interventions.

Control theory is about the processes underlying the behaviour of self-regulating agents. It proposes that behaviour is regulated by a negative feedback loop, in which the agent compares the perception of its current state against a goal state and will strive to reduce perceived discrepancies by modifying its behaviour. Although studies in health informatics often do not report the use of this theory, the principle of a negative feedback loop underlies many applications in the field. This chapter describes how control theory fits within health informatics, discussing its role in the development and assessment of audit and feedback interventions in healthcare. Control theory has been used to synthesise evidence of audit and feedback, and to design and evaluate interventions to improve the quality of blood transfusion practice, cardiac rehabilitation, and intensive care. This has driven progress in our understanding of the underlying mechanisms of audit and feedback for improving health care, and has helped to design better interventions.

Impact of audit and feedback with action implementation toolbox on improving ICU pain management: cluster-randomised controlled trial.

BACKGROUND: Audit and feedback (A&F) enjoys widespread use, but often achieves only marginal improvements in care. Providing recipients of A&F with suggested actions to overcome barriers (action implementation toolbox) may increase effectiveness. OBJECTIVE: To assess the impact of adding an action implementation toolbox to an electronic A&F intervention targeting quality of pain management in intensive care units (ICUs). TRIAL DESIGN: Two-armed cluster-randomised controlled trial. Randomisation was computer generated, with allocation concealment by a researcher, unaffiliated with the study. Investigators were not blinded to the group assignment of an ICU. PARTICIPANTS: Twenty-one Dutch ICUs and patients eligible for pain measurement. INTERVENTIONS: Feedback-only versus feedback with action implementation toolbox. OUTCOME: Proportion of patient-shift observations where pain management was adequate; composed by two process (measuring pain at least once per patient in each shift; re-measuring unacceptable pain scores within 1 hour) and two outcome indicators (acceptable pain scores; unacceptable pain scores normalised within 1 hour). RESULTS: 21 ICUs (feedback-only n=11; feedback-with-toolbox n=10) with a total of 253 530 patient-shift observations were analysed. We found absolute improvement on adequate pain management in the feedback-with-toolbox group (14.8%; 95% CI 14.0% to 15.5%) and the feedback-only group (4.8%; 95% CI 4.2% to 5.5%). Improvement was limited to the two process indicators. The feedback-with-toolbox group achieved larger effects than the feedback-only group both on the composite adequate pain management (p<0.05) and on measuring pain each shift (p<0.001). No important adverse effects have occurred. CONCLUSION: Feedback with toolbox improved the number of shifts where patients received adequate pain management compared with feedback alone, but only in process and not outcome indicators. TRIAL REGISTRATION NUMBER: NCT02922101.

Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research.

BACKGROUND: Providing health professionals with quantitative summaries of their clinical performance when treating specific groups of patients ("feedback") is a widely used quality improvement strategy, yet systematic reviews show it has varying success. Theory could help explain what factors influence feedback success, and guide approaches to enhance effectiveness. However, existing theories lack comprehensiveness and specificity to health care. To address this problem, we conducted the first systematic review and synthesis of qualitative evaluations of feedback interventions, using findings to develop a comprehensive new health care-specific feedback theory. METHODS: We searched MEDLINE, EMBASE, CINAHL, Web of Science, and Google Scholar from inception until 2016 inclusive. Data were synthesised by coding individual papers, building on pre-existing theories to formulate hypotheses, iteratively testing and improving hypotheses, assessing confidence in hypotheses using the GRADE-CERQual method, and summarising high-confidence hypotheses into a set of propositions. RESULTS: We synthesised 65 papers evaluating 73 feedback interventions from countries spanning five continents. From our synthesis we developed Clinical Performance Feedback Intervention Theory (CP-FIT), which builds on 30 pre-existing theories and has 42 high-confidence hypotheses. CP-FIT states that effective feedback works in a cycle of sequential processes; it becomes less effective if any individual process fails, thus halting progress round the cycle. Feedback's success is influenced by several factors operating via a set of common explanatory mechanisms: the feedback method used, health professional receiving feedback, and context in which feedback takes place. CP-FIT summarises these effects in three propositions: (1) health care professionals and organisations have a finite capacity to engage with feedback, (2) these parties have strong beliefs regarding how patient care should be provided that influence their interactions with feedback, and (3) feedback that directly supports clinical behaviours is most effective. CONCLUSIONS: This is the first qualitative meta-synthesis of feedback interventions, and the first comprehensive theory of feedback designed specifically for health care. Our findings contribute new knowledge about how feedback works and factors that influence its effectiveness. Internationally, practitioners, researchers, and policy-makers can use CP-FIT to design, implement, and evaluate feedback. Doing so could improve care for large numbers of patients, reduce opportunity costs, and improve returns on financial investments. TRIAL REGISTRATION: PROSPERO, CRD42015017541.

Clinical performance comparators in audit and feedback: a review of theory and evidence.

BACKGROUND: Audit and feedback (A&F) is a common quality improvement strategy with highly variable effects on patient care. It is unclear how A&F effectiveness can be maximised. Since the core mechanism of action of A&F depends on drawing attention to a discrepancy between actual and desired performance, we aimed to understand current and best practices in the choice of performance comparator. METHODS: We described current choices for performance comparators by conducting a secondary review of randomised trials of A&F interventions and identifying the associated mechanisms that might have implications for effective A&F by reviewing theories and empirical studies from a recent qualitative evidence synthesis. RESULTS: We found across 146 trials that feedback recipients' performance was most frequently compared against the performance of others (benchmarks; 60.3%). Other comparators included recipients' own performance over time (trends; 9.6%) and target standards (explicit targets; 11.0%), and 13% of trials used a combination of these options. In studies featuring benchmarks, 42% compared against mean performance. Eight (5.5%) trials provided a rationale for using a specific comparator. We distilled mechanisms of each comparator from 12 behavioural theories, 5 randomised trials, and 42 qualitative A&F studies. CONCLUSION: Clinical performance comparators in published literature were poorly informed by theory and did not explicitly account for mechanisms reported in qualitative studies. Based on our review, we argue that there is considerable opportunity to improve the design of performance comparators by (1) providing tailored comparisons rather than benchmarking everyone against the mean, (2) limiting the amount of comparators being displayed while providing more comparative information upon request to balance the feedback's credibility and actionability, (3) providing performance trends but not trends alone, and (4) encouraging feedback recipients to set personal, explicit targets guided by relevant information.

Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback.

Audit and feedback (A&F) is a commonly used quality improvement (QI) approach. A Cochrane review indicates that A&F is generally effective and leads to modest improvements in professional practice but with considerable variation in the observed effects. While we have some understanding of factors that enhance the effects of A&F, further research needs to explore when A&F is most likely to be effective and how to optimise it. To do this, we need to move away from two-arm trials of A&F compared with control in favour of head-to-head trials of different ways of providing A&F. This paper describes implementation laboratories involving collaborations between healthcare organisations providing A&F at scale, and researchers, to embed head-to-head trials into routine QI programmes. This can improve effectiveness while producing generalisable knowledge about how to optimise A&F. We also describe an international meta-laboratory that aims to maximise cross-laboratory learning and facilitate coordination of A&F research.

Measuring quality indicators to improve pain management in critically ill patients.

PURPOSE: To evaluate the quality of pain assessment in Dutch ICUs and its room for improvement. MATERIALS AND METHODS: We used a modified RAND method to develop pain assessment indicators. We measured performance on the indicators using retrospectively collected pain measurement data from Dutch ICUs, which are all mixed medical - surgical, of three months within October 2016-May 2017. We assessed the room for improvement, feasibility of data collection, and reliability of the indicators. RESULTS: We defined four pain assessment indicators. We analyzed 45,688 patient-shift observations from 15 ICUs. In 69.2% (IQR 58.7-84.9) of the patient-shifts pain was measured at least once (indicator 1); in 56.7% (IQR 49.6-73.5) pain scores were acceptable (indicator 2); in 11.7% (IQR 5.6-26.4) pain measurements with unacceptable scores were repeated within 1 h (indicator 3); and in 10.9% (IQR 5.1-20.1) unacceptable scores normalized within 1 h (indicator 4). We found data collection feasible because data were available for >79.3% of the admissions, and all indicators reliable as they produced consistent performance scores. CONCLUSIONS: There is substantial variation in pain assessment across Dutch ICUs, and ample room for improvement. With this study we took a first step towards quality assurance of pain assessment in Dutch ICUs.

SMASH! The Salford medication safety dashboard.

BACKGROUND: Patient safety is vital to well-functioning health systems. A key component is safe prescribing, particularly in primary care where most medications are prescribed. Previous research demonstrated that the number of patients exposed to potentially hazardous prescribing can be reduced by interrogating the electronic health record (EHR) database of general practices and providing feedback to general practitioners in a pharmacist-led intervention. We aimed to develop and roll out an online dashboard application that delivers this audit and feedback intervention in a continuous fashion. METHOD: Based on initial system requirements we designed the dashboard's user interface over 3 iterations with 6 general practitioners (GPs), 7 pharmacists and a member of the public. Prescribing safety indicators from previous work were implemented in the dashboard. Pharmacists were trained to use the intervention and deliver it to general practices. RESULTS: A web-based electronic dashboard was developed and linked to shared care records in Salford, UK. The completed dashboard was deployed in all but one (n=43) general practices in the region. By November 2017, 36 pharmacists had been trained in delivering the intervention to practices. There were 135 registered users of the dashboard, with an average of 91 user sessions a week. CONCLUSION: We have developed and successfully rolled out of a complex, pharmacist-led dashboard intervention in Salford, UK. System usage statistics indicate broad and sustained uptake of the intervention. The use of systems that provide regularly updated audit information may be an important contributor towards medication safety in primary care.

Development of actionable quality indicators and an action implementation toolbox for appropriate antibiotic use at intensive care units: A modified-RAND Delphi study.

INTRODUCTION: Extensive antibiotic use makes the intensive care unit (ICU) an important focus for antibiotic stewardship programs. The aim of this study was to develop a set of actionable quality indicators for appropriate antibiotic use at ICUs and an implementation toolbox, which can be used to assess and improve the appropriateness of antibiotic use in the treatment of adult patients at an ICU. METHODS: A four round modified-RAND Delphi procedure was used. Potential indicators were identified by a multidisciplinary panel of 15 Dutch experts, from international literature and guidelines. Using an online survey, the identified indicators were rated on three criteria: relevance, actionability and feasibility. Experts discussed and rated the indicators for the second time during a face-to-face consensus meeting. During a final consensus meeting the toolbox was developed, containing potential barriers and improvement strategies which were identified using a validated checklist by Flottorp et al., and if available also containing supporting material. RESULTS: The first round resulted in 24 potential indicators. After the final meeting a set of three process indicators, one structure indicator and one quantity metric remained: 1) perform at least two sets of blood cultures before start of empirical systemic therapy; 2) perform therapeutic drug monitoring in patients treated with vancomycin or aminoglycosides; 3) perform surveillance cultures if selective digestive or oropharyngeal decontamination is applied at the ICU; 4) biannual face-to-face meetings between ICU and microbiology staff in which local resistance rates are discussed; and 5) quantitative antibiotic use at the ICU expressed in days of therapy (DOT). The toolbox contains 24 unique barriers and 37 improvement strategies. CONCLUSIONS: Our study identified a set of four actionable quality indicators and one quantity metric, together with an implementation toolbox, to improve appropriate antibiotic use at ICUs.

Health professionals' perceptions about their clinical performance and the influence of audit and feedback on their intentions to improve practice: a theory-based study in Dutch intensive care units.

BACKGROUND: Audit and feedback aims to guide health professionals in improving aspects of their practice that need it most. Evidence suggests that feedback fails to increase accuracy of professional perceptions about clinical performance, which likely reduces audit and feedback effectiveness. This study investigates health professionals' perceptions about their clinical performance and the influence of feedback on their intentions to change practice. METHODS: We conducted an online laboratory experiment guided by Control Theory with 72 intensive care professionals from 21 units. For each of four new pain management indicators, we collected professionals' perceptions about their clinical performance; peer performance; targets; and improvement intentions before and after receiving first-time feedback. An electronic audit and feedback dashboard provided ICU's own performance, median and top 10% peer performance, and improvement recommendations. The experiment took place approximately 1 month before units enrolled into a cluster-randomised trial assessing the impact of adding a toolbox with suggested actions and materials to improve intensive care pain management. During the experiment, the toolbox was inaccessible; all participants accessed the same version of the dashboard. RESULTS: We analysed 288 observations. In 53.8%, intensive care professionals overestimated their clinical performance; but in only 13.5%, they underestimated it. On average, performance was overestimated by 22.9% (on a 0-100% scale). Professionals similarly overestimated peer performance, and set targets 20.3% higher than the top performance benchmarks. In 68.4% of cases, intentions to improve practice were consistent with actual gaps in performance, even before professionals had received feedback; which increased to 79.9% after receiving feedback (odds ratio, 2.41; 95% CI, 1.53 to 3.78). However, in 56.3% of cases, professionals still wanted to improve care aspects at which they were already top performers. Alternatively, in 8.3% of cases, they lacked improvement intentions because they did not consider indicators important; did not trust the data; or deemed benchmarks unrealistic. CONCLUSIONS: Audit and feedback helps health professionals to work on aspects for which improvement is recommended. Given the abundance of professionals' prior good improvement intentions, the limited effects typically found by audit and feedback studies are likely predominantly caused by barriers to translation of intentions into actual change in clinical practice. TRIAL REGISTRATION: ClinicalTrials.gov NCT02922101 . Registered 26 September 2016.

Informatics for Health 2017: Advancing both science and practice.

INTRODUCTION: The Informatics for Health congress, 24-26 April 2017, in Manchester, UK, brought together the Medical Informatics Europe (MIE) conference and the Farr Institute International Conference. This special issue of the Journal of Innovation in Health Informatics contains 113 presentation abstracts and 149 poster abstracts from the congress. DISCUSSION: The twin programmes of "Big Data" and "Digital Health" are not always joined up by coherent policy and investment priorities. Substantial global investment in health IT and data science has led to sound progress but highly variable outcomes. Society needs an approach that brings together the science and the practice of health informatics. The goal is multi-level Learning Health Systems that consume and intelligently act upon both patient data and organizational intervention outcomes. CONCLUSION: Informatics for Health demonstrated the art of the possible, seen in the breadth and depth of our contributions. We call upon policy makers, research funders and programme leaders to learn from this joined-up approach.

Electronic audit and feedback intervention with action implementation toolbox to improve pain management in intensive care: protocol for a laboratory experiment and cluster randomised trial.

BACKGROUND: Audit and feedback is often used as a strategy to improve quality of care, however, its effects are variable and often marginal. In order to learn how to design and deliver effective feedback, we need to understand their mechanisms of action. This theory-informed study will investigate how electronic audit and feedback affects improvement intentions (i.e. information-intention gap), and whether an action implementation toolbox with suggested actions and materials helps translating those intentions into action (i.e. intention-behaviour gap). The study will be executed in Dutch intensive care units (ICUs) and will be focused on pain management. METHODS AND DESIGN: We will conduct a laboratory experiment with individual ICU professionals to assess the impact of feedback on their intentions to improve practice. Next, we will conduct a cluster randomised controlled trial with ICUs allocated to feedback without or feedback with action implementation toolbox group. Participants will not be told explicitly what aspect of the intervention is randomised; they will only be aware that there are two variations of providing feedback. ICUs are eligible for participation if they submit indicator data to the Dutch National Intensive Care Evaluation (NICE) quality registry and agree to allocate a quality improvement team that spends 4 h per month on the intervention. All participating ICUs will receive access to an online quality dashboard that provides two functionalities: gaining insight into clinical performance on pain management indicators and developing action plans. ICUs with access to the toolbox can develop their action plans guided by a list of potential barriers in the care process, associated suggested actions, and supporting materials to facilitate implementation of the actions. The primary outcome measure for the laboratory experiment is the proportion of improvement intentions set by participants that are consistent with recommendations based on peer comparisons; for the randomised trial it is the proportion of patient shifts during which pain has been adequately managed. We will also conduct a process evaluation to understand how the intervention is implemented and used in clinical practice, and how implementation and use affect the intervention's impact. DISCUSSION: The results of this study will inform care providers and managers in ICU and other clinical settings how to use indicator-based performance feedback in conjunction with an action implementation toolbox to improve quality of care. Within the ICU context, this study will produce concrete and directly applicable knowledge with respect to what is or is not effective for improving pain management, and under which circumstances. The results will further guide future research that aims to understand the mechanisms behind audit and feedback and contribute to identifying the active ingredients of successful interventions. TRIAL REGISTRATION: ClinicalTrials.gov NCT02922101 . Registered 26 September 2016.

Development of a Web-Based Quality Dashboard Including a Toolbox to Improve Pain Management in Dutch Intensive Care.

Audit and feedback (A&F) is a common strategy to improve quality of care. Meta-analyses have indicated that A&F may be more effective in realizing desired change when baseline performance is low, it is delivered by a supervisor or colleague, it is provided frequently and in a timely manner, it is delivered in both verbal and written formats, and it includes specific targets and an action plan. However, there is little information to guide operationalization of these factors. Researchers have consequently called for A&F interventions featuring well-described and carefully justified components, with their theoretical rationale made explicit. This paper describes the rationale and development of a quality dashboard including an improvement toolbox for four previous developed pain indicators, guided by Control Theory.

How does audit and feedback influence intentions of health professionals to improve practice? A laboratory experiment and field study in cardiac rehabilitation.

OBJECTIVE: To identify factors that influence the intentions of health professionals to improve their practice when confronted with clinical performance feedback, which is an essential first step in the audit and feedback mechanism. METHODS: We conducted a theory-driven laboratory experiment with 41 individual professionals, and a field study in 18 centres in the context of a cluster-randomised trial of electronic audit and feedback in cardiac rehabilitation. Feedback reports were provided through a web-based application, and included performance scores and benchmark comparisons (high, intermediate or low performance) for a set of process and outcome indicators. From each report participants selected indicators for improvement into their action plan. Our unit of observation was an indicator presented in a feedback report (selected yes/no); we considered selecting an indicator to reflect an intention to improve. RESULTS: We analysed 767 observations in the laboratory experiment and 614 in the field study, respectively. Each 10% decrease in performance score increased the probability of an indicator being selected by 54% (OR, 1.54; 95% CI 1.29% to 1.83%) in the laboratory experiment, and 25% (OR, 1.25; 95% CI 1.13% to 1.39%) in the field study. Also, performance being benchmarked as low and intermediate increased this probability in laboratory settings. Still, participants ignored the benchmarks in 34% (laboratory experiment) and 48% (field study) of their selections. CONCLUSIONS: When confronted with clinical performance feedback, performance scores and benchmark comparisons influenced health professionals' intentions to improve practice. However, there was substantial variation in these intentions, because professionals disagreed with benchmarks, deemed improvement unfeasible or did not consider the indicator an essential aspect of care quality. These phenomena impede intentions to improve practice, and are thus likely to dilute the effects of audit and feedback interventions. TRIAL REGISTRATION NUMBER: NTR3251, pre-results.

Effect of a web-based audit and feedback intervention with outreach visits on the clinical performance of multidisciplinary teams: a cluster-randomized trial in cardiac rehabilitation.

BACKGROUND: The objective of this study was to assess the effect of a web-based audit and feedback (A&F) intervention with outreach visits to support decision-making by multidisciplinary teams. METHODS: We performed a multicentre cluster-randomized trial within the field of comprehensive cardiac rehabilitation (CR) in the Netherlands. Our participants were multidisciplinary teams in Dutch CR centres who were enrolled in the study between July 2012 and December 2013 and received the intervention for at least 1 year. The intervention included web-based A&F with feedback on clinical performance, facilities for goal setting and action planning, and educational outreach visits. Teams were randomized either to receive feedback that was limited to psychosocial rehabilitation (study group A) or to physical rehabilitation (study group B). The main outcome measure was the difference in performance between study groups in 11 care processes and six patient outcomes, measured at patient level. Secondary outcomes included effects on guideline concordance for the four main CR therapies. RESULTS: Data from 18 centres (14,847 patients) were analysed, of which 12 centres (9353 patients) were assigned to group A and six (5494 patients) to group B. During the intervention, a total of 233 quality improvement goals was identified by participating teams, of which 49 (21%) were achieved during the study period. Except for a modest improvement in data completeness (4.5% improvement per year; 95% CI 0.65 to 8.36), we found no effect of our intervention on any of our primary or secondary outcome measures. CONCLUSIONS: Within a multidisciplinary setting, our web-based A&F intervention engaged teams to define local performance improvement goals but failed to support them in actually completing the improvement actions that were needed to achieve those goals. Future research should focus on improving the actionability of feedback on clinical performance and on addressing the socio-technical perspective of the implementation process. TRIAL REGISTRATION: NTR3251.

Optimizing Digital Health Informatics Interventions Through Unobtrusive Quantitative Process Evaluations.

Health informatics interventions such as clinical decision support (CDS) and audit and feedback (A&F) are variably effective at improving care because the underlying mechanisms through which these interventions bring about change are poorly understood. This limits our possibilities to design better interventions. Process evaluations can be used to improve this understanding by assessing fidelity and quality of implementation, clarifying causal mechanisms, and identifying contextual factors associated with variation in outcomes. Coiera describes the intervention process as a series of stages extending from interactions to outcomes: the "information value chain". However, past process evaluations often did not assess the relationships between those stages. In this paper we argue that the chain can be measured quantitatively and unobtrusively in digital interventions thanks to the availability of electronic data that are a by-product of their use. This provides novel possibilities to study the mechanisms of informatics interventions in detail and inform essential design choices to optimize their efficacy.

Inside the Black Box of Audit and Feedback: a Laboratory Study to Explore Determinants of Improvement Target Selection by Healthcare Professionals in Cardiac Rehabilitation.

Audit and feedback (A&F) is widely used to aid healthcare professionals in improving clinical performance, but there is little understanding of the underlying mechanism that determines its effectiveness. The aim of this paper is to investigate the process by which healthcare professionals select indicators as improvement targets based on A&F. We performed a laboratory study among 41 healthcare professionals in the context of a web-based A&F intervention designed to improve the quality of cardiac rehabilitation care in the Netherlands. Feedback was provided on eighteen quality indicators, including a score and a colour (representing a recommendation for selection (red and yellow) or non-selection (green)). Indicators with more room for improvement were more likely to be selected, although this varied substantially between participants. In more than a quarter of the cases, participants did not select indicators with obvious room for improvement (yellow or red colour), or selected indicators without apparent room for improvement (green colour). We conclude that personal preferences and beliefs concerning quality and performance targets may dilute the efficiency of A&F.

Improving guideline concordance in multidisciplinary teams: preliminary results of a cluster-randomized trial evaluating the effect of a web-based audit and feedback intervention with outreach visits.

Despite their widespread use, audit and feedback (A&F) interventions show variable effectiveness on improving professional performance. Based on known facilitators of successful A&F interventions, we developed a web-based A&F intervention with indicator-based performance feedback, benchmark information, action planning and outreach visits. The goal of the intervention was to engage with multidisciplinary teams to overcome barriers to guideline concordance and to improve overall team performance in the field of cardiac rehabilitation (CR). To assess its effectiveness we conducted a cluster-randomized trial in 18 CR clinics (14,847 patients) already working with computerized decision support (CDS). Our preliminary results showed no increase in concordance with guideline recommendations regarding prescription of CR therapies. Future analyses will investigate whether our intervention did improve team performance on other quality indicators.