Grand rounds in methodology: key considerations for implementing machine learning solutions in quality improvement initiatives.

Machine learning (ML) solutions are increasingly entering healthcare. They are complex, sociotechnical systems that include data inputs, ML models, technical infrastructure and human interactions. They have promise for improving care across a wide range of clinical applications but if poorly implemented, they may disrupt clinical workflows, exacerbate inequities in care and harm patients. Many aspects of ML solutions are similar to other digital technologies, which have well-established approaches to implementation. However, ML applications present distinct implementation challenges, given that their predictions are often complex and difficult to understand, they can be influenced by biases in the data sets used to develop them, and their impacts on human behaviour are poorly understood. This manuscript summarises the current state of knowledge about implementing ML solutions in clinical care and offers practical guidance for implementation. We propose three overarching questions for potential users to consider when deploying ML solutions in clinical care: (1) Is a clinical or operational problem likely to be addressed by an ML solution? (2) How can an ML solution be evaluated to determine its readiness for deployment? (3) How can an ML solution be deployed and maintained optimally? The Quality Improvement community has an essential role to play in ensuring that ML solutions are translated into clinical practice safely, effectively, and ethically.

Temporal Clustering of Critical Illness Events on Medical Wards.

Importance: Recognizing and preventing patient deterioration is important for hospital safety. Objective: To investigate whether critical illness events (in-hospital death or intensive care unit [ICU] transfer) are associated with greater risk of subsequent critical illness events for other patients on the same medical ward. Design, Setting, and Participants: Retrospective cohort study in 5 hospitals in Toronto, Canada, including 118 529 hospitalizations. Patients were admitted to general internal medicine wards between April 1, 2010, and October 31, 2017. Data were analyzed between January 1, 2020, and April 10, 2023. Exposures: Critical illness events (in-hospital death or ICU transfer). Main Outcomes and Measures: The primary outcome was the composite of in-hospital death or ICU transfer. The association between critical illness events on the same ward across 6-hour intervals was studied using discrete-time survival analysis, adjusting for patient and situational factors. The association between critical illness events on different comparable wards in the same hospital was measured as a negative control. Results: The cohort included 118 529 hospitalizations (median age, 72 years [IQR, 56-83 years]; 50.7% male). Death or ICU transfer occurred in 8785 hospitalizations (7.4%). Patients were more likely to experience the primary outcome after exposure to 1 prior event (adjusted odds ratio [AOR], 1.39; 95% CI, 1.30-1.48) and more than 1 prior event (AOR, 1.49; 95% CI, 1.33-1.68) in the prior 6-hour interval compared with no exposure. The exposure was associated with increased odds of subsequent ICU transfer (1 event: AOR, 1.67; 95% CI, 1.54-1.81; >1 event: AOR, 2.05; 95% CI, 1.79-2.36) but not death alone (1 event: AOR, 1.08; 95% CI, 0.97-1.19; >1 event: AOR, 0.88; 95% CI, 0.71-1.09). There was no significant association between critical illness events on different wards within the same hospital. Conclusions and Relevance: Findings of this cohort study suggest that patients are more likely to be transferred to the ICU in the hours after another patient's critical illness event on the same ward. This phenomenon could have several explanations, including increased recognition of critical illness and preemptive ICU transfers, resource diversion to the first event, or fluctuations in ward or ICU capacity. Patient safety may be improved by better understanding the clustering of ICU transfers on medical wards.

Quality improvement strategies for diabetes care: Effects on outcomes for adults living with diabetes.

BACKGROUND: There is a large body of evidence evaluating quality improvement (QI) programmes to improve care for adults living with diabetes. These programmes are often comprised of multiple QI strategies, which may be implemented in various combinations. Decision-makers planning to implement or evaluate a new QI programme, or both, need reliable evidence on the relative effectiveness of different QI strategies (individually and in combination) for different patient populations. OBJECTIVES: To update existing systematic reviews of diabetes QI programmes and apply novel meta-analytical techniques to estimate the effectiveness of QI strategies (individually and in combination) on diabetes quality of care. SEARCH METHODS: We searched databases (CENTRAL, MEDLINE, Embase and CINAHL) and trials registers (ClinicalTrials.gov and WHO ICTRP) to 4 June 2019. We conducted a top-up search to 23 September 2021; we screened these search results and 42 studies meeting our eligibility criteria are available in the awaiting classification section. SELECTION CRITERIA: We included randomised trials that assessed a QI programme to improve care in outpatient settings for people living with diabetes. QI programmes needed to evaluate at least one system- or provider-targeted QI strategy alone or in combination with a patient-targeted strategy. - System-targeted: case management (CM); team changes (TC); electronic patient registry (EPR); facilitated relay of clinical information (FR); continuous quality improvement (CQI). - Provider-targeted: audit and feedback (AF); clinician education (CE); clinician reminders (CR); financial incentives (FI). - Patient-targeted: patient education (PE); promotion of self-management (PSM); patient reminders (PR). Patient-targeted QI strategies needed to occur with a minimum of one provider or system-targeted strategy. DATA COLLECTION AND ANALYSIS: We dual-screened search results and abstracted data on study design, study population and QI strategies. We assessed the impact of the programmes on 13 measures of diabetes care, including: glycaemic control (e.g. mean glycated haemoglobin (HbA1c)); cardiovascular risk factor management (e.g. mean systolic blood pressure (SBP), low-density lipoprotein cholesterol (LDL-C), proportion of people living with diabetes that quit smoking or receiving cardiovascular medications); and screening/prevention of microvascular complications (e.g. proportion of patients receiving retinopathy or foot screening); and harms (e.g. proportion of patients experiencing adverse hypoglycaemia or hyperglycaemia). We modelled the association of each QI strategy with outcomes using a series of hierarchical multivariable meta-regression models in a Bayesian framework. The previous version of this review identified that different strategies were more or less effective depending on baseline levels of outcomes. To explore this further, we extended the main additive model for continuous outcomes (HbA1c, SBP and LDL-C) to include an interaction term between each strategy and average baseline risk for each study (baseline thresholds were based on a data-driven approach; we used the median of all baseline values reported in the trials). Based on model diagnostics, the baseline interaction models for HbA1c, SBP and LDL-C performed better than the main model and are therefore presented as the primary analyses for these outcomes. Based on the model results, we qualitatively ordered each QI strategy within three tiers (Top, Middle, Bottom) based on its magnitude of effect relative to the other QI strategies, where 'Top' indicates that the QI strategy was likely one of the most effective strategies for that specific outcome. Secondary analyses explored the sensitivity of results to choices in model specification and priors.  Additional information about the methods and results of the review are available as Appendices in an online repository. This review will be maintained as a living systematic review; we will update our syntheses as more data become available. MAIN RESULTS: We identified 553 trials (428 patient-randomised and 125 cluster-randomised trials), including a total of 412,161 participants. Of the included studies, 66% involved people living with type 2 diabetes only. Participants were 50% female and the median age of participants was 58.4 years. The mean duration of follow-up was 12.5 months. HbA1c was the commonest reported outcome; screening outcomes and outcomes related to cardiovascular medications, smoking and harms were reported infrequently. The most frequently evaluated QI strategies across all study arms were PE, PSM and CM, while the least frequently evaluated QI strategies included AF, FI and CQI. Our confidence in the evidence is limited due to a lack of information on how studies were conducted.  Four QI strategies (CM, TC, PE, PSM) were consistently identified as 'Top' across the majority of outcomes. All QI strategies were ranked as 'Top' for at least one key outcome. The majority of effects of individual QI strategies were modest, but when used in combination could result in meaningful population-level improvements across the majority of outcomes. The median number of QI strategies in multicomponent QI programmes was three.  Combinations of the three most effective QI strategies were estimated to lead to the below effects:  - PR + PSM + CE: decrease in HbA1c by 0.41% (credibility interval (CrI) -0.61 to -0.22) when baseline HbA1c < 8.3%; - CM + PE + EPR: decrease in HbA1c by 0.62% (CrI -0.84 to -0.39) when baseline HbA1c > 8.3%;  - PE + TC + PSM: reduction in SBP by 2.14 mmHg (CrI -3.80 to -0.52) when baseline SBP < 136 mmHg; - CM + TC + PSM: reduction in SBP by 4.39 mmHg (CrI -6.20 to -2.56) when baseline SBP > 136 mmHg;  - TC + PE + CM: LDL-C lowering of 5.73 mg/dL (CrI -7.93 to -3.61) when baseline LDL < 107 mg/dL; - TC + CM + CR: LDL-C lowering by 5.52 mg/dL (CrI -9.24 to -1.89) when baseline LDL > 107 mg/dL. Assuming a baseline screening rate of 50%, the three most effective QI strategies were estimated to lead to an absolute improvement of 33% in retinopathy screening (PE + PR + TC) and 38% absolute increase in foot screening (PE + TC + Other). AUTHORS' CONCLUSIONS: There is a significant body of evidence about QI programmes to improve the management of diabetes. Multicomponent QI programmes for diabetes care (comprised of effective QI strategies) may achieve meaningful population-level improvements across the majority of outcomes. For health system decision-makers, the evidence summarised in this review can be used to identify strategies to include in QI programmes. For researchers, this synthesis identifies higher-priority QI strategies to examine in further research regarding how to optimise their evaluation and effects. We will maintain this as a living systematic review.

Potentially avoidable admissions to general internal medicine at an academic teaching hospital: an observational study.

BACKGROUND: Identifying potentially avoidable admissions to Canadian hospitals is an important health system goal. With general internal medicine (GIM) accounting for 40% of hospital admissions, we sought to develop a method to identify potentially avoidable admissions and characterize patient, provider and health system factors. METHODS: We conducted an observational study of GIM admissions at our institution from August 2019 to February 2020. We defined potentially avoidable admissions as admissions that could be managed in an appropriate and safe manner in the emergency department or ambulatory setting and asked staff physicians to screen admissions daily and flag candidates as potentially avoidable admissions. For each candidate, we prepared a case review and debriefed with members of the admitting team. We then reviewed each candidate with our research team, assigned an avoidability score (1 [low] to 4 [high]) and identified contributing factors for those with scores of 3 or more. RESULTS: We screened 601 total admissions and staff physicians flagged 117 (19.5%) of these as candidate potential avoidable admissions. Consensus review identified 67 candidates as potentially avoidable admissions (11.1%, 95% confidence interval 8.8%-13.9%); these patients were younger (mean age 65 yr v. 72 yr), had fewer comorbidities (Canadian Institute for Health Information Case Mix Group+ 0.42 v. 1.14), had lower resource-intensity weighting scores (0.72 v. 1.50) and shorter hospital lengths of stay (29 h v. 105 h) (p < 0.01). Common factors included diagnostic and therapeutic uncertainty, perceived need for short-term monitoring, government directive of a 4-hour limit for admission decision-making and subspecialist request to admit. INTERPRETATION: Our prospective method of screening, flagging and case review showed that 1 in 9 GIM admissions were potentially avoidable. Other institutions could consider adapting this methodology to ascertain their rate of potentially avoidable admissions and to understand contributing factors to inform improvement endeavours.

Striving for high reliability in healthcare: a qualitative study of the implementation of a hospital safety programme.

BACKGROUND: Healthcare leaders look to high-reliability organisations (HROs) for strategies to improve safety, despite questions about how to translate these strategies into practice. Weick and Sutcliffe describe five principles exhibited by HROs. Interventions aiming to foster these principles are common in healthcare; however, there have been few examinations of the perceptions of those who have planned or experienced these efforts. OBJECTIVE: This single-site qualitative study explores how healthcare professionals understand and enact the HRO principles in response to an HRO-inspired hospital-wide safety programme. METHODS: We interviewed 71 participants representing hospital executives, programme leadership, and staff and physicians from three clinical services. We observed and collected data from unit and hospital-wide quality and safety meetings and activities. We used thematic analysis to code and analyse the data. RESULTS: Participants reported enactment of the HRO principles 'preoccupation with failure', 'reluctance to simplify interpretations' and 'sensitivity to operations', and described the programme as adding legitimacy, training, and support. However, the programme was more often targeted at, and taken up by, nurses compared with other groups. Participants were less able to identify interventions that supported the HRO principles 'commitment to resilience' and 'deference to expertise' and reported limited examples of changes in practices related to these principles. Moreover, we identified inconsistent, and even conflicting, understanding of concepts related to the HRO principles, often related to social and professional norms and practices. Finally, an individualised rather than systemic approach hindered collective actions underlying high reliability. CONCLUSION: Our findings demonstrate that the safety programme supported some HRO principles more than others, and was targeted at, and perceived differently across professional groups leading to inconsistent understanding and enactments of the principles across the organisation. Combining HRO-inspired interventions with more targeted attention to each of the HRO principles could produce greater, more consistent high-reliability practices.

A Novel Collaborative Care Program to Augment Nursing Home Care During and After the COVID-19 Pandemic.

The 2019 novel coronavirus (COVID-19) pandemic created an immediate need to enhance current efforts to reduce transfers of nursing home (NH) residents to acute care. Long-Term Care Plus (LTC+), a collaborative care program developed and implemented during the COVID-19 pandemic, aimed to enhance care in the NH setting while also decreasing unnecessary acute care transfers. Using a hub-and-spoke model, LTC+ was implemented in 6 hospitals serving as central hubs to 54 geographically associated NHs with 9574 beds in Toronto, Canada. LTC+ provided NHs with the following: (1) virtual general internal medicine (GIM) consultations; (2) nursing navigator support; (3) rapid access to laboratory and diagnostic imaging services; and (4) educational resources. From April 2020 to June 2021, LTC+ provided 381 GIM consultations that addressed abnormal bloodwork (15%), cardiac problems (13%), and unexplained fever (11%) as the most common reasons for consultation. Sixty-five nurse navigator calls addressed requests for non-GIM specialist consultations (34%), wound care assessments (14%), and system navigation (12%). One hundred seventy-seven (46%, 95% CI 41%-52%) consults addressed care concerns sufficiently to avoid the need for acute care transfer. All 36 primary care physicians who consulted the LTC+ program reported strong satisfaction with the advice provided. Early results demonstrate the feasibility and acceptability of an integrated care model that enhances care delivery for NH residents where they reside and has the potential to positively impact the long-term care sector by ensuring equitable and timely access to care for people living in NHs. It represents an important step toward health system integration that values the expertise within the long-term care sector.

Association of Cardiology Billing Amounts With Health Care Utilization and Clinical Outcomes in Patients With Atrial Fibrillation.

Background The relationship between health care utilization and outcomes in patients with atrial fibrillation is unknown. The objective of this study was to investigate whether cardiologists' billing amounts in a fee-for-service environment are associated with better patient-level clinical outcomes. Methods and Results A retrospective cohort study was conducted using administrative claims data of cardiologists in Ontario, Canada between April 1, 2011 and March 31, 2016. The cardiologists were stratified into quintiles based on their median billing patterns per patient over the observation period. The primary outcomes were patient-level receipt of repeat visits, cardiac diagnostic tests, and medications ≤1 year of index date. The secondary clinical outcomes were death, emergency department visits, and all-cause hospitalization 1-year post-index visit. The patient cohort comprised 182 572 patients with atrial fibrillation (median age 74 years, 58% male) from 467 cardiologists. Patients with atrial fibrillation seen by higher-billing cardiologists were 26% more likely to have an echocardiogram (adjusted odds ratio [aOR], 1.26 [95% CI, 1.10-1.43] for quintile 5 versus 2), 28% a stress test (aOR, 1.28 [1.12-1.46] for quintile 5 versus 2), 25% continuous electrocardiographic monitoring (aOR, 1.25 [1.08-1.46] for quintile 4 versus 2), and 79% more likely to get a stress echocardiogram (aOR, 1.79 [1.32-2.42] for quintile 5 versus 2). They also had a higher rate of all-cause hospitalization (aOR, 1.13 [1.07-1.20]). Mortality rates were similar across cardiologists billing quintiles (eg, aOR, 0.98 [0.87-1.11] for quintile 4 versus 2). Conclusions Higher-billing cardiologists ordered more diagnostic tests per patient with atrial fibrillation but these are not associated with improvements in outcomes.

Implementation of a central-line bundle: a qualitative study of three clinical units.

BACKGROUND: Evidence for the central line-associated bloodstream infection (CLABSI) bundle effectiveness remains mixed, possibly reflecting implementation challenges and persistent ambiguities in how CLABSIs are counted and bundle adherence measured. In the context of a tertiary pediatric hospital that had reduced CLABSI by 30% as part of an international safety program, we aimed to examine unit-based socio-cultural factors influencing bundle practices and measurement, and how they come to be recognized and attended to by safety leaders over time in an organization-wide bundle implementation effort. METHODS: We used an interpretivist qualitative research approach, based on 74 interviews, approximately 50 h of observations, and documents. Data collection focused on hospital executives and safety leadership, and three clinical units: a medical specialty unit, an intensive care unit, and a surgical unit. We used thematic analysis and constant comparison methods for data analysis. RESULTS: Participants had variable beliefs about the central-line bundle as a quality improvement priority based on their professional roles and experiences and unit setting, which influenced their responses. Nursing leaders were particularly concerned about CLABSI being one of an overwhelming number of QI targets for which they were responsible. Bundle implementation strategies were initially reliant on unit-based nurse education. Over time there was recognition of the need for centralized education and reinforcement tactics. However, these interventions achieved limited impact given the influence of competing unit workflow demands and professional roles, interactions, and routines, which were variably targeted in the safety program. The auditing process, initially a responsibility of units, was performed in different ways based on individuals' approaches to the process. Given concerns about auditing reliability, a centralized approach was implemented, which continued to have its own variability. CONCLUSIONS: Our findings report on a contextualized, dynamic implementation approach that required movement between centralized and unit-based approaches and from a focus on standardization to some recognition of a role for customization. However, some factors related to bundle compliance and measurement remain unaddressed, including harder to change socio-cultural factors likely important to sustainability of the CLABSI reductions and fostering further improvements across a broader safety agenda.

Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials.

OBJECTIVE: To report the improvements achieved with clinical decision support systems and examine the heterogeneity from pooling effects across diverse clinical settings and intervention targets. DESIGN: Systematic review and meta-analysis. DATA SOURCES: Medline up to August 2019. ELIGIBILITY CRITERIA FOR SELECTING STUDIES AND METHODS: Randomised or quasi-randomised controlled trials reporting absolute improvements in the percentage of patients receiving care recommended by clinical decision support systems. Multilevel meta-analysis accounted for within study clustering. Meta-regression was used to assess the degree to which the features of clinical decision support systems and study characteristics reduced heterogeneity in effect sizes. Where reported, clinical endpoints were also captured. RESULTS: In 108 studies (94 randomised, 14 quasi-randomised), reporting 122 trials that provided analysable data from 1 203 053 patients and 10 790 providers, clinical decision support systems increased the proportion of patients receiving desired care by 5.8% (95% confidence interval 4.0% to 7.6%). This pooled effect exhibited substantial heterogeneity (I2=76%), with the top quartile of reported improvements ranging from 10% to 62%. In 30 trials reporting clinical endpoints, clinical decision support systems increased the proportion of patients achieving guideline based targets (eg, blood pressure or lipid control) by a median of 0.3% (interquartile range -0.7% to 1.9%). Two study characteristics (low baseline adherence and paediatric settings) were associated with significantly larger effects. Inclusion of these covariates in the multivariable meta-regression, however, did not reduce heterogeneity. CONCLUSIONS: Most interventions with clinical decision support systems appear to achieve small to moderate improvements in targeted processes of care, a finding confirmed by the small changes in clinical endpoints found in studies that reported them. A minority of studies achieved substantial increases in the delivery of recommended care, but predictors of these more meaningful improvements remain undefined.

Modelling resource requirements and physician staffing to provide virtual urgent medical care for residents of long-term care homes: a cross-sectional study.

BACKGROUND: The coronavirus disease 2019 (COVID-19) outbreak increases the importance of strategies to enhance urgent medical care delivery in long-term care (LTC) facilities that could potentially reduce transfers to emergency departments. The study objective was to model resource requirements to deliver virtual urgent medical care in LTC facilities. METHODS: We used data from all general medicine inpatient admissions at 7 hospitals in the Greater Toronto Area, Ontario, Canada, over a 7.5-year period (Apr. 1, 2010, to Oct. 31, 2017) to estimate historical patterns of hospital resource use by LTC residents. We estimated an upper bound of potentially avoidable transfers by combining data on short admissions (≤ 72 h) with historical data on the proportion of transfers from LTC facilities for which patients were discharged from the emergency department without admission. Regression models were used to extrapolate future resource requirements, and queuing models were used to estimate physician staffing requirements to perform virtual assessments. RESULTS: There were 235 375 admissions to general medicine wards, and residents of LTC facilities (age 16 yr or older) accounted for 9.3% (n = 21 948) of these admissions. Among the admissions of residents of LTC facilities, short admissions constituted 24.1% (n = 5297), and for 99.8% (n = 5284) of these admissions, the patient received laboratory testing, for 86.9% (n = 4604) the patient received plain radiography, for 41.5% (n = 2197) the patient received computed tomography and for 81.2% (n = 4300) the patient received intravenous medications. If all patients who have short admissions and are transferred from the emergency department were diverted to outpatient care, the average weekly demand for outpatient imaging per hospital would be 2.6 ultrasounds, 11.9 computed tomographic scans and 23.9 radiographs per week. The average daily volume of urgent medical virtual assessments would range from 2.0 to 5.8 per hospital. A single centralized virtual assessment centre staffed by 2 or 3 physicians would provide services similar in efficiency (measured by waiting time for physician assessment) to 7 separate centres staffed by 1 physician each. INTERPRETATION: The provision of acute medical care to LTC residents at their facility would probably require rapid access to outpatient diagnostic imaging, within-facility access to laboratory services and intravenous medication and virtual consultations with physicians. The results of this study can inform efforts to deliver urgent medical care in LTC facilities in light of a potential surge in COVID-19 cases.