Deployment of a human-centred clinical decision support system for pulmonary embolism: evaluation of impact on quality of diagnostic decisions.

Pulmonary embolism (PE) is a serious condition that presents a diagnostic challenge for which diagnostic errors often happen. The literature suggests that a gap remains between PE diagnostic guidelines and adherence in healthcare practice. While system-level decision support tools exist, the clinical impact of a human-centred design (HCD) approach of PE diagnostic tool design is unknown. DESIGN: Before-after (with a preintervention period as non-concurrent control) design study. SETTING: Inpatient units at two tertiary care hospitals. PARTICIPANTS: General internal medicine physicians and their patients who underwent PE workups. INTERVENTION: After a 6-month preintervention period, a clinical decision support system (CDSS) for diagnosis of PE was deployed and evaluated over 6 months. A CDSS technical testing phase separated the two time periods. MEASUREMENTS: PE workups were identified in both the preintervention and CDSS intervention phases, and data were collected from medical charts. Physician reviewers assessed workup summaries (blinded to the study period) to determine adherence to evidence-based recommendations. Adherence to recommendations was quantified with a score ranging from 0 to 1.0 (the primary study outcome). Diagnostic tests ordered for PE workups were the secondary outcomes of interest. RESULTS: Overall adherence to diagnostic pathways was 0.63 in the CDSS intervention phase versus 0.60 in the preintervention phase (p=0.18), with fewer workups in the CDSS intervention phase having very low adherence scores. Further, adherence was significantly higher when PE workups included the Wells prediction rule (median adherence score=0.76 vs 0.59, p=0.002). This difference was even more pronounced when the analysis was limited to the CDSS intervention phase only (median adherence score=0.80 when Wells was used vs 0.60 when Wells was not used, p=0.001). For secondary outcomes, using both the D-dimer blood test (42.9% vs 55.7%, p=0.014) and CT pulmonary angiogram imaging (61.9% vs 75.4%, p=0.005) was lower during the CDSS intervention phase. CONCLUSION: A clinical decision support intervention with an HCD improves some aspects of the diagnostic decision, such as the selection of diagnostic tests and the use of the Wells probabilistic prediction rule for PE.

Computerised clinical decision support system for the diagnosis of pulmonary thromboembolism: a preclinical pilot study.

BACKGROUND: Recommendations for the diagnosis of pulmonary embolism are available for healthcare providers. Yet, real practice data show existing gaps in the translation of evidence-based recommendations. This is a study to assess the effect of a computerised decision support system (CDSS) with an enhanced design based on best practices in content and reasoning representation for the diagnosis of pulmonary embolism. DESIGN: Randomised preclinical pilot study of paper-based clinical scenarios in the diagnosis of pulmonary embolism. Participants were clinicians (n=30) from three levels of experience: medical students, residents and physicians. Participants were randomised to two interventions for the diagnosis of pulmonary embolism: a didactic lecture versus a decision tree via a CDSS. The primary outcome of diagnostic pathway concordance (derived as a ratio of the number of correct diagnostic decision steps divided by the ideal number of diagnostic decision steps in diagnostic algorithms) was measured at baseline (five clinical scenarios) and after either intervention for a total of 10 clinical scenarios. RESULTS: The mean of diagnostic pathway concordance improved in both study groups: baseline mean=0.73, post mean for the CDSS group=0.90 (p<0.001, 95% CI 0.10-0.24); baseline mean=0.71, post mean for didactic lecture group=0.85 (p<0.001, 95% CI 0.07-0.2). There was no statistically significant difference between the two study groups or between the three levels of participants. INTERPRETATION: A computerised decision support system designed for both content and reasoning visualisation can improve clinicians' diagnostic decision-making.

Developing EMR-based algorithms to Identify hospital adverse events for health system performance evaluation and improvement: Study protocol.

BACKGROUND: Measurement of care quality and safety mainly relies on abstracted administrative data. However, it is well studied that administrative data-based adverse event (AE) detection methods are suboptimal due to lack of clinical information. Electronic medical records (EMR) have been widely implemented and contain detailed and comprehensive information regarding all aspects of patient care, offering a valuable complement to administrative data. Harnessing the rich clinical data in EMRs offers a unique opportunity to improve detection, identify possible risk factors of AE and enhance surveillance. However, the methodological tools for detection of AEs within EMR need to be developed and validated. The objectives of this study are to develop EMR-based AE algorithms from hospital EMR data and assess AE algorithm's validity in Canadian EMR data. METHODS: Patient EMR structured and text data from acute care hospitals in Calgary, Alberta, Canada will be linked with discharge abstract data (DAD) between 2010 and 2020 (n~1.5 million). AE algorithms development. First, a comprehensive list of AEs will be generated through a systematic literature review and expert recommendations. Second, these AEs will be mapped to EMR free texts using Natural Language Processing (NLP) technologies. Finally, an expert panel will assess the clinical relevance of the developed NLP algorithms. AE algorithms validation: We will test the newly developed AE algorithms on 10,000 randomly selected EMRs between 2010 to 2020 from Calgary, Alberta. Trained reviewers will review the selected 10,000 EMR charts to identify AEs that had occurred during hospitalization. Performance indicators (e.g., sensitivity, specificity, positive predictive value, negative predictive value, F1 score, etc.) of the developed AE algorithms will be assessed using chart review data as the reference standard. DISCUSSION: The results of this project can be widely implemented in EMR based healthcare system to accurately and timely detect in-hospital AEs.

A randomised controlled trial assessing the efficacy of an electronic discharge communication tool for preventing death or hospital readmission.

OBJECTIVE: To assess the efficacy of an electronic discharge communication tool (e-DCT) for preventing death or hospital readmission, as well as reducing patient-reported adverse events after hospital discharge. The e-DCT assessed has already been shown to yield high-quality discharge summaries with high levels of patient and physician satisfaction. METHODS: This two-arm randomised controlled trial was conducted in a Canadian tertiary care centre's internal medicine medical teaching units. Out of the 1953 patients approached and screened for inclusion, 1399 were randomised and available for data linkage for determination of the primary outcome. Participants were randomly assigned to e-DCT versus usual care (traditional discharge communication generated by dictation). The primary outcome was a composite of death or readmission within 90 days. The secondary outcome included any patient-reported adverse events within 30 days of discharge. RESULTS: Among 1399 randomised participants, 230 of 701 participants (32.8%) in the e-DCT group experienced the primary composite outcome of death or readmission within 90 days vs 205 of 698 participants (29.4%) in the usual care group (p=0.166). The incidence at 30 days of patient-reported adverse outcomes (35% for e-DCT vs 34% for usual care) and adverse events (2.1% for e-DCT vs 1.8% for usual care) also did not differ significantly between groups. CONCLUSIONS: The e-DCT tested did not reduce the composite endpoint of death or readmission at 90 days, nor the incidence of patient-reported adverse events at 30 days. This neutral finding for hard clinical endpoints needs to be considered in the context of high patient and physician satisfaction, and high quality of discharge summaries.

Deriving ICD-10 Codes for Patient Safety Indicators for Large-scale Surveillance Using Administrative Hospital Data.

BACKGROUND: Existing administrative data patient safety indicators (PSIs) have been limited by uncertainty around the timing of onset of included diagnoses. OBJECTIVE: We undertook de novo PSI development through a data-driven approach that drew upon "diagnosis timing" information available in some countries' administrative hospital data. RESEARCH DESIGN: Administrative database analysis and modified Delphi rating process. SUBJECTS: All hospitalized adults in Canada in 2009. MEASURES: We queried all hospitalizations for ICD-10-CA diagnosis codes arising during hospital stay. We then undertook a modified Delphi panel process to rate the extent to which each of the identified diagnoses has a potential link to suboptimal quality of care. We grouped the identified quality/safety-related diagnoses into relevant clinical categories. Lastly, we queried Alberta hospital discharge data to assess the frequency of the newly defined PSI events. RESULTS: Among 2,416,413 national hospitalizations, we found 2590 unique ICD-10-CA codes flagged as having arisen after admission. Seven panelists evaluated these in a 2-round review process, and identified a listing of 640 ICD-10-CA diagnosis codes judged to be linked to suboptimal quality of care and thus appropriate for inclusion in PSIs. These were then grouped by patient safety experts into 18 clinically relevant PSI categories. We then analyzed data on 2,381,652 Alberta hospital discharges from 2005 through 2012, and found that 134,299 (5.2%) hospitalizations had at least 1 PSI diagnosis. CONCLUSION: The resulting work creates a foundation for a new set of PSIs for routine large-scale surveillance of hospital and health system performance.

A framework to assess patient-reported adverse outcomes arising during hospitalization.

BACKGROUND: The assessment of adverse events from a patient-centered view includes patient-reported adverse outcomes. An adverse outcome refers to any suboptimal outcome experienced by the patient; when adverse outcomes are identified through a patient interview these are called patient-reported adverse outcomes. An adverse event is an adverse outcome that is more likely due to the processes of medical care rather than to the mere progression of disease. In the context of a large-scale study assessing post-hospitalization adverse events, we developed a conceptual framework to assess patient-reported adverse outcomes (PRAOs). This methodological manuscript describes this conceptual framework. METHODS: The PRAO framework builds on a validated adverse event ascertainment method including three phases: Phase 1 involves an inquiry to ascertain the occurrence of any patient-reported adverse outcome. It is completed by a structured telephone interview to obtain details - from a patient perspective - on symptoms that developed and/or worsened after hospitalization. Phase 2 involves the classification of PRAOs by physicians not involved in the patient care. Physician-reviewers then rate the PRAOs using well-adopted scales to determine whether the occurrence was the natural progression of the underlying illness or due to medical care. When the PRAO is rated as "due to medical care", it is then classified as an "adverse event". Phase 3 involves the classification of adverse events as preventable or ameliorable. RESULTS: Out of the 1347 patients contacted at 1-month post-discharge, 469 reported AOs and after reviewing 369 cases, 29 were classified as AEs. Observed agreement levels between raters were 87.3, 85.5, and 85.2 % respectively displaying a good agreement (k > 0.60). CONCLUSION: The framework incorporates PRAOs as a way to identify cases that need to be evaluated for adverse events. Further validation of this framework is warrant with the final aim of implementation at larger scale. The implementation of this framework will enable clinicians, researchers and healthcare institutions to compare outcome rates across providers and over time.

Using patient flow simulation to improve access at a multidisciplinary sleep centre.

The lack of timely access to diagnosis and treatment for sleep disorders is well described, but little attention has been paid to understanding how multiple system constraints contribute to long waiting times. The objectives of this study were to identify system constraints leading to long waiting times at a multidisciplinary sleep centre, and to use patient flow simulation modelling to test solutions that could improve access. Discrete-event simulation models of patient flow were constructed using historical data from 150 patients referred to the sleep centre, and used to both examine reasons for access delays and to test alternative system configurations that were predicted by administrators to reduce waiting times. Four possible solutions were modelled and compared with baseline, including addition of capacity to different areas at the sleep centre and elimination of prioritization by urgency. Within the model, adding physician capacity improved time from patient referral to initial physician appointment, but worsened time from polysomnography requisition to test completion, and had no effect on time from patient referral to treatment initiation. Adding respiratory therapist did not improve model performance compared with baseline. Eliminating triage prioritization worsened time to physician assessment and treatment initiation for urgent patients without improving waiting times overall. This study demonstrates that discrete-event simulation can identify multiple constraints in access-limited healthcare systems and allow suggested solutions to be tested before implementation. The model of this sleep centre predicted that investments in capacity expansion proposed by administrators would not reduce the time to a clinically meaningful patient outcome.

The costs of developing, implementing, and operating a safety learning system in community practice.

BACKGROUND: The cost of implementing safety systems in primary care has not been examined. One type of safety system is a safety learning system (SLS). An SLS has 2 components: a reporting that monitors patient safety incidents and a learning component that facilitated the development and implementation of improvement strategies. It is important to understand the costs of an SLS to determine if the improvement program is financially sustainable. OBJECTIVE: To determine the costs of the development, implementation, and operation of the community-based SLS. METHODS: Nineteen participating family physician clinics in Calgary, Alberta, were included (15 urban and 4 rural) consisting of 47 physicians, 53 office staff, 18 nurses, and 6 clinic managers. Costs of the SLS were determined by the ingredient method using micro-costing. The costs were divided into 3 stages: development, implementation, and operational. Development costs were processes required to create and initiate the SLS. Implementation costs were accrued as a result of establishing, running, and refining the SLS. Finally, operational costs were those related to maintaining the SLS. Costs were further broken down into fixed, marginal, and in kind; this approach will allow policy and decision makers to apply the appropriate costs to their own settings. RESULTS: The total development, implementation, and operational costs for the SLS in Canadian dollars were $77,011, $19,941, and $166,727, respectively, with a total cost of $263,679 over approximately a 4-year period. During this time, 270 incident reports were submitted, and 54 improvement cycles were implemented. CONCLUSIONS: The results provide quantitative data, which could be useful to legislators, policy makers, and other private and public sector payers of patient safety programs in determining the overall sustainability of an SLS.

Validity of AHRQ patient safety indicators derived from ICD-10 hospital discharge abstract data (chart review study).

OBJECTIVE: To assess if the Agency for Healthcare Research and Quality  patient safety indictors (PSIs) could be used for case findings in the International Classification of Disease 10th revision (ICD-10) hospital discharge abstract data. DESIGN: We identified and randomly selected 490 patients with a foreign body left during a procedure (PSI 5-foreign body), selected infections (IV site) due to medical care (PSI 7-infection), postoperative pulmonary embolism (PE) or deep vein thrombosis (DVT; PSI 12-PE/DVT), postoperative sepsis (PSI 13-sepsis)and accidental puncture or laceration (PSI 15-laceration) among patients discharged from three adult acute care hospitals in Calgary, Canada in 2007 and 2008. Their charts were reviewed for determining the presence of PSIs and used as the reference standard, positive predictive value (PPV) statistics were calculated to determine the proportion of positives in the administrative data representing 'true positives'. RESULTS: The PPV for PSI 5-foreign body was 62.5% (95% CI 35.4% to 84.8%), PSI 7-infection was 79.1% (67.4% to 88.1%), PSI 12-PE/DVT was 89.5% (66.9% to 98.7%), PSI 13-sepsis was 12.5% (1.6% to 38.4%) and PSI 15-laceration was 86.4% (75.0% to 94.0%) after excluding those who presented to the hospital with the condition. CONCLUSIONS: Several PSIs had high PPV in the ICD administrative data and are thus powerful tools for true positive case finding. The tools could be used to identify potential cases from the large volume of admissions for verification through chart reviews. In contrast, their sensitivity has not been well characterised and users of PSIs should be cautious if using them for 'quality of care reporting' presenting the rate of PSIs because under-coded data would generate falsely low PSI rates.

The Seamless Transfer-of-Care Protocol: a randomized controlled trial assessing the efficacy of an electronic transfer-of-care communication tool.

BACKGROUND: The transition between acute care and community care represents a vulnerable period in health care delivery. The vulnerability of this period has been attributed to changes to patients' medication regimens during hospitalization, failure to reconcile discrepancies between admission and discharge and the burdening of patients/families to take over care responsibilities at discharge and to relay important information to the primary care physician. Electronic communication platforms can provide an immediate link between acute care and community care physicians (and other community providers), designed to ensure consistent information transfer. This study examines whether a transfer-of-care (TOC) communication tool is efficacious and cost-effective for reducing hospital readmission, adverse events and adverse drug events as well as reducing death. METHODS: A randomized controlled trial conducted on the Medical Teaching Unit of a Canadian tertiary care centre will evaluate the efficacy and cost-effectiveness of a TOC communication tool. Medical in-patients admitted to the unit will be considered for this study. Data will be collected upon admission, and a total of 1400 patients will be randomized. The control group's acute care stay will be summarized using a traditional dictated summary, while the intervention group will have a summary generated using the TOC communication tool. The primary outcome will be a composite, at 3 months, of death or readmission to any Alberta acute-care hospital. Secondary outcomes will be the occurrence of post-discharge adverse events and adverse drug events at 1 month post discharge. Patients with adverse outcomes will have their cases reviewed by two Royal College certified internists or College-certified family physicians, blinded to patients' group assignments, to determine the type, severity, preventability and ameliorability of all detected adverse outcomes. An accompanying economic evaluation will assess the cost per life saved, cost per readmission avoided and cost per QALY gained with the TOC communication tool compared to traditional dictation summaries. DISCUSSION: This paper outlines the study protocol for a randomized controlled trial evaluating an electronic transfer-of-care communication tool, with sufficient statistical power to assess the impact of the tool on the significant outcomes of post-discharge death or readmission. The study findings will inform health systems around the world on the potential benefits of such tools, and the value for money associated with their widespread implementation. TRIAL REGISTRATION: ClinicalTrials.gov NCT01402609.

Development of a novel electronic surveillance system for monitoring of bloodstream infections.

BACKGROUND: Electronic surveillance systems (ESSs) that utilize existing information in databases are more efficient than conventional infection surveillance methods. OBJECTIVE: To develop an ESS for monitoring bloodstream infections (BSIs) and assess whether data obtained from the ESS were in agreement with data obtained by traditional manual medical-record review. METHODS: An ESS was developed by linking data from regional laboratory and hospital administrative databases. Definitions for excluding BSI episodes representing contamination and duplicate episodes were developed and applied. Infections were classified as nosocomial infections, healthcare-associated community-onset infections, or community-acquired infections. For a random sample of episodes, data in the ESS were compared with data obtained by independent medical chart review. RESULTS: From the records of the 306 patients whose infections were selected for comparative review, the ESS identified 323 episodes of BSI, of which 107 (33%) were classified as healthcare-associated community-onset infections, 108 (33%) were classified as community-acquired infections, 107 (33%) were classified as nosocomial infections, and 1 (0.3%) could not be classified. In comparison, 310 episodes were identified by use of medical chart review, of which 116 (37%) were classified as healthcare-associated community-onset infections, 95 (31%) as community-acquired infections, and 99 (32%) as nosocomial infections. For 302 episodes of BSI, there was concordance between the findings of the ESS and those of traditional manual chart review. Of the additional 21 discordant episodes that were identified by use of the ESS, 17 (81%) were classified as representing isolation of skin contaminants, by use of chart review. Of the additional 8 discordant episodes further identified by use of chart review, most were classified as repeat or polymicrobial episodes of disease. There was an overall 85% agreement between the findings of the ESS and those of chart review (kappa=0.78; standard error, kappa=0.04) for classification according to location of acquisition. CONCLUSION: Our novel ESS allows episodes of BSI to be identified and classified with a high degree of accuracy. This system requires validation in other cohorts and settings.

Advancing measurement of patient safety culture.

OBJECTIVE: To examine the psychometric and unit of analysis/strength of culture issues in patient safety culture (PSC) measurement. DATA SOURCE: Two cross-sectional surveys of health care staff in 10 Canadian health care organizations totaling 11,586 respondents. STUDY DESIGN: A cross-validation study of a measure of PSC using survey data gathered using the Modified Stanford PSC survey (MSI-2005 and MSI-2006); a within-group agreement analysis of MSI-2006 data. Extraction Methods. Exploratory factor analyses (EFA) of the MSI-05 survey data and confirmatory factor analysis (CFA) of the MSI-06 survey data; Rwg coefficients of homogeneity were calculated for 37 units and six organizations in the MSI-06 data set to examine within-group agreement. PRINCIPAL FINDINGS: The CFA did not yield acceptable levels of fit. EFA and reliability analysis of MSI-06 data suggest two reliable dimensions of PSC: Organization leadership for safety (alpha=0.88) and Unit leadership for safety (alpha=0.81). Within-group agreement analysis shows stronger within-unit agreement than within-organization agreement on assessed PSC dimensions. CONCLUSIONS: The field of PSC measurement has not been able to meet strict requirements for sound measurement using conventional approaches of CFA. Additional work is needed to identify and soundly measure key dimensions of PSC. The field would also benefit from further attention to strength of culture/unit of analysis issues.

Do hospital standardized mortality ratios measure patient safety? HSMRs in the Winnipeg Regional Health Authority.

The Canadian Institute for Health Information began publishing hospital standardized mortality ratio (HSMR) data for select Canadian hospitals in November 2007. This paper describes the experience of the Winnipeg Regional Health Authority in assessing the validity of the HSMR through statistical analysis, coding definitions and chart audits. We found a lack of empirical evidence supporting the use of the HSMR in measuring reductions in preventable deaths. We also found that limitations in standardization as well as differences in palliative care coding and place of death make inter-facility comparisons of HSMRs invalid. The results of our chart audit show that the HSMR is not a sensitive measure of adverse events as defined by "unexpected death" in the Canadian Adverse Events Study. It should not be viewed as an important indicator of patient safety or quality of care. We discuss the cumulative sum statistic as an alternative to the HSMR in monitoring in-hospital mortality.