Does Calculated Prognostic Estimation Lead to Different Outcomes Compared With Experience-Based Prognostication in the ICU? A Systematic Review.

Little is known about the impact of providing calculator/guideline based versus clinical experiential-based prognostic estimates to patients/caregivers in the ICU. We sought to determine whether studies have compared types of prognostic estimation in the ICU and associations with outcomes. DATA SOURCES: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, databases searched were PubMed, Embase, Web of Science, and Cochrane Library. The search was run on January 4, 2016, and April 12, 2017. References for included articles were searched. STUDY SELECTION: Studies meeting the following criteria were included in the analysis: communication of prognostic estimates, a comparator group, and in the adult ICU setting. DATA EXTRACTION: Titles/abstracts were reviewed by two researchers. We identified 10,704 articles of which 10 met inclusion criteria. Seven of the studies included estimates obtained from calculators/guidelines and three were based on subjective estimation wherein clinicians were asked to estimate prognosis based on experience. Only the seven using calculated/guideline based estimation were used for pooled analysis. Of these, one was a randomized trial, and six were nonrandomized before/after studies. All of the studies communicated the calculated/guideline-based estimates to the clinician. Two studies involved the communication of calculated prognostic estimates to the ICU physicians for all ICU patients. Four included identification of high-risk patients based on guidelines or review of historical local data which triggered a palliative care/ethics consultation, and one study included communication to physicians about guideline based likely outcomes for neurologic recovery for patients with out-of-hospital cardiac arrest survivors. The comparator arm in all studies was usual care without protocolized prognostication. DATA SYNTHESIS: Included studies were assessed for risk of bias. The most common outcomes measured were hospital mortality; do-not-resuscitate status; and medical ICU length of stay. In pooled analyses, there was an association between calculated/guideline based prognostic estimation and decreased medical ICU length of stay as well as increased do-not-resuscitate status, but no difference in hospital mortality. CONCLUSIONS: Protocolized assessment of calculator/guideline based prognosis in ICU patients is associated with decreased medical ICU length of stay and increased do-not-resuscitate status but does not have a significant effect on mortality. Future studies should explore how communicating these estimates to physicians changes behaviors including communication to patients/families and whether calculator/guideline based prognostication is associated with improved patient and family rated outcomes.

Avoiding alert fatigue in pulmonary embolism decision support: a new method to examine 'trigger rates'.

A clinical decision support system (CDSS) is integrated into the electronic health record (EHR) and allows physicians to easily use a clinical decision support (CDS) tool. However, often CDSSs are integrated into the EHR with poor adoption rates. One reason for this is secondary to 'trigger fatigue'. Therefore, we developed a new and innovative usability process named 'sensitivity and specificity trigger analysis' (SSTA) as part of our larger project around a pulmonary embolism decision support tool. SSTA will enable programmers to examine optimal trigger rates prior to the integration of a CDS tool into the EHR, by using a formal method of analysis. We performed a retrospective chart review. The outcome of interest was physician ordering of a CT angiography (CTA). Phrases that signify common symptoms associated with pulmonary embolism were assessed as possible triggers for the CDSS tool. We then analysed each trigger's ability to predict physician ordering of a CTA. We found that the most sensitive way to trigger the Pulmonary Embolism CDS tool while still maintaining a high specificity was by combining 1 or more pertinent symptoms with 1 or more elements of the Wells criteria. This study explored a unique methodology, SSTA, used to limit inaccurate triggering of a CDS tool prior to integration into the EHR. This methodology can be applied to other studies aiming to decrease triggering rates and increase adoption rates of previously validated CDSS tools.

Usability Testing of a National Substance Use Screening Tool Embedded in Electronic Health Records.

BACKGROUND: Screening, brief intervention, and referral to treatment (SBIRT) is currently being implemented into health systems nationally via paper and electronic methods. OBJECTIVE: The purpose of this study was to evaluate the integration of an electronic SBIRT tool into an existing paper-based SBIRT clinical workflow in a patient-centered medical home. METHODS: Usability testing was conducted in an academic ambulatory clinic. Two rounds of usability testing were done with medical office assistants (MOAs) using a paper and electronic version of the SBIRT tool, with two and four participants, respectively. Qualitative and quantitative data was analyzed to determine the impact of both tools on clinical workflow. A second round of usability testing was done with the revised electronic version and compared with the first version. RESULTS: Personal workflow barriers cited in the first round of testing were that the electronic health record (EHR) tool was disruptive to patient's visits. In Round 2 of testing, MOAs reported favoring the electronic version due to improved layout and the inclusion of an alert system embedded in the EHR. For example, using the system usability scale (SUS), MOAs reported a grade "1" for the statement, "I would like to use this system frequently" during the first round of testing but a "5" during the second round of analysis. CONCLUSIONS: The importance of testing usability of various mediums of tools used in health care screening is highlighted by the findings of this study. In the first round of testing, the electronic tool was reported as less user friendly, being difficult to navigate, and time consuming. Many issues faced in the first generation of the tool were improved in the second generation after usability was evaluated. This study demonstrates how usability testing of an electronic SBRIT tool can help to identify challenges that can impact clinical workflow. However, a limitation of this study was the small sample size of MOAs that participated. The results may have been biased to Northwell Health workers' perceptions of the SBIRT tool and their specific clinical workflow.

External validation of the IMPROVE Bleeding Risk Assessment Model in medical patients.

The IMPROVE Bleed Risk Assessment Model (RAM) remains the only bleed RAM in hospitalised medical patients using 11 clinical and laboratory factors. The aim of our study was to externally validate the IMPROVE Bleed RAM. A retrospective chart review was conducted between October 1, 2012 and July 31, 2014. We applied the point scoring system to compute risk scores for each patient in the validation sample. We then dichotomised the patients into those with a score <7 (low risk) vs ≥ 7 (high risk), as outlined in the original study, and compared the rates of any bleed, non-major bleed, and major bleed. Among the 12,082 subjects, there was an overall 2.6 % rate of any bleed within 14 days of admission. There was a 2.12 % rate of any bleed in those patients with a score of < 7 and a 4.68 % rate in those with a score ≥ 7 [Odds Ratio (OR) 2.3 (95 % CI=1.8-2.9), p<0.0001]. MB rates were 1.5 % in the patients with a score of < 7 and 3.2 % in the patients with a score of ≥ 7, [OR 2.2 (95 % CI=1.6-2.9), p<0.0001]. The ROC curve was 0.63 for the validation sample. This study represents the largest externally validated Bleed RAM in a hospitalised medically ill patient population. A cut-off point score of 7 or above was able to identify a high-risk patient group for MB and any bleed. The IMPROVE Bleed RAM has the potential to allow for more tailored approaches to thromboprophylaxis in medically ill hospitalised patients.

Developing a Clinical Prediction Rule for First Hospital-Onset Clostridium difficile Infections: A Retrospective Observational Study.

BACKGROUND The healthcare burden of hospital-acquired Clostridium difficile infection (CDI) demands attention and calls for a solution. Identifying patients' risk of developing a primary nosocomial CDI is a critical first step in reducing the development of new cases of CDI. OBJECTIVE To derive a clinical prediction rule that can predict a patient's risk of acquiring a primary CDI. DESIGN Retrospective cohort study. SETTING Large tertiary healthcare center. PATIENTS Total of 61,482 subjects aged at least 18 admitted over a 1-year period (2013). INTERVENTION None. METHODS Patient demographic characteristics, evidence of CDI, and other risk factors were retrospectively collected. To derive the CDI clinical prediction rule the patient population was divided into a derivation and validation cohort. A multivariable analysis was performed in the derivation cohort to identify risk factors individually associated with nosocomial CDI and was validated on the validation sample. RESULTS Among 61,482 subjects, CDI occurred in 0.46%. CDI outcome was significantly associated with age, admission in the past 60 days, mechanical ventilation, dialysis, history of congestive heart failure, and use of antibiotic medications. The sensitivity and specificity of the score, in the validation set, were 82.0% and 75.7%, respectively. The area under the receiver operating characteristic curve was 0.85. CONCLUSION This study successfully derived a clinical prediction rule that will help identify patients at high risk for primary CDI. This tool will allow physicians to systematically recognize those at risk for CDI and will allow for early interventional strategies. Infect Control Hosp Epidemiol 2016;37:896-900.

Formative assessment and design of a complex clinical decision support tool for pulmonary embolism.

Electronic health record (EHR)-based clinical decision support (CDS) tools are rolled out with the urgency to meet federal requirements without time for usability testing and refinement of the user interface. As part of a larger project to design, develop and integrate a pulmonary embolism CDS tool for emergency physicians, we conducted a formative assessment to determine providers' level of interest and input on designs and content. This was a study to conduct a formative assessment of emergency medicine (EM) physicians that included focus groups and key informant interviews. The focus of this study was twofold, to determine the general attitude towards CDS tool integration and the ideal integration point into the clinical workflow. To accomplish this, we first approached EM physicians in a focus group, then, during key informant interviews, we presented workflow designs and gave a scenario to help the providers visualise how the CDS tool works. Participants were asked questions regarding the trigger location, trigger words, integration into their workflow, perceived utility and heuristic of the tool. Results from the participants' survey responses to trigger location, perceived utility and efficiency, indicated that the providers felt the tool would be more of a hindrance than an aid. However, some providers commented that they had not had exposure to CDS tools but had used online calculators, and thought the tools would be helpful at the point-of-care if integrated into the EHR. Furthermore, there was a preference for an order entry wireframe. This study highlights several factors to consider when designing CDS tools: (1) formative assessment of EHR functionality and clinical environment workflow, (2) focus groups and key informative interviews to incorporate providers' perceptions of CDS and workflow integration and/or (3) the demonstration of proposed workflows through wireframes to help providers visualise design concepts.

Usability Testing of a Complex Clinical Decision Support Tool in the Emergency Department: Lessons Learned.

BACKGROUND: As the electronic health record (EHR) becomes the preferred documentation tool across medical practices, health care organizations are pushing for clinical decision support systems (CDSS) to help bring clinical decision support (CDS) tools to the forefront of patient-physician interactions. A CDSS is integrated into the EHR and allows physicians to easily utilize CDS tools. However, often CDSS are integrated into the EHR without an initial phase of usability testing, resulting in poor adoption rates. Usability testing is important because it evaluates a CDSS by testing it on actual users. This paper outlines the usability phase of a study, which will test the impact of integration of the Wells CDSS for pulmonary embolism (PE) diagnosis into a large urban emergency department, where workflow is often chaotic and high stakes decisions are frequently made. We hypothesize that conducting usability testing prior to integration of the Wells score into an emergency room EHR will result in increased adoption rates by physicians. OBJECTIVE: The objective of the study was to conduct usability testing for the integration of the Wells clinical prediction rule into a tertiary care center's emergency department EHR. METHODS: We conducted usability testing of a CDS tool in the emergency department EHR. The CDS tool consisted of the Wells rule for PE in the form of a calculator and was triggered off computed tomography (CT) orders or patients' chief complaint. The study was conducted at a tertiary hospital in Queens, New York. There were seven residents that were recruited and participated in two phases of usability testing. The usability testing employed a "think aloud" method and "near-live" clinical simulation, where care providers interacted with standardized patients enacting a clinical scenario. Both phases were audiotaped, video-taped, and had screen-capture software activated for onscreen recordings. RESULTS: Phase I: Data from the "think-aloud" phase of the study showed an overall positive outlook on the Wells tool in assessing a patient for a PE diagnosis. Subjects described the tool as "well-organized" and "better than clinical judgment". Changes were made to improve tool placement into the EHR to make it optimal for decision-making, auto-populating boxes, and minimizing click fatigue. Phase II: After incorporating the changes noted in Phase 1, the participants noted tool improvements. There was less toggling between screens, they had all the clinical information required to complete the tool, and were able to complete the patient visit efficiently. However, an optimal location for triggering the tool remained controversial. CONCLUSIONS: This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a CDS tool that will be implemented into the emergency room environment. Both methods proved useful in the assessment of the CDS tool and allowed us to refine tool usability and workflow.