Design and Implementation of an Electronic Health Record-Integrated Hypertension Management Application.

BACKGROUND: High blood pressure affects approximately 116 million adults in the United States. It is the leading risk factor for death and disability across the world. Unfortunately, over the past decade, hypertension control rates have decreased across the United States. Prediction models and clinical studies have shown that reducing clinician inertia alone is sufficient to reach the target of ≥80% blood pressure control. Digital health tools containing evidence-based algorithms that are able to reduce clinician inertia are a good fit for turning the tide in blood pressure control, but careful consideration should be taken in the design process to integrate digital health interventions into the clinical workflow. METHODS: We describe the development of a provider-facing hypertension management platform. We enumerate key steps of the development process, including needs finding, clinical workflow analysis, treatment algorithm creation, platform design and electronic health record integration. We interviewed and surveyed 5 Stanford clinicians from primary care, cardiology, and their clinical care team members (including nurses, advanced practice providers, medical assistants) to identify needs and break down the steps of clinician workflow analysis. The application design and development stage were aided by a team of approximately 15 specialists in the fields of primary care, hypertension, bioinformatics, and software development. CONCLUSIONS: Digital monitoring holds immense potential for revolutionizing chronic disease management. Our team developed a hypertension management platform at an academic medical center to address some of the top barriers to adoption and achieving clinical outcomes. The frameworks and processes described in this article may be used for the development of a diverse range of digital health tools in the cardiovascular space.

Lessons Learned from a Multi-Site, Team-Based Serious Illness Care Program Implementation at an Academic Medical Center.

Background: Patients with serious illness benefit from conversations to share prognosis and explore goals and values. To address this, we implemented Ariadne Labs' Serious Illness Care Program (SICP) at Stanford Health Care. Objective: Improve quantity, timing, and quality of serious illness conversations. Methods: Initial implementation followed Ariadne Labs' SICP framework. We later incorporated a team-based approach that included nonphysician care team members. Outcomes included number of patients with documented conversations according to clinician role and practice location. Machine learning algorithms were used in some settings to identify eligible patients. Results: Ambulatory oncology and hospital medicine were our largest implementation sites, engaging 4707 and 642 unique patients in conversations, respectively. Clinicians across eight disciplines engaged in these conversations. Identified barriers that included leadership engagement, complex workflows, and patient identification. Conclusion: Several factors contributed to successful SICP implementation across clinical sites: innovative clinical workflows, machine learning based predictive algorithms, and nonphysician care team member engagement.

Targeting repetitive laboratory testing with electronic health records-embedded predictive decision support: A pre-implementation study.

INTRODUCTION: Unnecessary laboratory testing contributes to patient morbidity and healthcare waste. Despite prior attempts at curbing such overutilization, there remains opportunity for improvement using novel data-driven approaches. This study presents the development and early evaluation of a clinical decision support tool that uses a predictive model to help providers reduce low-yield, repetitive laboratory testing in hospitalized patients. METHODS: We developed an EHR-embedded SMART on FHIR application that utilizes a laboratory test result prediction model based on historical laboratory data. A combination of semi-structured physician interviews, usability testing, and quantitative analysis on retrospective laboratory data were used to inform the tool's development and evaluate its acceptability and potential clinical impact. KEY RESULTS: Physicians identified culture and lack of awareness of repeat orders as key drivers for overuse of inpatient blood testing. Users expressed an openness to a lab prediction model and 13/15 physicians believed the tool would alter their ordering practices. The application received a median System Usability Scale score of 75, corresponding to the 75th percentile of software tools. On average, physicians desired a prediction certainty of 85% before discontinuing a routine recurring laboratory order and a higher certainty of 90% before being alerted. Simulation on historical lab data indicates that filtering based on accepted thresholds could have reduced âˆ¼22% of repeat chemistry panels. CONCLUSIONS: The use of a predictive algorithm as a means to calculate the utility of a diagnostic test is a promising paradigm for curbing laboratory test overutilization. An EHR-embedded clinical decision support tool employing such a model is a novel and acceptable intervention with the potential to reduce low-yield, repetitive laboratory testing.

Considerations in the reliability and fairness audits of predictive models for advance care planning.

Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a gap of operational guidance for performing reliability and fairness audits in practice. Following guideline recommendations, we conducted a reliability audit of two models based on model performance and calibration as well as a fairness audit based on summary statistics, subgroup performance and subgroup calibration. We assessed the Epic End-of-Life (EOL) Index model and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model in 3 practice settings: Primary Care, Inpatient Oncology and Hospital Medicine, using clinicians' answers to the surprise question ("Would you be surprised if [patient X] passed away in [Y years]?") as a surrogate outcome. For performance, the models had positive predictive value (PPV) at or above 0.76 in all settings. In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model had higher sensitivity (0.69, 0.89 respectively) than the EOL model (0.20, 0.27), and better calibration (O/E 1.5, 1.7) than the EOL model (O/E 2.5, 3.0). The Epic EOL model flagged fewer patients (11%, 21% respectively) than the Stanford HM ACP model (38%, 75%). There were no differences in performance and calibration by sex. Both models had lower sensitivity in Hispanic/Latino male patients with Race listed as "Other." 10 clinicians were surveyed after a presentation summarizing the audit. 10/10 reported that summary statistics, overall performance, and subgroup performance would affect their decision to use the model to guide care; 9/10 said the same for overall and subgroup calibration. The most commonly identified barriers for routinely conducting such reliability and fairness audits were poor demographic data quality and lack of data access. This audit required 115 person-hours across 8-10 months. Our recommendations for performing reliability and fairness audits include verifying data validity, analyzing model performance on intersectional subgroups, and collecting clinician-patient linkages as necessary for label generation by clinicians. Those responsible for AI models should require such audits before model deployment and mediate between model auditors and impacted stakeholders.

User-centred design for machine learning in health care: a case study from care management.

OBJECTIVES: Few machine learning (ML) models are successfully deployed in clinical practice. One of the common pitfalls across the field is inappropriate problem formulation: designing ML to fit the data rather than to address a real-world clinical pain point. METHODS: We introduce a practical toolkit for user-centred design consisting of four questions covering: (1) solvable pain points, (2) the unique value of ML (eg, automation and augmentation), (3) the actionability pathway and (4) the model's reward function. This toolkit was implemented in a series of six participatory design workshops with care managers in an academic medical centre. RESULTS: Pain points amenable to ML solutions included outpatient risk stratification and risk factor identification. The endpoint definitions, triggering frequency and evaluation metrics of the proposed risk scoring model were directly influenced by care manager workflows and real-world constraints. CONCLUSIONS: Integrating user-centred design early in the ML life cycle is key for configuring models in a clinically actionable way. This toolkit can guide problem selection and influence choices about the technical setup of the ML problem.

Team is brain: leveraging EHR audit log data for new insights into acute care processes.

OBJECTIVE: To determine whether novel measures of contextual factors from multi-site electronic health record (EHR) audit log data can explain variation in clinical process outcomes. MATERIALS AND METHODS: We selected one widely-used process outcome: emergency department (ED)-based team time to deliver tissue plasminogen activator (tPA) to patients with acute ischemic stroke (AIS). We evaluated Epic audit log data (that tracks EHR user-interactions) for 3052 AIS patients aged 18+ who received tPA after presenting to an ED at three Northern California health systems (Stanford Health Care, UCSF Health, and Kaiser Permanente Northern California). Our primary outcome was door-to-needle time (DNT) and we assessed bivariate and multivariate relationships with six audit log-derived measures of treatment team busyness and prior team experience. RESULTS: Prior team experience was consistently associated with shorter DNT; teams with greater prior experience specifically on AIS cases had shorter DNT (minutes) across all sites: (Site 1: -94.73, 95% CI: -129.53 to 59.92; Site 2: -80.93, 95% CI: -130.43 to 31.43; Site 3: -42.95, 95% CI: -62.73 to 23.17). Teams with greater prior experience across all types of cases also had shorter DNT at two sites: (Site 1: -6.96, 95% CI: -14.56 to 0.65; Site 2: -19.16, 95% CI: -36.15 to 2.16; Site 3: -11.07, 95% CI: -17.39 to 4.74). Team busyness was not consistently associated with DNT across study sites. CONCLUSIONS: EHR audit log data offers a novel, scalable approach to measure key contextual factors relevant to clinical process outcomes across multiple sites. Audit log-based measures of team experience were associated with better process outcomes for AIS care, suggesting opportunities to study underlying mechanisms and improve care through deliberate training, team-building, and scheduling to maximize team experience.

Predicting and Responding to Clinical Deterioration in Hospitalized Patients by Using Artificial Intelligence: Protocol for a Mixed Methods, Stepped Wedge Study.

BACKGROUND: The early identification of clinical deterioration in patients in hospital units can decrease mortality rates and improve other patient outcomes; yet, this remains a challenge in busy hospital settings. Artificial intelligence (AI), in the form of predictive models, is increasingly being explored for its potential to assist clinicians in predicting clinical deterioration. OBJECTIVE: Using the Systems Engineering Initiative for Patient Safety (SEIPS) 2.0 model, this study aims to assess whether an AI-enabled work system improves clinical outcomes, describe how the clinical deterioration index (CDI) predictive model and associated work processes are implemented, and define the emergent properties of the AI-enabled work system that mediate the observed clinical outcomes. METHODS: This study will use a mixed methods approach that is informed by the SEIPS 2.0 model to assess both processes and outcomes and focus on how physician-nurse clinical teams are affected by the presence of AI. The intervention will be implemented in hospital medicine units based on a modified stepped wedge design featuring three stages over 11 months-stage 0 represents a baseline period 10 months before the implementation of the intervention; stage 1 introduces the CDI predictions to physicians only and triggers a physician-driven workflow; and stage 2 introduces the CDI predictions to the multidisciplinary team, which includes physicians and nurses, and triggers a nurse-driven workflow. Quantitative data will be collected from the electronic health record for the clinical processes and outcomes. Interviews will be conducted with members of the multidisciplinary team to understand how the intervention changes the existing work system and processes. The SEIPS 2.0 model will provide an analytic framework for a mixed methods analysis. RESULTS: A pilot period for the study began in December 2020, and the results are expected in mid-2022. CONCLUSIONS: This protocol paper proposes an approach to evaluation that recognizes the importance of assessing both processes and outcomes to understand how a multifaceted AI-enabled intervention affects the complex team-based work of identifying and managing clinical deterioration. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/27532.

Developing a delivery science for artificial intelligence in healthcare.

Artificial Intelligence (AI) has generated a large amount of excitement in healthcare, mostly driven by the emergence of increasingly accurate machine learning models. However, the promise of AI delivering scalable and sustained value for patient care in the real world setting has yet to be realized. In order to safely and effectively bring AI into use in healthcare, there needs to be a concerted effort around not just the creation, but also the delivery of AI. This AI "delivery science" will require a broader set of tools, such as design thinking, process improvement, and implementation science, as well as a broader definition of what AI will look like in practice, which includes not just machine learning models and their predictions, but also the new systems for care delivery that they enable. The careful design, implementation, and evaluation of these AI enabled systems will be important in the effort to understand how AI can improve healthcare.

When order sets do not align with clinician workflow: assessing practice patterns in the electronic health record.

BACKGROUND: Order sets are widely used tools in the electronic health record (EHR) for improving healthcare quality. However, there is limited insight into how well they facilitate clinician workflow. We assessed four indicators based on order set usage patterns in the EHR that reflect potential misalignment between order set design and clinician workflow needs. METHODS: We used data from the EHR on all orders of medication, laboratory, imaging and blood product items at an academic hospital and an itemset mining approach to extract orders that frequently co-occurred with order set use. We identified the following four indicators: infrequent ordering of order set items, rapid retraction of medication orders from order sets, additional a la carte ordering of items not included in order sets and a la carte ordering of items despite being listed in the order set. RESULTS: There was significant variability in workflow alignment across the 11 762 order set items used in the 77 421 inpatient encounters from 2014 to 2017. The median ordering rate was 4.1% (IQR 0.6%-18%) and median medication retraction rate was 4% (IQR 2%-10%). 143 (5%) medications were significantly less likely while 68 (3%) were significantly more likely to be retracted than if the same medication was ordered a la carte. 214 (39%) order sets were associated with least one additional item frequently ordered a la carte and 243 (45%) order sets contained at least one item that was instead more often ordered a la carte. CONCLUSION: Order sets often do not align with what clinicians need at the point of care. Quantitative insights from EHRs may inform how order sets can be optimised to facilitate clinician workflow.

Impact of problem-based charting on the utilization and accuracy of the electronic problem list.

Objective: Problem-based charting (PBC) is a method for clinician documentation in commercially available electronic medical record systems that integrates note writing and problem list management. We report the effect of PBC on problem list utilization and accuracy at an academic intensive care unit (ICU). Materials and Methods: An interrupted time series design was used to assess the effect of PBC on problem list utilization, which is defined as the number of new problems added to the problem list by clinicians per patient encounter, and of problem list accuracy, which was determined by calculating the recall and precision of the problem list in capturing 5 common ICU diagnoses. Results: In total, 3650 and 4344 patient records were identified before and after PBC implementation at Stanford Hospital. An increase of 2.18 problems (>50% increase) in the mean number of new problems added to the problem list per patient encounter can be attributed to the initiation of PBC. There was a significant increase in recall attributed to the initiation of PBC for sepsis (ß = 0.45, P < .001) and acute renal failure (ß = 0.2, P = .007), but not for acute respiratory failure, pneumonia, or venous thromboembolism. Discussion: The problem list is an underutilized component of the electronic medical record that can be a source of clinician-structured data representing the patient's clinical condition in real time. PBC is a readily available tool that can integrate problem list management into physician workflow. Conclusion: PBC improved problem list utilization and accuracy at an academic ICU.

Discordance Between Apolipoprotein B and LDL-Cholesterol in Young Adults Predicts Coronary Artery Calcification: The CARDIA Study.

BACKGROUND: High levels of apolipoprotein B (apoB) have been shown to predict atherosclerotic cardiovascular disease (CVD) in adults even in the context of low levels of low-density lipoprotein cholesterol (LDL-C) or non-high-density lipoprotein cholesterol (non-HDL-C). OBJECTIVES: This study aimed to quantify the associations between apoB and the discordance between apoB and LDL-C or non-HDL-C in young adults and measured coronary artery calcium (CAC) in midlife. METHODS: Data were derived from a multicenter cohort study of young adults recruited at ages 18 to 30 years. All participants with complete baseline CVD risk factor data, including apoB and year 25 (Y25) CAC score, were entered into this study. Presence of CAC was defined as having a positive, nonzero Agatston score as determined by computed tomography. Baseline apoB values were divided into tertiles of 4 mutually exclusive concordant/discordant groups, based on median apoB and LDL-C or non-HDL-C. RESULTS: Analysis included 2,794 participants (mean age: 25 ± 3.6 years; body mass index: 24.5 ± 5 kg/m(2); and 44.4% male). Mean lipid values were as follows: total cholesterol: 177.3 ± 33.1 mg/dl; LDL-C: 109.9 ± 31.1 mg/dl; non-HDL-C: 124.0 ± 33.5 mg/dl; HDL-C: 53 ± 12.8 mg/dl; and apoB: 90.7 ± 24 mg/dl; median triglycerides were 61 mg/dl. Compared with the lowest apoB tertile, higher odds of developing Y25 CAC were seen in the middle (odds ratio [OR]: 1.53) and high (OR: 2.28) tertiles based on traditional risk factor-adjusted models. High apoB and low LDL-C or non-HDL-C discordance was also associated with Y25 CAC in adjusted models (OR: 1.55 and OR: 1.45, respectively). CONCLUSIONS: These data suggest a dose-response association between apoB in young adults and the presence of midlife CAC independent of baseline traditional CVD risk factors.