Choosing and Using Patient-Reported Outcome Measures in Clinical Practice.

The increasing use of patient-reported outcome (PRO) measures is forcing clinicians and health care systems to decide which to select and how to incorporate them into their records and clinical workflows. This overview addresses 3 topics related to these concerns. First, a literature review summarizes key psychometric and practical factors (such as reliability, responsiveness, computer adaptive testing, and interpretability) in choosing PROs for clinical practice. Second, 3 clinical decision support issues are highlighted: gathering PROs, electronic health record effect on providers, and incorporating PROs into clinical decision support design and implementation. Lastly, the salience of crosscutting domains as well as 9 key pragmatic decisions are reviewed. Crosscutting domains are those that are relevant across most medical and mental health conditions, such as the SPADE symptom pentad (sleep problems, pain, anxiety, depression, low energy/fatigue) and physical functioning. The 9 pragmatic decisions include (1) generic vs disease-specific scales; (2) single- vs multidomain scales; (3) universal scales vs user-choice selection; (4) number of domains to measure; (5) prioritization of domains when multiple domains are assessed; (6) action thresholds; (7) clinical purpose (screening vs monitoring); as well as the (8) frequency and (9) logistical aspects of PRO administration.

Study of EHR-mediated workflows using ethnography and process mining methods.

Rapid ethnography and data mining approaches have been used individually to study clinical workflows, but have seldom been used together to overcome the limitations inherent in either type of method. For rapid ethnography, how reliable are the findings drawn from small samples? For data mining, how accurate are the discoveries drawn from automatic analysis of big data, when compared with observable data? This paper explores the combined use of rapid ethnography and process mining, aka ethno-mining, to study and compare metrics of a typical clinical documentation task, vital signs charting. The task was performed with different electronic health records (EHRs) used in three different hospital sites. The individual methods revealed substantial discrepancies in task duration between sites. Specifically, means of 159.6(78.55), 38.2(34.9), and 431.3(283.04) seconds were captured with rapid ethnography. When process mining was used, means of 518.6(3,808), 345.5(660.6), and 119.74(210.3) seconds were found. When ethno-mining was applied instead, outliers could be identified, explained and removed. Without outliers, mean task duration was similar between sites (78.1(66.7), 72.5(78.5), and 71.7(75) seconds). Results from this work suggest that integrating rapid ethnography and data mining into a single process may provide more meaningful results than a siloed approach when studying of workflow.

A micro-analytic approach to understanding electronic health record navigation paths.

Clinician task performance is significantly impacted by the navigational efficiency of the system interface. Here we propose and evaluate a navigational complexity framework useful for examining differences in electronic health record (EHR) interface systems and their impact on task performance. The methodological approach includes 1) expert-based methods-specifically, representational analysis (focused on interface elements), keystroke level modeling (KLM), and cognitive walkthrough; and 2) quantitative analysis of interactive behaviors based on video-captured observations. Medication administration record (MAR) tasks completed by nurses during preoperative (PreOp) patient assessment were studied across three Mayo Clinic regional campuses and three different EHR systems. By analyzing the steps executed within the interfaces involved to complete the MAR tasks, we characterized complexities in EHR navigation. These complexities were reflected in time spent on task, click counts, and screen transitions, and were found to potentially influence nurses' performance. Two of the EHR systems, employing a single screen format, required less time to complete (mean 101.5, range 106-97 s), respectively, compared to one system employing multiple screens (176 s, 73% increase). These complexities surfaced through trade-offs in cognitive processes that could potentially influence nurses' performance. Factors such as perceptual-motor activity, visual search, and memory load impacted navigational complexity. An implication of this work is that small tractable changes in interface design can substantially improve EHR navigation, overall usability, and workflow.

A Task-Analytic Framework Comparing Preoperative Electronic Health Record-Mediated Nursing Workflow in Different Settings.

Preoperative care is a critical, yet complex, time-sensitive process. Optimization of workflow is challenging for many reasons, including a lack of standard workflow analysis methods. We sought to comprehensively characterize electronic health record-mediated preoperative nursing workflow. We employed a structured methodological framework to investigate and explain variations in the workflow. Video recording software captured 10 preoperative cases at Arizona and Florida regional referral centers. We compared the distribution of work for electronic health record tasks and off-screen tasks through quantitative analysis. Suboptimal patterns and reasons for variation were explored through qualitative analysis. Although both settings used the same electronic health record system, electronic health record tasks and off-screen tasks time distribution and patterns were notably different across two sites. Arizona nurses spent a longer time completing preoperative assessment. Electronic health record tasks occupied a higher proportion of time in Arizona, while off-screen tasks occupied a higher proportion in Florida. The contextual analysis helped to identify the variation associated with the documentation workload, preparation of the patient, and regional differences. These findings should seed hypotheses for future optimization efforts and research supporting standardization and harmonization of workflow across settings, post-electronic health record conversion.

We're Lost, But We are Making Good Time: Navigating Complex Pathways in a Patient-Order Management Task.

Patient order management (POM) is a mission-critical task for perioperative workflow. Interface complexity within different EHR systems result in poor usability, increasing documentation burden. POM interfaces were compared across two systems prior to (Cerner SurgiNet) and subsequent to an EHR conversion (Epic). Here we employ a navigational complexity framework useful for examining differences in EHR interface systems. The methodological approach includes 1) expert-based methods-specifically, functional analysis, keystroke level model (KLM) and cognitive walkthrough, and 2) quantitative analysis of observed interactive user behaviors. We found differences in relation to navigational complexity with the SurgiNet interface displaying a higher number of unused POM functions, with 12 in total whereas Epic displayed 7 total functions. As reflected in all measures, Epic facilitated a more streamlined task-focused user experience. The approach enabled us to scrutinize the impact of different EHR interfaces on task performance and usability barriers subsequent to system implementation.

EHR Conversion on the PreOp Care: A Pre-Post Workflow Comparison.

The impact of EHRs conversion on clinicians' daily work is crucial to evaluate the success of the intervention for Hospitals and to yield valuable insights into quality improvement. To assess the impact of different EHR systems on the preoperative nursing workflow, we used a structured framework combining quantitative time and motion study and qualitative cognitive analysis to characterize, visualize and explain the differences before and after an EHR conversion. The results showed that the EHR conversion brought a significant decrease in the patient case time and a reduced percentage of time using EHR. PreOp nurses spent a higher proportion of time caring for the patient, while the important tasks were completed in a more continuous pattern after the EHR conversion. The workflow variance was due to different nurse's cognitive process and the task time change was reduced because of some new interface features in the new EHR systems.

Mayo Clinic Registry of Operational Tasks (ROOT): A Paradigm Shift in Electronic Health Reco Implementation Evaluation.

OBJECTIVE: To systematically examine clinical workflows before and after a major electronic health record (EHR) implementation, we performed this study. EHR implementation and/or conversion are associated with many challenges, which are barriers to optimal care. Clinical workflows may be significantly affected by EHR implementations and conversions, resulting in provider frustration and reduced efficiency. PATIENTS AND METHODS: Our institution completed a large EHR conversion and workflow standardization converting from 3 EHRs (GE Centricity and 2 versions of Cerner) to a system-wide Epic platform. To study this quantitatively and qualitatively, we collected and curated clinical workflows through rapid ethnography, workflow observation, video ethnography, and log-file analyses of hundreds of providers, patients, and more than 100,000 log files. The study included 5 geographic sites in 4 states (Arizona, Minnesota, Florida, and Wisconsin). This project began in April 2016, and will be completed by December 2019. Our study began on May 1, 2016, and is ongoing. RESULTS: Salient themes include the importance of prioritizing clinical areas with the most intensive EHR use, the value of tools to identify bottlenecks in workflow that cause delays, and desire for additional training to optimize navigation. Video microanalyses identified marked differences in patterns of workflow and EHR navigation patterns across sites. Log-file analyses and social network analyses identified differences in personnel roles, which led to differences in patient-clinician interaction, time spent using the EHR, and paper-based artifacts. CONCLUSION: Assessing and curating workflow data before and after EHR conversion may provide opportunities for unexpected efficiencies in workflow optimization and information-system redesign. This project may be a model for capturing significant new knowledge in using EHRs to improve patient care, workflow efficiency, and outcomes.

Process Mining and Ethnography Study of Medication Reconciliation Tasks.

We studied the medication reconciliation (MedRec) task through analysis of computer logs and ethnographic data. Time spent by healthcare providers performing MedRec was compared between two different EHR systems used at four different regional perioperative settings. Only one of the EHRs used at two settings generated computer logs that supported automatic discovery of the MedRec task. At those two settings, 53 providers generated 383 MedRec instances. Findings from the computer logs were validated with ethnographic data, leading to the identification and removal of 47 outliers. Without outliers, one of the settings had slightly smaller mean (SD) time in seconds 67.3 (40.2) compared with the other, 92.1 (25). The difference in time metrics was statistically significant (p<.001). Reusability of an existing task-based analytic method allowed for rapid study of EHR-based workflow and task.

In Search of Vital Signs: A Comparative Study of EHR Documentation.

Vital sign documentation is an essential part of perioperative workflow. Health information technology can introduce complexity into all facets of documentation and burden clinicians with high cognitive load3-4. The Mayo Clinic enterprise is in the process of documenting current EHR-mediated workflow prior to a system-wide EHR conversion. We compared and evaluated three different vital sign documentation interfaces in pre-operative nursing assessments at three different Mayo Clinic sites. The interfaces differed in their modes of interaction, organization of patient information and cognitive support. Analyses revealed that accessing displays and the organization of interface elements are often unintuitive and inefficient, creating unnecessary complexities when interacting with the system. These differences surface through interface workflow models and interactive behavior measures for accessing, logging and reviewing patient information. Different designs differentially mediate task performance, which can ultimately mitigate errors for complex cognitive tasks, risking patient safety. Identifying barriers to interface usability and bottlenecks in EHR-mediated workflow can lead to system redesigns that minimize cognitive load while improving patient safety and efficiency.

Effect of a Novel Clinical Decision Support Tool on the Efficiency and Accuracy of Treatment Recommendations for Cholesterol Management.

BACKGROUND: The 2013 American College of Cardiology / American Heart Association Guidelines for the Treatment of Blood Cholesterol emphasize treatment based on cardiovascular risk. But finding time in a primary care visit to manually calculate cardiovascular risk and prescribe treatment based on risk is challenging. We developed an informatics-based clinical decision support tool, MayoExpertAdvisor, to deliver automated cardiovascular risk scores and guideline-based treatment recommendations based on patient-specific data in the electronic heath record. OBJECTIVE: To assess the impact of our clinical decision support tool on the efficiency and accuracy of clinician calculation of cardiovascular risk and its effect on the delivery of guideline-consistent treatment recommendations. METHODS: Clinicians were asked to review the EHR records of selected patients. We evaluated the amount of time and the number of clicks and keystrokes needed to calculate cardiovascular risk and provide a treatment recommendation with and without our clinical decision support tool. We also compared the treatment recommendation arrived at by clinicians with and without the use of our tool to those recommended by the guidelines. RESULTS: Clinicians saved 3 minutes and 38 seconds in completing both tasks with MayoExpertAdvisor, used 94 fewer clicks and 23 fewer key strokes, and improved accuracy from the baseline of 60.61% to 100% for both the risk score calculation and guideline-consistent treatment recommendation. CONCLUSION: Informatics solution can greatly improve the efficiency and accuracy of individualized treatment recommendations and have the potential to increase guideline compliance.

Knowledge as a Service at the Point of Care.

An electronic health record (EHR) can assist the delivery of high-quality patient care, in part by providing the capability for a broad range of clinical decision support, including contextual references (e.g., Infobuttons), alerts and reminders, order sets, and dashboards. All of these decision support tools are based on clinical knowledge; unfortunately, the mechanisms for managing rules, order sets, Infobuttons, and dashboards are often unrelated, making it difficult to coordinate the application of clinical knowledge to various components of the clinical workflow. Additional complexity is encountered when updating enterprise-wide knowledge bases and delivering the content through multiple modalities to different consumers. We present the experience of Mayo Clinic as a case study to examine the requirements and implementation challenges related to knowledge management across a large, multi-site medical center. The lessons learned through the development of our knowledge management and delivery platform will help inform the future development of interoperable knowledge resources.