Advanced Care Planning Content Encoding with Natural Language Processing.

While advanced care planning (ACP) is an essential practice for ensuring patient-centered care, its adoption remains poor and the completeness of its documentation variable. Natural language processing (NLP) approaches hold promise for supporting ACP, including its use for decision support to improve ACP gaps at the point of care. ACP themes were annotated on palliative care notes across four annotators (Fleiss kappa = 0.753) and supervised models trained (Huggingface models bert-base-uncased and Bio_ClinicalBERT) using 5-fold cross validation (F1=0.8, precision=0.75, recall=0.86, any theme). When applied across the full note corpus of 12,711 notes, we observed variability in documentation of ACP information. Our findings demonstrate the promise of NLP approaches for informatics-based approaches for ACP and patient-centered care.

Combined Expert and User-Driven Usability Assessment of Trauma Decision Support Systems Improves User-Centered Design.

BACKGROUND: Trauma clinical decision support systems improve adherence with evidence-based practice but suffer from poor usability and the lack of a user-centered design. The objective of this study was to compare the effectiveness of user and expert-driven usability testing methods to detect usability issues in a rib fracture clinical decision support system and identify guiding principles for trauma clinical decision support systems. METHODS: A user-driven and expert-driven usability investigation was conducted using a clinical decision support system developed for patients with rib fractures. The user-driven usability evaluation was as follows: 10 clinicians were selected for simulation-based usability testing using snowball sampling, and each clinician completed 3 simulations using a video-conferencing platform. End-users participated in a novel team-based approach that simulated realistic clinical workflows. The expert-driven heuristic evaluation was as follows: 2 usability experts conducted a heuristic evaluation of the clinical decision support system using 10 common usability heuristics. Usability issues were identified, cataloged, and ranked for severity using a 4-level ordinal scale. Thematic analysis was utilized to categorize the identified usability issues. RESULTS: Seventy-nine usability issues were identified; 63% were identified by experts and 48% by end-users. Notably, 58% of severe usability issues were identified by experts alone. Only 11% of issues were identified by both methods. Five themes were identified that could guide the design of clinical decision support systems-transparency, functionality and integration into workflow, automated and noninterruptive, flexibility, and layout and appearance. Themes were preferentially identified by different methods. CONCLUSION: We found that a dual-method usability evaluation involving usability experts and end-users drastically improved detection of usability issues over single-method alone. We identified 5 themes to guide trauma clinical decision support system design. Performing usability testing via a remote video-conferencing platform facilitated multi-site involvement despite a global pandemic.

The role of organizational culture in health information technology implementations: A scoping review.

Introduction: The exponential growth in health information technology (HIT) presents an immense opportunity for facilitating the data-to-knowledge-to-performance loop which supports learning health systems. This scoping review addresses the gap in knowledge around HIT implementation contextual factors such as organizational culture and provides a current state assessment. Methods: A search of 13 databases guided by Arskey and O'Malley's framework identified content on HIT implementations and organizational culture. The Consolidated Framework for Implementation Research (CFIR) was used to assess culture and to develop review criteria. Culture stress, culture effort, implementation climate, learning climate, readiness for implementation, leadership engagement, and available resources were the constructs examined. Rayyan and Qualtrics were used for screening and data extraction. Results: Fifty two studies included were mainly conducted in Academic Health Centers (n = 18, 35%) and at urban locations (n = 50, 96%). Interviews frequently used for data collection (n = 26, 50%) and guided by multiple frameworks (n = 34). Studies mostly focused on EHR implementations (n = 23, 44%) followed by clinical decision support (n = 9, 17%). About two-thirds (n = 34, 65%) reflected culture stress theme and 62% (21 of 34) acknowledged it as a barrier. Culture effort identified in 27 studies and was a facilitator in most (78%, 21 of 27). Leadership engagement theme in majority studies (71%, n = 37), with 35% (n = 13) noting it as a facilitator. Eighty percent (42 studies) noted available resources, 12 of which identified this as barrier to successful implementation. Conclusions: It is vital to determine the culture and other CFIR inner setting constructs that are significant to HIT implementation as facilitators or barriers. This scoping review presents a limited number of empirical studies in this topic highlighting the need for additional research to quantify the effects of culture. This will help build evidence and best practices that facilitate HIT implementations and hence serve as a platform to support robust learning health systems.

Application of Causal Discovery Algorithms in Studying the Nephrotoxicity of Remdesivir Using Longitudinal Data from the EHR.

Remdesivir has been widely used for the treatment of Coronavirus (COVID) in hospitalized patients, but its nephrotoxicity is still under investigation1. Given the paucity of knowledge regarding the mechanism and optimal treatment of the development of acute kidney injury (AKI) in the setting of COVID, we analyzed the role of remdesivir and built multifactorial causal models of COVID-AKI by applying causal discovery machine learning techniques. Risk factors of COVID-AKI and renal function measures were represented in a temporal sequence using longitudinal data from EHR. Our models successfully recreated known causal pathways to changes in renal function and interactions with each other and examined the consistency of high-level causal relationships over a 4-day course of remdesivir. Results indicated a need for assessment of renal function on day 2 and 3 use of remdesivir, while uncovering that remdesivir may pose less risk to AKI than existing conditions of chronic kidney disease.

Validation of Administrative Coding and Clinical Notes for Hospital-Acquired Acute Kidney Injury in Adults.

Acute kidney injury (AKI) is potentially catastrophic and commonly seen among inpatients. In the United States, the quality of administrative coding data for capturing AKI accurately is questionable and needs to be updated. This retrospective study validated the quality of administrative coding for hospital-acquired AKI and explored the opportunities to improve the phenotyping performance by utilizing additional data sources from the electronic health record (EHR). A total of34570 patients were included, and overall prevalence of AKI based on the KDIGO reference standard was 10.13%, We obtained significantly different quality measures (sensitivity.-0.486, specificity:0.947, PPV.0.509, NPV:0.942 in the full cohort) of administrative coding from the previously reported ones in the U.S. Additional use of clinical notes by incorporating automatic NLP data extraction has been found to increase the AUC in phenotyping AKI, and AKI was better recognized in patients with heart failure, indicating disparities in the coding and management of AKI.

Expert Perspectives on Definitions, Drivers and Informatics Contributions to Learning Health Systems.

Learning Health System (LHS) proposes a new paradigm in scientific enterprise to facilitate the rapid movement of data to knowledge (D2K) and knowledge to practice (K2P). Informatics can play a pivotal role in facilitating feedback loops and rapid cycles of learning across D2K and K2P. Though informatics has been acknowledged as a critical component of LHS, it remains unclear how leaders in informatics are conceptualizing its role in promoting LHS. This study sought to gain insights from informatics leaders and experts on their perspectives around role of informatics in LHS. We conducted semi-structured interviews with fourteen informatics leaders across different informatics domains and leadership positions. Our results revealed areas of agreement around key concepts related to LHS as well as opportunities to improve messaging and add clarity to the role of informatics in promoting LHS.

Rapid implementation of a COVID-19 remote patient monitoring program.

OBJECTIVE: The study sought to evaluate early lessons from a remote patient monitoring engagement and education technology solution for patients with coronavirus disease 2019 (COVID-19) symptoms. MATERIALS AND METHODS: A COVID-19-specific remote patient monitoring solution (GetWell Loop) was offered to patients with COVID-19 symptoms. The program engaged patients and provided educational materials and the opportunity to share concerns. Alerts were resolved through a virtual care workforce of providers and medical students. RESULTS: Between March 18 and April 20, 2020, 2255 of 3701 (60.93%) patients with COVID-19 symptoms enrolled, resulting in over 2303 alerts, 4613 messages, 13 hospital admissions, and 91 emergency room visits. A satisfaction survey was given to 300 patient respondents, 74% of whom would be extremely likely to recommend their doctor. DISCUSSION: This program provided a safe and satisfying experience for patients while minimizing COVID-19 exposure and in-person healthcare utilization. CONCLUSIONS: Remote patient monitoring appears to be an effective approach for managing COVID-19 symptoms at home.

Electronic Progress Note Reading Patterns: An Eye Tracking Analysis.

This study used eye-tracking to understand how the order of note sections influences the way physicians read electronic progress notes. Participants (n = 7) wore an eye-tracking device while reviewing progress notes for four patient cases and then provided a verbal summary. We reviewed and analyzed verbal summaries and eye tracking recordings. Wide variation in reading behaviors existed. There was no relationship between time spent reading a section and section origin of verbal summaries.

Challenges and Opportunities to Improve the Clinician Experience Reviewing Electronic Progress Notes.

BACKGROUND: High-quality clinical notes are essential to effective clinical communication. However, electronic clinical notes are often long, difficult to review, and contain information that is potentially extraneous or out of date. Additionally, many clinicians write electronic clinical notes using customized templates, resulting in notes with significant variability in structure. There is a need to understand better how clinicians review electronic notes and how note structure variability may impact clinicians' note-reviewing experiences. OBJECTIVE: This article aims to understand how physicians review electronic clinical notes and what impact section order has on note-reviewing patterns. MATERIALS AND METHODS: We conducted an experiment utilizing an electronic health record (EHR) system prototype containing four anonymized patient cases, each composed of nine progress notes that were presented with note sections organized in different orders to different subjects (i.e., Subjective, Objective, Assessment, and Plan, Assessment, Plan, Subjective, and Objective, Subjective, Assessment, Objective, and Plan, and Mixed). Participants, who were mid-level residents and fellows, reviewed the cases and provided a brief summary after reviewing each case. Time-related data were collected and analyzed using descriptive statistics. Surveys were administered and interviews regarding experiences reviewing notes were collected and analyzed qualitatively. RESULTS: Qualitatively, participants reported challenges related to reviewing electronic clinical notes. Experimentally, time spent reviewing notes varied based on the note section organization. Consistency in note section organization improved performance (e.g., less scrolling and searching) compared with Mixed section organization when reviewing progress notes. DISCUSSION: Clinicians face significant challenges reviewing electronic clinical notes. Our findings support minimizing extraneous information in notes, removing information that can be found in other parts of the EHR, and standardizing the display and order of note sections to improve clinicians' note review experience. CONCLUSION: Our findings support the need to improve EHR note design and presentation to support optimal note review patterns for clinicians.

Usability Evaluation of an Unstructured Clinical Document Query Tool for Researchers.

Natural Language Processing - Patient Information Extraction for Researchers (NLP-PIER) was developed for clinical researchers for self-service Natural Language Processing (NLP) queries with clinical notes. This study was to conduct a user-centered analysis with clinical researchers to gain insight into NLP-PIER's usability and to gain an understanding of the needs of clinical researchers when using an application for searching clinical notes. Clinical researcher participants (n=11) completed tasks using the system's two existing search interfaces and completed a set of surveys and an exit interview. Quantitative data including time on task, task completion rate, and survey responses were collected. Interviews were analyzed qualitatively. Survey scores, time on task and task completion proportions varied widely. Qualitative analysis indicated that participants found the system to be useful and usable in specific projects. This study identified several usability challenges and our findings will guide the improvement of NLP-PIER 's interfaces.

Using a Bedside Interactive Technology to Solicit and Record Pediatric Pain Reassessments: Parent and Nursing Perspectives on a Novel Workflow.

To measure the impact of a novel interactive inpatient pediatric pain management solution integrating our hospital's electronic health record system, the nurse communication phones, and the pharmacy dispensing system, we assessed parent and nurse perspectives on the tool's potential value, benefits, and challenges. A mixed-methods approach with survey instruments containing closed-ended and open-ended questions was administered to 30 parents and 59 nurses (66% and 23% response rate respectively). Overall, parents were more satisfied with the interactive technology experience (90%) compared to nurses (50%) with both indicating timely reassessments of pain being the most valuable feature. Qualitative analysis of nurses' responses yielded 6 themes for technology benefits and 12 for challenges. While patient-interactive technology solutions appear well-received particularly by parent end-users for pediatric hospital pain management, nurse training and interface improvements may result in higher efficacy, ultimately empowering patients/parents, promoting patient engagement and satisfaction.

Excess Cost of Cervical Cancer Screening Beyond Recommended Screening Ages or After Hysterectomy in a Single Institution.

OBJECTIVE: The aim of the study was to estimate the excess cost of guideline nonadherent cervical cancer screening in women beyond the recommended screening ages or posthysterectomy in a single healthcare system. MATERIALS AND METHODS: All Pap tests performed between September 1, 2012, and August 31, 2014, in women younger than 21 years, older than 65 years, or after hysterectomy, were coded as guideline adherent or nonadherent per the 2012 America Society of Colposcopy and Clinical Pathology guidelines. We assumed management of abnormal results per the 2013 America Society of Colposcopy and Clinical Pathology management guidelines. Costs were obtained from a literature review and Center for Medicare and Medicaid Services data and applied to nonadherent screening and subsequent diagnostic tests. RESULTS: During this period, 1,398 guideline nonadherent Pap tests were performed (257 in women <21 years, 536 in women >65 years, and 605 after hysterectomy), with 88 abnormal results: 35 (13.5%) in women younger than 21 years, 14 (2.6%) in women older than 65 years, and 39 (6.5%) in women after hysterectomy. The excess cost for initial screening, diagnostic tests, and follow-up was US $35,337 for 2 years in women younger than 21 years, US $54,378 for 5 years in women older than 65 years, and US $77,340 for 5 years in women after hysterectomy, resulting in a total excess cost of US $166,100 for 5 years. Of the 1,398 women who underwent guideline nonadherent screening, there were only 2 (0.1%) diagnoses of high-grade dysplasia (VaIN3). CONCLUSIONS: Guideline nonadherent cervical cancer screening in women beyond the recommended screening ages and posthysterectomy resulted in costs exceeding US $160,000 for screening, diagnostic tests, and follow-up with minimal improvement in detection of high-grade dysplasia.

Usability Evaluation of Electronic Health Record System around Clinical Notes Usage-An Ethnographic Study.

Background A substantial gap exists between current Electronic Health Record (EHR) usability and potential optimal usability. One of the fundamental reasons for this discrepancy is poor incorporation of a User-Centered Design (UCD) approach during the Graphical User Interface (GUI) development process. Objective To evaluate usability strengths and weaknesses of two widely implemented EHR GUIs for critical clinical notes usage tasks. Methods Twelve Internal Medicine resident physicians interacting with one of the two EHR systems (System-1 at Location-A and System-2 at Location-B) were observed by two usability evaluators employing an ethnographic approach. User comments and observer findings were analyzed for two critical tasks: (1) clinical notes entry and (2) related information-seeking tasks. Data were analyzed from two standpoints: (1) usability references categorized by usability evaluators as positive, negative, or equivocal and (2) usability impact of each feature measured through a 7-point severity rating scale. Findings were also validated by user responses to a post observation questionnaire. Results For clinical notes entry, System-1 surpassed System-2 with more positive (26% vs. 12%) than negative (12% vs. 34%) usability references. Greatest impact features on EHR usability (severity score pertaining to each feature) for clinical notes entry were: autopopulation (6), screen options (5.5), communication (5), copy pasting (4.5), error prevention (4.5), edit ability (4), and dictation and transcription (3.5). Both systems performed equally well on information-seeking tasks and features with greatest impacts on EHR usability were navigation for notes (7) and others (e.g., looking for ancillary data; 5.5). Ethnographic observations were supported by follow-up questionnaire responses. Conclusion This study provides usability-specific insights to inform future, improved, EHR interface that is better aligned with UCD approach.

Single Health System Adherence to 2012 Cervical Cancer Screening Guidelines at Extremes of Age and Posthysterectomy.

OBJECTIVE: To estimate the proportion of guideline nonadherent Pap tests in women aged younger than 21 years and older than 65 years and posthysterectomy in a single large health system. Secondary objectives were to describe temporal trends and patient and health care provider characteristics associated with screening in these groups. METHODS: A retrospective cross-sectional chart review was performed at Fairview Health Services and University of Minnesota Physicians. Reasons for testing and patient and health care provider information were collected. Tests were designated as indicated or nonindicated per the 2012 cervical cancer screening guidelines. Point estimates and descriptive statistics were calculated. Patient and health care provider characteristics were compared between indicated and nonindicated groups using χ and Wilcoxon rank-sum tests. RESULTS: A total of 3,920 Pap tests were performed between September 9, 2012, and August 31, 2014. A total of 257 (51%; 95% confidence interval [CI] 46.1-54.9%) of tests in the younger than 21 years group, 536 (40%; 95% CI 37.7-43.1%) in the older than 65 years group, and 605 (29%; 95% CI 27.1-31.0%) in the posthysterectomy group were not indicated. White race in the older than 65 years group was the only patient characteristic associated with receipt of a nonindicated Pap test (P=.007). Health care provider characteristics associated with nonindicated Pap tests varied by screening group. Temporal trends showed a decrease in the proportion of nonindicated tests in the younger than 21 years group but an increase in the posthysterectomy group. CONCLUSION: For women aged younger than 21 years and older than 65 years and posthysterectomy, 35% of Pap tests performed in our health system were not guideline-adherent. There were no patient or health care provider characteristics associated with guideline nonadherent screening across all groups.

Usability Evaluation of an EHR's Clinical Notes Interface from the Perspective of Attending and Resident Physicians: An Exploratory Study.

Usability gaps between current and future improved Electronic Health Record (EHR) system designs exist due to insufficient incorporation of User-Centered Design (UCD) principles during System Development Life Cycle (SDLC). Usability of a commercial, inpatient EHR clinical notes documentation interface was analyzed from standpoints of two provider groups employing two standardized patient cases. Both objective and subjective data were collected from attending (n = 6) and resident physicians (n = 8) through usability testing employing a mixed method approach. The study results suggested that (i) EHR usability and desirability is influenced by user characteristics, (ii) workloads associated with H&P and progress notes writing are perceived differently between two groups, (iii) repeated task performance improves user efficiency and (iv) user performance is correlated to their subjective system assessments. Understanding usability of clinical documentation interface from perspectives of two different user groups, provides interface designers with an opportunity to develop an EHR system centered on UCD principles.

Usability Evaluation of NLP-PIER: A Clinical Document Search Engine for Researchers.

NLP-PIER (Natural Language Processing - Patient Information Extraction for Research) is a self-service platform with a search engine for clinical researchers to perform natural language processing (NLP) queries using clinical notes. We conducted user-centered testing of NLP-PIER's usability to inform future design decisions. Quantitative and qualitative data were analyzed. Our findings will be used to improve the usability of NLP-PIER.

NLP-PIER: A Scalable Natural Language Processing, Indexing, and Searching Architecture for Clinical Notes.

Many design considerations must be addressed in order to provide researchers with full text and semantic search of unstructured healthcare data such as clinical notes and reports. Institutions looking at providing this functionality must also address the big data aspects of their unstructured corpora. Because these systems are complex and demand a non-trivial investment, there is an incentive to make the system capable of servicing future needs as well, further complicating the design. We present architectural best practices as lessons learned in the design and implementation NLP-PIER (Patient Information Extraction for Research), a scalable, extensible, and secure system for processing, indexing, and searching clinical notes at the University of Minnesota.

Usability Testing of Two Ambulatory EHR Navigators.

BACKGROUND: Despite widespread electronic health record (EHR) adoption, poor EHR system usability continues to be a significant barrier to effective system use for end users. One key to addressing usability problems is to employ user testing and user-centered design. OBJECTIVES: To understand if redesigning an EHR-based navigation tool with clinician input improved user performance and satisfaction. METHODS: A usability evaluation was conducted to compare two versions of a redesigned ambulatory navigator. Participants completed tasks for five patient cases using the navigators, while employing a think-aloud protocol. The tasks were based on Meaningful Use (MU) requirements. RESULTS: The version of navigator did not affect perceived workload, and time to complete tasks was longer in the redesigned navigator. A relatively small portion of navigator content was used to complete the MU-related tasks, though navigation patterns were highly variable across participants for both navigators. Preferences for EHR navigation structures appeared to be individualized. CONCLUSIONS: This study demonstrates the importance of EHR usability assessments to evaluate group and individual performance of different interfaces and preferences for each design.

A comparative observational study of inpatient clinical note-entry and reading/retrieval styles adopted by physicians.

OBJECTIVE: The objective of this study is to understand physicians' usage of inpatient notes by (i) ascertaining different clinical note-entry and reading/retrieval styles in two different and widely used Electronic Health Record (EHR) systems, (ii) extrapolating potential factors leading to adoption of various note-entry and reading/retrieval styles and (iii) determining the amount of time to task associated with documenting different types of clinical notes. METHODS: In order to answer "what" and "why" questions on physicians' adoption of certain-note-entry and reading/retrieval styles, an ethnographic study entailing Internal Medicine residents, with a mixed data analysis approach was performed. Participants were observed interacting with two different EHR systems in inpatient settings. Data was collected around the use and creation of History and Physical (H&P) notes, progress notes and discharge summaries. RESULTS: The highest variability in template styles was observed with progress notes and the least variability was within discharge summaries, while note-writing styles were most consistent for H&P notes. The first sections to be read in a H&P and progress note were the Chief Complaint and Assessment & Plan sections, respectively. The greatest note retrieval variability, with respect to the order of how note sections were reviewed, was observed with H&P and progress notes. Physician preference for adopting a certain reading/retrieval order appeared to be a function of what best fits their workflow while fulfilling the stimulus demands. The time spent entering H&P, discharge summaries and progress notes were similar in both EHRs. CONCLUSION: This research study unveils existing variability in clinical documentation processes and provides us with important information that could help in designing a next generation EHR Graphical User Interface (GUI) that is more congruent with physicians' mental models, task performance needs, and workflow requirements.

Characterizing Patient-Generated Clinical Data and Associated Implications for Electronic Health Records.

Patient-facing technologies are increasingly utilized for direct patient data entry for potential incorporation into the electronic health record. We analyzed patient-entered data during implementation of a patient-facing data entry technology using an online patient portal and clinic-based tablet computers at a University-based tertiary medical center clinic, including entries for past medical history, past surgical history, and social history. Entries were assessed for granularity, clinical accuracy, and the addition of novel information into the record. We found that over half of patient-generated diagnoses were duplicates of lesser or equal granularity compared to previous provider-entered diagnoses. Approximately one fifth of patient-generated diagnoses were found to meet the criteria for new, meaningful additions to the medical record. Our findings demonstrate that while patient-generated data provides important additional information, it may also present challenges including generating inaccurate or less granular information.