Engaging Multidisciplinary Clinical Users in the Design of an Artificial Intelligence-Powered Graphical User Interface for Intensive Care Unit Instability Decision Support.

BACKGROUND: Critical instability forecast and treatment can be optimized by artificial intelligence (AI)-enabled clinical decision support. It is important that the user-facing display of AI output facilitates clinical thinking and workflow for all disciplines involved in bedside care. OBJECTIVES: Our objective is to engage multidisciplinary users (physicians, nurse practitioners, physician assistants) in the development of a graphical user interface (GUI) to present an AI-derived risk score. METHODS: Intensive care unit (ICU) clinicians participated in focus groups seeking input on instability risk forecast presented in a prototype GUI. Two stratified rounds (three focus groups [only nurses, only providers, then combined]) were moderated by a focus group methodologist. After round 1, GUI design changes were made and presented in round 2. Focus groups were recorded, transcribed, and deidentified transcripts independently coded by three researchers. Codes were coalesced into emerging themes. RESULTS: Twenty-three ICU clinicians participated (11 nurses, 12 medical providers [3 mid-level and 9 physicians]). Six themes emerged: (1) analytics transparency, (2) graphical interpretability, (3) impact on practice, (4) value of trend synthesis of dynamic patient data, (5) decisional weight (weighing AI output during decision-making), and (6) display location (usability, concerns for patient/family GUI view). Nurses emphasized having GUI objective information to support communication and optimal GUI location. While providers emphasized need for recommendation interpretability and concern for impairing trainee critical thinking. All disciplines valued synthesized views of vital signs, interventions, and risk trends but were skeptical of placing decisional weight on AI output until proven trustworthy. CONCLUSION: Gaining input from all clinical users is important to consider when designing AI-derived GUIs. Results highlight that health care intelligent decisional support systems technologies need to be transparent on how they work, easy to read and interpret, cause little disruption to current workflow, as well as decisional support components need to be used as an adjunct to human decision-making.

Strategies for success in a nursing PhD program and beyond.

PURPOSE: The purpose of this article is to inform newly enrolled PhD students of program expectations, strategies for success, and next steps in the career of a nurse scientist. METHODS: We used empirical evidence and insights from the authors to describe strategies for success during a nursing PhD program and continued career development following graduation. FINDINGS: Measures of success included maintaining health, focus, integrity, and a supportive network, identifying mentors, pursuing new knowledge and advancing research to transform health outcomes. CONCLUSION: Nursing PhD programs help to shape future researchers and leaders. Choosing to obtain a PhD in nursing is an investment in oneself, the discipline, and the science. CLINICAL RELEVANCE: Nursing PhD programs offer opportunities to advance science, impact healthcare and health outcomes, and prepare for a variety of career opportunities. Informing newly enrolled PhD students may better prepare them for what lies ahead and facilitate student retention.

Engaging clinicians early during the development of a graphical user display of an intelligent alerting system at the bedside.

BACKGROUND: Artificial Intelligence (AI) is increasingly used to support bedside clinical decisions, but information must be presented in usable ways within workflow. Graphical User Interfaces (GUI) are front-facing presentations for communicating AI outputs, but clinicians are not routinely invited to participate in their design, hindering AI solution potential. PURPOSE: To inform early user-engaged design of a GUI prototype aimed at predicting future Cardiorespiratory Insufficiency (CRI) by exploring clinician methods for identifying at-risk patients, previous experience with implementing new technologies into clinical workflow, and user perspectives on GUI screen changes. METHODS: We conducted a qualitative focus group study to elicit iterative design feedback from clinical end-users on an early GUI prototype display. Five online focus group sessions were held, each moderated by an expert focus group methodologist. Iterative design changes were made sequentially, and the updated GUI display was presented to the next group of participants. RESULTS: 23 clinicians were recruited (14 nurses, 4 nurse practitioners, 5 physicians; median participant age ∼35 years; 60% female; median clinical experience 8 years). Five themes emerged from thematic content analysis: trend evolution, context (risk evolution relative to vital signs and interventions), evaluation/interpretation/explanation (sub theme: continuity of evaluation), clinician intuition, and clinical operations. Based on these themes, GUI display changes were made. For example, color and scale adjustments, integration of clinical information, and threshold personalization. CONCLUSIONS: Early user-engaged design was useful in adjusting GUI presentation of AI output. Next steps involve clinical testing and further design modification of the AI output to optimally facilitate clinician surveillance and decisions. Clinicians should be involved early and often in clinical decision support design to optimize efficacy of AI tools.

Accuracy of identifying hospital acquired venous thromboembolism by administrative coding: implications for big data and machine learning research.

Big data analytics research using heterogeneous electronic health record (EHR) data requires accurate identification of disease phenotype cases and controls. Overreliance on ground truth determination based on administrative data can lead to biased and inaccurate findings. Hospital-acquired venous thromboembolism (HA-VTE) is challenging to identify due to its temporal evolution and variable EHR documentation. To establish ground truth for machine learning modeling, we compared accuracy of HA-VTE diagnoses made by administrative coding to manual review of gold standard diagnostic test results. We performed retrospective analysis of EHR data on 3680 adult stepdown unit patients identifying HA-VTE. International Classification of Diseases, Ninth Revision (ICD-9-CM) codes for VTE were identified. 4544 radiology reports associated with VTE diagnostic tests were screened using terminology extraction and then manually reviewed by a clinical expert to confirm diagnosis. Of 415 cases with ICD-9-CM codes for VTE, 219 were identified with acute onset type codes. Test report review identified 158 new-onset HA-VTE cases. Only 40% of ICD-9-CM coded cases (n = 87) were confirmed by a positive diagnostic test report, leaving the majority of administratively coded cases unsubstantiated by confirmatory diagnostic test. Additionally, 45% of diagnostic test confirmed HA-VTE cases lacked corresponding ICD codes. ICD-9-CM coding missed diagnostic test-confirmed HA-VTE cases and inaccurately assigned cases without confirmed VTE, suggesting dependence on administrative coding leads to inaccurate HA-VTE phenotyping. Alternative methods to develop more sensitive and specific VTE phenotype solutions portable across EHR vendor data are needed to support case-finding in big-data analytics.

Intensive Care Unit Scoring Systems.

BACKGROUND: Illness severity scoring systems are commonly used in critical care. When applied to the populations for whom they were developed and validated, these tools can facilitate mortality prediction and risk stratification, optimize resource use, and improve patient outcomes. OBJECTIVE: To describe the characteristics and applications of the scoring systems most frequently applied to critically ill patients. METHODS: A literature search was performed using MEDLINE to identify original articles on intensive care unit scoring systems published in the English language from 1980 to 2020. Search terms associated with critical care scoring systems were used alone or in combination to find relevant publications. RESULTS: Two types of scoring systems are most frequently applied to critically ill patients: those that predict risk of in-hospital mortality at the time of intensive care unit admission (Acute Physiology and Chronic Health Evaluation, Simplified Acute Physiology Score, and Mortality Probability Models) and those that assess and characterize current degree of organ dysfunction (Multiple Organ Dysfunction Score, Sequential Organ Failure Assessment, and Logistic Organ Dysfunction System). This article details these systems' differing features and timing of use, score calculation, patient populations, and comparative performance data. CONCLUSION: Critical care nurses must be aware of the strengths, limitations, and specific characteristics of severity scoring systems commonly used in intensive care unit patients to effectively employ these tools in clinical practice and critically appraise research findings based on their use.

A developmental trajectory supporting the evaluation and achievement of competencies: Articulating the Mastery Rubric for the nurse practitioner (MR-NP) program curriculum.

BACKGROUND: Advanced practice registered nursing (APRN) competencies exist, but there is no structure supporting the operationalization of the competencies by APRN educators. The development of a Mastery Rubric (MR) for APRNs provides a developmental trajectory that supports educational institutions, educators, students, and APRNs. A MR describes the explicit knowledge, skills, and abilities as performed by the individual moving from novice (student) through graduation and into the APRN career. METHOD: A curriculum development tool, the Mastery Rubric (MR), was created to structure the curriculum and career of the nurse practitioner (NP), the MR-NP. Cognitive task analysis (CTA) yielded the first of the three required elements for any MR: a list of knowledge, skills, and abilities (KSAs) to be established through the curriculum. The European guild structure and Bloom's taxonomy of cognitive behaviors provided the second element of the MR, the specific developmental stages that are relevant for the curriculum. The Body of Work method of standard setting was used to create the third required element of the MR, performance level descriptors (PLDs) for each KSA at each of these stages. Although the CTA was informed by the competencies, it was still necessary to formally assess the alignment of competencies with the resulting KSAs; this was achieved via Degrees of Freedom Analysis (DoFA). Validity evidence was obtained from this Analysis and from the DoFA of the KSAs' alignment with principles of andragogy, and with learning outcomes assessment criteria. These analyses are the first time the national competencies for the NP have been evaluated in this manner. RESULTS: CTA of the 43 NP Competencies led to seven KSAs that support a developmental trajectory for instruction and documenting achievement towards independent performance on the competencies. The Competencies were objectively evaluable for the first time since their publication due to the psychometric validity attributes of the PLD-derived developmental trajectory. Three qualitatively distinct performance levels for the independent practitioner make the previously implicit developmental requirements of the competencies explicit for the first time. DISCUSSION: The MR-NP provides the first articulated and observable developmental trajectory for the NP competencies, during and beyond the formal curriculum. A focus on psychometric validity was brought to bear on how learners would demonstrate their development, and ultimately their achievement, of the competencies. The MR-NP goes beyond the competencies with trajectories and PLDs that can engage both learner and instructor in this developmental process throughout the career.

A call to alarms: Current state and future directions in the battle against alarm fatigue.

Research demonstrates that the majority of alarms derived from continuous bedside monitoring devices are non-actionable. This avalanche of unreliable alerts causes clinicians to experience sensory overload when attempting to sort real from false alarms, causing desensitization and alarm fatigue, which in turn leads to adverse events when true instability is neither recognized nor attended to despite the alarm. The scope of the problem of alarm fatigue is broad, and its contributing mechanisms are numerous. Current and future approaches to defining and reacting to actionable and non-actionable alarms are being developed and investigated, but challenges in impacting alarm modalities, sensitivity and specificity, and clinical activity in order to reduce alarm fatigue and adverse events remain. A multi-faceted approach involving clinicians, computer scientists, industry, and regulatory agencies is needed to battle alarm fatigue.