A novel use of an artificially intelligent Chatbot and a live, synchronous virtual question-and answer session for fellowship recruitment.

INTRODUCTION: Academic departments universally communicate information about their programs using static websites. In addition to websites, some programs have even ventured out into social media (SM). These bidirectional forms of SM interaction show great promise; even hosting a live Question and Answer (Q&A) session has the potential for program branding. Artificial Intelligence (AI) usage in the form of a chatbot has expanded on websites and in SM. The potential use of chatbots, for the purposes of trainee recruitment, is novel and underutilized. With this pilot study, we aimed to answer the question; can the use of an Artificially Intelligent Chatbot and a Virtual Question-and-Answer Session aid in recruitment in a Post-COVID-19 era? METHODS: We held three structured Question-and-Answer Sessions over a period of 2 weeks. This preliminary study was performed after completion of the three Q&A sessions, in March-May, 2021. All 258 applicants to the pain fellowship program were invited via email to participate in the survey after attending one of the Q&A sessions. A 16-item survey assessing participants' perception of the chatbot was administered. RESULTS: Forty-eight pain fellowship applicants completed the survey, for an average response rate of 18.6%. In all, 35 (73%) of survey respondents had used the website chatbot, and 84% indicated that it had found them the information they were seeking. CONCLUSION: We employed an artificially intelligent chatbot on the department website to engage in a bidirectional exchange with users to adapt to changes brought on by the pandemic. SM engagement via chatbot and Q&A sessions can leave a favorable impression and improve the perception of a program.

Effect of Remote Cardiac Monitoring System Design on Response Time to Critical Arrhythmias.

INTRODUCTION: In many hospitals across the country, electrocardiograms of multiple at-risk patients are monitored remotely by telemetry monitor watchers in a central location. However, there is limited evidence regarding best practices for designing these cardiac monitoring systems to ensure prompt detection and response to life-threatening events. To identify factors that may affect monitoring efficiency, we simulated critical arrhythmias in inpatient units with different monitoring systems and compared their efficiency in communicating the arrhythmias to a first responder. METHODS: This was a multicenter cross-sectional in situ simulation study. Simulation participants were monitor watchers and first responders (usually nurses) in 2 inpatient units in each of 3 hospitals. Manipulated variables included: (1) number of communication nodes between monitor watchers and first responders; (2) central monitoring station location-on or off the patient care unit; (3) monitor watchers' workload; (4) nurses' workload; and (5) participants' experience. RESULTS: We performed 62 arrhythmia simulations to measure response times of monitor watchers and 128 arrhythmia simulations to measure response times in patient care units. We found that systems in which an intermediary between monitor watchers and nurses communicated critical events had faster response times to simulated arrhythmias than systems in which monitor watchers communicated directly with nurses. Responses were also faster in units colocated with central monitoring stations than in those located remotely. As the perceived workload of nurses increased, response latency also increased. Experience did not affect response times. CONCLUSIONS: Although limited in our ability to isolate the effects of these factors from extraneous factors on central monitoring system efficiency, our study provides a roadmap for using in situ arrhythmia simulations to assess and improve monitoring performance.

Iterative heuristic design of temporal graphic displays with clinical domain experts.

Conventional electronic health record information displays are not optimized for efficient information processing. Graphical displays that integrate patient information can improve information processing, especially in data-rich environments such as critical care. We propose an adaptable and reusable approach to patient information display with modular graphical components (widgets). We had two study objectives. First, reduce numerous widget prototype alternatives to preferred designs. Second, derive widget design feature recommendations. Using iterative human-centered design methods, we interviewed experts to hone design features of widgets displaying frequently measured data elements, e.g., heart rate, for acute care patient monitoring and real-time clinical decision-making. Participant responses to design queries were coded to calculate feature-set agreement, average prototype score, and prototype agreement. Two iterative interview cycles covering 64 design queries and 86 prototypes were needed to reach consensus on six feature sets. Interviewers agreed that line graphs with a smoothed or averaged trendline, 24-h timeframe, and gradient coloring for urgency were useful and informative features. Moreover, users agreed that widgets should include key functions: (1) adjustable reference ranges, (2) expandable timeframes, and (3) access to details on demand. Participants stated graphical widgets would be used to identify correlating patterns and compare abnormal measures across related data elements at a specific time. Combining theoretical principles and validated design methods was an effective and reproducible approach to designing widgets for healthcare displays. The findings suggest our widget design features and recommendations match critical care clinician expectations for graphical information display of continuous and frequently updated patient data.

Impact of integrated graphical display on expert and novice diagnostic performance in critical care.

OBJECTIVE: To determine the impact of a graphical information display on diagnosing circulatory shock. MATERIALS AND METHODS: This was an experimental study comparing integrated and conventional information displays. Participants were intensivists or critical care fellows (experts) and first-year medical residents (novices). RESULTS: The integrated display was associated with higher performance (87% vs 82%; P < .001), less time (2.9 vs 3.5 min; P = .008), and more accurate etiology (67% vs 54%; P = .048) compared to the conventional display. When stratified by experience, novice physicians using the integrated display had higher performance (86% vs 69%; P < .001), less time (2.9 vs 3.7 min; P = .03), and more accurate etiology (65% vs 42%; P = .02); expert physicians using the integrated display had nonsignificantly improved performance (87% vs 82%; P = .09), time (2.9 vs 3.3; P = .28), and etiology (69% vs 67%; P = .81). DISCUSSION: The integrated display appeared to support efficient information processing, which resulted in more rapid and accurate circulatory shock diagnosis. Evidence more strongly supported a difference for novices, suggesting that graphical displays may help reduce expert-novice performance gaps.

Organizing Audible Alarm Sounds in the Hospital: A Card-Sorting Study.

In hospitals, clinicians are presented with varied and disorganized alarm sounds from disparate devices. While there has been attention to reducing inactionable alarms to address alarm overload, little effort has focused on organizing, simplifying, or improving the informativeness of alarms. We sought to elicit nurses' tacit interpretation of alarm events to create an organizational structure to inform the design of advanced alarm sounds or integrated alert systems. We used open card sorting to evaluate nurses' perception of the relatedness of different alarm events. Seventy hospital nurses sorted 89 alarm events into groups they believed could or should be indicated by the same sound. We conducted factor analysis on a similarity matrix of frequency of alarm event pairings to interpret how strongly alarm events loaded on different alarm groups (factors). We interpreted participants' grouping rationale from their group labels and comments. Urgency of response was the most common grouping rationale. Participants also grouped: 1) monitoring-related events, 2) device-related events, and 3) events related to calls and patients. Our findings support standardization and integration of alarm sounds across devices toward a simpler and more informative hospital alarm environment.

Critical care information display approaches and design frameworks: A systematic review and meta-analysis.

OBJECTIVE: To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians. METHODS: We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author. RESULTS: Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible. CONCLUSION: Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.

Novel displays of patient information in critical care settings: a systematic review.

OBJECTIVE: Clinician information overload is prevalent in critical care settings. Improved visualization of patient information may help clinicians cope with information overload, increase efficiency, and improve quality. We compared the effect of information display interventions with usual care on patient care outcomes. MATERIALS AND METHODS: We conducted a systematic review including experimental and quasi-experimental studies of information display interventions conducted in critical care and anesthesiology settings. Citations from January 1990 to June 2018 were searched in PubMed and IEEE Xplore. Reviewers worked independently to screen articles, evaluate quality, and abstract primary outcomes and display features. RESULTS: Of 6742 studies identified, 22 studies evaluating 17 information displays met the study inclusion criteria. Information display categories included comprehensive integrated displays (3 displays), multipatient dashboards (7 displays), physiologic and laboratory monitoring (5 displays), and expert systems (2 displays). Significant improvement on primary outcomes over usual care was reported in 12 studies for 9 unique displays. Improvement was found mostly with comprehensive integrated displays (4 of 6 studies) and multipatient dashboards (5 of 7 studies). Only 1 of 5 randomized controlled trials had a positive effect in the primary outcome. CONCLUSION: We found weak evidence suggesting comprehensive integrated displays improve provider efficiency and process outcomes, and multipatient dashboards improve compliance with care protocols and patient outcomes. Randomized controlled trials of physiologic and laboratory monitoring displays did not show improvement in primary outcomes, despite positive results in simulated settings. Important research translation gaps from laboratory to actual critical care settings exist.

Critical care information display approaches and design frameworks: A systematic review and meta-analysis.

OBJECTIVE: To systematically review original user evaluations of patient information displays relevant to critical care and understand the impact of design frameworks and information presentation approaches on decision-making, efficiency, workload, and preferences of clinicians. METHODS: We included studies that evaluated information displays designed to support real-time care decisions in critical care or anesthesiology using simulated tasks. We searched PubMed and IEEExplore from 1/1/1990 to 6/30/2018. The search strategy was developed iteratively with calibration against known references. Inclusion screening was completed independently by two authors. Extraction of display features, design processes, and evaluation method was completed by one and verified by a second author. RESULTS: Fifty-six manuscripts evaluating 32 critical care and 22 anesthesia displays were included. Primary outcome metrics included clinician accuracy and efficiency in recognizing, diagnosing, and treating problems. Implementing user-centered design (UCD) processes, especially iterative evaluation and redesign, resulted in positive impact in outcomes such as accuracy and efficiency. Innovative display approaches that led to improved human-system performance in critical care included: (1) improving the integration and organization of information, (2) improving the representation of trend information, and (3) implementing graphical approaches to make relationships between data visible. CONCLUSION: Our review affirms the value of key principles of UCD. Improved information presentation can facilitate faster information interpretation and more accurate diagnoses and treatment. Improvements to information organization and support for rapid interpretation of time-based relationships between related quantitative data is warranted. Designers and developers are encouraged to involve users in formal iterative design and evaluation activities in the design of electronic health records (EHRs), clinical informatics applications, and clinical devices.

Development and usability testing of an audit and feedback tool for anesthesiologists.

BACKGROUND: We describe the creation and evaluation of a personal audit and feedback (A&F) tool for anesthesiologists. METHODS: A survey aimed at capturing barriers for personal improvement efforts and feedback preferences was administered to attending anesthesiologists. The results informed the design and implementation of 4 dashboards that display information on individual practice characteristics as well as comparative performance on several quality metrics. The dashboards' usability was then tested using the human-centered design framework. RESULTS: Anesthesiologists listed lack of information on current practice as the main barrier for improvement. Regarding usability, participants gave the dashboards an average score of 3.8 (scale 1-5) on consistency, learnability, and information organization, and performed the assigned tasks well, with an average score of 89% (range 79-100%). CONCLUSIONS: We describe the design, implementation, and usability testing of an innovative tool that utilizes data derived from the electronic health record (EHR) system to provide A&F to anesthesiology providers.

Patient information organization in the intensive care setting: expert knowledge elicitation with card sorting methods.

Introduction: Many electronic health records fail to support information uptake because they impose low-level information organization tasks on users. Clinical concept-oriented views have shown information processing improvements, but the specifics of this organization for critical care are unclear. Objective: To determine high-level cognitive processes and patient information organization schema in critical care. Methods: We conducted an open card sort of 29 patient data elements and a modified Delphi card sort of 65 patient data elements. Study participants were 39 clinicians with varied critical care training and experience. We analyzed the open sort with a hierarchical cluster analysis (HCA) and factor analysis (FA). The Delphi sort was split into three initiating groups that resulted in three unique solutions. We compared results between open sort analyses (HCA and FA), between card sorting exercises (open and Delphi), and across the Delphi solutions. Results: Between the HCA and FA, we observed common constructs including cardiovascular and hemodynamics, infectious disease, medications, neurology, patient overview, respiratory, and vital signs. The more comprehensive Delphi sort solutions also included gastrointestinal, renal, and imaging constructs. Conclusions: We identified primarily system-based groupings (e.g., cardiovascular, respiratory). Source-based (e.g., medications, laboratory) groups became apparent when participants were asked to sort a longer list of concepts. These results suggest a hybrid approach to information organization, which may combine systems, source, or problem-based groupings, best supports clinicians' mental models. These results can contribute to the design of information displays to better support clinicians' access and interpretation of information for critical care decisions.

Triangulating Methodologies from Software, Medicine and Human Factors Industries to Measure Usability and Clinical Efficacy of Medication Data Visualization in an Electronic Health Record System.

Within the last decade, use of Electronic Health Record (EHR) systems has become intimately integrated into healthcare practice in the United States. However, large gaps remain in the study of clinical usability and require rigorous and innovative approaches for testing usability principles. In this study, validated tools from the core functions that EHRs serve-software, medicine and human factors-were combined to holistically understand and objectively measure usability of medication data displays. The first phase of this study included 132 medical trainee participants who were randomized to one of two simulated EHR environments with either a medication list or a medication timeline visualization. Within these environments human-computer interaction metrics, clinical reasoning and situation awareness tests, and usability surveys captured their multi-faceted interactions. Results showed no statistically significant differences in the two displays from software and situation awareness perspectives, though there were higher statistically significant usability scores of the medication timeline (intervention) as compared to the medication list (control). This first phase of a novel design in triangulating methodologies revealed several limitations from which future experiments will be adjusted with hopes of yielding further insight and a generalizable testing platform for evolving EHR interfaces.

Toward Designing Information Display to Support Critical Care. A Qualitative Contextual Evaluation and Visioning Effort.

OBJECTIVES: Electronic health information overload makes it difficult for providers to quickly find and interpret information to support care decisions. The purpose of this study was to better understand how clinicians use information in critical care to support the design of improved presentation of electronic health information. METHODS: We conducted a contextual analysis and visioning project. We used an eye-tracker to record 20 clinicians' information use activities in critical care settings. We played video recordings back to clinicians in retrospective cued interviews and queried: 1) context and goals of information use, 2) impacts of current display design on use, and 3) processes related to information use. We analyzed interview transcripts using grounded theory-based content analysis techniques and identified emerging themes. From these, we conducted a visioning activity with a team of subject matter experts and identified key areas for focus of design and research for future display designs. RESULTS: Analyses revealed four unique critical care information use activities including new patient assessment, known patient status review, specific directed information seeking, and review and prioritization of multiple patients. Emerging themes were primarily related to a need for better representation of dynamic data such as vital signs and laboratory results, usability issues associated with reducing cognitive load and supporting efficient interaction, and processes for managing information. Visions for the future included designs that: 1) provide rapid access to new information, 2) organize by systems or problems as well as by current versus historical patient data, and 3) apply intelligence toward detecting and representing change and urgency. CONCLUSIONS: The results from this study can be used to guide the design of future acute care electronic health information display. Additional research and collaboration is needed to refine and implement intelligent graphical user interfaces to improve clinical information organization and prioritization to support care decisions.

Operating Room-to-ICU Patient Handovers: A Multidisciplinary Human-Centered Design Approach.

BACKGROUND: Patient handovers (handoffs) following surgery have often been characterized by poor teamwork, unclear procedures, unstructured processes, and distractions. A study was conducted to apply a human-centered approach to the redesign of operating room (OR)-to-ICU patient handovers in a broad surgical ICU (SICU) population. This approach entailed (1) the study of existing practices, (2) the redesign of the handover on the basis of the input of hand over participants and evidence in the medical literature, and (3) the study of the effects of this change on processes and communication. METHODS: The Durham [North Carolina] Veterans Affairs Medical Center SICU is an 11-bed mixed surgical specialty unit. To understand the existing process for receiving postoperative patients in the SICU, ethnographic methods-a series of observations, surveys, interviews, and focus groups-were used. The handover process was redesigned to better address providers' work flow, information needs, and expectations, as well as concerns identified in the literature. RESULTS: Technical and communication flaws were uncovered, and the handover was redesigned to address them. For the 49 preintervention and 49 postintervention handovers, the information transfer score and number of interruptions were not significantly different. However, staff workload and team behaviors scores improved significantly, while the hand over duration was not prolonged by the new process. Handover participants were also significantly more satisfied with the new handover method. CONCLUSIONS: An HCD approach led to improvements in the patient handover process from the OR to the ICU in a mixed adult surgical population. Although the specific handover process would unlikely be optimal in another clinical setting if replicated exactly, the HCD foundation behind the redesign process is widely applicable.

In situ simulated cardiac arrest exercises to detect system vulnerabilities.

INTRODUCTION: Sudden cardiac arrest is the leading cause of death in the United States. Despite new therapies, progress in this area has been slow, and outcomes remain poor even in the hospital setting, where providers, drugs, and devices are readily available. This is partly attributed to the quality of resuscitation, which is an important determinant of survival for patients who experience cardiac arrest. Systems problems, such as deficiencies in the physical space or equipment design, hospital-level policies, work culture, and poor leadership and teamwork, are now known to contribute significantly to the quality of resuscitation provided. METHODS: We describe an in situ simulation-based quality improvement program that was designed to continuously monitor the cardiac arrest response process for hazards and defects and to detect opportunities for system optimization. RESULTS: A total of 72 simulated unannounced cardiac arrest exercises were conducted between October 2010 and September 2013 at various locations throughout our medical center and at different times of the day. We detected several environmental, human-machine interface, culture, and policy hazards and defects. We used the Systems Engineering Initiative for Patient Safety (SEIPS) model to understand the structure, processes, and outcomes related to the hospital's emergency response system. Multidisciplinary solutions were crafted for each of the hazards detected, and the simulation program was used to iteratively test the redesigned processes before implementation in real clinical settings. CONCLUSIONS: We describe an ongoing program that uses in situ simulation to identify and mitigate latent hazards and defects in the hospital emergency response system. The SEIPS model provides a framework for describing and analyzing the structure, processes, and outcomes related to these events.

Patient load effects on response time to critical arrhythmias in cardiac telemetry: a randomized trial.

OBJECTIVES: Remotely monitored patients may be at risk for a delayed response to critical arrhythmias if the telemetry watchers who monitor them are subject to an excessive patient load. There are no guidelines or studies regarding the appropriate number of patients that a single watcher may safely and effectively monitor. Our objective was to determine the impact of increasing the number of patients monitored on response time to simulated cardiac arrest. DESIGN: Randomized trial. SETTING: Laboratory-based experiment. SUBJECTS: Forty-two remote telemetry technicians and nurses from cardiac units. INTERVENTIONS: Number of patients monitored in a simulation of cardiac telemetry monitoring work. MEASUREMENTS AND MAIN RESULTS: We carried out a study to compare response times to ventricular fibrillation across five patient loads: 16, 24, 32, 40, and 48 patients. The simulation replicated the work of telemetry watchers using a combination of real recorded patient electrocardiogram signals and a simulated patient experiencing ventricular fibrillation. Study participants were assigned to one of the five patient loads and completed a 4-hour monitoring session, during which they performed tasks-including event documentation and phone calls to report events-similar to real monitoring work. When the simulated patient sustained ventricular fibrillation, the time required to report this arrhythmia was recorded. As patient loads increased, there was a statistically significant increase in response times to the ventricular fibrillation. In addition, frequency of failure to meet a response time goal of less than 20 seconds was significantly higher in the 48-patient condition than in all other conditions. Task performance decreased as patient load increased. CONCLUSIONS: As participants monitored more patients in a laboratory setting, their performance with respect to recognizing critical and noncritical events declined. This study has implications for the design of remote telemetry work and other patient monitoring tasks in critical and intermediate care units.

Standardized assessment for evaluation of team skills: validity and feasibility.

INTRODUCTION: The authors developed a Standardized Assessment for Evaluation of Team Skills (SAFE-TeamS) in which actors portray health care team members in simulated challenging teamwork scenarios. Participants are scored on scenario-specific ideal behaviors associated with assistance, conflict resolution, communication, assertion, and situation assessment. This research sought to provide evidence of the validity and feasibility of SAFE-TeamS as a tool to support the advancement of science related to team skills training. METHODS: Thirty-eight medical and nursing students were assessed using SAFE-TeamS before and after team skills training. The SAFE-TeamS pretraining and posttraining scores were compared, and participants were surveyed. Generalizability analysis was used to estimate the variance in scores associated with the following: examinee, scenario, rater, pretraining/posttraining, examinee type, rater type (actor-live vs. external rater-videotape), actor team, and scenario order. RESULTS: The SAFE-TeamS scores reflected improvement after training and were sensitive to individual differences. Score variance due to rater was low. Variance due to scenario was moderate. Estimates of relative reliability for 2 raters and 8 scenarios ranged from 0.6 to 0.7. With fixed scenarios and raters, 2 raters and 2 scenarios, reliability is greater than 0.8. Raters believed SAFE-TeamS assessed relevant team skills. Examinees' responses were mixed. CONCLUSIONS: The SAFE-TeamS was sensitive to individual differences and team skill training, providing evidence for validity. It is not clear whether different scenarios measure different skills and whether the scenarios cover the necessary breadth of skills. Use of multiple scenarios will support assessment across a broader range of skills. Future research is required to determine whether assessments using SAFE-TeamS will translate to performance in clinical practice.

A Cognitive Modeling Approach to Decision Support Tool Design for Anesthesia Provider Crisis Management.

Prior research has revealed existing operating room (OR) patient monitors to provide limited support for prompt and accurate decision making by anesthesia providers during crises. Decision support tools (DSTs) developed for this purpose typically alert the anesthesia provider to existence of a problem but do not recommend a treatment plan. There is a need for a human-centered approach to the design and development of a crisis management DST. A hierarchical task analysis was conducted to identify anesthesia provider procedures in detecting, diagnosing, and treating a critical incident and a cognitive task analysis to elicit goals, decisions, and information requirements. This information was coded in a computational cognitive model using GOMS (Goals, Operators, Methods, Selection rules) Language. An OR monitor interface was prototyped to present output from the cognitive model following ecological interface design principles. A preliminary assessment of the DST was performed with anesthesiology and usability experts. The anesthesiologists indicated they would use the tool in the perioperative environment and would recommend its use by junior anesthesia providers. Future research will focus on formal validation of the DST design approach and comparison of tool output to actual anesthesia provider decisions in real or simulated crises.