A Coding Framework for Usability Evaluation of Digital Health Technologies.

Several studies have reported low adherence and high resistance from clinicians to adopt digital health technologies into clinical practice, particularly the use of computer-based clinical decision support systems. Poor usability and lack of integration with the clinical workflow have been identified as primary issues. Few guidelines exist on how to analyze the collected data associated with the usability of digital health technologies. In this study, we aimed to develop a coding framework for the systematic evaluation of users' feedback generated during focus groups and interview sessions with clinicians, underpinned by fundamental usability principles and design components. This codebook also included a coding category to capture the user's clinical role associated with each specific piece of feedback, providing a better understanding of role-specific challenges and perspectives, as well as the level of shared understanding across the multiple clinical roles. Furthermore, a voting system was created to quantitatively inform modifications of the digital system based on usability data. As a use case, we applied this method to an electronic cognitive aid designed to improve coordination and communication in the cardiac operating room, showing that this framework is feasible and useful not only to better understand suboptimal usability aspects, but also to recommend relevant modifications in the design and development of the system from different perspectives, including clinical, technical, and usability teams. The framework described herein may be applied in other highly complex clinical settings, in which digital health systems may play an important role in improving patient care and enhancing patient safety.

Analysis of Dynamic Changes in Cognitive Workload During Cardiac Surgery Perfusionists' Interactions With the Cardiopulmonary Bypass Pump.

OBJECTIVE: This novel preliminary study sought to capture dynamic changes in heart rate variability (HRV) as a proxy for cognitive workload among perfusionists while operating the cardiopulmonary bypass (CPB) pump during real-life cardiac surgery. BACKGROUND: Estimations of operators' cognitive workload states in naturalistic settings have been derived using noninvasive psychophysiological measures. Effective CPB pump operation by perfusionists is critical in maintaining the patient's homeostasis during open-heart surgery. Investigation into dynamic cognitive workload fluctuations, and their relationship with performance, is lacking in the literature. METHOD: HRV and self-reported cognitive workload were collected from three Board-certified cardiac perfusionists (N = 23 cases). Five HRV components were analyzed in consecutive nonoverlapping 1-min windows from skin incision through sternal closure. Cases were annotated according to predetermined phases: prebypass, three phases during bypass, and postbypass. Values from all 1min time windows within each phase were averaged. RESULTS: Cognitive workload was at its highest during the time between initiating bypass and clamping the aorta (preclamp phase during bypass), and decreased over the course of the bypass period. CONCLUSION: We identified dynamic, temporal fluctuations in HRV among perfusionists during cardiac surgery corresponding to subjective reports of cognitive workload. Not only does cognitive workload differ for perfusionists during bypass compared with pre- and postbypass phases, but differences in HRV were also detected within the three bypass phases. APPLICATION: These preliminary findings suggest the preclamp phase of CPB pump interaction corresponds to higher cognitive workload, which may point to an area warranting further exploration using passive measurement.

Dissecting Cardiac Surgery: A Video-based Recall Protocol to Elucidate Team Cognitive Processes in the Operating Room.

OBJECTIVE: The aim of this study was to elucidate the cognitive processes involved in surgical procedures from the perspective of different team roles (surgeon, anesthesiologist, and perfusionist) and provide a comprehensive compilation of intraoperative cognitive processes. SUMMARY BACKGROUND DATA: Nontechnical skills play a crucial role in surgical team performance and understanding the cognitive processes underlying the intraoperative phase of surgery is essential to improve patient safety in the operating room (OR). METHODS: A mixed-methods approach encompassing semistructured interviews with 9 subject-matter experts. A cognitive task analysis was built upon a hierarchical segmentation of coronary artery bypass grafting procedures and a cued-recall protocol using video vignettes was used. RESULTS: A total of 137 unique surgical cognitive processes were identified, including 33 decision points, 23 critical communications, 43 pitfalls, and 38 strategies. Self-report cognitive workload varied substantially, depending on team role and surgical step. A web-based dashboard was developed, providing an integrated visualization of team cognitive processes in the OR that allows readers to intuitively interact with the study findings. CONCLUSIONS: This study advances the current body of knowledge by making explicit relevant cognitive processes involved during the intraoperative phase of cardiac surgery from the perspective of multiple OR team members. By displaying the research findings in an interactive dashboard, we provide trainees with new knowledge in an innovative fashion that could be used to enhance learning outcomes. In addition, the approach used in the present study can be used to deeply understand the cognitive factors underlying surgical adverse events and errors in the OR.

Digital Cognitive Aids to Support Adaptation of Surgical Processes to COVID-19 Protective Policies.

Surgical processes are rapidly being adapted to address the COVID-19 pandemic, with changes in procedures and responsibilities being made to protect both patients and medical teams. These process changes put new cognitive demands on the medical team and increase the likelihood of miscommunication, lapses in judgment, and medical errors. We describe two process model driven cognitive aids, referred to as the Narrative View and the Smart Checklist View, generated automatically from models of the processes. The immediate perceived utility of these cognitive aids is to support medical simulations, particularly when frequent adaptations are needed to quickly respond to changing operating room guidelines.

Process-Model-Driven Guidance to Reduce Surgical Procedure Errors: An Expert Opinion.

This paper explains how a detailed, precise surgical process model can help reduce errors by fostering better understanding, providing guidance during surgery, helping train newcomers, and by supporting process improvement. It describes the features that a process-modeling language should have in order to support the precise specification of such models.

Development of an Interactive Dashboard to Analyze Cognitive Workload of Surgical Teams During Complex Procedural Care.

In the surgical setting, team members constantly deal with a high-demand operative environment that requires simultaneously processing a large amount of information. In certain situations, high demands imposed by surgical tasks and other sources may exceed team member's cognitive capacity, leading to cognitive overload which may place patient safety at risk. In the present study, we describe a novel approach to integrate an objective measure of team member's cognitive load with procedural, behavioral and contextual data from real-life cardiac surgeries. We used heart rate variability analysis, capturing data simultaneously from multiple team members (surgeon, anesthesiologist and perfusionist) in a real-time and unobtrusive manner. Using audio-video recordings, behavioral coding and a hierarchical surgical process model, we integrated multiple data sources to create an interactive surgical dashboard, enabling the analysis of the cognitive load imposed by specific steps, substeps and/or tasks. The described approach enables us to detect cognitive load fluctuations over time, under specific conditions (e.g. emergencies, teaching) and in situations that are prone to errors. This in-depth understanding of the relationship between cognitive load, task demands and error occurrence is essential for the development of cognitive support systems to recognize and mitigate errors during complex surgical care in the operating room.

Intelligent Interruption Management System to Enhance Safety and Performance in Complex Surgical and Robotic Procedures.

Procedural flow disruptions secondary to interruptions play a key role in error occurrence during complex medical procedures, mainly because they increase mental workload among team members, negatively impacting team performance and patient safety. Since certain types of interruptions are unavoidable, and consequently the need for multitasking is inherent to complex procedural care, this field can benefit from an intelligent system capable of identifying in which moment flow interference is appropriate without generating disruptions. In the present study we describe a novel approach for the identification of tasks imposing low cognitive load and tasks that demand high cognitive effort during real-life cardiac surgeries. We used heart rate variability analysis as an objective measure of cognitive load, capturing data in a real-time and unobtrusive manner from multiple team members (surgeon, anesthesiologist and perfusionist) simultaneously. Using audio-video recordings, behavioral coding and a hierarchical surgical process model, we integrated multiple data sources to create an interactive surgical dashboard, enabling the identification of specific steps, substeps and tasks that impose low cognitive load. An interruption management system can use these low demand situations to guide the surgical team in terms of the appropriateness of flow interruptions. The described approach also enables us to detect cognitive load fluctuations over time, under specific conditions (e.g. emergencies) or in situations that are prone to errors. An in-depth understanding of the relationship between cognitive overload states, task demands, and error occurrence will drive the development of cognitive supporting systems that recognize and mitigate errors efficiently and proactively during high complex procedures.

Toward Improving Surgical Outcomes by Incorporating Cognitive Load Measurement into Process-Driven Guidance.

This paper summarizes the accomplishments and recent directions of our medical safety project. Our process-based approach uses a detailed, rigorously-defined, and carefully validated process model to provide a dynamically updated, context-aware and thus, "Smart" Checklist to help process performers understand and manage their pending tasks [7]. This paper focuses on support for teams of performers, working independently as well as in close collaboration, in stressful situations that are life critical. Our recent work has three main thrusts: provide effective real-time guidance for closely collaborating teams; develop and evaluate techniques for measuring cognitive load based on biometric observations and human surveys; and, using these measurements plus analysis and discrete event process simulation, predict cognitive load throughout the process model and propose process modifications to help performers better manage high cognitive load situations. This project is a collaboration among software engineers, surgical team members, human factors researchers, and medical equipment instrumentation experts. Experimental prototype capabilities are being built and evaluated based upon process models of two cardiovascular surgery processes, Aortic Valve Replacement (AVR) and Coronary Artery Bypass Grafting (CABG). In this paper we describe our approach for each of the three research thrusts by illustrating our work for heparinization, a common subprocess of both AVR and CABG. Heparinization is a high-risk error-prone procedure that involves complex team interactions and thus highlights the importance of this work for improving patient outcomes.

Process Driven Guidance for Complex Surgical Procedures.

Surgical team processes are known to be complex and error prone. This paper describes an approach that uses a detailed, validated model of a medical process to provide the clinicians who carry out that complex process with offline and online guidance to help reduce errors. Offline guidance is in the form of a hypertext document describing all the ways the process can be carried out. Online guidance is in the form of a context-sensitive and continually updated electronic "checklist" that lists next steps and needed resources, as well as completed steps. In earlier work, we focused on providing such guidance for single-clinician or single-team processes. This paper describes guiding the collaboration of multiple teams of clinicians through complex processes with significant concurrency, complicated exception handling, and precise and timely communication. We illustrate this approach by applying it to a highly complex, high risk subprocess of cardiac surgery.

Cognitive Support During High-Consequence Episodes of Care in Cardiovascular Surgery.

Despite significant efforts to reduce preventable adverse events in medical processes, such events continue to occur at unacceptable rates. This paper describes a computer science approach that uses formal process modeling to provide situationally aware monitoring and management support to medical professionals performing complex processes. These process models represent both normative and non-normative situations, and are validated by rigorous automated techniques such as model checking and fault tree analysis, in addition to careful review by experts. Context-aware Smart Checklists are then generated from the models, providing cognitive support during high-consequence surgical episodes. The approach is illustrated with a case study in cardiovascular surgery.