Strategies for case-based training with virtual patients: An experimental study of the impact of integrating mental model articulation and self-reflection.

Resilient system performance in high-stakes settings, which includes the ability to monitor, respond, anticipate, and learn, can be enhanced for trainees through simulation of realistic scenarios enhanced by augmented reality. Active learning strategies can enhance simulation-based training, particularly the mental model articulation principle where students are prompted to anticipate what will happen next and the reflection principle where students self-assess their performance compared to a gold standard expert model. In this paper, we compared simulation-based training for trauma care with and without active learning strategies during pauses in the simulated action for progressively deteriorating patients. The training was conducted online and real-time without a facilitator, with 42 medical students viewing training materials and then immediately taking an online quiz for three types of trauma cases: hemorrhage, airway obstruction, and tension pneumothorax. Participants were randomly assigned to either the experimental or control condition in a between-subjects design. We compared performance in the control and experimental conditions based on: A) the proportion of cues correctly recognized, B) the proportion of accurate diagnoses, C) the proportion of appropriate treatment interventions, and D) verbal briefing quality on a 1-5 scale. We found that the training intervention increased recognition of subtle cues critical for accurate diagnosis and appropriate treatment interventions; the training did not improve the accuracy of diagnoses or the quality of the verbal briefing. We conclude that incorporating active learning strategies in simulation-based training improved foundational capabilities in detecting subtle cues and intervening to rescue deteriorating patients that can increase the readiness for trainees to contribute to resilient system performance in the high-stakes setting of emergency care in hospitals.

Adapting Cognitive Task Analysis Methods for Use in a Large Sample Simulation Study of High-Risk Healthcare Events.

Cognitive task analysis (CTA) methods are traditionally used to conduct small-sample, in-depth studies. In this case study, CTA methods were adapted for a large multi-site study in which 102 anesthesiologists worked through four different high-fidelity simulated high-consequence incidents. Cognitive interviews were used to elicit decision processes following each simulated incident. In this paper, we highlight three practical challenges that arose: (1) standardizing the interview techniques for use across a large, distributed team of diverse backgrounds; (2) developing effective training; and (3) developing a strategy to analyze the resulting large amount of qualitative data. We reflect on how we addressed these challenges by increasing standardization, developing focused training, overcoming social norms that hindered interview effectiveness, and conducting a staged analysis. We share findings from a preliminary analysis that provides early validation of the strategy employed. Analysis of a subset of 64 interview transcripts using a decompositional analysis approach suggests that interviewers successfully elicited descriptions of decision processes that varied due to the different challenges presented by the four simulated incidents. A holistic analysis of the same 64 transcripts revealed individual differences in how anesthesiologists interpreted and managed the same case.

Decision-Making During High-Risk Events: A Systematic Literature Review.

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

Visual displays for cyber network defense.

Five computer network Defence displays (one Alphanumeric and four graphical displays: Radial Traffic Analyser, Bar Graph, Cube, and Treemap) were evaluated. Two experiments were conducted using different methodological procedures. Participants responded to questions that were structured to approximate various ways in which analysts might need to consider network traffic. Numerous significant effects were obtained and a fairly clear rank ordering of performance for the four graphical displays was obtained across experiments (from best to worst): Bar Graph, Cube, Radial Traffic Analyser, and Treemap. The results are interpreted from the perspective of ecological interface design: the quality of performance is directly related to the quality of semantic mapping between work domain, display, and human constraints. Factors that may have contributed to the poor performance for the Radial Traffic Analyser and Treemap displays are discussed. General implications for display and interface design are provided. Practitioner summary: Proposed displays for computer network Defence are evaluated; the results are interpreted from the perspective of ecological interface design. The associated design principles are applicable to all analogical graphical displays. Abbreviation: CND: cyber network defence; CSE: cognitive systems engineering; EID: ecological interface design; ICMP: internet control message protocol; IP: internet protocol; RTA: radial traffic analyzer; TCP: transmission control protocol; UDP: user datagram protocol.

Ecological Interface Design for Computer Network Defense.

OBJECTIVE: A prototype ecological interface for computer network defense (CND) was developed. BACKGROUND: Concerns about CND run high. Although there is a vast literature on CND, there is some indication that this research is not being translated into operational contexts. Part of the reason may be that CND has historically been treated as a strictly technical problem, rather than as a socio-technical problem. METHODS: The cognitive systems engineering (CSE)/ecological interface design (EID) framework was used in the analysis and design of the prototype interface. A brief overview of CSE/EID is provided. EID principles of design (i.e., direct perception, direct manipulation and visual momentum) are described and illustrated through concrete examples from the ecological interface. RESULTS: Key features of the ecological interface include (a) a wide variety of alternative visual displays, (b) controls that allow easy, dynamic reconfiguration of these displays, (c) visual highlighting of functionally related information across displays, (d) control mechanisms to selectively filter massive data sets, and (e) the capability for easy expansion. Cyber attacks from a well-known data set are illustrated through screen shots. CONCLUSION: CND support needs to be developed with a triadic focus (i.e., humans interacting with technology to accomplish work) if it is to be effective. Iterative design and formal evaluation is also required. The discipline of human factors has a long tradition of success on both counts; it is time that HF became fully involved in CND. APPLICATION: Direct application in supporting cyber analysts.

Evaluating a Modular Decision Support Application For Colorectal Cancer Screening.

BACKGROUND: There is a need for health information technology evaluation that goes beyond randomized controlled trials to include consideration of usability, cognition, feedback from representative users, and impact on efficiency, data quality, and clinical workflow. This article presents an evaluation illustrating one approach to this need using the Decision-Centered Design framework. OBJECTIVE: To evaluate, through a Decision-Centered Design framework, the ability of the Screening and Surveillance App to support primary care clinicians in tracking and managing colorectal cancer testing. METHODS: We leveraged two evaluation formats, online and in-person, to obtain feedback from a range primary care clinicians and obtain comparative data. Both the online and in-person evaluations used mock patient data to simulate challenging patient scenarios. Primary care clinicians responded to a series of colorectal cancer-related questions about each patient and made recommendations for screening. We collected data on performance, perceived workload, and usability. Key elements of Decision-Centered Design include evaluation in the context of realistic, challenging scenarios and measures designed to explore impact on cognitive performance. RESULTS: Comparison of means revealed increases in accuracy, efficiency, and usability and decreases in perceived mental effort and workload when using the Screening and Surveillance App. CONCLUSION: The results speak to the benefits of using the Decision-Centered Design approach in the analysis, design, and evaluation of Health Information Technology. Furthermore, the Screening and Surveillance App shows promise for filling decision support gaps in current electronic health records.

Designing Colorectal Cancer Screening Decision Support: A Cognitive Engineering Enterprise.

Adoption of clinical decision support has been limited. Important barriers include an emphasis on algorithmic approaches to decision support that do not align well with clinical work flow and human decision strategies, and the expense and challenge of developing, implementing, and refining decision support features in existing electronic health records (EHRs). We applied decision-centered design to create a modular software application to support physicians in managing and tracking colorectal cancer screening. Using decision-centered design facilitates a thorough understanding of cognitive support requirements from an end user perspective as a foundation for design. In this project, we used an iterative design process, including ethnographic observation and cognitive task analysis, to move from an initial design concept to a working modular software application called the Screening & Surveillance App. The beta version is tailored to work with the Veterans Health Administration's EHR Computerized Patient Record System (CPRS). Primary care providers using the beta version Screening & Surveillance App more accurately answered questions about patients and found relevant information more quickly compared to those using CPRS alone. Primary care providers also reported reduced mental effort and rated the Screening & Surveillance App positively for usability.