The use of fault reporting of medical equipment to identify latent design flaws.

BACKGROUND AND PURPOSE: Poor device design that fails to adequately account for user needs, cognition, and behavior is often responsible for use errors resulting in adverse events. This poor device design is also often latent, and could be responsible for "No Fault Found" (NFF) reporting, in which medical devices sent for repair by clinical users are found to be operating as intended. Unresolved NFF reports may contribute to incident under reporting, clinical user frustration, and biomedical engineering technologist inefficacy. This study uses human factors engineering methods to investigate the relationship between NFF reporting frequency and device usability. MATERIAL AND METHODS: An analysis of medical equipment maintenance data was conducted to identify devices with a high NFF reporting frequency. Subsequently, semi-structured interviews and heuristic evaluations were performed in order to identify potential usability issues. Finally, usability testing was conducted in order to validate that latent usability related design faults result in a higher frequency of NFF reporting. RESULTS: The analysis of medical equipment maintenance data identified six devices with a high NFF reporting frequency. Semi-structured interviews, heuristic evaluations and usability testing revealed that usability issues caused a significant portion of the NFF reports. Other factors suspected to contribute to increased NFF reporting include accessory issues, intermittent faults and environmental issues. Usability testing conducted on three of the devices revealed 23 latent usability related design faults. CONCLUSIONS: These findings demonstrate that latent usability related design faults manifest themselves as an increase in NFF reporting and that devices containing usability related design faults can be identified through an analysis of medical equipment maintenance data.

Modeling a medical environment: an ontology for integrated medical informatics design.

Modern medical environments have seen an increase in technological complexity and pressures of handling more patients with fewer resources, resulting in higher demands on medical practitioners. Medical informatics designers will have to focus on the problem of organizing medical information more effectively to enable practitioners to cope with these challenges. This article addresses this research problem for the particular area of medical problem solving in patient care. First, we describe a traditional modeling approach for medical reasoning used as a basis for developing some decision support systems. We argue these models may be faithful to what is known about biomedical knowledge, but they have limitations for human problem solving, especially in unanticipated situations. Second, we present an ontological framework, known as the abstraction hierarchy (Rasmussen, IEEE Trans. Man. Cybernetics 15 (1985) 234-243), for integrating patient representations that are faithful to existing biomedical knowledge and that are consistent with what is known about human problem solving. Through an example of a critical event in the operating room, we reveal how this framework can support medical problem solving in unanticipated situations. Third, we show how to use these representations as a frame of reference for mapping medical roles, responsibilities, sensors, and controls in an operating room context. Finally, we provide some insight for medical informatics designers in using this framework to design novel training programs and human-computer displays.

Patient safety, potential adverse drug events, and medical device design: a human factors engineering approach.

Adverse drug events are the single leading threat to patient safety. Human factors engineering has been repeatedly proposed, but largely untested, as the key to improving patient safety. The value of this approach was investigated in the context of a commercially available patient-controlled analgesia device that has been linked with several alleged patient injuries and deaths. Several reports have stated that errors in programming drug concentration were made during these adverse drug events. A simulation of the commercially available interface was compared experimentally with a simulated prototype of a new interface designed according to a human factors process. Professional nurses, averaging over 5 years of clinical experience with the commercially available interface and only minimal experience with the new interface, programmed both interfaces. The new interface eliminated drug concentration errors, whereas the simulated commercially available interface did not. Also, the new interface led to significantly fewer total errors and faster performance. These findings may have broad implications for the design, regulation, and procurement of biomedical devices, products, or systems that improve patient safety in clinical settings.

Making the most of ecological interface design: the role of individual differences.

As advanced control rooms for new process control plants are being designed, the question arises as to whether operators of the future need to have a particular set of cognitive characteristics to make the most of those advanced control rooms. This issue was investigated by examining the interaction between ecological interface design (EID) and individual differences in the context of a process control microworld. A number of potential predictors of performance were investigated, including: demographic data, type of interface, type of instruction, and data from two cognitive style tests. Eight linear regression analyses were conducted to determine which variables were the strongest predictors of performance. The results indicate that the strongest and most consistent predictor of performance was the interaction between a holist cognitive style score and an interface based on the principles of EID. That is, individuals who used an EID interface and who had high holist scores were the best performers. It seems that these individuals have the relational thinking ability that is required to exploit the value of the higher-order functional information provided by an EID interface. This empirical result has practical implications for operator selection.

There is more to monitoring a nuclear power plant than meets the eye.

A fundamental challenge in studying cognitive systems in context is how to move from the specific work setting studied to a more general understanding of distributed cognitive work and how to support it. We present a series of cognitive field studies that illustrate one response to this challenge. Our focus was on how nuclear power plant (NPP) operators monitor plant state during normal operating conditions. We studied operators at two NPPs with different control room interfaces. We identified strong consistencies with respect to factors that made monitoring difficult and the strategies that operators have developed to facilitate monitoring. We found that what makes monitoring difficult is not the need to identify subtle abnormal indications against a quiescent background, but rather the need to identify and pursue relevant findings against a noisy background. Operators devised proactive strategies to make important information more salient or reduce meaningless change, create new information, and off-load some cognitive processing onto the interface. These findings emphasize the active problem-solving nature of monitoring, and highlight the use of strategies for knowledge-driven monitoring and the proactive adaptation of the interface to support monitoring. Potential applications of this research include control room design for process control and alarm systems and user interfaces for complex systems.

A framework for epistemological analysis in empirical (laboratory and field) studies.

In their search for generalizable behavioral patterns and design principles, cognitive field researchers should reflect on the epistemological limitations of empirical studies. In this paper we describe a framework for epistemological analysis that can help serve this purpose and discuss its application to two prototypical cases of cognitive engineering research: laboratory experiments and field studies. The framework examines two, often implicit, processes in empirical research: the abstraction from empirical data and the substantiation of theoretical constructs and principles. By explicitly considering these two processes in several systematic steps, we can gain appreciation for the epistemological contribution of empirical studies to cognitive engineering research. The framework and its application also provide guidance to such important issues as generalizability of results and external validity. Possible applications of this research include providing guidance to researchers and practitioners in evaluating design principles or conducting field studies.

Revisiting the constraint attunement hypothesis: reply to Ericsson, Patel, and Kintsch (2000) and Simon and Gobet (2000).

This article is part of an exchange concerning the contributions of the constraint attunement hypothesis (CAH) to the understanding of expertise effects in memory recall. K. A. Ericsson, V. Patel, and W. Kintsch (2000) and H. A. Simon and F. Gobet (2000) claim that the CAH is not novel and that existing theories of this phenomenon do not have the limitations that were attributed to them. In this reply, the CAH is argued to be the only theory of expertise effects in memory recall to adopt the abstraction hierarchy as a theory of the environment, a feature that has important theoretical implications. Also, other theories focus on psychological mechanisms but have not satisfied the burden of scientific proof required of process theories. Progress can be made by integrating the complementary advantages of existing theories into a unified theory that acknowledges the equally important roles of the organism and the environment.

A participant-observer study of ergonomics in engineering design: how constraints drive design process.

Too often, ergonomics is relegated to being a "post-design" evaluation, leaving ergonomists little opportunity to make significant and important design changes. One way to start attacking this problem is to study the process of design and, in particular, ergonomics in design. This article describes the findings from a four-month long participant-observer study of the relationship between ergonomics and engineering design. The study was conducted in the context of a large, interdisciplinary project consisting of design of a control room for a nuclear power plant. It was observed that designers and ergonomists must negotiate through a changing web of constraints from many sources. The impact that these constraints had on the course of the design was documented. A model is developed based on the abstraction hierarchy (Rasmussen, 1985, IEEE Trans. Systems Man Cybernet. SMC-15, 234-243; 1990, Int. J. Ind. Ergon. 5, 5-16) which shows the interaction of conflicting goals as ergonomists and other designers attempt to solve a complex design problem. This model leads to several insights: (1) locally optimal ergonomic designs may not be globally optimal, (2) ergonomists can improve their solutions by understanding the goals of other designers, and (3) future tools to aid ergonomists must be compatible with the constraint-rich environments in which they work.

Applying human factors to the design of medical equipment: patient-controlled analgesia.

OBJECTIVE: Medical instruments commonly have poorly designed user interfaces that promote human errors with life-threatening consequences. The primary hypothesis of this study was that a specific user interface could be made safer and more efficient if redesigned using human factors techniques and principles. METHODS: The user interface of a commercially available patient-controlled analgesia (PCA) pump, the Abbott Lifecare 4100 PCA Plus II infuser, was evaluated using a cognitive task analysis of bench tests and field observations. Based on this analysis, the user interface was redesigned. Important elements of the new design include a dialog structure with fewer steps, a dialog overview showing the user's location in the programming sequence, better command feedback, easier error recovery, and clearer labels and messages. The changes were evaluated by comparing a computer prototype of the new interface with a computer simulation of the old one. Twelve student nurses performed six programming tasks with each interface. Task completion time, number of errors, and subjective mental workload were collected for each trial. RESULTS: The results showed significantly faster programming times (F(1,11) = 6.85, P < 0.025), lower mental workload ratings (chi2(1) = 4.45, p < 0.025, one-tailed), and fewer errors (chi2(1) = 3.33, p < 0.05, one-tailed) with the new interface. CONCLUSION: Adopting a human factors approach to redesigning the PCA interface led to significantly faster, easier, and more reliable performance. These findings have important implications for improving the design of other computer-based medical equipment.

Measures of operator performance in complex, dynamic microworlds: advancing the state of the art.

Microworld research provides a useful complement to field studies and highly controlled laboratory studies, aiming to strike a balance between representativeness and experimental control. Yet microworld research has associated methodological difficulties, particularly the problem of performance measurement. Researchers generally adopt a variety of measures to provide converging evidence concerning questions of interest. To confront problems with existing measures, this paper examines a series of objective measures used to characterize the performance of human operators in process control. These measures include novel, quantitative extensions to existing graphical analyses and new graphical representations. The measures are applied in the context of a 6-month longitudinal study using an interactive, thermal-hydraulic process control microworld (DURESS II). The following measures are discussed: steady-state time, action transition graph complexity, the path length in state space diagrams, the area under distance-to-goals graphs, divergence from the temperature goal line in mass inventory versus energy inventory graphs, and the proportion of control actions near the beginning of the trials represented by timelines. Two case studies emphasize the performance and strategy differences of individual operators across the battery of measures.

An ecological theory of expertise effects in memory recall.

Previous research has shown a significant correlation between domain expertise and memory recall performance after a very brief exposure time. Despite the large number of such studies, several findings in the literature have no satisfactory theoretical explanation. A novel theory based on an ecological approach is proposed to explain these results. This constraint attunement hypothesis provides a framework for identifying and representing the various levels of goal-relevant constraint in a domain. The theory predicts that there will be a memory expertise advantage in cases in which experts are attuned to the goal-relevant constraints in the material to be recalled and that the more constraint available, the greater the expertise advantage can be. The theory explains a number of diverse empirical findings in the literature in a coherent, unique, and parsimonious fashion and suggests a number of promising issues for future research.

Towards viable, useful and usable human factors design guidance.

This paper investigates the factors relevant to producing effective human factors design guidance, using the Engineering Data Compendium (EDC) as a research vehicle. A series of three exploratory experiments focusing on the factors that affect the usability, usefulness and viability of human factors handbooks was conducted. The results of these studies were interpreted in the context of the process by which the EDC was developed, leading to the following recommendations: (a) human factors guidance should be organized in a manner that is stepped in context; (b) human factors guidance should be based on an explicit requirements analysis; (c) the calibration of designers' perceptions of the cost of obtaining human factors information must be improved; (d) organizational policies must be changed to induce more effective information search behaviour.

Reconstructive remembering of the scientific literature.

In this paper we investigate the role of reconstructive memory in citation errors that occur in the scientific literature. We focus on the case of de Groot's (1946) studies of the memory for chess positions by chess experts. Previous work has shown that this research is very often cited incorrectly. In Experiment 1 we show that free recall of this work by research psychologists replicates most of the errors found in the published literature. Experiment 2 shows that undergraduates reading a correct account of the de Groot study also make the same set of errors in recall. We interpret these findings as showing that consistent errors in secondary accounts of experimental findings are frequently reconstructive memory errors due to source confusion and schema-based processes. Analysis of a number of other examples of scientific literature that have been frequently cited incorrectly add additional support to the reconstructive account. We conclude that scientists should be aware of the tendency of reconstructive memory errors to cause violations of the scientific norm of accurate reporting of the scientific literature.

Memory recall in a process control system: a measure of expertise and display effectiveness.

Previous research has shown that memory-recall performance is correlated with domain expertise. In this study, a process control system was selected as a vehicle for conducting research on memory recall. The primary purposes of the present work were to determine if the classic expertise effects originally obtained in chess generalize to this novel domain and to evaluate the validity of memory recall as a measure of display effectiveness. Experts and novices viewed dynamic event sequences showing the behavior of a thermal-hydraulic system with two different displays, one that only contained information about the physical components in the system (P) and another that also contained information about higher order functional variables (P+F). There were three types of trials: normal, where the system was operating correctly; fault, where a single fault was introduced; and random, where the system's behavior did not obey physical laws. On each trial, subjects were asked to recall the final state of the system and to diagnose the system state. The P+F display resulted in superior diagnosis performance compared with the P display. With regard to memory, there was some evidence of an interaction between trial type and expertise, with experts outperforming novices but primarily on meaningful trials. In addition, memory for the subset of variables most critical to diagnosis was better with the P+F display than with the P display, thereby indicating that memory recall can be a sensitive measure of display effectiveness. The results also clarify a theoretical problem that has existed for some time in the literature, namely, the conditions under which expertise advantages are to be expected in memory-recall tasks. Collectively, these findings point to the potential benefits of adopting an applied context as a test bed for basic research issues.