Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet.

Accident analysis methods are used to model the multifactorial cause of adverse incidents. Methods such as AcciMap, STAMP-CAST and recently AcciNet, are systemic approaches that support the identification of safety interventions across sociotechnical system levels. Despite their growing popularity, little is known about how reliable systems-based methods are when used to describe, model and classify contributory factors and relationships. Here, we conducted an intra-rater and inter-rater reliability assessment of AcciMap, STAMP-CAST and AcciNet using the Signal Detection Theory (SDT) paradigm. A total of 180 hours' worth of analyses across 360 comparisons were performed by 30 expert analysts. Findings revealed that all three methods produced a weak to moderate positive correlation coefficient, however the inter-rater reliability of STAMP-CAST was significantly higher compared to AcciMap and AcciNet. No statistically significant or practically meaningful differences were found between methods in the overall intra-rater reliability analyses. Implications and future research directions are discussed.

Practitioners who undertake accident analysis within their organisations should consider the use of STAMP-CAST due to the significantly higher inter-rater reliability findings obtained in this study compared to AcciMap and AcciNet, particularly if they tend to work alone and/or part of relatively small teams.

Towards a unified model of accident causation: refining and validating the systems thinking safety tenets.

The systems thinking tenets were developed based on a synthesis of contemporary accident causation theory, models and approaches and encapsulate 15 features of complex systems that interact to create both safety and adverse events. Whilst initial testing provided supportive evidence, the tenets have not yet been subject to formal validation. This article presents the findings from a three-round Delphi study undertaken to refine and validate the tenets and assess their suitability for inclusion in a unified model of accident causation. Participants with expertise in accident causation and systems thinking provided feedback on the tenets and associated definitions until an acceptable level of consensus was achieved. The results reduced the original 15 tenets to 14 and 10 were identified as important to include in unified model of accident causation. The refined systems thinking tenets are presented along with future research directions designed to facilitate their use in safety practice.Practitioner summary: This article presents a refined and validated set of systems thinking tenets which describe features of complex systems that interact to create adverse events. The tenets can be used by practitioners to proactively identify safety leading indicators and contributory factors during adverse event analysis.

Festschrift in honour of Professor Neville Stanton: A lone crusader in a world of driving simulators.

The automotive future has always pointed to a world of intelligent co-pilots and robot cars, but perhaps no more so than Knight Rider. In this 1980's television series the fictional Knight Industries Two Thousand (KITT) was a supercomputer on wheels with 1000 megabytes of memory. The protagonist was Michael Knight, a young loner on a crusade to champion the cause of the innocent and the helpless. This was a shadowy flight into the trials and tribulations of different levels of automation, re-claiming control when automation failed, and a wilful, chatty computer co-driver. An amusing metaphor, perhaps, for the research impact made by Neville Stanton in the field of vehicle automation. Without question - to paraphrase the Knight Rider outro - "one man can make a difference". This festschrift in Neville's honour tells the story of how.

Testing the reliability and validity of risk assessment methods in Human Factors and Ergonomics.

There is growing interest in the use of systems-based risk assessment methods in Human Factors and Ergonomics (HFE). The purpose of this study was to test the intra-rater reliability and criterion-referenced concurrent validity of three systems-based risk assessment approaches: (i) the Systems-Theoretic Process Analysis (STPA) method; (ii) the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method; and, (iii) the Network Hazard Analysis and Risk Management System (Net-HARMS) method. Reliability and validity measures were obtained using the Signal Detection Theory (SDT) paradigm. Whilst STPA identified the highest number of risks, the findings indicate a weak to moderate level of reliability and validity for STPA, EAST-BL and Net-HARMS. There were no statistically significant differences between the methods across analyses. The results suggest that there is merit to the continued use of systems-based risk assessment methods following a series of methodological extensions that aim to enhance the reliability and validity of future applications. Practitioner summary The three risk assessment methods produced weak to moderate levels of stability and accuracy regarding their capability to predict risks. There is a pressing need to further test the reliability and validity of safety methods in Human Factors and Ergonomics.

State of science: evolving perspectives on 'human error'.

This paper reviews the key perspectives on human error and analyses the core theories and methods developed and applied over the last 60 years. These theories and methods have sought to improve our understanding of what human error is, and how and why it occurs, to facilitate the prediction of errors and use these insights to support safer work and societal systems. Yet, while this area of Ergonomics and Human Factors (EHF) has been influential and long-standing, the benefits of the 'human error approach' to understanding accidents and optimising system performance have been questioned. This state of science review analyses the construct of human error within EHF. It then discusses the key conceptual difficulties the construct faces in an era of systems EHF. Finally, a way forward is proposed to prompt further discussion within the EHF community. Practitioner statement This state-of-science review discusses the evolution of perspectives on human error as well as trends in the theories and methods applied to understand, prevent and mitigate error. It concludes that, although a useful contribution has been made, we must move beyond a focus on an individual error to systems failure to understand and optimise whole systems.

Complexity theory in accident causation: using AcciMap to identify the systems thinking tenets in 11 catastrophes.

The quest to explain and understand the cause of accidents is both ever-present and ongoing amongst the safety science community. In an attempt to advance the theory and science of accident causation, researchers have recently formalised a set of '15 systems thinking tenets' that cover the conditions and characteristics of work systems that are believed to contribute to the cause of accidents. The purpose of this study was to attempt to identify the systems thinking tenets across a range of different systems and accidents using the Accident Mapping (AcciMap) method. The findings suggest that the tenets can be attributed to play a role in accident causation, however as a result of this process, the capability of AcciMap has been brought into question. Implications and directions for future research are described. Practitioner Summary: This study is an extension of previous work that suggested there was a need to test for the 'systems thinking tenets of accident causation' in a multi-incident dataset. We used AcciMap to evaluate whether it has the capability to support ongoing accident analysis activities in ergonomics research.

Systems thinking-based risk assessment methods applied to sports performance: A comparison of STPA, EAST-BL, and Net-HARMS in the context of elite women's road cycling.

There is increasing interest in applying systems Human Factors and Ergonomics (HFE) methods in sport. Risk assessment (RA) methods can be used identify risks which may impact the performance of individual athletes, teams, and overall sports systems; however, they have not yet been tested in sport. This study sets out to apply and compare three systems thinking-based RA methods in the context of elite sports performance and report on the frequency and types of the risks identified. The Systems-Theoretic Process Analysis (STPA) method, the Event Analysis of Systemic Teamwork Broken Links (EAST-BL) method, and the Networked Hazard Analysis and Risk Management System (Net-HARMS) method were applied to elite women's road cycling to identify all the credible risks that could degrade optimal team performance. The findings demonstrate that all three methods appear to provide useful results in a context other than safety, and that multiple risks threatening the performance of the cycling team were identified. Whilst the frequency and types of risks differed across the methods applied, there are additional theoretical, methodological, and practical implications to be considered prior to the selection and use of systems thinking-based RA approaches. Recommendations and directions for future HFE and sports science research are discussed.

Interaction-centred design: an end user evaluation of road intersection concepts developed using the cognitive work analysis design toolkit (CWA-DT).

Crashes at intersections represent an important road safety problem. Interactions between different road user types, such as between vehicles and vulnerable road users, are a particular concern. It has been suggested driver-centric road design plays a role in crashes. A multi-road user evaluation of three novel intersection designs is described. The designs were generated using the Cognitive Work Analysis Design Toolkit, underpinned by sociotechnical systems theory. The desktop evaluation involved drivers, motorcyclists, cyclists and pedestrians rating the design concepts against alignment with design goals, sociotechnical systems theory and usability, and providing feedback on the positive and negative aspects. Two concepts received more positive ratings and feedback in comparison to a concept that provided more user autonomy. The evaluation results also highlight clear differences in needs across road user groups. The design and evaluation process demonstrates how sociotechnical systems values and principles can be applied in the design of public spaces. Practitioner Summary: This study involved a participatory evaluation of novel road intersection designs, based on sociotechnical systems theory. The results identified important differences in needs and preferences across road user groups and demonstrate the value of sociotechnical systems theory and user participation in road transport design and evaluation processes. Abbreviations: CWA-DT: cognitive work analysis design toolkit; WDA: work domain analysis; SUS: system usability scale.

Walking the talk: Comparing pedestrian 'activity as imagined' with 'activity as done'.

The safety of vulnerable road users, including pedestrians, is an important issue worldwide. In line with the shift towards systems thinking in transport safety, the aim of this study was to compare the normal performance of pedestrians as they navigate the road system with that imagined by road system managers to gain insights into how safety management can be improved for this vulnerable road user group. The Event Analysis of Systemic Teamwork framework was used to compare pedestrian activity 'as imagined' and 'as done' at signalised road intersections and railway level crossings. Data regarding 'activity as imagined' was derived from documentation review, and data on 'activity as done' was derived from a semi-naturalistic study of ten participants. It is concluded that in both environments pedestrians exhibited more diversity and variability than anticipated by system managers. Insights for improving the design of the road environment for pedestrians are provided. Further, it is argued that wider changes to the processes used in the design and management of road systems are needed.

Fitting methods to paradigms: are ergonomics methods fit for systems thinking?

The issues being tackled within ergonomics problem spaces are shifting. Although existing paradigms appear relevant for modern day systems, it is worth questioning whether our methods are. This paper asks whether the complexities of systems thinking, a currently ubiquitous ergonomics paradigm, are outpacing the capabilities of our methodological toolkit. This is achieved through examining the contemporary ergonomics problem space and the extent to which ergonomics methods can meet the challenges posed. Specifically, five key areas within the ergonomics paradigm of systems thinking are focused on: normal performance as a cause of accidents, accident prediction, system migration, systems concepts and ergonomics in design. The methods available for pursuing each line of inquiry are discussed, along with their ability to respond to key requirements. In doing so, a series of new methodological requirements and capabilities are identified. It is argued that further methodological development is required to provide researchers and practitioners with appropriate tools to explore both contemporary and future problems. Practitioner Summary: Ergonomics methods are the cornerstone of our discipline. This paper examines whether our current methodological toolkit is fit for purpose given the changing nature of ergonomics problems. The findings provide key research and practice requirements for methodological development.

Quantum ergonomics: shifting the paradigm of the systems agenda.

A paradigm is an accepted world view. If we do not continually question our paradigm then wider trends and movements will overtake the discipline leaving it ill adapted to future challenges. This Special Issue is an opportunity to keep systems thinking at the forefront of ergonomics theory and practice. Systems thinking prompts us to ask whether ergonomics, as a discipline, has been too timid? Too preoccupied with the resolution of immediate problems with industrial-age methods when, approaching fast, are developments which could render these operating assumptions an irrelevance. Practical case studies are presented to show how abstract systems problems can be tackled head-on to deliver highly innovative and cost-effective insights. The strategic direction of the discipline foregrounds high-quality systems problems. These are something the discipline is well able to respond to provided that the appropriate operating paradigms are selected. Practitioner Summary: High-quality systems problems are the future of the discipline. How do we convert obtuse sounding systems concepts into practical interventions? In this paper, the essence of systems thinking is distilled and practical case studies used to demonstrate the benefits of this new paradigm.

Using the decision ladder to understand road user decision making at actively controlled rail level crossings.

Rail level crossings (RLXs) represent a key strategic risk for railways worldwide. Despite enforcement and engineering countermeasures, user behaviour at RLXs can often confound expectations and erode safety. Research in this area is limited by a relative absence of insights into actual decision making processes and a focus on only a subset of road user types. One-hundred and sixty-six road users (drivers, motorcyclists, cyclists and pedestrians) completed a diary entry for each of 457 naturalistic encounters with RLXs when a train was approaching. The final eligible sample comprised 94 participants and 248 encounters at actively controlled crossings where a violation of the active warnings was possible. The diary incorporated Critical Decision Method probe questions, which enabled user responses to be mapped onto Rasmussen's decision ladder. Twelve percent of crossing events were non-compliant. The underlying decision making was compared to compliant events and a reference decision model to reveal important differences in the structure and type of decision making within and between road user groups. The findings show that engineering countermeasures intended to improve decision making (e.g. flashing lights), may have the opposite effect for some users because the system permits a high level of flexibility for circumvention. Non-motorised users were more likely to access information outside of the warning signals because of their ability to achieve greater proximity to the train tracks and the train itself. The major conundrum in resolving these issues is whether to restrict the amount of time and information available to users so that it cannot be used for circumventing the system or provide more information to help users make safe decisions.

Can Link Analysis Be Applied to Identify Behavioral Patterns in Train Recorder Data?

OBJECTIVE: A proof-of-concept analysis was conducted to establish whether link analysis could be applied to data from on-train recorders to detect patterns of behavior that could act as leading indicators of potential safety issues. BACKGROUND: On-train data recorders capture data about driving behavior on thousands of routine journeys every day and offer a source of untapped data that could be used to offer insights into human behavior. METHOD: Data from 17 journeys undertaken by six drivers on the same route over a 16-hr period were analyzed using link analysis, and four key metrics were examined: number of links, network density, diameter, and sociometric status. RESULTS: The results established that link analysis can be usefully applied to data captured from on-vehicle recorders. The four metrics revealed key differences in normal driver behavior. These differences have promising construct validity as leading indicators. CONCLUSION: Link analysis is one method that could be usefully applied to exploit data routinely gathered by on-vehicle data recorders. It facilitates a proactive approach to safety based on leading indicators, offers a clearer understanding of what constitutes normal driving behavior, and identifies trends at the interface of people and systems, which is currently a key area of strategic risk. APPLICATION: These research findings have direct applications in the field of transport data monitoring. They offer a means of automatically detecting patterns in driver behavior that could act as leading indicators of problems during operation and that could be used in the proactive monitoring of driver competence, risk management, and even infrastructure design.

More than meets the eye: Using cognitive work analysis to identify design requirements for future rail level crossing systems.

An increasing intensity of operations means that the longstanding safety issue of rail level crossings is likely to become worse in the transport systems of the future. It has been suggested that the failure to prevent collisions may be, in part, due to a lack of systems thinking during design, crash analysis, and countermeasure development. This paper presents a systems analysis of current active rail level crossing systems in Victoria, Australia that was undertaken to identify design requirements to improve safety in future rail level crossing environments. Cognitive work analysis was used to analyse rail level crossing systems using data derived from a range of activities. Overall the analysis identified a range of instances where modification or redesign in line with systems thinking could potentially improve behaviour and safety. A notable finding is that there are opportunities for redesign outside of the physical rail level crossing infrastructure, including improved data systems, in-vehicle warnings and modifications to design processes, standards and guidelines. The implications for future rail level crossing systems are discussed.

Multiple driver distractions: a systemic transport problem.

Strategies to contend with driver distraction may no longer be sufficient for the emerging variety of contemporary driver distractions. A more systematic and systemic approach holds promise for improved road safety but is not currently being developed. This systematic review of multiple driver distractions aims to address this gap and presents two key findings. Systematic classification of distracting tasks with respect to driving is challenging, and engagement with Multiple-Additional-to-Driving (MAD) tasks is almost universally detrimental to driving performance. A model is presented to assist in systematically characterising multiple driver demands. Identified literature is placed into context using the model and shortfalls are identified.

Using the event analysis of systemic teamwork (EAST) to explore conflicts between different road user groups when making right hand turns at urban intersections.

Collisions between different types of road users at intersections form a substantial component of the road toll. This paper presents an analysis of driver, cyclist, motorcyclist and pedestrian behaviour at intersections that involved the application of an integrated suite of ergonomics methods, the Event Analysis of Systemic Teamwork (EAST) framework, to on-road study data. EAST was used to analyse behaviour at three intersections using data derived from an on-road study of driver, cyclist, motorcyclist and pedestrian behaviour. The analysis shows the differences in behaviour and cognition across the different road user groups and pinpoints instances where this may be creating conflicts between different road users. The role of intersection design in creating these differences in behaviour and resulting conflicts is discussed. It is concluded that currently intersections are not designed in a way that supports behaviour across the four forms of road user studied. Interventions designed to improve intersection safety are discussed. Practitioner Summary: Intersection safety currently represents a key road safety issue worldwide. This paper presents a novel application of a framework of ergonomics methods for studying differences in road user behaviour at intersections. The findings support development of interventions that consider all road users as opposed to one group in isolation.

Exploring schema-driven differences in situation awareness between road users: an on-road study of driver, cyclist and motorcyclist situation awareness.

Collisions between different road users make a substantial contribution to road trauma. Although evidence suggests that different road users interpret the same road situations differently, it is not clear how road users' situation awareness differs, nor is it clear which differences might lead to conflicts. This article presents the findings from an on-road study conducted to examine driver, motorcyclist and cyclist situation awareness in different road environments. The findings suggest that, in addition to minor differences in the structure of different road users' situation awareness (i.e. amount of information and how it is integrated), the actual content of situation awareness in terms of road user schemata, the resulting interaction with the world and the information underpinning situation awareness is markedly different. Further examination indicates that the differences are likely to be compatible along arterial roads, shopping strips and at roundabouts, but that they may create conflicts between different road users at intersections. Interventions designed to support compatible situation awareness and behaviour between different road users are discussed.

Human performance under two different command and control paradigms.

The paradoxical behaviour of a new command and control concept called Network Enabled Capability (NEC) provides the motivation for this paper. In it, a traditional hierarchical command and control organisation was pitted against a network centric alternative on a common task, played thirty times, by two teams. Multiple regression was used to undertake a simple form of time series analysis. It revealed that whilst the NEC condition ended up being slightly slower than its hierarchical counterpart, it was able to balance and optimise all three of the performance variables measured (task time, enemies neutralised and attrition). From this it is argued that a useful conceptual response is not to consider NEC as an end product comprised of networked computers and standard operating procedures, nor to regard the human system interaction as inherently stable, but rather to view it as a set of initial conditions from which the most adaptable component of all can be harnessed: the human.