Operator selection for human-automation teaming: The role of manual task skill in predicting automation failure intervention.

Humans working in modern work systems are increasingly required to supervise task automation. We examined whether manual aircraft conflict detection skill predicted participants' ability to respond to conflict detection automation failures in simulated air traffic control. In a conflict discrimination task (to assess manual skill), participants determined whether pairs of aircraft were in conflict or not by judging their relative-arrival time at common intersection points. Then in a simulated air traffic control task, participants supervised automation which either partially or fully detected and resolved conflicts on their behalf. Automation supervision required participants to detect when automation may have failed and effectively intervene. When automation failed, participants who had better manual conflict detection skill were faster and more accurate to intervene. However, a substantial proportion of variance in failure intervention was not explained by manual conflict detection skill, potentially reflecting that future research should consider other cognitive skills underlying automation supervision.

How do humans learn about the reliability of automation?

In a range of settings, human operators make decisions with the assistance of automation, the reliability of which can vary depending upon context. Currently, the processes by which humans track the level of reliability of automation are unclear. In the current study, we test cognitive models of learning that could potentially explain how humans track automation reliability. We fitted several alternative cognitive models to a series of participants' judgements of automation reliability observed in a maritime classification task in which participants were provided with automated advice. We examined three experiments including eight between-subjects conditions and 240 participants in total. Our results favoured a two-kernel delta-rule model of learning, which specifies that humans learn by prediction error, and respond according to a learning rate that is sensitive to environmental volatility. However, we found substantial heterogeneity in learning processes across participants. These outcomes speak to the learning processes underlying how humans estimate automation reliability and thus have implications for practice.

Prospective memory decision control: A computational model of context effects on prospective memory.

Prospective memory (PM) tasks require remembering to perform a deferred action and can be associated with predictable contexts. We present a theory and computational model, prospective memory decision control (PMDC), of the cognitive processes by which context supports PM. Under control conditions, participants completed lexical decisions. Under PM conditions, participants had the additional PM task of responding to letter strings containing certain syllables. Stimuli were presented in one of two colors, with color potentially changing after each set of four trials. A pretrial colored fixation was presented before each set. Under control and PM standard conditions, fixation color was meaningless. Under PM context conditions, fixation color indicated whether a PM target could occur within the next set. We replicated prior findings of higher PM accuracy for context compared to standard conditions, and the expected variation in PM costs (slowed lexical decisions) as a function of context relevance. PMDC, which formalizes PM as a process of evidence accumulation among ongoing and PM task responses, accounted for the impact of context on PM costs and accuracy via proactive and reactive cognitive control. Increased ongoing task thresholds and decreased PM thresholds in relevant contexts indicated proactive control. With context provision, PM accumulation rates on PM trials increased, as did inhibition of accumulation to competing responses, indicating reactive control. Although an observed capacity-sharing effect explained some portion of PM costs, we found no evidence that participants redirected more capacity from the ongoing to the PM task when contextually cued to relevant contexts. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

An extension of the shifted Wald model of human response times: Capturing the time dynamic properties of human cognition : Trial-varying Wald model.

Despite the ubiquitous nature of evidence accumulation models in cognitive and experimental psychology, there has been a comparatively limited uptake of such techniques in the applied literature. While quantifying latent cognitive processing properties has significant potential for applied domains such as adaptive work systems, accumulator models often fall short in practical applications. Two primary reasons for these shortcomings are the complexities and time needed for the application of cognitive models, and the failure of current models to capture systematic trial-to-trial variability in parameters. In this manuscript, we develop a novel, trial-varying extension of the shifted Wald model to address these concerns. By leveraging conjugate properties of the Wald distribution, we derive computationally efficient solutions for threshold and drift parameters which can be updated instantaneously with new data. The resulting model allows the quantification of systematic variation in latent cognitive parameters across trials and we demonstrate the utility of such analyses through simulations and an exemplar application to an existing data set. The analytic nature of our solutions opens the door for real-world applications, significantly extending the reach of computational models of behavioral responses.

Meaningful Communication but not Superficial Anthropomorphism Facilitates Human-Automation Trust Calibration: The Human-Automation Trust Expectation Model (HATEM).

OBJECTIVE: The objective was to demonstrate anthropomorphism needs to communicate contextually useful information to increase user confidence and accurately calibrate human trust in automation. BACKGROUND: Anthropomorphism is believed to improve human-automation trust but supporting evidence remains equivocal. We test the Human-Automation Trust Expectation Model (HATEM) that predicts improvements to trust calibration and confidence in accepted advice arising from anthropomorphism will be weak unless it aids naturalistic communication of contextually useful information to facilitate prediction of automation failures. METHOD: Ninety-eight undergraduates used a submarine periscope simulator to classify ships, aided by the Ship Automated Modelling (SAM) system that was 50% reliable. A between-subjects 2 × 3 design compared SAM appearance (anthropomorphic avatar vs. camera eye) and voice inflection (monotone vs. meaningless vs. meaningful), with the meaningful inflections communicating contextually useful information about automated advice regarding certainty and uncertainty. RESULTS: Avatar SAM appearance was rated as more anthropomorphic than camera eye, and meaningless and meaningful inflections were both rated more anthropomorphic than monotone. However, for subjective trust, trust calibration, and confidence in accepting SAM advice, there was no evidence of anthropomorphic appearance having any impact, while there was decisive evidence that meaningful inflections yielded better outcomes on these trust measures than monotone and meaningless inflections. CONCLUSION: Anthropomorphism had negligible impact on human-automation trust unless its execution enhanced communication of relevant information that allowed participants to better calibrate expectations of automation performance. APPLICATION: Designers using anthropomorphism to calibrate trust need to consider what contextually useful information will be communicated via anthropomorphic features.

Transparent Automated Advice to Mitigate the Impact of Variation in Automation Reliability.

OBJECTIVE: To examine the extent to which increased automation transparency can mitigate the potential negative effects of low and high automation reliability on disuse and misuse of automated advice, and perceived trust in automation. BACKGROUND: Automated decision aids that vary in the reliability of their advice are increasingly used in workplaces. Low-reliability automation can increase disuse of automated advice, while high-reliability automation can increase misuse. These effects could be reduced if the rationale underlying automated advice is made more transparent. METHODS: Participants selected the optimal UV to complete missions. The Recommender (automated decision aid) assisted participants by providing advice; however, it was not always reliable. Participants determined whether the Recommender provided accurate information and whether to accept or reject advice. The level of automation transparency (medium, high) and reliability (low: 65%, high: 90%) were manipulated between-subjects. RESULTS: With high- compared to low-reliability automation, participants made more accurate (correctly accepted advice and identified whether information was accurate/inaccurate) and faster decisions, and reported increased trust in automation. Increased transparency led to more accurate and faster decisions, lower subjective workload, and higher usability ratings. It also eliminated the increased automation disuse associated with low-reliability automation. However, transparency did not mitigate the misuse associated with high-reliability automation. CONCLUSION: Transparency protected against low-reliability automation disuse, but not against the increased misuse potentially associated with the reduced monitoring and verification of high-reliability automation. APPLICATION: These outcomes can inform the design of transparent automation to improve human-automation teaming under conditions of varied automation reliability.

Reducing the Costs of Automation Failure by Providing Voluntary Automation Checking Tools.

OBJECTIVE: We investigated the extent to which a voluntary-use range and bearing line (RBL) tool improves return-to-manual performance when supervising high-degree conflict detection automation in simulated air traffic control. BACKGROUND: High-degree automation typically benefits routine performance and reduces workload, but can degrade return-to-manual performance if automation fails. We reasoned that providing a voluntary checking tool (RBL) would support automation failure detection, but also that automation induced complacency could extend to nonoptimal use of such tools. METHOD: Participants were assigned to one of three conditions, where conflict detection was either performed: manually, with RBLs available to use (Manual + RBL), automatically with RBLs (Auto + RBL), or automatically without RBLs (Auto). Voluntary-use RBLs allowed participants to reliably check aircraft conflict status. Automation failed once. RESULTS: RBLs improved automation failure detection - with participants intervening faster and making fewer false alarms when provided RBLs compared to not (Auto + RBL vs Auto). However, a cost of high-degree automation remained, with participants slower to intervene to the automation failure than to an identical manual conflict event (Auto + RBL vs Manual + RBL). There was no difference in RBL engagement time between Auto + RBL and Manual + RBL conditions, suggesting participants noticed the conflict event at the same time. CONCLUSIONS: The cost of automation may have arisen from participants' reconciling which information to trust: the automation (which indicated no conflict and had been perfectly reliable prior to failing) or the RBL (which indicated a conflict). APPLICATIONS: Providing a mechanism for checking the validity of high-degree automation may facilitate human supervision of automation.

Do concurrent task demands impact the benefit of automation transparency?

Automated decision aids typically improve decision-making, but incorrect advice risks automation misuse or disuse. We examined the novel question of whether increased automation transparency improves the accuracy of automation use under conditions with/without concurrent (non-automated assisted) task demands. Participants completed an uninhabited vehicle (UV) management task whereby they assigned the best UV to complete missions. Automation advised the best UV but was not always correct. Concurrent non-automated task demands decreased the accuracy of automation use, and increased decision time and perceived workload. With no concurrent task demands, increased transparency which provided more information on how the automation made decisions, improved the accuracy of automation use. With concurrent task demands, increased transparency led to higher trust ratings, faster decisions, and a bias towards agreeing with automation. These outcomes indicate increased reliance on highly transparent automation under conditions with concurrent task demands and have potential implications for human-automation teaming design.

Automated decision aids: When are they advisors and when do they take control of human decision making?

We applied a computational model to examine the extent to which participants used an automated decision aid as an advisor, as compared to a more autonomous trigger of responding, at varying levels of decision aid reliability. In an air traffic control conflict detection task, we found higher accuracy when the decision aid was correct, and more errors when the decision aid was incorrect, as compared to a manual condition (no decision aid). Responses that were correct despite incorrect automated advice were slower than matched manual responses. Decision aids set at lower reliability (75%) had smaller effects on choices and response times, and were subjectively trusted less, than decision aids set at higher reliability (95%). We fitted an evidence accumulation model to choices and response times to measure how information processing was affected by decision aid inputs. Participants primarily treated low-reliability decision aids as an advisor rather than directly accumulating evidence based on its advice. Participants directly accumulated evidence based upon the advice of high-reliability decision aids, consistent with granting decision aids more autonomous influence over decisions. Individual differences in the level of direct accumulation correlated with subjective trust, suggesting a cognitive mechanism by which trust impacts human decisions. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Lower prospective memory is associated with higher neurocognitive dispersion in two samples of people with HIV: A conceptual replication study.

OBJECTIVES: People living with HIV (PLWH) often experience deficits in the strategic/executive aspects of prospective memory (PM) that can interfere with instrumental activities of daily living. This study used a conceptual replication design to determine whether cognitive intraindividual variability, as measured by dispersion (IIV-dispersion), contributes to PM performance and symptoms among PLWH. METHODS: Study 1 included 367 PLWH who completed a comprehensive clinical neuropsychological test battery, the Memory for Intentions Test (MIsT), and the Prospective and Retrospective Memory Questionnaire (PRMQ). Study 2 included 79 older PLWH who completed the Cogstate cognitive battery, the Cambridge Prospective Memory Test (CAMPROMPT), an experimental measure of time-based PM, and the PRMQ. In both studies, a mean-adjusted coefficient of variation was derived to measure IIV-dispersion using normative T-scores from the cognitive battery. RESULTS: Higher IIV-dispersion was significantly associated with lower time-based PM performance at small-to-medium effect sizes in both studies (mean r s  = -0.30). The relationship between IIV-dispersion and event-based PM performance was comparably small in magnitude in both studies (r s  = -0.19, -0.20), but it was only statistically significant in Study 1. IIV-dispersion showed very small, nonsignificant relationships with self-reported PM symptoms in both samples (r s < 0.10). CONCLUSIONS: Extending prior work in healthy adults, these findings suggest that variability in performance across a cognitive battery contributes to laboratory-based PM accuracy, but not perceived PM symptoms, among PLWH. Future studies might examine whether daily fluctuations in cognition or other aspects of IIV (e.g., inconsistency) play a role in PM failures in everyday life.

The Impact of Transparency and Decision Risk on Human-Automation Teaming Outcomes.

OBJECTIVE: Examine the effects of decision risk and automation transparency on the accuracy and timeliness of operator decisions, automation verification rates, and subjective workload. BACKGROUND: Decision aids typically benefit performance, but can provide incorrect advice due to contextual factors, creating the potential for automation disuse or misuse. Decision aids can reduce an operator's manual problem evaluation, and it can also be strategic for operators to minimize verifying automated advice in order to manage workload. METHOD: Participants assigned the optimal unmanned vehicle to complete missions. A decision aid provided advice but was not always reliable. Two levels of decision aid transparency were manipulated between participants. The risk associated with each decision was manipulated using a financial incentive scheme. Participants could use a calculator to verify automated advice; however, this resulted in a financial penalty. RESULTS: For high- compared with low-risk decisions, participants were more likely to reject incorrect automated advice and were more likely to verify automation and reported higher workload. Increased transparency did not lead to more accurate decisions and did not impact workload but decreased automation verification and eliminated the increased decision time associated with high decision risk. CONCLUSION: Increased automation transparency was beneficial in that it decreased automation verification and decreased decision time. The increased workload and automation verification for high-risk missions is not necessarily problematic given the improved automation correct rejection rate. APPLICATION: The findings have potential application to the design of interfaces to improve human-automation teaming, and for anticipating the impact of decision risk on operator behavior.

Subjective cognitive decline disrupts aspects of prospective memory in older adults with HIV disease.

Subjective cognitive decline (SCD) is a risk factor for dementia that may occur at higher rates in people with HIV (PWH). Prospective memory (PM) is an aspect of cognition that may help us better understand how SCD impacts daily life. Paricipants were 62 PWH aged ≥ 50 years and 33 seronegative individuals. SCD was operationalized as normatively elevated cognitive symptoms on standardized questionnaires, but with normatively unimpaired performance-based cognition and no current affective disorders. PM was measured with the Comprehensive Assessment of Prospective Memory (CAPM), the Cambridge Test of Prospective Memory (CAMPROMPT), and an experimental computerized time-based PM task. A logistic regression revealed that older PWH had a three-fold increased likelihood for SCD. Among the PWH, SCD was associated with more frequent PM symptoms and poorer accuracy on the time-based scale of the CAMPROMPT. These findings suggest that SCD disrupts PM in older PWH.

Detecting a Single Automation Failure: The Impact of Expected (But Not Experienced) Automation Reliability.

OBJECTIVE: Examine the impact of expected automation reliability on trust, workload, task disengagement, nonautomated task performance, and the detection of a single automation failure in simulated air traffic control. BACKGROUND: Prior research has focused on the impact of experienced automation reliability. However, many operational settings feature automation that is reliable to the extent that operators will seldom experience automation failures. Despite this, operators must remain aware of when automation is at greater risk of failing. METHOD: Participants performed the task with or without conflict detection/resolution automation. Automation failed to detect/resolve one conflict (i.e., an automation miss). Expected reliability was manipulated via instructions such that the expected level of reliability was (a) constant or variable, and (b) the single automation failure occurred when expected reliability was high or low. RESULTS: Trust in automation increased with time on task prior to the automation failure. Trust was higher when expecting high relative to low reliability. Automation failure detection was improved when the failure occurred under low compared with high expected reliability. Subjective workload decreased with automation, but there was no improvement to nonautomated task performance. Automation increased perceived task disengagement. CONCLUSIONS: Both automation reliability expectations and task experience played a role in determining trust. Automation failure detection was improved when the failure occurred at a time it was expected to be more likely. Participants did not effectively allocate any spared capacity to nonautomated tasks. APPLICATIONS: The outcomes are applicable because operators in field settings likely form contextual expectations regarding the reliability of automation.

The Impact of Information Integration in a Simulation of Future Submarine Command and Control.

OBJECTIVE: Examine the extent to which increasing information integration across displays in a simulated submarine command and control room can reduce operator workload, improve operator situation awareness, and improve team performance. BACKGROUND: In control rooms, the volume and number of sources of information are increasing, with the potential to overwhelm operator cognitive capacity. It is proposed that by distributing information to maximize relevance to each operator role (increasing information integration), it is possible to not only reduce operator workload but also improve situation awareness and team performance. METHOD: Sixteen teams of six novice participants were trained to work together to combine data from multiple sensor displays to build a tactical picture of surrounding contacts at sea. The extent that data from one display were available to operators at other displays was manipulated (information integration) between teams. Team performance was assessed as the accuracy of the generated tactical picture. RESULTS: Teams built a more accurate tactical picture, and individual team members had better situation awareness and lower workload, when provided with high compared with low information integration. CONCLUSION: A human-centered design approach to integrating information in command and control settings can result in lower workload, and enhanced situation awareness and team performance. APPLICATION: The design of modern command and control rooms, in which operators must fuse increasing volumes of complex data from displays, may benefit from higher information integration based on a human-centered design philosophy, and a fundamental understanding of the cognitive work that is carried out by operators.

The Perception of Automation Reliability and Acceptance of Automated Advice.

OBJECTIVE: Examine (1) the extent to which humans can accurately estimate automation reliability and calibrate to changes in reliability, and how this is impacted by the recent accuracy of automation; and (2) factors that impact the acceptance of automated advice, including true automation reliability, reliability perception, and the difference between an operator's perception of automation reliability and perception of their own reliability. BACKGROUND: Existing evidence suggests humans can adapt to changes in automation reliability but generally underestimate reliability. Cognitive science indicates that humans heavily weight evidence from more recent experiences. METHOD: Participants monitored the behavior of maritime vessels (contacts) in order to classify them, and then received advice from automation regarding classification. Participants were assigned to either an initially high (90%) or low (60%) automation reliability condition. After some time, reliability switched to 75% in both conditions. RESULTS: Participants initially underestimated automation reliability. After the change in true reliability, estimates in both conditions moved towards the common true reliability, but did not reach it. There were recency effects, with lower future reliability estimates immediately following incorrect automation advice. With lower initial reliability, automation acceptance rates tracked true reliability more closely than perceived reliability. A positive difference between participant assessments of the reliability of automation and their own reliability predicted greater automation acceptance. CONCLUSION: Humans underestimate the reliability of automation, and we have demonstrated several critical factors that impact the perception of automation reliability and automation use. APPLICATION: The findings have potential implications for training and adaptive human-automation teaming.

Evidence accumulation modelling in the wild: understanding safety-critical decisions.

Evidence accumulation models (EAMs) are a class of computational cognitive model used to understand the latent cognitive processes that underlie human decisions and response times (RTs). They have seen widespread application in cognitive psychology and neuroscience. However, historically, the application of these models was limited to simple decision tasks. Recently, researchers have applied these models to gain insight into the cognitive processes that underlie observed behaviour in applied domains, such as air-traffic control (ATC), driving, forensic and medical image discrimination, and maritime surveillance. Here, we discuss how this modelling approach helps researchers understand how the cognitive system adapts to task demands and interventions, such as task automation. We also discuss future directions and argue for wider adoption of cognitive modelling in Human Factors research.

Return-to-Manual Performance can be Predicted Before Automation Fails.

OBJECTIVE: This study aimed to examine operator state variables (workload, fatigue, and trust in automation) that may predict return-to-manual (RTM) performance when automation fails in simulated air traffic control. BACKGROUND: Prior research has largely focused on triggering adaptive automation based on reactive indicators of performance degradation or operator strain. A more direct and effective approach may be to proactively engage/disengage automation based on predicted operator RTM performance (conflict detection accuracy and response time), which requires analyses of within-person effects. METHOD: Participants accepted and handed-off aircraft from their sector and were assisted by imperfect conflict detection/resolution automation. To avoid aircraft conflicts, participants were required to intervene when automation failed to detect a conflict. Participants periodically rated their workload, fatigue and trust in automation. RESULTS: For participants with the same or higher average trust than the sample average, an increase in their trust (relative to their own average) slowed their subsequent RTM response time. For participants with lower average fatigue than the sample average, an increase in their fatigue (relative to own average) improved their subsequent RTM response time. There was no effect of workload on RTM performance. CONCLUSIONS: RTM performance degraded as trust in automation increased relative to participants' own average, but only for individuals with average or high levels of trust. APPLICATIONS: Study outcomes indicate a potential for future adaptive automation systems to detect vulnerable operator states in order to predict subsequent RTM performance decrements.

Human behavioral response to fluctuating automation reliability.

Human perception of automation reliability and automation acceptance behaviours are key to effective human-automation teaming. This study examined factors that impact perceptions of automation reliability over time and the acceptance of automated advice. Participants completed a maritime vessel classification task in which they classified vessels (contacts) with the assistance of automation. In Experiment 1 automation reliability successively switched from high to low (or vice versa). In Experiment 2 automation reliability decreased by varying magnitudes before returning to high. Participants did not initially calibrate to true reliability and experiencing low automation reliability reduced future reliability estimates when experiencing subsequent high reliability. Automation acceptance was predicted by positive differences between participant perception of automation reliability and confidence in their own manual classification reliability. Experiencing low automation reliability caused perceptions of reliability and automation acceptance rates to diverge. These findings have important implications for training and adaptive human-automation teaming in complex work environments.

Should We Just Let the Machines Do It? The Benefit and Cost of Action Recommendation and Action Implementation Automation.

OBJECTIVE: To examine the effects of action recommendation and action implementation automation on performance, workload, situation awareness (SA), detection of automation failure, and return-to-manual performance in a submarine track management task. BACKGROUND: Theory and meta-analytic evidence suggest that with increasing degrees of automation (DOA), operator performance improves and workload decreases, but SA and return-to-manual performance declines. METHOD: Participants monitored the location and heading of contacts in order to classify them, mark their closest point of approach (CPA), and dive when necessary. Participants were assigned either no automation, action recommendation automation, or action implementation automation. An automation failure occurred late in the task, whereby the automation provided incorrect classification advice or implemented incorrect classification actions. RESULTS: Compared to no automation, action recommendation automation benefited automated task performance and lowered workload, but cost nonautomated task performance. Action implementation automation resulted in perfect automated task performance (by default) and lowered workload, with no costs to nonautomated task performance, SA, or return-to-manual performance compared to no automation. However, participants provided action implementation automation were less likely to detect the automation failure compared to those provided action recommendations, and made less accurate classifications immediately after the automation failure, compared to those provided no automation. CONCLUSION: Action implementation automation produced the anticipated benefits but also caused poorer automation failure detection. APPLICATION: While action implementation automation may be effective for some task contexts, system designers should be aware that operators may be less likely to detect automation failures and that performance may suffer until such failures are detected.

Target learning in event-based prospective memory.

Event-based prospective memory (PM) tasks require individuals to remember to perform a previously planned action when they encounter a specific event. Often, the natural environments in which PM tasks occur are embedded are constantly changing, requiring humans to adapt by learning. We examine one such adaptation by integrating PM target learning with the prospective memory decision control (PMDC) cognitive model. We apply this augmented model to an experiment that manipulated exposure to PM targets, comparing a single-target PM condition where the target was well learned from the outset, to a multiple-target PM condition with less initial PM target exposure, allowing us to examine the effect of continued target learning opportunities. Single-target PM accuracy was near ceiling whereas multiple-target PM accuracy was initially poorer but improved throughout the course of the experiment. PM response times were longer for the multiple- compared with single-target PM task but this difference also decreased over time. The model indicated that PM trial evidence accumulation rates, and the inhibition of competing responses, were initially higher for single compared to multiple PM targets, but that this difference decreased over time due to the learning of multiple-targets over the target repetitions. These outcomes provide insight into how the processes underlying event-based PM can dynamically evolve over time, and a modeling framework to further investigate the effect of learning on event-based PM decision processes. (PsycInfo Database Record (c) 2022 APA, all rights reserved).