Identifying inefficient strategies in automation-aided signal detection.

Automated diagnostic aids can assist human operators in signal detection tasks, providing alarms, warnings, or diagnoses. Operators often use decision aids poorly, though, falling short of best possible performance levels. Previous research has suggested that operators interact with binary signal detection aids using a sluggish contingent cutoff (CC) strategy (Robinson & Sorkin, 1985), shifting their response criterion in the direction stipulated by the aid's diagnosis each trial but making adjustments that are smaller than optimal. The present study tested this model by examining the efficiency of automation-aided signal detection under different levels of task difficulty. In a pair of experiments, participants performed a numeric decision-making task requiring them to make signal or noise judgments on the basis of probabilistic readings. The mean reading values of signal and noise states differed between groups of participants, producing two levels of task difficulty. Data were fit with the CC model and two alternative accounts of automation-aided strategy: a discrete deference (DD) model, which assumed participants defer to the aid on a subset of trials and a mixture model, which assumed that participants choose randomly between the CC and DD strategies every trial. Model fits favored the mixture model. The results indicate multiple forms of inefficiency in operators' strategies for using signal detection aids. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Psychometric curves reveal changes in bias, lapse rate, and guess rate in an online vigilance task.

When human monitors are required to detect infrequent signals among noise, they typically exhibit a decline in correct detections over time. Researchers have attributed this vigilance decrement to three alternative mechanisms: shifts in response bias, losses of sensitivity, and attentional lapses. The current study examined the extent to which changes in these mechanisms contributed to the vigilance decrement in an online monitoring task. Participants in two experiments (N = 102, N = 192) completed an online signal detection task, judging whether the separation between two probes each trial exceeded a criterion value. Separation was varied across trials and data were fit with logistic psychometric curves using Bayesian hierarchical parameter estimation. Parameters representing sensitivity, response bias, attentional lapse rate, and guess rate were compared across the first and last 4 minutes of the vigil. Data gave decisive evidence of conservative bias shifts, an increased attentional lapse rate, and a decreased positive guess rate over time on task, but no strong evidence for or against an effect of sensitivity. Sensitivity decrements appear less robust than criterion shifts or attention lapses as causes of the vigilance loss.

Simple manipulations of anthropomorphism fail to induce perceptions of humanness or improve trust in an automated agent.

Although automation is employed as an aid to human performance, operators often interact with automated decision aids inefficiently. The current study investigated whether anthropomorphic automation would engender higher trust and use, subsequently improving human-automation team performance. Participants performed a multi-element probabilistic signal detection task in which they diagnosed a hypothetical nuclear reactor as in a state of safety or danger. The task was completed unassisted and assisted by a 93%-reliable agent varying in anthropomorphism. Results gave no evidence that participants' perceptions of anthropomorphism differed between conditions. Further, anthropomorphic automation failed to bolster trust and automation-aided performance. Findings suggest that the benefits of anthropomorphism may be limited in some contexts.

Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task.

OBJECTIVE: The present study replicated and extended prior findings of suboptimal automation use in a signal detection task, benchmarking automation-aided performance to the predictions of several statistical models of collaborative decision making. BACKGROUND: Though automated decision aids can assist human operators to perform complex tasks, operators often use the aids suboptimally, achieving performance lower than statistically ideal. METHOD: Participants performed a simulated security screening task requiring them to judge whether a target (a knife) was present or absent in a series of colored X-ray images of passenger baggage. They completed the task both with and without assistance from a 93%-reliable automated decision aid that provided a binary text diagnosis. A series of three experiments varied task characteristics including the timing of the aid's judgment relative to the raw stimuli, target certainty, and target prevalence. RESULTS AND CONCLUSION: Automation-aided performance fell closest to the predictions of the most suboptimal model under consideration, one which assumes the participant defers to the aid's diagnosis with a probability of 50%. Performance was similar across experiments. APPLICATION: Results suggest that human operators' performance when undertaking a naturalistic search task falls far short of optimal and far lower than prior findings using an abstract signal detection task.

Psychometric Curves Reveal Three Mechanisms of Vigilance Decrement.

The vigilance decrement is a decline in signal detection rate that occurs over time on a sustained-attention task. The effect has typically been ascribed to conservative shifts of response bias and losses of perceptual sensitivity. Recent work, though, has suggested that sensitivity losses in vigilance tasks are spurious, and other findings have implied that attentional lapses contribute to vigilance failures. To test these possibilities, we used Bayesian hierarchical modeling to compare psychometric curves for the first and last blocks of a visual vigilance task. Participants were a convenience sample of 99 young adults. Data showed evidence for all three postulated mechanisms of vigilance loss: a conservative shift of response bias, a decrease in perceptual sensitivity, and a tendency toward more frequent attentional lapses. Results confirm that sensitivity losses are possible in a sustained-attention task but indicate that mental lapses can also contribute to the vigilance decrement.

Shared Gaze Fails to Improve Team Visual Monitoring.

OBJECTIVE: The aim was to test the value of shared gaze as a way to improve team performance in a visual monitoring task. BACKGROUND: Teams outperform individuals in monitoring tasks, but fall short of achievable levels. Shared-gaze displays offer a potential method of improving team efficiency. Within a shared-gaze arrangement, operators collaborate on a visual task, and each team member's display includes a cursor to represent the other teammates' point of regard. Past work has suggested that shared gaze allows operators to better communicate and coordinate their attentional scanning in a visual search task. The current experiments sought to replicate and extend earlier findings of inefficient team performance in a visual monitoring task, and asked whether shared gaze would improve team efficiency. METHOD: Participants performed a visual monitoring task framed as a sonar operation. Displays were matrices of luminance patches varying in intensity. The participants' task was to monitor for occasional critical signals, patches of high luminance. In Experiment 1, pairs of participants performed the task independently, or working as teams. In Experiment 2, teams of two participants performed the task with or without shared-gaze displays. RESULTS: In Experiment 1, teams detected more critical signals than individuals, but were statistically inefficient; detection rates were lower than predicted by a control model that assumed pairs of operators searching in isolation. In Experiment 2, shared gaze failed to increase target detection rates. CONCLUSION AND APPLICATION: Operators collaborate inefficiently in visual monitoring tasks, and shared gaze does not improve their performance.

Ironic efficiency in automation-aided signal detection.

Decision makers often make poor use of the information provided by an automated signal detection aid; recent studies have found that participants assisted by an automated aid fell well short of best-possible sensitivity levels. The present study tested the generalisability of this finding over varying levels of aid reliability. Participants performed a binary signal detection task either unaided or with assistance from a decision aid that was 60%, 85%, or 96%-reliable. Assistance from a highly reliable aid (85% or 96%) improved discrimination performance, while assistance from a low-reliability aid (60%) did not. Because their ideal strategy is to place less weight on less reliable cues, however, the decision makers' tendency to disuse the aid became more appropriate as the aid's reliability declined. Automation-aided efficiency was thus near to optimal when the aid was close to chance but became highly inefficient, ironically, as the aid's reliability increased. Practitioner Summary: Investigating operators' automation-aided information integration strategies allows human factors practitioners to predict the level of performance the operator will attain. Ironically, in an aided signal detection task, performance when assisted by a highly reliable aid is far less efficient than that obtained when assisted by a far less reliable aid. Abbreviations: OW: optimal weighting; UW: uniform weighting; CC: contingent criterion; BD: best decides; CF: coin flip; PM: probability matching; HDI: highest density interval; MCMC: markov chain monte carlo; HR: hit rate; FAR: false alarm rate.

Collaboration improves unspeeded search in the absence of precise target information.

Two-person teams outperform individuals in search tasks, and even exceed expectations based on statistical limitations. Here, we aimed to replicate and extend this result. We used Bayesian hierarchical modelling of receiver operating characteristics to examine collaborative performance in a visual search task wherein top-down target information was constrained. Participants (N = 16 teams per experiment in Experiments 1 and 2; N = 24 teams in Experiment 3), working independently or collaboratively, performed a search task framed as a medical image reading task. Stimuli were polygons generated by randomly distorting a prototype shape. Observers judged whether an extreme distortion was present among a set of low-distortion distractor objects. Team members' individual sensitivity levels were used to predict collaborative sensitivity using two versions of a uniform judgment-weighting (UW) model, one that assumed stochastically independent judgments and one that accounted for correlations in the team members' judgments. Collaborative search was better than that from single observers in all three experiments, and consistently trended higher than predictions of the correlated UW model. Results imply that collaborative search can be highly efficient even when target foreknowledge is limited.

Collaborative search in a mock baggage screening task.

Signal detection theory provides models of information integration that allow researchers to predict and benchmark collaborative performance in a visual search task. Naturalistic stimuli, however, may not conform to the simplifying assumptions-specifically, assumptions of equal-variance signal and noise distributions and stochastically independent observers-that are often made to make collaborative signal detection models tractable. Here, we used Bayesian hierarchical modeling of receiver operating characteristics to circumvent this difficulty. Participants (N = 28-32 per experiment) performed a simulated baggage x-ray screening task, working alone or in teams of 2. Team performance was compared with the predictions of 2 versions of a uniform weighting model of information integration, 1 that assumed stochastically independent judgments from the 2 members of a team and 1 that allowed for correlated judgments. Across 4 experiments, teams fell short of the uncorrelated-judgment model's predictions, but outperformed predictions based on the observed correlations in individual judgments. Results imply motivational effects that improve individual searchers' effort under collaborative conditions, or collaborative strategies that effectively decorrelate the individual searchers' judgments. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

No Effect of Cue Format on Automation Dependence in an Aided Signal Detection Task.

OBJECTIVE: To investigate whether manipulating the format of an automated decision aid's cues can improve participants' information integration strategies in a signal detection task. BACKGROUND: Automation-aided decision making is often suboptimal, falling well short of statistically ideal levels. The choice of format in which the cues from the aid are displayed may help users to better understand and integrate the aid's judgments with their own. METHOD: Participants performed a signal detection task that asked them to classify random dot images as either blue or orange dominant. They made their judgments either unaided or with assistance from a 93% reliable automated decision aid. The aid provided a binary judgment, along with an estimate of signal strength in the form of either a raw value, a likelihood ratio, or a confidence rating (Experiments 1 and 2) or a binary judgment along with either a verbal or verbal-visuospatial expression of confidence (Experiment 3). Aided sensitivity was benchmarked to the predictions of various statistical models of collaborative decision making. RESULTS AND CONCLUSION: Aided performance was suboptimal, matching the predictions of some of the least efficient models. Most importantly, performance was similar across cue formats. APPLICATION: Results indicate that changes to the format in which cues from a signal detection aid are rendered are unlikely to dramatically improve the efficiency of automation-aided decision making.

Does wearable device bring distraction closer to drivers? Comparing smartphones and Google Glass.

BACKGROUND: Head-up and wearable displays, such as Google Glass™, are sometimes marketed as safe in-vehicle alternatives to phone-based displays, as they allow drivers to receive messages without eye-off-the-road glances. However, head-up displays can still compromise driver performance (e.g., He et al., 2015b), as the distracting effect of interacting with any device will depend on the user's multitasking strategies. The present experiment examined drivers' interaction with a head-down smartphone display and a wearable head-up display. METHOD: Participants performed a simulated driving task while receiving and responding to text messages via smartphone or the head-mounted display (HMD) on the Google Glass™. Incoming messages were signaled by an auditory alert, and responses were made vocally. RESULTS: When using Google Glass, participants' responses were quicker than that of smartphone, and the time to engage in a task did not vary according to lane-keeping difficulty. Results suggest that a willingness to engage more readily in distracting tasks may offset the potential safety benefits of wearable devices.

Redundant-target processing is robust against changes to task load.

Monitoring visual displays while performing other tasks is commonplace in many operational environments. Although dividing attention between tasks can impair monitoring accuracy and response times, it is unclear whether it also reduces processing efficiency for visual targets. Thus, the current three experiments examined the effects of dual-tasking on target processing in the visual periphery. A total of 120 undergraduate students performed a redundant-target task either by itself (Experiment 1a) or in conjunction with a manual tracking task (Experiments 1b-3). Target processing efficiency was assessed using measures of workload resilience. Processing of redundant targets in Experiments 1-2 was less efficient than predicted by a standard parallel race model, giving evidence for limited-capacity, parallel processing. However, when stimulus characteristics forced participants to process targets in serial (Experiment 3), processing efficiency became super-capacity. Across the three experiments, dual-tasking had no effect on target processing efficiency. Results suggest that a central task slows target detection in the display periphery, but does not change the efficiency with which multiple concurrent targets are processed.

Effects of Task Difficulty and Display Format on Automation Usage Strategy: A Workload Capacity Analysis.

Objective An experiment used workload capacity analysis to quantify automation usage strategy across different task difficulty and display format types in a speeded task. Background Workload capacity measures the efficiency of concurrent information processing and can serve as a gauge of automation usage strategy in speeded decision tasks. The present study used workload capacity analysis to investigate automation usage strategy while information display format and task difficulty were manipulated. Method Subjects performed a speeded judgment task assisted by an automated aid that issued decision cues at varying onset times. Response time distributions were converted to measures of workload capacity. Results Two variants of a workload capacity measure, CzOR and CzAND, gave evidence that operators moderated their own decision times both in anticipation of and following the arrival of the aid's diagnosis under difficult task conditions regardless of display format. Conclusion Assistance from an automated decision aid may cause operators to delay their own responses in a speeded decision task, producing joint response time distributions that are slower than optimal. Application Even when it renders its own judgments quickly and with high accuracy, an automated decision aid may slow responses from a user. Automation designers should consider the relative costs and benefits of response accuracy and time when choosing whether and how to implement an automated decision aid.

Benchmarking Aided Decision Making in a Signal Detection Task.

OBJECTIVE: A series of experiments examined human operators' strategies for interacting with highly (93%) reliable automated decision aids in a binary signal detection task. BACKGROUND: Operators often interact with automated decision aids in a suboptimal way, achieving performance levels lower than predicted by a statistically ideal model of information integration. To better understand operators' inefficient use of decision aids, we compared participants' automation-aided performance levels with the predictions of seven statistical models of collaborative decision making. METHOD: Participants performed a binary signal detection task that asked them to classify random dot images as either blue or orange dominant. They made their judgments either unaided or with assistance from a 93% reliable automated decision aid that provided either graded (Experiments 1 and 3) or binary (Experiment 2) cues. We compared automation-aided performance with the predictions of seven statistical models of collaborative decision making, including a statistically optimal model and Robinson and Sorkin's contingent criterion model. RESULTS AND CONCLUSION: Automation-aided sensitivity hewed closest to the predictions of the two least efficient collaborative models, well short of statistically ideal levels. Performance was similar whether the aid provided graded or binary judgments. Model comparisons identified potential strategies by which participants integrated their judgments with the aid's. APPLICATION: Results lend insight into participants' automation-aided decision strategies and provide benchmarks for predicting automation-aided performance levels.

Workload Capacity: A Response Time-Based Measure of Automation Dependence.

OBJECTIVE: An experiment used the workload capacity measure C(t) to quantify the processing efficiency of human-automation teams and identify operators' automation usage strategies in a speeded decision task. BACKGROUND: Although response accuracy rates and related measures are often used to measure the influence of an automated decision aid on human performance, aids can also influence response speed. Mean response times (RTs), however, conflate the influence of the human operator and the automated aid on team performance and may mask changes in the operator's performance strategy under aided conditions. The present study used a measure of parallel processing efficiency, or workload capacity, derived from empirical RT distributions as a novel gauge of human-automation performance and automation dependence in a speeded task. METHOD: Participants performed a speeded probabilistic decision task with and without the assistance of an automated aid. RT distributions were used to calculate two variants of a workload capacity measure, COR(t) and CAND(t). RESULTS: Capacity measures gave evidence that a diagnosis from the automated aid speeded human participants' responses, and that participants did not moderate their own decision times in anticipation of diagnoses from the aid. CONCLUSION AND APPLICATION: Workload capacity provides a sensitive and informative measure of human-automation performance and operators' automation dependence in speeded tasks.

Texting while driving using Google Glass™: Promising but not distraction-free.

Texting while driving is risky but common. This study evaluated how texting using a Head-Mounted Display, Google Glass, impacts driving performance. Experienced drivers performed a classic car-following task while using three different interfaces to text: fully manual interaction with a head-down smartphone, vocal interaction with a smartphone, and vocal interaction with Google Glass. Fully manual interaction produced worse driving performance than either of the other interaction methods, leading to more lane excursions and variable vehicle control, and higher workload. Compared to texting vocally with a smartphone, texting using Google Glass produced fewer lane excursions, more braking responses, and lower workload. All forms of texting impaired driving performance compared to undistracted driving. These results imply that the use of Google Glass for texting impairs driving, but its Head-Mounted Display configuration and speech recognition technology may be safer than texting using a smartphone.

Lane keeping under cognitive load: performance changes and mechanisms.

OBJECTIVE: A pair of simulated driving experiments studied the effects of cognitive load on drivers' lane-keeping performance. BACKGROUND: Cognitive load while driving often reduces the variability of lane position. However, there is no agreement as to whether this effect should be interpreted as a performance loss, consistent with other effects of distraction on driving, or as an anomalous performance gain. METHOD: Participants in a high-fidelity driving simulator performed a lane-keeping task in lateral wind,with instructions to keep a steady lane position. Under high load conditions, participants performed a concurrent working memory task with auditory stimuli. Cross-spectral analysis measured the relationship between wind force and steering inputs. RESULTS: Cognitive load reduced the variability of lane position and increased the coupling between steering wheel position and crosswind strength. CONCLUSION: Although cognitive load disrupts driver performance in a variety of ways, it produces a performance gain in lane keeping.This effect appears to reflect drivers' efforts to protect lateral control against the risk of distraction, at the apparent neglect of other elements of driving performance. APPLICATION: Results may inform educational efforts to help drivers understand the risks of distraction and the inadequacies of compensatory driving strategies.

Metacognition of multitasking: How well do we predict the costs of divided attention?

Risky multitasking, such as texting while driving, may occur because people misestimate the costs of divided attention. In two experiments, participants performed a computerized visual-manual tracking task in which they attempted to keep a mouse cursor within a small target that moved erratically around a circular track. They then separately performed an auditory n-back task. After practicing both tasks separately, participants received feedback on their single-task tracking performance and predicted their dual-task tracking performance before finally performing the 2 tasks simultaneously. Most participants correctly predicted reductions in tracking performance under dual-task conditions, with a majority overestimating the costs of dual-tasking. However, the between-subjects correlation between predicted and actual performance decrements was near 0. This combination of results suggests that people do anticipate costs of multitasking, but have little metacognitive insight on the extent to which they are personally vulnerable to the risks of divided attention, relative to other people.

Great expectations: top-down attention modulates the costs of clutter and eccentricity.

An experiment and modeling effort examined interactions between bottom-up and top-down attentional control in visual alert detection. Participants performed a manual tracking task while monitoring peripheral display channels for alerts of varying salience, eccentricity, and spatial expectancy. Spatial expectancy modulated the influence of salience and eccentricity; alerts in low-probability locations engendered higher miss rates, longer detection times, and larger costs of visual clutter and eccentricity, indicating that top-down attentional control offset the costs of poor bottom-up stimulus quality. Data were compared to the predictions of a computational model of scanning and noticing that incorporates bottom-up and top-down sources of attentional control. The model accounted well for the overall pattern of miss rates and response times, predicting each of the observed main effects and interactions. Empirical results suggest that designers should expect the costs of poor bottom-up visibility to be greater for low expectancy signals, and that the placement of alerts within a display should be determined based on the combination of alert expectancy and response priority. Model fits suggest that the current model can serve as a useful tool for exploring a design space as a precursor to empirical data collection and for generating hypotheses for future experiments.

Change detection: training and transfer.

Observers often fail to notice even dramatic changes to their environment, a phenomenon known as change blindness. If training could enhance change detection performance in general, then it might help to remedy some real-world consequences of change blindness (e.g. failing to detect hazards while driving). We examined whether adaptive training on a simple change detection task could improve the ability to detect changes in untrained tasks for young and older adults. Consistent with an effective training procedure, both young and older adults were better able to detect changes to trained objects following training. However, neither group showed differential improvement on untrained change detection tasks when compared to active control groups. Change detection training led to improvements on the trained task but did not generalize to other change detection tasks.