Near-Perfect Automation: Investigating Performance, Trust, and Visual Attention Allocation.

OBJECTIVE: Assess performance, trust, and visual attention during the monitoring of a near-perfect automated system. BACKGROUND: Research rarely attempts to assess performance, trust, and visual attention in near-perfect automated systems even though they will be relied on in high-stakes environments. METHODS: Seventy-three participants completed a 40-min supervisory control task where they monitored three search feeds. All search feeds were 100% reliable with the exception of two automation failures: one miss and one false alarm. Eye-tracking and subjective trust data were collected. RESULTS: Thirty-four percent of participants correctly identified the automation miss, and 67% correctly identified the automation false alarm. Subjective trust increased when participants did not detect the automation failures and decreased when they did. Participants who detected the false alarm had a more complex scan pattern in the 2 min centered around the automation failure compared with those who did not. Additionally, those who detected the failures had longer dwell times in and transitioned to the center sensor feed significantly more often. CONCLUSION: Not only does this work highlight the limitations of the human when monitoring near-perfect automated systems, it begins to quantify the subjective experience and attentional cost of the human. It further emphasizes the need to (1) reevaluate the role of the operator in future high-stakes environments and (2) understand the human on an individual level and actively design for the given individual when working with near-perfect automated systems. APPLICATION: Multiple operator-level measures should be collected in real-time in order to monitor an operator's state and leverage real-time, individualized assistance.

Workload transition rate matters: Evidence from growth curve modeling.

This research examined three specific gaps in the workload transition literature: (1) the impact of workload transition rate, (2) the applicability of current theoretical explanations, and (3) the variability of performance overall and over time. Sixty Naval flight students multitasked in an unmanned aerial vehicle control testbed and workload transitioned at three rates: slow, medium, or fast. Response time and accuracy were analyzed via growth curve modeling. Slow transitions had the largest decline in performance over time. Medium transitions had some of the slowest, but most accurate and consistent performance. Fast transitions had some of the fastest, but least accurate performance. However, all performance trends significantly varied, suggesting multiple theoretical explanations may apply and performance may also depend on the individual. Design guidance on how to maximize performance goals with transition rate is provided, but future research needs to study the theoretical explanations and impact of individual differences further.

An investigation of cardiac vagal tone over time and its relation to vigilance performance: a growth curve modeling approach.

Introduction: Research over the last couple of decades has demonstrated a relationship between psychophysiological measures, specifically cardiac functions, and cognitive performance. Regulation of the cardiac system under parasympathetic control is commonly referred to as cardiac vagal tone and is associated with the regulation of cognitive and socioemotional states. The goal of the current study was to capture the dynamic relationship between cardiac vagal tone and performance in a vigilance task. Method/Results: We implemented a longitudinal growth curve modeling approach which unveiled a relationship between cardiac vagal tone and vigilance that was non-monotonic and dependent upon each person. Discussion: The findings suggest that cardiac vagal tone may be a process-based physiological measure that further explains how the vigilance decrement manifests over time and differs across individuals. This contributes to our understanding of vigilance by modeling individual differences in cardiac vagal tone changes that occur over the course of the vigilance task.

Scan-based eye tracking measures are predictive of workload transition performance.

Given there is no unifying theory or design guidance for workload transitions, this work investigated how visual attention allocation patterns could inform both topics, by understanding if scan-based eye tracking metrics could predict workload transition performance trends in a context-relevant domain. The eye movements of sixty Naval flight students were tracked as workload transitioned at a slow, medium, and fast pace in an unmanned aerial vehicle testbed. Four scan-based metrics were significant predictors across the different growth curve models of response time and accuracy. Stationary gaze entropy (a measure of how dispersed visual attention transitions are across tasks) was predictive across all three transition rates. The other three predictive scan-based metrics captured different aspects of visual attention, including its spread, directness, and duration. The findings specify several missing details in both theory and design guidance, which is unprecedented, and serves as a basis of future workload transition research.

An examination of relations between baseline pupil measures and cognitive abilities.

Examining individual differences in pupil size and pupillary dynamics have revealed important insights into the nature of individual differences in cognitive abilities like working memory capacity, long-term memory, attention control, and fluid intelligence. These findings are often tied to the locus coeruleus-norepinephrine (LC-NE) system, as this system has a tight temporal correlation with pupil diameter. Some recent research has demonstrated positive correlations between resting pupil size and cognitive ability, specifically fluid intelligence. The present study attempted to replicate such relations. Across three studies, a large sample of participants (N = 845) completed batteries of cognitive ability measures and measures of resting pupil size and pupillary hippus (fluctuations in pupil diameter). The cognitive measures comprised tasks previously used to measure attention control, visual short-term memory capacity, fluid intelligence, working memory capacity, and visuospatial ability. At the factor level, cognitive ability and pupil size correlated near zero. We did observe some limited evidence for a negative correlation between resting pupillary hippus and cognitive ability. Given the null findings in the present data, we encourage further replication of relations between resting pupil measures and cognitive abilities before making any strong theoretical conclusions about such relations.

Limitations of current spatial ability testing for military aviators.

Spatial ability has long been considered an important attribute when identifying military aviators. This paper examines the Direction Orientation Task (DOT), which is currently used by the US military to assess spatial ability in aviation applicants. Several limitations of the test, such as a limited number of trials and the availability of practice trials online, make it subject to potential ceiling effects. The paper presents historical data of all Naval Aviator applicants over a six-year time period and revealed that 22% of applicants answered 90% or more of the questions correctly. Furthermore, test performance has significantly increased in the years since the test was first administered and there is evidence that DOT is no longer contributing incremental validity. A follow-up empirical study looked at DOT performance and strategy in a group of military student aviators and student air traffic controllers. The results of the empirical investigation reveal that the use of an analytic strategy was associated with higher performance on the DOT, whereas the use of a spatial strategy was not associated with performance. Taken together, the improved performance data over time and the data on strategy use suggest the test's ability to measure spatial ability may be diminishing, ultimately reducing its construct and incremental validity. This is problematic and should be addressed, since the DOT is the only measure of spatial ability used by the Navy to assess aviation applicants.

Detecting automation failures in a simulated supervisory control environment.

The goal of this research was to determine how individuals perform and allocate their visual attention when monitoring multiple automated displays that differ in automation reliability. Ninety-six participants completed a simulated supervisory control task where each automated display had a different level of reliability (namely 70%, 85% and 95%). In addition, participants completed a high and low workload condition. The performance data revealed that (1) participants' failed to detect automation misses approximately 2.5 times more than automation false alarms, (2) participants' had worse automation failure detection in the high workload condition and (3) participant automation failure detection remained mostly static across reliability. The eye tracking data revealed that participants spread their attention relatively equally across all three of the automated displays for the duration of the experiment. Together, these data support a system-wide trust approach as the default position of an individual monitoring multiple automated displays. Practitioner Summary: Given the rapid growth of automation throughout the workforce, there is an immediate need to better understand how humans monitor multiple automated displays concurrently. The data in this experiment support a system-wide trust approach as the default position of an individual monitoring multiple automated displays. Abbreviations: DoD: Department of Defense; UA: unmanned aircraft; SCOUT: Supervisory Control Operations User Testbed; UAV: unmanned aerial vehicle; AOI: areas of interest.

Ongoing task delays affect prospective memory more powerfully than filler task delays.

The effect of delay on prospective memory (PM) is mixed. Research has typically shown that PM either decreases or remains unchanged as the time increases between intention formation and encounter with a PM cue. However, the results of one study demonstrated that PM sometimes increases with increasing delays (Hicks, Marsh, & Russell, 2000). Hicks et al. hypothesised that increasing the delay may afford an opportunity for people to spontaneously rehearse the intention, or to be reminded of the intention. In the present work, we tested delays of 6 minutes, 21 minutes, and 36 minutes. Two factors were orthogonally manipulated between-subjects. One was the duration of the filler task that came between intention formation and the beginning of the ongoing task in which PM cues were embedded. The second was the duration of the ongoing task prior to the presentation of the first PM cue. Lengthening the ongoing task delay decreased PM. However, lengthening the filler task nominally increased PM. These results suggest that delays within the ongoing task replicate the effects traditionally found in retrospective memory work. In contrast, delays between intention formation and the beginning of the ongoing task may not have straightforward effects on PM retrieval.

Source memory for unrecognized items: predictions from multivariate signal detection theory.

We report three experiments investigating source memory for words that were called "new" on a recognition test. In each experiment, participants could accurately specify the source of words that they failed to recognize. Results also demonstrated that source memory for unrecognized items varied with the bias to respond "old" in recognition decisions: Participants displayed unrecognized source memory when they were told that 25% of the recognition test words were old (promoting conservative responding) but not when they were told that 75% of the test words were old (promoting liberal responding). Our results were successfully predicted by a multivariate signal detection approach to recognition/source memory.