Measuring Individual Differences in the Perfect Automation Schema.

OBJECTIVE: A self-report measure of the perfect automation schema (PAS) is developed and tested. BACKGROUND: Researchers have hypothesized that the extent to which users possess a PAS is associated with greater decreases in trust after users encounter automation errors. However, no measure of the PAS currently exists. We developed a self-report measure assessing two proposed PAS factors: high expectations and all-or-none thinking about automation performance. METHOD: In two studies, participants responded to our PAS measure, interacted with imperfect automated aids, and reported trust. RESULTS: Each of the two PAS measure factors demonstrated fit to the hypothesized factor structure and convergent and discriminant validity when compared with propensity to trust machines and trust in a specific aid. However, the high expectations and all-or-none thinking scales showed low intercorrelations and differential relationships with outcomes, suggesting that they might best be considered two separate constructs rather than two subfactors of the PAS. All-or-none thinking had significant associations with decreases in trust following aid errors, whereas high expectations did not. Results therefore suggest that the all-or-none thinking scale may best represent the PAS construct. CONCLUSION: Our PAS measure (specifically, the all-or-none thinking scale) significantly predicted the severe trust decreases thought to be associated with high PAS. Further, it demonstrated acceptable psychometric properties across two samples. APPLICATION: This measure may be used in future work to assess levels of PAS in users of automated systems in either research or applied settings.

Are well-calibrated users effective users? Associations between calibration of trust and performance on an automation-aided task.

OBJECTIVE: We present alternative operationalizations of trust calibration and examine their associations with predictors and outcomes. BACKGROUND: It is thought that trust calibration (correspondence between aid reliability and user trust in the aid) is a key to effective human-automation performance. We propose that calibration can be operationalized in three ways. Perceptual accuracy is the extent to which the user perceives the aid's reliability accurately at one point in time. Perceptual sensitivity and trust sensitivity reflect user adjustment of perceived reliability and trust as the aid's actual reliability changes over time. METHOD: One hundred fifty-five students completed an X-ray screening task with an automated screener. Awareness of the aid's accuracy trajectory and error type was examined as predictors, and task performance and aid failure detection were examined as outcomes. RESULTS: Awareness of accuracy trajectory was significantly associated with all three operationalizations of calibration, but awareness of error type was not when considered in conjunction with accuracy trajectory. Contrary to expectations, only perceptual accuracy was significantly associated with task performance and failure detection, and combined, the three operationalizations accounted for only 9% and 4% of the variance in these outcomes, respectively. CONCLUSION: Our results suggest that the potential importance of trust calibration warrants further examination. Moderators may exist. APPLICATION: Users who were better able to perform the task unaided were better able to identify and correct aid failure, suggesting that user task training and expertise may benefit human-automation performance.