The Effects of Captioning Errors, Background Noise, and Hearing Loss on Memory for Text-Captioned Speech.

PURPOSE: Previous studies have suggested that the negative effects of acoustic challenge on speech memory can be attenuated with assistive text captions, particularly among older adults with hearing impairment. However, no studies have systematically examined the effects of text-captioning errors, which are common in automated speech recognition (ASR) systems. METHOD: In two experiments, we examined memory for text-captioned speech (with and without background noise) when captions had no errors (control) or had one of three common ASR errors: substitution, deletion, or insertion errors. RESULTS: In both Experiment 1 (young adults with normal hearing) and Experiment 2 (older adults with varying hearing acuity), we observed similar additive effects of caption errors and background noise, such that increased background noise and the presence of captioning errors negatively impacted memory outcomes. Notably, the negative effects of captioning errors were largest among older adults with increased hearing thresholds, suggesting that older adults with hearing loss may show an increased reliance on text captions compared to adults with normal hearing. CONCLUSION: Our findings show that even a single-word error can be deleterious to memory for text-captioned speech, especially in older adults with hearing loss. Therefore, to produce the greatest benefit to memory, it is crucial that text captions are accurate.

Text Captioning Buffers Against the Effects of Background Noise and Hearing Loss on Memory for Speech.

OBJECTIVE: Everyday speech understanding frequently occurs in perceptually demanding environments, for example, due to background noise and normal age-related hearing loss. The resulting degraded speech signals increase listening effort, which gives rise to negative downstream effects on subsequent memory and comprehension, even when speech is intelligible. In two experiments, we explored whether the presentation of realistic assistive text captioned speech offsets the negative effects of background noise and hearing impairment on multiple measures of speech memory. DESIGN: In Experiment 1, young normal-hearing adults (N = 48) listened to sentences for immediate recall and delayed recognition memory. Speech was presented in quiet or in two levels of background noise. Sentences were either presented as speech only or as text captioned speech. Thus, the experiment followed a 2 (caption vs no caption) × 3 (no noise, +7 dB signal-to-noise ratio, +3 dB signal-to-noise ratio) within-subjects design. In Experiment 2, a group of older adults (age range: 61 to 80, N = 31), with varying levels of hearing acuity completed the same experimental task as in Experiment 1. For both experiments, immediate recall, recognition memory accuracy, and recognition memory confidence were analyzed via general(ized) linear mixed-effects models. In addition, we examined individual differences as a function of hearing acuity in Experiment 2. RESULTS: In Experiment 1, we found that the presentation of realistic text-captioned speech in young normal-hearing listeners showed improved immediate recall and delayed recognition memory accuracy and confidence compared with speech alone. Moreover, text captions attenuated the negative effects of background noise on all speech memory outcomes. In Experiment 2, we replicated the same pattern of results in a sample of older adults with varying levels of hearing acuity. Moreover, we showed that the negative effects of hearing loss on speech memory in older adulthood were attenuated by the presentation of text captions. CONCLUSIONS: Collectively, these findings strongly suggest that the simultaneous presentation of text can offset the negative effects of effortful listening on speech memory. Critically, captioning benefits extended from immediate word recall to long-term sentence recognition memory, a benefit that was observed not only for older adults with hearing loss but also young normal-hearing listeners. These findings suggest that the text captioning benefit to memory is robust and has potentially wide applications for supporting speech listening in acoustically challenging environments.