Early or synchronized gestures facilitate speech recall-a study based on motion capture data.

Introduction: Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with. Methods: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall. Results: Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall. Discussion: These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.

Motion capture-based animated characters for the study of speech-gesture integration.

Digitally animated characters are promising tools in research studying how we integrate information from speech and visual sources such as gestures because they allow specific gesture features to be manipulated in isolation. We present an approach combining motion capture and 3D-animated characters that allows us to manipulate natural individual gesture strokes for experimental purposes, for example to temporally shift and present gestures in ecologically valid sequences. We exemplify how such stimuli can be used in an experiment investigating implicit detection of speech-gesture (a) synchrony, and discuss the general applicability of the workflow for research in this domain.

A virtual speaker in noisy classroom conditions: supporting or disrupting children's listening comprehension?

AIM: Seeing a speaker's face facilitates speech recognition, particularly under noisy conditions. Evidence for how it might affect comprehension of the content of the speech is more sparse. We investigated how children's listening comprehension is affected by multi-talker babble noise, with or without presentation of a digitally animated virtual speaker, and whether successful comprehension is related to performance on a test of executive functioning. MATERIALS AND METHODS: We performed a mixed-design experiment with 55 (34 female) participants (8- to 9-year-olds), recruited from Swedish elementary schools. The children were presented with four different narratives, each in one of four conditions: audio-only presentation in a quiet setting, audio-only presentation in noisy setting, audio-visual presentation in a quiet setting, and audio-visual presentation in a noisy setting. After each narrative, the children answered questions on the content and rated their perceived listening effort. Finally, they performed a test of executive functioning. RESULTS: We found significantly fewer correct answers to explicit content questions after listening in noise. This negative effect was only mitigated to a marginally significant degree by audio-visual presentation. Strong executive function only predicted more correct answers in quiet settings. CONCLUSIONS: Altogether, our results are inconclusive regarding how seeing a virtual speaker affects listening comprehension. We discuss how methodological adjustments, including modifications to our virtual speaker, can be used to discriminate between possible explanations to our results and contribute to understanding the listening conditions children face in a typical classroom.

Is children's listening effort in background noise influenced by the speaker's voice quality?

The present study aims at exploring the influence of voice quality on listening effort in children performing a language comprehension test with sentences of increasing difficulty. Listening effort is explored in relation to gender ( = cisgender). The study has a between-groups design. Ninety-three mainstreamed children aged 8;2 to 9;3 with typical language development participated. The children were randomly assigned to two groups (n = 46/47) with equal allocation of boys and girls and for the analysis to four groups depending of gender and voice condition. Working memory capacity and executive functions were tested in quiet. A digital version of a language comprehension test (the TROG-2) was used to measure the effect of voice quality on listening effort, measured as response time in a forced-choice paradigm. The groups listened to sentences through recordings of the same female voice, one group with a typical voice and one with a dysphonic voice, both in competing multi-talker babble noise. Response times were logged after a time buffer between the sentence-ending and indication of response. There was a significant increase in response times with increased task difficulty and response times between the two voice conditions differed significantly. The girls in the dysphonic condition were slower with increasing task difficulty. A dysphonic voice clearly adds to the noise burden and listening effort is greater in girls than in boys when the teacher speaks with dysphonic voice in a noisy background. These findings might mirror gender differences as for coping strategies in challenging contexts and have important implications for education.

Children's Subjective Ratings and Opinions of Typical and Dysphonic Voice After Performing a Language Comprehension Task in Background Noise.

OBJECTIVE: Previous studies suggest that dysphonic voices are perceived more negatively than typical voices, and it is possible that vocal perception influences academic achievement and comprehension. This study assessed children's subjective ratings and opinions of a typical and dysphonic voice after performing a language comprehension task in background noise. It also explored if subjective ratings were related to performance in the comprehension task. STUDY DESIGN: Ninety-three 8-year-old children participated and were randomly allocated to two groups. METHODS: A digital version of the Test for Reception of Grammar, version 2 (TROG-2) was used. In TROG-2, the child listens to a sentence and then, out of four possible pictures, selects the picture that corresponds to the sentence content. Each sentence assesses a specific grammatical construction. Test instructions were recorded using one speaker with two different voice qualities (typical voice and provoked dysphonic voice). One group of children performed TROG-2 with the typical voice and the other with the dysphonic voice, both presented in a multitalker babble noise using earphones. All children rated their subjective perception of the voice. RESULTS: The dysphonic voice received poorer ratings than the typical voice. Children making more positive ratings of the voice performed better on earlier blocks in TROG-2, but only in the typical voice group. CONCLUSIONS: Children perceived the dysphonic voice more negatively even when listening to the same speaker. Performance is related to the perception of the voice, but only when the perceptual load of the task-relevant stimuli is sufficiently low.

Are children with stronger cognitive capacity more or less disturbed by classroom noise and dysphonic teachers?

PURPOSE: This study aimed to explore if dysphonic voice quality affects children's performance on a language comprehension test, the Test for Reception of Grammar-2 (TROG-2), performed in simultaneous background noise (non-semantic babble). A further aim was to investigate the role of Working Memory Capacity (WMC) and Executive Functioning (EF) in coping with the voice against a background of babble conditions. METHOD: Ninety-three mainstreamed 8 year old children with typical language development were tested for WMC and EF. Two groups of children (n = 47/46) were formed and presented with recordings of TROG-2 instructions read by one female speaker: one group was presented with recordings with induced dysphonic voice quality, the other with recordings of typical voice. Both groups listened to the voice recordings in competing babble noise at a Signal-to-Noise Ratio of + 10 dB. RESULT: Significant differences were found for the interaction between cognitive capacity and the TROG-2 results in relation to the voice conditions. In the dysphonic voice condition, children with better WMC results scored higher at the easier comprehension tasks. In the typical voice condition, children with better WMC and EF results scored higher on the more difficult tasks. Seventeen per cent of the variance for the TROG-2 results was explained by the WMC and EF results. There was no overall effect on the children's performance depending of voice condition. CONCLUSION: The effect of the speaker's voice quality on children's performance varies depending on the prevalence of background babble noise and on the task demands. The dysphonic voice and babble noise seem to demand allocation of cognitive capacities at the cost of language comprehension.

Does the speaker's voice quality influence children's performance on a language comprehension test?

PURPOSE: A small number of studies have explored children's perception of speakers' voice quality and its possible influence on language comprehension. The aim of this explorative study was to investigate the relationship between the examiner's voice quality, the child's performance on a digital version of a language comprehension test, the Test for Reception of Grammar (TROG-2), and two measures of cognitive functioning. METHOD: The participants were (n = 86) mainstreamed 8-year old children with typical language development. Two groups of children (n = 41/45) were presented with the TROG-2 through recordings of one female speaker: one group was presented with a typical voice and the other with a simulated dysphonic voice. RESULT: Significant associations were found between executive functioning and language comprehension. The results also showed that children listening to the dysphonic voice achieved significantly lower scores for more difficult sentences ("the man but not the horse jumps") and used more self-corrections on simpler sentences ("the girl is sitting"). CONCLUSION: Findings suggest that a dysphonic speaker's voice may force the child to allocate capacity to the processing of the voice signal at the expense of comprehension. The findings have implications for clinical and research settings where standardized language tests are used.

The slower the better? Does the speaker's speech rate influence children's performance on a language comprehension test?

The aim of this study was to examine the effects of speech rate on children's performance on a widely used language comprehension test, the Test for Reception of Grammar, version 2 (TROG'2), and to explore how test performance interacts with task difficulty and with the child's working memory capacity. Participants were 102 typically-developing Swedish-speaking children randomly assigned to one of the three conditions; the TROG'2 sentences spoken by a speech-language pathologist with slow, normal or fast speech rate. Results showed that the fast speech rate had a negative effect on the TROG'2 scores and that slow rate was more beneficial in general. However, for more difficult tasks the beneficial effect of slow speech was only pronounced for children with better scores on a working memory task. The interpretation is that slow speech is particularly helpful when children do not yet fully master a task but are just about to grasp it. These results emphasize the necessity of careful considerations of the role dynamic aspects of examiner's speech might play in test administration and favour digitalized procedures in standardized language comprehension assessment.