Early or synchronized gestures facilitate speech recall-a study based on motion capture data.

Introduction: Temporal co-ordination between speech and gestures has been thoroughly studied in natural production. In most cases gesture strokes precede or coincide with the stressed syllable in words that they are semantically associated with. Methods: To understand whether processing of speech and gestures is attuned to such temporal coordination, we investigated the effect of delaying, preposing or eliminating individual gestures on the memory for words in an experimental study in which 83 participants watched video sequences of naturalistic 3D-animated speakers generated based on motion capture data. A target word in the sequence appeared (a) with a gesture presented in its original position synchronized with speech, (b) temporally shifted 500 ms before or (c) after the original position, or (d) with the gesture eliminated. Participants were asked to retell the videos in a free recall task. The strength of recall was operationalized as the inclusion of the target word in the free recall. Results: Both eliminated and delayed gesture strokes resulted in reduced recall rates compared to synchronized strokes, whereas there was no difference between advanced (preposed) and synchronized strokes. An item-level analysis also showed that the greater the interval between the onsets of delayed strokes and stressed syllables in target words, the greater the negative effect was on recall. Discussion: These results indicate that speech-gesture synchrony affects memory for speech, and that temporal patterns that are common in production lead to the best recall. Importantly, the study also showcases a procedure for using motion capture-based 3D-animated speakers to create an experimental paradigm for the study of speech-gesture comprehension.

Development of an Auditory Passage Comprehension Task for Swedish Primary School Children of Cultural and Linguistic Diversity.

Purpose This study reports on the development of an auditory passage comprehension task for Swedish primary school children of cultural and linguistic diversity. It also reports on their performance on the task in quiet and in noise. Method Eighty-eight children aged 7-9 years and showing normal hearing participated. The children were divided into three groups based on presumed language exposure: 13 children were categorized as Swedish-speaking monolinguals, 19 children were categorized as simultaneous bilinguals, and 56 children were categorized as sequential bilinguals. No significant difference in working memory capacity was seen between the three language groups. Two passages and associated multiple-choice questions were developed. During development of the passage comprehension task, steps were taken to reduce the impact of culture-specific prior experience and knowledge on performance. This was achieved by using the story grammar principles, universal topics and plots, and simple language that avoided complex or unusual grammatical structures and words. Results The findings indicate no significant difference between the two passages and similar response distributions. Passage comprehension performance was significantly better in quiet than in noise, regardless of language exposure group. The monolinguals outperformed both simultaneous and sequential bilinguals in both listening conditions. Conclusions Because the task was designed to minimize the effect of cultural knowledge on auditory passage comprehension, this suggests that compared with monolinguals, both simultaneous and sequential bilinguals have a disadvantage in auditory passage comprehension. As expected, the findings demonstrate that noise has a negative effect on auditory passage comprehension. The magnitude of this effect does not relate to language exposure. The developed auditory passage comprehension task seems suitable for assessing auditory passage comprehension in primary school children of linguistic and cultural diversity.

Motion capture-based animated characters for the study of speech-gesture integration.

Digitally animated characters are promising tools in research studying how we integrate information from speech and visual sources such as gestures because they allow specific gesture features to be manipulated in isolation. We present an approach combining motion capture and 3D-animated characters that allows us to manipulate natural individual gesture strokes for experimental purposes, for example to temporally shift and present gestures in ecologically valid sequences. We exemplify how such stimuli can be used in an experiment investigating implicit detection of speech-gesture (a) synchrony, and discuss the general applicability of the workflow for research in this domain.

A virtual speaker in noisy classroom conditions: supporting or disrupting children's listening comprehension?

AIM: Seeing a speaker's face facilitates speech recognition, particularly under noisy conditions. Evidence for how it might affect comprehension of the content of the speech is more sparse. We investigated how children's listening comprehension is affected by multi-talker babble noise, with or without presentation of a digitally animated virtual speaker, and whether successful comprehension is related to performance on a test of executive functioning. MATERIALS AND METHODS: We performed a mixed-design experiment with 55 (34 female) participants (8- to 9-year-olds), recruited from Swedish elementary schools. The children were presented with four different narratives, each in one of four conditions: audio-only presentation in a quiet setting, audio-only presentation in noisy setting, audio-visual presentation in a quiet setting, and audio-visual presentation in a noisy setting. After each narrative, the children answered questions on the content and rated their perceived listening effort. Finally, they performed a test of executive functioning. RESULTS: We found significantly fewer correct answers to explicit content questions after listening in noise. This negative effect was only mitigated to a marginally significant degree by audio-visual presentation. Strong executive function only predicted more correct answers in quiet settings. CONCLUSIONS: Altogether, our results are inconclusive regarding how seeing a virtual speaker affects listening comprehension. We discuss how methodological adjustments, including modifications to our virtual speaker, can be used to discriminate between possible explanations to our results and contribute to understanding the listening conditions children face in a typical classroom.

Listening Comprehension and Listening Effort in the Primary School Classroom.

In the primary school classroom, children are exposed to multiple factors that combine to create adverse conditions for listening to and understanding what the teacher is saying. Despite the ubiquity of these conditions, there is little knowledge concerning the way in which various factors combine to influence listening comprehension and the effortfulness of listening. The aim of the present study was to investigate the combined effects of background noise, voice quality, and visual cues on children's listening comprehension and effort. To achieve this aim, we performed a set of four well-controlled, yet ecologically valid, experiments with 245 eight-year-old participants. Classroom listening conditions were simulated using a digitally animated talker with a dysphonic (hoarse) voice and background babble noise composed of several children talking. Results show that even low levels of babble noise interfere with listening comprehension, and there was some evidence that this effect was reduced by seeing the talker's face. Dysphonia did not significantly reduce listening comprehension scores, but it was considered unpleasant and made listening seem difficult, probably by reducing motivation to listen. We found some evidence that listening comprehension performance under adverse conditions is positively associated with individual differences in executive function. Overall, these results suggest that multiple factors combine to influence listening comprehension and effort for child listeners in the primary school classroom. The constellation of these room, talker, modality, and listener factors should be taken into account in the planning and design of educational and learning activities.

Domiciliary VR-Based Therapy for Functional Recovery and Cortical Reorganization: Randomized Controlled Trial in Participants at the Chronic Stage Post Stroke.

BACKGROUND: Most stroke survivors continue to experience motor impairments even after hospital discharge. Virtual reality-based techniques have shown potential for rehabilitative training of these motor impairments. Here we assess the impact of at-home VR-based motor training on functional motor recovery, corticospinal excitability and cortical reorganization. OBJECTIVE: The aim of this study was to identify the effects of home-based VR-based motor rehabilitation on (1) cortical reorganization, (2) corticospinal tract, and (3) functional recovery after stroke in comparison to home-based occupational therapy. METHODS: We conducted a parallel-group, controlled trial to compare the effectiveness of domiciliary VR-based therapy with occupational therapy in inducing motor recovery of the upper extremities. A total of 35 participants with chronic stroke underwent 3 weeks of home-based treatment. A group of subjects was trained using a VR-based system for motor rehabilitation, while the control group followed a conventional therapy. Motor function was evaluated at baseline, after the intervention, and at 12-weeks follow-up. In a subgroup of subjects, we used Navigated Brain Stimulation (NBS) procedures to measure the effect of the interventions on corticospinal excitability and cortical reorganization. RESULTS: Results from the system's recordings and clinical evaluation showed significantly greater functional recovery for the experimental group when compared with the control group (1.53, SD 2.4 in Chedoke Arm and Hand Activity Inventory). However, functional improvements did not reach clinical significance. After the therapy, physiological measures obtained from a subgroup of subjects revealed an increased corticospinal excitability for distal muscles driven by the pathological hemisphere, that is, abductor pollicis brevis. We also observed a displacement of the centroid of the cortical map for each tested muscle in the damaged hemisphere, which strongly correlated with improvements in clinical scales. CONCLUSIONS: These findings suggest that, in chronic stages, remote delivery of customized VR-based motor training promotes functional gains that are accompanied by neuroplastic changes. TRIAL REGISTRATION: International Standard Randomized Controlled Trial Number NCT02699398 (Archived by ClinicalTrials.gov at https://clinicaltrials.gov/ct2/show/NCT02699398?term=NCT02699398&rank=1).

Erratum to: The visual amplification of goal-oriented movements counteracts acquired non-use in hemiparetic stroke patients.

Unfortunately, in the original version of this article [1] the sentence "This project was supported through ERC project cDAC (FP7-IDEAS-ERC 341196), EC H2020 project socSMCs (H2020-EU.1.2.2. 641321) and MINECO project SANAR (Gobierno de España)" was missing from the acknowledgements.The acknowledgements have been correctly included in full in this erratum.

The visual amplification of goal-oriented movements counteracts acquired non-use in hemiparetic stroke patients.

BACKGROUND: Stroke-induced impairments result from both primary and secondary causes, i.e. damage to the brain and the acquired non-use of the impaired limbs. Indeed, stroke patients often under-utilize their paretic limb despite sufficient residual motor function. We hypothesize that acquired non-use can be overcome by reinforcement-based training strategies. METHODS: Hemiparetic stroke patients (n = 20, 11 males, 9 right-sided hemiparesis) were asked to reach targets appearing in either the real world or in a virtual environment. Sessions were divided into 3 phases: baseline, intervention and washout. During the intervention the movement of the virtual representation of the patients' paretic limb was amplified towards the target. RESULTS: We found that the probability of using the paretic limb during washout was significantly higher in comparison to baseline. Patients showed generalization of these results by displaying a more substantial workspace in real world task. These gains correlated with changes in effector selection patterns. CONCLUSIONS: The amplification of the movement of the paretic limb in a virtual environment promotes the use of the paretic limb in stroke patients. Our findings indicate that reinforcement-based therapies may be an effective approach for counteracting learned non-use and may modulate motor performance in the real world.

Adaptive rehabilitation gaming system: on-line individualization of stroke rehabilitation.

The effects of stroke differ considerably in degree and symptoms for different patients. It has been shown that specific, individualized and varied therapy favors recovery. The Rehabilitation Gaming System (RGS) is a Virtual Reality (VR) based rehabilitation system designed following these principles. We have developed two algorithms to control the level of task difficulty that a user of the RGS is exposed to, as well as providing controlled variation in the therapy. In this paper, we compare the two algorithms by running numerical simulations and a study with healthy subjects. We show that both algorithms allow for individualization of the challenge level of the task. Further, the results reveal that the algorithm that iteratively learns a user model for each subject also allows a high variation of the task.