Gesture in the eye of the beholder: An eye-tracking study on factors determining the attention for gestures produced by people with aphasia.

Co-speech hand gestures are an ubiquitous form of nonverbal communication, which can express additional information that is not present in speech. Hand gestures may become more relevant when verbal production is impaired, as in speakers with post-stroke aphasia. In fact, speakers with aphasia produce more gestures than non-brain damaged speakers. Furthermore, there is evidence to suggest that speakers with aphasia produce gestures that convey information essential to understand their communication. In the present study, we addressed the question whether these gestures catch the attention of their addressees. Healthy volunteers (observers) watched short video clips while their eye movements were recorded. These video clips featured speakers with aphasia and non-brain damaged speakers describing two different scenarios (buying a sweater or having witnessed an accident). Our results show that hand gestures produced by speakers with aphasia are on average attended to longer than gestures produced by non-brain damaged speakers. This effect was significant even when we controlled for the longer duration of the gestural movements in speakers with aphasia. Further, the amount of information in speech was also correlated with gesture attention. That is gestures produced by speakers with less informative speech were attended to more frequently. In conclusion, our findings suggest that listeners reallocate their attention and focus more strongly on non-verbal information from co-speech gestures if speech comprehension becomes challenging due to the speaker's verbal production deficits. These findings support a communicative function of co-speech gestures and advocate for instructing people with aphasia to communicate things in the form of gestures that cannot be expressed verbally because interlocutors take notice of these gestures.

The comprehensibility of pantomimes produced by people with aphasia.

BACKGROUND: People with aphasia (PWA) use pantomime, gesture in absence of speech, differently from non-brain-damaged people (NBDP). AIMS: To evaluate through an exploratory study the comprehensibility of PWA's pantomimes and to find out whether they can compensate for information PWA are unable to convey in speech. METHODS & PROCEDURES: A total of 273 naïve observers participated in one of two judgement tasks: forced-choice and open-ended questions. These were used to determine the comprehensibility of pantomimes produced to depict objects by PWA as compared with NBDP. Furthermore, we compared the information conveyed in pantomime with the information in speech. We looked into factors influencing pantomime's comprehensibility: individual factors, manner of depiction and information needed to be depicted. OUTCOME & RESULTS: Although comprehensibility scores for PWA's pantomimes were lower than for those produced by NBDP, all PWA were able to convey information in pantomime that they could not convey in speech. Comprehensibility of pantomimes was predicted by apraxia. The inability to use the right hand related to slightly lower comprehensibility scores. Objects for which individuals depicted its use were best understood. CONCLUSION & IMPLICATIONS: Our findings highlight the potential benefit of pantomime for clinical practice. Pantomimes, even though sometimes impaired, can convey information that PWA cannot convey in speech. Clinical implications are discussed.

Production and Comprehension of Pantomimes Used to Depict Objects.

Pantomime, gesture in absence of speech, has no conventional meaning. Nevertheless, individuals seem to be able to produce pantomimes and derive meaning from pantomimes. A number of studies has addressed the use of co-speech gesture, but little is known on pantomime. Therefore, the question of how people construct and understand pantomimes arises in gesture research. To determine how people use pantomimes, we asked participants to depict a set of objects using pantomimes only. We annotated what representation techniques people produced. Furthermore, using judgment tasks, we assessed the pantomimes' comprehensibility. Analyses showed that similar techniques were used to depict objects across individuals. Objects with a default depiction method were better comprehended than objects for which there was no such default. More specifically, tools and objects depicted using a handling technique were better understood. The open-answer experiment showed low interpretation accuracy. Conversely, the forced-choice experiment showed ceiling effects. These results suggest that across individuals, similar strategies are deployed to produce pantomime, with the handling technique as the apparent preference. This might indicate that the production of pantomimes is based on mental representations which are intrinsically similar. Furthermore, pantomime conveys semantically rich, but ambiguous, information, and its interpretation is much dependent on context. This pantomime database is available online: https://dataverse.nl/dataset.xhtml?persistentId=hdl:10411/QZHO6M. This can be used as a baseline with which we can compare clinical groups.

How Much Information Do People With Aphasia Convey via Gesture?

PURPOSE: People with aphasia (PWA) face significant challenges in verbally expressing their communicative intentions. Different types of gestures are produced spontaneously by PWA, and a potentially compensatory function of these gestures has been discussed. The current study aimed to investigate how much information PWA communicate through 3 types of gesture and the communicative effectiveness of such gestures. METHOD: Listeners without language impairment rated the information content of short video clips taken from PWA in conversation. Listeners were asked to rate communication within a speech-only condition and a gesture + speech condition. RESULTS: The results revealed that the participants' interpretations of the communicative intentions expressed in the clips of PWA were significantly more accurate in the gesture + speech condition for all tested gesture types. CONCLUSION: It was concluded that all 3 gesture types under investigation contributed to the expression of semantic meaning communicated by PWA. Gestures are an important communicative means for PWA and should be regarded as such by their interlocutors. Gestures have been shown to enhance listeners' interpretation of PWA's overall communication.

Pantomime Production by People With Aphasia: What Are Influencing Factors?

PURPOSE: The present article aimed to inform clinical practice on whether people with aphasia (PWA) deploy pantomime techniques similarly to participants without brain damage (PWBD) and if not, what factors influence these differences. METHOD: We compared 38 PWA to 20 PWBD in their use of 6 representation techniques (handling, enact, object, shape, deictic, and other) when pantomiming objects, and determined whether PWA used the same defaults as PWBD. We assessed the influence of (non-)dominant arm use, ideomotor apraxia, semantic processing, aphasia severity, and oral naming. RESULTS: PWA used various pantomime techniques. Enact, deictic, and other were used infrequently. No differences were found for the use of shape techniques, but PWA used fewer handling and object techniques than PWBD and they did not use these for the same objects as PWBD did. No influence was found for (non-)dominant arm use. All other variables correlated with the use of handling, object, and defaults. CONCLUSION: In our study, PWA were able to use various pantomime techniques. As a group, they used these techniques differently from PWBD and relied more heavily on the use of shape techniques. This was not influenced by a hemiparesis, but seemed dependent on semantic processing. Clinical implications are discussed.

Should pantomime and gesticulation be assessed separately for their comprehensibility in aphasia? A case study.

BACKGROUND: Gesticulation (gestures accompanying speech) and pantomime (gestures in the absence of speech) can each be comprehensible. Little is known about the differences between these two gesture modes in people with aphasia. AIMS: To discover whether there are differences in the communicative use of gesticulation and pantomime in QH, a person with severe fluent aphasia. METHODS & PROCEDURES: QH performed two tasks: naming objects and retelling a story. He did this once in a verbal condition (enabling gesticulation) and once in a pantomime condition. For both conditions, the comprehensibility of gestures was analysed in a forced-choice task by naïve judges. Secondly, a comparison was made between QH and healthy controls for the representation techniques used. OUTCOMES & RESULTS: Pantomimes produced by QH for naming objects were significantly more comprehensible than chance, whereas his gesticulation was not. For retelling a story the opposite pattern was found. When naming objects QH gesticulated much more than did healthy controls. His pantomimes for this task were simpler than those used by the control group. For retelling a story no differences were found. CONCLUSIONS & IMPLICATIONS: Although QH did not make full use of each gesture modes' potential, both did contribute to QH's comprehensibility. Crucially, the benefits of each mode differed across tasks. This implies that both gesture modes should be taken into account separately in models of speech and gesture production and in clinical practice for different communicative settings.