Metaphor processing is supramodal semantic processing: The role of the bilateral lateral temporal regions in multimodal communication.

This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying gestures ("multimodal metaphors"). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, "She felt bad" while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, "Her spirits fell" while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.

Coordinated Collaboration and Nonverbal Social Interactions: A Formal and Functional Analysis of Gaze, Gestures, and Other Body Movements in a Contemporary Dance Improvisation Performance.

This study presents a microanalysis of what information performers "give" and "give off" to each other via their bodies during a contemporary dance improvisation. We compare what expert performers and non-performers (sufficiently trained to successfully perform) do with their bodies during a silent, multiparty improvisation exercise, in order to identify any differences and to provide insight into nonverbal communication in a less conventional setting. The coordinated collaboration of the participants (two groups of six) was examined in a frame-by-frame analysis focusing on all body movements, including gaze shifts as well as the formal and functional movement units produced in the head-face, upper-, and lower-body regions. The Methods section describes in detail the annotation process and inter-rater agreement. The results of this study indicate that expert performers during the improvisation are in "performance mode" and have embodied other social cognitive strategies and skills (e.g., endogenous orienting, gaze avoidance, greater motor control) that the non-performers do not have available. Expert performers avoid using intentional communication, relying on information to be inferentially communicated in order to coordinate collaboratively, with silence and stillness being construed as meaningful in that social practice and context. The information that expert performers produce is quantitatively less (i.e., producing fewer body movements) and qualitatively more inferential than intentional compared to a control group of non-performers, which affects the quality of the performance.

Handling or being the concept: An fMRI study on metonymy representations in coverbal gestures.

In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses attention on a specific aspect of the entity. Whereas metaphors, another figurative language device, are substitutions based on similarity, metonymy involves substitutions based on associations. Both are figures of speech but are also expressed in coverbal gestures during multimodal communication. The closest neuropsychological studies of metonymy in gestures have been nonlinguistic tool-use, illustrated by the classic apraxic problem of body-part-as-object (BPO, equivalent to an internal metonymy representation of the tool) vs. pantomimed action (external metonymy representation of the absent object/tool). Combining these research domains with concepts in cognitive linguistic research on gestures, we conducted an fMRI study to investigate metonymy resolution in coverbal gestures. Given the greater difficulty in developmental and apraxia studies, perhaps explained by the more complex semantic inferencing involved for external metonymy than for internal metonymy representations, we hypothesized that external metonymy resolution requires greater processing demands and that the neural resources supporting metonymy resolution would modulate regions involved in semantic processing. We found that there are indeed greater activations for external than for internal metonymy resolution in the temporoparietal junction (TPJ). This area is posterior to the lateral temporal regions recruited by metaphor processing. Effective connectivity analysis confirmed our hypothesis that metonymy resolution modulates areas implicated in semantic processing. We interpret our results in an interdisciplinary view of what metonymy in action can reveal about abstract cognition.