Dissociable effects of prediction and integration during language comprehension: evidence from a large-scale study using brain potentials.

Composing sentence meaning is easier for predictable words than for unpredictable words. Are predictable words genuinely predicted, or simply more plausible and therefore easier to integrate with sentence context? We addressed this persistent and fundamental question using data from a recent, large-scale (n = 334) replication study, by investigating the effects of word predictability and sentence plausibility on the N400, the brain's electrophysiological index of semantic processing. A spatio-temporally fine-grained mixed-effect multiple regression analysis revealed overlapping effects of predictability and plausibility on the N400, albeit with distinct spatio-temporal profiles. Our results challenge the view that the predictability-dependent N400 reflects the effects of either prediction or integration, and suggest that semantic facilitation of predictable words arises from a cascade of processes that activate and integrate word meaning with context into a sentence-level meaning. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Bound together: Social binding leads to faster processing, spatial distortion, and enhanced memory of interacting partners.

The binding of features into perceptual wholes is a well-established phenomenon, which has previously only been studied in the context of early vision and low-level features, such as color or proximity. We hypothesized that a similar binding process, based on higher level information, could bind people into interacting groups, facilitating faster processing and enhanced memory of social situations. To investigate this possibility we used 3 experimental approaches to explore grouping effects in displays involving interacting people. First, using a visual search task we demonstrate more rapid processing for interacting (vs. noninteracting) pairs in an odd-quadrant paradigm (Experiments 1a and 1b). Second, using a spatial judgment task, we show that interacting individuals are remembered as physically closer than are noninteracting individuals (Experiments 2a and 2b). Finally, we show that memory retention of group-relevant and irrelevant features are enhanced when recalling interacting partners in a surprise memory task (Experiments 3a and 3b). Each of these results is consistent with the social binding hypothesis, and alternative explanations based on low level perceptual features and attentional effects are ruled out. We conclude that automatic midlevel grouping processes bind individuals into groups on the basis of their perceived interaction. Such social binding could provide the basis for more sophisticated social processing. Identifying the automatic encoding of social interactions in visual search, distortions of spatial working memory, and facilitated retrieval of object properties from longer-term memory, opens new approaches to studying social cognition with possible practical applications. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Imagining Sounds and Images: Decoding the Contribution of Unimodal and Transmodal Brain Regions to Semantic Retrieval in the Absence of Meaningful Input.

In the absence of sensory information, we can generate meaningful images and sounds from representations in memory. However, it remains unclear which neural systems underpin this process and whether tasks requiring the top-down generation of different kinds of features recruit similar or different neural networks. We asked people to internally generate the visual and auditory features of objects, either in isolation (car, dog) or in specific and complex meaning-based contexts (car/dog race). Using an fMRI decoding approach, in conjunction with functional connectivity analysis, we examined the role of auditory/visual cortex and transmodal brain regions. Conceptual retrieval in the absence of external input recruited sensory and transmodal cortex. The response in transmodal regions-including anterior middle temporal gyrus-was of equal magnitude for visual and auditory features yet nevertheless captured modality information in the pattern of response across voxels. In contrast, sensory regions showed greater activation for modality-relevant features in imagination (even when external inputs did not differ). These data are consistent with the view that transmodal regions support internally generated experiences and that they play a role in integrating perceptual features encoded in memory.

Activation of the language control network in bilingual visual word recognition.

Research into bilingual language production has identified a language control network that subserves control operations when bilinguals produce speech. Here we explore which brain areas are recruited for control purposes in bilingual language comprehension. In two experimental fMRI sessions, Dutch-English unbalanced bilinguals read words that differed in cross-linguistic form and meaning overlap across their two languages. The need for control operations was further manipulated by varying stimulus list composition across the two experimental sessions. We observed activation of the language control network in bilingual language comprehension as a function of both cross-linguistic form and meaning overlap and stimulus list composition. These findings suggest that the language control network is shared across bilingual language production and comprehension. We argue that activation of the language control network in language comprehension allows bilinguals to quickly and efficiently grasp the context-relevant meaning of words.

Large-scale replication study reveals a limit on probabilistic prediction in language comprehension.

Do people routinely pre-activate the meaning and even the phonological form of upcoming words? The most acclaimed evidence for phonological prediction comes from a 2005 Nature Neuroscience publication by DeLong, Urbach and Kutas, who observed a graded modulation of electrical brain potentials (N400) to nouns and preceding articles by the probability that people use a word to continue the sentence fragment ('cloze'). In our direct replication study spanning 9 laboratories (N=334), pre-registered replication-analyses and exploratory Bayes factor analyses successfully replicated the noun-results but, crucially, not the article-results. Pre-registered single-trial analyses also yielded a statistically significant effect for the nouns but not the articles. Exploratory Bayesian single-trial analyses showed that the article-effect may be non-zero but is likely far smaller than originally reported and too small to observe without very large sample sizes. Our results do not support the view that readers routinely pre-activate the phonological form of predictable words.

Distant from input: Evidence of regions within the default mode network supporting perceptually-decoupled and conceptually-guided cognition.

The default mode network supports a variety of mental operations such as semantic processing, episodic memory retrieval, mental time travel and mind-wandering, yet the commonalities between these functions remains unclear. One possibility is that this system supports cognition that is independent of the immediate environment; alternatively or additionally, it might support higher-order conceptual representations that draw together multiple features. We tested these accounts using a novel paradigm that separately manipulated the availability of perceptual information to guide decision-making and the representational complexity of this information. Using task based imaging we established regions that respond when cognition combines both stimulus independence with multi-modal information. These included left and right angular gyri and the left middle temporal gyrus. Although these sites were within the default mode network, they showed a stronger response to demanding memory judgements than to an easier perceptual task, contrary to the view that they support automatic aspects of cognition. In a subsequent analysis, we showed that these regions were located at the extreme end of a macroscale gradient, which describes gradual transitions from sensorimotor to transmodal cortex. This shift in the focus of neural activity towards transmodal, default mode, regions might reflect a process of where the functional distance from specific sensory enables conceptually rich and detailed cognitive states to be generated in the absence of input.

"I know something you don't know": Discourse and social context effects on the N400 in adolescents.

Adolescence is a time of great cognitive and social development. Despite this, relatively few studies to date have investigated how perspective taking affects on-line language comprehension in adolescents. In the current study, we addressed this gap in the literature, making use of a Joint Comprehension Task in which two individuals with differing background knowledge jointly attend to linguistic stimuli. Using event-related potentials, we investigated adolescents' electrophysiological responses to (a) semantically anomalous sentence stimuli in discourse context and (b) semantically plausible sentence stimuli that the participants believe another individual finds semantically implausible. Our results demonstrate that a robust "N400 effect" (i.e., a well-established event-related potential, known to be sensitive to lexical-semantic integration difficulties)  is elicited by semantically anomalous sentences; this N400 effect is subsequently attenuated by discourse context. Lastly, a "social N400 effect" is elicited by sentences that are semantically plausible for the participants if they believe that another individual finds the sentences implausible. The results suggest that adolescents integrate the perspective of others during on-line language comprehension via simulation; that is, adolescents use their own language processing system to interpret language input from the perspective of other jointly attending individuals.

Fractionating the anterior temporal lobe: MVPA reveals differential responses to input and conceptual modality.

Words activate cortical regions in accordance with their modality of presentation (i.e., written vs. spoken), yet there is a long-standing debate about whether patterns of activity in any specific brain region capture modality-invariant conceptual information. Deficits in patients with semantic dementia highlight the anterior temporal lobe (ATL) as an amodal store of semantic knowledge but these studies do not permit precise localisation of this function. The current investigation used multiple imaging methods in healthy participants to examine functional dissociations within ATL. Multi-voxel pattern analysis identified spatially segregated regions: a response to input modality in anterior superior temporal gyrus (aSTG) and a response to meaning in more ventral anterior temporal lobe (vATL). This functional dissociation was supported by resting-state connectivity that found greater coupling for aSTG with primary auditory cortex and vATL with the default mode network. A meta-analytic decoding of these connectivity patterns implicated aSTG in processes closely tied to auditory processing (such as phonology and language) and vATL in meaning-based tasks (such as comprehension or social cognition). Thus we provide converging evidence for the segregation of meaning and input modality in the ATL.

Neuronal interactions between mentalising and action systems during indirect request processing.

Human communication relies on the ability to process linguistic structure and to map words and utterances onto our environment. Furthermore, as what we communicate is often not directly encoded in our language (e.g. in the case of irony, jokes or indirect requests), we need to extract additional cues to infer the beliefs and desires of our conversational partners. Although the functional interplay between language and the ability to mentalise has been discussed in theoretical accounts in the past, the neurobiological underpinnings of these dynamics are currently not well understood. Here, we address this issue using functional imaging (fMRI). Participants listened to question-reply dialogues. In these dialogues, a reply is interpreted as a direct reply, an indirect reply or a request for action, depending on the question. We show that inferring meaning from indirect replies engages parts of the mentalising network (mPFC) while requests for action also activate the cortical motor system (IPL). Subsequent connectivity analysis using Dynamic Causal Modelling (DCM) revealed that this pattern of activation is best explained by an increase in effective connectivity from the mentalising network (mPFC) to the action system (IPL). These results are an important step towards a more integrative understanding of the neurobiological basis of indirect speech processing.

(How) observed eye-contact modulates gaze following. An fMRI study.

Humans are highly sensitive to directional gaze cues and rapidly shift attention in accordance with others' gaze (i.e., gaze following). Besides providing information about the physical environment, for instance, the location of an object, gaze direction can be used to extract information about the social environment, such as whether or not two people are interacting with each other. In the present fMRI study we investigated how these two different types of information conveyed by gaze direction interact with one another. Participants saw two faces that were either looking at each other or away from each other before jointly shifting gaze toward one of two target locations. Targets either appeared at the gazed at or the non-gazed at location. Behaviorally, gaze following (faster responses to congruent versus incongruent trials) was more prominent after observing eye contact than after observing no eye contact. In line with behavioral findings, neuroimaging results revealed enhanced activation in fronto-parietal and temporal areas in congruent trials when faces had looked at each other versus away from each other. These findings demonstrate that observing an attentional relation between others augments processing of their subsequent gaze cues.

Neural correlates of observing joint actions with shared intentions.

Studies on the neural bases of action perception have largely focused on the perception of individual actions. Little is known about perception of joint actions where two or more individuals coordinate their actions based on a shared intention. In this fMRI study we asked whether observing situations where two individuals act on a shared intention elicits a different neural response than observing situations where individuals act on their independent parallel intentions. We compared the neural response to perceptually identical yet intentionally ambiguous actions observed in varying contexts. A dialog between two individuals conveyed either a shared intention or two independent parallel intentions. The dialogs were followed by an identical video clip where the two individuals performed certain actions. In one task condition participants tracked the intentions of the actors, in the other, they monitored moving colored dots placed on the same videos. We found that in the intention task versus the color task, observing joint actions based on shared intentions activated the temporal poles, precuneus, and the ventral striatum compared to observing interactions based on parallel intentions. Precuneus and the temporal poles are thought to support mental state reasoning, the latter with a more specific role in retrieving memories associated with social scripts. Activation in the ventral striatum, an area involved in reward processing, likely indicates a hedonistic response to observed shared intentional relations similarly to those experienced when personally sharing mental states with others.

Feature activation during word recognition: action, visual, and associative-semantic priming effects.

Embodied theories of language postulate that language meaning is stored in modality-specific brain areas generally involved in perception and action in the real world. However, the temporal dynamics of the interaction between modality-specific information and lexical-semantic processing remain unclear. We investigated the relative timing at which two types of modality-specific information (action-based and visual-form information) contribute to lexical-semantic comprehension. To this end, we applied a behavioral priming paradigm in which prime and target words were related with respect to (1) action features, (2) visual features, or (3) semantically associative information. Using a Go/No-Go lexical decision task, priming effects were measured across four different inter-stimulus intervals (ISI = 100, 250, 400, and 1000 ms) to determine the relative time course of the different features. Notably, action priming effects were found in ISIs of 100, 250, and 1000 ms whereas a visual priming effect was seen only in the ISI of 1000 ms. Importantly, our data suggest that features follow different time courses of activation during word recognition. In this regard, feature activation is dynamic, measurable in specific time windows but not in others. Thus the current study (1) demonstrates how multiple ISIs can be used within an experiment to help chart the time course of feature activation and (2) provides new evidence for embodied theories of language.

Shared neural processes support semantic control and action understanding.

Executive-semantic control and action understanding appear to recruit overlapping brain regions but existing evidence from neuroimaging meta-analyses and neuropsychology lacks spatial precision; we therefore manipulated difficulty and feature type (visual vs. action) in a single fMRI study. Harder judgements recruited an executive-semantic network encompassing medial and inferior frontal regions (including LIFG) and posterior temporal cortex (including pMTG). These regions partially overlapped with brain areas involved in action but not visual judgements. In LIFG, the peak responses to action and difficulty were spatially identical across participants, while these responses were overlapping yet spatially distinct in posterior temporal cortex. We propose that the co-activation of LIFG and pMTG allows the flexible retrieval of semantic information, appropriate to the current context; this might be necessary both for semantic control and understanding actions. Feature selection in difficult trials also recruited ventral occipital-temporal areas, not implicated in action understanding.

The Social N400 effect: how the presence of other listeners affects language comprehension.

During conversation, it is necessary to keep track of what others can and cannot understand. Previous research has focused largely on understanding the time course along which knowledge about interlocutors influences language comprehension/production rather than the cognitive process by which interlocutors take each other's perspective. In addition, most work has looked at the effects of knowledge about a speaker on a listener's comprehension, and not on the possible effects of other listeners on a participant's comprehension process. In the current study, we introduce a novel joint comprehension paradigm that addresses the cognitive processes underlying perspective taking during language comprehension. Specifically, we show that participants who understand a language stimulus, but are simultaneously aware that someone sitting next to them does not understand the same stimulus, show an electrophysiological marker of semantic integration difficulty (i.e., an N400-effect). Crucially, in a second group of participants, we demonstrate that presenting exactly the same sentences to the participant alone (i.e. without a co-listener) results in no N400-effect. Our results suggest that (1) information about co-listeners as well as the speaker affect language comprehension, and (2) the cognitive process by which we understand what others comprehend mirrors our own language comprehension processes.

Neural networks involved in learning lexical-semantic and syntactic information in a second language.

The present study used functional magnetic resonance imaging (fMRI) to investigate the neural correlates of language acquisition in a realistic learning environment. Japanese native speakers were trained in a miniature version of German prior to fMRI scanning. During scanning they listened to (1) familiar sentences, (2) sentences including a novel sentence structure, and (3) sentences containing a novel word while visual context provided referential information. Learning-related decreases of brain activation over time were found in a mainly left-hemispheric network comprising classical frontal and temporal language areas as well as parietal and subcortical regions and were largely overlapping for novel words and the novel sentence structure in initial stages of learning. Differences occurred at later stages of learning during which content-specific activation patterns in prefrontal, parietal and temporal cortices emerged. The results are taken as evidence for a domain-general network supporting the initial stages of language learning which dynamically adapts as learners become proficient.

Oscillatory neuronal activity reflects lexical-semantic feature integration within and across sensory modalities in distributed cortical networks.

Research from the previous decade suggests that word meaning is partially stored in distributed modality-specific cortical networks. However, little is known about the mechanisms by which semantic content from multiple modalities is integrated into a coherent multisensory representation. Therefore we aimed to characterize differences between integration of lexical-semantic information from a single modality compared with two sensory modalities. We used magnetoencephalography in humans to investigate changes in oscillatory neuronal activity while participants verified two features for a given target word (e.g., "bus"). Feature pairs consisted of either two features from the same modality (visual: "red," "big") or different modalities (auditory and visual: "red," "loud"). The results suggest that integrating modality-specific features of the target word is associated with enhanced high-frequency power (80-120 Hz), while integrating features from different modalities is associated with a sustained increase in low-frequency power (2-8 Hz). Source reconstruction revealed a peak in the anterior temporal lobe for low-frequency and high-frequency effects. These results suggest that integrating lexical-semantic knowledge at different cortical scales is reflected in frequency-specific oscillatory neuronal activity in unisensory and multisensory association networks.

Flexibility in embodied language processing: context effects in lexical access.

According to embodied theories of language (ETLs), word meaning relies on sensorimotor brain areas, generally dedicated to acting and perceiving in the real world. More specifically, words denoting actions are postulated to make use of neural motor areas, while words denoting visual properties draw on the resources of visual brain areas. Therefore, there is a direct correspondence between word meaning and the experience a listener has had with a word's referent on the brain level. Behavioral and neuroimaging studies have provided evidence in favor of ETLs; however, recent studies have also shown that sensorimotor information is recruited in a flexible manner during language comprehension (e.g., Raposo et al.; Van Dam et al.,), leaving open the question as to what level of language processing sensorimotor activations contribute. In this study, we investigated the time course of modality-specific contributions (i.e., the contribution of action information) as to word processing by manipulating both (a) the linguistic and (b) the action context in which target words were presented. Our results demonstrate that processes reflecting sensorimotor information play a role early in word processing (i.e., within 200 ms of word presentation), but that they are sensitive to the linguistic context in which a word is presented. In other words, when sensorimotor information is activated, it is activated quickly; however, specific words do not reliably activate a consistent sensorimotor pattern.

Cross-modal integration of lexical-semantic features during word processing: evidence from oscillatory dynamics during EEG.

In recent years, numerous studies have provided converging evidence that word meaning is partially stored in modality-specific cortical networks. However, little is known about the mechanisms supporting the integration of this distributed semantic content into coherent conceptual representations. In the current study we aimed to address this issue by using EEG to look at the spatial and temporal dynamics of feature integration during word comprehension. Specifically, participants were presented with two modality-specific features (i.e., visual or auditory features such as silver and loud) and asked to verify whether these two features were compatible with a subsequently presented target word (e.g., WHISTLE). Each pair of features described properties from either the same modality (e.g., silver, tiny  =  visual features) or different modalities (e.g., silver, loud  =  visual, auditory). Behavioral and EEG data were collected. The results show that verifying features that are putatively represented in the same modality-specific network is faster than verifying features across modalities. At the neural level, integrating features across modalities induces sustained oscillatory activity around the theta range (4-6 Hz) in left anterior temporal lobe (ATL), a putative hub for integrating distributed semantic content. In addition, enhanced long-range network interactions in the theta range were seen between left ATL and a widespread cortical network. These results suggest that oscillatory dynamics in the theta range could be involved in integrating multimodal semantic content by creating transient functional networks linking distributed modality-specific networks and multimodal semantic hubs such as left ATL.

Embodied language in first- and second-language speakers: neural correlates of processing motor verbs.

The involvement of neural motor and sensory systems in the processing of language has so far mainly been studied in native (L1) speakers. In an fMRI experiment, we investigated whether non-native (L2) semantic representations are rich enough to allow for activation in motor and somatosensory brain areas. German learners of Dutch and a control group of Dutch native speakers made lexical decisions about visually presented Dutch motor and non-motor verbs. Region-of-interest (ROI) and whole-brain analyses indicated that L2 speakers, like L1 speakers, showed significantly increased activation for simple motor compared to non-motor verbs in motor and somatosensory regions. This effect was not restricted to Dutch-German cognate verbs, but was also present for non-cognate verbs. These results indicate that L2 semantic representations are rich enough for motor-related activations to develop in motor and somatosensory areas.

Observing, performing, and understanding actions: revisiting the role of cortical motor areas in processing of action words.

Language content and action/perception have been shown to activate common brain areas in previous neuroimaging studies. However, it is unclear whether overlapping cortical activation reflects a common neural source or adjacent, but distinct, sources. We address this issue by using multivoxel pattern analysis on fMRI data. Specifically, participants were instructed to engage in five tasks: (1) execute hand actions (AE), (2) observe hand actions (AO), (3) observe nonbiological motion (MO), (4) read action verbs, and (5) read nonaction verbs. A classifier was trained to distinguish between data collected from neural motor areas during (1) AE versus MO and (2) AO versus MO. These two algorithms were then used to test for a distinction between data collected during the reading of action versus nonaction verbs. The results show that the algorithm trained to distinguish between AE and MO distinguishes between word categories using signal recorded from the left parietal cortex and pre-SMA, but not from ventrolateral premotor cortex. In contrast, the algorithm trained to distinguish between AO and MO discriminates between word categories using the activity pattern in the left premotor and left parietal cortex. This shows that the sensitivity of premotor areas to language content is more similar to the process of observing others acting than to acting oneself. Furthermore, those parts of the brain that show comparable neural pattern for action execution and action word comprehension are high-level integrative motor areas rather than low-level motor areas.