The fate of the unexpected: Consequences of misprediction assessed using ERP repetition effects.

Amid increasing interest in the role of prediction in language comprehension, there remains a gap in our understanding of what happens when predictions are disconfirmed. Are unexpected words harder to process and encode because of interference from the original prediction? Or, because of their relevance for learning, do expectation violations strengthen the representations of unexpected words? In two experiments, we used event-related potentials to probe the downstream consequences of prediction violations. Critical words were unexpected but plausible completions of either strongly constraining sentences, wherein they constituted a prediction violation, or weakly constraining sentences that did not afford a clear prediction. Three sentences later the critical word was repeated at the end of a different, weakly constraining sentence. In Experiment 1, repeated words elicited a reduced N400 and an enhanced late positive complex (LPC) compared to words seen for the first time. Critically, there was no effect of initial sentence constraint on the size of the repetition effect in either time window. Thus, prediction violations did not accrue either costs or benefits for later processing. Experiment 2 used the same critical items and added strongly constraining filler sentences with expected endings to further promote prediction. Again, there was no effect of initial sentence constraint on either the N400 or the LPC to repeated critical words. When taken with prior findings, the results suggest that prediction is both powerful and flexible: It can facilitate processing of predictable information by reducing encoding effort without causing processing difficulties for unexpected inputs.

Definitely saw it coming? The dual nature of the pre-nominal prediction effect.

In well-known demonstrations of lexical prediction during language comprehension, pre-nominal articles that mismatch a likely upcoming noun's gender elicit different neural activity than matching articles. However, theories differ on what this pre-nominal prediction effect means and on what is being predicted. Does it reflect mismatch with a predicted article, or 'merely' revision of the noun prediction? We contrasted the 'article prediction mismatch' hypothesis and the 'noun prediction revision' hypothesis in two ERP experiments on Dutch mini-story comprehension, with pre-registered data collection and analyses. We capitalized on the Dutch gender system, which marks gender on definite articles ('de/het') but not on indefinite articles ('een'). If articles themselves are predicted, mismatching gender should have little effect when readers expected an indefinite article without gender marking. Participants read contexts that strongly suggested either a definite or indefinite noun phrase as its best continuation, followed by a definite noun phrase with the expected noun or an unexpected, different gender noun phrase ('het boek/de roman', the book/the novel). Experiment 1 (N = 48) showed a pre-nominal prediction effect, but evidence for the article prediction mismatch hypothesis was inconclusive. Informed by exploratory analyses and power analyses, direct replication Experiment 2 (N = 80) yielded evidence for article prediction mismatch at a newly pre-registered occipital region-of-interest. However, at frontal and posterior channels, unexpectedly definite articles also elicited a gender-mismatch effect, and this support for the noun prediction revision hypothesis was further strengthened by exploratory analyses: ERPs elicited by gender-mismatching articles correlated with incurred constraint towards a new noun (next-word entropy), and N400s for initially unpredictable nouns decreased when articles made them more predictable. By demonstrating its dual nature, our results reconcile two prevalent explanations of the pre-nominal prediction effect.

Word predictability blurs the lines between production and comprehension: Evidence from the production effect in memory.

Predictions about likely upcoming input may promote rapid language processing, but the mechanisms by which such predictions are generated remain unclear. One hypothesis is that comprehenders use their production system to covertly produce what they would say if they were the speaker. If reading predictable words involves covert production, this act might have consequences for memory. The present study capitalized on the production effect, which is the observation that words read aloud are remembered better than words read silently. Participants read sentence-final predictable and unpredictable words aloud or silently, followed by a surprise recognition memory task. If reading predictable words involves covert production, the memory improvement from actually producing the words should be smaller for predictable words than for unpredictable words. This was confirmed in Experiment 1, which tested item memory using old/new judgments. Experiment 2 followed the same procedure, except that participants now made aloud/silent judgments probing their memory for prior acts of production. Here the hypothesis was that, relative to unpredictable words, it should be more difficult to remember whether predictable words had been read aloud or silently. Indeed, word predictability tended to make it harder to tell the difference, suggesting that predictability blurred the lines between production and comprehension. Taken together, the findings support the idea that reading predictable words can involve covert production and show that this act has consequences for what readers retain.

How a question context aids word production: Evidence from the picture-word interference paradigm.

Difficulties in saying the right word at the right time arise at least in part because multiple response candidates are simultaneously activated in the speaker's mind. The word selection process has been simulated using the picture-word interference task, in which participants name pictures while ignoring a superimposed written distractor word. However, words are usually produced in context, in the service of achieving a communicative goal. Two experiments addressed the questions whether context influences word production, and if so, how. We embedded the picture-word interference task in a dialogue-like setting, in which participants heard a question and named a picture as an answer to the question while ignoring a superimposed distractor word. The conversational context was either constraining or nonconstraining towards the answer. Manipulating the relationship between the picture name and the distractor, we focused on two core processes of word production: retrieval of semantic representations (Experiment 1) and phonological encoding (Experiment 2). The results of both experiments showed that naming reaction times (RTs) were shorter when preceded by constraining contexts as compared with nonconstraining contexts. Critically, constraining contexts decreased the effect of semantically related distractors but not the effect of phonologically related distractors. This suggests that conversational contexts can help speakers with aspects of the meaning of to-be-produced words, but phonological encoding processes still need to be performed as usual.

Downstream Behavioral and Electrophysiological Consequences of Word Prediction on Recognition Memory.

When people process language, they can use context to predict upcoming information, influencing processing and comprehension as seen in both behavioral and neural measures. Although numerous studies have shown immediate facilitative effects of confirmed predictions, the downstream consequences of prediction have been less explored. In the current study, we examined those consequences by probing participants' recognition memory for words after they read sets of sentences. Participants read strongly and weakly constraining sentences with expected or unexpected endings ("I added my name to the list/basket"), and later were tested on their memory for the sentence endings while EEG was recorded. Critically, the memory test contained words that were predictable ("list") but were never read (participants saw "basket"). Behaviorally, participants showed successful discrimination between old and new items, but false alarmed to the expected-item lures more often than to new items, showing that predicted words or concepts can linger, even when predictions are disconfirmed. Although false alarm rates did not differ by constraint, event-related potentials (ERPs) differed between false alarms to strongly and weakly predictable words. Additionally, previously unexpected (compared to previously expected) endings that appeared on the memory test elicited larger N1 and LPC amplitudes, suggesting greater attention and episodic recollection. In contrast, highly predictable sentence endings that had been read elicited reduced LPC amplitudes during the memory test. Thus, prediction can facilitate processing in the moment, but can also lead to false memory and reduced recollection for predictable information.

Lingering expectations: A pseudo-repetition effect for words previously expected but not presented.

Prediction can help support rapid language processing. However, it is unclear whether prediction has downstream consequences, beyond processing in the moment. In particular, when a prediction is disconfirmed, does it linger, or is it suppressed? This study manipulated whether words were actually seen or were only expected, and probed their fate in memory by presenting the words (again) a few sentences later. If disconfirmed predictions linger, subsequent processing of the previously expected (but never presented) word should be similar to actual word repetition. At initial presentation, electrophysiological signatures of prediction disconfirmation demonstrated that participants had formed expectations. Further downstream, relative to unseen words, repeated words elicited a strong N400 decrease, an enhanced late positive complex (LPC), and late alpha band power decreases. Critically, like repeated words, words previously expected but not presented also attenuated the N400. This "pseudo-repetition effect" suggests that disconfirmed predictions can linger at some stages of processing, and demonstrates that prediction has downstream consequences beyond rapid on-line processing.

Predictability's aftermath: Downstream consequences of word predictability as revealed by repetition effects.

Stimulus processing in language and beyond is shaped by context, with predictability having a particularly well-attested influence on the rapid processes that unfold during the presentation of a word. But does predictability also have downstream consequences for the quality of the constructed representations? On the one hand, the ease of processing predictable words might free up time or cognitive resources, allowing for relatively thorough processing of the input. On the other hand, predictability might allow the system to run in a top-down "verification mode", at the expense of thorough stimulus processing. This electroencephalogram (EEG) study manipulated word predictability, which reduced N400 amplitude and inter-trial phase clustering (ITPC), and then probed the fate of the (un)predictable words in memory by presenting them again. More thorough processing of predictable words should increase repetition effects, whereas less thorough processing should decrease them. Repetition was reflected in N400 decreases, late positive complex (LPC) enhancements, and late alpha/beta band power decreases. Critically, prior predictability tended to reduce the repetition effect on the N400, suggesting less priming, and eliminated the repetition effect on the LPC, suggesting a lack of episodic recollection. These findings converge on a top-down verification account, on which the brain processes more predictable input less thoroughly. More generally, the results demonstrate that predictability has multifaceted downstream consequences beyond processing in the moment.

Lesion evidence for a critical role of left posterior but not frontal areas in alpha-beta power decreases during context-driven word production.

Different frequency bands in the electroencephalogram are postulated to support distinct language functions. Studies have suggested that alpha-beta power decreases may index word-retrieval processes. In context-driven word retrieval, participants hear lead-in sentences that either constrain the final word ('He locked the door with the') or not ('She walked in here with the'). The last word is shown as a picture to be named. Previous studies have consistently found alpha-beta power decreases prior to picture onset for constrained relative to unconstrained sentences, localised to the left lateral-temporal and lateral-frontal lobes. However, the relative contribution of temporal versus frontal areas to alpha-beta power decreases is unknown. We recorded the electroencephalogram from patients with stroke lesions encompassing the left lateral-temporal and inferior-parietal regions or left-lateral frontal lobe and from matched controls. Individual participant analyses revealed a behavioural sentence context facilitation effect in all participants, except for in the two patients with extensive lesions to temporal and inferior parietal lobes. We replicated the alpha-beta power decreases prior to picture onset in all participants, except for in the two same patients with extensive posterior lesions. Thus, whereas posterior lesions eliminated the behavioural and oscillatory context effect, frontal lesions did not. Hierarchical clustering analyses of all patients' lesion profiles, and behavioural and electrophysiological effects identified those two patients as having a unique combination of lesion distribution and context effects. These results indicate a critical role for the left lateral-temporal and inferior parietal lobes, but not frontal cortex, in generating the alpha-beta power decreases underlying context-driven word production.

Alpha and theta band dynamics related to sentential constraint and word expectancy.

Despite strong evidence for prediction during language comprehension, the underlying mechanisms, and the extent to which they are specific to language, remain unclear. Re-analyzing an ERP study, we examined responses in the time-frequency domain to expected and unexpected (but plausible) words in strongly and weakly constraining sentences, and found results similar to those reported in nonverbal domains. Relative to expected words, unexpected words elicited an increase in the theta band (4-7 Hz) in strongly constraining contexts, suggesting the involvement of control processes to deal with the consequences of having a prediction disconfirmed. Prior to critical word onset, strongly constraining sentences exhibited a decrease in the alpha band (8-12 Hz) relative to weakly constraining sentences, suggesting that comprehenders can take advantage of predictive sentence contexts to prepare for the input. The results suggest that the brain recruits domain-general preparation and control mechanisms when making and assessing predictions during sentence comprehension.

Lateralized electrical brain activity reveals covert attention allocation during speaking.

Speakers usually begin to speak while only part of the utterance has been planned. Earlier work has shown that speech planning processes are reflected in speakers' eye movements as they describe visually presented objects. However, to-be-named objects can be processed to some extent before they have been fixated upon, presumably because attention can be allocated to objects covertly, without moving the eyes. The present study investigated whether EEG could track speakers' covert attention allocation as they produced short utterances to describe pairs of objects (e.g., "dog and chair"). The processing difficulty of each object was varied by presenting it in upright orientation (easy) or in upside down orientation (difficult). Background squares flickered at different frequencies in order to elicit steady-state visual evoked potentials (SSVEPs). The N2pc component, associated with the focusing of attention on an item, was detectable not only prior to speech onset, but also during speaking. The time course of the N2pc showed that attention shifted to each object in the order of mention prior to speech onset. Furthermore, greater processing difficulty increased the time speakers spent attending to each object. This demonstrates that the N2pc can track covert attention allocation in a naming task. In addition, an effect of processing difficulty at around 200-350ms after stimulus onset revealed early attention allocation to the second to-be-named object. The flickering backgrounds elicited SSVEPs, but SSVEP amplitude was not influenced by processing difficulty. These results help complete the picture of the coordination of visual information uptake and motor output during speaking.

Withholding planned speech is reflected in synchronized beta-band oscillations.

When engaged in a conversation, speakers sometimes have to withhold a planned response, for example, before it is their turn to speak. In the present study, using magnetoencephalography (MEG) outside of a conversational setting, we investigate the oscillatory brain mechanisms involved in the process of withholding a planned verbal response until it is time to speak. Our participants viewed a sequence of four random consonant strings and one pseudoword, which they had to pronounce when the fifth string (the imperative stimulus) was presented. The pseudoword appeared either as the fourth or fifth stimulus in the sequence, creating two conditions. In the withhold condition, the pseudoword was the fourth string and the verbal response was withheld until the imperative stimulus was presented. In the control condition, the fifth string was the pseudoword, so no response was withheld. We compared oscillatory responses to the withhold relative to the control condition in the time period preceding speech. Alpha-beta power (8-30 Hz) decreased over occipital sensors in the withhold condition relative to the control condition. Source-level analysis indicated a posterior source (i.e., occipital cortex) associated with the alpha-beta power decreases. This occipital alpha-beta desynchronization likely reflects attentional allocation to the upcoming imperative stimulus. Moreover, beta (12-20 Hz) power increased over frontal sensors. Source-level analysis indicated a frontal source (i.e., middle and superior frontal gyri) associated with the beta-power increases. We interpret the frontal beta synchronization to reflect a mechanism aiding the maintenance of the current motor or cognitive state. Our results provide a window into a possible oscillatory mechanism implementing the ability of speakers to withhold a planned verbal response until they have to speak.

Beta oscillations reflect memory and motor aspects of spoken word production.

Two major components form the basis of spoken word production: the access of conceptual and lexical/phonological information in long-term memory, and motor preparation and execution of an articulatory program. Whereas the motor aspects of word production have been well characterized as reflected in alpha-beta desynchronization, the memory aspects have remained poorly understood. Using magnetoencephalography, we investigated the neurophysiological signature of not only motor but also memory aspects of spoken-word production. Participants named or judged pictures after reading sentences. To probe the involvement of the memory component, we manipulated sentence context. Sentence contexts were either constraining or nonconstraining toward the final word, presented as a picture. In the judgment task, participants indicated with a left-hand button press whether the picture was expected given the sentence. In the naming task, they named the picture. Naming and judgment were faster with constraining than nonconstraining contexts. Alpha-beta desynchronization was found for constraining relative to nonconstraining contexts pre-picture presentation. For the judgment task, beta desynchronization was observed in left posterior brain areas associated with conceptual processing and in right motor cortex. For the naming task, in addition to the same left posterior brain areas, beta desynchronization was found in left anterior and posterior temporal cortex (associated with memory aspects), left inferior frontal cortex, and bilateral ventral premotor cortex (associated with motor aspects). These results suggest that memory and motor components of spoken word production are reflected in overlapping brain oscillations in the beta band.

Verbal and nonverbal predictors of language-mediated anticipatory eye movements.

During language comprehension, listeners often anticipate upcoming information. This can draw listeners' overt attention to visually presented objects before the objects are referred to. We investigated to what extent the anticipatory mechanisms involved in such language-mediated attention rely on specific verbal factors and on processes shared with other domains of cognition. Participants listened to sentences ending in a highly predictable word (e.g., "In 1969 Neil Armstrong was the first man to set foot on the moon") while viewing displays containing three unrelated distractor objects and a critical object, which was either the target object (e.g., a moon), an object with a similar shape (e.g., a tomato), or an unrelated control object (e.g., rice). Language-mediated anticipatory eye movements were observed to targets and to shape competitors. Importantly, looks to the shape competitor were systematically related to individual differences in anticipatory attention, as indexed by a spatial cueing task: Participants whose responses were most strongly facilitated by predictive arrow cues also showed the strongest effects of predictive language input on their eye movements. By contrast, looks to the target were related to individual differences in vocabulary size and verbal fluency. The results suggest that verbal and nonverbal factors contribute to different types of language-mediated eye movements. The findings are consistent with multiple-mechanism accounts of predictive language processing.

Object shape and orientation do not routinely influence performance during language processing.

The role of visual representations during language processing remains unclear: They could be activated as a necessary part of the comprehension process, or they could be less crucial and influence performance in a task-dependent manner. In the present experiments, participants read sentences about an object. The sentences implied that the object had a specific shape or orientation. They then either named a picture of that object (Experiments 1 and 3) or decided whether the object had been mentioned in the sentence (Experiment 2). Orientation information did not reliably influence performance in any of the experiments. Shape representations influenced performance most strongly when participants were asked to compare a sentence with a picture or when they were explicitly asked to use mental imagery while reading the sentences. Thus, in contrast to previous claims, implied visual information often does not contribute substantially to the comprehension process during normal reading.

The contents of predictions in sentence comprehension: activation of the shape of objects before they are referred to.

When comprehending concrete words, listeners and readers can activate specific visual information such as the shape of the words' referents. In two experiments we examined whether such information can be activated in an anticipatory fashion. In Experiment 1, listeners' eye movements were tracked while they were listening to sentences that were predictive of a specific critical word (e.g., "moon" in "In 1969 Neil Armstrong was the first man to set foot on the moon"). 500 ms before the acoustic onset of the critical word, participants were shown four-object displays featuring three unrelated distractor objects and a critical object, which was either the target object (e.g., moon), an object with a similar shape (e.g., tomato), or an unrelated control object (e.g., rice). In a time window before shape information from the spoken target word could be retrieved, participants already tended to fixate both the target and the shape competitors more often than they fixated the control objects, indicating that they had anticipatorily activated the shape of the upcoming word's referent. This was confirmed in Experiment 2, which was an ERP experiment without picture displays. Participants listened to the same lead-in sentences as in Experiment 1. The sentence-final words corresponded to the predictable target, the shape competitor, or the unrelated control object (yielding, for instance, "In 1969 Neil Armstrong was the first man to set foot on the moon/tomato/rice"). N400 amplitude in response to the final words was significantly attenuated in the shape-related compared to the unrelated condition. Taken together, these results suggest that listeners can activate perceptual attributes of objects before they are referred to in an utterance.

Context-dependent semantic processing in the human brain: evidence from idiom comprehension.

Language comprehension involves activating word meanings and integrating them with the sentence context. This study examined whether these routines are carried out even when they are theoretically unnecessary, namely, in the case of opaque idiomatic expressions, for which the literal word meanings are unrelated to the overall meaning of the expression. Predictable words in sentences were replaced by a semantically related or unrelated word. In literal sentences, this yielded previously established behavioral and electrophysiological signatures of semantic processing: semantic facilitation in lexical decision, a reduced N400 for semantically related relative to unrelated words, and a power increase in the gamma frequency band that was disrupted by semantic violations. However, the same manipulations in idioms yielded none of these effects. Instead, semantic violations elicited a late positivity in idioms. Moreover, gamma band power was lower in correct idioms than in correct literal sentences. It is argued that the brain's semantic expectancy and literal word meaning integration operations can, to some extent, be "switched off" when the context renders them unnecessary. Furthermore, the results lend support to models of idiom comprehension that involve unitary idiom representations.

Using the visual world paradigm to study language processing: a review and critical evaluation.

We describe the key features of the visual world paradigm and review the main research areas where it has been used. In our discussion we highlight that the paradigm provides information about the way language users integrate linguistic information with information derived from the visual environment. Therefore the paradigm is well suited to study one of the key issues of current cognitive psychology, namely the interplay between linguistic and visual information processing. However, conclusions about linguistic processing (e.g., about activation, competition, and timing of access of linguistic representations) in the absence of relevant visual information must be drawn with caution.