Tracking Lexical and Semantic Prediction Error Underlying the N400 Using Artificial Neural Network Models of Sentence Processing.

Recent research has shown that the internal dynamics of an artificial neural network model of sentence comprehension displayed a similar pattern to the amplitude of the N400 in several conditions known to modulate this event-related potential. These results led Rabovsky et al. (2018) to suggest that the N400 might reflect change in an implicit predictive representation of meaning corresponding to semantic prediction error. This explanation stands as an alternative to the hypothesis that the N400 reflects lexical prediction error as estimated by word surprisal (Frank et al., 2015). In the present study, we directly model the amplitude of the N400 elicited during naturalistic sentence processing by using as predictor the update of the distributed representation of sentence meaning generated by a sentence gestalt model (McClelland et al., 1989) trained on a large-scale text corpus. This enables a quantitative prediction of N400 amplitudes based on a cognitively motivated model, as well as quantitative comparison of this model to alternative models of the N400. Specifically, we compare the update measure from the sentence gestalt model to surprisal estimated by a comparable language model trained on next-word prediction. Our results suggest that both sentence gestalt update and surprisal predict aspects of N400 amplitudes. Thus, we argue that N400 amplitudes might reflect two distinct but probably closely related sub-processes that contribute to the processing of a sentence.

Error-based Implicit Learning in Language: The Effect of Sentence Context and Constraint in a Repetition Paradigm.

Prediction errors drive implicit learning in language, but the specific mechanisms underlying these effects remain debated. This issue was addressed in an EEG study manipulating the context of a repeated unpredictable word (repetition of the complete sentence or repetition of the word in a new sentence context) and sentence constraint. For the manipulation of sentence constraint, unexpected words were presented either in high-constraint (eliciting a precise prediction) or low-constraint sentences (not eliciting any specific prediction). Repetition-induced reduction of N400 amplitudes and of power in the alpha/beta frequency band was larger for words repeated with their sentence context as compared with words repeated in a new low-constraint context, suggesting that implicit learning happens not only at the level of individual items but additionally improves sentence-based predictions. These processing benefits for repeated sentences did not differ between constraint conditions, suggesting that sentence-based prediction update might be proportional to the amount of unpredicted semantic information, rather than to the precision of the prediction that was violated. In addition, the consequences of high-constraint prediction violations, as reflected in a frontal positivity and increased theta band power, were reduced with repetition. Overall, our findings suggest a powerful and specific adaptation mechanism that allows the language system to quickly adapt its predictions when unexpected semantic information is processed, irrespective of sentence constraint, and to reduce potential costs of strong predictions that were violated.

Are the P600 and P3 ERP components linked to the task-evoked pupillary response as a correlate of norepinephrine activity?

During language comprehension, anomalies and ambiguities in the input typically elicit the P600 event-related potential component. Although traditionally interpreted as a specific signal of combinatorial operations in sentence processing, the component has alternatively been proposed to be a variant of the oddball-sensitive, domain-general P3 component. In particular, both components might reflect phasic norepinephrine release from the locus coeruleus (LC/NE) to motivationally significant stimuli. In this preregistered study, we tested this hypothesis by relating both components to the task-evoked pupillary response, a putative biomarker of LC/NE activity. 36 participants completed a sentence comprehension task (containing 25% morphosyntactic violations) and a non-linguistic oddball task (containing 20% oddballs), while the EEG and pupil size were co-registered. Our results showed that the task-evoked pupillary response and the ERP amplitudes of both components were similarly affected by both experimental tasks. In the oddball task, there was also a temporally specific relationship between the P3 and the pupillary response beyond the shared oddball effect, thereby further linking the P3 to NE. Because this link was less reliable in the linguistic context, we did not find conclusive evidence for or against a relationship between the P600 and the pupillary response. Still, our findings further stimulate the debate on whether language-related ERPs are indeed specific to linguistic processes or shared across cognitive domains. However, further research is required to verify a potential link between the two ERP positivities and the LC/NE system as the common neural generator.

Perceived similarity as a window into representations of integrated sentence meaning.

When perceiving the world around us, we are constantly integrating pieces of information. The integrated experience consists of more than just the sum of its parts. For example, visual scenes are defined by a collection of objects as well as the spatial relations amongst them and sentence meaning is computed based on individual word semantic but also syntactic configuration. Having quantitative models of such integrated representations can help evaluate cognitive models of both language and scene perception. Here, we focus on language, and use a behavioral measure of perceived similarity as an approximation of integrated meaning representations. We collected similarity judgments of 200 subjects rating nouns or transitive sentences through an online multiple arrangement task. We find that perceived similarity between sentences is most strongly modulated by the semantic action category of the main verb. In addition, we show how non-negative matrix factorization of similarity judgment data can reveal multiple underlying dimensions reflecting both semantic as well as relational role information. Finally, we provide an example of how similarity judgments on sentence stimuli can serve as a point of comparison for artificial neural networks models (ANNs) by comparing our behavioral data against sentence similarity extracted from three state-of-the-art ANNs. Overall, our method combining the multiple arrangement task on sentence stimuli with matrix factorization can capture relational information emerging from integration of multiple words in a sentence even in the presence of strong focus on the verb.

Distrust before first sight? Examining knowledge- and appearance-based effects of trustworthiness on the visual consciousness of faces.

The present EEG study with 32 healthy participants investigated whether affective knowledge about a person influences the visual awareness of their face, additionally considering the impact of facial appearance. Faces differing in perceived trustworthiness based on appearance were associated with negative or neutral social information and shown as target stimuli in an attentional blink task. As expected, participants showed enhanced awareness of faces associated with negative compared to neutral social information. On the neurophysiological level, this effect was connected to differences in the time range of the early posterior negativity (EPN)-a component associated with enhanced attention and facilitated processing of emotional stimuli. The findings indicate that the social-affective relevance of a face based on emotional knowledge is accessed during a phase of attentional enhancement for conscious perception and can affect prioritization for awareness. In contrast, no clear evidence for influences of facial trustworthiness during the attentional blink was found.

The N400 is Elicited by Meaning Changes but not Synonym Substitutions: Evidence From Persian Phrasal Verbs.

We tested two accounts of the cognitive process underlying the N400 event-related potential component: one that it reflects meaning-based processing and one that it reflects the processing of specific words. The experimental design utilized separable Persian phrasal verbs, which form a strongly probabilistic, long-distance dependency, ideal for the study of probabilistic processing. In sentences strongly constraining for a particular continuation, we show evidence that between two low-probability words, only the word that changed the expected meaning of the sentence increased N400 amplitude, while a synonym of the most probable target word did not. The findings support an account of the N400 in which its underlying process is driven by the processing of meaning rather than of specific words.

Spontaneous mind wandering impairs model-based decision making.

BACKGROUND: If our attention wanders to other thoughts while making a decision, then the decision might not be directed towards future goals, reflecting a lack of model-based decision making, but may instead be driven by habits, reflecting model-free decision making. Here we aimed to investigate if and how model-based versus model-free decision making is reduced by trait spontaneous mind wandering. METHODS AND FINDINGS: We used a sequential two-step Markov decision task and a self-report questionnaire assessing trait spontaneous and deliberate mind wandering propensity, to investigate how trait mind wandering relates to model-free as well as model-based decisions. We estimated parameters of a computational neurocognitive dual-control model of decision making. Analyzing estimated model parameters, we found that trait spontaneous mind wandering was related to impaired model-based decisions, while model-free choice stayed unaffected. CONCLUSIONS: Our findings suggest trait spontaneous mind wandering is associated with impaired model-based decision making, and it may reflect model-based offline replay for other tasks (e.g., real-life goals) outside the current lab situation.

Social distancing-induced deceleration can enhance subjective well-being.

Modern societies provide an abundance of opportunities, which could lead to acceleration and time poverty, thereby paradoxically limiting well-being. This study examines this issue using social distancing measures introduced during the COVID-19 pandemic. We analyzed a data set of over four million responses, collected by the German online newspaper "ZEIT ONLINE," where people responded to the question "How are you today?" with "good" or "bad," assessing subjective well-being, and an optional self-descriptive adjective of mood. The results showed that subjective well-being significantly increased with the onset of social distancing regulations. This increase was closely accompanied by a rise in adjectives associated with deceleration, the daily usage of which best predicted daily well-being during COVID-19. Factor analysis showed that Factor 1 best predicted daily well-being and was effectively described by adjectives associated with deceleration. An analysis of potential mechanisms of deceleration during the pandemic revealed lower stress levels during workdays and weekends, as well as better sleep. These findings provide large-scale support to theories suggesting that acceleration and time poverty in modern societies may impair well-being.

Sustained Attention as Measured by Reaction Time Variability Is a Strong Modulator for the P600, but Not the N400.

The functional significance of the two prominent language-related ERP components N400 and P600 is still under debate. It has recently been suggested that one important dimension along which the two vary is in terms of automaticity versus attentional control, with N400 amplitudes reflecting more automatic and P600 amplitudes reflecting more controlled aspects of sentence comprehension. The availability of executive resources necessary for controlled processes depends on sustained attention, which fluctuates over time. Here, we thus tested whether P600 and N400 amplitudes depend on the level of sustained attention. We reanalyzed EEG and behavioral data from a sentence processing task by Sassenhagen and Bornkessel-Schlesewsky [The P600 as a correlate of ventral attention network reorientation. Cortex, 66, A3-A20, 2015], which included sentences with morphosyntactic and semantic violations. Participants read sentences phrase by phrase and indicated whether a sentence contained any type of anomaly as soon as they had the relevant information. To quantify the varying degrees of sustained attention, we extracted a moving reaction time coefficient of variation over the entire course of the task. We found that the P600 amplitude was significantly larger during periods of low reaction time variability (high sustained attention) than in periods of high reaction time variability (low sustained attention). In contrast, the amplitude of the N400 was not affected by reaction time variability. These results thus suggest that the P600 component is sensitive to sustained attention whereas the N400 component is not, which provides independent evidence for accounts suggesting that P600 amplitudes reflect more controlled and N400 amplitudes reflect more automatic aspects of sentence comprehension.

Electrophysiological Chronometry of Graded Consciousness during the Attentional Blink.

One of the ongoing debates about visual consciousness is whether it can be considered as an all-or-none or a graded phenomenon. While there is increasing evidence for the existence of graded states of conscious awareness based on paradigms such as visual masking, only little and mixed evidence is available for the attentional blink paradigm, specifically in regard to electrophysiological measures. Thereby, the all-or-none pattern reported in some attentional blink studies might have originated from specifics of the experimental design, suggesting the need to examine the generalizability of results. In the present event-related potential (ERP) study (N = 32), visual awareness of T2 face targets was assessed via subjective visibility ratings on a perceptual awareness scale in combination with ERPs time-locked to T2 onset (components P1, N1, N2, and P3). Furthermore, a classification task preceding visibility ratings allowed to track task performance. The behavioral results indicate a graded rather than an all-or-none pattern of visual awareness. Corresponding graded differences in the N1, N2, and P3 components were observed for the comparison of visibility levels. These findings suggest that conscious perception during the attentional blink can occur in a graded fashion.

The N400 ERP component reflects an error-based implicit learning signal during language comprehension.

The functional significance of the N400 evoked-response component is still actively debated. An increasing amount of theoretical and computational modelling work is built on the interpretation of the N400 as a prediction error. In neural network modelling work, it was proposed that the N400 component can be interpreted as the change in a probabilistic representation of meaning that drives the continuous adaptation of an internal model of the statistics of the environment. These results imply that increased N400 amplitudes should correspond to greater adaptation, which can be measured via implicit memory. To investigate this model derived hypothesis, the current study manipulated expectancy in a sentence reading task to influence N400 amplitudes and subsequently presented the previously expected vs. unexpected words in a perceptual identification task to measure implicit memory. As predicted, reaction times in the perceptual identification task were significantly faster for previously unexpected words that induced larger N400 amplitudes in the previous sentence reading task. Additionally, it could be demonstrated that this adaptation seems to specifically depend on the process underlying N400 amplitudes, as participants with larger N400 differences during sentence reading also exhibited a larger implicit memory benefit in the perceptual identification task. These findings support the interpretation of the N400 as an implicit learning signal driving adaptation in language processing.

Semantic richness and density effects on language production: Electrophysiological and behavioral evidence.

Language production ultimately aims to convey meaning. Yet words differ widely in the richness and density of their semantic representations, and these differences impact conceptual and lexical processes during speech planning. Here, we replicated the recent finding that semantic richness, measured as the number of associated semantic features according to semantic feature production norms, facilitates object naming. In contrast, intercorrelational semantic feature density, measured as the degree of intercorrelation of a concept's features, presumably resulting in the coactivation of closely related concepts, has an inhibitory influence. We replicated the behavioral effects and investigated their relative time course and electrophysiological correlates. Both the facilitatory effect of high semantic richness and the inhibitory influence of high feature density were reflected in an increased posterior positivity starting at about 250 ms, in line with previous reports of posterior positivities in paradigms employing contextual manipulations to induce semantic interference during language production. Furthermore, amplitudes at the same posterior electrode sites were positively correlated with object naming times between about 230 and 380 ms. The observed effects follow naturally from the assumption of conceptual facilitation and simultaneous lexical competition and are difficult to explain by language production theories dismissing lexical competition. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Change in a probabilistic representation of meaning can account for N400 effects on articles: A neural network model.

Increased N400 amplitudes on indefinite articles (a/an) incompatible with expected nouns have been initially taken as strong evidence for probabilistic pre-activation of phonological word forms, and recently been intensely debated because they have been difficult to replicate. Here, these effects are simulated using a neural network model of sentence comprehension that we previously used to simulate a broad range of empirical N400 effects. The model produces the effects when the cue validity of the articles concerning upcoming noun meaning in the learning environment is high, but fails to produce the effects when the cue validity of the articles is low due to adjectives presented between articles and nouns during training. These simulations provide insight into one of the factors potentially contributing to the small size of the effects in empirical studies and generate predictions for cross-linguistic differences in article induced N400 effects based on articles' cue validity. The model accounts for article induced N400 effects without assuming pre-activation of word forms, and instead simulates these effects as the stimulus-induced change in a probabilistic representation of meaning corresponding to an implicit semantic prediction error.

Dissociating neural learning signals in human sign- and goal-trackers.

Individuals differ in how they learn from experience. In Pavlovian conditioning models, where cues predict reinforcer delivery at a different goal location, some animals-called sign-trackers-come to approach the cue, whereas others, called goal-trackers, approach the goal. In sign-trackers, model-free phasic dopaminergic reward-prediction errors underlie learning, which renders stimuli 'wanted'. Goal-trackers do not rely on dopamine for learning and are thought to use model-based learning. We demonstrate this double dissociation in 129 male humans using eye-tracking, pupillometry and functional magnetic resonance imaging informed by computational models of sign- and goal-tracking. We show that sign-trackers exhibit a neural reward prediction error signal that is not detectable in goal-trackers. Model-free value only guides gaze and pupil dilation in sign-trackers. Goal-trackers instead exhibit a stronger model-based neural state prediction error signal. This model-based construct determines gaze and pupil dilation more in goal-trackers.

Quasi-compositional mapping from form to meaning: a neural network-based approach to capturing neural responses during human language comprehension.

We argue that natural language can be usefully described as quasi-compositional and we suggest that deep learning-based neural language models bear long-term promise to capture how language conveys meaning. We also note that a successful account of human language processing should explain both the outcome of the comprehension process and the continuous internal processes underlying this performance. These points motivate our discussion of a neural network model of sentence comprehension, the Sentence Gestalt model, which we have used to account for the N400 component of the event-related brain potential (ERP), which tracks meaning processing as it happens in real time. The model, which shares features with recent deep learning-based language models, simulates N400 amplitude as the automatic update of a probabilistic representation of the situation or event described by the sentence, corresponding to a temporal difference learning signal at the level of meaning. We suggest that this process happens relatively automatically, and that sometimes a more-controlled attention-dependent process is necessary for successful comprehension, which may be reflected in the subsequent P600 ERP component. We relate this account to current deep learning models as well as classic linguistic theory, and use it to illustrate a domain general perspective on some specific linguistic operations postulated based on compositional analyses of natural language. This article is part of the theme issue 'Towards mechanistic models of meaning composition'.

Clear judgments based on unclear evidence: Person evaluation is strongly influenced by untrustworthy gossip.

Affective information about other people's social behavior may prejudice social interactions and bias person judgments. The trustworthiness of person-related information, however, can vary considerably, as in the case of gossip, rumors, lies, or "fake news." Here, we investigated how spontaneous person likability and explicit person judgments are influenced by trustworthiness, employing event-related potentials as indices of emotional brain responses. Social-emotional information about the (im)moral behavior of previously unknown persons was verbally presented as trustworthy fact (e.g., "He bullied his apprentice") or marked as untrustworthy gossip (by adding, e.g., allegedly), using verbal qualifiers that are frequently used in conversations, news, and social media to indicate the questionable trustworthiness of the information and as a precaution against wrong accusations. In Experiment 1, spontaneous likability, deliberate person judgments, and electrophysiological measures of emotional person evaluation were strongly influenced by negative information yet remarkably unaffected by the trustworthiness of the information. Experiment 2 replicated these findings and extended them to positive information. Our findings demonstrate a tendency for strong emotional evaluations and person judgments even when they are knowingly based on unclear evidence. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Semantic Knowledge Enhances Conscious Awareness of Visual Objects.

It is becoming increasingly established that information from long-term memory can influence early perceptual processing, a finding that is in line with recent theoretical approaches to cognition such as the predictive coding framework. Notwithstanding, the impact of semantic knowledge on conscious perception and the temporal dynamics of such an influence remain unclear. To address this question, we presented pictures of novel objects to participants as the second of two targets in an attentional blink paradigm. We found that associating newly acquired semantic knowledge to objects increased overall conscious detection in comparison to objects associated with minimal knowledge while controlling for object familiarity. Additionally, event-related brain potentials revealed a corresponding modulation beginning 100 msec after stimulus presentation in the P1 component. Furthermore, the size of this modulation was correlated with participant's subjective reports of conscious perception. These findings suggest that semantic knowledge can shape the contents of consciousness by affecting early stages of perceptual processing.

Attentional modulation of orthographic neighborhood effects during reading: Evidence from event-related brain potentials in a psychological refractory period paradigm.

It is often assumed that word reading proceeds automatically. Here, we tested this assumption by recording event-related potentials during a psychological refractory period (PRP) paradigm, requiring lexical decisions about written words. Specifically, we selected words differing in their orthographic neighborhood size-the number of words that can be obtained from a target by exchanging a single letter-and investigated how influences of this variable depend on the availability of central attention. As expected, when attentional resources for lexical decisions were unconstrained, words with many orthographic neighbors elicited larger N400 amplitudes than those with few neighbors. However, under conditions of high temporal overlap with a high priority primary task, the N400 effect was not statistically different from zero. This finding indicates strong attentional influences on processes sensitive to orthographic neighbors during word reading, providing novel evidence against the full automaticity of processes involved in word reading. Furthermore, in conjunction with the observation of an underadditive interaction between stimulus onset asynchrony (SOA) and orthographic neighborhood size in lexical decision performance, commonly taken to indicate automaticity, our results raise issues concerning the standard logic of cognitive slack in the PRP paradigm.

Modelling the N400 brain potential as change in a probabilistic representation of meaning.

The N400 component of the event-related brain potential has aroused much interest because it is thought to provide an online measure of meaning processing in the brain. However, the underlying process remains incompletely understood and actively debated. Here we present a computationally explicit account of this process and the emerging representation of sentence meaning. We simulate N400 amplitudes as the change induced by an incoming stimulus in an implicit and probabilistic representation of meaning captured by the hidden unit activation pattern in a neural network model of sentence comprehension, and we propose that the process underlying the N400 also drives implicit learning in the network. The model provides a unified account of 16 distinct findings from the N400 literature and connects human language comprehension with recent deep learning approaches to language processing.