Hierarchical dynamic coding coordinates speech comprehension in the brain.

Speech comprehension requires the human brain to transform an acoustic waveform into meaning. To do so, the brain generates a hierarchy of features that converts the sensory input into increasingly abstract language properties. However, little is known about how these hierarchical features are generated and continuously coordinated. Here, we propose that each linguistic feature is dynamically represented in the brain to simultaneously represent successive events. To test this 'Hierarchical Dynamic Coding' (HDC) hypothesis, we use time-resolved decoding of brain activity to track the construction, maintenance, and integration of a comprehensive hierarchy of language features spanning acoustic, phonetic, sub-lexical, lexical, syntactic and semantic representations. For this, we recorded 21 participants with magnetoencephalography (MEG), while they listened to two hours of short stories. Our analyses reveal three main findings. First, the brain incrementally represents and simultaneously maintains successive features. Second, the duration of these representations depend on their level in the language hierarchy. Third, each representation is maintained by a dynamic neural code, which evolves at a speed commensurate with its corresponding linguistic level. This HDC preserves the maintenance of information over time while limiting the interference between successive features. Overall, HDC reveals how the human brain continuously builds and maintains a language hierarchy during natural speech comprehension, thereby anchoring linguistic theories to their biological implementations.

Introducing MEG-MASC a high-quality magneto-encephalography dataset for evaluating natural speech processing.

The "MEG-MASC" dataset provides a curated set of raw magnetoencephalography (MEG) recordings of 27 English speakers who listened to two hours of naturalistic stories. Each participant performed two identical sessions, involving listening to four fictional stories from the Manually Annotated Sub-Corpus (MASC) intermixed with random word lists and comprehension questions. We time-stamp the onset and offset of each word and phoneme in the metadata of the recording, and organize the dataset according to the 'Brain Imaging Data Structure' (BIDS). This data collection provides a suitable benchmark to large-scale encoding and decoding analyses of temporally-resolved brain responses to speech. We provide the Python code to replicate several validations analyses of the MEG evoked responses such as the temporal decoding of phonetic features and word frequency. All code and MEG, audio and text data are publicly available to keep with best practices in transparent and reproducible research.

Neural dynamics of phoneme sequences reveal position-invariant code for content and order.

Speech consists of a continuously-varying acoustic signal. Yet human listeners experience it as sequences of discrete speech sounds, which are used to recognise discrete words. To examine how the human brain appropriately sequences the speech signal, we recorded two-hour magnetoencephalograms from 21 participants listening to short narratives. Our analyses show that the brain continuously encodes the three most recently heard speech sounds in parallel, and maintains this information long past its dissipation from the sensory input. Each speech sound representation evolves over time, jointly encoding both its phonetic features and the amount of time elapsed since onset. As a result, this dynamic neural pattern encodes both the relative order and phonetic content of the speech sequence. These representations are active earlier when phonemes are more predictable, and are sustained longer when lexical identity is uncertain. Our results show how phonetic sequences in natural speech are represented at the level of populations of neurons, providing insight into what intermediary representations exist between the sensory input and sub-lexical units. The flexibility in the dynamics of these representations paves the way for further understanding of how such sequences may be used to interface with higher order structure such as lexical identity.

Early Form-Based Morphological Decomposition in Tagalog: MEG Evidence from Reduplication, Infixation, and Circumfixation.

Neuro- and psycholinguistic experimentation supports the early decomposition of morphologically complex words within the ventral processing stream, which MEG has localized to the M170 response in the (left) visual word form area (VWFA). Decomposition into an exhaustive parse of visual morpheme forms extends beyond words like farmer to those imitating complexity (e.g., brother; Lewis et al., 2011), and to "unique" stems occurring in only one word but following the syntax and semantics of their affix (e.g., vulnerable; Gwilliams & Marantz, 2018). Evidence comes primarily from suffixation; other morphological processes have been under-investigated. This study explores circumfixation, infixation, and reduplication in Tagalog. In addition to investigating whether these are parsed like suffixation, we address an outstanding question concerning semantically empty morphemes. Some words in Tagalog resemble English winter as decomposition is not supported (wint-er); these apparently reduplicated pseudoreduplicates lack the syntactic and semantic features of reduplicated forms. However, unlike winter, these words exhibit phonological behavior predicted only if they involve a reduplicating morpheme. If these are decomposed, this provides evidence that words are analyzed as complex, like English vulnerable, when the grammar demands it. In a lexical decision task with MEG, we find that VWFA activity correlates with stem:word transition probability for circumfixed, infixed, and reduplicated words. Furthermore, a Bayesian analysis suggests that pseudoreduplicates with reduplicate-like phonology are also decomposed; other pseudoreduplicates are not. These findings are consistent with an interpretation that decomposition is modulated by phonology in addition to syntax and semantics.

Left posterior temporal cortex is sensitive to syntax within conceptually matched Arabic expressions.

During language comprehension, the brain processes not only word meanings, but also the grammatical structure-the "syntax"-that strings words into phrases and sentences. Yet the neural basis of syntax remains contentious, partly due to the elusiveness of experimental designs that vary structure independently of meaning-related variables. Here, we exploit Arabic's grammatical properties, which enable such a design. We collected magnetoencephalography (MEG) data while participants read the same noun-adjective expressions with zero, one, or two contiguously-written definite articles (e.g., 'chair purple'; 'the-chair purple'; 'the-chair the-purple'), representing equivalent concepts, but with different levels of syntactic complexity (respectively, indefinite phrases: 'a purple chair'; sentences: 'The chair is purple.'; definite phrases: 'the purple chair'). We expected regions processing syntax to respond differently to simple versus complex structures. Single-word controls ('chair'/'purple') addressed definiteness-based accounts. In noun-adjective expressions, syntactic complexity only modulated activity in the left posterior temporal lobe (LPTL), ~ 300 ms after each word's onset: indefinite phrases induced more MEG-measured positive activity. The effects disappeared in single-word tokens, ruling out non-syntactic interpretations. In contrast, left anterior temporal lobe (LATL) activation was driven by meaning. Overall, the results support models implicating the LPTL in structure building and the LATL in early stages of conceptual combination.

Adaptation to mis-pronounced speech: evidence for a prefrontal-cortex repair mechanism.

Speech is a complex and ambiguous acoustic signal that varies significantly within and across speakers. Despite the processing challenge that such variability poses, humans adapt to systematic variations in pronunciation rapidly. The goal of this study is to uncover the neurobiological bases of the attunement process that enables such fluent comprehension. Twenty-four native English participants listened to words spoken by a "canonical" American speaker and two non-canonical speakers, and performed a word-picture matching task, while magnetoencephalography was recorded. Non-canonical speech was created by including systematic phonological substitutions within the word (e.g. [s] → [sh]). Activity in the auditory cortex (superior temporal gyrus) was greater in response to substituted phonemes, and, critically, this was not attenuated by exposure. By contrast, prefrontal regions showed an interaction between the presence of a substitution and the amount of exposure: activity decreased for canonical speech over time, whereas responses to non-canonical speech remained consistently elevated. Grainger causality analyses further revealed that prefrontal responses serve to modulate activity in auditory regions, suggesting the recruitment of top-down processing to decode non-canonical pronunciations. In sum, our results suggest that the behavioural deficit in processing mispronounced phonemes may be due to a disruption to the typical exchange of information between the prefrontal and auditory cortices as observed for canonical speech.

Memory for affixes in a long-lag priming paradigm.

Psycholinguistic research on the processing of morphologically complex words has largely focused on debates about how/if lexical stems are recognized, stored, and retrieved. Comparatively little processing research has investigated similar issues for functional affixes. In Word or Lexeme Based Morphology (Aronoff 1994), affixes are not representational units on par with stems or roots. This view is in stark contrast to the claims of linguistic theories like Distributed Morphology (Halle & Marantz 1993), which assign rich representational content to affixes. We conducted a series of eight visual lexical decision studies, evaluating effects of derivational affix priming along with stem priming, identity priming, form priming, and semantic priming at long and short lags. We find robust and consistent affix priming (but not semantic or form priming) with lags up to 33 items, supporting the position that affixes are morphemes, i.e., representational units on par with stems. Intriguingly, we find only weaker evidence for the long-lag stem priming effect found in other studies. We interpret this potential asymmetry in terms of the salience of different morphological contexts for recollection memory.

Modeling Human Morphological Competence.

One of the central debates in the cognitive science of language has revolved around the nature of human linguistic competence. Whether syntactic competence should be characterized by abstract hierarchical structures or reduced to surface linear strings has been actively debated, but the nature of morphological competence has been insufficiently appreciated despite the parallel question in the cognitive science literature. In this paper, in order to investigate whether morphological competence should be characterized by abstract hierarchical structures, we conducted a crowdsourced acceptability judgment experiment on morphologically complex words and evaluated five computational models of morphological competence against human acceptability judgments: Character Markov Models (Character), Syllable Markov Models (Syllable), Morpheme Markov Models (Morpheme), Hidden Markov Models (HMM), and Probabilistic Context-Free Grammars (PCFG). Our psycholinguistic experimentation and computational modeling demonstrated that "morphous" computational models with morpheme units outperformed "amorphous" computational models without morpheme units and, importantly, PCFG with hierarchical structures most accurately explained human acceptability judgments on several evaluation metrics, especially for morphologically complex words with nested morphological structures. Those results strongly suggest that human morphological competence should be characterized by abstract hierarchical structures internally generated by the grammar, not reduced to surface linear strings externally attested in large corpora.

Left occipital and right frontal involvement in syntactic category prediction: MEG evidence from Standard Arabic.

Though recent years have seen a growth in research on predictive processes in language comprehension, their scope and mechanisms remain partially elusive. While mechanisms involved in predicting specific words are relatively well understood, those underlying syntactic prediction are still unclear. In part, this is because of the difficulty in designing experiments that manipulate syntactic predictability while controlling other variables. In this MEG study, we achieved this with a manipulation of syntactic category predictability within fully well-formed expressions of Standard Arabic. Participants read sentences beginning with a subject-adjective context, in which the presence of at least one of two possible cues (gender-incongruity and/or an intervening relative pronoun) was sufficient for predicting a target word's syntactic category. Absence of both cues (i.e., congruent subject-adjective context with no relative pronoun) increased uncertainty about the target's syntactic category. Using source analysis, we compared activity evoked by targets with predictable and unpredictable categories in the occipital lobe. We found an interaction effect consistent with previous findings: in the primary visual cortex, an early evoked component (visual M100) is enhanced only when the syntactic category was unpredictable. We also compared responses to pre-target predictive and unpredictive contexts across five bilateral frontal and temporal regions. In the right-hemispheric frontal region, we found a temporal cluster (~230 ms after adjective onset), where unpredictive contexts elicited more activation than predictive contexts. By hypothesis elimination, we conclude that the most likely variable driving this effect is syntactic entropy. Our results show that predictive mechanisms recruited during reading also involve predicting upcoming syntactic categories, implicating at least two cortical regions: the left visual cortex and the right frontal cortex.

Prefix Stripping Re-Re-Revisited: MEG Investigations of Morphological Decomposition and Recomposition.

We revisit a long-standing question in the psycholinguistic and neurolinguistic literature on comprehending morphologically complex words: are prefixes and suffixes processed using the same cognitive mechanisms? Recent work using Magnetoencephalography (MEG) to uncover the dynamic temporal and spatial responses evoked by visually presented complex suffixed single words provide us with a comprehensive picture of morphological processing in the brain, from early, form-based decomposition, through lexical access, grammatically constrained recomposition, and semantic interpretation. In the present study, we find that MEG responses to prefixed words reveal interesting early differences in the lateralization of the form-based decomposition response compared to the effects reported in the literature for suffixed words, but a very similar post-decomposition profile. These results not only address a question stretching back to the earliest days of modern psycholinguistics, but also add critical support and nuance to our much newer emerging understanding of spatial organization and temporal dynamics of morphological processing in the human brain.

The Form of Morphemes: MEG Evidence From Masked Priming of Two Hebrew Templates.

Studies of lexical access have benefited from comparisons between languages like English, which shows concatenative morphology, and Semitic languages showing non-concatenative morphology of roots and patterns. Morphological decomposition in Semitic has previously been probed using masked priming, originally developed to investigate concatenative morphology. However, studies conducted on Semitic languages have often targeted Semitic-specific questions, such as whether the root and the verbal template prime lexical access. The overall consequence of these studies for our understanding of lexical access remains unclear. In two experiments on Hebrew using MEG, we demonstrate that a verbal form which is orthographically and phonologically indistinguishable from non-verbal forms is primed by other verbs in the same template but not by similar nouns and adjectives. These results suggest that masked priming taps into more than just visual forms but reflects morphological content, even if this content is abstract, showing no distinct orthographic or phonological marking.

Building words and phrases in the left temporal lobe.

A central part of knowing a language is the ability to combine basic linguistic units to form complex representations. While our neurobiological understanding of how words combine into larger structures has significantly advanced in recent years, the combinatory operations that build words themselves remain unknown. Are complex words such as tombstone and starlet built with the same mechanisms that construct phrases from words, such as grey stone or bright star? Here we addressed this with two magnetoencephalography (MEG) experiments, which simultaneously varied demands associated with phrasal composition, and the processing of morphological complexity in compound and suffixed nouns. Replicating previous findings, we show that portions of the left anterior temporal lobe (LATL) are engaged in the combination of modifiers and monomorphemic nouns in phrases (e.g., brown rabbit). As regards compounding, we show that semantically transparent compounds (e.g., tombstone) also engage left anterior temporal cortex, though the spatiotemporal details of this effect differed from phrasal composition. Further, when a phrase was constructed from a modifier and a transparent compound (e.g., granite tombstone), the typical LATL phrasal composition response appeared at a delayed latency, which follows if an initial within-word operation (tomb + stone) must take place before the combination of the compound with the preceding modifier (granite + tombstone). In contrast to compounding, suffixation (i.e., star + let) did not engage the LATL in any consistent way, suggesting a distinct processing route. Finally, our results suggest an intriguing generalization that morpho-orthographic complexity that does not recruit the LATL may block the engagement of the LATL in subsequent phrase building. In sum, our findings offer a detailed spatiotemporal characterization of the lowest level combinatory operations that ultimately feed the composition of full sentences.

In Spoken Word Recognition, the Future Predicts the Past.

Speech is an inherently noisy and ambiguous signal. To fluently derive meaning, a listener must integrate contextual information to guide interpretations of the sensory input. Although many studies have demonstrated the influence of prior context on speech perception, the neural mechanisms supporting the integration of subsequent context remain unknown. Using MEG to record from human auditory cortex, we analyzed responses to spoken words with a varyingly ambiguous onset phoneme, the identity of which is later disambiguated at the lexical uniqueness point. Fifty participants (both male and female) were recruited across two MEG experiments. Our findings suggest that primary auditory cortex is sensitive to phonological ambiguity very early during processing at just 50 ms after onset. Subphonemic detail is preserved in auditory cortex over long timescales and re-evoked at subsequent phoneme positions. Commitments to phonological categories occur in parallel, resolving on the shorter timescale of ∼450 ms. These findings provide evidence that future input determines the perception of earlier speech sounds by maintaining sensory features until they can be integrated with top-down lexical information.SIGNIFICANCE STATEMENT The perception of a speech sound is determined by its surrounding context in the form of words, sentences, and other speech sounds. Often, such contextual information becomes available later than the sensory input. The present study is the first to unveil how the brain uses this subsequent information to aid speech comprehension. Concretely, we found that the auditory system actively maintains the acoustic signal in auditory cortex while concurrently making guesses about the identity of the words being said. Such a processing strategy allows the content of the message to be accessed quickly while also permitting reanalysis of the acoustic signal to minimize parsing mistakes.

Syntactic Priming As a Test of Argument Structure: A Self-paced Reading Experiment.

Using data from a behavioral structural priming experiment, we test two competing theoretical approaches to argument structure, which attribute different configurations to (in)transitive structures. These approaches make different claims about the relationship between unergatives and transitive structures selecting either a DP complement or a small clause complement in structurally unambiguous sentences, thus making different predictions about priming relations between them. Using statistical tools that combine a factorial 6 × 6 within subjects ANOVA, a mixed effects ANCOVA and a linear mixed effects regression model, we report syntactic priming effects in comprehension, which suggest a stronger predictive contribution of a model that supports an interpretive semantics view of syntax, whereby syntactic structures do not necessarily reflect argument/event structure in semantically unambiguous configurations. They also contribute novel experimental evidence that correlate representational complexity with language processing in the mind and brain. Our study further upholds the validity of combining quantitative methods and theoretical approaches to linguistics for advancing our knowledge of syntactic phenomena.

Lexical Preactivation in Basic Linguistic Phrases.

Many previous studies have shown that predictable words are read faster and lead to reduced neural activation, consistent with a model of reading in which words are activated in advance of being encountered. The nature of such preactivation, however, has typically been studied indirectly through its subsequent effect on word recognition. Here, we use magnetoencephalography to study the dynamics of prediction within serially presented adjective-noun phrases, beginning at the point at which the predictive information is first available to the reader. Using corpus transitional probability to estimate the predictability of a noun, we found an increase in activity in the left middle temporal gyrus in response to the presentation of highly predictive adjectives (i.e., adjectives that license a strong noun prediction). Moreover, we found that adjective predictivity and expected noun frequency interacted, such that the response to the highly predictive adjectives (e.g., stainless) was modulated by the frequency of the expected noun (steel). These results likely reflect preactivation of nouns in highly predictive contexts. The fact that the preactivation process was modulated by the frequency of the predicted item is argued to provide support for a frequency-sensitive lexicon.

Decomposition, lookup, and recombination: MEG evidence for the full decomposition model of complex visual word recognition.

There is much evidence that visual recognition of morphologically complex words (e.g., teacher) proceeds via a decompositional route, first involving recognition of their component morphemes (teach + -er). According to the Full Decomposition model, after the visual decomposition stage, followed by morpheme lookup, there is a final "recombination" stage, in which the decomposed morphemes are combined and the well-formedness of the complex form is evaluated. Here, we use MEG to provide evidence for the temporally-differentiated stages of this model. First, we demonstrate an early effect of derivational family entropy, corresponding to the stem lookup stage; this is followed by a surface frequency effect, corresponding to the later recombination stage. We also demonstrate a late effect of a novel statistical measure, semantic coherence, which quantifies the gradient semantic well-formedness of complex words. Our findings illustrate the usefulness of corpus measures in investigating the component processes within visual word recognition.

The role of morphology in phoneme prediction: evidence from MEG.

There is substantial neural evidence for the role of morphology (word-internal structure) in visual word recognition. We extend this work to auditory word recognition, drawing on recent evidence that phoneme prediction is central to this process. In a magnetoencephalography (MEG) study, we crossed morphological complexity (bruis-er vs. bourbon) with the predictability of the word ending (bourbon vs. burble). High prediction error (surprisal) led to increased auditory cortex activity. This effect was enhanced for morphologically complex words. Additionally, we calculated for each timepoint the surprisal corresponding to the phoneme perceived at that timepoint, as well as the cohort entropy, which quantifies the competition among words compatible with the string prefix up to that timepoint. Higher surprisal increased neural activity at the end of the word, and higher entropy decreased neural activity shortly after word onset. These results reinforce the role of morphology and phoneme prediction in spoken word recognition.

MEG masked priming evidence for form-based decomposition of irregular verbs.

To what extent does morphological structure play a role in early processing of visually presented English past tense verbs? Previous masked priming studies have demonstrated effects of obligatory form-based decomposition for genuinely affixed words (teacher-TEACH) and pseudo-affixed words (corner-CORN), but not for orthographic controls (brothel-BROTH). Additionally, MEG single word reading studies have demonstrated that the transition probability from stem to affix (in genuinely affixed words) modulates an early evoked response known as the M170; parallel findings have been shown for the transition probability from stem to pseudo-affix (in pseudo-affixed words). Here, utilizing the M170 as a neural index of visual form-based morphological decomposition, we ask whether the M170 demonstrates masked morphological priming effects for irregular past tense verbs (following a previous study which obtained behavioral masked priming effects for irregulars). Dual mechanism theories of the English past tense predict a rule-based decomposition for regulars but not for irregulars, while certain single mechanism theories predict rule-based decomposition even for irregulars. MEG data was recorded for 16 subjects performing a visual masked priming lexical decision task. Using a functional region of interest (fROI) defined on the basis of repetition priming and regular morphological priming effects within the left fusiform and inferior temporal regions, we found that activity in this fROI was modulated by the masked priming manipulation for irregular verbs, during the time window of the M170. We also found effects of the scores generated by the learning model of Albright and Hayes (2003) on the degree of priming for irregular verbs. The results favor a single mechanism account of the English past tense, in which even irregulars are decomposed into stems and affixes prior to lexical access, as opposed to a dual mechanism model, in which irregulars are recognized as whole forms.

A MEG investigation of single-word auditory comprehension in aphasia.

PURPOSE: To explore whether individuals with aphasia exhibit differences in the M350, an electrophysiological marker of lexical activation, compared with healthy controls. METHOD: Seven people with aphasia, 9 age-matched controls, and 10 younger controls completed an auditory lexical decision task while cortical activity was recorded with magnetoencephalography. There were 2 stimulus conditions of interest: identity primed (i.e., a word preceded by itself) and semantic primed (i.e., a word preceded by a semantically related word). Latency and amplitude of the M350 response as well as reaction time were measured. RESULTS: Consistent with the age-matched control group, the group with aphasia showed both identity and semantic priming behaviorally. In contrast to the control groups, the group with aphasia did not show either semantic or identity priming of the M350 response. This group also demonstrated longer M350 latencies than either control group. Furthermore, within this group, M350 latency was positively correlated with a measure of semantic impairment. CONCLUSIONS: These findings highlight the usefulness of temporally sensitive measures when studying aphasia and demonstrate that the latency of electrophysiological markers is of interest in this population. In particular, increased M350 latency appears to be indicative of a semantic processing impairment.

The neural basis of obligatory decomposition of suffixed words.

Recent neurolinguistic studies present somewhat conflicting evidence concerning the role of the inferior temporal cortex (IT) in visual word recognition within the first 200 ms after presentation. On the one hand, fMRI studies of the Visual Word Form Area (VWFA) suggest that the IT might recover representations of the orthographic form of words. On the other hand, influential MEG studies of responses from the occipito-temporal regions around 150 ms post-stimulus onset indicate recognition of letters as opposed to symbols but not a sensitivity to statistical properties of letter strings associated with word form representations. Recent MEG experiments support the position that the IT does represent the visual word forms of morphemes and performs morphological decomposition modulated by the statistical relations between morphemes by 170 ms post presentation (at the M170 response). Responses to heteronyms show that the M170 does not make contact with the mental lexicon where word forms are connected to meanings. We report here an MEG study of pseudo-affixed words like brother, which masked priming studies have shown are decomposed in recognition. If the M170 response from IT does index obligatory morphological decomposition based on visual word forms but not lexical entries, we should find that the statistical relation between pseudo-stem and pseudo-suffix modulates the M170 for pseudo-affixed words, as for truly affixed words. The results of this experiment confirm this prediction. In addition, surface form frequency for these words also modulates the M170, providing some support for dual route recognition for words for which decomposition is a garden path.