The relationship of speech perception and speech production: It's complicated.

A widely held belief is that speech perception and speech production are tightly linked, with each modality available to help with learning in the other modality. This positive relationship is often summarized as perception and production being "two sides of the same coin." There are, indeed, many situations that have shown this mutually supportive relationship. However, there is a growing body of research showing very different results, with the modalities operating independently, or even in opposition to each other. We review the now-sizeable literature demonstrating the negative effect that speech production can have on perceptual learning of speech, at multiple levels (particularly at the lexical and sublexical levels). By comparing the situations that show this pattern with ones in which more positive interactions occur, we provide an initial account of why the different outcomes are found, identifying factors that lead to either positive or negative effects of production on perception. The review clarifies the complex relationship that exists between the two modalities: They are indeed linked, but their relationship is more complicated than is suggested by the notion that they are two sides of the same coin.

Phonetic categorization in phonological lexical neighborhoods: Facilitatory and inhibitory effects.

Phonetic processing, whereby the bottom-up speech signal is translated into higher-level phonological representations such as phonemes, has been demonstrated to be influenced by phonological lexical neighborhoods. Previous studies show facilitatory effects of lexicality and phonological neighborhood density on phonetic categorization. However, given the evidence for lexical competition in spoken word recognition, we hypothesize that there are concurrent facilitatory and inhibitory effects of phonological lexical neighborhoods on phonetic processing. In Experiments 1 and 2, participants categorized the onset phoneme in word-nonword and nonword-word acoustic continua. The results show that the target word of the continuum exhibits facilitatory lexical influences whereas rhyme neighbors inhibit phonetic categorization. The results support the hypothesis that sublexical phonetic processing is affected by multiple facilitatory and inhibitory lexical forces in the processing stream.

The development of real-time spoken and word recognition derives from changes in ability, not maturation.

In typical adults, recognizing both spoken and written words is thought to be served by a process of competition between candidates in the lexicon. In recent years, work has used eye-tracking in the visual world paradigm to characterize this competition process over development. It has shown that both spoken and written word recognition continue to develop through adolescence (Rigler et al., 2015). It is still unclear what drives these changes in real-time word recognition over the school years, as there are dramatic changes in language, the onset of reading instruction, and gains in domain general function during this time. This study began to address these issues by asking whether changes in real-time word recognition derive from changes in overall language and reading ability or reflect more general age-related development. This cross-sectional study examined 278 school-age children (Grades 1-3) using the Visual World Paradigm to assess both spoken and written word recognition, along with multiple measures of language, reading and phonology. A structural equation model applied to these ability measures found three factors representing language, reading, and phonology. Multiple regression analyses were used to understand how these three factors relate to real-time spoken and written word recognition as well as a non-linguistic variant of the VWP intended to capture decision speed, eye-movement factors, and other non-language/reading differences. We found that for both spoken and written word recognition, the speed of activating target words in both domains was more closely tied to the relevant ability (e.g., reading for written word recognition) than was age. We also examined competition resolution (how fully competitors were suppressed late in processing). Here, spoken word recognition showed only small, developmental effects that were only related to phonological processing, suggesting links to developmental language disorder. However, in written word recognition, competitor resolution showed large impacts of development which were strongly linked to reading. This suggests the dimensionality of real-time lexical processing may differ across domains. Importantly, neither spoken nor written word recognition is fully described by changes in non-linguistic skills assessed with non-linguistic VWP, and the non-linguistic VWP was linked to differences in language and reading. These findings suggest that spoken and written word recognition continue past the first year of life and are mostly driven by ability and not only by overall maturation.

Word Forms Reflect Trade-Offs Between Speaker Effort and Robust Listener Recognition.

How do cognitive pressures shape the lexicons of natural languages? Here, we reframe George Kingsley Zipf's proposed "law of abbreviation" within a more general framework that relates it to cognitive pressures that affect speakers and listeners. In this new framework, speakers' drive to reduce effort (Zipf's proposal) is counteracted by the need for low-frequency words to have word forms that are sufficiently distinctive to allow for accurate recognition by listeners. To support this framework, we replicate and extend recent work using the prevalence of subword phonemic sequences (phonotactic probability) to measure speakers' production effort in place of Zipf's measure of length. Across languages and corpora, phonotactic probability is more strongly correlated with word frequency than word length. We also show this measure of ease of speech production (phonotactic probability) is strongly correlated with a measure of perceptual difficulty that indexes the degree of competition from alternative interpretations in word recognition. This is consistent with the claim that there must be trade-offs between these two factors, and is inconsistent with a recent proposal that phonotactic probability facilitates both perception and production. To our knowledge, this is the first work to offer an explanation why long, phonotactically improbable word forms remain in the lexicons of natural languages.

Phase-dependent word perception emerges from region-specific sensitivity to the statistics of language.

Neural oscillations reflect fluctuations in excitability, which biases the percept of ambiguous sensory input. Why this bias occurs is still not fully understood. We hypothesized that neural populations representing likely events are more sensitive, and thereby become active on earlier oscillatory phases, when the ensemble itself is less excitable. Perception of ambiguous input presented during less-excitable phases should therefore be biased toward frequent or predictable stimuli that have lower activation thresholds. Here, we show such a frequency bias in spoken word recognition using psychophysics, magnetoencephalography (MEG), and computational modelling. With MEG, we found a double dissociation, where the phase of oscillations in the superior temporal gyrus and medial temporal gyrus biased word-identification behavior based on phoneme and lexical frequencies, respectively. This finding was reproduced in a computational model. These results demonstrate that oscillations provide a temporal ordering of neural activity based on the sensitivity of separable neural populations.

Lexical Feedback in the Time-Invariant String Kernel (TISK) Model of Spoken Word Recognition.

The Time-Invariant String Kernel (TISK) model of spoken word recognition (Hannagan, Magnuson & Grainger, 2013; You & Magnuson, 2018) is an interactive activation model with many similarities to TRACE (McClelland & Elman, 1986). However, by replacing most time-specific nodes in TRACE with time-invariant open-diphone nodes, TISK uses orders of magnitude fewer nodes and connections than TRACE. Although TISK performed remarkably similarly to TRACE in simulations reported by Hannagan et al., the original TISK implementation did not include lexical feedback, precluding simulation of top-down effects, and leaving open the possibility that adding feedback to TISK might fundamentally alter its performance. Here, we demonstrate that when lexical feedback is added to TISK, it gains the ability to simulate top-down effects without losing the ability to simulate the fundamental phenomena tested by Hannagan et al. Furthermore, with feedback, TISK demonstrates graceful degradation when noise is added to input, although parameters can be found that also promote (less) graceful degradation without feedback. We review arguments for and against feedback in cognitive architectures, and conclude that feedback provides a computationally efficient basis for robust constraint-based processing.

The development of lexical processing: Real-time phonological competition and semantic activation in school age children.

Prior research suggests that the development of speech perception and word recognition stabilises in early childhood. However, recent work suggests that development of these processes continues throughout adolescence. This study aimed to investigate whether these developmental changes are based solely within the lexical system or are due to domain general changes, and to extend this investigation to lexical-semantic processing. We used two Visual World Paradigm tasks: one to examine phonological and semantic processing, one to capture non-linguistic domain-general skills. We tested 43 seven- to nine-year-olds, 42 ten- to thirteen-year-olds, and 30 sixteen- to seventeen-year-olds. Older children were quicker to fixate the target word and exhibited earlier onset and offset of fixations to both semantic and phonological competitors. Visual/cognitive skills explained significant, but not all, variance in the development of these effects. Developmental changes in semantic activation were largely attributable to changes in upstream phonological processing. These results suggest that the concurrent development of linguistic processes and broader visual/cognitive skills lead to developmental changes in real-time phonological competition, while semantic activation is more stable across these ages.

Resolving competing predictions in speech: How qualitatively different cues and cue reliability contribute to phoneme identification.

Listeners have many sources of information available in interpreting speech. Numerous theoretical frameworks and paradigms have established that various constraints impact the processing of speech sounds, but it remains unclear how listeners might simultaneously consider multiple cues, especially those that differ qualitatively (i.e., with respect to timing and/or modality) or quantitatively (i.e., with respect to cue reliability). Here, we establish that cross-modal identity priming can influence the interpretation of ambiguous phonemes (Exp. 1, N = 40) and show that two qualitatively distinct cues - namely, cross-modal identity priming and auditory co-articulatory context - have additive effects on phoneme identification (Exp. 2, N = 40). However, we find no effect of quantitative variation in a cue - specifically, changes in the reliability of the priming cue did not influence phoneme identification (Exp. 3a, N = 40; Exp. 3b, N = 40). Overall, we find that qualitatively distinct cues can additively influence phoneme identification. While many existing theoretical frameworks address constraint integration to some degree, our results provide a step towards understanding how information that differs in both timing and modality is integrated in online speech perception.

Examining listeners' perception of spoken words with different face masks.

The COVID-19 pandemic made face masks part of daily life. While masks protect against the virus, it is important to understand the impact masks have on listeners' recognition of spoken words. We examined spoken word recognition under three different mask conditions (no mask; cloth mask; Kn95 mask) and in both easy (low density, high phonotactic probability) and hard (high density, low phonotactic probability) words in a lexical decision task. In Experiment 1, participants heard all words and nonwords under all three mask conditions. In Experiment 2, participants heard each word and nonword only once under one of the mask conditions. The reaction time and accuracy results were consistent between Experiments 1 and 2. The pattern of results was such that the no mask condition produced the fastest and most accurate responses followed by the Kn95 mask condition and the cloth mask condition, respectively. Furthermore, there was a trend towards a speed-accuracy trade-off with Word Type. Easy words produced faster but less accurate responses relative to hard words. The finding that cloth masks had a more detrimental impact on spoken word recognition than Kn95 masks is consistent with previous research, and the current results further demonstrate that this effect extends to individual word recognition tasks with only audio presentation.

Contra assertions, feedback improves word recognition: How feedback and lateral inhibition sharpen signals over noise.

Whether top-down feedback modulates perception has deep implications for cognitive theories. Debate has been vigorous in the domain of spoken word recognition, where competing computational models and agreement on at least one diagnostic experimental paradigm suggest that the debate may eventually be resolvable. Norris and Cutler (2021) revisit arguments against lexical feedback in spoken word recognition models. They also incorrectly claim that recent computational demonstrations that feedback promotes accuracy and speed under noise (Magnuson et al., 2018) were due to the use of the Luce choice rule rather than adding noise to inputs (noise was in fact added directly to inputs). They also claim that feedback cannot improve word recognition because feedback cannot distinguish signal from noise. We have two goals in this paper. First, we correct the record about the simulations of Magnuson et al. (2018). Second, we explain how interactive activation models selectively sharpen signals via joint effects of feedback and lateral inhibition that boost lexically-coherent sublexical patterns over noise. We also review a growing body of behavioral and neural results consistent with feedback and inconsistent with autonomous (non-feedback) architectures, and conclude that parsimony supports feedback. We close by discussing the potential for synergy between autonomous and interactive approaches.

Evidence for conflict monitoring during speech recognition in noise.

During difficult tasks, conflict can benefit performance on a subsequent trial. One theory for such performance adjustments is that people monitor for conflict and reactively engage cognitive control. This hypothesis has been challenged because tasks that control for associative learning do not show such "cognitive control" effects. The current study experimentally controlled associative learning by presenting a novel stimulus on every trial of a picture-speech conflict task and found that performance adjustments still occur. Thirty-one healthy young adults listened to and repeated words presented in background noise while viewing pictures that were congruent or incongruent (i.e., phonological neighbors) with the word. Following conflict, participants had higher word recognition (+17% points) on incongruent but not congruent trials. This result was not attributable to posterror effects nor a speed-accuracy trade-off. An analysis of erroneous responses showed that participants made more phonologically related errors than nonrelated errors only on incongruent trials, demonstrating elevated phonological conflict when the picture was a neighbor of the target word. Additionally, postconflict improvements appear to be due to better resolution of phonological conflict in the mental lexicon rather than decreased attention to the picture or increased attention to the speech signal. Our findings provide new evidence for conflict monitoring and suggest that cognitive control helps resolve phonological conflict during speech recognition in noise.

Tracking the time-course of spoken word recognition of Cantonese Chinese in sentence context: Evidence from eye movements.

In this study, we conducted an eye-tracking experiment to investigate the effects of sentence context and tonal information on spoken word recognition processes in Cantonese Chinese. We recruited 60 native Cantonese listeners to participate in the eye-tracking experiment. The target words (phonologically similar words) were manipulated to either (1) a congruent context or (2) an incongruent context in the experiment. The resulting eye-movement patterns in the incongruent context condition clearly revealed that (1) sentence context produced a garden-path effect in the initial stage of the spoken word recognition processes and then (2) the lexical tone of the word (bottom-up information) overrode the contextual effects to help listeners to discriminate between different similar-sounding words during lexical access. In conclusion, the patterns of eye-tracking data show the interactive processes between the lexical tone (an acoustic cue within a Cantonese word) and sentence context played in different phases to the spoken word recognition of Cantonese Chinese.

When phonological neighbours cooperate during spoken sentence processing.

This study examined for the first time the impact of the presence of a phonological neighbour on word recognition when the target word and its neighbour co-occur in a spoken sentence. To do so, we developed a new task, the verb detection task, in which participants were instructed to respond as soon as they detected a verb in a sequence of words, thus allowing us to probe spoken word recognition processes in real time. We found that participants were faster at detecting a verb when it was phonologically related to the preceding noun than when it was phonologically unrelated. This effect was found with both correct sentences (Experiment 1) and with ungrammatical sequences of words (Experiment 2). The effect was also found in Experiment 3 where adjacent phonologically related words were included in the non-verb condition (i.e., word sequences not containing a verb), thus ruling out any strategic influences. These results suggest that activation persists across different words during spoken sentence processing such that processing of a word at position n + 1 benefits from the sublexical phonology activated during processing of the word at position n. We discuss how different models of spoken word recognition might be able (or not) to account for these findings.

Influence of face masks on recalibration of phonetic categories.

Previous research demonstrates listeners dynamically adjust phonetic categories in line with lexical context. While listeners show flexibility in adapting speech categories, recalibration may be constrained when variability can be attributed externally. It has been hypothesized that when listeners attribute atypical speech input to a causal factor, phonetic recalibration is attenuated. The current study investigated this theory directly by examining the influence of face masks, an external factor that affects both visual and articulatory cues, on the magnitude of phonetic recalibration. Across four experiments, listeners completed a lexical decision exposure phase in which they heard an ambiguous sound in either /s/-biasing or /ʃ/-biasing lexical contexts, while simultaneously viewing a speaker with a mask off, mask on the chin, or mask over the mouth. Following exposure, all listeners completed an auditory phonetic categorization test along an /ʃ/-/s/ continuum. In Experiment 1 (when no face mask was present during exposure trials), Experiment 2 (when the face mask was on the chin), Experiment 3 (when the face mask was on the mouth during ambiguous items), and Experiment 4 (when the face mask was on the mouth during the entire exposure phase), listeners showed a robust and equivalent phonetic recalibration effect. Recalibration manifested as greater proportion /s/ responses for listeners in the /s/-biased exposure group, relative to listeners in the /ʃ/-biased exposure group. Results support the notion that listeners do not causally attribute face masks with speech idiosyncrasies, which may reflect a general speech learning adjustment during the COVID-19 pandemic.

Spoken word recognition in a second language: The importance of phonetic details.

Spoken word recognition depends on variations in fine-grained phonetics as listeners decode speech. However, many models of second language (L2) speech perception focus on units such as isolated syllables, and not on words. In two eye-tracking experiments, we investigated how fine-grained phonetic details (i.e. duration of nasalization on contrastive and coarticulatory nasalized vowels in Canadian French) influenced spoken word recognition in an L2, as compared to a group of native (L1) listeners. Results from L2 listeners (English-native speakers) indicated that fine-grained phonetics impacted the recognition of words, i.e. they were able to use nasalization duration variability in a way similar to L1-French listeners, providing evidence that lexical representations can be highly specified in an L2. Specifically, L2 listeners were able to distinguish minimal word pairs (differentiated by the presence of phonological vowel nasalization in French) and were able to use variability in a way approximating L1-French listeners. Furthermore, the robustness of the French "nasal vowel" category in L2 listeners depended on age of exposure. Early bilinguals displayed greater sensitivity to some ambiguity in the stimuli than late bilinguals, suggesting that early bilinguals had greater sensitivity to small variations in the signal and thus better knowledge of the phonetic cue associated with phonological vowel nasalization in French, similarly to L1 listeners.

Spoken Word Recognition across Language Boundary: ERP Evidence of Prosodic Transfer Driven by Pitch.

Extensive research has explored the perception of English lexical stress by Chinese EFL learners and tried to unveil the underlying mechanism of the prosodic transfer from a native tonal language to a non-native stress language. However, the role of the pitch as the shared cue by lexical stress and lexical tone during the transfer remains controversial when the segmental cue (i.e., reduced vowel) is absent. By employing event-related potential (ERP) measurements, the current study aimed to further investigate the role of the pitch during the prosodic transfer from L1 lexical tone to L2 lexical stress and the underlying neural responses. Two groups of adult Chinese EFL learners were compared, as both Mandarin and Cantonese are tonal languages with different levels of complexity. The results showed that Cantonese speakers relied more than Mandarin speakers on pitch cues, not only in their processing of English lexical stress but also in word recognition. Our findings are consistent with the arguments of Cue Weighting and attest to the influence of native tonal language experience on second language acquisition. The results may have implications on pedagogical methods that pitch could be an important clue in second language teaching.

Full-form vs. combinatorial processing of Chinese compound words: Evidence from mismatch negativity.

This study examined whether Chinese spoken compound words are processed via full-form access or combination through morphemes by recording mismatch negativity (MMN). MMN has been shown to be larger for linguistic units that involves full-form access (lexical MMN enhancement) and smaller for separate but combinable units (combinatorial MMN reduction). Chinse compound words were compared against pseudocompounds, which do not have full-form representations in the long-term memory and are "illegal" combinations. All stimuli were disyllabic (bimorphemic). Word frequency was manipulated with the prediction that low-frequency compounds are more likely processed combinatorially, while high-frequency ones are more likely accessed in full forms. The results showed that low-frequency words elicited smaller MMNs than pseudocompounds, which supported the prediction of combinatorial processing. However, neither MMN enhancement nor reduction was found for high-frequency words. These results were interpreted within the dual-route model framework that assumes simultaneous access to words and morphemes.

Relative Contributions of Social, Contextual, and Lexical Factors in Speech Processing.

This exploratory study examined the simultaneous interactions and relative contributions of bottom-up social information (regional dialect, speaking style), top-down contextual information (semantic predictability), and the internal dynamics of the lexicon (neighborhood density, lexical frequency) to lexical access and word recognition. Cross-modal matching and intelligibility in noise tasks were conducted with a community sample of adults at a local science museum. Each task featured one condition in which keywords were presented in isolation and one condition in which they were presented within a multiword phrase. Lexical processing was slower and more accurate when keywords were presented in their phrasal context, and was both faster and more accurate for auditory stimuli produced in the local Midland dialect. In both tasks, interactions were observed among stimulus dialect, speaking style, semantic predictability, phonological neighborhood density, and lexical frequency. These interactions revealed that bottom-up social information and top-down contextual information contribute more to speech processing than the internal dynamics of the lexicon. Moreover, the relatively stronger bottom-up social effects were observed in both the isolated word and multiword phrase conditions, suggesting that social variation is central to speech processing, even in non-interactive laboratory tasks. At the same time, the specific interactions observed differed between the two experiments, reflecting task-specific demands related to processing time constraints and signal degradation.

Effects of consonant-vowel status on transposed-phoneme priming.

Nonwords created by transposing two phonemes of auditory words (e.g., /buʒãle/) are more effective primes for the corresponding base word target (/bulãʒe/) than nonword primes created by substituting two phonemes (e.g., /buvãʀe/). In one in-lab experiment and one online experiment using the short-term phonological priming paradigm, here, we examine the role of vowels and consonants in driving transposed-phoneme priming effects. Results showed that facilitatory transposed-phoneme priming occurs when the transposed phonemes are consonants (/buʒãle/-/bulãʒe/; /lubãʒe/-/bulãʒe/), but not when they are vowels (/bãluʒe/-/bulãʒe/; /buleʒã/-/bulãʒe/). These results add to existing findings showing differences in the processing of vowels and consonants during spoken and visual word recognition. We suggest that differences in the speed of processing of consonants and vowels combined with differences in the amount of information provided by consonants and vowels relative to the identity of the word being recognized provide a complete account of the present findings.

The development of lexical competition in written- and spoken-word recognition.

Efficient word recognition depends on the ability to overcome competition from overlapping words. The nature of the overlap depends on the input modality: spoken words have temporal overlap from other words that share phonemes in the same positions, whereas written words have spatial overlap from other words with letters in the same places. It is unclear how these differences in input format affect the ability to recognise a word and the types of competitors that become active while doing so. This study investigates word recognition in both modalities in children between 7 and 15. Children complete a visual-world paradigm eye-tracking task that measures competition from words with several types of overlap, using identical word lists between modalities. Results showed correlated developmental changes in the speed of target recognition in both modalities. In addition, developmental changes were seen in the efficiency of competitor suppression for some competitor types in the spoken modality. These data reveal some developmental continuity in the process of word recognition independent of modality but also some instances of independence in how competitors are activated. Stimuli, data, and analyses from this project are available at: https://osf.io/eav72.