Attention modulates perceptual learning of non-native-accented speech.

Listeners readily adapt to variation in non-native-accented speech, learning to disambiguate between talker-specific and accent-based variation. We asked (1) which linguistic and indexical features of the spoken utterance are relevant for this learning to occur and (2) whether task-driven attention to these features affects the extent to which learning generalizes to novel utterances and voices. In two experiments, listeners heard English sentences (Experiment 1) or words (Experiment 2) produced by Spanish-accented talkers during an exposure phase. Listeners' attention was directed to lexical content (transcription), indexical cues (talker identification), or both (transcription + talker identification). In Experiment 1, listeners' test transcription of novel English sentences spoken by Spanish-accented talkers showed generalized perceptual learning to previously unheard voices and utterances for all training conditions. In Experiment 2, generalized learning occurred only in the transcription + talker identification condition, suggesting that attention to both linguistic and indexical cues optimizes listeners' ability to distinguish between individual talker- and group-based variation, especially with the reduced availability of sentence-length prosodic information. Collectively, these findings highlight the role of attentional processes in the encoding of speech input and underscore the interdependency of indexical and lexical characteristics in spoken language processing.

Neural basis of sound-symbolic pseudoword-shape correspondences.

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants (n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration.

Neural Basis Of Sound-Symbolic Pseudoword-Shape Correspondences.

Non-arbitrary mapping between the sound of a word and its meaning, termed sound symbolism, is commonly studied through crossmodal correspondences between sounds and visual shapes, e.g., auditory pseudowords, like 'mohloh' and 'kehteh', are matched to rounded and pointed visual shapes, respectively. Here, we used functional magnetic resonance imaging (fMRI) during a crossmodal matching task to investigate the hypotheses that sound symbolism (1) involves language processing; (2) depends on multisensory integration; (3) reflects embodiment of speech in hand movements. These hypotheses lead to corresponding neuroanatomical predictions of crossmodal congruency effects in (1) the language network; (2) areas mediating multisensory processing, including visual and auditory cortex; (3) regions responsible for sensorimotor control of the hand and mouth. Right-handed participants ( n = 22) encountered audiovisual stimuli comprising a simultaneously presented visual shape (rounded or pointed) and an auditory pseudoword ('mohloh' or 'kehteh') and indicated via a right-hand keypress whether the stimuli matched or not. Reaction times were faster for congruent than incongruent stimuli. Univariate analysis showed that activity was greater for the congruent compared to the incongruent condition in the left primary and association auditory cortex, and left anterior fusiform/parahippocampal gyri. Multivoxel pattern analysis revealed higher classification accuracy for the audiovisual stimuli when congruent than when incongruent, in the pars opercularis of the left inferior frontal (Broca's area), the left supramarginal, and the right mid-occipital gyri. These findings, considered in relation to the neuroanatomical predictions, support the first two hypotheses and suggest that sound symbolism involves both language processing and multisensory integration. HIGHLIGHTS: fMRI investigation of sound-symbolic correspondences between auditory pseudowords and visual shapesFaster reaction times for congruent than incongruent audiovisual stimuliGreater activation in auditory and visual cortices for congruent stimuliHigher classification accuracy for congruent stimuli in language and visual areasSound symbolism involves language processing and multisensory integration.

Neural Basis of the Sound-Symbolic Crossmodal Correspondence Between Auditory Pseudowords and Visual Shapes.

Sound symbolism refers to the association between the sounds of words and their meanings, often studied using the crossmodal correspondence between auditory pseudowords, e.g., 'takete' or 'maluma', and pointed or rounded visual shapes, respectively. In a functional magnetic resonance imaging study, participants were presented with pseudoword-shape pairs that were sound-symbolically congruent or incongruent. We found no significant congruency effects in the blood oxygenation level-dependent (BOLD) signal when participants were attending to visual shapes. During attention to auditory pseudowords, however, we observed greater BOLD activity for incongruent compared to congruent audiovisual pairs bilaterally in the intraparietal sulcus and supramarginal gyrus, and in the left middle frontal gyrus. We compared this activity to independent functional contrasts designed to test competing explanations of sound symbolism, but found no evidence for mediation via language, and only limited evidence for accounts based on multisensory integration and a general magnitude system. Instead, we suggest that the observed incongruency effects are likely to reflect phonological processing and/or multisensory attention. These findings advance our understanding of sound-to-meaning mapping in the brain.

Consistency and strength of grapheme-color associations are separable aspects of synesthetic experience.

Consistency of synesthetic associations over time is a widely used test of synesthesia. Since many studies suggest that consistency is not a completely reliable feature, we compared the consistency and strength of synesthetes' grapheme-color associations. Consistency was measured by scores on the Synesthesia Battery and by the Euclidean distance in color space for the specific graphemes tested for each participant. Strength was measured by congruency magnitudes on the Implicit Association Test. The strength of associations was substantially greater for synesthetes than non-synesthetes, suggesting that this is a novel, objective marker of synesthesia. Although, intuitively, strong associations should also be consistent, consistency and strength were uncorrelated, indicating that they are likely independent, at least for grapheme-color synesthesia. These findings have implications for our understanding of synesthesia and for estimates of its prevalence since synesthetes who experience strong, but inconsistent, associations may not be identified by tests that focus solely on consistency.

A second chance for a first impression: Sensitivity to cumulative input statistics for lexically guided perceptual learning.

Listeners use lexical knowledge to modify the mapping from acoustics to speech sounds, but the timecourse of experience that informs lexically guided perceptual learning is unknown. Some data suggest that learning is contingent on initial exposure to atypical productions, while other data suggest that learning reflects only the most recent exposure. Here we seek to reconcile these findings by assessing the type and timecourse of exposure that promote robust lexcially guided perceptual learning. In three experiments, listeners (n = 560) heard 20 critical productions interspersed among 200 trials during an exposure phase and then categorized items from an ashi-asi continuum in a test phase. In Experiment 1, critical productions consisted of ambiguous fricatives embedded in either /s/- or /ʃ/-biasing contexts. Learning was observed; the /s/-bias group showed more asi responses compared to the /ʃ/-bias group. In Experiment 2, listeners heard ambiguous and clear productions in a consistent context. Order and lexical bias were manipulated between-subjects, and perceptual learning occurred regardless of the order in which the clear and ambiguous productions were heard. In Experiment 3, listeners heard ambiguous fricatives in both /s/- and /ʃ/-biasing contexts. Order differed between two exposure groups, and no difference between groups was observed at test. Moreover, the results showed a monotonic decrease in learning across experiments, in line with decreasing exposure to stable lexically biasing contexts, and were replicated across novel stimulus sets. In contrast to previous findings showing that either initial or most recent experience are critical for lexically guided perceptual learning, the current results suggest that perceptual learning reflects cumulative experience with a talker's input over time.

Stimulus Parameters Underlying Sound-Symbolic Mapping of Auditory Pseudowords to Visual Shapes.

Sound symbolism refers to non-arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as "maluma" and "takete" with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics-to-noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound-to-shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross-modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.

Communicative Context Affects Use of Referential Prosody.

The current study assessed the extent to which the use of referential prosody varies with communicative demand. Speaker-listener dyads completed a referential communication task during which speakers attempted to indicate one of two color swatches (one bright, one dark) to listeners. Speakers' bright sentences were reliably higher pitched than dark sentences for ambiguous (e.g., bright red versus dark red) but not unambiguous (e.g., bright red versus dark purple) trials, suggesting that speakers produced meaningful acoustic cues to brightness when the accompanying linguistic content was underspecified (e.g., "Can you get the red one?"). Listening partners reliably chose the correct corresponding swatch for ambiguous trials when lexical information was insufficient to identify the target, suggesting that listeners recruited prosody to resolve lexical ambiguity. Prosody can thus be conceptualized as a type of vocal gesture that can be recruited to resolve referential ambiguity when there is communicative demand to do so.

Specificity and generalization in perceptual adaptation to accented speech.

The present study investigated the degree to which perceptual adaptation to foreign-accented speech is specific to the regularities in pronunciation associated with a particular accent. Across experiments, the conditions under which generalization of learning did or did not occur were evaluated. In Experiment 1, listeners trained on word-length utterances in Korean-accented English and tested with words produced by the same or different set of Korean-accented speakers. Listeners performed better than untrained controls when tested with novel words from the same or different speakers. In Experiment 2, listeners were trained with Spanish-, Korean-, or mixed-accented speech and transcribed novel words produced by unfamiliar Korean- or Spanish-accented speakers at test. The findings revealed relative specificity of learning. Listeners trained and tested on the same variety of accented speech showed better transcription at test than those trained with a different accent or untrained controls. Performance after mixed-accent training was intermediate. Patterns of errors and analysis of acoustic properties for accented vowels suggested perceptual improvement for regularities arising from each accent, with learning dependent on the relative similarity of linguistic form within and across accents.

Vocal alignment to native and non-native speakers of English.

Research on vocal alignment, the tendency for language users to match another individual's speech productions, suggests that multiple factors contribute to this behavior. Social and motivational goals, aspects of cognitive architecture, and linguistic flexibility may all affect the extent to which vocal alignment occurs, suggesting complex underlying mechanisms. The present study capitalized on the social and linguistic characteristics of Spanish-accented English to examine the relationship among these contributors to vocal alignment. American English-speaking adults participated in a shadowing task. Degree of vocal alignment was assessed by both acoustic measures and independent raters' judgments. Participants aligned to both native English and Spanish-accented productions, despite differences in attitudes to and intelligibility of the different accents. Individual differences in shadowers' vowel dispersion were also related to extent of vocal alignment, with greater dispersion associated with greater alignment. Acoustic measures were related to perceptual assessments of alignment and differed by accent type, suggesting that patterns of alignment may differ across accents. Overall, the current study demonstrates vocal alignment between talkers of differing language backgrounds and highlights the importance of acoustic and linguistic components of alignment behavior.

Neural basis of the crossmodal correspondence between auditory pitch and visuospatial elevation.

Crossmodal correspondences refer to associations between otherwise unrelated stimulus features in different sensory modalities. For example, high and low auditory pitches are associated with high and low visuospatial elevation, respectively. The neural mechanisms underlying crossmodal correspondences are currently unknown. Here, we used functional magnetic resonance imaging (fMRI) to investigate the neural basis of the pitch-elevation correspondence. Pitch-elevation congruency effects were observed bilaterally in the inferior frontal and insular cortex, the right frontal eye field and right inferior parietal cortex. Independent functional localizers failed to provide strong evidence for any of three proposed mechanisms for crossmodal correspondences: semantic mediation, magnitude estimation, and multisensory integration. Instead, pitch-elevation congruency effects overlapped with areas selective for visually presented non-word strings relative to sentences, and with regions sensitive to audiovisual asynchrony. Taken together with the prior literature, the observed congruency effects are most consistent with mediation by multisensory attention.

Eye movements reveal persistent sensitivity to sound symbolism during word learning.

Although the relationship between sound and meaning in language is assumed to be largely arbitrary, reliable correspondences between sound and meaning in natural language appear to facilitate word learning. Using a set of independently normed pseudoword and shape stimuli, we examined the real-time effects of sound-to-shape correspondences at initial presentation and throughout an extended learning process resulting in high accuracy. In addition to accuracy and response time (RT) measures, we monitored participants' eye movements to investigate the extent to which visual orienting to objects is influenced by the sound symbolic characteristics of novel labels at initial exposure and throughout learning. Over the course of word learning, congruency of sound and shape properties affected both accuracy and RT with higher accuracy and faster responses for congruent than incongruent items. Eye tracking data reveal that congruent targets were fixated faster than incongruent targets throughout learning and that nontargets consistent with the sound symbolic properties of the word remained attractive distracters, even after overt behavioral differences in accuracy disappeared. This demonstrates the sustained influence of sound symbolism and the importance of sensitive, continuous measures of assessing sound symbolic effects in word learning and lexical processing. Arbitrariness resulted in better final individuation performance only when the arbitrary items were more phonologically distinct than the sound symbolic stimuli. These findings suggest that the advantages of sound symbolism may persist beyond early word learning and serve to significantly influence online lexical processing. (PsycINFO Database Record

Developmental change in children's sensitivity to sound symbolism.

The current study examined developmental change in children's sensitivity to sound symbolism. Three-, five-, and seven-year-old children heard sound symbolic novel words and foreign words meaning round and pointy and chose which of two pictures (one round and one pointy) best corresponded to each word they heard. Task performance varied as a function of both word type and age group such that accuracy was greater for novel words than for foreign words, and task performance increased with age for both word types. For novel words, children in all age groups reliably chose the correct corresponding picture. For foreign words, 3-year-olds showed chance performance, whereas 5- and 7-year-olds showed reliably above-chance performance. Results suggest increased sensitivity to sound symbolic cues with development and imply that although sensitivity to sound symbolism may be available early and facilitate children's word-referent mappings, sensitivity to subtler sound symbolic cues requires greater language experience.

The Specificity of Sound Symbolic Correspondences in Spoken Language.

Although language has long been regarded as a primarily arbitrary system, sound symbolism, or non-arbitrary correspondences between the sound of a word and its meaning, also exists in natural language. Previous research suggests that listeners are sensitive to sound symbolism. However, little is known about the specificity of these mappings. This study investigated whether sound symbolic properties correspond to specific meanings, or whether these properties generalize across semantic dimensions. In three experiments, native English-speaking adults heard sound symbolic foreign words for dimensional adjective pairs (big/small, round/pointy, fast/slow, moving/still) and for each foreign word, selected a translation among English antonyms that either matched or mismatched with the correct meaning dimension. Listeners agreed more reliably on the English translation for matched relative to mismatched dimensions, though reliable cross-dimensional mappings did occur. These findings suggest that although sound symbolic properties generalize to meanings that may share overlapping semantic features, sound symbolic mappings offer semantic specificity.

The role of training structure in perceptual learning of accented speech.

Foreign-accented speech contains multiple sources of variation that listeners learn to accommodate. Extending previous findings showing that exposure to high-variation training facilitates perceptual learning of accented speech, the current study examines to what extent the structure of training materials affects learning. During training, native adult speakers of American English transcribed sentences spoken in English by native Spanish-speaking adults. In Experiment 1, training stimuli were blocked by speaker, sentence, or randomized with respect to speaker and sentence (Variable training). At test, listeners transcribed novel English sentences produced by unfamiliar Spanish-accented speakers. Listeners' transcription accuracy was highest in the Variable condition, suggesting that varying both speaker identity and sentence across training trials enabled listeners to generalize their learning to novel speakers and linguistic content. Experiment 2 assessed the extent to which ordering of training tokens by a single factor, speaker intelligibility, would facilitate speaker-independent accent learning, finding that listeners' test performance did not reliably differ from that in the no-training control condition. Overall, these results suggest that the structure of training exposure, specifically trial-to-trial variation on both speaker's voice and linguistic content, facilitates learning of the systematic properties of accented speech. The current findings suggest a crucial role of training structure in optimizing perceptual learning. Beyond characterizing the types of variation listeners encode in their representations of spoken utterances, theories of spoken language processing should incorporate the role of training structure in learning lawful variation in speech. (PsycINFO Database Record

Cross-linguistic sound symbolism and crossmodal correspondence: Evidence from fMRI and DTI.

Non-arbitrary correspondences between spoken words and categories of meanings exist in natural language, with mounting evidence that listeners are sensitive to this sound symbolic information. Native English speakers were asked to choose the meaning of spoken foreign words from one of four corresponding antonym pairs selected from a previously developed multi-language stimulus set containing both sound symbolic and non-symbolic stimuli. In behavioral (n=9) and fMRI (n=15) experiments, participants showed reliable sensitivity to the sound symbolic properties of the stimulus set, selecting the consistent meaning for the sound symbolic words at above chances rates. There was increased activation for sound symbolic relative to non-symbolic words in left superior parietal cortex, and a cluster in left superior longitudinal fasciculus showed a positive correlation between fractional anisotropy (FA) and an individual's sensitivity to sound symbolism. These findings support the idea that crossmodal correspondences underlie sound symbolism in spoken language.

Tone of voice guides word learning in informative referential contexts.

Listeners infer which object in a visual scene a speaker refers to from the systematic variation of the speaker's tone of voice (ToV). We examined whether ToV also guides word learning. During exposure, participants heard novel adjectives (e.g., "daxen") spoken with a ToV representing hot, cold, strong, weak, big, or small while viewing picture pairs representing the meaning of the adjective and its antonym (e.g., elephant-ant for big-small). Eye fixations were recorded to monitor referent detection and learning. During test, participants heard the adjectives spoken with a neutral ToV, while selecting referents from familiar and unfamiliar picture pairs. Participants were able to learn the adjectives' meanings, and, even in the absence of informative ToV, generalize them to new referents. A second experiment addressed whether ToV provides sufficient information to infer the adjectival meaning or needs to operate within a referential context providing information about the relevant semantic dimension. Participants who saw printed versions of the novel words during exposure performed at chance during test. ToV, in conjunction with the referential context, thus serves as a cue to word meaning. ToV establishes relations between labels and referents for listeners to exploit in word learning.

Say it like you mean it: mothers' use of prosody to convey word meaning.

Prosody plays a variety of roles in infants' communicative development, aiding in attention modulation, speech segmentation, and syntax acquisition. This study investigates the extent to which parents also spontaneously modulate prosodic aspects of infant directed speech in ways that distinguish semantic aspects of language. Fourteen mothers of two-year-old children read a picture book to their children in which they labeled pictures using dimensional adjectives (e.g., big, small, hot, cold). Recordings of the mothers' input to their children were analyzed acoustically and antonyms within each dimension were compared. Mothers modulated aspects of their prosody including amplitude and duration of target words and sentences to distinguish dimensional adjectives. Mothers appear to recruit prosody in the service of word learning.

Charm or Harm: Effect of Passage Content on Listener Attitudes toward American English Accents.

This study examined how passage content influences attitudes towards American English Accents. Participants listened to passages differing in topic content spoken in an American Southern English or Standard American English accent. Although Southern-accented speakers were rated higher in sociality, but lower in status, than standard-accented speakers, sociality ratings varied as a function of passage topic only for Standard-accented speakers. Linguistic content appeared most likely to influence listeners' attitudes when preexisting assumptions based on regional accent were absent.

The developing role of prosody in novel word interpretation.

This study examined whether children use prosodic correlates to word meaning when interpreting novel words. For example, do children infer that a word spoken in a deep, slow, loud voice refers to something larger than a word spoken in a high, fast, quiet voice? Participants were 4- and 5-year-olds who viewed picture pairs that varied along a single dimension (e.g., big vs. small flower) and heard a recorded voice asking them, for example, "Can you get the blicket one?" spoken with either meaningful or neutral prosody. The 4-year-olds failed to map prosodic cues to their corresponding meaning, whereas the 5-year-olds succeeded (Experiment 1). However, 4-year-olds successfully mapped prosodic cues to word meaning following a training phase that reinforced children's attention to prosodic information (Experiment 2). These studies constitute the first empirical demonstration that young children are able to use prosody-to-meaning correlates as a cue to novel word interpretation.