Multi-modal cross-linguistic perception of Mandarin tones in clear speech.

Clearly enunciated speech (relative to conversational, plain speech) involves articulatory and acoustic modifications that enhance auditory-visual (AV) segmental intelligibility. However, little research has explored clear-speech effects on the perception of suprasegmental properties such as lexical tone, particularly involving visual (facial) perception. Since tone production does not primarily rely on vocal tract configurations, tones may be less visually distinctive. Questions thus arise as to whether clear speech can enhance visual tone intelligibility, and if so, whether any intelligibility gain can be attributable to tone-specific category-enhancing (code-based) clear-speech cues or tone-general saliency-enhancing (signal-based) cues. The present study addresses these questions by examining the identification of clear and plain Mandarin tones with visual-only, auditory-only, and AV input modalities by native (Mandarin) and nonnative (English) perceivers. Results show that code-based visual and acoustic clear tone modifications, although limited, affect both native and nonnative intelligibility, with category-enhancing cues increasing intelligibility and category-blurring cues decreasing intelligibility. In contrast, signal-based cues, which are extensively available, do not benefit native intelligibility, although they contribute to nonnative intelligibility gain. These findings demonstrate that linguistically relevant visual tonal cues are existent. In clear speech, such tone category-enhancing cues are incorporated with saliency-enhancing cues across AV modalities for intelligibility improvements.

Plain-to-clear speech video conversion for enhanced intelligibility.

Clearly articulated speech, relative to plain-style speech, has been shown to improve intelligibility. We examine if visible speech cues in video only can be systematically modified to enhance clear-speech visual features and improve intelligibility. We extract clear-speech visual features of English words varying in vowels produced by multiple male and female talkers. Via a frame-by-frame image-warping based video generation method with a controllable parameter (displacement factor), we apply the extracted clear-speech visual features to videos of plain speech to synthesize clear speech videos. We evaluate the generated videos using a robust, state of the art AI Lip Reader as well as human intelligibility testing. The contributions of this study are: (1) we successfully extract relevant visual cues for video modifications across speech styles, and have achieved enhanced intelligibility for AI; (2) this work suggests that universal talker-independent clear-speech features may be utilized to modify any talker's visual speech style; (3) we introduce "displacement factor" as a way of systematically scaling the magnitude of displacement modifications between speech styles; and (4) the high definition generated videos make them ideal candidates for human-centric intelligibility and perceptual training studies.

Tonal properties in a non-tonal language: The case of Indonesian.

This study investigates the tonal properties of Indonesian, specifically examining intonation contours in Indonesian at the sentence level and focusing on how the tonal system is used to indicate different pragmatic meanings in Indonesian. Four distinct intonation contours with four distinct meanings were contrasted: strong agreement to the truth of the relevant word (emphasizing) (H), interrogative meaning (LH), doubting the fact of the word in question (L), and a neutral conceptual meaning of the relevant word (HL). The stimuli for the study were four Indonesian words (two verbs makan 'eat' and tahu 'know' and two adjectives cantik 'beautiful' and marah 'angry'). Stimuli were recorded to capture the four-way contrast in pitch contour. A comprehension task to identify the distinct meanings was conducted. Forty-nine participants were asked to listen to the four words with the four different pitch contours and select their respective meanings. The data show that the participants were able to apply the Mandarin four-way contrast pitch contour to Indonesian to accurately indicate four different (pragmatic) meanings. The most difficult contour for the listeners/participants was in distinguishing the interrogative intonation contour (LH) from the low-dipping intonation contour (L) signaling doubting the conceptual truth of the lexical items used. The overall study suggests that a tonal system based on Mandarin tonal contrasts can be applied to Indonesian intonational expressions.

Production and Perception Evidence of a Merger: [l] and [n] in Fuzhou Min.

The current study investigated the merger-in-progress between word-initial nasal and lateral consonants in Fuzhou Min, examining the linguistic and social factors that modulate the merger. First, the acoustic cues to the l-n distinction were examined in Fuzhou Min. Acoustic analyses suggested a collapse of phonemic contrast between prescriptive L and N (phonemes in the unmerged system), with none of the six acoustic cues showing any difference across L and N. Linear discriminant analysis did identify acoustically distinct [l] and [n] tokens, although the mapping onto the phonetic space of prescriptive L and N substantially overlapped. Speakers of all ages and both genders tended to produce [l], and low vowels correlated with more [n]-like classification. In perception, AX discrimination data showed Fuzhou Min listeners confused both prescriptive L and N and acoustic [l] and [n]. Greater sensitivity to the acoustic differences occurred in the context of low vowels and a nasal coda, supported by the acoustics of the stimuli, and younger listeners were more sensitive to the difference between [l] and [n] than older listeners. In two-alternative forced choice (2AFC) identification, Fuzhou Min listeners also identified the merged form as L more frequently than N, with more L responses elicited in the context of low vowels and in the absence of nasal codas. Overall, although Fuzhou Min speakers produced some acoustically distinct [l] and [n] tokens in the context of a sound merger, these productions did not map onto prescriptive L and N. In addition, younger listeners were more sensitive to the acoustic distinction than older listeners, suggesting an emerging acoustic contrast possibly arising due to contact with Mandarin.

The contrast between clear and plain speaking style for Mandarin tones.

We examine the acoustic characteristics of clear and plain conversational productions of Mandarin tones. Twenty-one native Mandarin speakers were asked to produce a selection of Mandarin words in both plain and clear speaking styles. Several tokens were gathered for each of the four tones giving a total of 2045 productions. Six critical tonal cues were computed for each production: fundamental frequency (F0) mean, slope, and second derivative, duration, mean intensity, and a binary variable coding whether the production involved creaky voice. A linear mixed-effects regression model was used to explore how these cues changed with respect to the clear versus plain distinction for each tone, with speaking style as the fixed effect and speaker being a random effect. The strongest effects detected were that duration and mean intensity increased in clear speech across speakers and tones. Tones 2 and 3 increased in mean F0 and Tone 4 increased its slope. An additional finding was that, for contour tones, speakers accomplished the increase in duration by stretching out the tone contours in time while largely not changing the F0 range. These results are discussed in terms of signal-based (affecting all tones) and code-based (enhancing contrast between tones) change.

Investigating the Lexical Representation of Mandarin Tone 3 Phonological Alternations.

Phonological alternations pose challenges for models of spoken word recognition in how surface information is mapped onto stored representations in the lexicon. In the current study, an auditory-auditory priming lexical decision experiment was conducted to investigate the alternating representations of Mandarin Tone 3 in both half-third and third tone sandhi contexts. In Mandarin, a full Tone 3 (213) is reduced to an abridged tone (21) when followed by Tone 1, Tone 2, or Tone 4 (half-third tone sandhi), and Tone 3 is replaced by Tone 2 when followed by another Tone 3 (third tone sandhi). In the half-third sandhi block, disyllabic targets with a half-third (21) or full-third (213) tone FIRST syllable and a Tone 2 (35) or Tone 4 (51) second syllable were preceded by either a half-third prime, a full-third prime, or a control prime. In the third tone sandhi block, third-tone sandhi disyllabic targets with a half-third or full-third SECOND syllable were preceded by either a half-third prime, a full-third prime, or a control prime. Results showed that both half-third and full-third primes elicited significantly faster reaction times relative to the control Tone 1 condition. The size of the facilitation was not influenced by prime condition, target frequency, targets' first syllable tone or targets' second syllable tone. These data suggest that Mandarin T3 may be a more abstract tone and stored as the first syllable for both types of sandhi words.

ADFAC: Automatic detection of facial articulatory features.

Using computer-vision and image processing techniques, we aim to identify specific visual cues as induced by facial movements made during monosyllabic speech production. The method is named ADFAC: Automatic Detection of Facial Articulatory Cues. Four facial points of interest were detected automatically to represent head, eyebrow and lip movements: nose tip (proxy for head movement), medial point of left eyebrow, and midpoints of the upper and lower lips. The detected points were then automatically tracked in the subsequent video frames. Critical features such as the distance, velocity, and acceleration describing local facial movements with respect to the resting face of each speaker were extracted from the positional profiles of each tracked point. In this work, a variant of random forest is proposed to determine which facial features are significant in classifying speech sound categories. The method takes in both video and audio as input and extracts features from any video with a plain or simple background. The method is implemented in MATLAB and scripts are made available on GitHub for easy access.•Using innovative computer-vision and image processing techniques to automatically detect and track keypoints on the face during speech production in videos, thus allowing more natural articulation than previous sensor-based approaches.•Measuring multi-dimensional and dynamic facial movements by extracting time-related, distance-related and kinematics-related features in speech production.•Adopting the novel random forest classification approach to determine and rank the significance of facial features toward accurate speech sound categorization.

Multi-modal cross-linguistic perception of fricatives in clear speech.

Research shows that acoustic modifications in clearly enunciated fricative consonants (relative to the plain, conversational productions) facilitate auditory fricative perception, particularly for auditorily salient sibilant fricatives and for native perception. However, clear-speech effects on visual fricative perception have received less attention. A comparison of auditory and visual (facial) clear-fricative perception is particularly interesting since sibilant fricatives in English are more auditorily salient while non-sibilants are more visually salient. This study thus examines clear-speech effects on multi-modal perception of English sibilant and non-sibilant fricatives. Native English perceivers and non-native (Mandarin, Korean) perceivers with different fricative inventories in their native languages (L1s) identified clear and conversational fricative-vowel syllables in audio-only, visual-only, and audio-visual (AV) modes. The results reveal an overall positive clear-speech effect when visual information is involved. Considering the factor of AV saliency, clear speech benefits sibilants more in the auditory domain and non-sibilants more in the visual domain. With respect to language background, non-native (Mandarin and Korean) perceivers benefit from visual as well as auditory information, even for fricatives non-existent in their respective L1s, but the patterns of clear-speech gains are affected by the relative AV weighting and "nativeness" of the fricatives. These findings are discussed in terms of how saliency-enhancing and category-distinctive cues of speech sounds are adopted in AV perception to improve intelligibility.

Characterizing the distinctive acoustic cues of Mandarin tones.

This study aims to characterize distinctive acoustic features of Mandarin tones based on a corpus of 1025 monosyllabic words produced by 21 native Mandarin speakers. For each tone, 22 acoustic cues were extracted. Besides standard F0, duration, and intensity measures, further cues were determined by fitting two mathematical functions to the pitch contours. The first function is a parabola, which gives three parameters: a mean F0, an F0 slope, and an F0 second derivative. The second is a broken-line function, which models the contour as a continuous curve consisting of two lines with a single breakpoint. Cohen's d, sparse Principal Component Analysis, and other statistical measures are used to identify which of the cues, and which combinations of the cues, are important for distinguishing each tone from each other among all the speakers. Although the specific cues that best characterize the tone contours depend on the particular tone and the statistical measure used, this paper shows that the three cues obtained by fitting a parabola to the tone contour are broadly effective. This research suggests using these three cues as a canonical choice for defining tone characteristics.

The Role of Surface and Underlying Forms When Processing Tonal Alternations in Mandarin Chinese: A Mismatch Negativity Study.

Phonological alternation (sound change depending on the phonological environment) poses challenges to spoken word recognition models. Mandarin Chinese T3 sandhi is such a phenomenon in which a tone 3 (T3) changes into a tone 2 (T2) when followed by another T3. In a mismatch negativity (MMN) study examining Mandarin Chinese T3 sandhi, participants passively listened to either a T2 word [tʂu2 je4] /tʂu2 je4/, a T3 word [tʂu3 je4] /tʂu3 je4/, a sandhi word [tʂu2 jen3] /tʂu3 jen3/, or a mix of T3 and sandhi word standards. The deviant in each condition was a T2 word [tʂu2]. Results showed an MMN only in the T2 and T3 conditions but not in the Sandhi or Mix conditions. All conditions also yielded omission MMNs. This pattern cannot be explained based on the surface forms of standards and deviants; rather these data suggest an underspecified or underlying T3 stored linguistic representation used in spoken word processing.

Training Children to Perceive Non-native Lexical Tones: Tone Language Background, Bilingualism, and Auditory-Visual Information.

This study investigates the role of language background and bilingual status in the perception of foreign lexical tones. Eight groups of participants, consisting of children of 6 and 8 years from one of four language background (tone or non-tone) × bilingual status (monolingual or bilingual)-Thai monolingual, English monolingual, English-Thai bilingual, and English-Arabic bilingual were trained to perceive the four Mandarin lexical tones. Half the children in each of these eight groups were given auditory-only (AO) training and half auditory-visual (AV) training. In each group Mandarin tone identification was tested before and after (pre- and post-) training with both auditory-only test (ao-test) and auditory-visual test (av test). The effect of training on Mandarin tone identification was minimal for 6-year-olds. On the other hand, 8-year-olds, particularly those with tone language experience showed greater pre- to post-training improvement, and this was best indexed by ao-test trials. Bilingual vs. monolingual background did not facilitate overall improvement due to training, but it did modulate the efficacy of the Training mode: for bilinguals both AO and AV training, and especially AO, resulted in performance gain; but for monolinguals training was most effective with AV stimuli. Again this effect was best indexed by ao-test trials. These results suggest that tone language experience, be it monolingual or bilingual, is a strong predictor of learning unfamiliar tones; that monolinguals learn best from AV training trials and bilinguals from AO training trials; and that there is no metalinguistic advantage due to bilingualism in learning to perceive lexical tones.

Cross-modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers.

Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers' performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning.

Just noticeable differences for pitch direction, height, and slope for Mandarin and English listeners.

Previous studies on tones suggest that Mandarin listeners are more sensitive to pitch direction and slope while English listeners primarily attend to pitch height. In this study, just noticeable differences were established for pitch discrimination using a three-interval, forced-choice procedure with a two-down, one-up staircase design. A high rising and a high falling Mandarin tone were manipulated in terms of pitch direction, height, and slope. Results indicate that, overall, Mandarin listeners are more sensitive to pitch slope and English listeners to pitch height. However, these effects are modulated by both the direction (falling/rising) and slope of the pitch contours.

What's in a Word: Observing the Contribution of Underlying and Surface Representations.

underlying representations play a crucial role in capturing predictable relations among different phonetic categories in phonological theory. Tone sandhi is a tonal alternation phenomenon in which a tone changes to a different tone in certain phonological environments. This study investigates whether Taiwanese listeners are more sensitive to the surface form of the tones or the underlying tonal representations of tone sandhi words. An auditory lexical decision experiment was conducted to examine priming effects between monosyllabic primes and disyllabic target words (tone sandhi T51 → T55 and sandhi T24 → T33). Each target was preceded by either a surface-tone prime (e.g., ping55-ping55tsun24; pue33-pue33jong51), an underlying-tone prime (e.g., ping51-ping55tsun24; pue24-pue33jong51), or an unrelated control (e.g., ping21-ping55tsun24; pue21-pue33jong51). Results showed significant differences in the natue of the priming effects across the two sandhi types, with productivity of the tone sandhi rule influencing how listeners' process and represent tone sandhi words.

Acoustic characteristics of clearly spoken English tense and lax vowels.

Clearly produced vowels exhibit longer duration and more extreme spectral properties than plain, conversational vowels. These features also characterize tense relative to lax vowels. This study explored the interaction of clear-speech and tensity effects by comparing clear and plain productions of three English tense-lax vowel pairs (/i-ɪ/, /ɑ-ʌ/, /u-ʊ/ in /kVd/ words). Both temporal and spectral acoustic features were examined, including vowel duration, vowel-to-word duration ratio, formant frequency, and dynamic spectral characteristics. Results revealed that the tense-lax vowel difference was generally enhanced in clear relative to plain speech, but clear-speech modifications for tense and lax vowels showed a trade-off in the use of temporal and spectral cues. While plain-to-clear vowel lengthening was greater for tense than lax vowels, clear-speech modifications in spectral change were larger for lax than tense vowels. Moreover, peripheral tense vowels showed more consistent clear-speech modifications in the temporal than spectral domain. Presumably, articulatory constraints limit the spectral variation of these extreme vowels, so clear-speech modifications resort to temporal features and reserve the primary spectral features for tensity contrasts. These findings suggest that clear-speech and tensity interactions involve compensatory modifications in different acoustic domains.

The Contribution of Segmental and Tonal Information in Mandarin Spoken Word Processing.

Two priming experiments examined the separate contribution of lexical tone and segmental information in the processing of spoken words in Mandarin Chinese. Experiment I contrasted four types of prime-target pairs: tone-and-segment overlap (ru4-ru4), segment-only overlap (ru3-ru4), tone-only overlap (sha4-ru4) and unrelated (qin 1 -ru4) in an auditory lexical decision task with 48 native Mandarin listeners. Experiment 2 further investigated the minimal segmental overlap needed to trigger priming when tonal information is present. Four prime-target conditions were contrasted: tone-and-segment overlap (ru4-ru4), only onset segment overlap (re4-ru4), only rime overlap (pu4-ru4) and unrelated (qin 1 -ru4) in an auditory lexical decision task with 68 native Mandarin listeners. The results showed significant priming effects when both tonal and segmental information overlapped or, although to a lesser extent, when only segmental information overlapped, with no priming found when only tones matched. Moreover, any partial segmental overlap, even with matching tonal cues, resulted in significant inhibition. These data clearly indicate that lexical tones are processed differently from segments, with syllabic structure playing a critical role. These findings are discussed in terms of the overall architecture of the processing system that emerges in Mandarin lexical access.

Intelligibility of foreign-accented speech for older adults with and without hearing loss.

BACKGROUND: Numerous studies have demonstrated that the negative effect of noise and other distortions on speech understanding is greater for older adults than for younger adults. Anecdotal evidence suggests that older adults may also be disproportionately negatively affected by foreign accent. While two previous studies found no interaction between foreign accent and listener age, these studies reported no audiometric data and assessed speech understanding in quiet only. PURPOSE: To examine the effects of foreign accent, listening condition, and listener age and hearing status on word identification. RESEARCH DESIGN: A cross-sectional descriptive study. STUDY SAMPLE: Experiments 1 and 2 tested young adults with normal hearing (n = 20 and n = 5, respectively), older adults with essentially normal hearing (n = 20 and n = 10, respectively), and older adults with sloping sensorineural hearing loss (n = 20 and n = 10, respectively). DATA COLLECTION AND ANALYSIS: The intelligibility of English words produced by a native speaker of English and by a native speaker of Spanish was assessed. In Experiment 1, word intelligibility was measured in quiet, in noise (+3 dB signal-to-babble ratio, or SBR), and in a telephone filter condition. In Experiment 2, intelligibility was measured in three additional noise conditions (+6, +9, and +12 dB SBR). RESULTS: English words produced by the native speaker of English were significantly more intelligible than those produced by the native speaker of Spanish. While the negative effect of noise was significantly greater for older listeners than for younger listeners, the effect of foreign accent was independent of listener age, listener hearing status, and listening condition. CONCLUSION: The results suggest that, unlike with other forms of distortion, older adults are not disproportionately affected by foreign accent. This suggests, in turn, that talker-related distortions of the speech signal have a qualitatively different impact on speech perception than distortions that are applied to the signal after it has been produced. The nature of these different types of distortion may be a fruitful area for future investigations of speech understanding in older adults.

Word length and lexical competition: longer is the same as shorter.

Neighborhood density refers to the number of words that sound similar to a given word. Previous studies have found that neighborhood density influences the recognition of spoken words (Luce & Pisoni, 1998); however, this work has focused almost exclusively on monosyllabic words in English. To investigate the effects of neighborhood density on longer words, bisyllabic words varying in neighborhood density were presented auditorily to participants in a perceptual identification task and a lexical decision task. In the perceptual identification task, words with sparse neighborhoods were more accurately identified than words with dense neighborhoods. In the lexical decision task, words with sparse neighborhoods were responded to more quickly and more accurately than words with dense neighborhoods. These results are similar to those found in studies examining the influence of neighborhood density on the recognition of monosyllabic words in English. In order to better understand lexical processing, models of spoken word recognition must account for the processing of words of all types.

A comparison of semantic and syntactic event related potentials generated by children and adults.

The present study employs event related potentials (ERPs) to verify the utility of using electrophysiological measures to study developmental questions within the field of language comprehension. Established ERP components (N400 and P600) that reflect semantic and syntactic processing were examined. Fifteen adults and 14 children (ages 8-13) processed spoken stimuli containing either semantic or syntactic anomalies. Adult participants showed a significant N400 in response to semantic anomalies and P600 components in response to syntactic anomalies. Children also show evidence of both ERP components. The children's N400 component differed from the adults' in scalp location, latency, and component amplitude. The children's P600 was remarkably similar to the P600 shown by adults in scalp location, component amplitude, and component latency. Theoretical implication for theories of language comprehension in adults and children will be discussed.

fMRI evidence for cortical modification during learning of Mandarin lexical tone.

Functional magnetic resonance imaging was employed before and after six native English speakers completed lexical tone training as part of a program to learn Mandarin as a second language. Language-related areas including Broca's area, Wernicke's area, auditory cortex, and supplementary motor regions were active in all subjects before and after training and did not vary in average location. Across all subjects, improvements in performance were associated with an increase in the spatial extent of activation in left superior temporal gyrus (Brodmann's area 22, putative Wernicke's area), the emergence of activity in adjacent Brodmann's area 42, and the emergence of activity in right inferior frontal gyrus (Brodmann's area 44), a homologue of putative Broca's area. These findings demonstrate a form of enrichment plasticity in which the early cortical effects of learning a tone-based second language involve both expansion of preexisting language-related areas and recruitment of additional cortical regions specialized for functions similar to the new language functions.