Auditory-visual processing represented in the human superior temporal gyrus.

In natural face-to-face communication, speech perception utilizes both auditory and visual information. We described previously an acoustically responsive area on the posterior lateral surface of the superior temporal gyrus (field PLST) that is distinguishable on physiological grounds from other auditory fields located within the superior temporal plane. Considering the empirical findings in humans and non-human primates of cortical locations responsive to heard sounds and/or seen sound-sources, we reasoned that area PLST would also contain neural signals reflecting audiovisual speech interactions. To test this hypothesis, event related potentials (ERPs) were recorded from area PLST using chronically implanted multi-contact subdural surface-recording electrodes in patient-subjects undergoing diagnosis and treatment of medically intractable epilepsy, and cortical ERP maps were acquired during five contrasting auditory, visual and bimodal speech conditions. Stimulus conditions included consonant-vowel (CV) syllable sounds alone, silent seen speech or CV sounds paired with a female face articulating matched or mismatched syllables. Data were analyzed using a MANOVA framework, with the results from planned comparisons used to construct cortical significance maps. Our findings indicate that evoked responses recorded from area PLST to auditory speech stimuli are influenced significantly by the addition of visual images of the moving lower face and lips, either articulating the audible syllable or carrying out a meaningless (gurning) motion. The area of cortex exhibiting this audiovisual influence was demonstrably greater in the speech-dominant hemisphere.

Auditory cortical spatial receptive fields.

Neurons in the primary auditory cortex (AI) of anesthetized cats were studied for their sensitivity to directions of transient sounds in virtual acoustic space under a variety of conditions. An effective transient sound evokes a single spike or short burst of spikes with a precisely timed onset. The aggregate of effective directions forms a spatial receptive field. Typically, spatial receptive fields are large, often occupying a quadrant or more of acoustic space. Within the receptive field onset latency varies systematically with direction thereby providing information about source direction. This receptive field structure is highly robust, remaining relatively stable under conditions of competing sounds. Maximum likelihood analysis suggests that psychophysical spatial acuity can be achieved with a relatively small ensemble of AI neurons with broad receptive fields having response gradients of latency. Using reverse correlation and white-noise analysis receptive fields were mapped in space and time. This analysis revealed that spatial receptive fields of AI neurons need not be static but may exhibit marked temporal dynamics. This suggests a sensitivity for direction and speed of moving sound sources.

Listening through different ears alters spatial response fields in ferret primary auditory cortex.

The localization of sounds in space is based on spatial cues that arise from the acoustical properties of the head and external ears. Individual differences in localization cue values result from variability in the shape and dimensions of these structures. We have mapped spatial response fields of high-frequency neurons in ferret primary auditory cortex using virtual sound sources based either on the animal's own ears or on the ears of other subjects. For 73% of units, the response fields measured using the animals' own ears differed significantly in shape and/or position from those obtained using spatial cues from another ferret. The observed changes correlated with individual differences in the acoustics. These data are consistent with previous reports showing that humans localize less accurately when listening to virtual sounds from other individuals. Together these findings support the notion that neural mechanisms underlying auditory space perception are calibrated by experience to the properties of the individual.

Auditory space-time receptive field dynamics revealed by spherical white-noise analysis.

Numerous studies have investigated the spatial sensitivity of cat auditory cortical neurons, but possible dynamic properties of the spatial receptive fields have been largely ignored. Given the considerable amount of evidence that implicates the primary auditory field in the neural pathways responsible for the perception of sound source location, a logical extension to earlier observations of spectrotemporal receptive fields, which characterize the dynamics of frequency tuning, is a description that uses sound source direction, rather than sound frequency, to examine the evolution of spatial tuning over time. The object of this study was to describe auditory space-time receptive field dynamics using a new method based on cross-correlational techniques and white-noise analysis in spherical auditory space. This resulted in a characterization of auditory receptive fields in two spherical dimensions of space (azimuth and elevation) plus a third dimension of time. Further analysis has revealed that spatial receptive fields of neurons in auditory cortex, like those in the visual system, are not static but can exhibit marked temporal dynamics. This might result, for example, in a neuron becoming selective for the direction and speed of moving auditory sound sources. Our results show that approximately 14% of AI neurons evidence significant space-time interaction (inseparability).

Correlated cortical populations can enhance sound localization performance.

Neurons within cortical populations often evidence some degree of response correlation. Correlation has generally been regarded as detrimental to the decoding performance of a theoretical vector-averaging observer making inferences about the physical world-for example, an observer estimating the location of a sound source. However, if an alternative decoder is considered, in this case a Maximum Likelihood estimator, performance can improve when responses in the population are correlated. Improvement in sound localization performance is demonstrated analytically using Fisher information, and is also shown using Monte Carlo simulations based on recordings from single neurons in cat primary auditory cortex.

Modeling of auditory spatial receptive fields with spherical approximation functions.

A spherical approximation technique is presented that affords a mathematical characterization of a virtual space receptive field (VSRF) based on first-spike latency in the auditory cortex of cat. Parameterizing directional sensitivity in this fashion is much akin to the use of difference-of-Gaussian (DOG) functions for modeling neural responses in visual cortex. Artificial neural networks and approximation techniques typically have been applied to problems conforming to a multidimensional Cartesian input space. The problem with using classical planar Gaussians is that radial symmetry and consistency on the plane actually translate into directionally dependent distortion on spherical surfaces. An alternative set of spherical basis functions, the von Mises basis function (VMBF), is used to eliminate spherical approximation distortion. Unlike the Fourier transform or spherical harmonic expansions, the VMBFs are nonorthogonal, and hence require some form of gradient-descent search for optimal estimation of parameters in the modeling of the VSRF. The optimization equations required to solve this problem are presented. Three descriptive classes of VSRF (contralateral, frontal, and ipsilateral) approximations are investigated, together with an examination of the residual error after parameter optimization. The use of the analytic receptive field model in computational models of population coding of sound direction is discussed, together with the importance of quantifying receptive field gradients. Because spatial hearing is by its very nature three dimensional or, more precisely, two dimensional (directional) on the sphere, we find that spatial receptive field models are best developed on the sphere.

A backpropagation network model of the monaural localization information available in the bat echolocation system.

The information echolocating bats receive is a combination of the properties of the sound they emit and the sound they receive at the eardrum. Convolving the emission and the external ear transfer functions produces the full spectral information contained in the echolocation combination. Spatially dependent changes in the magnitude spectra of the emission, external ear transfer functions, and the echolocation combination of Eptesicus fuscus could provide localization information to the bat. Principal component analysis was used to reduce the dimensionality of these complex spectral data sets. The first eight principal component weights were normalized, rotated, and used as the input to a backpropagation network model which examined the relative directionality of the emission, ear, and the echolocation combination. The model was able to localize more accurately when provided with the directional information of the echolocation combination compared to either the emission or ear information alone.

The combination of echolocation emission and ear reception enhances directional spectral cues of the big brown bat, Eptesicus fuscus.

The acoustic information used by bats is produced by a combination of the properties of the sound emission and the reception at the eardrum. The potential localization cues used by bats can only be fully revealed when the magnitude spectra of the emission and the external ear are convolved to produce the echolocation combination magnitude spectra. The spatially dependent changes in the magnitude spectra of the echolocation combination of Eptesicus fuscus are described. The emission and external ear magnitude spectra act together to enhance the potential localization cues. In the echolocation combination, the spectral peaks are sharpened and there is greater contrast in intensity between peaks and notches when compared to the spectra of the ear alone. The spectral localization cues in the echolocation combination appear to be restricted to a cone of space of approximately +/-30 degrees.

Perception of voicing for syllable-initial stops at different intensities: does synchrony capture signal voiceless stop consonants?

In response to stop consonants with longer F1-cutback duration, the dominant synchronization of mid- and high-CF chinchilla auditory-nerve fibers changes from frequencies near F2 to frequencies near F1 at onset of voicing [D. G. Sinex and L. P. McDonald, J. Acoust. Soc. Am. 85, 1995-2004 (1989)]. If this change in neural synchronization is perceptually relevant for human listeners, then it may be predicted that changes in stimulus intensity and changes in the frequency difference between lower (F1) and higher (F2/F3) stimulus components should both affect perception of voicing. In a series of experiments, multiple continua of synthesized CVs varying in F1 cutback of the consonantal portion were played to listeners at levels ranging from 40 to 80 dB SPL. Across experiments, the frequency difference between F1 and F2 was manipulated by changing the onset frequency of F1 or F2. Subjects labeled more initial stops as voiceless as a function of increasing stimulus level and of decreasing frequency difference between F1 and F2. There was also an interaction between stimulus intensity and the frequency difference between F1 and F2 such that the effect of intensity was greater for smaller differences. These effects were reliable across a number of synthetic F1-cutback series, and the effect of intensity extended to a digitally edited series of hybrid CVs in which F1-cutback was varied by cross splicing naturally produced /da/ and /ta/. The effect of overall stimulus intensity was not affected by amplitude of prevocalic aspiration energy or by the presence or absence of release bursts. The results provide evidence for the perceptual significance of synchrony encoding of voicing for stop consonants.

A comparison of the von Mises and Gaussian basis functions for approximating spherical acoustic scatter.

This paper compares the approximation accuracy of two basis functions that share a common radial basis function (RBF) neural network architecture used for approximating a known function on the unit sphere. The basis function types considered are that of a new spherical basis function, the von Mises function, and the now well-known Gaussian basis function. Gradient descent learning rules were applied to optimize (learn) the solution for both approximating basis functions. A benchmark approximation problem was used to compare the performance of the two types of basis functions, in this case the mathematical expression for the scattering of an acoustic wave striking a rigid sphere.

Effects of glide slope, noise intensity, and noise duration on the extrapolation of FM glides through noise.

Listeners are quite adept at maintaining integrated perceptual events in environments that are frequently noisy. Three experiments were conducted to assess the mechanisms by which listeners maintain continuity for upward sinusoidal glides that are interrupted by a period of broadband noise. The first two experiments used stimulus complexes consisting of three parts: prenoise glide, broadband noise interval, and postnoise glide. For a given prenoise glide and noise interval, the subject's task was to adjust the onset frequency of a same-slope postnoise glide so that, together with the prenoise glide and noise, the complex sounded as "smooth and continuous as possible." The slope of the glides (1.67, 3.33, 5, and 6.67 Bark/sec) as well as the duration (50, 200, and 350 msec) and relative level of the interrupting noise (0, -6, and -12 dB S/N) were varied. For all but the shallowest glides, subjects consistently adjusted the offset portion of the glide to frequencies lower than predicted by accurate interpolation of the prenoise portion. Curiously, for the shallowest glides, subjects consistently selected postnoise glide onset-frequency values higher than predicted by accurate extrapolation of the prenoise glide. There was no effect of noise level on subjects' adjustments in the first two experiments. The third experiment used a signal detection task to measure the phenomenal experience of continuity through the noise. Frequency glides were either present or absent during the noise for stimuli like those use in the first two experiments as well as for stimuli that had no prenoise or postnoise glides. Subjects were more likely to report the presence of glides in the noise when none occurred (false positives) when noise was shorter or of greater relative level and when glides were present adjacent to the noise.

A composite model of the auditory periphery for the processing of speech based on the filter response functions of single auditory-nerve fibers.

A composite model of the auditory periphery, based upon a unique analysis technique for deriving filter response characteristics from cat auditory-nerve fibers, is presented. The model is distinctive in its ability to capture a significant broadening of auditory-nerve fiber frequency selectivity as a function of increasing sound-pressure level within a computationally tractable time-invariant structure. The output of the model shows the tonotopic distribution of synchrony activity of single fibers in response to the steady-state vowel [e] presented over a 40-dB range of sound-pressure levels and is compared with the population-response data of Young and Sachs (1979). The model, while limited by its time invariance, accurately captures most of the place-synchrony response patterns reported by the Johns Hopkins group. In both the physiology and in the model, auditory-nerve fibers spanning a broad tonotopic range synchronize to the first formant (F1), with the proportion of units phase-locked to F1 increasing appreciably at moderate to high sound-pressure levels. A smaller proportion of fibers maintain phase locking to the second and third formants across the same intensity range. At sound-pressure levels of 60 dB and above, the vast majority of fibers with characteristic frequencies greater than 3 kHz synchronize to F1 (512 Hz), rather than to frequencies in the most sensitive portion of their response range. On the basis of these response patterns it is suggested that neural synchrony is the dominant auditory-nerve representation of formant information under "normal" listening conditions in which speech signals occur across a wide range of intensities and against a background of unpredictable and frequently intense acoustic interference.

Auditory space expansion via linear filtering.

A signal-processing algorithm that modifies the interaural time delays associated with directional sources is described. Signals received at two microphones are processed by four linear filters arranged in a lattice configuration to produce two outputs, one for each ear. Since the processing is linear, the method is equally applicable to single or multiple directional sources. The filters are designed to minimize the average squared error between a user specified desired space warping function and the actual warping function that they implement. Two classes of filters are considered: filters whose frequency response is unconstrained and filters constrained to be causal with finite impulse response. In both cases the solution of the least-squares problem is given and properties of the actual space warping function are examined. Perceptual experiments and analysis of acoustic waveforms are utilized to demonstrate the effectiveness of the algorithm. Extension of this method for utilizing more than two microphones is described.

Evaluation of three strategies for fitting hearing aids binaurally.

Three strategies for evaluating optimum frequency shaping and noise reduction in binaural digital hearing aids were compared in a repeated-measures design, using a new preference-based prescriptive fitting method. These strategies consisted of using preferred frequency shaping and noise reduction values binaurally: (1) based on monaural testing; (2) based on separate evaluations of each ear; and (3) based on evaluation of a second ear while subjects wore an aid programmed with the preferred values in the first ear. Individually preferred characteristics were programmed for 17 hearing-impaired subjects, most of whom exhibited symmetrical sensorineural hearing loss. Each subject was administered intelligibility estimation and midplane localization measurements in the laboratory, as well as a questionnaire survey based on situational listening in the real world. No statistically significant differences in preferences for either frequency shaping or noise reduction were found for the three fitting strategies, suggesting that monaural testing is sufficient in symmetrical cases to provide information for binaural fitting. Related to this finding, differences across binaural conditions were minimal for both intelligibility estimation and localization results. A significant improvement in localization performance under binaural conditions over monaural listening, however, was documented by both the laboratory and the real world data. A strong overall preference for binaural over monaural amplification was also documented under real world conditions.