A biologically oriented algorithm for spatial sound segregation.

Listening in an acoustically cluttered scene remains a difficult task for both machines and hearing-impaired listeners. Normal-hearing listeners accomplish this task with relative ease by segregating the scene into its constituent sound sources, then selecting and attending to a target source. An assistive listening device that mimics the biological mechanisms underlying this behavior may provide an effective solution for those with difficulty listening in acoustically cluttered environments (e.g., a cocktail party). Here, we present a binaural sound segregation algorithm based on a hierarchical network model of the auditory system. In the algorithm, binaural sound inputs first drive populations of neurons tuned to specific spatial locations and frequencies. The spiking response of neurons in the output layer are then reconstructed into audible waveforms via a novel reconstruction method. We evaluate the performance of the algorithm with a speech-on-speech intelligibility task in normal-hearing listeners. This two-microphone-input algorithm is shown to provide listeners with perceptual benefit similar to that of a 16-microphone acoustic beamformer. These results demonstrate the promise of this biologically inspired algorithm for enhancing selective listening in challenging multi-talker scenes.

Effects of Hearing Loss on Interaural Time Difference Sensitivity at Low and High Frequencies.

While many studies have reported a loss of sensitivity to interaural time differences (ITDs) carried in the fine structure of low-frequency signals for listeners with hearing loss, relatively few data are available on the perception of ITDs carried in the envelope of high-frequency signals in this population. The relevant studies found stronger effects of hearing loss at high frequencies than at low frequencies in most cases, but small subject numbers and several confounding effects prevented strong conclusions from being drawn. In the present study, we revisited this question while addressing some of the issues identified in previous studies. Participants were ten young adults with normal hearing (NH) and twenty adults with sensorineural hearing impairment (HI) spanning a range of ages. ITD discrimination thresholds were measured for octave-band-wide "rustle" stimuli centered at 500 Hz or 4000 Hz, which were presented at 20 or 40 dB sensation level. Broadband rustle stimuli and 500-Hz pure-tone stimuli were also tested. Thresholds were poorer on average for the HI group than the NH group. The ITD deficit, relative to the NH group, was similar at low and high frequencies for most HI participants. For a small number of participants, however, the deficit was strongly frequency-dependent. These results provide new insights into the binaural perception of complex sounds and may inform binaural models that incorporate effects of hearing loss.

Intracranial lateralization bias observed in the presence of symmetrical hearing thresholds.

It is generally assumed that listeners with normal audiograms have relatively symmetric hearing, and more specifically that diotic stimuli (having zero interaural differences) are heard as centered in the head. While measuring intracranial lateralization with a visual pointing task for tones and 50-Hz-wide narrowband noises from 300 to 700 Hz, examples of systematic and large (>50% from midline to the ear) lateralization biases were found. In a group of ten listeners, five showed consistent lateralization bias to the right or left side at all or a subset of frequencies. Asymmetries in hearing, not apparent in audiometric thresholds, may explain these lateralization biases.

Binaural Recordings in Natural Acoustic Environments: Estimates of Speech-Likeness and Interaural Parameters.

Binaural acoustic recordings were made in multiple natural environments, which were chosen to be similar to those reported to be difficult for listeners with impaired hearing. These environments include natural conversations that take place in the presence of other sound sources as found in restaurants, walking or biking in the city, and so on. Sounds from these environments were recorded binaurally with in-the-ear microphones and were analyzed with respect to speech-likeness measures and interaural difference measures. The speech-likeness measures were based on amplitude-modulation patterns within frequency bands and were estimated for 1-s time-slices. The interaural difference measures included interaural coherence, interaural time difference, and interaural level difference, which were estimated for time-slices of 20-ms duration. These binaural measures were documented for one-fourth-octave frequency bands centered at 500 Hz and for the envelopes of one-fourth-octave bands centered at 2000 Hz. For comparison purposes, the same speech-likeness and interaural difference measures were computed for a set of virtual recordings that mimic typical clinical test configurations. These virtual recordings were created by filtering anechoic waveforms with available head-related transfer functions and combining them to create multiple source combinations. Overall, the speech-likeness results show large variability within and between environments, and they demonstrate the importance of having information from both ears available. Furthermore, the interaural parameter results show that the natural recordings contain a relatively small proportion of time-slices with high coherence compared with the virtual recordings; however, when present, binaural cues might be used for selecting intervals with good speech intelligibility for individual sources.

A Physiologically Inspired Model for Solving the Cocktail Party Problem.

At a cocktail party, we can broadly monitor the entire acoustic scene to detect important cues (e.g., our names being called, or the fire alarm going off), or selectively listen to a target sound source (e.g., a conversation partner). It has recently been observed that individual neurons in the avian field L (analog to the mammalian auditory cortex) can display broad spatial tuning to single targets and selective tuning to a target embedded in spatially distributed sound mixtures. Here, we describe a model inspired by these experimental observations and apply it to process mixtures of human speech sentences. This processing is realized in the neural spiking domain. It converts binaural acoustic inputs into cortical spike trains using a multi-stage model composed of a cochlear filter-bank, a midbrain spatial-localization network, and a cortical network. The output spike trains of the cortical network are then converted back into an acoustic waveform, using a stimulus reconstruction technique. The intelligibility of the reconstructed output is quantified using an objective measure of speech intelligibility. We apply the algorithm to single and multi-talker speech to demonstrate that the physiologically inspired algorithm is able to achieve intelligible reconstruction of an "attended" target sentence embedded in two other non-attended masker sentences. The algorithm is also robust to masker level and displays performance trends comparable to humans. The ideas from this work may help improve the performance of hearing assistive devices (e.g., hearing aids and cochlear implants), speech-recognition technology, and computational algorithms for processing natural scenes cluttered with spatially distributed acoustic objects.

Noise edge pitch and models of pitch perception.

Monaural noise edge pitch (NEP) is evoked by a broadband noise with a sharp falling edge in the power spectrum. The pitch is heard near the spectral edge frequency but shifted slightly into the frequency region of the noise. Thus, the pitch of a lowpass (LP) noise is matched by a pure tone typically 2%-5% below the edge, whereas the pitch of highpass (HP) noise is matched a comparable amount above the edge. Musically trained listeners can recognize musical intervals between NEPs. The pitches can be understood from a temporal pattern-matching model of pitch perception based on the peaks of a simplified autocorrelation function. The pitch shifts arise from limits on the autocorrelation window duration. An alternative place-theory approach explains the pitch shifts as the result of lateral inhibition. Psychophysical experiments using edge frequencies of 100 Hz and below find that LP-noise pitches exist but HP-noise pitches do not. The result is consistent with a temporal analysis in tonotopic regions outside the noise band. LP and HP experiments with high-frequency edges find that pitch tends to disappear as the edge frequency approaches 5000 Hz, as expected from a timing theory, though exceptional listeners can go an octave higher.

The fMRI Data of Thompson et al. (2006) Do Not Constrain How the Human Midbrain Represents Interaural Time Delay.

This commentary provides an alternate interpretation of the fMRI data that were presented in a communication to the journal Nature Neuroscience (Thompson et al., Nat. Neurosci. 9: 1096-1098, 2006 ). The authors argued that their observations demonstrated that traditional models of binaural hearing which incorporate "internal delays," such as the coincidence-counting mechanism proposed by Jeffress and quantified by Colburn, are invalid, and that a new model for human interaural time delay processing must be developed. We argue that the fMRI data presented do not strongly favor either the refutation or the retention of the traditional models, although they may be useful in constraining the physiological sites of various processing stages. The conclusions of Thompson et al. are based on the locations of maximal activity in the midbrain in response to selected binaural signals. These locations are inconsistent with well-known perceptual attributes of the stimuli under consideration, as is noted by the authors, which suggests that further processing is involved in forming the percept of subjective lateral position.

Comparison of a target-equalization-cancellation approach and a localization approach to source separation.

Interaural differences are important for listeners to be able to maintain focus on a sound source of interest in the presence of multiple sources. Because interaural differences are sound localization cues, most binaural-cue-based source separation algorithms attempt separation by localizing each time-frequency (T-F) unit to one of the possible source directions using interaural differences. By assembling T-F units that are assigned to one direction, the sound stream from that direction is enhanced. In this paper, a different type of binaural cue for source-separation purposes is proposed. For each T-F unit, the target-direction signal is cancelled by applying the equalization-cancellation (EC) operation to cancel the signal from the target direction; then, the dominance of the target in each T-F unit is determined by the effectiveness of the cancellation. Specifically, the energy change from cancellation is used as the criterion for target dominance for each T-F unit. Source-separation performance using the target-EC cue is compared with performance using localization cues. With simulated multi-talker and diffuse-babble interferers, the algorithm based on target-EC cues yields better source-separation performance than the algorithm based on localization cues, both in direct comparison with the ideal binary mask and in measured speech intelligibility for the separated target streams.

Asymmetries in behavioral and neural responses to spectral cues demonstrate the generality of auditory looming bias.

Studies of auditory looming bias have shown that sources increasing in intensity are more salient than sources decreasing in intensity. Researchers have argued that listeners are more sensitive to approaching sounds compared with receding sounds, reflecting an evolutionary pressure. However, these studies only manipulated overall sound intensity; therefore, it is unclear whether looming bias is truly a perceptual bias for changes in source distance, or only in sound intensity. Here we demonstrate both behavioral and neural correlates of looming bias without manipulating overall sound intensity. In natural environments, the pinnae induce spectral cues that give rise to a sense of externalization; when spectral cues are unnatural, sounds are perceived as closer to the listener. We manipulated the contrast of individually tailored spectral cues to create sounds of similar intensity but different naturalness. We confirmed that sounds were perceived as approaching when spectral contrast decreased, and perceived as receding when spectral contrast increased. We measured behavior and electroencephalography while listeners judged motion direction. Behavioral responses showed a looming bias in that responses were more consistent for sounds perceived as approaching than for sounds perceived as receding. In a control experiment, looming bias disappeared when spectral contrast changes were discontinuous, suggesting that perceived motion in distance and not distance itself was driving the bias. Neurally, looming bias was reflected in an asymmetry of late event-related potentials associated with motion evaluation. Hence, both our behavioral and neural findings support a generalization of the auditory looming bias, representing a perceptual preference for approaching auditory objects.

Relating interaural difference sensitivities for several parameters measured in normal-hearing and hearing-impaired listeners.

Just-noticeable differences (JNDs) in interaural time delay (ITD), interaural level difference (ILD), and interaural cross-correlation (ICC) were measured with low- and high-frequency noise bands over multiple sessions for 10 normal-hearing (NH) and 11 hearing-impaired (HI) listeners. Individual subject thresholds tended to improve with training then stabilize. Measured JNDs varied over these experienced listeners, for both subject groups and all tasks. Group JNDs were seldom predictable from hearing level. Individual listeners' JNDs were highly correlated across frequency for each task and group, except for ICC in the HI listeners. Further, ITD JNDs almost always significantly correlated with ILD JNDs within a group. Finally, although the ICC JNDs always significantly correlated with the ITD or ILD JNDs for the NH listeners, they often did not for the HI listeners. These findings suggest that little information about binaural sensitivity is added for NH listeners with multiple ITD, ILD, and ICC measures. For HI listeners, however, while ITD and ILD measures are well correlated, information is added with ICC measures. In general, the results suggest that less information is added with JND measures for NH listeners (15 significant correlations) than for HI listeners (six significant correlations).

A Binaural Grouping Model for Predicting Speech Intelligibility in Multitalker Environments.

Spatially separating speech maskers from target speech often leads to a large intelligibility improvement. Modeling this phenomenon has long been of interest to binaural-hearing researchers for uncovering brain mechanisms and for improving signal-processing algorithms in hearing-assistive devices. Much of the previous binaural modeling work focused on the unmasking enabled by binaural cues at the periphery, and little quantitative modeling has been directed toward the grouping or source-separation benefits of binaural processing. In this article, we propose a binaural model that focuses on grouping, specifically on the selection of time-frequency units that are dominated by signals from the direction of the target. The proposed model uses Equalization-Cancellation (EC) processing with a binary decision rule to estimate a time-frequency binary mask. EC processing is carried out to cancel the target signal and the energy change between the EC input and output is used as a feature that reflects target dominance in each time-frequency unit. The processing in the proposed model requires little computational resources and is straightforward to implement. In combination with the Coherence-based Speech Intelligibility Index, the model is applied to predict the speech intelligibility data measured by Marrone et al. The predicted speech reception threshold matches the pattern of the measured data well, even though the predicted intelligibility improvements relative to the colocated condition are larger than some of the measured data, which may reflect the lack of internal noise in this initial version of the model.

Cortical Transformation of Spatial Processing for Solving the Cocktail Party Problem: A Computational Model(1,2,3).

In multisource, "cocktail party" sound environments, human and animal auditory systems can use spatial cues to effectively separate and follow one source of sound over competing sources. While mechanisms to extract spatial cues such as interaural time differences (ITDs) are well understood in precortical areas, how such information is reused and transformed in higher cortical regions to represent segregated sound sources is not clear. We present a computational model describing a hypothesized neural network that spans spatial cue detection areas and the cortex. This network is based on recent physiological findings that cortical neurons selectively encode target stimuli in the presence of competing maskers based on source locations (Maddox et al., 2012). We demonstrate that key features of cortical responses can be generated by the model network, which exploits spatial interactions between inputs via lateral inhibition, enabling the spatial separation of target and interfering sources while allowing monitoring of a broader acoustic space when there is no competition. We present the model network along with testable experimental paradigms as a starting point for understanding the transformation and organization of spatial information from midbrain to cortex. This network is then extended to suggest engineering solutions that may be useful for hearing-assistive devices in solving the cocktail party problem.

Modeling binaural responses in the auditory brainstem to electric stimulation of the auditory nerve.

Bilateral cochlear implants (CIs) provide improvements in sound localization and speech perception in noise over unilateral CIs. However, the benefits arise mainly from the perception of interaural level differences, while bilateral CI listeners' sensitivity to interaural time difference (ITD) is poorer than normal. To help understand this limitation, a set of ITD-sensitive neural models was developed to study binaural responses to electric stimulation. Our working hypothesis was that central auditory processing is normal with bilateral CIs so that the abnormality in the response to electric stimulation at the level of the auditory nerve fibers (ANFs) is the source of the limited ITD sensitivity. A descriptive model of ANF response to both acoustic and electric stimulation was implemented and used to drive a simplified biophysical model of neurons in the medial superior olive (MSO). The model's ITD sensitivity was found to depend strongly on the specific configurations of membrane and synaptic parameters for different stimulation rates. Specifically, stronger excitatory synaptic inputs and faster membrane responses were required for the model neurons to be ITD-sensitive at high stimulation rates, whereas weaker excitatory synaptic input and slower membrane responses were necessary at low stimulation rates, for both electric and acoustic stimulation. This finding raises the possibility of frequency-dependent differences in neural mechanisms of binaural processing; limitations in ITD sensitivity with bilateral CIs may be due to a mismatch between stimulation rate and cell parameters in ITD-sensitive neurons.

Application of a short-time version of the Equalization-Cancellation model to speech intelligibility experiments with speech maskers.

A short-time-processing version of the Equalization-Cancellation (EC) model of binaural processing is described and applied to speech intelligibility tasks in the presence of multiple maskers, including multiple speech maskers. This short-time EC model, called the STEC model, extends the model described by Wan et al. [J. Acoust. Soc. Am. 128, 3678-3690 (2010)] to allow the EC model's equalization parameters τ and α to be adjusted as a function of time, resulting in improved masker cancellation when the dominant masker location varies in time. Using the Speech Intelligibility Index, the STEC model is applied to speech intelligibility with maskers that vary in number, type, and spatial arrangements. Most notably, when maskers are located on opposite sides of the target, this STEC model predicts improved thresholds when the maskers are modulated independently with speech-envelope modulators; this includes the most relevant case of independent speech maskers. The STEC model describes the spatial dependence of the speech reception threshold with speech maskers better than the steady-state model. Predictions are also improved for independently speech-modulated noise maskers but are poorer for reversed-speech maskers. In general, short-term processing is useful, but much remains to be done in the complex task of understanding speech in speech maskers.

Dual sensitivity of inferior colliculus neurons to ITD in the envelopes of high-frequency sounds: experimental and modeling study.

Human listeners are sensitive to interaural time differences (ITDs) in the envelopes of sounds, which can serve as a cue for sound localization. Many high-frequency neurons in the mammalian inferior colliculus (IC) are sensitive to envelope-ITDs of sinusoidally amplitude-modulated (SAM) sounds. Typically, envelope-ITD-sensitive IC neurons exhibit either peak-type sensitivity, discharging maximally at the same delay across frequencies, or trough-type sensitivity, discharging minimally at the same delay across frequencies, consistent with responses observed at the primary site of binaural interaction in the medial and lateral superior olives (MSO and LSO), respectively. However, some high-frequency IC neurons exhibit dual types of envelope-ITD sensitivity in their responses to SAM tones, that is, they exhibit peak-type sensitivity at some modulation frequencies and trough-type sensitivity at other frequencies. Here we show that high-frequency IC neurons in the unanesthetized rabbit can also exhibit dual types of envelope-ITD sensitivity in their responses to SAM noise. Such complex responses to SAM stimuli could be achieved by convergent inputs from MSO and LSO onto single IC neurons. We test this hypothesis by implementing a physiologically explicit, computational model of the binaural pathway. Specifically, we examined envelope-ITD sensitivity of a simple model IC neuron that receives convergent inputs from MSO and LSO model neurons. We show that dual envelope-ITD sensitivity emerges in the IC when convergent MSO and LSO inputs are differentially tuned for modulation frequency.

A modeling study of the responses of the lateral superior olive to ipsilateral sinusoidally amplitude-modulated tones.

The lateral superior olive (LSO) is a brainstem nucleus that is classically understood to encode binaural information in high-frequency sounds. Previous studies have shown that LSO cells are sensitive to envelope interaural time difference in sinusoidally amplitude-modulated (SAM) tones (Joris and Yin, J Neurophysiol 73:1043-1062, 1995; Joris, J Neurophysiol 76:2137-2156, 1996) and that a subpopulation of LSO neurons exhibit low-threshold potassium currents mediated by Kv1 channels (Barnes-Davies et al., Eur J Neurosci 19:325-333, 2004). It has also been shown that in many LSO cells the average response rate to ipsilateral SAM tones decreases with modulation frequency above a few hundred Hertz (Joris and Yin, J Neurophysiol 79:253-269, 1998). This low-pass feature is not directly inherited from the inputs to the LSO since the response rate of these input neurons changes little with increasing modulation frequency. In the current study, an LSO cell model is developed to investigate mechanisms consistent with the responses described above, notably the emergent rate decrease with increasing frequency. The mechanisms explored included the effects of after-hyperpolarization (AHP) channels, the dynamics of low-threshold potassium channels (KLT), and the effects of background inhibition. In the model, AHP channels alone were not sufficient to induce the observed rate decrease at high modulation frequencies. The model also suggests that the background inhibition alone, possibly from the medial nucleus of the trapezoid body, can account for the small rate decrease seen in some LSO neurons, but could not explain the large rate decrease seen in other LSO neurons at high modulation frequencies. In contrast, both the small and large rate decreases were replicated when KLT channels were included in the LSO neuron model. These results support the conclusion that KLT channels may play a major role in the large rate decreases seen in some units and that background inhibition may be a contributing factor, a factor that could be adequate for small decreases.

Auditory sensitivity may require dynamically unstable spike generators: evidence from a model of electrical stimulation.

The response of the auditory nerve to electrical stimulation is highly sensitive to small modulations (<0.5%). This report demonstrates that dynamical instability (i.e., a positive Lyapunov exponent) can account for this sensitivity in a modified FitzHugh-Nagumo model of spike generation, so long as the input noise is not too large. This finding suggests both that spike generator instability is necessary to account for auditory nerve sensitivity and that the amplitude of physiological noise, such as that produced by the random behavior of voltage-gated sodium channels, is small. Based on these results with direct electrical stimulation, it is hypothesized that spike generator instability may be the mechanism that reconciles high sensitivity with the cross-fiber independence observed under acoustic stimulation.

Physiological and psychophysical modeling of the precedence effect.

Many past studies of sound localization explored the precedence effect (PE), in which a pair of brief, temporally close sounds from different directions is perceived as coming from a location near that of the first-arriving sound. Here, a computational model of low-frequency inferior colliculus (IC) neurons accounts for both physiological and psychophysical responses to PE click stimuli. In the model, IC neurons have physiologically plausible inputs, receiving excitation from the ipsilateral medial superior olive (MSO) and long-lasting inhibition from both ipsilateral and contralateral MSOs, relayed through the dorsal nucleus of the lateral lemniscus. In this model, physiological suppression of the lagging response depends on the inter-stimulus delay (ISD) between the lead and lag as well as their relative locations. Psychophysical predictions are generated from a population of model neurons. At all ISDs, predicted lead localization is good. At short ISDs, the estimated location of the lag is near that of the lead, consistent with subjects perceiving both lead and lag from the lead location. As ISD increases, the estimated lag location moves closer to the true lag location, consistent with listeners' perception of two sounds from separate locations. Together, these simulations suggest that location-dependent suppression in IC neurons can explain the behavioral phenomenon known as the precedence effect.

A modeling study of the effects of membrane afterhyperpolarization on spike interval statistics and on ILD encoding in the lateral superior olive.

The lateral superior olive (LSO) is the first nucleus in the ascending auditory pathway that encodes acoustic level information from both ears, the interaural level difference (ILD). This sensitivity is believed to result from the relative strengths of ipsilateral excitation and contralateral inhibition. The study reported here simulated sound-evoked responses of LSO chopper units with a focus on the role of the heterogeneity in membrane afterhyperpolarization (AHP) channels on spike interval statistics and on ILD encoding. A relatively simplified cell model was used so that the effects of interest could be isolated. Specifically, the amplitude and time constant of the AHP conductance within a leaky integrate-and-fire (LIF) cell model were studied. This extends the work of others who used a more physiologically detailed model. Results show that differences in these two parameters lead to both the distinctive chopper response patterns and to the level-dependent interval statistics as observed in vivo. In general, diverse AHP characteristics enable an enhanced contrast across population responses with respect to rate gain and temporal correlations. This membrane heterogeneity provides an internal, cell-specific dimension for the neural representation of stimulus information, allowing sensitivity to ILDs of dynamic stimuli.

Application of an extended equalization-cancellation model to speech intelligibility with spatially distributed maskers.

An extended version of the equalization-cancellation (EC) model of binaural processing is described and applied to speech intelligibility tasks in the presence of multiple maskers. The model incorporates time-varying jitters, both in time and amplitude, and implements the equalization and cancellation operations in each frequency band independently. The model is consistent with the original EC model in predicting tone-detection performance for a large set of configurations. When the model is applied to speech, the speech intelligibility index is used to predict speech intelligibility performance in a variety of conditions. Specific conditions addressed include different types of maskers, different numbers of maskers, and different spatial locations of maskers. Model predictions are compared with empirical measurements reported by Hawley et al. [J. Acoust. Soc. Am. 115, 833-843 (2004)] and by Marrone et al. [J. Acoust. Soc. Am. 124, 1146-1158 (2008)]. The model succeeds in predicting speech intelligibility performance when maskers are speech-shaped noise or broadband-modulated speech-shaped noise but fails when the maskers are speech or reversed speech.