The neural representation of an auditory spatial cue in the primate cortex.

Humans make use of small differences in the timing of sounds at the two ears-interaural time differences (ITDs)-to locate their sources. Despite extensive investigation, however, the neural representation of ITDs in the human brain is contentious, particularly the range of ITDs explicitly represented by dedicated neural detectors. Here, using magneto- and electro-encephalography (MEG and EEG), we demonstrate evidence of a sparse neural representation of ITDs in the human cortex. The magnitude of cortical activity to sounds presented via insert earphones oscillated as a function of increasing ITD-within and beyond auditory cortical regions-and listeners rated the perceptual quality of these sounds according to the same oscillating pattern. This pattern was accurately described by a population of model neurons with preferred ITDs constrained to the narrow, sound-frequency-dependent range evident in other mammalian species. When scaled for head size, the distribution of ITD detectors in the human cortex is remarkably like that recorded in vivo from the cortex of rhesus monkeys, another large primate that uses ITDs for source localization. The data solve a long-standing issue concerning the neural representation of ITDs in humans and suggest a representation that scales for head size and sound frequency in an optimal manner.

Deep Factor Learning for Accurate Brain Neuroimaging Data Analysis on Discrimination for Structural MRI and Functional MRI.

Analysis of neuroimaging data (e.g., Magnetic Resonance Imaging, structural and functional MRI) plays an important role in monitoring brain dynamics and probing brain structures. Neuroimaging data are multi-featured and non-linear by nature, and it is a natural way to organise these data as tensors prior to performing automated analyses such as discrimination of neurological disorders like Parkinson's Disease (PD) and Attention Deficit and Hyperactivity Disorder (ADHD). However, the existing approaches are often subject to performance bottlenecks (e.g., conventional feature extraction and deep learning based feature construction), as these can lose the structural information that correlates multiple data dimensions or/and demands excessive empirical and application-specific settings. This study proposes a Deep Factor Learning model on a Hilbert Basis tensor (namely, HB-DFL) to automatically derive latent low-dimensional and concise factors of tensors. This is achieved through the application of multiple Convolutional Neural Networks (CNNs) in a non-linear manner along all possible dimensions with no assumed a priori knowledge. HB-DFL leverages the Hilbert basis tensor to enhance the stability of the solution by regularizing the core tensor to allow any component in a certain domain to interact with any component in the other dimensions. The final multi-domain features are handled through another multi-branch CNN to achieve reliable classification, exemplified here using MRI discrimination as a typical case. A case study of MRI discrimination has been performed on public MRI datasets for discrimination of PD and ADHD. Results indicate that 1) HB-DFL outperforms the counterparts in terms of FIT, mSIR and stability (mSC and umSC) of factor learning; 2) HB-DFL identifies PD and ADHD with an accuracy significantly higher than state-of-the-art methods do. Overall, HB-DFL has significant potentials for neuroimaging data analysis applications with its stability of automatic construction of structural features.

Functional connectivity learning via Siamese-based SPD matrix representation of brain imaging data.

Measurement of brain functional connectivity has become a dominant approach to explore the interaction dynamics between brain regions of subjects under examination. Conventional functional connectivity measures largely originate from deterministic models on empirical analysis, usually demanding application-specific settings (e.g., Pearson's Correlation and Mutual Information). To bridge the technical gap, this study proposes a Siamese-based Symmetric Positive Definite (SPD) Matrix Representation framework (SiameseSPD-MR) to derive the functional connectivity of brain imaging data (BID) such as Electroencephalography (EEG), thus the alternative application-independent measure (in the form of SPD matrix) can be automatically learnt: (1) SiameseSPD-MR first exploits graph convolution to extract the representative features of BID with the adjacency matrix computed considering the anatomical structure; (2) Adaptive Gaussian kernel function then applies to obtain the functional connectivity representations from the deep features followed by SPD matrix transformation to address the intrinsic functional characteristics; and (3) Two-branch (Siamese) networks are combined via an element-wise product followed by a dense layer to derive the similarity between the pairwise inputs. Experimental results on two EEG datasets (autism spectrum disorder, emotion) indicate that (1) SiameseSPD-MR can capture more significant differences in functional connectivity between neural states than the state-of-the-art counterparts do, and these findings properly highlight the typical EEG characteristics of ASD subjects, and (2) the obtained functional connectivity representations conforming to the proposed measure can act as meaningful markers for brain network analysis and ASD discrimination.

Disentangled and Side-Aware Unsupervised Domain Adaptation for Cross-Dataset Subjective Tinnitus Diagnosis.

EEG-based tinnitus classification is a valuable tool for tinnitus diagnosis, research, and treatments. Most current works are limited to a single dataset where data patterns are similar. But EEG signals are highly non-stationary, resulting in model's poor generalization to new users, sessions or datasets. Thus, designing a model that can generalize to new datasets is beneficial and indispensable. To mitigate distribution discrepancy across datasets, we propose to achieve Disentangled and Side-aware Unsupervised Domain Adaptation (DSUDA) for cross-dataset tinnitus diagnosis. A disentangled auto-encoder is developed to decouple class-irrelevant information from the EEG signals to improve the classifying ability. The side-aware unsupervised domain adaptation module adapts the class-irrelevant information as domain variance to a new dataset and excludes the variance to obtain the class-distill features for the new dataset classification. It also aligns signals of left and right ears to overcome inherent EEG pattern difference. We compare DSUDA with state-of-the-art methods, and our model achieves significant improvements over competitors regarding comprehensive evaluation criteria. The results demonstrate our model can successfully generalize to a new dataset and effectively diagnose tinnitus.

Cross-modal functional connectivity supports speech understanding in cochlear implant users.

Sensory deprivation can lead to cross-modal cortical changes, whereby sensory brain regions deprived of input may be recruited to perform atypical function. Enhanced cross-modal responses to visual stimuli observed in auditory cortex of postlingually deaf cochlear implant (CI) users are hypothesized to reflect increased activation of cortical language regions, but it is unclear if this cross-modal activity is "adaptive" or "mal-adaptive" for speech understanding. To determine if increased activation of language regions is correlated with better speech understanding in CI users, we assessed task-related activation and functional connectivity of auditory and visual cortices to auditory and visual speech and non-speech stimuli in CI users (n = 14) and normal-hearing listeners (n = 17) and used functional near-infrared spectroscopy to measure hemodynamic responses. We used visually presented speech and non-speech to investigate neural processes related to linguistic content and observed that CI users show beneficial cross-modal effects. Specifically, an increase in connectivity between the left auditory and visual cortices-presumed primary sites of cortical language processing-was positively correlated with CI users' abilities to understand speech in background noise. Cross-modal activity in auditory cortex of postlingually deaf CI users may reflect adaptive activity of a distributed, multimodal speech network, recruited to enhance speech understanding.

Side-Aware Meta-Learning for Cross-Dataset Listener Diagnosis With Subjective Tinnitus.

With the development of digital technology, machine learning has paved the way for the next generation of tinnitus diagnoses. Although machine learning has been widely applied in EEG-based tinnitus analysis, most current models are dataset-specific. Each dataset may be limited to a specific range of symptoms, overall disease severity, and demographic attributes; further, dataset formats may differ, impacting model performance. This paper proposes a side-aware meta-learning for cross-dataset tinnitus diagnosis, which can effectively classify tinnitus in subjects of divergent ages and genders from different data collection processes. Owing to the superiority of meta-learning, our method does not rely on large-scale datasets like conventional deep learning models. Moreover, we design a subject-specific training process to assist the model in fitting the data pattern of different patients or healthy people. Our method achieves a high accuracy of 73.8% in the cross-dataset classification. We conduct an extensive analysis to show the effectiveness of side information of ears in enhancing model performance and side-aware meta-learning in improving the quality of the learned features.

Helping People Hear Better with "Smart" Hearing Devices.

Millions of people around the world have difficulty hearing. Hearing aids and cochlear implants help people hear better, especially in quiet places. Unfortunately, these devices do not always help in noisy situations like busy classrooms or restaurants. This means that a person with hearing loss may struggle to follow a conversation with friends or family and may avoid going out. We used methods from the field of artificial intelligence to develop "smart" hearing aids and cochlear implants that can get rid of background noise. We play many different sounds into a computer program, which learns to pick out the speech sounds and filter out unwanted background noises. Once the computer program has been trained, it is then tested on new examples of noisy speech and can be incorporated into hearing aids or cochlear implants. These "smart" approaches can help people with hearing loss understand speech better in noisy situations.

Deep reinforcement learning guided graph neural networks for brain network analysis.

Modern neuroimaging techniques enable us to construct human brains as brain networks or connectomes. Capturing brain networks' structural information and hierarchical patterns is essential for understanding brain functions and disease states. Recently, the promising network representation learning capability of graph neural networks (GNNs) has prompted related methods for brain network analysis to be proposed. Specifically, these methods apply feature aggregation and global pooling to convert brain network instances into vector representations encoding brain structure induction for downstream brain network analysis tasks. However, existing GNN-based methods often neglect that brain networks of different subjects may require various aggregation iterations and use GNN with a fixed number of layers to learn all brain networks. Therefore, how to fully release the potential of GNNs to promote brain network analysis is still non-trivial. In our work, a novel brain network representation framework, BN-GNN, is proposed to solve this difficulty, which searches for the optimal GNN architecture for each brain network. Concretely, BN-GNN employs deep reinforcement learning (DRL) to automatically predict the optimal number of feature propagations (reflected in the number of GNN layers) required for a given brain network. Furthermore, BN-GNN improves the upper bound of traditional GNNs' performance in eight brain network disease analysis tasks.

Modulation Depth Discrimination by Cochlear Implant Users.

Cochlear implants (CIs) convey the amplitude envelope of speech by modulating high-rate pulse trains. However, not all of the envelope may be necessary to perceive amplitude modulations (AMs); the effective envelope depth may be limited by forward and backward masking from the envelope peaks. Three experiments used modulated pulse trains to measure which portions of the envelope can be effectively processed by CI users as a function of AM frequency. Experiment 1 used a three-interval forced-choice task to test the ability of CI users to discriminate less-modulated pulse trains from a fully modulated standard, without controlling for loudness. The stimuli in experiment 2 were identical, but a two-interval task was used in which participants were required to choose the less-modulated interval, ignoring loudness. Catch trials, in which judgements based on level or modulation depth would give opposing answers, were included. Experiment 3 employed novel stimuli whose modulation envelope could be modified below a variable point in the dynamic range, without changing the loudness of the stimulus. Overall, results showed that substantial portions of the envelope are not accurately encoded by CI users. In experiment 1, where loudness cues were available, participants on average were insensitive to changes in the bottom 30% of their dynamic range. In experiment 2, where loudness was controlled, participants appeared insensitive to changes in the bottom 50% of the dynamic range. In experiment 3, participants were insensitive to changes in the bottom 80% of the dynamic range. We discuss potential reasons for this insensitivity and implications for CI speech-processing strategies.

Understanding degraded speech leads to perceptual gating of a brainstem reflex in human listeners.

The ability to navigate "cocktail party" situations by focusing on sounds of interest over irrelevant, background sounds is often considered in terms of cortical mechanisms. However, subcortical circuits such as the pathway underlying the medial olivocochlear (MOC) reflex modulate the activity of the inner ear itself, supporting the extraction of salient features from auditory scene prior to any cortical processing. To understand the contribution of auditory subcortical nuclei and the cochlea in complex listening tasks, we made physiological recordings along the auditory pathway while listeners engaged in detecting non(sense) words in lists of words. Both naturally spoken and intrinsically noisy, vocoded speech-filtering that mimics processing by a cochlear implant (CI)-significantly activated the MOC reflex, but this was not the case for speech in background noise, which more engaged midbrain and cortical resources. A model of the initial stages of auditory processing reproduced specific effects of each form of speech degradation, providing a rationale for goal-directed gating of the MOC reflex based on enhancing the representation of the energy envelope of the acoustic waveform. Our data reveal the coexistence of 2 strategies in the auditory system that may facilitate speech understanding in situations where the signal is either intrinsically degraded or masked by extrinsic acoustic energy. Whereas intrinsically degraded streams recruit the MOC reflex to improve representation of speech cues peripherally, extrinsically masked streams rely more on higher auditory centres to denoise signals.

Generalizable Sample-Efficient Siamese Autoencoder for Tinnitus Diagnosis in Listeners With Subjective Tinnitus.

Electroencephalogram (EEG)-based neurofeedback has been widely studied for tinnitus therapy in recent years. Most existing research relies on experts' cognitive prediction, and studies based on machine learning and deep learning are either data-hungry or not well generalizable to new subjects. In this paper, we propose a robust, data-efficient model for distinguishing tinnitus from the healthy state based on EEG-based tinnitus neurofeedback. We propose trend descriptor, a feature extractor with lower fineness, to reduce the effect of electrode noises on EEG signals, and a siamese encoder-decoder network boosted in a supervised manner to learn accurate alignment and to acquire high-quality transferable mappings across subjects and EEG signal channels. Our experiments show the proposed method significantly outperforms state-of-the-art algorithms when analyzing subjects' EEG neurofeedback to 90dB and 100dB sound, achieving an accuracy of 91.67%-94.44% in predicting tinnitus and control subjects in a subject-independent setting. Our ablation studies on mixed subjects and parameters show the method's stability in performance.

A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers.

Brain signals refer to the biometric information collected from the human brain. The research on brain signals aims to discover the underlying neurological or physical status of the individuals by signal decoding. The emerging deep learning techniques have improved the study of brain signals significantly in recent years. In this work, we first present a taxonomy of non-invasive brain signals and the basics of deep learning algorithms. Then, we provide the frontiers of applying deep learning for non-invasive brain signals analysis, by summarizing a large number of recent publications. Moreover, upon the deep learning-powered brain signal studies, we report the potential real-world applications which benefit not only disabled people but also normal individuals. Finally, we discuss the opening challenges and future directions.

Hidden Hearing Loss Impacts the Neural Representation of Speech in Background Noise.

Many individuals with seemingly normal hearing abilities struggle to understand speech in noisy backgrounds. To understand why this might be the case, we investigated the neural representation of speech in the auditory midbrain of gerbils with "hidden hearing loss" through noise exposure that increased hearing thresholds only temporarily. In noise-exposed animals, we observed significantly increased neural responses to speech stimuli, with a more pronounced increase at moderate than at high sound intensities. Noise exposure reduced discriminability of neural responses to speech in background noise at high sound intensities, with impairment most severe for tokens with relatively greater spectral energy in the noise-exposure frequency range (2-4 kHz). At moderate sound intensities, discriminability was surprisingly improved, which was unrelated to spectral content. A model combining damage to high-threshold auditory nerve fibers with increased response gain of central auditory neurons reproduced these effects, demonstrating that a specific combination of peripheral damage and central compensation could explain listening difficulties despite normal hearing thresholds.

Tolerable delay for speech production and perception: effects of hearing ability and experience with hearing aids.

OBJECTIVE: Processing delay is one of the important factors that limit the development of novel algorithms for hearing devices. In this study, both normal-hearing listeners and listeners with hearing loss were tested for their tolerance of processing delay up to 50 ms using a real-time setup for own-voice and external-voice conditions based on linear processing to avoid confounding effects of time-dependent gain. DESIGN: Participants rated their perceived subjective annoyance for each condition on a 7-point Likert scale. STUDY SAMPLE: Twenty normal-hearing participants and twenty participants with a range of mild to moderate hearing losses. RESULTS: Delay tolerance was significantly greater for the participants with hearing loss in two out of three voice conditions. The average slopes of annoyance ratings were negatively correlated with the degree of hearing loss across participants. A small trend of higher tolerance of delay by experienced users of hearing aids in comparison to new users was not significant. CONCLUSION: The increased tolerance of processing delay for speech production and perception with hearing loss and reduced sensitivity to changes in delay with stronger hearing loss may be beneficial for novel algorithms for hearing devices but the setup used in this study differed from commercial hearing aids.

Authenticity and Fashionable Disease in Eighteenth-Century Britain.

Throughout the eighteenth century the issue of authenticity shaped portrayals of fashionable diseases. From the very beginning of the century, writers satirized the behavior of elite invalids who paraded their delicacy as a sign of their status. As disorders such as the spleen came to be regarded as "fashionable," the legitimacy of patients' claims to suffer from distinguished diseases was called further into question, with some observers questioning the validity of the disease categories themselves. During the early and middle decades of the century, criticism was largely confined to periodicals, plays, and poetry, while medical writers wrote in defense of the authenticity of such conditions. The adoption of fashionable ailments and nervous sensibility grew increasingly popular, however, and from the 1770s onwards practitioners and novelists increasingly suggested that such diseases should not be trusted as signifiers of personal qualities or social status.

Influence of envelope waveform on ITD sensitivity of neurons in the auditory midbrain.

Interaural time differences (ITDs) conveyed by the modulated envelopes of high-frequency sounds can serve as a cue for localizing a sound source. Klein-Hennig et al. (J Acoust Soc Am 129: 3856, 2011) demonstrated the envelope attack (the rate at which stimulus energy in the envelope increases) and the duration of the pause (the interval between successive envelope pulses) as important factors affecting sensitivity to envelope ITDs in human listeners. Modulated sounds with rapid attacks and long pauses produce the lowest ITD discrimination thresholds. The duration of the envelope's sustained component (sustain) and the rate at which stimulus energy falls at the offset of the envelope (decay) are only minor factors. We assessed the responses of 71 single neurons, recorded from the midbrains of 15 urethane-anesthetized tri-colored guinea pigs, to envelope shapes in which the four envelope components, i.e., attack, sustain, decay, and pause, were systematically varied. We confirmed the importance of the attack and pause components in generating ITD-sensitive responses. Analysis of neural firing rates demonstrated more neurons (49/71) show ITD sensitivity in response to "damped" stimuli (fast attack and slow decay) compared with "ramped" stimuli (slow attack and fast decay) (14/71). Furthermore, the lowest threshold for the damped stimulus (91 µs) was lower by a factor of 4 than that for the temporally reversed ramped envelope shape (407 µs). The data confirm the importance of fast attacks and optimal pause durations in generating sensitivity to ITDs conveyed in the modulated envelopes of high-frequency sounds and are incompatible with models of ITD processing based on the integration of sound energy over time.NEW & NOTEWORTHY Using single-neuron electrophysiology, we show that the precise shape of a sound's "energy envelope" is a critical factor in determining how well midbrain neurons are able to convey information about auditory spatial cues. Consistent with human behavioral performance, sounds with rapidly rising energy and relatively long intervals between energy bursts are best at conveying spatial information. The data suggest specific sound energy patterns that might best be applied to hearing devices to aid spatial listening.

Auditory inspired machine learning techniques can improve speech intelligibility and quality for hearing-impaired listeners.

Machine-learning based approaches to speech enhancement have recently shown great promise for improving speech intelligibility for hearing-impaired listeners. Here, the performance of three machine-learning algorithms and one classical algorithm, Wiener filtering, was compared. Two algorithms based on neural networks were examined, one using a previously reported feature set and one using a feature set derived from an auditory model. The third machine-learning approach was a dictionary-based sparse-coding algorithm. Speech intelligibility and quality scores were obtained for participants with mild-to-moderate hearing impairments listening to sentences in speech-shaped noise and multi-talker babble following processing with the algorithms. Intelligibility and quality scores were significantly improved by each of the three machine-learning approaches, but not by the classical approach. The largest improvements for both speech intelligibility and quality were found by implementing a neural network using the feature set based on auditory modeling. Furthermore, neural network based techniques appeared more promising than dictionary-based, sparse coding in terms of performance and ease of implementation.

Speech enhancement based on neural networks improves speech intelligibility in noise for cochlear implant users.

Speech understanding in noisy environments is still one of the major challenges for cochlear implant (CI) users in everyday life. We evaluated a speech enhancement algorithm based on neural networks (NNSE) for improving speech intelligibility in noise for CI users. The algorithm decomposes the noisy speech signal into time-frequency units, extracts a set of auditory-inspired features and feeds them to the neural network to produce an estimation of which frequency channels contain more perceptually important information (higher signal-to-noise ratio, SNR). This estimate is used to attenuate noise-dominated and retain speech-dominated CI channels for electrical stimulation, as in traditional n-of-m CI coding strategies. The proposed algorithm was evaluated by measuring the speech-in-noise performance of 14 CI users using three types of background noise. Two NNSE algorithms were compared: a speaker-dependent algorithm, that was trained on the target speaker used for testing, and a speaker-independent algorithm, that was trained on different speakers. Significant improvements in the intelligibility of speech in stationary and fluctuating noises were found relative to the unprocessed condition for the speaker-dependent algorithm in all noise types and for the speaker-independent algorithm in 2 out of 3 noise types. The NNSE algorithms used noise-specific neural networks that generalized to novel segments of the same noise type and worked over a range of SNRs. The proposed algorithm has the potential to improve the intelligibility of speech in noise for CI users while meeting the requirements of low computational complexity and processing delay for application in CI devices.

A method to enhance the use of interaural time differences for cochlear implants in reverberant environments.

The ability of normal-hearing (NH) listeners to exploit interaural time difference (ITD) cues conveyed in the modulated envelopes of high-frequency sounds is poor compared to ITD cues transmitted in the temporal fine structure at low frequencies. Sensitivity to envelope ITDs is further degraded when envelopes become less steep, when modulation depth is reduced, and when envelopes become less similar between the ears, common factors when listening in reverberant environments. The vulnerability of envelope ITDs is particularly problematic for cochlear implant (CI) users, as they rely on information conveyed by slowly varying amplitude envelopes. Here, an approach to improve access to envelope ITDs for CIs is described in which, rather than attempting to reduce reverberation, the perceptual saliency of cues relating to the source is increased by selectively sharpening peaks in the amplitude envelope judged to contain reliable ITDs. Performance of the algorithm with room reverberation was assessed through simulating listening with bilateral CIs in headphone experiments with NH listeners. Relative to simulated standard CI processing, stimuli processed with the algorithm generated lower ITD discrimination thresholds and increased extents of laterality. Depending on parameterization, intelligibility was unchanged or somewhat reduced. The algorithm has the potential to improve spatial listening with CIs.