Human EEG and Recurrent Neural Networks Exhibit Common Temporal Dynamics During Speech Recognition.

Recent deep-learning artificial neural networks have shown remarkable success in recognizing natural human speech, however the reasons for their success are not entirely understood. Success of these methods might be because state-of-the-art networks use recurrent layers or dilated convolutional layers that enable the network to use a time-dependent feature space. The importance of time-dependent features in human cortical mechanisms of speech perception, measured by electroencephalography (EEG) and magnetoencephalography (MEG), have also been of particular recent interest. It is possible that recurrent neural networks (RNNs) achieve their success by emulating aspects of cortical dynamics, albeit through very different computational mechanisms. In that case, we should observe commonalities in the temporal dynamics of deep-learning models, particularly in recurrent layers, and brain electrical activity (EEG) during speech perception. We explored this prediction by presenting the same sentences to both human listeners and the Deep Speech RNN and considered the temporal dynamics of the EEG and RNN units for identical sentences. We tested whether the recently discovered phenomenon of envelope phase tracking in the human EEG is also evident in RNN hidden layers. We furthermore predicted that the clustering of dissimilarity between model representations of pairs of stimuli would be similar in both RNN and EEG dynamics. We found that the dynamics of both the recurrent layer of the network and human EEG signals exhibit envelope phase tracking with similar time lags. We also computed the representational distance matrices (RDMs) of brain and network responses to speech stimuli. The model RDMs became more similar to the brain RDM when going from early network layers to later ones, and eventually peaked at the recurrent layer. These results suggest that the Deep Speech RNN captures a representation of temporal features of speech in a manner similar to human brain.

Brain electrical dynamics in speech segmentation depends upon prior experience with the language.

It remains unclear whether the process of speech tracking, which facilitates speech segmentation, reflects top-down mechanisms related to prior linguistic models or stimulus-driven mechanisms, or possibly both. To address this, we recorded electroencephalography (EEG) responses from native and non-native speakers of English that had different prior experience with the English language but heard acoustically identical stimuli. Despite a significant difference in the ability to segment and perceive speech, our EEG results showed that theta-band tracking of the speech envelope did not depend significantly on prior experience with language. However, tracking in the theta-band did show changes across repetitions of the same sentence, suggesting a priming effect. Furthermore, native and non-native speakers showed different phase dynamics at word boundaries, suggesting differences in segmentation mechanisms. Finally, we found that the correlation between higher frequency dynamics reflecting phoneme-level processing and perceptual segmentation of words might depend on prior experience with the spoken language.

Speech Interaction to Control a Hands-Free Delivery Robot for High-Risk Health Care Scenarios.

The Covid-19 pandemic has had a widespread effect across the globe. The major effect on health-care workers and the vulnerable populations they serve has been of particular concern. Near-complete lockdown has been a common strategy to reduce the spread of the pandemic in environments such as live-in care facilities. Robotics is a promising area of research that can assist in reducing the spread of covid-19, while also preventing the need for complete physical isolation. The research presented in this paper demonstrates a speech-controlled, self-sanitizing robot that enables the delivery of items from a visitor to a resident of a care facility. The system is automated to reduce the burden on facility staff, and it is controlled entirely through hands-free audio interaction in order to reduce transmission of the virus. We demonstrate an end-to-end delivery test, and an in-depth evaluation of the speech interface. We also recorded a speech dataset with two conditions: the talker wearing a face mask and the talker not wearing a face mask. We then used this dataset to evaluate the speech recognition system. This enabled us to test the effect of face masks on speech recognition interfaces in the context of autonomous systems.

Phase of electroencephalography theta oscillation during stimulus encoding affects accuracy of memory recall.

Oscillatory activity is a ubiquitous property of brain signals, and yet relatively few studies have investigated how the phase of such ongoing oscillations affects our cognition. One of the main findings in this field is that the phase of electroencephalography (EEG) in the alpha band can affect perception of milliseconds-long stimuli. However, the importance of the phase of EEG for processing more naturalistic stimuli, which have a much longer duration, is still not clear. To address this question here, we presented word-nonword pairs, each of which was visible for 5 s and measured the effect of EEG phase during stimulus onset on later memory recall. The task consisted of an encoding (learning) phase in which 20 novel word-nonword pairs were presented, followed by a test phase in which participants were shown one of the seen words with four target nonwords to choose from. We found that memory recall performance was higher when the words during encoding were presented at a descending phase of the theta oscillation. This effect was the strongest in the frontal cortex. These results suggest that the phase of ongoing cortical activity can affect memorization of seconds-long stimuli that are an integral part of many daily tasks.

The effects of distractor set-size on neural tracking of attended speech.

Attention is crucial to speech comprehension in real-world, noisy environments. Selective phase-tracking between low-frequency brain dynamics and the envelope of target speech is a proposed mechanism to reject competing distractors. Studies have supported this theory in the case of a single distractor, but have not considered how tracking is systematically affected by varying distractor set sizes. We recorded electroencephalography (EEG) during selective listening to both natural and vocoded speech as distractor set-size varied from two to six voices. Increasing set-size reduced performance and attenuated EEG tracking of target speech. Further, we found that intrusions of distractor speech into perception were not accompanied by sustained tracking of the distractor stream. Our results support the theory that tracking of speech dynamics is a mechanism for selective attention, and that the mechanism of distraction is not simple stimulus-driven capture of sustained entrainment of auditory mechanisms by the acoustics of distracting speech.

The effects of periodic interruptions on cortical entrainment to speech.

Speech is perceived as a continuous stream of words despite consisting of a discontinuous, quasi-periodic signal of interleaved sounds and silences. Speech perception is surprisingly robust to interference by interruption, however speech that is replaced by gaps of silence is difficult to understand. When those silences are filled with noise, the speech is once again perceived as continuous even when the underlying speech sounds are removed completely. This is a phenomenon known as phonemic restoration. Perception of normal speech is accompanied by robust phase-locking of EEG signals to acoustic and linguistic features of speech. In this study we test the theory that interrupting speech with silence impairs perception by interfering with neural speech tracking. Further, we test the theory that we can restore perception and phase-tracking of the original acoustics by inserting noise in the interruptions. We find that disruptions of the acoustic envelope reduce the tracking of both acoustic and phonemic features. By inserting amplitude modulated noise such that the original broadband envelope is restored, we improved perception of the degraded speech and restored the magnitude of the speech tracking response; however, topographic analysis suggests that the neural response to noise-interrupted speech may recruit systematically different brain areas. The acoustic envelope seems to be an important physical component of speech that facilitates the dynamic neural mechanisms for perception of spoken language, particularly in adverse listening conditions.

Implicit Valuation of the Near-Miss is Dependent on Outcome Context.

Gambling studies have described a "near-miss effect" wherein the experience of almost winning increases gambling persistence. The near-miss has been proposed to inflate the value of preceding actions through its perceptual similarity to wins. We demonstrate here, however, that it acts as a conditioned stimulus to positively or negatively influence valuation, dependent on reward expectation and cognitive engagement. When subjects are asked to choose between two simulated slot machines, near-misses increase valuation of machines with a low payout rate, whereas they decrease valuation of high payout machines. This contextual effect impairs decisions and persists regardless of manipulations to outcome feedback or financial incentive provided for good performance. It is consistent with proposals that near-misses cause frustration when wins are expected, and we propose that it increases choice stochasticity and overrides avoidance of low-valued options. Intriguingly, the near-miss effect disappears when subjects are required to explicitly value machines by placing bets, rather than choosing between them. We propose that this task increases cognitive engagement and recruits participation of brain regions involved in cognitive processing, causing inhibition of otherwise dominant systems of decision-making. Our results reveal that only implicit, rather than explicit strategies of decision-making are affected by near-misses, and that the brain can fluidly shift between these strategies according to task demands.

Rendering visual events as sounds: Spatial attention capture by auditory augmented reality.

Many salient visual events tend to coincide with auditory events, such as seeing and hearing a car pass by. Information from the visual and auditory senses can be used to create a stable percept of the stimulus. Having access to related coincident visual and auditory information can help for spatial tasks such as localization. However not all visual information has analogous auditory percepts, such as viewing a computer monitor. Here, we describe a system capable of detecting and augmenting visual salient events into localizable auditory events. The system uses a neuromorphic camera (DAVIS 240B) to detect logarithmic changes of brightness intensity in the scene, which can be interpreted as salient visual events. Participants were blindfolded and asked to use the device to detect new objects in the scene, as well as determine direction of motion for a moving visual object. Results suggest the system is robust enough to allow for the simple detection of new salient stimuli, as well accurately encoding direction of visual motion. Future successes are probable as neuromorphic devices are likely to become faster and smaller in the future, making this system much more feasible.

Theta-band phase tracking in the two-talker problem.

It is usually easy to understand speech, but when several people are talking at once it becomes difficult. The brain must select one speech stream and ignore distracting streams. We tested a theory about the neural and computational mechanisms of attentional selection. The theory is that oscillating signals in brain networks phase-lock with amplitude fluctuations in speech. By doing this, brain-wide networks acquire information from the selected speech, but ignore other speech signals on the basis of their non-preferred dynamics. Two predictions were supported: first, attentional selection boosted the power of neuroelectric signals that were phase-locked with attended speech, but not ignored speech. Second, this phase selectivity was associated with better recall of the attended speech.

Brain dynamics encode the spectrotemporal boundaries of auditory objects.

Perception of objects in the scene around us is effortless and intuitive, yet entails profound computational challenges. Progress has been made in understanding some mechanisms by which the brain encodes the boundaries and surfaces of visual objects. However, in the auditory domain, these mechanisms are poorly understood. We investigated differences between neural responses to spectrotemporal boundaries in the auditory scene. We used iterated rippled noise to create perceptual boundaries with and without energy transients. In contrast to boundaries marked by energy transients, second-order boundaries were characterized by an absence of early components in the event-related potential. First-order energy boundaries triggered a transient evoked gamma-band response and a well-defined P90 component of the event-related potential, whereas second-order boundaries evoked only the later N1 component. Furthermore, the N1 component was delayed when evoked by second-order boundaries and theta-band electroencephalography activity at this latency exhibited significant phase lag for second-order compared to first-order boundaries. We speculate that boundaries defined by sharp energy transients can be registered by early feed-forward mechanisms. By contrast, boundaries defined only by discontinuities at discrete frequency bands require integration across the tonotopic representation of the frequency spectrum and require time-consuming interaction between auditory areas.

Mental rotational ability is correlated with spatial but not verbal working memory performance and P300 amplitude in males.

This study investigated how both sex and individual differences in a mental rotation test (MRT) influence performance on working memory (WM). To identify the neural substrate supporting these differences, brain electrical activity was measured using the event-related potential technique. No significant sex differences were observed in a test of verbal WM, however males were significantly faster than females to respond to probe stimuli in a test of spatial WM. This difference was no longer significant after controlling for differences in MRT score, suggesting that rotational ability mediates performance in the spatial memory task for both sexes. A posterior P300 was observed in both tasks as participants encoded information into memory, however the amplitude of the P300 correlated with RT in the spatial task but not in the verbal task. Individual differences in the MRT also correlated with RT and with the amplitude of the P300, but again only in the spatial task. After splitting the analysis by sex, partial correlations controlling for MRT revealed that for males, individual differences in rotational ability completely mediated the correlation between the P300 and RT in the spatial task. This mediating effect was not observed for the female participants. The results therefore suggest a relatively stronger association in males between innate mental rotational ability, spatial memory performance, and brain electrophysiological processes supporting spatial memory.

Dynamics of distraction: competition among auditory streams modulates gain and disrupts inter-trial phase coherence in the human electroencephalogram.

Auditory distraction is a failure to maintain focus on a stream of sounds. We investigated the neural correlates of distraction in a selective-listening pitch-discrimination task with high (competing speech) or low (white noise) distraction. High-distraction impaired performance and reduced the N1 peak of the auditory Event-Related Potential evoked by probe tones. In a series of simulations, we explored two theories to account for this effect: disruption of sensory gain or a disruption of inter-trial phase consistency. When compared to these simulations, our data were consistent with both effects of distraction. Distraction reduced the gain of the auditory evoked potential and disrupted the inter-trial phase consistency with which the brain responds to stimulus events. Tones at a non-target, unattended frequency were more susceptible to the effects of distraction than tones within an attended frequency band.

Temporal variability of the N2pc during efficient and inefficient visual search.

Efficient and inefficient visual search are characterized by the difference in the time required to find the target. Efficient "popout" search time is relatively unaffected by increases in the number of search items, whereas inefficient "non-popout" search time requires more time increases in duration. Electrophysiological investigations of the neural correlates of visual search have revealed a component of the event-related potential (ERP) known as the N2pc. The N2pc is thought to reflect the orienting of attention to the target during visual search. If the N2pc is more closely associated in time with the moment of target selection than the onset of the search display, it may be predicted that the N2pc would be delayed or more temporally variable during inefficient compared to efficient visual search. In the present study, we contrasted efficient "popout" search with inefficient "non-popout" search to investigate the hypothesis that the N2pc is temporally associated with the moment of search completion, and this would be reflected as a delayed or attenuated N2pc during non-popout search compared to popout. In Experiment 1, we observed a robust N2pc during popout search, but not during non-popout search. However, sorting trials by the N2pc latency recovered the N2pc during non-popout search. In Experiment 2 we replicated this observation and demonstrated that popout and non-popout search reflected differences in the temporal variability of the N2pc latency. Further investigation using time-spectral analysis suggests that evoked posterior-contralateral theta (4-6 Hz) underlies the N2pc during both popout and non-popout search.

Neural correlates of auditory distraction revealed in θ-band EEG.

Selective attention involves the exclusion of irrelevant information in order to optimize perception of a single source of sensory input; failure to do so often results in the familiar phenomenon of distraction. The term 'distraction' broadly refers to a perceptual phenomenon. In the present study we attempted to find the electrophysiological correlates of distraction using an auditory discrimination task. EEG and event-related potential responses to identical stimuli were compared under two levels of distraction (continuous broad-band noise or continuous speech). Relative to broad-band noise, the presence of a continuous speech signal in the unattended ear impaired task performance and also attenuated the N1 peak evoked by nontarget stimuli in the attended ear. As the magnitude of a peak in the event-related potential waveform can be modulated by differences in intertrial power but also by differences in the stability of EEG phase across trials, we sought to characterize the effect of distraction on intertrial power and intertrial phase locking around the latency of the N1. The presence of continuous speech resulted in a prominent reduction of theta EEG band intertrial phase locking around the latency of the N1. This suggests that distraction may act not only to disrupt a sensory gain mechanism but also to disrupt the temporal fidelity with which the brain responds to stimulus events.

Problem gamblers exhibit reward hypersensitivity in medial frontal cortex during gambling.

Problem gambling (PG) is increasingly conceptualized as an addiction akin to substance abuse, rather than an impulse control disorder, however the mechanism of addiction remains unclear. Neuroimaging investigations have supported a "reward deficiency" hypothesis for PG by suggesting a blunted response to gambling, particularly in the striatum. Here we describe electrophysiological evidence of a hypersensitive response to gambling feedback in problem gamblers. Previous research in healthy participants has shown that feedback during gambling tasks triggers stereotypical neural responses including the Feedback-Related Mediofrontal Negativity (FRN), the feedback-related P300, and an increase in induced theta-band (4-8 Hz) power. We tested the theory that abnormal feedback processing characterizes brain activity in problem gamblers while gambling. EEG was recorded from non-gamblers and self-identified gamblers as they engaged in a computerized version of the Iowa Gambling Task. Feedback about valence (win vs. loss) triggered a FRN in both groups, but in gamblers this was preceded by an early-latency hypersensitive fronto-central difference to feedback. This early FRN was correlated with gambling severity and was localized to medial frontal cortex using distributed source imaging (CLARA). Gamblers also differed in responses to risk, showing a blunted P300 component and less EEG power in the theta band. Here we suggest that a more nuanced interpretation of reward deficiency is called for with respect to PG. For certain aspects of brain function, gamblers may exhibit hypersensitivity to reward feedback more akin to drug sensitization than reward deficiency. Our results also suggest that the neurologically normal brain employs dissociable systems in the processing of feedback from tasks involving risky decision making.

A lateralized auditory evoked potential elicited when auditory objects are defined by spatial motion.

Scene analysis involves the process of segmenting a field of overlapping objects from each other and from the background. It is a fundamental stage of perception in both vision and hearing. The auditory system encodes complex cues that allow listeners to find boundaries between sequential objects, even when no gap of silence exists between them. In this sense, object perception in hearing is similar to perceiving visual objects defined by isoluminant color, motion or binocular disparity. Motion is one such cue: when a moving sound abruptly disappears from one location and instantly reappears somewhere else, the listener perceives two sequential auditory objects. Smooth reversals of motion direction do not produce this segmentation. We investigated the brain electrical responses evoked by this spatial segmentation cue and compared them to the familiar auditory evoked potential elicited by sound onsets. Segmentation events evoke a pattern of negative and positive deflections that are unlike those evoked by onsets. We identified a negative component in the waveform - the Lateralized Object-Related Negativity - generated by the hemisphere contralateral to the side on which the new sound appears. The relationship between this component and similar components found in related paradigms is considered.

Selective attention modulates electrical responses to reversals of optic-flow direction.

Attended stimuli typically evoke larger event-related potentials (ERPs) than unattended stimuli. We previously reported an exception when an optic-flow pattern is interleaved with stationary dots. Reversals of motion direction evoked a larger N200 peak when attention was directed to the stationary dots. We replicated and further characterized this result: the N200 enhancement was eliminated when the dots moved randomly rather than in optic flow. The effect was also attenuated with isoluminant stimuli. Electrical source analysis suggested the attentional modulation of a configuration of dorsal extrastriate generators. The ERP evoked by reversals of optic flow may reflect the operation of independently configurable attentional filters within visual cortex.

Right frontal cortex generates reward-related theta-band oscillatory activity.

When participants in a gambling game are given feedback as to whether they won or lost the previous bet, a series of stereotypical brain electrical responses can be observed in the electroencephalogram (EEG) and the stimulus-locked Event-Related Potential (ERP). These include the Feedback-Related Mediofrontal Negativity (FRN), a posterior P300, and a feedback-induced increase in power at the theta (4 to 8 Hz) band over frontal scalp. Although the generators of the FRN and P300 have been studied previously, little is known about the generator of feedback-induced theta. We employed a gambling game in which participants chose either high-risk/high-reward or low-risk/low-reward bets to investigate these feedback-related responses. The FRN was not modulated by the riskiness of the bet, but both P300 and feedback-induced theta were of greater amplitude following high- relative to low-risk bets. Using a bilateral multi-source Beamformer approach, we localized the induced theta-band responses following wins and losses to partially overlapping regions in the right medial frontal cortex, possibly including the Anterior Cingulate. Using a dipole-fitting approach, we found that the generators of feedback-induced theta are anatomically distinct from those of the FRN and P300.

Attention modulates responses to motion reversals in human visual cortex.

Selective attention modulates brain responses in visual cortex. A common finding, using functional magnetic resonance imaging or event-related potentials, is that responses to attended relative to unattended stimuli are potentiated. We report an exceptional circumstance in a motion-processing paradigm. Participants viewed superimposed stationary and moving dots and were instructed to attend to one or the other subset. Changes in the direction of dot motion triggered an event-related potential over posterior scalp sites, with a prominent negative peak at 200 ms that was larger when attention was directed at the stationary dots. This effect was localized to extrastriate visual cortex and may be due to reflexive effects of attention orienting triggered by unattended peripheral motion.

Early phase of spatial mismatch negativity is localized to a posterior "where" auditory pathway.

The auditory mismatch negativity (MMN) is an event-related potential that reflects early processing of changes in acoustic stimulus features. Although the MMN has been well characterized by previous work, the number, roles, and anatomical locations of its cortical generators remain unresolved. Here, we report that the MMN elicited by occasional deviations in sound location is comprised of two temporally and anatomically distinct phases: an early phase with a generator posterior to auditory cortex and contralateral to the deviant stimulus, and a later phase with generators that are more frontal and bilaterally symmetric. The posterior location of the early-phase generator suggests the engagement of neurons within a putative "where" pathway for processing spatial auditory information.