The noisy encoding of disparity model predicts perception of the McGurk effect in native Japanese speakers.

In the McGurk effect, visual speech from the face of the talker alters the perception of auditory speech. The diversity of human languages has prompted many intercultural studies of the effect in both Western and non-Western cultures, including native Japanese speakers. Studies of large samples of native English speakers have shown that the McGurk effect is characterized by high variability in the susceptibility of different individuals to the illusion and in the strength of different experimental stimuli to induce the illusion. The noisy encoding of disparity (NED) model of the McGurk effect uses principles from Bayesian causal inference to account for this variability, separately estimating the susceptibility and sensory noise for each individual and the strength of each stimulus. To determine whether variation in McGurk perception is similar between Western and non-Western cultures, we applied the NED model to data collected from 80 native Japanese-speaking participants. Fifteen different McGurk stimuli that varied in syllable content (unvoiced auditory "pa" + visual "ka" or voiced auditory "ba" + visual "ga") were presented interleaved with audiovisual congruent stimuli. The McGurk effect was highly variable across stimuli and participants, with the percentage of illusory fusion responses ranging from 3 to 78% across stimuli and from 0 to 91% across participants. Despite this variability, the NED model accurately predicted perception, predicting fusion rates for individual stimuli with 2.1% error and for individual participants with 2.4% error. Stimuli containing the unvoiced pa/ka pairing evoked more fusion responses than the voiced ba/ga pairing. Model estimates of sensory noise were correlated with participant age, with greater sensory noise in older participants. The NED model of the McGurk effect offers a principled way to account for individual and stimulus differences when examining the McGurk effect in different cultures.

Synthetic faces generated with the facial action coding system or deep neural networks improve speech-in-noise perception, but not as much as real faces.

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking faces generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the real vs. synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces.

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System.

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking face generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the real vs. synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces, providing a roadmap for improving communication in this rapidly developing domain.

Benefits of sharing neurophysiology data from the BRAIN Initiative Research Opportunities in Humans Consortium.

Sharing human brain data can yield scientific benefits, but because of various disincentives, only a fraction of these data is currently shared. We profile three successful data-sharing experiences from the NIH BRAIN Initiative Research Opportunities in Humans (ROH) Consortium and demonstrate benefits to data producers and to users.

YAEL: Your Advanced Electrode Localizer.

Intracranial electroencephalography (iEEG) provides a unique opportunity to record and stimulate neuronal populations in the human brain. A key step in neuroscience inference from iEEG is localizing the electrodes relative to individual subject anatomy and identified regions in brain atlases. We describe a new software tool, Your Advanced Electrode Localizer (YAEL), that provides an integrated solution for every step of the electrode localization process. YAEL is compatible with all common data formats to provide an easy-to-use, drop-in replacement for problematic existing workflows that require users to grapple with multiple programs and interfaces. YAEL's automatic extrapolation and interpolation functions speed localization, especially important in patients with many implanted stereotactic (sEEG) electrode shafts. The graphical user interface is presented in a web browser for broad compatibility and includes an interactive 3D viewer for easier localization of nearby sEEG contacts. After localization is complete, users may enter or import data into YAEL's 3D viewer to create publication-ready visualizations of electrodes and brain anatomy, including identified brain areas from atlases; the response to experimental tasks measured with iEEG; and clinical measures such as epileptiform activity or the results of electrical stimulation mapping. YAEL is free and open source and does not depend on any commercial software. Installation instructions for Mac, Windows, and Linux are available at https://yael.wiki.

A comparison of neuroelectrophysiology databases.

As data sharing has become more prevalent, three pillars - archives, standards, and analysis tools - have emerged as critical components in facilitating effective data sharing and collaboration. This paper compares four freely available intracranial neuroelectrophysiology data repositories: Data Archive for the BRAIN Initiative (DABI), Distributed Archives for Neurophysiology Data Integration (DANDI), OpenNeuro, and Brain-CODE. The aim of this review is to describe archives that provide researchers with tools to store, share, and reanalyze both human and non-human neurophysiology data based on criteria that are of interest to the neuroscientific community. The Brain Imaging Data Structure (BIDS) and Neurodata Without Borders (NWB) are utilized by these archives to make data more accessible to researchers by implementing a common standard. As the necessity for integrating large-scale analysis into data repository platforms continues to grow within the neuroscientific community, this article will highlight the various analytical and customizable tools developed within the chosen archives that may advance the field of neuroinformatics.

A Comparison of Neuroelectrophysiology Databases.

As data sharing has become more prevalent, three pillars - archives, standards, and analysis tools - have emerged as critical components in facilitating effective data sharing and collaboration. This paper compares four freely available intracranial neuroelectrophysiology data repositories: Data Archive for the BRAIN Initiative (DABI), Distributed Archives for Neurophysiology Data Integration (DANDI), OpenNeuro, and Brain-CODE. The aim of this review is to describe archives that provide researchers with tools to store, share, and reanalyze both human and non-human neurophysiology data based on criteria that are of interest to the neuroscientific community. The Brain Imaging Data Structure (BIDS) and Neurodata Without Borders (NWB) are utilized by these archives to make data more accessible to researchers by implementing a common standard. As the necessity for integrating large-scale analysis into data repository platforms continues to grow within the neuroscientific community, this article will highlight the various analytical and customizable tools developed within the chosen archives that may advance the field of neuroinformatics.

Multivariate fMRI responses in superior temporal cortex predict visual contributions to, and individual differences in, the intelligibility of noisy speech.

Humans have the unique ability to decode the rapid stream of language elements that constitute speech, even when it is contaminated by noise. Two reliable observations about noisy speech perception are that seeing the face of the talker improves intelligibility and the existence of individual differences in the ability to perceive noisy speech. We introduce a multivariate BOLD fMRI measure that explains both observations. In two independent fMRI studies, clear and noisy speech was presented in visual, auditory and audiovisual formats to thirty-seven participants who rated intelligibility. An event-related design was used to sort noisy speech trials by their intelligibility. Individual-differences multidimensional scaling was applied to fMRI response patterns in superior temporal cortex and the dissimilarity between responses to clear speech and noisy (but intelligible) speech was measured. Neural dissimilarity was less for audiovisual speech than auditory-only speech, corresponding to the greater intelligibility of noisy audiovisual speech. Dissimilarity was less in participants with better noisy speech perception, corresponding to individual differences. These relationships held for both single word and entire sentence stimuli, suggesting that they were driven by intelligibility rather than the specific stimuli tested. A neural measure of perceptual intelligibility may aid in the development of strategies for helping those with impaired speech perception.

Stability of Ti3C2Tx MXene Films and Devices under Clinical Sterilization Processes.

MXenes are being heavily investigated in biomedical research, with applications ranging from regenerative medicine to bioelectronics. To enable the adoption and integration of MXenes into therapeutic platforms and devices, however, their stability under standard sterilization procedures must be established. Here, we present a comprehensive investigation of the electrical, chemical, structural, and mechanical effects of common thermal (autoclave) and chemical (ethylene oxide (EtO) and H2O2 gas plasma) sterilization protocols on both thin-film Ti3C2Tx MXene microelectrodes and mesoscale arrays made from Ti3C2Tx-infused cellulose-elastomer composites. We also evaluate the effectiveness of the sterilization processes in eliminating all pathogens from the Ti3C2Tx films and composites. Post-sterilization analysis revealed that autoclave and EtO did not alter the DC conductivity, electrochemical impedance, surface morphology, or crystallographic structure of Ti3C2Tx and were both effective at eliminating E. coli from both types of Ti3C2Tx-based devices. On the other end, exposure to H2O2 gas plasma sterilization for 45 min induced severe degradation of the structure and properties of Ti3C2Tx films and composites. The stability of the Ti3C2Tx after EtO and autoclave sterilization and the complete removal of pathogens establish the viability of both sterilization processes for Ti3C2Tx-based technologies.

Functional group bridge for simultaneous regression and support estimation.

This paper is motivated by studying differential brain activities to multiple experimental condition presentations in intracranial electroencephalography (iEEG) experiments. Contrasting effects of experimental conditions are often zero in most regions and nonzero in some local regions, yielding locally sparse functions. Such studies are essentially a function-on-scalar regression problem, with interest being focused not only on estimating nonparametric functions but also on recovering the function supports. We propose a weighted group bridge approach for simultaneous function estimation and support recovery in function-on-scalar mixed effect models, while accounting for heterogeneity present in functional data. We use B-splines to transform sparsity of functions to its sparse vector counterpart of increasing dimension, and propose a fast nonconvex optimization algorithm using nested alternative direction method of multipliers (ADMM) for estimation. Large sample properties are established. In particular, we show that the estimated coefficient functions are rate optimal in the minimax sense under the L2 norm and resemble a phase transition phenomenon. For support estimation, we derive a convergence rate under the L ∞ $L_{\infty }$ norm that leads to a selection consistency property under δ-sparsity, and obtain a result under strict sparsity using a simple sufficient regularity condition. An adjusted extended Bayesian information criterion is proposed for parameter tuning. The developed method is illustrated through simulations and an application to a novel iEEG data set to study multisensory integration.

Percepts evoked by multi-electrode stimulation of human visual cortex.

BACKGROUND: Direct electrical stimulation of early visual cortex evokes the perception of small spots of light known as phosphenes. Previous studies have examined the location, size, and brightness of phosphenes evoked by stimulation of single electrodes. While it has been envisioned that concurrent stimulation of many electrodes could be used as the basis for a visual cortical prosthesis, the percepts resulting from multi-electrode stimulation have not been fully characterized. OBJECTIVE: To understand the rules governing perception of phosphenes evoked by multi-electrode stimulation of visual cortex. METHODS: Multi-electrode stimulation was conducted in human epilepsy patients. We examined the number and spatial arrangement of phosphenes evoked by stimulation of individual multi-electrode groups (n = 8), and the ability of subjects to discriminate between the pattern of phosphenes generated by stimulation of different multi-electrode groups (n = 7). RESULTS: Simultaneous stimulation of pairs of electrodes separated by greater than 4 mm tended to produce perception of two distinct phosphenes. Simultaneous stimulation of three electrodes gave rise to a consistent spatial pattern of phosphenes, but with significant variation in the absolute location, size, and orientation of that pattern perceived on each trial. Although multi-electrode stimulation did not produce perception of recognizable forms, subjects could use the pattern of phosphenes evoked by stimulation to perform simple discriminations. CONCLUSIONS: The number of phosphenes produced by multi-electrode stimulation can be predicted using a model for spread of activity in early visual cortex, but there are additional subtle effects that must be accounted for.

Ethical commitments, principles, and practices guiding intracranial neuroscientific research in humans.

Leveraging firsthand experience, BRAIN-funded investigators conducting intracranial human neuroscience research propose two fundamental ethical commitments: (1) maintaining the integrity of clinical care and (2) ensuring voluntariness. Principles, practices, and uncertainties related to these commitments are offered for future investigation.

Biased Orientation and Color Tuning of the Human Visual Gamma Rhythm.

Narrowband γ oscillations (NBG: ∼20-60 Hz) in visual cortex reflect rhythmic fluctuations in population activity generated by underlying circuits tuned for stimulus location, orientation, and color. A variety of theories posit a specific role for NBG in encoding and communicating this information within visual cortex. However, recent findings suggest a more nuanced role for NBG, given its dependence on certain stimulus feature configurations, such as coherent-oriented edges and specific hues. Motivated by these factors, we sought to quantify the independent and joint tuning properties of NBG to oriented and color stimuli using intracranial recordings from the human visual cortex (male and female). NBG was shown to display a cardinal orientation bias (horizontal) and also an end- and mid-spectral color bias (red/blue and green). When jointly probed, the cardinal bias for orientation was attenuated and an end-spectral preference for red and blue predominated. This loss of mid-spectral tuning occurred even for recording sites showing large responses to uniform green stimuli. Our results demonstrate the close, yet complex, link between the population dynamics driving NBG oscillations and known feature selectivity biases for orientation and color within visual cortex. Such a bias in stimulus tuning imposes new constraints on the functional significance of the visual γ rhythm. More generally, these biases in population electrophysiology will need to be considered in experiments using orientation or color features to examine the role of visual cortex in other domains, such as working memory and decision-making.SIGNIFICANCE STATEMENT Oscillations in electrophysiological activity occur in visual cortex in response to stimuli that strongly drive the orientation or color selectivity of visual neurons. The significance of this induced "γ rhythm" to brain function remains unclear. Answering this question requires understanding how and why some stimuli can reliably generate oscillatory γ activity while others do not. We examined how different orientations and colors independently and jointly modulate γ oscillations in the human brain. Our data show that γ oscillations are greatest for certain orientations and colors that reflect known response biases in visual cortex. Such findings complicate the functional significance of γ oscillations but open new avenues for linking circuits to population dynamics in visual cortex.

3D-printed microchip electrophoresis device containing spiral electrodes for integrated capacitively coupled contactless conductivity detection.

In this work, we demonstrate for the first time the design and fabrication of microchip electrophoresis devices containing cross-shaped channels and spiral electrodes around the separation channel for microchip electrophoresis and capacitively coupled contactless conductivity detection. The whole device was prepared in a digital light processing-based 3D printer in poly(ethylene glycol) diacrylate resin. Outstanding X-Y resolution of the customized 3D printer ensured the fabrication of 40-µm cross section channels. The spiral channels were filled with melted gallium to form conductive electrodes around the separation channel. We demonstrate the applicability of the device on the separation of sodium, potassium, and lithium cations by microchip electrophoresis. Graphical abstract.

Intelligibility of audiovisual sentences drives multivoxel response patterns in human superior temporal cortex.

Regions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech, and neural responses in pSTG/S may underlie the perceptual benefit of visual speech for the comprehension of noisy auditory speech. We examined this possibility through the lens of multivoxel pattern responses in pSTG/S. BOLD fMRI data was collected from 22 participants presented with speech consisting of English sentences presented in five different formats: visual-only; auditory with and without added auditory noise; and audiovisual with and without auditory noise. Participants reported the intelligibility of each sentence with a button press and trials were sorted post-hoc into those that were more or less intelligible. Response patterns were measured in regions of the pSTG/S identified with an independent localizer. Noisy audiovisual sentences with very similar physical properties evoked very different response patterns depending on their intelligibility. When a noisy audiovisual sentence was reported as intelligible, the pattern was nearly identical to that elicited by clear audiovisual sentences. In contrast, an unintelligible noisy audiovisual sentence evoked a pattern like that of visual-only sentences. This effect was less pronounced for noisy auditory-only sentences, which evoked similar response patterns regardless of intelligibility. The successful integration of visual and auditory speech produces a characteristic neural signature in pSTG/S, highlighting the importance of this region in generating the perceptual benefit of visual speech.

Raising the stakes for cortical visual prostheses.

In this issue of the JCI, the dream of restoring useful vision to blind individuals with neurotechnology moves one step closer to realization. Fernández et al. implanted an electrode array with 96 penetrating electrodes in the visual cortex of a blind patient who had been without light perception for 16 years due to optic neuropathy. Remarkably, the patient was able to perceive visual patterns created by passing current through array electrodes. The use of a penetrating electrode array meant that action potentials from single neurons could be recorded to study the neural response to stimulation. Compared with electrodes resting on the cortical surface, penetrating electrodes require one-tenth the current to create a visual percept. However, patterned electrical stimulation often fails to produce the expected percept for penetrating and surface electrode arrays, highlighting the need for further research to untangle the relationship between stimulus and perception.

Multi-electrode stimulation evokes consistent spatial patterns of phosphenes and improves phosphene mapping in blind subjects.

BACKGROUND: Visual cortical prostheses (VCPs) have the potential to restore visual function to patients with acquired blindness. Successful implementation of VCPs requires the ability to reliably map the location of the phosphene produced by stimulation of each implanted electrode. OBJECTIVE: To evaluate the efficacy of different approaches to phosphene mapping and propose simple improvements to mapping strategy. METHODS: We stimulated electrodes implanted in the visual cortex of five blind and fifteen sighted patients. We tested two fixation strategies, unimanual fixation, where subjects placed a single index finger on a tactile fixation point and bimanual fixation, where subjects overlaid their right index finger over their left on the tactile point. In addition, we compared absolute mapping in which a single electrode was stimulated on each trial, and relative mapping with sequences containing stimulation of three to five phosphenes on each trial. Trial-to-trial variability present in relative mapping sequences was quantified. RESULTS: Phosphene mapping was less precise in blind subjects than in sighted subjects (2DRMS, 16 ± 2.9° vs. 1.9 ± 0.93°; t (18) = 18, p = <0.001). Within blind subjects, bimanual fixation resulted in more consistent phosphene localization than unimanual fixation (BS1: 4.0 ± 2.6° vs. 19 ± 4.7°, t (79) = 24, p < 0.001; BS2 4.1 ± 2.0° vs. 12 ± 2.7°, t (65) = 19, p < 0.001). Multi-point relative mapping had similar baseline precision to absolute mapping (BS1: 4.7 ± 2.6° vs. 3.9 ± 2.0°; BS2: 4.1 ± 2.0° vs. 3.2 ± 1.1°) but improved significantly when trial-to-trial translational variability was removed. Although multi-point mapping methods did reveal more of the functional organization expected in early visual cortex, subjects tended to artificially regularize the spacing between phosphenes. We attempt to address this issue by fitting a standard logarithmic map to relative multi-point sequences. CONCLUSIONS: Relative mapping methods, combined with bimanual fixation, resulted in the most precise estimates of phosphene organization. These techniques, combined with use of a standard logarithmic model of visual cortex, may provide a practical way to improve the implementation of a VCP.

Face and voice perception: Monkey see, monkey hear.

Primate brains contain specialized areas for perceiving social cues. New research shows that only some of these areas integrate visual faces with auditory voices.

Weak observer-level correlation and strong stimulus-level correlation between the McGurk effect and audiovisual speech-in-noise: A causal inference explanation.

The McGurk effect is a widely used measure of multisensory integration during speech perception. Two observations have raised questions about the validity of the effect as a tool for understanding speech perception. First, there is high variability in perception of the McGurk effect across different stimuli and observers. Second, across observers there is low correlation between McGurk susceptibility and recognition of visual speech paired with auditory speech-in-noise, another common measure of multisensory integration. Using the framework of the causal inference of multisensory speech (CIMS) model, we explored the relationship between the McGurk effect, syllable perception, and sentence perception in seven experiments with a total of 296 different participants. Perceptual reports revealed a relationship between the efficacy of different McGurk stimuli created from the same talker and perception of the auditory component of the McGurk stimuli presented in isolation, both with and without added noise. The CIMS model explained this strong stimulus-level correlation using the principles of noisy sensory encoding followed by optimal cue combination within a common representational space across speech types. Because the McGurk effect (but not speech-in-noise) requires the resolution of conflicting cues between modalities, there is an additional source of individual variability that can explain the weak observer-level correlation between McGurk and noisy speech. Power calculations show that detecting this weak correlation requires studies with many more participants than those conducted to-date. Perception of the McGurk effect and other types of speech can be explained by a common theoretical framework that includes causal inference, suggesting that the McGurk effect is a valid and useful experimental tool.