Comparison of electrical microstimulation artifact removal methods for high-channel-count prostheses.

BACKGROUND: Neuroprostheses are used to electrically stimulate the brain, modulate neural activity and restore sensory and motor function following injury or disease, such as blindness, paralysis, and other movement and psychiatric disorders. Recordings are often made simultaneously with stimulation, allowing the monitoring of neural signals and closed-loop control of devices. However, stimulation-evoked artifacts may obscure neural activity, particularly when stimulation and recording sites are nearby. Several methods have been developed to remove stimulation artifacts, but it remains challenging to validate and compare these methods because the 'ground-truth' of the neuronal signals may be contaminated by artifacts. NEW METHOD: Here, we delivered stimulation to the visual cortex via a high-channel-count prosthesis while recording neuronal activity and stimulation artifacts. We quantified the waveforms and temporal properties of stimulation artifacts from the cortical visual prosthesis (CVP) and used them to build a dataset, in which we simulated the neuronal activity and the stimulation artifacts. We illustrate how to use the simulated data to evaluate the performance of six software-based artifact removal methods (Template subtraction, Linear interpolation, Polynomial fitting, Exponential fitting, SALPA and ERAASR) in a CVP application scenario. RESULTS: We here focused on stimulation artifacts caused by electrical stimulation through a high-channel-count cortical prosthesis device. We find that the Polynomial fitting and Exponential fitting methods outperform the other methods in recovering spikes and multi-unit activity. Linear interpolation and Template subtraction recovered the local-field potentials. CONCLUSION: Polynomial fitting and Exponential fitting provided a good trade-off between the quality of the recovery of spikes and multi-unit activity (MUA) and the computational complexity for a cortical prosthesis.

Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain.

A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.

Recurrent neural networks that learn multi-step visual routines with reinforcement learning.

Many cognitive problems can be decomposed into series of subproblems that are solved sequentially by the brain. When subproblems are solved, relevant intermediate results need to be stored by neurons and propagated to the next subproblem, until the overarching goal has been completed. We will here consider visual tasks, which can be decomposed into sequences of elemental visual operations. Experimental evidence suggests that intermediate results of the elemental operations are stored in working memory as an enhancement of neural activity in the visual cortex. The focus of enhanced activity is then available for subsequent operations to act upon. The main question at stake is how the elemental operations and their sequencing can emerge in neural networks that are trained with only rewards, in a reinforcement learning setting. We here propose a new recurrent neural network architecture that can learn composite visual tasks that require the application of successive elemental operations. Specifically, we selected three tasks for which electrophysiological recordings of monkeys' visual cortex are available. To train the networks, we used RELEARNN, a biologically plausible four-factor Hebbian learning rule, which is local both in time and space. We report that networks learn elemental operations, such as contour grouping and visual search, and execute sequences of operations, solely based on the characteristics of the visual stimuli and the reward structure of a task. After training was completed, the activity of the units of the neural network elicited by behaviorally relevant image items was stronger than that elicited by irrelevant ones, just as has been observed in the visual cortex of monkeys solving the same tasks. Relevant information that needed to be exchanged between subroutines was maintained as a focus of enhanced activity and passed on to the subsequent subroutines. Our results demonstrate how a biologically plausible learning rule can train a recurrent neural network on multistep visual tasks.

An integrative, multiscale view on neural theories of consciousness.

How is conscious experience related to material brain processes? A variety of theories aiming to answer this age-old question have emerged from the recent surge in consciousness research, and some are now hotly debated. Although most researchers have so far focused on the development and validation of their preferred theory in relative isolation, this article, written by a group of scientists representing different theories, takes an alternative approach. Noting that various theories often try to explain different aspects or mechanistic levels of consciousness, we argue that the theories do not necessarily contradict each other. Instead, several of them may converge on fundamental neuronal mechanisms and be partly compatible and complementary, so that multiple theories can simultaneously contribute to our understanding. Here, we consider unifying, integration-oriented approaches that have so far been largely neglected, seeking to combine valuable elements from various theories.

Flexible Polymer Electrodes for Stable Prosthetic Visual Perception in Mice.

Brain interfaces that can stimulate neurons, cause minimal damage, and work for a long time will be central for future neuroprosthetics. Here, the long-term performance of highly flexible, thin polyimide shanks with several small (<15 µm) electrodes during electrical microstimulation of the visual cortex, is reported. The electrodes exhibit a remarkable stability when several billions of electrical pulses are applied in vitro. When the devices are implanted in the primary visual cortex (area V1) of mice and the animals are trained to detect electrical microstimulation, it is found that the perceptual thresholds are 2-20 microamperes (µA), which is far below the maximal currents that the electrodes can withstand. The long-term functionality of the devices in vivo is excellent, with stable performance for up to more than a year and little damage to the brain tissue. These results demonstrate the potential of thin floating electrodes for the long-term restoration of lost sensory functions.

Towards biologically plausible phosphene simulation for the differentiable optimization of visual cortical prostheses.

Blindness affects millions of people around the world. A promising solution to restoring a form of vision for some individuals are cortical visual prostheses, which bypass part of the impaired visual pathway by converting camera input to electrical stimulation of the visual system. The artificially induced visual percept (a pattern of localized light flashes, or 'phosphenes') has limited resolution, and a great portion of the field's research is devoted to optimizing the efficacy, efficiency, and practical usefulness of the encoding of visual information. A commonly exploited method is non-invasive functional evaluation in sighted subjects or with computational models by using simulated prosthetic vision (SPV) pipelines. An important challenge in this approach is to balance enhanced perceptual realism, biologically plausibility, and real-time performance in the simulation of cortical prosthetic vision. We present a biologically plausible, PyTorch-based phosphene simulator that can run in real-time and uses differentiable operations to allow for gradient-based computational optimization of phosphene encoding models. The simulator integrates a wide range of clinical results with neurophysiological evidence in humans and non-human primates. The pipeline includes a model of the retinotopic organization and cortical magnification of the visual cortex. Moreover, the quantitative effects of stimulation parameters and temporal dynamics on phosphene characteristics are incorporated. Our results demonstrate the simulator's suitability for both computational applications such as end-to-end deep learning-based prosthetic vision optimization as well as behavioral experiments. The modular and open-source software provides a flexible simulation framework for computational, clinical, and behavioral neuroscientists working on visual neuroprosthetics.

Involvement of superior colliculus in complex figure detection of mice.

Object detection is an essential function of the visual system. Although the visual cortex plays an important role in object detection, the superior colliculus can support detection when the visual cortex is ablated or silenced. Moreover, it has been shown that superficial layers of mouse SC (sSC) encode visual features of complex objects, and that this code is not inherited from the primary visual cortex. This suggests that mouse sSC may provide a significant contribution to complex object vision. Here, we use optogenetics to show that mouse sSC is involved in figure detection based on differences in figure contrast, orientation, and phase. Additionally, our neural recordings show that in mouse sSC, image elements that belong to a figure elicit stronger activity than those same elements when they are part of the background. The discriminability of this neural code is higher for correct trials than for incorrect trials. Our results provide new insight into the behavioral relevance of the visual processing that takes place in sSC.

Experience shapes chandelier cell function and structure in the visual cortex.

Detailed characterization of interneuron types in primary visual cortex (V1) has greatly contributed to understanding visual perception, yet the role of chandelier cells (ChCs) in visual processing remains poorly characterized. Using viral tracing we found that V1 ChCs predominantly receive monosynaptic input from local layer 5 pyramidal cells and higher-order cortical regions. Two-photon calcium imaging and convolutional neural network modeling revealed that ChCs are visually responsive but weakly selective for stimulus content. In mice running in a virtual tunnel, ChCs respond strongly to events known to elicit arousal, including locomotion and visuomotor mismatch. Repeated exposure of the mice to the virtual tunnel was accompanied by reduced visual responses of ChCs and structural plasticity of ChC boutons and axon initial segment length. Finally, ChCs only weakly inhibited pyramidal cells. These findings suggest that ChCs provide an arousal-related signal to layer 2/3 pyramidal cells that may modulate their activity and/or gate plasticity of their axon initial segments during behaviorally relevant events.

The representation of occluded image regions in area V1 of monkeys and humans.

Neuronal activity in the primary visual cortex (V1) is driven by feedforward input from within the neurons' receptive fields (RFs) and modulated by contextual information in regions surrounding the RF. The effect of contextual information on spiking activity occurs rapidly and is therefore challenging to dissociate from feedforward input. To address this challenge, we recorded the spiking activity of V1 neurons in monkeys viewing either natural scenes or scenes where the information in the RF was occluded, effectively removing the feedforward input. We found that V1 neurons responded rapidly and selectively to occluded scenes. V1 responses elicited by occluded stimuli could be used to decode individual scenes and could be predicted from those elicited by non-occluded images, indicating that there is an overlap between visually driven and contextual responses. We used representational similarity analysis to show that the structure of V1 representations of occluded scenes measured with electrophysiology in monkeys correlates strongly with the representations of the same scenes in humans measured with functional magnetic resonance imaging (fMRI). Our results reveal that contextual influences rapidly alter V1 spiking activity in monkeys over distances of several degrees in the visual field, carry information about individual scenes, and resemble those in human V1. VIDEO ABSTRACT.

Chronic stability of a neuroprosthesis comprising multiple adjacent Utah arrays in monkeys.

Objective. Electrical stimulation of visual cortex via a neuroprosthesis induces the perception of dots of light ('phosphenes'), potentially allowing recognition of simple shapes even after decades of blindness. However, restoration of functional vision requires large numbers of electrodes, and chronic, clinical implantation of intracortical electrodes in the visual cortex has only been achieved using devices of up to 96 channels. We evaluated the efficacy and stability of a 1024-channel neuroprosthesis system in non-human primates (NHPs) over more than 3 years to assess its suitability for long-term vision restoration.Approach.We implanted 16 microelectrode arrays (Utah arrays) consisting of 8 × 8 electrodes with iridium oxide tips in the primary visual cortex (V1) and visual area 4 (V4) of two sighted macaques. We monitored the animals' health and measured electrode impedances and neuronal signal quality by calculating signal-to-noise ratios of visually driven neuronal activity, peak-to-peak voltages of the waveforms of action potentials, and the number of channels with high-amplitude signals. We delivered cortical microstimulation and determined the minimum current that could be perceived, monitoring the number of channels that successfully yielded phosphenes. We also examined the influence of the implant on a visual task after 2-3 years of implantation and determined the integrity of the brain tissue with a histological analysis 3-3.5 years post-implantation.Main results. The monkeys remained healthy throughout the implantation period and the device retained its mechanical integrity and electrical conductivity. However, we observed decreasing signal quality with time, declining numbers of phosphene-evoking electrodes, decreases in electrode impedances, and impaired performance on a visual task at visual field locations corresponding to implanted cortical regions. Current thresholds increased with time in one of the two animals. The histological analysis revealed encapsulation of arrays and cortical degeneration. Scanning electron microscopy on one array revealed degradation of IrOxcoating and higher impedances for electrodes with broken tips.Significance. Long-term implantation of a high-channel-count device in NHP visual cortex was accompanied by deformation of cortical tissue and decreased stimulation efficacy and signal quality over time. We conclude that improvements in device biocompatibility and/or refinement of implantation techniques are needed before future clinical use is feasible.

Solving the binding problem: Assemblies form when neurons enhance their firing rate-they don't need to oscillate or synchronize.

When we look at an image, its features are represented in our visual system in a highly distributed manner, calling for a mechanism that binds them into coherent object representations. There have been different proposals for the neuronal mechanisms that can mediate binding. One hypothesis is that binding is achieved by oscillations that synchronize neurons representing features of the same perceptual object. This view allows separate communication channels between different brain areas. Another hypothesis is that binding of features that are represented in different brain regions occurs when the neurons in these areas that respond to the same object simultaneously enhance their firing rate, which would correspond to directing object-based attention to these features. This review summarizes evidence in favor of and against these two hypotheses, examining the neuronal correlates of binding and assessing the time course of perceptual grouping. I conclude that enhanced neuronal firing rates bind features into coherent object representations, whereas oscillations and synchrony are unrelated to binding.

Inversion of pop-out for a distracting feature dimension in monkey visual cortex.

During visual search, it is important to reduce the interference of distracting objects in the scene. The neuronal responses elicited by the search target stimulus are typically enhanced. However, it is equally important to suppress the representations of distracting stimuli, especially if they are salient and capture attention. We trained monkeys to make an eye movement to a unique "pop-out" shape stimulus among an array of distracting stimuli. One of these distractors had a salient color that varied across trials and differed from the color of the other stimuli, causing it to also pop-out. The monkeys were able to select the pop-out shape target with high accuracy and actively avoided the pop-out color distractor. This behavioral pattern was reflected in the activity of neurons in area V4. Responses to the shape targets were enhanced, while the activity evoked by the pop-out color distractor was only briefly enhanced, directly followed by a sustained period of pronounced suppression. These behavioral and neuronal results demonstrate a cortical selection mechanism that rapidly inverts a pop-out signal to "pop-in" for an entire feature dimension thereby facilitating goal-directed visual search in the presence of salient distractors.

Contextual drive of neuronal responses in mouse V1 in the absence of feedforward input.

Neurons in the primary visual cortex (V1) respond to stimuli in their receptive field (RF), which is defined by the feedforward input from the retina. However, V1 neurons are also sensitive to contextual information outside their RF, even if the RF itself is unstimulated. Here, we examined the cortical circuits for V1 contextual responses to gray disks superimposed on different backgrounds. Contextual responses began late and were strongest in the feedback-recipient layers of V1. They differed between the three main classes of inhibitory neurons, with particularly strong contextual drive of VIP neurons, indicating a contribution of disinhibitory circuits to contextual drive. Contextual drive was strongest when the gray disk was perceived as figure, occluding its background, rather than a hole. Our results link contextual drive in V1 to perceptual organization and provide previously unknown insight into how recurrent processing shapes the response of sensory neurons to facilitate figure perception.

Purkinje Cell Activity in the Medial and Lateral Cerebellum During Suppression of Voluntary Eye Movements in Rhesus Macaques.

Volitional suppression of responses to distracting external stimuli enables us to achieve our goals. This volitional inhibition of a specific behavior is supposed to be mainly mediated by the cerebral cortex. However, recent evidence supports the involvement of the cerebellum in this process. It is currently not known whether different parts of the cerebellar cortex play differential or synergistic roles in the planning and execution of this behavior. Here, we measured Purkinje cell (PC) responses in the medial and lateral cerebellum in two rhesus macaques during pro- and anti-saccade tasks. During an antisaccade trial, non-human primates (NHPs) were instructed to make a saccadic eye movement away from a target, rather than toward it, as in prosaccade trials. Our data show that the cerebellum plays an important role not only during the execution of the saccades but also during the volitional inhibition of eye movements toward the target. Simple spike (SS) modulation during the instruction and execution periods of pro- and anti-saccades was prominent in PCs of both the medial and lateral cerebellum. However, only the SS activity in the lateral cerebellar cortex contained information about stimulus identity and showed a strong reciprocal interaction with complex spikes (CSs). Moreover, the SS activity of different PC groups modulated bidirectionally in both of regions, but the PCs that showed facilitating and suppressive activity were predominantly associated with instruction and execution, respectively. These findings show that different cerebellar regions and PC groups contribute to goal-directed behavior and volitional inhibition, but with different propensities, highlighting the rich repertoire of the cerebellar control in executive functions.

1024-channel electrophysiological recordings in macaque V1 and V4 during resting state.

Co-variations in resting state activity are thought to arise from a variety of correlated inputs to neurons, such as bottom-up activity from lower areas, feedback from higher areas, recurrent processing in local circuits, and fluctuations in neuromodulatory systems. Most studies have examined resting state activity throughout the brain using MRI scans, or observed local co-variations in activity by recording from a small number of electrodes. We carried out electrophysiological recordings from over a thousand chronically implanted electrodes in the visual cortex of non-human primates, yielding a resting state dataset with unprecedentedly high channel counts and spatiotemporal resolution. Such signals could be used to observe brain waves across larger regions of cortex, offering a temporally detailed picture of brain activity. In this paper, we provide the dataset, describe the raw and processed data formats and data acquisition methods, and indicate how the data can be used to yield new insights into the 'background' activity that influences the processing of visual information in our brain.

Population receptive fields in nonhuman primates from whole-brain fMRI and large-scale neurophysiology in visual cortex.

Population receptive field (pRF) modeling is a popular fMRI method to map the retinotopic organization of the human brain. While fMRI-based pRF maps are qualitatively similar to invasively recorded single-cell receptive fields in animals, it remains unclear what neuronal signal they represent. We addressed this question in awake nonhuman primates comparing whole-brain fMRI and large-scale neurophysiological recordings in areas V1 and V4 of the visual cortex. We examined the fits of several pRF models based on the fMRI blood-oxygen-level-dependent (BOLD) signal, multi-unit spiking activity (MUA), and local field potential (LFP) power in different frequency bands. We found that pRFs derived from BOLD-fMRI were most similar to MUA-pRFs in V1 and V4, while pRFs based on LFP gamma power also gave a good approximation. fMRI-based pRFs thus reliably reflect neuronal receptive field properties in the primate brain. In addition to our results in V1 and V4, the whole-brain fMRI measurements revealed retinotopic tuning in many other cortical and subcortical areas with a consistent increase in pRF size with increasing eccentricity, as well as a retinotopically specific deactivation of default mode network nodes similar to previous observations in humans.

A neuronal basis of iconic memory in macaque primary visual cortex.

After a briefly presented visual stimulus disappears, observers retain a detailed representation of this stimulus for a short period of time. This sensory storage is called iconic memory. We measured iconic memory in the perception of monkeys and its neuronal correlates in the primary visual cortex (area V1). We determined how many milliseconds extra viewing time iconic memory is worth and how it decays by varying the duration of a brief stimulus and the timing of a mask. The V1 activity that persists after the disappearance of a stimulus predicted accuracy, with a time course resembling the worth and decay of iconic memory. Finally, we examined how iconic memory interacts with attention. A cue presented after the stimulus disappears boosts attentional influences pertaining to a relevant part of the stimulus but only if it appears before iconic memory decayed. Our results relate iconic memory to neuronal activity in early visual cortex.

Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex.

BACKGROUNDA long-held goal of vision therapy is to transfer information directly to the visual cortex of blind individuals, thereby restoring a rudimentary form of sight. However, no clinically available cortical visual prosthesis yet exists.METHODSWe implanted an intracortical microelectrode array consisting of 96 electrodes in the visual cortex of a 57-year-old person with complete blindness for a 6-month period. We measured thresholds and the characteristics of the visual percepts elicited by intracortical microstimulation.RESULTSImplantation and subsequent explantation of intracortical microelectrodes were carried out without complications. The mean stimulation threshold for single electrodes was 66.8 ± 36.5 µA. We consistently obtained high-quality recordings from visually deprived neurons and the stimulation parameters remained stable over time. Simultaneous stimulation via multiple electrodes was associated with a significant reduction in thresholds (P < 0.001, ANOVA) and evoked discriminable phosphene percepts, allowing the blind participant to identify some letters and recognize object boundaries.CONCLUSIONSOur results demonstrate the safety and efficacy of chronic intracortical microstimulation via a large number of electrodes in human visual cortex, showing its high potential for restoring functional vision in the blind.TRIAL REGISTRATIONClinicalTrials.gov identifier NCT02983370.FUNDINGThe Spanish Ministerio de Ciencia Innovación y Universidades, the Generalitat Valenciana (Spain), the Europan Union's Horizon 2020 programme, the Bidons Egara Research Chair of the University Miguel Hernández (Spain), and the John Moran Eye Center of the University of Utah.

Theta-phase dependent neuronal coding during sequence learning in human single neurons.

The ability to maintain a sequence of items in memory is a fundamental cognitive function. In the rodent hippocampus, the representation of sequentially organized spatial locations is reflected by the phase of action potentials relative to the theta oscillation (phase precession). We investigated whether the timing of neuronal activity relative to the theta brain oscillation also reflects sequence order in the medial temporal lobe of humans. We used a task in which human participants learned a fixed sequence of pictures and recorded single neuron and local field potential activity with implanted electrodes. We report that spikes for three consecutive items in the sequence (the preferred stimulus for each cell, as well as the stimuli immediately preceding and following it) were phase-locked at distinct phases of the theta oscillation. Consistent with phase precession, spikes were fired at progressively earlier phases as the sequence advanced. These findings generalize previous findings in the rodent hippocampus to the human temporal lobe and suggest that encoding stimulus information at distinct oscillatory phases may play a role in maintaining sequential order in memory.

A Direct Comparison of Spatial Attention and Stimulus-Response Compatibility between Mice and Humans.

Mice are becoming an increasingly popular model for investigating the neural substrates of visual processing and higher cognitive functions. To validate the translation of mouse visual attention and sensorimotor processing to humans, we compared their performance in the same visual task. Mice and human participants judged the orientation of a grating presented on either the right or left side in the visual field. To induce shifts of spatial attention, we varied the stimulus probability on each side. As expected, human participants showed faster RTs and a higher accuracy for the side with a higher probability, a well-established effect of visual attention. The attentional effect was only present in mice when their response was slow. Although the task demanded a judgment of grating orientation, the accuracy of the mice was strongly affected by whether the side of the stimulus corresponded to the side of the behavioral response. This stimulus-response compatibility (Simon) effect was much weaker in humans and only significant for their fastest responses. Both species exhibited a speed-accuracy trade-off in their responses, because slower responses were more accurate than faster responses. We found that mice typically respond very fast, which contributes to the stronger stimulus-response compatibility and weaker attentional effects, which were only apparent in the trials with slowest responses. Humans responded slower and had stronger attentional effects, combined with a weak influence of stimulus-response compatibility, which was only apparent in trials with fast responses. We conclude that spatial attention and stimulus-response compatibility influence the responses of humans and mice but that strategy differences between species determine the dominance of these effects.