Subtle adversarial image manipulations influence both human and machine perception.

Although artificial neural networks (ANNs) were inspired by the brain, ANNs exhibit a brittleness not generally observed in human perception. One shortcoming of ANNs is their susceptibility to adversarial perturbations-subtle modulations of natural images that result in changes to classification decisions, such as confidently mislabelling an image of an elephant, initially classified correctly, as a clock. In contrast, a human observer might well dismiss the perturbations as an innocuous imaging artifact. This phenomenon may point to a fundamental difference between human and machine perception, but it drives one to ask whether human sensitivity to adversarial perturbations might be revealed with appropriate behavioral measures. Here, we find that adversarial perturbations that fool ANNs similarly bias human choice. We further show that the effect is more likely driven by higher-order statistics of natural images to which both humans and ANNs are sensitive, rather than by the detailed architecture of the ANN.

Individual variability of neural computations in the primate retina.

Variation in the neural code contributes to making each individual unique. We probed neural code variation using ∼100 population recordings from major ganglion cell types in the macaque retina, combined with an interpretable computational representation of individual variability. This representation captured variation and covariation in properties such as nonlinearity, temporal dynamics, and spatial receptive field size and preserved invariances such as asymmetries between On and Off cells. The covariation of response properties in different cell types was associated with the proximity of lamination of their synaptic input. Surprisingly, male retinas exhibited higher firing rates and faster temporal integration than female retinas. Exploiting data from previously recorded retinas enabled efficient characterization of a new macaque retina, and of a human retina. Simulations indicated that combining a large dataset of retinal recordings with behavioral feedback could reveal the neural code in a living human and thus improve vision restoration with retinal implants.

A model-based spike sorting algorithm for removing correlation artifacts in multi-neuron recordings.

We examine the problem of estimating the spike trains of multiple neurons from voltage traces recorded on one or more extracellular electrodes. Traditional spike-sorting methods rely on thresholding or clustering of recorded signals to identify spikes. While these methods can detect a large fraction of the spikes from a recording, they generally fail to identify synchronous or near-synchronous spikes: cases in which multiple spikes overlap. Here we investigate the geometry of failures in traditional sorting algorithms, and document the prevalence of such errors in multi-electrode recordings from primate retina. We then develop a method for multi-neuron spike sorting using a model that explicitly accounts for the superposition of spike waveforms. We model the recorded voltage traces as a linear combination of spike waveforms plus a stochastic background component of correlated Gaussian noise. Combining this measurement model with a Bernoulli prior over binary spike trains yields a posterior distribution for spikes given the recorded data. We introduce a greedy algorithm to maximize this posterior that we call "binary pursuit". The algorithm allows modest variability in spike waveforms and recovers spike times with higher precision than the voltage sampling rate. This method substantially corrects cross-correlation artifacts that arise with conventional methods, and substantially outperforms clustering methods on both real and simulated data. Finally, we develop diagnostic tools that can be used to assess errors in spike sorting in the absence of ground truth.

Efficient coding of spatial information in the primate retina.

Sensory neurons have been hypothesized to efficiently encode signals from the natural environment subject to resource constraints. The predictions of this efficient coding hypothesis regarding the spatial filtering properties of the visual system have been found consistent with human perception, but they have not been compared directly with neural responses. Here, we analyze the information that retinal ganglion cells transmit to the brain about the spatial information in natural images subject to three resource constraints: the number of retinal ganglion cells, their total response variances, and their total synaptic strengths. We derive a model that optimizes the transmitted information and compare it directly with measurements of complete functional connectivity between cone photoreceptors and the four major types of ganglion cells in the primate retina, obtained at single-cell resolution. We find that the ganglion cell population exhibited 80% efficiency in transmitting spatial information relative to the model. Both the retina and the model exhibited high redundancy (~30%) among ganglion cells of the same cell type. A novel and unique prediction of efficient coding, the relationships between projection patterns of individual cones to all ganglion cells, was consistent with the observed projection patterns in the retina. These results indicate a high level of efficiency with near-optimal redundancy in visual signaling by the retina.

Modeling the impact of common noise inputs on the network activity of retinal ganglion cells.

Synchronized spontaneous firing among retinal ganglion cells (RGCs), on timescales faster than visual responses, has been reported in many studies. Two candidate mechanisms of synchronized firing include direct coupling and shared noisy inputs. In neighboring parasol cells of primate retina, which exhibit rapid synchronized firing that has been studied extensively, recent experimental work indicates that direct electrical or synaptic coupling is weak, but shared synaptic input in the absence of modulated stimuli is strong. However, previous modeling efforts have not accounted for this aspect of firing in the parasol cell population. Here we develop a new model that incorporates the effects of common noise, and apply it to analyze the light responses and synchronized firing of a large, densely-sampled network of over 250 simultaneously recorded parasol cells. We use a generalized linear model in which the spike rate in each cell is determined by the linear combination of the spatio-temporally filtered visual input, the temporally filtered prior spikes of that cell, and unobserved sources representing common noise. The model accurately captures the statistical structure of the spike trains and the encoding of the visual stimulus, without the direct coupling assumption present in previous modeling work. Finally, we examined the problem of decoding the visual stimulus from the spike train given the estimated parameters. The common-noise model produces Bayesian decoding performance as accurate as that of a model with direct coupling, but with significantly more robustness to spike timing perturbations.

Correlated firing among major ganglion cell types in primate retina.

Retinal ganglion cells exhibit substantial correlated firing: a tendency to fire nearly synchronously at rates different from those expected by chance. These correlations suggest that network interactions significantly shape the visual signal transmitted from the eye to the brain. This study describes the degree and structure of correlated firing among the major ganglion cell types in primate retina. Correlated firing among ON and OFF parasol, ON and OFF midget, and small bistratified cells, which together constitute roughly 75% of the input to higher visual areas, was studied using large-scale multi-electrode recordings. Correlated firing in the presence of constant, spatially uniform illumination exhibited characteristic strength, time course and polarity within and across cell types. Pairs of nearby cells with the same light response polarity were positively correlated; cells with the opposite polarity were negatively correlated. The strength of correlated firing declined systematically with distance for each cell type, in proportion to the degree of receptive field overlap. The pattern of correlated firing across cell types was similar at photopic and scotopic light levels, although additional slow correlations were present at scotopic light levels. Similar results were also observed in two other retinal ganglion cell types. Most of these observations are consistent with the hypothesis that shared noise from photoreceptors is the dominant cause of correlated firing. Surprisingly, small bistratified cells, which receive ON input from S cones, fired synchronously with ON parasol and midget cells, which receive ON input primarily from L and M cones. Collectively, these results provide an overview of correlated firing across cell types in the primate retina, and constraints on the underlying mechanisms.

Functional connectivity in the retina at the resolution of photoreceptors.

To understand a neural circuit requires knowledge of its connectivity. Here we report measurements of functional connectivity between the input and ouput layers of the macaque retina at single-cell resolution and the implications of these for colour vision. Multi-electrode technology was used to record simultaneously from complete populations of the retinal ganglion cell types (midget, parasol and small bistratified) that transmit high-resolution visual signals to the brain. Fine-grained visual stimulation was used to identify the location, type and strength of the functional input of each cone photoreceptor to each ganglion cell. The populations of ON and OFF midget and parasol cells each sampled the complete population of long- and middle-wavelength-sensitive cones. However, only OFF midget cells frequently received strong input from short-wavelength-sensitive cones. ON and OFF midget cells showed a small non-random tendency to selectively sample from either long- or middle-wavelength-sensitive cones to a degree not explained by clumping in the cone mosaic. These measurements reveal computations in a neural circuit at the elementary resolution of individual neurons.

High-sensitivity rod photoreceptor input to the blue-yellow color opponent pathway in macaque retina.

Small bistratified cells (SBCs) in the primate retina carry a major blue-yellow opponent signal to the brain. We found that SBCs also carry signals from rod photoreceptors, with the same sign as S cone input. SBCs exhibited robust responses under low scotopic conditions. Physiological and anatomical experiments indicated that this rod input arose from the AII amacrine cell-mediated rod pathway. Rod and cone signals were both present in SBCs at mesopic light levels. These findings have three implications. First, more retinal circuits may multiplex rod and cone signals than were previously thought to, efficiently exploiting the limited number of optic nerve fibers. Second, signals from AII amacrine cells may diverge to most or all of the approximately 20 retinal ganglion cell types in the peripheral primate retina. Third, rod input to SBCs may be the substrate for behavioral biases toward perception of blue at mesopic light levels.

Receptive fields in primate retina are coordinated to sample visual space more uniformly.

In the visual system, large ensembles of neurons collectively sample visual space with receptive fields (RFs). A puzzling problem is how neural ensembles provide a uniform, high-resolution visual representation in spite of irregularities in the RFs of individual cells. This problem was approached by simultaneously mapping the RFs of hundreds of primate retinal ganglion cells. As observed in previous studies, RFs exhibited irregular shapes that deviated from standard Gaussian models. Surprisingly, these irregularities were coordinated at a fine spatial scale: RFs interlocked with their neighbors, filling in gaps and avoiding large variations in overlap. RF shapes were coordinated with high spatial precision: the observed uniformity was degraded by angular perturbations as small as 15 degrees, and the observed populations sampled visual space with more than 50% of the theoretical ideal uniformity. These results show that the primate retina encodes light with an exquisitely coordinated array of RF shapes, illustrating a higher degree of functional precision in the neural circuitry than previously appreciated.

Uniform signal redundancy of parasol and midget ganglion cells in primate retina.

The collective representation of visual space in high resolution visual pathways was explored by simultaneously measuring the receptive fields of hundreds of ON and OFF midget and parasol ganglion cells in isolated primate retina. As expected, the receptive fields of all four cell types formed regular mosaics uniformly tiling the visual scene. Surprisingly, comparison of all four mosaics revealed that the overlap of neighboring receptive fields was nearly identical, for both the excitatory center and inhibitory surround components of the receptive field. These observations contrast sharply with the large differences in the dendritic overlap between the parasol and midget cell populations, revealing a surprising lack of correspondence between the anatomical and functional architecture in the dominant circuits of the primate retina.

The structure of large-scale synchronized firing in primate retina.

Synchronized firing among neurons has been proposed to constitute an elementary aspect of the neural code in sensory and motor systems. However, it remains unclear how synchronized firing affects the large-scale patterns of activity and redundancy of visual signals in a complete population of neurons. We recorded simultaneously from hundreds of retinal ganglion cells in primate retina, and examined synchronized firing in completely sampled populations of approximately 50-100 ON-parasol cells, which form a major projection to the magnocellular layers of the lateral geniculate nucleus. Synchronized firing in pairs of cells was a subset of a much larger pattern of activity that exhibited local, isotropic spatial properties. However, a simple model based solely on interactions between adjacent cells reproduced 99% of the spatial structure and scale of synchronized firing. No more than 20% of the variability in firing of an individual cell was predictable from the activity of its neighbors. These results held both for spontaneous firing and in the presence of independent visual modulation of the firing of each cell. In sum, large-scale synchronized firing in the entire population of ON-parasol cells appears to reflect simple neighbor interactions, rather than a unique visual signal or a highly redundant coding scheme.

Synchronized firing in the retina.

Synchronized firing in neural populations has been proposed to constitute an elementary aspect of the neural code, but a complete understanding of its origins and significance has been elusive. Synchronized firing has been extensively documented in retinal ganglion cells, the output neurons of the retina. However, differences in synchronized firing across species and cell types have led to varied conclusions about its mechanisms and role in visual signaling. Recent work on two identified cell populations in the primate retina, the ON-parasol and OFF-parasol cells, permits a more unified understanding. Intracellular recordings reveal that synchronized firing in these cell types arises primarily from common synaptic input to adjacent pairs of cells. Statistical analysis indicates that local pairwise interactions can explain the pattern of synchronized firing in the entire parasol cell population. Computational analysis reveals that the aggregate impact of synchronized firing on the visual signal is substantial. Thus, in the parasol cells, the origin and impact of synchronized firing on the neural code may be understood as locally shared input which influences the visual signals transmitted from eye to brain.

Spatio-temporal correlations and visual signalling in a complete neuronal population.

Statistical dependencies in the responses of sensory neurons govern both the amount of stimulus information conveyed and the means by which downstream neurons can extract it. Although a variety of measurements indicate the existence of such dependencies, their origin and importance for neural coding are poorly understood. Here we analyse the functional significance of correlated firing in a complete population of macaque parasol retinal ganglion cells using a model of multi-neuron spike responses. The model, with parameters fit directly to physiological data, simultaneously captures both the stimulus dependence and detailed spatio-temporal correlations in population responses, and provides two insights into the structure of the neural code. First, neural encoding at the population level is less noisy than one would expect from the variability of individual neurons: spike times are more precise, and can be predicted more accurately when the spiking of neighbouring neurons is taken into account. Second, correlations provide additional sensory information: optimal, model-based decoding that exploits the response correlation structure extracts 20% more information about the visual scene than decoding under the assumption of independence, and preserves 40% more visual information than optimal linear decoding. This model-based approach reveals the role of correlated activity in the retinal coding of visual stimuli, and provides a general framework for understanding the importance of correlated activity in populations of neurons.

Spatial properties and functional organization of small bistratified ganglion cells in primate retina.

The primate visual system consists of parallel pathways initiated by distinct cell types in the retina that encode different features of the visual scene. Small bistratified cells (SBCs), which form a major projection to the thalamus, exhibit blue-ON/yellow-OFF [S-ON/(L+M)-OFF] light responses thought to be important for high-acuity color vision. However, the spatial processing properties of individual SBCs and their spatial arrangement across the visual field are poorly understood. The present study of peripheral primate retina reveals that contrary to previous suggestions, SBCs exhibit center-surround spatial structure, with the (L+M)-OFF component of the receptive field approximately 50% larger in diameter than the S-ON component. Analysis of response kinetics shows that the (L+M)-OFF response in SBCs is slower than the S-ON response and significantly less transient than that of simultaneously recorded OFF-parasol cells. The (L+M)-OFF response in SBCs was eliminated by bath application of the metabotropic glutamate receptor agonist L-APB. These observations indicate that the (L+M)-OFF response of SBCs is not formed by OFF-bipolar cell input as has been suspected and suggest that it arises from horizontal cell feedback. Finally, the receptive fields of SBCs form orderly mosaics, with overlap and regularity similar to those of ON-parasol cells. Thus, despite their distinctive morphology and chromatic properties, SBCs exhibit two features of other retinal ganglion cell types: center-surround antagonism and regular mosaic sampling of visual space.

Identification and characterization of a Y-like primate retinal ganglion cell type.

The primate retina communicates visual information to the brain via a set of parallel pathways that originate from at least 22 anatomically distinct types of retinal ganglion cells. Knowledge of the physiological properties of these ganglion cell types is of critical importance for understanding the functioning of the primate visual system. Nonetheless, the physiological properties of only a handful of retinal ganglion cell types have been studied in detail. Here we show, using a newly developed multielectrode array system for the large-scale recording of neural activity, the existence of a physiologically distinct population of ganglion cells in the primate retina with distinctive visual response properties. These cells, which we will refer to as upsilon cells, are characterized by large receptive fields, rapid and transient responses to light, and significant nonlinearities in their spatial summation. Based on the measured properties of these cells, we speculate that they correspond to the smooth/large radiate cells recently identified morphologically in the primate retina and may therefore provide visual input to both the lateral geniculate nucleus and the superior colliculus. We further speculate that the upsilon cells may be the primate retina's counterparts of the Y-cells observed in the cat and other mammalian species.

Estimating information rates with confidence intervals in neural spike trains.

Information theory provides a natural set of statistics to quantify the amount of knowledge a neuron conveys about a stimulus. A related work (Kennel, Shlens, Abarbanel, & Chichilnisky, 2005) demonstrated how to reliably estimate, with a Bayesian confidence interval, the entropy rate from a discrete, observed time series. We extend this method to measure the rate of novel information that a neural spike train encodes about a stimulus--the average and specific mutual information rates. Our estimator makes few assumptions about the underlying neural dynamics, shows excellent performance in experimentally relevant regimes, and uniquely provides confidence intervals bounding the range of information rates compatible with the observed spike train. We validate this estimator with simulations of spike trains and highlight how stimulus parameters affect its convergence in bias and variance. Finally, we apply these ideas to a recording from a guinea pig retinal ganglion cell and compare results to a simple linear decoder.

The structure of multi-neuron firing patterns in primate retina.

Current understanding of many neural circuits is limited by our ability to explore the vast number of potential interactions between different cells. We present a new approach that dramatically reduces the complexity of this problem. Large-scale multi-electrode recordings were used to measure electrical activity in nearly complete, regularly spaced mosaics of several hundred ON and OFF parasol retinal ganglion cells in macaque monkey retina. Parasol cells exhibited substantial pairwise correlations, as has been observed in other species, indicating functional connectivity. However, pairwise measurements alone are insufficient to determine the prevalence of multi-neuron firing patterns, which would be predicted from widely diverging common inputs and have been hypothesized to convey distinct visual messages to the brain. The number of possible multi-neuron firing patterns is far too large to study exhaustively, but this problem may be circumvented if two simple rules of connectivity can be established: (1) multi-cell firing patterns arise from multiple pairwise interactions, and (2) interactions are limited to adjacent cells in the mosaic. Using maximum entropy methods from statistical mechanics, we show that pairwise and adjacent interactions accurately accounted for the structure and prevalence of multi-neuron firing patterns, explaining approximately 98% of the departures from statistical independence in parasol cells and approximately 99% of the departures that were reproducible in repeated measurements. This approach provides a way to define limits on the complexity of network interactions and thus may be relevant for probing the function of many neural circuits.

Estimating entropy rates with Bayesian confidence intervals.

The entropy rate quantifies the amount of uncertainty or disorder produced by any dynamical system. In a spiking neuron, this uncertainty translates into the amount of information potentially encoded and thus the subject of intense theoretical and experimental investigation. Estimating this quantity in observed, experimental data is difficult and requires a judicious selection of probabilistic models, balancing between two opposing biases. We use a model weighting principle originally developed for lossless data compression, following the minimum description length principle. This weighting yields a direct estimator of the entropy rate, which, compared to existing methods, exhibits significantly less bias and converges faster in simulation. With Monte Carlo techinques, we estimate a Bayesian confidence interval for the entropy rate. In related work, we apply these ideas to estimate the information rates between sensory stimuli and neural responses in experimental data (Shlens, Kennel, Abarbanel, & Chichilnisky, in preparation).