Residual dynamics resolves recurrent contributions to neural computation.

Relating neural activity to behavior requires an understanding of how neural computations arise from the coordinated dynamics of distributed, recurrently connected neural populations. However, inferring the nature of recurrent dynamics from partial recordings of a neural circuit presents considerable challenges. Here we show that some of these challenges can be overcome by a fine-grained analysis of the dynamics of neural residuals-that is, trial-by-trial variability around the mean neural population trajectory for a given task condition. Residual dynamics in macaque prefrontal cortex (PFC) in a saccade-based perceptual decision-making task reveals recurrent dynamics that is time dependent, but consistently stable, and suggests that pronounced rotational structure in PFC trajectories during saccades is driven by inputs from upstream areas. The properties of residual dynamics restrict the possible contributions of PFC to decision-making and saccade generation and suggest a path toward fully characterizing distributed neural computations with large-scale neural recordings and targeted causal perturbations.

Deep-learning based identification, tracking, pose estimation, and behavior classification of interacting primates and mice in complex environments.

The quantification of behaviors of interest from video data is commonly used to study brain function, the effects of pharmacological interventions, and genetic alterations. Existing approaches lack the capability to analyze the behavior of groups of animals in complex environments. We present a novel deep learning architecture for classifying individual and social animal behavior, even in complex environments directly from raw video frames, while requiring no intervention after initial human supervision. Our behavioral classifier is embedded in a pipeline (SIPEC) that performs segmentation, identification, pose-estimation, and classification of complex behavior, outperforming the state of the art. SIPEC successfully recognizes multiple behaviors of freely moving individual mice as well as socially interacting non-human primates in 3D, using data only from simple mono-vision cameras in home-cage setups.

Prefrontal cortex exhibits multidimensional dynamic encoding during decision-making.

Recent work has suggested that the prefrontal cortex (PFC) plays a key role in context-dependent perceptual decision-making. In this study, we addressed that role using a new method for identifying task-relevant dimensions of neural population activity. Specifically, we show that the PFC has a multidimensional code for context, decisions and both relevant and irrelevant sensory information. Moreover, these representations evolve in time, with an early linear accumulation phase followed by a phase with rotational dynamics. We identify the dimensions of neural activity associated with these phases and show that they do not arise from distinct populations but from a single population with broad tuning characteristics. Finally, we use model-based decoding to show that the transition from linear to rotational dynamics coincides with a plateau in decoding accuracy, revealing that rotational dynamics in the PFC preserve sensory choice information for the duration of the stimulus integration period.

Nearest neighbours reveal fast and slow components of motor learning.

Changes in behaviour resulting from environmental influences, development and learning1-5 are commonly quantified on the basis of a few hand-picked features2-4,6,7 (for example, the average pitch of acoustic vocalizations3), assuming discrete classes of behaviours (such as distinct vocal syllables)2,3,8-10. However, such methods generalize poorly across different behaviours and model systems and may miss important components of change. Here we present a more-general account of behavioural change that is based on nearest-neighbour statistics11-13, and apply it to song development in a songbird, the zebra finch3. First, we introduce the concept of 'repertoire dating', whereby each rendition of a behaviour (for example, each vocalization) is assigned a repertoire time, reflecting when similar renditions were typical in the behavioural repertoire. Repertoire time isolates the components of vocal variability that are congruent with long-term changes due to vocal learning and development, and stratifies the behavioural repertoire into 'regressions', 'anticipations' and 'typical renditions'. Second, we obtain a holistic, yet low-dimensional, description of vocal change in terms of a stratified 'behavioural trajectory', revealing numerous previously unrecognized components of behavioural change on fast and slow timescales, as well as distinct patterns of overnight consolidation1,2,4,14,15 across the behavioral repertoire. We find that diurnal changes in regressions undergo only weak consolidation, whereas anticipations and typical renditions consolidate fully. Because of its generality, our nonparametric description of how behaviour evolves relative to itself-rather than to a potentially arbitrary, experimenter-defined goal2,3,14,16-appears well suited for comparing learning and change across behaviours and species17,18, as well as biological and artificial systems5.

Context-dependent computation by recurrent dynamics in prefrontal cortex.

Prefrontal cortex is thought to have a fundamental role in flexible, context-dependent behaviour, but the exact nature of the computations underlying this role remains largely unknown. In particular, individual prefrontal neurons often generate remarkably complex responses that defy deep understanding of their contribution to behaviour. Here we study prefrontal cortex activity in macaque monkeys trained to flexibly select and integrate noisy sensory inputs towards a choice. We find that the observed complexity and functional roles of single neurons are readily understood in the framework of a dynamical process unfolding at the level of the population. The population dynamics can be reproduced by a trained recurrent neural network, which suggests a previously unknown mechanism for selection and integration of task-relevant inputs. This mechanism indicates that selection and integration are two aspects of a single dynamical process unfolding within the same prefrontal circuits, and potentially provides a novel, general framework for understanding context-dependent computations.

Functional mechanisms shaping lateral geniculate responses to artificial and natural stimuli.

Functional models of the early visual system should predict responses not only to simple artificial stimuli but also to sequences of complex natural scenes. An ideal testbed for such models is the lateral geniculate nucleus (LGN). Mechanisms shaping LGN responses include the linear receptive field and two fast adaptation processes, sensitive to luminance and contrast. We propose a compact functional model for these mechanisms that operates on sequences of arbitrary images. With the same parameters that fit the firing rate responses to simple stimuli, it predicts the bulk of the firing rate responses to complex stimuli, including natural scenes. Further improvements could result by adding a spiking mechanism, possibly one capable of bursts, but not by adding mechanisms of slow adaptation. We conclude that up to the LGN the responses to natural scenes can be largely explained through insights gained with simple artificial stimuli.

How MT cells analyze the motion of visual patterns.

Neurons in area MT (V5) are selective for the direction of visual motion. In addition, many are selective for the motion of complex patterns independent of the orientation of their components, a behavior not seen in earlier visual areas. We show that the responses of MT cells can be captured by a linear-nonlinear model that operates not on the visual stimulus, but on the afferent responses of a population of nonlinear V1 cells. We fit this cascade model to responses of individual MT neurons and show that it robustly predicts the separately measured responses to gratings and plaids. The model captures the full range of pattern motion selectivity found in MT. Cells that signal pattern motion are distinguished by having convergent excitatory input from V1 cells with a wide range of preferred directions, strong motion opponent suppression and a tuned normalization that may reflect suppressive input from the surround of V1 cells.

The statistical computation underlying contrast gain control.

In the early visual system, a contrast gain control mechanism sets the gain of responses based on the locally prevalent contrast. The measure of contrast used by this adaptation mechanism is commonly assumed to be the standard deviation of light intensities relative to the mean (root-mean-square contrast). A number of alternatives, however, are possible. For example, the measure of contrast might depend on the absolute deviations relative to the mean, or on the prevalence of the darkest or lightest intensities. We investigated the statistical computation underlying this measure of contrast in the cat's lateral geniculate nucleus, which relays signals from retina to cortex. Borrowing a method from psychophysics, we recorded responses to white noise stimuli whose distribution of intensities was precisely varied. We varied the standard deviation, skewness, and kurtosis of the distribution of intensities while keeping the mean luminance constant. We found that gain strongly depends on the standard deviation of the distribution. At constant standard deviation, moreover, gain is invariant to changes in skewness or kurtosis. These findings held for both ON and OFF cells, indicating that the measure of contrast is independent of the range of stimulus intensities signaled by the cells. These results confirm the long-held assumption that contrast gain control computes root-mean-square contrast. They also show that contrast gain control senses the full distribution of intensities and leaves unvaried the relative responses of the different cell types. The advantages to visual processing of this remarkably specific computation are not entirely known.

Do we know what the early visual system does?

We can claim that we know what the visual system does once we can predict neural responses to arbitrary stimuli, including those seen in nature. In the early visual system, models based on one or more linear receptive fields hold promise to achieve this goal as long as the models include nonlinear mechanisms that control responsiveness, based on stimulus context and history, and take into account the nonlinearity of spike generation. These linear and nonlinear mechanisms might be the only essential determinants of the response, or alternatively, there may be additional fundamental determinants yet to be identified. Research is progressing with the goals of defining a single "standard model" for each stage of the visual pathway and testing the predictive power of these models on the responses to movies of natural scenes. These predictive models represent, at a given stage of the visual pathway, a compact description of visual computation. They would be an invaluable guide for understanding the underlying biophysical and anatomical mechanisms and relating neural responses to visual perception.

The suppressive field of neurons in lateral geniculate nucleus.

The responses of neurons in lateral geniculate nucleus (LGN) exhibit powerful suppressive phenomena such as contrast saturation, size tuning, and masking. These phenomena cannot be explained by the classical center-surround receptive field and have been ascribed to a variety of mechanisms, including feedback from cortex. We asked whether these phenomena might all be explained by a single mechanism, contrast gain control, which is inherited from retina and possibly strengthened in thalamus. We formalized an intuitive model of retinal contrast gain control that explicitly predicts gain as a function of local contrast. In the model, the output of the receptive field is divided by the output of a suppressive field, which computes the local root-mean-square contrast. The model provides good fits to LGN responses to a variety of stimuli; with a single set of parameters, it captures saturation, size tuning, and masking. It also correctly predicts that responses to small stimuli grow proportionally with contrast: were it not for the suppressive field, LGN responses would be linear. We characterized the suppressive field and found that it is similar in size to the surround of the classical receptive field (which is eight times larger than commonly estimated), it is not selective for stimulus orientation, and it responds to a wide range of frequencies, including very low spatial frequencies and high temporal frequencies. The latter property is hardly consistent with feedback from cortex. These measurements thoroughly describe the visual properties of contrast gain control in LGN and provide a parsimonious explanation for disparate suppressive phenomena.

Independence of luminance and contrast in natural scenes and in the early visual system.

The early visual system is endowed with adaptive mechanisms that rapidly adjust gain and integration time based on the local luminance (mean intensity) and contrast (standard deviation of intensity relative to the mean). Here we show that these mechanisms are matched to the statistics of the environment. First, we measured the joint distribution of luminance and contrast in patches selected from natural images and found that luminance and contrast were statistically independent of each other. This independence did not hold for artificial images with matched spectral characteristics. Second, we characterized the effects of the adaptive mechanisms in lateral geniculate nucleus (LGN), the direct recipient of retinal outputs. We found that luminance gain control had the same effect at all contrasts and that contrast gain control had the same effect at all mean luminances. Thus, the adaptive mechanisms for luminance and contrast operate independently, reflecting the very independence encountered in natural images.

Mapping of stimulus energy in primary visual cortex.

A recent optical imaging study of primary visual cortex (V1) by Basole, White, and Fitzpatrick demonstrated that maps of preferred orientation depend on the choice of stimuli used to measure them. These authors measured population responses expressed as a function of the optimal orientation of long drifting bars. They then varied bar length, direction, and speed and found that stimuli of a same orientation can elicit different population responses and stimuli with different orientation can elicit similar population responses. We asked whether these results can be explained from known properties of V1 receptive fields. We implemented an "energy model" where a receptive field integrates stimulus energy over a region of three-dimensional frequency space. The population of receptive fields defines a volume of visibility, which covers all orientations and a plausible range of spatial and temporal frequencies. This energy model correctly predicts the population response to bars of different length, direction, and speed and explains the observations made with optical imaging. The model also readily explains a related phenomenon, the appearance of motion streaks for fast-moving dots. We conclude that the energy model can be applied to activation maps of V1 and predicts phenomena that may otherwise appear to be surprising. These results indicate that maps obtained with optical imaging reflect the layout of neurons selective for stimulus energy, not for isolated stimulus features such as orientation, direction, and speed.