Activity in primate visual cortex is minimally driven by spontaneous movements.

Organisms process sensory information in the context of their own moving bodies, an idea referred to as embodiment. This idea is important for developmental neuroscience, robotics and systems neuroscience. The mechanisms supporting embodiment are unknown, but a manifestation could be the observation in mice of brain-wide neuromodulation, including in the primary visual cortex, driven by task-irrelevant spontaneous body movements. We tested this hypothesis in macaque monkeys (Macaca mulatta), a primate model for human vision, by simultaneously recording visual cortex activity and facial and body movements. We also sought a direct comparison using an analogous approach to those used in mouse studies. Here we found that activity in the primate visual cortex (V1, V2 and V3/V3A) was associated with the animals' own movements, but this modulation was largely explained by the impact of the movements on the retinal image, that is, by changes in visual input. These results indicate that visual cortex in primates is minimally driven by spontaneous movements and may reflect species-specific sensorimotor strategies.

Serotonergic modulation of local network processing in V1 mirrors previously reported signatures of local network modulation by spatial attention.

Sensory processing is influenced by neuromodulators such as serotonin, thought to relay behavioural state. Recent work has shown that the modulatory effect of serotonin itself differs with the animal's behavioural state. In primates, including humans, the serotonin system is anatomically important in the primary visual cortex (V1). We previously reported that in awake fixating macaques, serotonin reduces the spiking activity by decreasing response gain in V1. But the effect of serotonin on the local network is unknown. Here, we simultaneously recorded single-unit activity and local field potentials (LFPs) while iontophoretically applying serotonin in V1 of alert monkeys fixating on a video screen for juice rewards. The reduction in spiking response we observed previously is the opposite of the known increase of spiking activity with spatial attention. Conversely, in the local network (LFP), the application of serotonin resulted in changes mirroring the local network effects of previous reports in macaques directing spatial attention to the receptive field. It reduced the LFP power and the spike-field coherence, and the LFP became less predictive of spiking activity, consistent with reduced functional connectivity. We speculate that together, these effects may reflect the sensory side of a serotonergic contribution to quiet vigilance: The lower gain reduces the salience of stimuli to suppress an orienting reflex to novel stimuli, whereas at the network level, visual processing is in a state comparable to that of spatial attention.

Decision-related feedback in visual cortex lacks spatial selectivity.

Feedback in the brain is thought to convey contextual information that underlies our flexibility to perform different tasks. Empirical and computational work on the visual system suggests this is achieved by targeting task-relevant neuronal subpopulations. We combine two tasks, each resulting in selective modulation by feedback, to test whether the feedback reflected the combination of both selectivities. We used visual feature-discrimination specified at one of two possible locations and uncoupled the decision formation from motor plans to report it, while recording in macaque mid-level visual areas. Here we show that although the behavior is spatially selective, using only task-relevant information, modulation by decision-related feedback is spatially unselective. Population responses reveal similar stimulus-choice alignments irrespective of stimulus relevance. The results suggest a common mechanism across tasks, independent of the spatial selectivity these tasks demand. This may reflect biological constraints and facilitate generalization across tasks. Our findings also support a previously hypothesized link between feature-based attention and decision-related activity.

Sleep does not aid the generalisation of binocular disparity-based learning to the other visual hemifield.

Visual perceptual learning refers to long-lasting performance improvements on a visual skill - an ability supported by plastic changes in early visual brain areas. Visual perceptual learning has been shown to be induced by training and to benefit from consolidation during sleep, presumably via the reactivation of learning-associated neuronal firing patterns. However, previous studies have almost exclusively relied on a single paradigm, the texture discrimination task, on which performance improvements may rely on higher-order rather than lower-level perceptual skills. In the present study, we tested whether sleep has beneficial effects on a visual disparity discrimination task. We confirm previous findings in showing that the ability to discriminate different disparities is unaffected by sleep during a 12-hr retention period after training. Importantly, we extend these results by providing evidence against an effect of sleep on the generalisation of improved disparity discrimination across the vertical meridian. By relying on a between-subject design, we further exclude carry-over effects as a possible confound present in previous findings. These data argue against sleep as an important factor in the consolidation of a low-level perceptual skill. This sets important constraints on models of the role of sleep and sleep-associated neural reactivation in the consolidation of non-declarative memories.

Neuroscience needs behavior: inferring psychophysical strategy trial by trial.

Tools quantifying dynamic behavior are important to understand brain function. In this issue of Neuron, Roy et al. (2021) extended the available repertoire with PsyTrack, which tracks, trial by trial, how subjects performing psychophysical tasks adjust the way they weigh stimuli and task covariates or change their biases.

Binocular disparity-based learning is retinotopically specific and independent of sleep.

Sleep supports the consolidation of recently encoded declarative and procedural memories. An important component of this effect is the repeated reactivation of neuronal ensemble activity elicited during memory encoding. For perceptual learning, however, sleep benefits have only been reported for specific tasks and it is not clear whether sleep targets low-level perceptual, higher-order temporal or attentional aspects of performance. Here, we employed a coarse binocular disparity discrimination task, known to rely on low-level stereoscopic vision. We show that human subjects improve over training and retain the same performance level across a 12-h retention period. Improvements do not generalize to other parts of the visual field and are unaffected by whether the retention period contains sleep or not. These results are compatible with the notion that behavioural improvements in binocular disparity discrimination do not additionally benefit from sleep when compared with the same time spent awake. We hypothesize that this might generalize to other strictly low-level perceptual tasks. This article is part of the Theo Murphy meeting issue 'Memory reactivation: replaying events past, present and future'.

Choice (-history) correlations in sensory cortex: cause or consequence?

One challenge in neuroscience, as in other areas of science, is to make inferences about the underlying causal structure from correlational data. Here, we discuss this challenge in the context of choice correlations in sensory neurons, that is, trial-by-trial correlations, unexplained by the stimulus, between the activity of sensory neurons and an animal's perceptual choice. Do these choice-correlations reflect feedforward, feedback signalling, both, or neither? We highlight recent results of correlational and causal examinations of choice and choice-history signals in sensory, and in part sensorimotor, cortex and address formal statistical frameworks to infer causal interactions from data.

Manipulating patterns of dynamic deformation elicits the impression of cloth with varying stiffness.

Cloth is a common material, and humans can visually estimate its mechanical properties by observing how it deforms under external forces. Here, we ask whether and how dynamic deformation can affect the perception of mechanical properties of cloth. In Experiment 1, we find that both intrinsic mechanical properties and optical properties affect stiffness perception when the stimuli are presented as images. By contrast, in videos, humans can partially discount the effect of optical appearances and exhibit higher sensitivity to stiffness. We further identified an idiosyncratic deformation pattern (i.e., movement uniformity) to differentiate stiffness, which can be reliably measured by six optical flow features. In Experiment 2, we isolate the deformation by creating dynamic dot stimuli from the 3-D mesh of the cloth. We directly alter the movement pattern by manipulating the uniformity of the displacement vectors on the dot stimuli and show that changing the pattern of dynamic deformation alone can alter the perceived stiffness of cloth in a variety of scene setups. Furthermore, by analyzing optical flow fields extracted from the manipulated dynamic dot stimuli, we confirmed the same six optical flow features can be diagnostic of the degree of stiffness of moving cloth across different scenes. Overall, our study demonstrates that manipulating patterns of dynamic deformation alone can elicit the impression of cloth with varying stiffness, suggesting that the human visual system might rely on the idiosyncratic pattern of dynamic deformation for estimating stiffness.

Human-level saccade detection performance using deep neural networks.

Saccades are ballistic eye movements that rapidly shift gaze from one location of visual space to another. Detecting saccades in eye movement recordings is important not only for studying the neural mechanisms underlying sensory, motor, and cognitive processes, but also as a clinical and diagnostic tool. However, automatically detecting saccades can be difficult, particularly when such saccades are generated in coordination with other tracking eye movements, like smooth pursuits, or when the saccade amplitude is close to eye tracker noise levels, like with microsaccades. In such cases, labeling by human experts is required, but this is a tedious task prone to variability and error. We developed a convolutional neural network to automatically detect saccades at human-level accuracy and with minimal training examples. Our algorithm surpasses state of the art according to common performance metrics and could facilitate studies of neurophysiological processes underlying saccade generation and visual processing. NEW & NOTEWORTHY Detecting saccades in eye movement recordings can be a difficult task, but it is a necessary first step in many applications. We present a convolutional neural network that can automatically identify saccades with human-level accuracy and with minimal training examples. We show that our algorithm performs better than other available algorithms, by comparing performance on a wide range of data sets. We offer an open-source implementation of the algorithm as well as a web service.

Differentiating between Models of Perceptual Decision Making Using Pupil Size Inferred Confidence.

During perceptual decisions, subjects often rely more strongly on early, rather than late, sensory evidence, even in tasks when both are equally informative about the correct decision. This early psychophysical weighting has been explained by an integration-to-bound decision process, in which the stimulus is ignored after the accumulated evidence reaches a certain bound, or confidence level. Here, we derive predictions about how the average temporal weighting of the evidence depends on a subject's decision confidence in this model. To test these predictions empirically, we devised a method to infer decision confidence from pupil size in 2 male monkeys performing a disparity discrimination task. Our animals' data confirmed the integration-to-bound predictions, with different internal decision bounds and different levels of correlation between pupil size and decision confidence accounting for differences between animals. However, the data were less compatible with two alternative accounts for early psychophysical weighting: attractor dynamics either within the decision area or due to feedback to sensory areas, or a feedforward account due to neuronal response adaptation. This approach also opens the door to using confidence more broadly when studying the neural basis of decision making.SIGNIFICANCE STATEMENT An animal's ability to adjust decisions based on its level of confidence, sometimes referred to as "metacognition," has generated substantial interest in neuroscience. Here, we show how measurements of pupil diameter in macaques can be used to infer their confidence. This technique opens the door to more neurophysiological studies of confidence because it eliminates the need for training on behavioral paradigms to evaluate confidence. We then use this technique to test predictions from competing explanations of why subjects in perceptual decision making often rely more strongly on early evidence: the way in which the strength of this effect should depend on a subject's decision confidence. We find that a bounded decision formation process best explains our empirical data.

Monoaminergic Neuromodulation of Sensory Processing.

All neuronal circuits are subject to neuromodulation. Modulatory effects on neuronal processing and resulting behavioral changes are most commonly reported for higher order cognitive brain functions. Comparatively little is known about how neuromodulators shape processing in sensory brain areas that provide the signals for downstream regions to operate on. In this article, we review the current knowledge about how the monoamine neuromodulators serotonin, dopamine and noradrenaline influence the representation of sensory stimuli in the mammalian sensory system. We review the functional organization of the monoaminergic brainstem neuromodulatory systems in relation to their role for sensory processing and summarize recent neurophysiological evidence showing that monoamines have diverse effects on early sensory processing, including changes in gain and in the precision of neuronal responses to sensory inputs. We also highlight the substantial evidence for complementarity between these neuromodulatory systems with different patterns of innervation across brain areas and cortical layers as well as distinct neuromodulatory actions. Studying the effects of neuromodulators at various target sites is a crucial step in the development of a mechanistic understanding of neuronal information processing in the healthy brain and in the generation and maintenance of mental diseases.

Estimating mechanical properties of cloth from videos using dense motion trajectories: Human psychophysics and machine learning.

Humans can visually estimate the mechanical properties of deformable objects (e.g., cloth stiffness). While much of the recent work on material perception has focused on static image cues (e.g., textures and shape), little is known about whether humans can integrate information over time to make a judgment. Here we investigated the effect of spatiotemporal information across multiple frames (multiframe motion) on estimating the bending stiffness of cloth. Using high-fidelity cloth animations, we first examined how the perceived bending stiffness changed as a function of the physical bending stiffness defined in the simulation model. Using maximum-likelihood difference-scaling methods, we found that the perceived stiffness and physical bending stiffness were highly correlated. A second experiment in which we scrambled the frame sequences diminished this correlation. This suggests that multiframe motion plays an important role. To provide further evidence for this finding, we extracted dense motion trajectories from the videos across 15 consecutive frames and used the trajectory descriptors to train a machine-learning model with the measured perceptual scales. The model can predict human perceptual scales in new videos with varied winds, optical properties of cloth, and scene setups. When the correct multiframe was removed (using either scrambled videos or two-frame optical flow to train the model), the predictions significantly worsened. Our findings demonstrate that multiframe motion information is important for both humans and machines to estimate the mechanical properties. In addition, we show that dense motion trajectories are effective features to build a successful automatic cloth-estimation system.

Cognitive Computational Neuroscience: A New Conference for an Emerging Discipline.

Understanding the computational principles that underlie complex behavior is a central goal in cognitive science, artificial intelligence, and neuroscience. In an attempt to unify these disconnected communities, we created a new conference called Cognitive Computational Neuroscience (CCN). The inaugural meeting revealed considerable enthusiasm but significant obstacles remain.

Can Serial Dependencies in Choices and Neural Activity Explain Choice Probabilities?

During perceptual decisions the activity of sensory neurons covaries with choice, a covariation often quantified as "choice-probability". Moreover, choices are influenced by a subject's previous choice (serial dependence) and neuronal activity often shows temporal correlations on long (seconds) timescales. Here, we test whether these findings are linked. Using generalized linear models, we analyze simultaneous measurements of behavior and V2 neural activity in macaques performing a visual discrimination task. Both, decisions and spiking activity show substantial temporal correlations and cross-correlations but seem to reflect two mostly separate processes. Indeed, removing history effects using semipartial correlation analysis leaves choice probabilities largely unchanged. The serial dependencies in choices and neural activity therefore cannot explain the observed choice probability. Rather, serial dependencies in choices and spiking activity reflect two predominantly separate but parallel processes, which are coupled on each trial by covariations between choices and activity. These findings provide important constraints for computational models of perceptual decision-making that include feedback signals.SIGNIFICANCE STATEMENT Correlations, unexplained by the sensory input, between the activity of sensory neurons and an animal's perceptual choice ("choice probabilities") have received attention from both a systems and computational neuroscience perspective. Conversely, whereas temporal correlations for both spiking activity ("non-stationarities") and for a subject's choices in perceptual tasks ("serial dependencies") have long been established, they have typically been ignored when measuring choice probabilities. Some accounts of choice probabilities incorporating feedback predict that these observations are linked. Here, we explore the extent to which this is the case. We find that, contrasting with these predictions, choice probabilities are largely independent of serial dependencies, which adds new constraints to accounts of choice probabilities that include feedback.

Serotonin Decreases the Gain of Visual Responses in Awake Macaque V1.

Serotonin, an important neuromodulator in the brain, is implicated in affective and cognitive functions. However, its role even for basic cortical processes is controversial. For example, in the mammalian primary visual cortex (V1), heterogenous serotonergic modulation has been observed in anesthetized animals. Here, we combined extracellular single-unit recordings with iontophoresis in awake animals. We examined the role of serotonin on well-defined tuning properties (orientation, spatial frequency, contrast, and size) in V1 of two male macaque monkeys. We find that in the awake macaque the modulatory effect of serotonin is surprisingly uniform: it causes a mainly multiplicative decrease of the visual responses and a slight increase in the stimulus-selective response latency. Moreover, serotonin neither systematically changes the selectivity or variability of the response, nor the interneuronal correlation unexplained by the stimulus ("noise-correlation"). The modulation by serotonin has qualitative similarities with that for a decrease in stimulus contrast, but differs quantitatively from decreasing contrast. It can be captured by a simple additive change to a threshold-linear spiking nonlinearity. Together, our results show that serotonin is well suited to control the response gain of neurons in V1 depending on the animal's behavioral or motivational context, complementing other known state-dependent gain-control mechanisms.SIGNIFICANCE STATEMENT Serotonin is an important neuromodulator in the brain and a major target for drugs used to treat psychiatric disorders. Nonetheless, surprisingly little is known about how it shapes information processing in sensory areas. Here we examined the serotonergic modulation of visual processing in the primary visual cortex of awake behaving macaque monkeys. We found that serotonin mainly decreased the gain of the visual responses, without systematically changing their selectivity, variability, or covariability. This identifies a simple computational function of serotonin for state-dependent sensory processing, depending on the animal's affective or motivational state.

Decision-Related Activity in Macaque V2 for Fine Disparity Discrimination Is Not Compatible with Optimal Linear Readout.

Fine judgments of stereoscopic depth rely mainly on relative judgments of depth (relative binocular disparity) between objects, rather than judgments of the distance to where the eyes are fixating (absolute disparity). In macaques, visual area V2 is the earliest site in the visual processing hierarchy for which neurons selective for relative disparity have been observed (Thomas et al., 2002). Here, we found that, in macaques trained to perform a fine disparity discrimination task, disparity-selective neurons in V2 were highly selective for the task, and their activity correlated with the animals' perceptual decisions (unexplained by the stimulus). This may partially explain similar correlations reported in downstream areas. Although compatible with a perceptual role of these neurons for the task, the interpretation of such decision-related activity is complicated by the effects of interneuronal "noise" correlations between sensory neurons. Recent work has developed simple predictions to differentiate decoding schemes (Pitkow et al., 2015) without needing measures of noise correlations, and found that data from early sensory areas were compatible with optimal linear readout of populations with information-limiting correlations. In contrast, our data here deviated significantly from these predictions. We additionally tested this prediction for previously reported results of decision-related activity in V2 for a related task, coarse disparity discrimination (Nienborg and Cumming, 2006), thought to rely on absolute disparity. Although these data followed the predicted pattern, they violated the prediction quantitatively. This suggests that optimal linear decoding of sensory signals is not generally a good predictor of behavior in simple perceptual tasks. SIGNIFICANCE STATEMENT: Activity in sensory neurons that correlates with an animal's decision is widely believed to provide insights into how the brain uses information from sensory neurons. Recent theoretical work developed simple predictions to differentiate decoding schemes, and found support for optimal linear readout of early sensory populations with information-limiting correlations. Here, we observed decision-related activity for neurons in visual area V2 of macaques performing fine disparity discrimination, as yet the earliest site for this task. These findings, and previously reported results from V2 in a different task, deviated from the predictions for optimal linear readout of a population with information-limiting correlations. Our results suggest that optimal linear decoding of early sensory information is not a general decoding strategy used by the brain.

Feedforward and feedback sources of choice probability in neural population responses.

How the processing of signals carried by sensory neurons supports perceptual decisions is a long-standing question in neuroscience. The ability to record neuronal activity in awake animals while they perform psychophysical tasks near threshold has been a key advance in studying these questions. Trial-to-trial correlations between the activity of sensory neurons and the decisions reported by animals ('choice probabilities'), even when measured across repeated presentations of an identical stimulus provide insights into this problem. But understanding the sources of such co-variability between sensory neurons and behavior has proven more difficult than it initially appeared. Below, we discuss our current understanding of what gives rise to these correlations.

Belief states as a framework to explain extra-retinal influences in visual cortex.

The activity of sensory neurons is modulated by non-sensory influences, but the role of these influences in cognition is only partially understood. Here we review how the large-scale recording of neuronal activity within and across brain regions allows researchers to examine the interactions between simultaneously recorded neurons as they are jointly influenced by fluctuations in an animal's mental state. We focus on studies on the visual cortex of non-human primates to examine the relationship between extra-retinal influences and beliefs about the state of the sensory world. We explore how these influences can be understood within theoretical frameworks that propose how the continuous updating of belief states supports perceptual inference.

Decision-related activity in sensory neurons may depend on the columnar architecture of cerebral cortex.

Many studies have reported correlations between the activity of sensory neurons and animals' judgments in discrimination tasks. Here, we suggest that such neuron-behavior correlations may require a cortical map for the task relevant features. This would explain why studies using discrimination tasks based on disparity in area V1 have not found these correlations: V1 contains no map for disparity. This scheme predicts that activity of V1 neurons correlates with decisions in an orientation-discrimination task. To test this prediction, we trained two macaque monkeys in a coarse orientation discrimination task using band-pass-filtered dynamic noise. The two orientations were always 90° apart and task difficulty was controlled by varying the orientation bandwidth of the filter. While the trained animals performed this task, we recorded from orientation-selective V1 neurons (n = 82, n = 31 for Monkey 1, n = 51 for Monkey 2). For both monkeys, we observed significant correlation (quantified as "choice probabilities") of the V1 activity with the monkeys' perceptual judgments (mean choice probability 0.54, p = 10(-5)). In one of these animals, we had previously measured choice probabilities in a disparity discrimination task in V1, which had been at chance (0.49, not significantly different from 0.5). The choice probabilities in this monkey for the orientation discrimination task were significantly larger than those for the disparity discrimination task (p = 0.032). These results are predicted by our suggestion that choice probabilities are only observed for cortical sensory neurons that are organized in maps for the task-relevant feature.

Contrast dependence and differential contributions from somatostatin- and parvalbumin-expressing neurons to spatial integration in mouse V1.

A characteristic feature in the primary visual cortex is that visual responses are suppressed as a stimulus extends beyond the classical receptive field. Here, we examined the role of inhibitory neurons expressing somatostatin (SOM⁺) or parvalbumin (PV⁺) on surround suppression and preferred receptive field size. We recorded multichannel extracellular activity in V1 of transgenic mice expressing channelrhodopsin in SOM⁺ neurons or PV⁺ neurons. Preferred size and surround suppression were measured using drifting square-wave gratings of varying radii and at two contrasts. Consistent with findings in primates, we found that the preferred size was larger for lower contrasts across all cortical depths, whereas the suppression index (SI) showed a trend to decrease with contrast. We then examined the effect of these metrics on units that were suppressed by photoactivation of either SOM⁺ or PV⁺ neurons. When activating SOM⁺ neurons, we found a significant increase in SI at cortical depths >400 µm, whereas activating PV⁺ neurons caused a trend toward lower SIs regardless of cortical depth. Conversely, activating PV⁺ neurons significantly increased preferred size across all cortical depths, similar to lowering contrast, whereas activating SOM⁺ neurons had no systematic effect on preferred size across all depths. These data suggest that SOM⁺ and PV⁺ neurons contribute differently to spatial integration. Our findings are compatible with the notion that SOM⁺ neurons mediate surround suppression, particularly in deeper cortex, whereas PV⁺ activation decreases the drive of the input to cortex and therefore resembles the effects on spatial integration of lowering contrast.