Brain representations of motion and position in the double-drift illusion.

In the 'double-drift' illusion, local motion within a window moving in the periphery of the visual field alters the window's perceived path. The illusion is strong even when the eyes track a target whose motion matches the window so that the stimulus remains stable on the retina. This implies that the illusion involves the integration of retinal signals with non-retinal eye-movement signals. To identify where in the brain this integration occurs, we measured BOLD fMRI responses in visual cortex while subjects experienced the double-drift illusion. We then used a combination of univariate and multivariate decoding analyses to identify (1) which brain areas were sensitive to the illusion and (2) whether these brain areas contained information about the illusory stimulus trajectory. We identified a number of cortical areas that responded more strongly during the illusion than a control condition that was matched for low-level stimulus properties. Only in area hMT+ was it possible to decode the illusory trajectory. We additionally performed a number of important controls that rule out possible low-level confounds. Concurrent eye tracking confirmed that subjects accurately tracked the moving target; we were unable to decode the illusion trajectory using eye position measurements recorded during fMRI scanning, ruling out explanations based on differences in oculomotor behavior. Our results provide evidence for a perceptual representation in human visual cortex that incorporates extraretinal information.

Pupil Size Is Sensitive to Low-Level Stimulus Features, Independent of Arousal-Related Modulation.

Similar to a camera aperture, pupil size adjusts to the surrounding luminance. Unlike a camera, pupil size is additionally modulated both by stimulus properties and by cognitive processes, including attention and arousal, though the interdependence of these factors is unclear. We hypothesized that different stimulus properties interact to jointly modulate pupil size while remaining independent from the impact of arousal. We measured pupil responses from human observers to equiluminant stimuli during a demanding rapid serial visual presentation (RSVP) task at fixation and tested how response amplitude depends on contrast, spatial frequency, and reward level. We found that under constant luminance, unattended stimuli evoke responses that are separable from changes caused by general arousal or attention. We further uncovered a double-dissociation between task-related responses and stimulus-evoked responses, suggesting that different sources of pupil size modulation are independent of one another. Our results shed light on neural pathways underlying pupillary response.

Representations in human primary visual cortex drift over time.

Primary sensory regions are believed to instantiate stable neural representations, yet a number of recent rodent studies suggest instead that representations drift over time. To test whether sensory representations are stable in human visual cortex, we analyzed a large longitudinal dataset of fMRI responses to images of natural scenes. We fit the fMRI responses using an image-computable encoding model and tested how well the model generalized across sessions. We found systematic changes in model fits that exhibited cumulative drift over many months. Convergent analyses pinpoint changes in neural responsivity as the source of the drift, while population-level representational dissimilarities between visual stimuli were unchanged. These observations suggest that downstream cortical areas may read-out a stable representation, even as representations within V1 exhibit drift.

Natural scene sampling reveals reliable coarse-scale orientation tuning in human V1.

Orientation selectivity in primate visual cortex is organized into cortical columns. Since cortical columns are at a finer spatial scale than the sampling resolution of standard BOLD fMRI measurements, analysis approaches have been proposed to peer past these spatial resolution limitations. It was recently found that these methods are predominantly sensitive to stimulus vignetting - a form of selectivity arising from an interaction of the oriented stimulus with the aperture edge. Beyond vignetting, it is not clear whether orientation-selective neural responses are detectable in BOLD measurements. Here, we leverage a dataset of visual cortical responses measured using high-field 7T fMRI. Fitting these responses using image-computable models, we compensate for vignetting and nonetheless find reliable tuning for orientation. Results further reveal a coarse-scale map of orientation preference that may constitute the neural basis for known perceptual anisotropies. These findings settle a long-standing debate in human neuroscience, and provide insights into functional organization principles of visual cortex.

Task-related hemodynamic responses in human early visual cortex are modulated by task difficulty and behavioral performance.

Early visual cortex exhibits widespread hemodynamic responses in the absence of visual stimulation, which are entrained to the timing of a task and not predicted by local spiking or local field potential. Such task-related responses (TRRs) covary with reward magnitude and physiological signatures of arousal. It is unknown, however, if TRRs change on a trial-to-trial basis according to behavioral performance and task difficulty. If so, this would suggest that TRRs reflect arousal on a trial-to-trial timescale and covary with critical task and behavioral variables. We measured functional magnetic resonance imaging blood-oxygen-level-dependent (fMRI-BOLD) responses in the early visual cortex of human observers performing an orientation discrimination task consisting of separate easy and hard runs of trials. Stimuli were presented in a small portion of one hemifield, but the fMRI response was measured in the ipsilateral hemisphere, far from the stimulus representation and focus of spatial attention. TRRs scaled in amplitude with task difficulty, behavioral accuracy, reaction time, and lapses across trials. These modulations were not explained by the influence of respiration, cardiac activity, or head movement on the fMRI signal. Similar modulations with task difficulty and behavior were observed in pupil size. These results suggest that TRRs reflect arousal and behavior on the timescale of individual trials.

Spatiotopic and retinotopic memory in the context of natural images.

Neural responses throughout the visual cortex encode stimulus location in a retinotopic (i.e., eye-centered) reference frame, and memory for stimulus position is most precise in retinal coordinates. Yet visual perception is spatiotopic: objects are perceived as stationary, even though eye movements cause frequent displacement of their location on the retina. Previous studies found that, after a single saccade, memory of retinotopic locations is more accurate than memory of spatiotopic locations. However, it is not known whether various aspects of natural viewing affect the retinotopic reference frame advantage. We found that the retinotopic advantage may in part depend on a retinal afterimage, which can be effectively nullified through backwards masking. Moreover, in the presence of natural scenes, spatiotopic memory is more accurate than retinotopic memory, but only when subjects are provided sufficient time to process the scene before the eye movement. Our results demonstrate that retinotopic memory is not always more accurate than spatiotopic memory and that the fidelity of memory traces in both reference frames are sensitive to the presence of contextual cues.

Task-related activity in human visual cortex.

The brain exhibits widespread endogenous responses in the absence of visual stimuli, even at the earliest stages of visual cortical processing. Such responses have been studied in monkeys using optical imaging with a limited field of view over visual cortex. Here, we used functional MRI (fMRI) in human participants to study the link between arousal and endogenous responses in visual cortex. The response that we observed was tightly entrained to task timing, was spatially extensive, and was independent of visual stimulation. We found that this response follows dynamics similar to that of pupil size and heart rate, suggesting that task-related activity is related to arousal. Finally, we found that higher reward increased response amplitude while decreasing its trial-to-trial variability (i.e., the noise). Computational simulations suggest that increased temporal precision underlies both of these observations. Our findings are consistent with optical imaging studies in monkeys and support the notion that arousal increases precision of neural activity.

Stimulus vignetting and orientation selectivity in human visual cortex.

Neural selectivity to orientation is one of the simplest and most thoroughly-studied cortical sensory features. Here, we show that a large body of research that purported to measure orientation tuning may have in fact been inadvertently measuring sensitivity to second-order changes in luminance, a phenomenon we term 'vignetting'. Using a computational model of neural responses in primary visual cortex (V1), we demonstrate the impact of vignetting on simulated V1 responses. We then used the model to generate a set of predictions, which we confirmed with functional MRI experiments in human observers. Our results demonstrate that stimulus vignetting can wholly determine the orientation selectivity of responses in visual cortex measured at a macroscopic scale, and suggest a reinterpretation of a well-established literature on orientation processing in visual cortex.

Functional MRI Representational Similarity Analysis Reveals a Dissociation between Discriminative and Relative Location Information in the Human Visual System.

Neural responses in visual cortex are governed by a topographic mapping from retinal locations to cortical responses. Moreover, at the voxel population level early visual cortex (EVC) activity enables accurate decoding of stimuli locations. However, in many cases information enabling one to discriminate between locations (i.e., discriminative information) may be less relevant than information regarding the relative location of two objects (i.e., relative information). For example, when planning to grab a cup, determining whether the cup is located at the same retinal location as the hand is hardly relevant, whereas the location of the cup relative to the hand is crucial for performing the action. We have previously used multivariate pattern analysis techniques to measure discriminative location information, and found the highest levels in EVC, in line with other studies. Here we show, using representational similarity analysis, that availability of discriminative information in fMRI activation patterns does not entail availability of relative information. Specifically, we find that relative location information can be reliably extracted from activity patterns in posterior intraparietal sulcus (pIPS), but not from EVC, where we find the spatial representation to be warped. We further show that this variability in relative information levels between regions can be explained by a computational model based on an array of receptive fields. Moreover, when the model's receptive fields are extended to include inhibitory surround regions, the model can account for the spatial warping in EVC. These results demonstrate how size and shape properties of receptive fields in human visual cortex contribute to the transformation of discriminative spatial representations into relative spatial representations along the visual stream.

Position and Identity Information Available in fMRI Patterns of Activity in Human Visual Cortex.

Parietal cortex is often implicated in visual processing of actions. Action understanding is essentially abstract, specific to the type or goal of action, but greatly independent of variations in the perceived position of the action. If certain parietal regions are involved in action understanding, then we expect them to show these generalization and selectivity properties. However, additional functions of parietal cortex, such as self-action control, may impose other demands by requiring an accurate representation of the location of graspable objects. Therefore, the dimensions along which responses are modulated may indicate the functional role of specific parietal regions. Here, we studied the degree of position invariance and hand/object specificity during viewing of tool-grasping actions. To that end, we characterize the information available about location, hand, and tool identity in the patterns of fMRI activation in various cortical areas: early visual cortex, posterior intraparietal sulcus, anterior superior parietal lobule, and the ventral object-specific lateral occipital complex. Our results suggest a gradient within the human dorsal stream: along the posterior-anterior axis, position information is gradually lost, whereas hand and tool identity information is enhanced. This may reflect a gradual transformation of visual input from an initial retinotopic representation in early visual areas to an abstract, position-invariant representation of viewed action in anterior parietal cortex. SIGNIFICANCE STATEMENT: Since the seminal study of Goodale and Milner (1992), there is general agreement that visual processing is largely divided between a ventral and dorsal stream specializing in object recognition and vision for action, respectively. Here, we address the specific representation of viewed actions. Specifically, we study the degree of position invariance and hand/object manipulation specificity in the human visual pathways, characterizing the information available in patterns of fMRI activation during viewing of object-grasping videos, which appeared in different retinal locations. We find converging evidence for a gradient within the dorsal stream: along the posterior-anterior axis, position information is gradually lost, whereas hand and action identity information is enhanced, leading to an abstract, position-invariant representation of viewed action in the anterior parietal cortex.

Fingerprints of Learned Object Recognition Seen in the fMRI Activation Patterns of Lateral Occipital Complex.

One feature of visual processing in the ventral stream is that cortical responses gradually depart from the physical aspects of the visual stimulus and become correlated with perceptual experience. Thus, unlike early retinotopic areas, the responses in the object-related lateral occipital complex (LOC) are typically immune to parameter changes (e.g., contrast, location, etc.) when these do not affect recognition. Here, we use a complementary approach to highlight changes in brain activity following a shift in the perceptual state (in the absence of any alteration in the physical image). Specifically, we focus on LOC and early visual cortex (EVC) and compare their functional magnetic resonance imaging (fMRI) responses to degraded object images, before and after fast perceptual learning that renders initially unrecognized objects identifiable. Using 3 complementary analyses, we find that, in LOC, unlike EVC, learned recognition is associated with a change in the multivoxel response pattern to degraded object images, such that the response becomes significantly more correlated with that evoked by the intact version of the same image. This provides further evidence that the coding in LOC reflects the recognition of visual objects.