Individual differences link sensory processing and motor control.

Research on saccadic and pursuit eye movements led to great advances in our understanding of sensorimotor processing and human behavior. However, studies often have focused on isolated saccadic and pursuit eye movements measured with respect to different sensory information (static vs. dynamic targets). Here, we leveraged interindividual differences across a carefully balanced combination of different tasks to demonstrate that critical links in the control of oculomotor behavior were previously missed. We observed correlations in eye movement behavior across tasks, but only when compared with the same sensory information (e.g., pursuit gain and accuracy of saccades to moving targets). Within the same task, the coordination of saccadic and pursuit eye movements was tailored to the strengths of the individual: observers with more accurate saccades to moving targets rely on them more to catch up with moving targets. Our results have profound implications for the theoretical understanding of sensorimotor processing for oculomotor control. They necessitate a reevaluation of previous data used to map brain circuits for saccadic and pursuit eye movements measured with different types of relevant sensory information. Additionally, they underscore the importance of moving beyond average observations to embrace individual differences as a rich source of information. These individual differences not only reveal the strengths and weaknesses of observers. When combined across different tasks, they allow insights about why observers behave differently in a given task. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Oculomotor routines for perceptual judgments.

In everyday life we frequently make simple visual judgments about object properties, for example, how big or wide is a certain object? Our goal is to test whether there are also task-specific oculomotor routines that support perceptual judgments, similar to the well-established exploratory routines for haptic perception. In a first study, observers saw different scenes with two objects presented in a photorealistic virtual reality environment. Observers were asked to judge which of two objects was taller or wider while gaze was tracked. All tasks were performed with the same set of virtual objects in the same scenes, so that we can compare spatial characteristics of exploratory gaze behavior to quantify oculomotor routines for each task. Width judgments showed fixations around the center of the objects with larger horizontal spread. In contrast, for height judgments, gaze was shifted toward the top of the objects with larger vertical spread. These results suggest specific strategies in gaze behavior that presumably are used for perceptual judgments. To test the causal link between oculomotor behavior and perception, in a second study, observers could freely gaze at the object or we introduced a gaze-contingent setup forcing observers to fixate specific positions on the object. Discrimination performance was similar between free-gaze and the gaze-contingent conditions for width and height judgments. These results suggest that although gaze is adapted for different tasks, performance seems to be based on a perceptual strategy, independent of potential cues that can be provided by the oculomotor system.

The execution of saccadic eye movements suppresses visual processing of both color and luminance in the early visual cortex of humans.

Our eyes execute rapid, directional movements known as saccades, occurring several times per second, to focus on objects of interest in our environment. During these movements, visual sensitivity is temporarily reduced. Despite numerous studies on this topic, the underlying mechanism remains elusive, including a lingering debate on whether saccadic suppression affects the parvocellular visual pathway. To address this issue, we conducted a study employing steady-state visual evoked potentials (SSVEPs) elicited by chromatic and luminance stimuli while observers performed saccadic eye movements. We also employed an innovative analysis pipeline to enhance the signal-to-noise ratio, yielding superior results compared to the previous method. Our findings revealed a clear suppression effect on SSVEP signals during saccades compared to fixation periods. Notably, this suppression effect was comparable for both chromatic and luminance stimuli. We went further to measure the suppression effect across various contrast levels, which enabled us to model SSVEP responses with contrast response functions. The results suggest that saccades primarily reduce response gain without significantly affecting contrast gain and that this reduction applies uniformly to both chromatic and luminance pathways. In summary, our study provides robust evidence that saccades similarly suppress visual processing in both the parvocellular and magnocellular pathways within the human early visual cortex, as indicated by SSVEP responses. The observation that saccadic eye movements impact response gain rather than contrast gain implies that they influence visual processing through a multiplicative mechanism.NEW & NOTEWORTHY The present study demonstrates that saccadic eye movements reduce the processing of both luminance and chromatic stimuli in the early visual cortex of humans. By modeling the contrast response function, the study further shows that saccades affect visual processing by reducing the response gain rather than altering the contrast gain, suggesting that a multiplicative mechanism of visual attenuation affects both parvocellular and magnocellular pathways.

Color constancy mechanisms in virtual reality environments.

Prior research has demonstrated high levels of color constancy in real-world scenarios featuring single light sources, extensive fields of view, and prolonged adaptation periods. However, exploring the specific cues humans rely on becomes challenging, if not unfeasible, with actual objects and lighting conditions. To circumvent these obstacles, we employed virtual reality technology to craft immersive, realistic settings that can be manipulated in real time. We designed forest and office scenes illuminated by five colors. Participants selected a test object most resembling a previously shown achromatic reference. To study color constancy mechanisms, we modified scenes to neutralize three contributors: local surround (placing a uniform-colored leaf under test objects), maximum flux (keeping the brightest object constant), and spatial mean (maintaining a neutral average light reflectance), employing two methods for the latter: changing object reflectances or introducing new elements. We found that color constancy was high in conditions with all cues present, aligning with past research. However, removing individual cues led to varied impacts on constancy. Local surrounds significantly reduced performance, especially under green illumination, showing strong interaction between greenish light and rose-colored contexts. In contrast, the maximum flux mechanism barely affected performance, challenging assumptions used in white balancing algorithms. The spatial mean experiment showed disparate effects: Adding objects slightly impacted performance, while changing reflectances nearly eliminated constancy, suggesting human color constancy relies more on scene interpretation than pixel-based calculations.

Color constancy in real-world settings.

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

A deep new look at color.

Bowers et al. counter deep neural networks (DNNs) as good models of human visual perception. From our color perspective we feel their view is based on three misconceptions: A misrepresentation of the state-of-the-art of color perception; the type of model required to move the field forward; and the attribution of shortcomings to DNN research that are already being resolved.

Context effects on the perception of saturation of fruit colors in still-life paintings.

Still-life painters, especially of the so-called Golden Age (17th century) in the Netherlands, are famous for their masterful techniques of rendering reality. Their amazing abilities to depict different material properties of fruits and flowers are stunning. But how important are these careful arrangements of different objects for the perception of an individual item? Is the perceived color saturation of a single fruit influenced by its surrounding context? We selected fruits in still-life paintings as stimuli to investigate whether and how perceived saturations of fruits were affected by their original contexts. In our study, we focused especially on effects of five context properties: complementary colors, chromatic and luminance contrast, object overlap, and surround variance. Six fruit varieties depicted in high-quality digital reproductions of 48 classic and eight varieties in 64 more recent, modern still-life paintings were selected. In a single trial, eight images of fruits of the same variety appeared on a neutral gray background; half were single fruit cutouts, and the other half were the same fruits embedded in their circular contexts. Fifteen participants ranked all eight images according to perceived color saturations of the fruits. Saturation ratings showed a high agreement of 77%. Surrounding contexts led to an increase in perceived saturation of central fruits. This effect was mainly driven by object overlap, the presence of the central fruit type also in the context, and surround variance. Chroma contrast between fruits and contexts decreased saturation significantly. No significant context effects were found for complementary colors or luminance contrast. Our results show that in paintings, many of the cues that are usually experimentally isolated occur in interesting combinations and lead to an increase in perceived saturation that makes fruit objects more appealing and convincing.

Cues for predictive eye movements in naturalistic scenes.

We previously compared following of the same trajectories with eye movements, but either as an isolated targets or embedded in a naturalistic scene-in this case, the movement of a puck in an ice hockey game. We observed that the oculomotor system was able to leverage the contextual cues available in the naturalistic scene to produce predictive eye movements. In this study, we wanted to assess which factors are critical for achieving this predictive advantage by manipulating four factors: the expertise of the viewers, the amount of available peripheral information, and positional and kinematic cues. The more peripheral information became available (by manipulating the area of the video that was visible), the better the predictions of all observers. However, expert ice hockey fans were consistently better at predicting than novices and used peripheral information more effectively for predictive saccades. Artificial cues about player positions did not lead to a predictive advantage, whereas impairing the causal structure of kinematic cues by playing the video in reverse led to a severe impairment. When videos were flipped vertically to introduce more difficult kinematic cues, predictive behavior was comparable to watching the original videos. Together, these results demonstrate that, when contextual information is available in naturalistic scenes, the oculomotor system is successfully integrating them and is not relying only on low-level information about the target trajectory. Critical factors for successful prediction seem to be the amount of available information, experience with the stimuli, and the availability of intact kinematic cues for player movements.

Color and gloss constancy under diverse lighting environments.

When we look at an object, we simultaneously see how glossy or matte it is, how light or dark, and what color. Yet, at each point on the object's surface, both diffuse and specular reflections are mixed in different proportions, resulting in substantial spatial chromatic and luminance variations. To further complicate matters, this pattern changes radically when the object is viewed under different lighting conditions. The purpose of this study was to simultaneously measure our ability to judge color and gloss using an image set capturing diverse object and illuminant properties. Participants adjusted the hue, lightness, chroma, and specular reflectance of a reference object so that it appeared to be made of the same material as a test object. Critically, the two objects were presented under different lighting environments. We found that hue matches were highly accurate, except for under a chromatically atypical illuminant. Chroma and lightness constancy were generally poor, but these failures correlated well with simple image statistics. Gloss constancy was particularly poor, and these failures were only partially explained by reflection contrast. Importantly, across all measures, participants were highly consistent with one another in their deviations from constancy. Although color and gloss constancy hold well in simple conditions, the variety of lighting and shape in the real world presents significant challenges to our visual system's ability to judge intrinsic material properties.

Laplacian reference is optimal for steady-state visual-evoked potentials.

Steady-state visual-evoked potentials (SSVEPs) are widely used in human neuroscience studies and applications such as brain-computer interfaces (BCIs). Surprisingly, no previous study has systematically evaluated different reference methods for SSVEP analysis, despite that signal reference is crucial for the proper assessment of neural activities. In the present study, using four datasets from our previous SSVEP studies (Chen J, Valsecchi M, Gegenfurtner KR. J Neurophysiol 118: 749-754, 2017; Chen J, Valsecchi M, Gegenfurtner KR. Neuropsychologia 102: 206-216, 2017; Chen J, McManus M, Valsecchi M, Harris LR, Gegenfurtner KR. J Vis 19: 8, 2019) and three public datasets from other studies (Baker DH, Vilidaite G, Wade AR. PLoS Comput Biol 17: e1009507, 2021; Lygo FA, Richard B, Wade AR, Morland AB, Baker DH. NeuroImage 230: 117780, 2021; Vilidaite G, Norcia AM, West RJH, Elliott CJH, Pei F, Wade AR, Baker DH. Proc R Soc B 285: 20182255, 2018), we compared four reference methods: monopolar reference, common average reference, averaged-mastoids reference, and Laplacian reference. The quality of the resulting SSVEP signals was compared in terms of both signal-to-noise ratios (SNRs) and reliability. The results showed that Laplacian reference, which uses signals at the maximally activated electrode after subtracting the average of the nearby electrodes to reduce common noise, gave rise to the highest SNRs. Furthermore, the Laplacian reference resulted in SSVEP signals that were highly reliable across recording sessions or trials. These results suggest that Laplacian reference is optimal for SSVEP studies and applications. Laplacian reference is especially advantageous for SSVEP experiments where short preparation time is preferred as it requires only data from the maximally activated electrode and a few surrounding electrodes.NEW & NOTEWORTHY The present study provides a comprehensive evaluation of the use of different reference methods for steady-state visual-evoked potentials (SSVEPs) and has found that Laplacian reference increases signal-to-noise ratios (SNRs) and enhances reliabilities of SSVEP signals. Thus, the results suggest that Laplacian reference is optimal for SSVEP analysis.

Object-based color constancy in a deep neural network.

Color constancy refers to our capacity to see consistent colors under different illuminations. In computer vision and image processing, color constancy is often approached by explicit estimation of the scene's illumination, followed by an image correction. In contrast, color constancy in human vision is typically measured as the capacity to extract color information about objects and materials in a scene consistently throughout various illuminations, which goes beyond illumination estimation and might require some degree of scene and color understanding. Here, we pursue an approach with deep neural networks that tries to assign reflectances to individual objects in the scene. To circumvent the lack of massive ground truth datasets labeled with reflectances, we used computer graphics to render images. This study presents a model that recognizes colors in an image pixel by pixel under different illumination conditions.

Perception of saturation in natural objects.

The distribution of colors across a surface depends on the interaction between its surface properties, its shape, and the lighting environment. Shading, chroma, and lightness are positively correlated: points on the object that have high luminance also have high chroma. Saturation, typically defined as the ratio of chroma to lightness, is therefore relatively constant across an object. Here we explored to what extent this relationship affects perceived saturation of an object. Using images of hyperspectral fruit and rendered matte objects, we manipulated the lightness-chroma correlation (positive or negative) and asked observers which of two objects appeared more saturated. Despite the negative-correlation stimulus having greater mean and maximum chroma, lightness, and saturation than the positive, observers overwhelmingly chose the positive as more saturated. This suggests that simple colorimetric statistics do not accurately represent perceived saturation of objects-observers likely base their judgments on interpretations about the cause of the color distribution.

The time course of chromatic adaptation in human early visual cortex revealed by SSVEPs.

Previous studies have identified at least two components of chromatic adaptation: a rapid component with a time scale between tens of milliseconds to a few seconds, and a slow component with a half-life of about 10 to 30 seconds. The basis of the rapid adaptation probably lies in receptor adaptation at the retina. The neural substrate for the slow adaptation remains unclear, although previous psychophysical results hint at the early visual cortex. A promising approach to investigate adaptation effects in the visual cortex is to analyze steady-state visual evoked potentials (SSVEPs) elicited by chromatic stimuli, which typically use long durations of stimulation. Here, we re-analyzed the data from two previous pattern-reversal SSVEP studies. In these experiments (N = 49 observers in total), SSVEPs were elicited by counter-phase flickering color- or luminance-defined grating stimuli for 150 seconds in each trial. By analyzing SSVEPs with short time windows, we found that chromatic SSVEP responses decreased with increasing stimulation duration and reached a lower asymptote within a minute of stimulation. The luminance SSVEPs did not show any systematic adaptation. The time course of chromatic SSVEPs can be well described by an exponential decay function with a half-life of about 20 seconds, which is very close to previous psychophysical reports. Despite the difference in stimuli between the current and previous studies, the coherent time course may indicate a more general adaptation mechanism in the early visual cortex. In addition, the current result also provides a guide for future color SSVEP studies in terms of either avoiding or exploiting this adaptation effect.

Contrast sensitivity function in deep networks.

The contrast sensitivity function (CSF) is a fundamental signature of the visual system that has been measured extensively in several species. It is defined by the visibility threshold for sinusoidal gratings at all spatial frequencies. Here, we investigated the CSF in deep neural networks using the same 2AFC contrast detection paradigm as in human psychophysics. We examined 240 networks pretrained on several tasks. To obtain their corresponding CSFs, we trained a linear classifier on top of the extracted features from frozen pretrained networks. The linear classifier is exclusively trained on a contrast discrimination task with natural images. It has to find which of the two input images has higher contrast. The network's CSF is measured by detecting which one of two images contains a sinusoidal grating of varying orientation and spatial frequency. Our results demonstrate characteristics of the human CSF are manifested in deep networks both in the luminance channel (a band-limited inverted U-shaped function) and in the chromatic channels (two low-pass functions of similar properties). The exact shape of the networks' CSF appears to be task-dependent. The human CSF is better captured by networks trained on low-level visual tasks such as image-denoising or autoencoding. However, human-like CSF also emerges in mid- and high-level tasks such as edge detection and object recognition. Our analysis shows that human-like CSF appears in all architectures but at different depths of processing, some at early layers, while others in intermediate and final layers. Overall, these results suggest that (i) deep networks model the human CSF faithfully, making them suitable candidates for applications of image quality and compression, (ii) efficient/purposeful processing of the natural world drives the CSF shape, and (iii) visual representation from all levels of visual hierarchy contribute to the tuning curve of the CSF, in turn implying a function which we intuitively think of as modulated by low-level visual features may arise as a consequence of pooling from a larger set of neurons at all levels of the visual system.

Emergent color categorization in a neural network trained for object recognition.

Color is a prime example of categorical perception, yet it is unclear why and how color categories emerge. On the one hand, prelinguistic infants and several animals treat color categorically. On the other hand, recent modeling endeavors have successfully utilized communicative concepts as the driving force for color categories. Rather than modeling categories directly, we investigate the potential emergence of color categories as a result of acquiring visual skills. Specifically, we asked whether color is represented categorically in a convolutional neural network (CNN) trained to recognize objects in natural images. We systematically trained new output layers to the CNN for a color classification task and, probing novel colors, found borders that are largely invariant to the training colors. The border locations were confirmed using an evolutionary algorithm that relies on the principle of categorical perception. A psychophysical experiment on human observers, analogous to our primary CNN experiment, shows that the borders agree to a large degree with human category boundaries. These results provide evidence that the development of basic visual skills can contribute to the emergence of a categorical representation of color.

Deep neural models for color classification and color constancy.

Color constancy is our ability to perceive constant colors across varying illuminations. Here, we trained deep neural networks to be color constant and evaluated their performance with varying cues. Inputs to the networks consisted of two-dimensional images of simulated cone excitations derived from three-dimensional (3D) rendered scenes of 2,115 different 3D shapes, with spectral reflectances of 1,600 different Munsell chips, illuminated under 278 different natural illuminations. The models were trained to classify the reflectance of the objects. Testing was done with four new illuminations with equally spaced CIEL*a*b* chromaticities, two along the daylight locus and two orthogonal to it. High levels of color constancy were achieved with different deep neural networks, and constancy was higher along the daylight locus. When gradually removing cues from the scene, constancy decreased. Both ResNets and classical ConvNets of varying degrees of complexity performed well. However, DeepCC, our simplest sequential convolutional network, represented colors along the three color dimensions of human color vision, while ResNets showed a more complex representation.

The role of color in the perception of three-dimensional shape.

The human visual system can derive information about three-dimensional (3D) shape from the structure of light reflected by surfaces. Most research on single static images has focused on the 3D shape information contained in variations of brightness caused by interactions between the illumination and local surface orientation ("shading").1-6 Although color can enhance the recovery of surface shading when color and brightness vary independently,7-9 there is no evidence that color alone provides any information about 3D shape. Here, we show that the wavelength-dependent reflectance of chromatic materials provides information about the 3D shape of translucent materials. We show that different wavelengths of light undergo varying degrees of subsurface light transport, which generates multiple forms of spatial structure: wavelengths that are weakly reflected generate shading-like image structure, linked to 3D surface orientation, whereas wavelengths that penetrate more deeply into the material are primarily constrained by the direction of surface curvature (convexities and concavities).10 Psychophysical experiments demonstrate that the enhanced perception of 3D shape in chromatic translucent surfaces arises from the shading structure generated by weakly reflected wavelengths, which, in turn, generates correlated spatial variations in saturation. These results demonstrate a new functional role for color in the perception of the 3D shape of translucent materials.

Colour Calibration of a Head Mounted Display for Colour Vision Research Using Virtual Reality.

Virtual reality (VR) technology offers vision researchers the opportunity to conduct immersive studies in simulated real-world scenes. However, an accurate colour calibration of the VR head mounted display (HMD), both in terms of luminance and chromaticity, is required to precisely control the presented stimuli. Such a calibration presents significant new challenges, for example, due to the large field of view of the HMD, or the software implementation used for scene rendering, which might alter the colour appearance of objects. Here, we propose a framework for calibrating an HMD using an imaging colorimeter, the I29 (Radiant Vision Systems, Redmond, WA, USA). We examine two scenarios, both with and without using a rendering software for visualisation. In addition, we present a colour constancy experiment design for VR through a gaming engine software, Unreal Engine 4. The colours of the objects of study are chosen according to the previously defined calibration. Results show a high-colour constancy performance among participants, in agreement with recent studies performed on real-world scenarios. Our studies show that our methodology allows us to control and measure the colours presented in the HMD, effectively enabling the use of VR technology for colour vision research.

A change in perspective: The interaction of saccadic and pursuit eye movements in oculomotor control and perception.

Due to the close relationship between oculomotor behavior and visual processing, eye movements have been studied in many different areas of research over the last few decades. While these studies have brought interesting insights, specialization within each research area comes at the potential cost of a narrow and isolated view of the oculomotor system. In this review, we want to expand this perspective by looking at the interactions between the two most important types of voluntary eye movements: saccades and pursuit. Recent evidence indicates multiple interactions and shared signals at the behavioral and neurophysiological level for oculomotor control and for visual perception during pursuit and saccades. Oculomotor control seems to be based on shared position- and velocity-related information, which leads to multiple behavioral interactions and synergies. The distinction between position- and velocity-related information seems to be also present at the neurophysiological level. In addition, visual perception seems to be based on shared efferent signals about upcoming eye positions and velocities, which are to some degree independent of the actual oculomotor response. This review suggests an interactive perspective on the oculomotor system, based mainly on different types of sensory input, and less so on separate subsystems for saccadic or pursuit eye movements.

Electrophysiological evidence for higher-level chromatic mechanisms in humans.

Color vision in humans starts with three types of cones (short [S], medium [M], and long [L] wavelengths) in the retina and three retinal and subcortical cardinal mechanisms, which linearly combine cone signals into the luminance channel (L + M), the red-green channel (L - M), and the yellow-blue channel (S-(L + M)). Chromatic mechanisms at the cortical level, however, are less well characterized. The present study investigated such higher-order chromatic mechanisms by recording electroencephalograms (EEGs) on human observers in a noise masking paradigm. Observers viewed colored stimuli that consisted of a target embedded in noise. Color directions of the target and noise varied independently and systematically in an isoluminant plane of color space. The target was flickering on-off at 3 Hz, eliciting steady-state visual evoked potential (SSVEP) responses. As a result, the masking strength could be estimated from the SSVEP amplitude in the presence of 6 Hz noise. Masking was strongest (i.e. target eliciting smallest SSVEPs) when the target and noise were along the same color direction, and was weakest (i.e. target eliciting highest SSVEPs) when the target and noise were along orthogonal directions. This pattern of results was observed both when the target color varied along the cardinal and intermediate directions, which is evidence for higher-order chromatic mechanisms tuned to intermediate axes. The SSVEP result can be well predicted by a model with multiple broadly tuned chromatic mechanisms. In contrast, a model with only cardinal mechanisms failed to account for the data. These results provide strong electrophysiological evidence for multiple chromatic mechanisms in the early visual cortex of humans.