Oculomotor feature discrimination is cortically mediated.

Eye movements are often directed toward stimuli with specific features. Decades of neurophysiological research has determined that this behavior is subserved by a feature-reweighting of the neural activation encoding potential eye movements. Despite the considerable body of research examining feature-based target selection, no comprehensive theoretical account of the feature-reweighting mechanism has yet been proposed. Given that such a theory is fundamental to our understanding of the nature of oculomotor processing, we propose an oculomotor feature-reweighting mechanism here. We first summarize the considerable anatomical and functional evidence suggesting that oculomotor substrates that encode potential eye movements rely on the visual cortices for feature information. Next, we highlight the results from our recent behavioral experiments demonstrating that feature information manifests in the oculomotor system in order of featural complexity, regardless of whether the feature information is task-relevant. Based on the available evidence, we propose an oculomotor feature-reweighting mechanism whereby (1) visual information is projected into the oculomotor system only after a visual representation manifests in the highest stage of the cortical visual processing hierarchy necessary to represent the relevant features and (2) these dynamically recruited cortical module(s) then perform feature discrimination via shifting neural feature representations, while also maintaining parity between the feature representations in cortical and oculomotor substrates by dynamically reweighting oculomotor vectors. Finally, we discuss how our behavioral experiments may extend to other areas in vision science and its possible clinical applications.

Motion distractors perturb saccade programming later in time than static distractors.

The mechanism that reweights oculomotor vectors based on visual features is unclear. However, the latency of oculomotor visual activations gives insight into their antecedent featural processing. We compared the oculomotor processing time course of grayscale, task-irrelevant static and motion distractors during target selection by continuously measuring a battery of human saccadic behavioral metrics as a function of time after distractor onset. The motion direction was towards or away from the target and the motion speed was fast or slow. We compared static and motion distractors and observed that both distractors elicited curved saccades and shifted endpoints at short latencies (∼25 ms). After 50 ms, saccade trajectory biasing elicited by motion distractors lagged static distractor trajectory biasing by 10 ms. There were no such latency differences between distractor motion directions or motion speeds. This pattern suggests that additional processing of motion stimuli occurred prior to the propagation of visual information into the oculomotor system. We examined the interaction of distractor processing time (DPT) with two additional factors: saccadic reaction time (SRT) and saccadic amplitude. Shorter SRTs were associated with shorter DPT latencies of biased saccade trajectories. Both SRT and saccadic amplitude were associated with the magnitude of saccade trajectory biases.

Target-Distractor Competition Modulates Saccade Trajectories in Space and Object Space.

Saccade planning and execution can be affected by a multitude of factors present in a target selection task. Recent studies have shown that the similarity between a target and nearby distractors affects the curvature of saccade trajectories, because of target-distractor competition. To further understand the nature of this competition, we varied the distance between and the similarity of complex target and distractor objects in a delayed match-to-sample task to examine their effects on human saccade trajectories and better understand the underlying neural circuitry. For trials with short saccadic reaction times (SRTs) when target-distractor competition is still active, the distractor is attractive and saccade trajectories are deviated toward the distractor. We found a robust effect of distance consistent with saccade vector averaging, whereas the effect of similarity suggested the existence of an object-based suppressive surround. At longer SRTs, there was sufficient time for competition between the objects to complete and the distractor to be repulsive, which resulted in saccade trajectory deviations away from the distractor exhibiting the effects of a spatial suppressive surround. In terms of similarity, as the target-distractor similarity decreased, the initial saccade angle shifted toward the target, reflecting stronger distractor inhibition. There were no interactions between distance and similarity at any point in the time course of target-distractor competition. Together, saccade trajectories reflect target-distractor competition that is affected independently by both spatial and object space suppressive surrounds. The differences in saccade trajectories at short and long SRTs distinguish between active and completed decision-making processes.

Attention to visual motion suppresses neuronal and behavioral sensitivity in nearby feature space.

BACKGROUND: Feature-based attention prioritizes the processing of the attended feature while strongly suppressing the processing of nearby ones. This creates a non-linearity or "attentional suppressive surround" predicted by the Selective Tuning model of visual attention. However, previously reported effects of feature-based attention on neuronal responses are linear, e.g., feature-similarity gain. Here, we investigated this apparent contradiction by neurophysiological and psychophysical approaches. RESULTS: Responses of motion direction-selective neurons in area MT/MST of monkeys were recorded during a motion task. When attention was allocated to a stimulus moving in the neurons' preferred direction, response tuning curves showed its minimum for directions 60-90° away from the preferred direction, an attentional suppressive surround. This effect was modeled via the interaction of two Gaussian fields representing excitatory narrowly tuned and inhibitory widely tuned inputs into a neuron, with feature-based attention predominantly increasing the gain of inhibitory inputs. We further showed using a motion repulsion paradigm in humans that feature-based attention produces a similar non-linearity on motion discrimination performance. CONCLUSIONS: Our results link the gain modulation of neuronal inputs and tuning curves examined through the feature-similarity gain lens to the attentional impact on neural population responses predicted by the Selective Tuning model, providing a unified framework for the documented effects of feature-based attention on neuronal responses and behavior.

Oculomotor system can differentially process red and green colors during saccade programming in the presence of a competing distractor.

Selective attention filters irrelevant information entering our brain to allow for fine-tuning of the relevant information processing. In the visual domain, shifts of attention are most often followed by a saccadic eye movement to objects and places of high relevance. Recent studies have shown that the stimulus color can affect saccade target selection and saccade trajectories. While those saccade modulations are based on perceptual color space, the level in the visual processing hierarchy at which color selection biases saccade programming remains unclear. As color has also been shown to influence manual response inhibition which is a key function of the prefrontal cortex, we hypothesized that the effects of color on executive functions would also inherently affect saccade programming. To test this hypothesis, we measured behavioral performance and saccade metrics during a modified saccadic Stroop task which reflects competition between color words ("RED" and "GREEN") and their color at the level of the prefrontal cortex. Our results revealed that the oculomotor system can differentially process red and green colors when planning a saccade in the presence of a competing distractor.

The role of temporal cortex in the control of attention.

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

Oculomotor target selection is mediated by complex objects.

Oculomotor target selection often requires discriminating visual features, but it remains unclear how oculomotor substrates encoding saccade vectors functionally contribute to this process. One possibility is that oculomotor vector representations (observed directly as physiological activation or inferred from behavioral interference) of potential targets are continuously reweighted by task relevance computed elsewhere in specialized visual modules, whereas an alternative possibility is that oculomotor modules use local featural analyses to actively discriminate potential targets. Strengthening the former account, oculomotor vector representations have longer onset latencies for ventral- (i.e., color) than dorsal-stream features (i.e., luminance), suggesting that oculomotor vector representations originate from featurally relevant specialized visual modules. Here, we extended this reasoning by behaviorally examining whether the onset latency of saccadic interference elicited by visually complex stimuli is greater than is commonly observed for simple stimuli. We measured human saccade metrics (saccade curvature, endpoint deviations, saccade frequency, and error proportion) as a function of time after abrupt distractor onset. Distractors were novel, visually complex, and had to be discriminated from targets to guide saccades. The earliest saccadic interference latency was ∼110 ms, considerably longer than previous experiments, suggesting that sensory representations projected into the oculomotor system are gated to allow for sufficient featural processing to satisfy task demands. Surprisingly, initial oculomotor vector representations encoded features, as we manipulated the visual similarity between targets and distractors and observed increased vector modulation response magnitude and duration when the distractor was highly similar to the target. Oculomotor vector modulation was gradually extinguished over the time course of the experiment.NEW & NOTEWORTHY We challenge the role of the oculomotor system in discriminating features during saccadic target selection. Our data suggest that the onset latency of oculomotor vector representations is scaled by task difficulty and featural complexity, suggesting that featural computations are performed outside of the oculomotor system, which receives the output of these computations only after sufficient visual and cognitive processing. We also challenge the convention that initial oculomotor vector representations are feature invariant, as they encoded task relevance.

Gaze patterns during presentation of fixed and random phase radial frequency patterns.

Radial frequency (RF) patterns, circles which have had their radius modulated as a function of their polar angle, have been used in the examination of the integration of contour information around closed contour patterns. Typically, these patterns have been presented in a random orientation from trial-to-trial in order to maintain spatial uncertainty as to the location of the deformation on the pattern, as it may affect observer strategy and performance. However, the effect of fixed and random orientation (phase) on observer gaze strategies used to discriminate RF patterns has not been directly tested. This study compared fixation patterns across four conditions: fixed phase single cycle; random phase single cycle; fixed phase three cycle; and random phase three cycle RF3 patterns. The results showed that observers fixated on the known location of deformation for the fixed phase single cycle condition but used a more central fixation for the other three conditions. This strategy had a significant effect on observer thresholds for the fixed phase single cycle condition, with greater adherence to the strategy resulting in lower thresholds. It was also found that for the single cycle patterns observers tended to fixate on different locations on the pattern: on the maximum orientation difference from circular for the fixed phase pattern; and on the point of maximum curvature for the random phase pattern. These differences in gaze patterns are likely driven by the underlying local or global processing of the fixed or random phase single cycle patterns, respectively.

Color Modulates Feature Integration.

Bayesian models of object recognition propose the resolution of ambiguity through probabilistic integration of prior experience with available sensory information. Color, even when task-irrelevant, has been shown to modulate high-level cognitive control tasks. However, it remains unclear how color modulations affect lower-level perceptual processing. We investigated whether color affects feature integration using the flash-jump illusion. This illusion occurs when an apparent motion stimulus, a rectangular bar appearing at different locations along a motion trajectory, changes color at a single position. Observers misperceive this color change as occurring farther along the trajectory of motion. This mislocalization error is proposed to be produced by a Bayesian perceptual framework dependent on responses in area V4. Our results demonstrated that the color of the flash modulated the magnitude of the flash-jump illusion such that participants reported less of a shift, i.e., a more veridical flash location, for both red and blue flashes, as compared to green and yellow. Our findings extend color-dependent modulation effects found in higher-order executive functions into lower-level Bayesian perceptual processes. Our results also support the theory that feature integration is a Bayesian process. In this framework, color modulations play an inherent and automatic role as different colors have different weights in Bayesian perceptual processing.

Long-term memory and hippocampal function support predictive gaze control during goal-directed search.

Eye movements during visual search change with prior experience for search stimuli. Previous studies measured these gaze effects shortly after initial viewing, typically during free viewing; it remains open whether the effects are preserved across long delays and for goal-directed search, and which memory system guides gaze. In Experiment 1, we analyzed eye movements of healthy adults viewing novel and repeated scenes while searching for a scene-embedded target. The task was performed across different time points to examine the repetition effects in long-term memory, and memory types were grouped based on explicit recall of targets. In Experiment 2, an amnesic person with bilateral extended hippocampal damage and the age-matched control group performed the same task with shorter intervals to determine whether or not the repetition effects depend on hippocampal function. When healthy adults explicitly remembered repeated target-scene pairs, search time and fixation duration decreased, and gaze was directed closer to the target region, than when they forgot targets. These effects were seen even after a one-month delay from their initial viewing, suggesting the effects are associated with long-term, explicit memory. Saccadic amplitude was not strongly modulated by scene repetition or explicit recall of targets. The amnesic person did not show explicit recall or implicit repetition effects, whereas his control group showed similar patterns to those seen in Experiment 1. The results reveal several aspects of gaze control that are influenced by long-term memory. The dependence of gaze effects on medial temporal lobe integrity support a role for this region in predictive gaze control.

Feed-forward visual processing suffices for coarse localization but fine-grained localization in an attention-demanding context needs feedback processing.

It is well known that simple visual tasks, such as object detection or categorization, can be performed within a short period of time, suggesting the sufficiency of feed-forward visual processing. However, more complex visual tasks, such as fine-grained localization may require high-resolution information available at the early processing levels in the visual hierarchy. To access this information using a top-down approach, feedback processing would need to traverse several stages in the visual hierarchy and each step in this traversal takes processing time. In the present study, we compared the processing time required to complete object categorization and localization by varying presentation duration and complexity of natural scene stimuli. We hypothesized that performance would be asymptotic at shorter presentation durations when feed-forward processing suffices for visual tasks, whereas performance would gradually improve as images are presented longer if the tasks rely on feedback processing. In Experiment 1, where simple images were presented, both object categorization and localization performance sharply improved until 100 ms of presentation then it leveled off. These results are a replication of previously reported rapid categorization effects but they do not support the role of feedback processing in localization tasks, indicating that feed-forward processing enables coarse localization in relatively simple visual scenes. In Experiment 2, the same tasks were performed but more attention-demanding and ecologically valid images were used as stimuli. Unlike in Experiment 1, both object categorization performance and localization precision gradually improved as stimulus presentation duration became longer. This finding suggests that complex visual tasks that require visual scrutiny call for top-down feedback processing.

Higher order, multifeatural object encoding by the oculomotor system.

Previous behavioral and physiological research has demonstrated that as the behavioral relevance of potential saccade goals increases, they elicit more competition during target selection processing as evidenced by increased saccade curvature and neural activity. However, these effects have only been demonstrated for lower order feature singletons, and it remains unclear whether more complicated featural differences between higher order objects also elicit vector modulation. Therefore, we measured human saccades curvature elicited by distractors bilaterally flanking a target during a visual search saccade task and systematically varied subsets of features shared between the two distractors and the target, referred to as objective similarity (OS). Our results demonstrate that saccades deviated away from the distractor highest in OS to the target and that there was a linear relationship between the magnitude of saccade deviation and the number of feature differences between the most similar distractor and the target. Furthermore, an analysis of curvature over the time course of the saccade demonstrated that curvature only occurred in the first 20-30 ms of the movement. Given the multifeatural complexity of the novel stimuli, these results suggest that saccadic target selection processing involves dynamically reweighting vector representations for movement planning to several possible targets based on their behavioral relevance. NEW & NOTEWORTHY We demonstrate that small featural differences between unfamiliar, higher order object representations modulate vector weights during saccadic target selection processing. Such effects have previously only been demonstrated for familiar, simple feature singletons (e.g., color) in which features characterize entire objects. The complexity and novelty of our stimuli suggest that the oculomotor system dynamically receives visual/cognitive information processed in the higher order representational networks of the cortical visual processing hierarchy and integrates this information for saccadic movement planning.

The Attentional Suppressive Surround: Eccentricity, Location-Based and Feature-Based Effects and Interactions.

The Selective Tuning model of visual attention (Tsotsos, 1990) has proposed that the focus of attention is surrounded by an inhibitory zone, eliciting a center-surround attentional distribution. This attentional suppressive surround inhibits irrelevant information which is located close to attended information in physical space (e.g., Cutzu and Tsotsos, 2003; Hopf et al., 2010) or in feature space (e.g., Tombu and Tsotsos, 2008; Störmer and Alvarez, 2014; Bartsch et al., 2017). In Experiment 1, we investigate the interaction between location-based and feature-based surround suppression and hypothesize that the attentional surround suppression would be maximized when spatially adjacent stimuli are also represented closely within a feature map. Our results demonstrate that perceptual discrimination is worst when two similar orientations are presented in proximity to each other, suggesting the interplay of the two surround suppression mechanisms. The Selective Tuning model also predicts that the size of the attentional suppressive surround is determined by the receptive field size of the neuron which optimally processes the attended information. The receptive field size of the processing neurons is tightly associated with stimulus size and eccentricity. Therefore, Experiment 2 tested the hypothesis that the size of the attentional suppressive surround would become larger as stimulus size and eccentricity increase, corresponding to an increase in the neuron's receptive field size. We show that stimulus eccentricity but not stimulus size modulates the size of the attentional suppressive surround. These results are consistent for both low- and high-level features (e.g., orientation and human faces). Overall, the present study supports the existence of the attentional suppressive surround and reveals new properties of this selection mechanism.

Perceptual Color Space Representations in the Oculomotor System Are Modulated by Surround Suppression and Biased Selection.

The oculomotor system utilizes color extensively for planning saccades. Therefore, we examined how the oculomotor system actually encodes color and several factors that modulate these representations: attention-based surround suppression and inherent biases in selecting and encoding color categories. We measured saccade trajectories while human participants performed a memory-guided saccade task with color targets and distractors and examined whether oculomotor target selection processing was functionally related to the CIE (x,y) color space distances between color stimuli and whether there were hierarchical differences between color categories in the strength and speed of encoding potential saccade goals. We observed that saccade planning was modulated by the CIE (x,y) distances between stimuli thus demonstrating that color is encoded in perceptual color space by the oculomotor system. Furthermore, these representations were modulated by (1) cueing attention to a particular color thereby eliciting surround suppression in oculomotor color space and (2) inherent selection and encoding biases based on color category independent of cueing and perceptual discriminability. Since surround suppression emerges from recurrent feedback attenuation of sensory projections, observing oculomotor surround suppression suggested that oculomotor encoding of behavioral relevance results from integrating sensory and cognitive signals that are pre-attenuated based on task demands and that the oculomotor system therefore does not functionally contribute to this process. Second, although perceptual discriminability did partially account for oculomotor processing differences between color categories, we also observed preferential processing of the red color category across various behavioral metrics. This is consistent with numerous previous studies and could not be simply explained by perceptual discriminability. Since we utilized a memory-guided saccade task, this indicates that the biased processing of the red color category does not rely on sustained sensory input and must therefore involve cortical areas associated with the highest levels of visual processing involved in visual working memory.

Rapid accumulation of inhibition accounts for saccades curved away from distractors.

Saccades curved toward a distractor are accompanied by a burst of neuronal activation at the distractor locus in the intermediate layers of the superior colliculus (SCi) ~30 ms before the initiation of a saccade. Although saccades curve away from inactivated SCi loci, whether inhibition is restricted to a similar critical epoch for saccades curved away from a distractor remains unclear. We examined this possibility by modeling human saccade curvature as a function of the time between onset of a task irrelevant luminance- or color-modulated distractor and initiation of an impending saccade, referred to as saccade distractor onset asynchrony (SDOA). Our results demonstrated that 70 ms of luminance-modulated distractor processing or 90 ms of color-modulated distractor processing was required to modulate saccade trajectories. As these behavioral, feature-based differences were temporally consistent with the cortically mediated neurophysiological differences in visual onset latencies between luminance and color stimuli observed in the oculomotor and visual system, this method provides a noninvasive means to estimate the timing of peak activation in the oculomotor system. As such, we modeled SDOA functions separately for saccades curved toward and away from distractors and observed that a similar temporal process determined the magnitude of saccade curvatures in both contexts, suggesting that saccades deviate away from a distractor due to a rapid accumulation of inhibition in the critical epoch before saccade initiation.NEW & NOTEWORTHY In this research article, we propose a novel, noninvasive approach to behaviorally model the time course of competitive oculomotor processing. Our results highly resembled those from previously published neurophysiological experiments utilizing similar oculomotor processing contexts, thus validating our approach. Furthermore, this methodology provided new insights into the underlying neural mechanism subserving oculomotor processing given that we applied it to a context with which the neural mechanism is more contentious, and the results clearly favored one view.

Effector-based attention systems.

Visual processing is known to be enhanced at the end point of eye movements. Feedback within the oculomotor system has been shown to drive these alterations in visual processing. However, we do not simply view the world; we also reach out and interact using our hands. Consequently, it is not surprising that visual processing has also been shown to be altered in near-hand space. A growing body of work documents a myriad of alterations in near-hand visual processing, with little consensus on the neural underpinnings of the effect of the hand. Since movement of the eyes and hands is governed by parallel frontoparietal networks and since within the oculomotor system feedback from these motor control regions has been shown to drive enhanced visual processing at saccade end points, it is plausible that a similar feedback mechanism is at play in near-hand improvements in visual processing. Here, we compare and contrast oculomotor-driven and hand-driven changes in visual processing and provide support for the hypothesis that feedback within the reaching and grasping systems enhances visual processing near the hand in a novel way.

An Eye in the Palm of Your Hand: Alterations in Visual Processing Near the Hand, a Mini-Review.

Feedback within the oculomotor system improves visual processing at eye movement end points, also termed a visual grasp. We do not just view the world around us however, we also reach out and grab things with our hands. A growing body of literature suggests that visual processing in near-hand space is altered. The control systems for moving either the eyes or the hands rely on parallel networks of fronto-parietal regions, which have feedback connections to visual areas. Since the oculomotor system effects on visual processing occur through feedback, both through the motor plan and the motor efference copy, a parallel system where reaching and/or grasping motor-related activity also affects visual processing is likely. Areas in the posterior parietal cortex, for example, receive proprioceptive and visual information used to guide actions, as well as motor efference signals. This trio of information channels is all that would be necessary to produce spatial allocation of reach-related visual attention. We review evidence from behavioral and neurophysiological studies that support the hypothesis that feedback from the reaching and/or grasping motor control networks affects visual processing while noting ways in which it differs from that seen within the oculomotor system. We also suggest that object affordances may represent the neural mechanism through which certain object features are selected for preferential processing when stimuli are near the hand. Finally, we summarize the two effector-based feedback systems and discuss how having separate but parallel effector systems allows for efficient decoupling of eye and hand movements.

Response Inhibition Is Facilitated by a Change to Red Over Green in the Stop Signal Paradigm.

Actions are informed by the complex interactions of response execution and inhibition networks. These networks integrate sensory information with internal states and behavioral goals to produce an appropriate action or to update an ongoing action. Recent investigations have shown that, behaviorally, attention is captured through a hierarchy of colors. These studies showed how the color hierarchy affected visual processing. To determine whether the color hierarchy can be extended to higher level executive functions such as response execution and inhibition, we conducted several experiments using the stop-signal task (SST). In the first experiment, we modified the classic paradigm so that the go signals could vary in task-irrelevant color, with an auditory stop signal. We found that the task-irrelevant color of the go signals did not differentially affect response times. In the second experiment we determined that making the color of the go signal relevant for response selection still did not affect reaction times(RTs) and, thus, execution. In the third experiment, we modified the paradigm so that the stop signal was a task relevant change in color of the go signal. The mean RT to the red stop signal was approximately 25 ms faster than to the green stop signal. In other words, red stop signals facilitated response inhibition more than green stop signals, however, there was no comparative facilitation of response execution. These findings suggest that response inhibition, but not execution, networks are sensitive to differences in color salience. They also suggest that the color hierarchy is based on attentional networks and not simply on early sensory processing.