Retinal eccentricity modulates saliency-driven but not relevance-driven visual selection.

Where we move our eyes during visual search is controlled by the relative saliency and relevance of stimuli in the visual field. However, the visual field is not homogeneous, as both sensory representations and attention change with eccentricity. Here we present an experiment investigating how eccentricity differences between competing stimuli affect saliency- and relevance-driven selection. Participants made a single eye movement to a predefined orientation singleton target that was simultaneously presented with an orientation singleton distractor in a background of multiple homogenously oriented other items. The target was either more or less salient than the distractor. Moreover, each of the two singletons could be presented at one of three different retinal eccentricities, such that both were presented at the same eccentricity, one eccentricity value apart, or two eccentricity values apart. The results showed that selection was initially determined by saliency, followed after about 300 ms by relevance. In addition, observers preferred to select the closer over the more distant singleton, and this central selection bias increased with increasing eccentricity difference. Importantly, it largely emerged within the same time window as the saliency effect, thereby resulting in a net reduction of the influence of saliency on the selection outcome. In contrast, the relevance effect remained unaffected by eccentricity. Together, these findings demonstrate that eccentricity is a major determinant of selection behavior, even to the extent that it modifies the relative contribution of saliency in determining where people move their eyes.

Does crowding predict conjunction search? An individual differences approach.

Searching for objects in the visual environment is an integral part of human behavior. Most of the information used during such visual search comes from the periphery of our vision, and understanding the basic mechanisms of search therefore requires taking into account the inherent limitations of peripheral vision. Our previous work using an individual differences approach has shown that one of the major factors limiting peripheral vision (crowding) is predictive of single feature search, as reflected in response time and eye movement measures. Here we extended this work, by testing the relationship between crowding and visual search in a conjunction-search paradigm. Given that conjunction search involves more fine-grained discrimination and more serial behavior, we predicted it would be strongly affected by crowding. We tested sixty participants with regard to their sensitivity to both orientation and color-based crowding (as measured by critical spacing) and their efficiency in searching for a color/orientation conjunction (as indicated by manual response times and eye movements). While the correlations between the different crowding tasks were high, the correlations between the different crowding measures and search performance were relatively modest, and no higher than those previously observed for single-feature search. Instead, observers showed very strong color selectivity during search. The results suggest that conjunction search behavior relies more on top-down guidance (here by color) and is therefore relatively less determined by individual differences in sensory limitations as caused by crowding.

The effects of eccentricity on attentional capture.

Visual attention may be captured by an irrelevant yet salient distractor, thereby slowing search for a relevant target. This phenomenon has been widely studied using the additional singleton paradigm in which search items are typically all presented at one and the same eccentricity. Yet, differences in eccentricity may well bias the competition between target and distractor. Here we investigate how attentional capture is affected by the relative eccentricities of a target and a distractor. Participants searched for a shape-defined target in a grid of homogeneous nontargets of the same color. On 75% of trials, one of the nontarget items was replaced by a salient color-defined distractor. Crucially, target and distractor eccentricities were independently manipulated across three levels of eccentricity (i.e., near, middle, and far). Replicating previous work, we show that the presence of a distractor slows down search. Interestingly, capture as measured by manual reaction times was not affected by target and distractor eccentricity, whereas capture as measured by the eyes was: items close to fixation were more likely to be selected than items presented further away. Furthermore, the effects of target and distractor eccentricity were largely additive, suggesting that the competition between saliency- and relevance-driven selection was modulated by an independent eccentricity-based spatial component. Implications of the dissociation between manual and oculomotor responses are also discussed.

A CODE model bridging crowding in sparse and dense displays.

Visual crowding is arguably the strongest limitation imposed on extrafoveal vision, and is a relatively well-understood phenomenon. However, most investigations and theories are based on sparse displays consisting of a target and at most a handful of flanker objects. Recent findings suggest that the laws thought to govern crowding may not hold for densely cluttered displays, and that grouping and nearest neighbour effects may be more important. Here we present a computational model that accounts for crowding effects in both sparse and dense displays. The model is an adaptation and extension of an earlier model that has previously successfully accounted for spatial clustering, numerosity and object-based attention phenomena. Our model combines grouping by proximity and similarity with a nearest neighbour rule, and defines crowding as the extent to which target and flankers fail to segment. We show that when the model is optimized for explaining crowding phenomena in classic, sparse displays, it also does a good job in capturing novel crowding patterns in dense displays, in both existing and new data sets. The model thus ties together different principles governing crowding, specifically Bouma's law, grouping, and nearest neighbour similarity effects.

A matter of availability: sharper tuning for memorized than for perceived stimulus features.

Our visual environment is relatively stable over time. An optimized visual system could capitalize on this by devoting less representational resources to objects that are physically present. The vividness of subjective experience, however, suggests that externally available (perceived) information is more strongly represented in neural signals than memorized information. To distinguish between these opposing predictions, we use EEG multivariate pattern analysis to quantify the representational strength of task-relevant features in anticipation of a change-detection task. Perceptual availability was manipulated between experimental blocks by either keeping the stimulus available on the screen during a 2-s delay period (perception) or removing it shortly after its initial presentation (memory). We find that task-relevant (attended) memorized features are more strongly represented than irrelevant (unattended) features. More importantly, we find that task-relevant features evoke significantly weaker representations when they are perceptually available compared with when they are unavailable. These findings demonstrate that, contrary to what subjective experience suggests, vividly perceived stimuli elicit weaker neural representations (in terms of detectable multivariate information) than the same stimuli maintained in visual working memory. We hypothesize that an efficient visual system spends little of its limited resources on the internal representation of information that is externally available anyway.

How do people find pairs?

Humans continuously scan their visual environment for relevant information. Such visual search behavior has typically been studied with tasks in which the search goal is constant and well-defined, requiring relatively little interplay between memory and orienting. Here we studied a situation in which the target is not known in advance, and instead, memory needs to be dynamically updated during the actual search. Observers compared two simultaneously presented arrays of objects for any matching pair of items-a task that requires continuous comparisons between what is seen now and what was seen a few moments ago. To manipulate the balance between memorizing and scanning, we ran two versions of the task. In an eye-tracking version, the objects were continuously available and could be scanned with relative ease. The results suggested that observers preferred scanning over memorizing. In a mouse-tracking version, perceptual availability was limited, and scanning was slowed. Now observers substantially increased their memory use. Thus, the results revealed a flexible and dynamic interplay between memory and perception. The findings aid in further bridging the research fields of attention and memory. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The eyes prefer targets nearby fixation: Quantifying eccentricity-dependent attentional biases in oculomotor selection.

An important function of peripheral vision is to provide the target of the next eye movement. Here we investigate the extent to which the eyes are biased to select a target closer to fixation over one further away. Participants were presented with displays containing two identical singleton targets and were asked to move their eyes to either one of them. The targets could be presented at three different eccentricities relative to central fixation. In one condition both singletons were presented at the same eccentricity, providing an estimate of the speed of selection at each of the eccentricities. The saccadic latency distributions from this same-eccentricity condition were then used to predict the selection bias when both targets were presented at different eccentricities. The results show that when targets are presented at different eccentricities, participants are biased to select the item closest to fixation. This eccentricity-based bias was considerably stronger than predicted on the basis of saccadic latency distributions in the same-eccentricity condition. This rules out speed of processing per se as a sole explanation for such a bias. Instead, the results are consistent with attentional competition being weighted in favour of items close to fixation.

Visual working memory representations bias attention more when they are the target of an action plan.

Attention has frequently been regarded as an emergent property of linking sensory representations to action plans. It has recently been proposed that similar mechanisms may operate within visual working memory (VWM), such that linking an object in VWM to an action plan strengthens its sensory memory representation, which then expresses as an attentional bias. Here we directly tested this hypothesis by comparing attentional biases induced by VWM representations which were the target of a future action, to those induced by VWM representations that were equally task-relevant, but not the direct target of action. We predicted that the first condition would result in a more prioritized memory state and hence stronger attentional biases. Specifically, participants memorized a geometric shape for a subsequent memory test. At test, in case of a match, participants either had to perform a grip movement on the matching object (action condition), or perform the same movement, but on an unrelated object (control condition). To assess any attentional biases, during the delay period between memorandum and test, participants performed a visual selection task in which either the target was surrounded by the memorized shape (congruent trials) or a distractor (incongruent trials). Eye movements were measured as a proxy for attentional priority. We found a significant interaction for saccade latencies between action condition and shape congruency, reflecting more pronounced VWM-based attentional biases in the action condition. Our results are consistent with the idea that action plans prioritize sensory representations in VWM.

Multi-res: An interface for improving reading without central vision.

Loss of sharp foveal vision, as is inherent to Macular Degeneration (MD), severely impacts reading. One strategy for preserving patients' reading ability involves a one-by-one serial visual presentation (SVP) of words, whereby words are viewed extrafoveally. However, the method is limited as patients often retain the natural tendency to foveate words, thus bringing those words in the scotomal region. Additionally, SVP offers no compensation for the fact that orthographic input is degraded outside the fovea. Addressing these issues, here we tested a novel interface wherein texts are presented word-by-word, but with multiple repetitions (Multi-Res) of each word being displayed simultaneously around the fovea. We hypothesized that the Multi-Res setup would lead readers to make fewer detrimental eye movements, and to recognize words faster as a consequence of multiplied orthographic input. We used eye-tracking to simulate a gaze-contingent foveal scotoma in normally-sighted participants, who read words either in classic SVP or in Multi-Res mode. In line with our hypotheses, reading was drastically better in the Multi-Res condition, with faster recognition, fewer saccades and increased oculomotor stability. We surmise that the Multi-Res method has good potential for improving reading in central vision loss, over and above classic SVP techniques.

Visual Working Memory Adapts to the Nature of Anticipated Interference.

Visual working memory has been proven to be relatively robust against interference. However, little is known on whether such robust coding is obligatory, or can be flexibly recruited depending on its expected usefulness. To address this, participants remembered both the color and orientation of a grating. During the maintenance, we inserted a secondary color/orientation memory task, interfering with the primary task. Crucially, we varied the expectations of the type of interference by varying the probability of the two types of intervening task. Behavioral data indicate that to-be-remembered features for which interference is expected are bolstered, whereas to-be-remembered features for which no interference is expected are left vulnerable. This was further supported by fMRI data obtained from visual cortex. In conclusion, the flexibility of visual working memory allows it to strengthen memories for which it anticipates the highest risk of interference.

An attentional limbo: Saccades become momentarily non-selective in between saliency-driven and relevance-driven selection.

Human vision involves selectively directing the eyes to potential objects of interest. According to most prominent theories, selection is the quantal outcome of an ongoing competition between saliency-driven signals on the one hand, and relevance-driven signals on the other, with both types of signals continuously and concurrently projecting onto a common priority map. Here, we challenge this view. We asked participants to make a speeded eye movement towards a target orientation, which was presented together with a non-target of opposing tilt. In addition to the difference in relevance, the target and non-target also differed in saliency, with the target being either more or less salient than the non-target. We demonstrate that saliency- and relevance-driven eye movements have highly idiosyncratic temporal profiles, with saliency-driven eye movements occurring rapidly after display onset while relevance-driven eye movements occur only later. Remarkably, these types of eye movements can be fully separated in time: We find that around 250 ms after display onset, eye movements are no longer driven by saliency differences between potential targets, but also not yet driven by relevance information, resulting in a period of non-selectivity, which we refer to as the attentional limbo. Binomial modeling further confirmed that visual selection is not necessarily the outcome of a direct battle between saliency- and relevance-driven signals. Instead, selection reflects the dynamic changes in the underlying saliency- and relevance-driven processes themselves, and the time at which an action is initiated then determines which of the two will emerge as the driving force of behavior.

Neural Repetition Suppression Modulates Time Perception: Evidence From Electrophysiology and Pupillometry.

Human time perception is malleable and subject to many biases. For example, it has repeatedly been shown that stimuli that are physically intense or that are unexpected seem to last longer. Two competing hypotheses have been proposed to account for such biases: One states that these temporal illusions are the result of increased levels of arousal that speeds up neural clock dynamics, whereas the alternative "magnitude coding" account states that the magnitude of sensory responses causally modulates perceived durations. Common experimental paradigms used to study temporal biases cannot dissociate between these accounts, as arousal and sensory magnitude covary and modulate each other. Here, we present two temporal discrimination experiments where two flashing stimuli demarcated the start and end of a to-be-timed interval. These stimuli could be either in the same or a different location, which led to different sensory responses because of neural repetition suppression. Crucially, changes and repetitions were fully predictable, which allowed us to explore effects of sensory response magnitude without changes in arousal or surprise. Intervals with changing markers were perceived as lasting longer than those with repeating markers. We measured EEG (Experiment 1) and pupil size (Experiment 2) and found that temporal perception was related to changes in ERPs (P2) and pupil constriction, both of which have been related to responses in the sensory cortex. Conversely, correlates of surprise and arousal (P3 amplitude and pupil dilation) were unaffected by stimulus repetitions and changes. These results demonstrate, for the first time, that sensory magnitude affects time perception even under constant levels of arousal.

Memory for Stimulus Duration Is Not Bound to Spatial Information.

Different theories have been proposed to explain how the human brain derives an accurate sense of time. One specific class of theories, intrinsic clock theories, postulate that temporal information of a stimulus is represented much like other features such as color and location, bound together to form a coherent percept. Here, we explored to what extent this holds for temporal information after it has been perceived and is held in working memory for subsequent comparison. We recorded EEG of participants who were asked to time stimuli at lateral positions of the screen followed by comparison stimuli presented in the center. Using well-established markers of working memory maintenance, we investigated whether the usage of temporal information evoked neural signatures that were indicative of the location where the stimuli had been presented, both during maintenance and during comparison. Behavior and neural measures including the contralateral delay activity, lateralized alpha suppression, and decoding analyses through time all supported the same conclusion: The representation of location was strongly involved during perception of temporal information, but when temporal information was to be used for comparison, it no longer showed a relation to spatial information. These results support a model where the initial perception of a stimulus involves intrinsic computations, but that this information is subsequently translated to a stimulus-independent format to be used to further guide behavior.

Individual differences in crowding predict visual search performance.

Visual search is an integral part of human behavior and has proven important to understanding mechanisms of perception, attention, memory, and oculomotor control. Thus far, the dominant theoretical framework posits that search is mainly limited by covert attentional mechanisms, comprising a central bottleneck in visual processing. A different class of theories seeks the cause in the inherent limitations of peripheral vision, with search being constrained by what is known as the functional viewing field (FVF). One of the major factors limiting peripheral vision, and thus the FVF, is crowding. We adopted an individual differences approach to test the prediction from FVF theories that visual search performance is determined by the efficacy of peripheral vision, in particular crowding. Forty-four participants were assessed with regard to their sensitivity to crowding (as measured by critical spacing) and their search efficiency (as indicated by manual responses and eye movements). This revealed substantial correlations between the two tasks, as stronger susceptibility to crowding was predictive of slower search, more eye movements, and longer fixation durations. Our results support FVF theories in showing that peripheral vision is an important determinant of visual search efficiency.

The dynamics of saliency-driven and goal-driven visual selection as a function of eccentricity.

Both saliency and goal information are important factors in driving visual selection. Saliency-driven selection occurs primarily in early responses, whereas goal-driven selection happens predominantly in later responses. Here, we investigated how eccentricity affects the time courses of saliency-driven and goal-driven visual selection. In three experiments, we asked people to make a speeded eye movement toward a predefined target singleton which was simultaneously presented with a non-target singleton in a background of multiple homogeneously oriented other items. The target singleton could be either more or less salient than the non-target singleton. Both singletons were presented at one of three eccentricities (i.e., near, middle, or far). The results showed that, even though eccentricity had only little effect on overall selection performance, the underlying time courses of saliency-driven and goal-driven selection altered such that saliency effects became protracted and relevance effects became delayed for far eccentricity conditions. The protracted saliency effect was shown to be modulated by expectations as induced by the preceding trial. The results demonstrate the importance of incorporating both time and eccentricity as factors in models of visual selection.

High-pass filtering artifacts in multivariate classification of neural time series data.

BACKGROUND: Traditionally, EEG/MEG data are high-pass filtered and baseline-corrected to remove slow drifts. Minor deleterious effects of high-pass filtering in traditional time-series analysis have been well-documented, including temporal displacements. However, its effects on time-resolved multivariate pattern classification analyses (MVPA) are largely unknown. NEW METHOD: To prevent potential displacement effects, we extend an alternative method of removing slow drift noise - robust detrending - with a procedure in which we mask out all cortical events from each trial. We refer to this method as trial-masked robust detrending. RESULTS: In both real and simulated EEG data of a working memory experiment, we show that both high-pass filtering and standard robust detrending create artifacts that result in the displacement of multivariate patterns into activity silent periods, particularly apparent in temporal generalization analyses, and especially in combination with baseline correction. We show that trial-masked robust detrending is free from such displacements. COMPARISON WITH EXISTING METHOD(S): Temporal displacement may emerge even with modest filter cut-off settings such as 0.05 Hz, and even in regular robust detrending. However, trial-masked robust detrending results in artifact-free decoding without displacements. Baseline correction may unwittingly obfuscate spurious decoding effects and displace them to the rest of the trial. CONCLUSIONS: Decoding analyses benefit from trial-masked robust detrending, without the unwanted side effects introduced by filtering or regular robust detrending. However, for sufficiently clean data sets and sufficiently strong signals, no filtering or detrending at all may work adequately. Implications for other types of data are discussed, followed by a number of recommendations.

Content or status: Frontal and posterior cortical representations of object category and upcoming task goals in working memory.

To optimize task sequences, the brain must differentiate between current and prospective goals. We previously showed that currently and prospectively relevant object representations in working memory can be dissociated within object-selective cortex. Based on other recent studies indicating that a range of brain areas may be involved in distinguishing between currently relevant and prospectively relevant information in working memory, here we conducted multivoxel pattern analyses of fMRI activity in additional posterior areas (specifically early visual cortex and the intraparietal sulcus) as well as frontal areas (specifically the frontal eye fields and lateral prefrontal cortex). We assessed whether these areas represent the memory content, the current versus prospective status of the memory, or both. On each trial, participants memorized an object drawn from three different categories. The object was the target for either a first task (currently relevant), a second task (prospectively relevant), or for neither task (irrelevant). The results revealed a division of labor across brain regions: While posterior areas preferentially coded for content (i.e., the category), frontal areas carried information about the current versus prospective relevance status of the memory, irrespective of the category. Intraparietal sulcus revealed both strong category- and status-sensitivity, consistent with its hub function of combining stimulus and priority signals. Furthermore, cross-decoding analyses revealed that while current and prospective representations were similar prior to search, they became dissimilar during search, in posterior as well as frontal areas. The findings provide further evidence for a dissociation between content and control networks in working memory.

Flexible Working Memory Through Selective Gating and Attentional Tagging.

Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.

Attention for action in visual working memory.

From the conception of Baddeley's visuospatial sketchpad, visual working memory and visual attention have been closely linked concepts. An attractive model has advocated unity of the two cognitive functions, with attention serving the active maintenance of sensory representations. However, empirical evidence from various paradigms and dependent measures has now firmly established an at least partial dissociation between visual attention and visual working memory maintenance - thus leaving unclear what the relationship between the two concepts is. Moreover, a focus on sensory storage has treated visual working memory as a reflection of the past, with attention as a limiting resource. This view ignores what storage is for: immediate or future action. We argue that rather than serving sensory storage, attention emerges from coupling relevant sensory and action representations within working memory. Importantly, this coupling is bidirectional: First, through recurrent feedback mechanisms, action coupling results in the enhancement of the appropriate sensory memory representation. Under this view, unattended memories are currently not coupled to an action plan, but are not necessarily lost and remain available for future tasks when necessary. Second, through the very same feedback projections, attention serves as the credit assignment mechanism for the action's outcome. When the action is successful, the associated representations are being reinforced, leading to more robust consolidation and more rapid retrieval in the future - thus explaining performance benefits for attended memories without assuming that attention serves as the maintenance mechanism. By firmly grounding VWM in the action system, the new framework integrates a range of behavioural and neurophysiological findings and avoids circularity in explaining the role of attention in working memory.

Oscillatory Control over Representational States in Working Memory.

In the visual world, attention is guided by perceptual goals activated in visual working memory (VWM). However, planning multiple-task sequences also requires VWM to store representations for future goals. These future goals need to be prevented from interfering with the current perceptual task. Recent findings have implicated neural oscillations as a control mechanism serving the implementation and switching of different states of prioritization of VWM representations. We review recent evidence that posterior alpha-band oscillations underlie the flexible activation and deactivation of VWM representations and that frontal delta-to-theta-band oscillations play a role in the executive control of this process. That is, frontal delta-to-theta appears to orchestrate posterior alpha through long-range oscillatory networks to flexibly set up and change VWM states during multitask sequences.