Neural mechanisms of resource allocation in working memory.

To mitigate capacity limits of working memory, people allocate resources according to an item's relevance. However, the neural mechanisms supporting such a critical operation remain unknown. Here, we developed computational neuroimaging methods to decode and demix neural responses associated with multiple items in working memory with different priorities. In striate and extrastriate cortex, the gain of neural responses tracked the priority of memoranda. Higher-priority memoranda were decoded with smaller error and lower uncertainty. Moreover, these neural differences predicted behavioral differences in memory prioritization. Remarkably, trialwise variability in the magnitude of delay activity in frontal cortex predicted differences in decoded precision between low and high-priority items in visual cortex. These results suggest a model in which feedback signals broadcast from frontal cortex sculpt the gain of memory representations in visual cortex according to behavioral relevance, thus, identifying a neural mechanism for resource allocation.

Neural population dynamics of human working memory.

The activity of neurons in macaque prefrontal cortex (PFC) persists during working memory (WM) delays, providing a mechanism for memory.1,2,3,4,5,6,7,8,9,10,11 Although theory,11,12 including formal network models,13,14 assumes that WM codes are stable over time, PFC neurons exhibit dynamics inconsistent with these assumptions.15,16,17,18,19 Recently, multivariate reanalyses revealed the coexistence of both stable and dynamic WM codes in macaque PFC.20,21,22,23 Human EEG studies also suggest that WM might contain dynamics.24,25 Nonetheless, how WM dynamics vary across the cortical hierarchy and which factors drive dynamics remain unknown. To elucidate WM dynamics in humans, we decoded WM content from fMRI responses across multiple cortical visual field maps.26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 We found coexisting stable and dynamic neural representations of WM during a memory-guided saccade task. Geometric analyses of neural subspaces revealed that early visual cortex exhibited stronger dynamics than high-level visual and frontoparietal cortex. Leveraging models of population receptive fields, we visualized and made the neural dynamics interpretable. We found that during WM delays, V1 population initially encoded a narrowly tuned bump of activation centered on the peripheral memory target. Remarkably, this bump then spread inward toward foveal locations, forming a vector along the trajectory of the forthcoming memory-guided saccade. In other words, the neural code transformed into an abstraction of the stimulus more proximal to memory-guided behavior. Therefore, theories of WM must consider both sensory features and their task-relevant abstractions because changes in the format of memoranda naturally drive neural dynamics.

Joint representation of working memory and uncertainty in human cortex.

Neural representations of visual working memory (VWM) are noisy, and thus, decisions based on VWM are inevitably subject to uncertainty. However, the mechanisms by which the brain simultaneously represents the content and uncertainty of memory remain largely unknown. Here, inspired by the theory of probabilistic population codes, we test the hypothesis that the human brain represents an item maintained in VWM as a probability distribution over stimulus feature space, thereby capturing both its content and uncertainty. We used a neural generative model to decode probability distributions over memorized locations from fMRI activation patterns. We found that the mean of the probability distribution decoded from retinotopic cortical areas predicted memory reports on a trial-by-trial basis. Moreover, in several of the same mid-dorsal stream areas, the spread of the distribution predicted subjective trial-by-trial uncertainty judgments. These results provide evidence that VWM content and uncertainty are jointly represented by probabilistic neural codes.

To look or not to look: dissociating presaccadic and covert spatial attention.

Attention is a central neural process that enables selective and efficient processing of visual information. Individuals can attend to specific visual information either overtly, by making an eye movement to an object of interest, or covertly, without moving their eyes. We review behavioral, neuropsychological, neurophysiological, and computational evidence of presaccadic attentional modulations that occur while preparing saccadic eye movements, and highlight their differences from those of covert spatial endogenous (voluntary) and exogenous (involuntary) attention. We discuss recent studies and experimental procedures on how these different types of attention impact visual performance, alter appearance, differentially modulate the featural representation of basic visual dimensions (orientation and spatial frequency), engage different neural computations, and recruit partially distinct neural substrates. We conclude that presaccadic attention and covert attention are dissociable.

Different computations underlie overt presaccadic and covert spatial attention.

Perception and action are tightly coupled: visual responses at the saccade target are enhanced right before saccade onset. This phenomenon, presaccadic attention, is a form of overt attention-deployment of visual attention with concurrent eye movements. Presaccadic attention is well-documented, but its underlying computational process remains unknown. This is in stark contrast to covert attention-deployment of visual attention without concurrent eye movements-for which the computational processes are well characterized by a normalization model. Here, a series of psychophysical experiments reveal that presaccadic attention modulates visual performance only via response gain changes. A response gain change was observed even when attention field size increased, violating the predictions of a normalization model of attention. Our empirical results and model comparisons reveal that the perceptual modulations by overt presaccadic and covert spatial attention are mediated through different computations.

Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis.

Decision confidence reflects our ability to evaluate the quality of decisions and guides subsequent behavior. Experiments on confidence reports have almost exclusively focused on two-alternative decision-making. In this realm, the leading theory is that confidence reflects the probability that a decision is correct (the posterior probability of the chosen option). There is, however, another possibility, namely that people are less confident if the best two options are closer to each other in posterior probability, regardless of how probable they are in absolute terms. This possibility has not previously been considered because in two-alternative decisions, it reduces to the leading theory. Here, we test this alternative theory in a three-alternative visual categorization task. We found that confidence reports are best explained by the difference between the posterior probabilities of the best and the next-best options, rather than by the posterior probability of the chosen (best) option alone, or by the overall uncertainty (entropy) of the posterior distribution. Our results upend the leading notion of decision confidence and instead suggest that confidence reflects the observer's subjective probability that they made the best possible decision.

How exogenous spatial attention affects visual representation.

Orienting covert spatial attention to a target location enhances visual sensitivity and benefits performance in many visual tasks. How these attention-related improvements in performance affect the underlying visual representation of low-level visual features is not fully understood. Here we focus on characterizing how exogenous spatial attention affects the feature representations of orientation and spatial frequency. We asked observers to detect a vertical grating embedded in noise and performed psychophysical reverse correlation. Doing so allowed us to make comparisons with previous studies that utilized the same task and analysis to assess how endogenous attention and presaccadic modulations affect visual representations. We found that exogenous spatial attention improved performance and enhanced the gain of the target orientation without affecting orientation tuning width. Moreover, we found no change in spatial frequency tuning. We conclude that covert exogenous spatial attention alters performance by strictly boosting gain of orientation-selective filters, much like covert endogenous spatial attention.

Presaccadic attention improves or impairs performance by enhancing sensitivity to higher spatial frequencies.

Right before we move our eyes, visual performance and neural responses for the saccade target are enhanced. This effect, presaccadic attention, is considered to prioritize the saccade target and to enhance behavioral performance for the saccade target. Recent evidence has shown that presaccadic attention modulates the processing of feature information. Hitherto, it remains unknown whether presaccadic modulations on feature information are flexible, to improve performance for the task at hand, or automatic, so that they alter the featural representation similarly regardless of the task. Using a masking procedure, here we report that presaccadic attention can either improve or impair performance depending on the spatial frequency content of the visual input. These counterintuitive modulations were significant at a time window right before saccade onset. Furthermore, merely deploying covert attention within the same temporal interval without preparing a saccade did not affect performance. This study reveals that presaccadic attention not only prioritizes the saccade target, but also automatically modifies its featural representation.

Attention model of binocular rivalry.

When the corresponding retinal locations in the two eyes are presented with incompatible images, a stable percept gives way to perceptual alternations in which the two images compete for perceptual dominance. As perceptual experience evolves dynamically under constant external inputs, binocular rivalry has been used for studying intrinsic cortical computations and for understanding how the brain regulates competing inputs. Converging behavioral and EEG results have shown that binocular rivalry and attention are intertwined: binocular rivalry ceases when attention is diverted away from the rivalry stimuli. In addition, the competing image in one eye suppresses the target in the other eye through a pattern of gain changes similar to those induced by attention. These results require a revision of the current computational theories of binocular rivalry, in which the role of attention is ignored. Here, we provide a computational model of binocular rivalry. In the model, competition between two images in rivalry is driven by both attentional modulation and mutual inhibition, which have distinct selectivity (feature vs. eye of origin) and dynamics (relatively slow vs. relatively fast). The proposed model explains a wide range of phenomena reported in rivalry, including the three hallmarks: (i) binocular rivalry requires attention; (ii) various perceptual states emerge when the two images are swapped between the eyes multiple times per second; (iii) the dominance duration as a function of input strength follows Levelt's propositions. With a bifurcation analysis, we identified the parameter space in which the model's behavior was consistent with experimental results.

Proactive Control: Neural Oscillatory Correlates of Conflict Anticipation and Response Slowing.

Proactive control allows us to anticipate environmental changes and adjust behavioral strategy. In the laboratory, investigators have used a number of different behavioral paradigms, including the stop-signal task (SST), to examine the neural processes of proactive control. Previous functional MRI studies of the SST have demonstrated regional responses to conflict anticipation-the likelihood of a stop signal or P(stop) as estimated by a Bayesian model-and reaction time (RT) slowing and how these responses are interrelated. Here, in an electrophysiological study, we investigated the time-frequency domain substrates of proactive control. The results showed that conflict anticipation as indexed by P(stop) was positively correlated with the power in low-theta band (3-5 Hz) in the fixation (trial onset)-locked interval, and go-RT was negatively correlated with the power in delta-theta band (2-8 Hz) in the go-locked interval. Stimulus prediction error was positively correlated with the power in the low-beta band (12-22 Hz) in the stop-locked interval. Further, the power of the P(stop) and go-RT clusters was negatively correlated, providing a mechanism relating conflict anticipation to RT slowing in the SST. Source reconstruction with beamformer localized these time-frequency activities close to brain regions as revealed by functional MRI in earlier work. These are the novel results to show oscillatory electrophysiological substrates in support of trial-by-trial behavioral adjustment for proactive control.

Saccade Preparation Reshapes Sensory Tuning.

Human observers make large rapid eye movements-saccades-to bring behaviorally relevant information into the fovea, where spatial resolution is high. In some visual tasks [1-4], performance at the location of a saccade target improves before the eyes move. Although these findings provide evidence that extra-retinal signals evoked by saccades can enhance visual perception, it remains unknown whether and how presaccadic modulations change the processing of feature information and thus modulate visual representations. To answer this question, one must go beyond the use of methods that only probe performance accuracy (d') in different tasks. Here, using a psychophysical reverse correlation approach [5-8], we investigated how saccade preparation influences the processing of orientation and spatial frequency-two building blocks of early vision. We found that saccade preparation selectively enhanced the gain of high spatial frequency information and narrowed orientation tuning at the upcoming saccade landing position. These modulations were time locked to saccade onset, peaking right before the eyes moved (-50-0 ms). Moreover, merely deploying covert attention within the same temporal interval without preparing a saccade did not alter performance. The observed presaccadic tuning changes may correspond to the presaccadic enhancement [9-11] and receptive field shifts reported in neurophysiological studies [12-14]. Saccade preparation may support transaccadic integration by reshaping the representation of the saccade target to be more fovea-like just before the eyes move. The presaccadic modulations on spatial frequency and orientation processing illustrate a strong perception-action coupling by revealing that the visual system dynamically reshapes feature selectivity contingent upon eye movements.

Deconstructing Interocular Suppression: Attention and Divisive Normalization.

In interocular suppression, a suprathreshold monocular target can be rendered invisible by a salient competitor stimulus presented in the other eye. Despite decades of research on interocular suppression and related phenomena (e.g., binocular rivalry, flash suppression, continuous flash suppression), the neural processing underlying interocular suppression is still unknown. We developed and tested a computational model of interocular suppression. The model included two processes that contributed to the strength of interocular suppression: divisive normalization and attentional modulation. According to the model, the salient competitor induced a stimulus-driven attentional modulation selective for the location and orientation of the competitor, thereby increasing the gain of neural responses to the competitor and reducing the gain of neural responses to the target. Additional suppression was induced by divisive normalization in the model, similar to other forms of visual masking. To test the model, we conducted psychophysics experiments in which both the size and the eye-of-origin of the competitor were manipulated. For small and medium competitors, behavioral performance was consonant with a change in the response gain of neurons that responded to the target. But large competitors induced a contrast-gain change, even when the competitor was split between the two eyes. The model correctly predicted these results and outperformed an alternative model in which the attentional modulation was eye specific. We conclude that both stimulus-driven attention (selective for location and feature) and divisive normalization contribute to interocular suppression.

Perigenual anterior cingulate event-related potential precedes stop signal errors.

Momentary lapses in attention disrupt goal-directed behavior. Attentional lapse has been associated with increased "default-mode" network (DMN) activity. In our previous fMRI study of a stop signal task (SST), greater activation of the perigenual anterior cingulate cortex (pgACC) - an important node of the DMN - predicts stop signal errors. In event-related potential (ERP) studies, the amplitude of an error-preceding positivity (EPP) also predicts response error. However, it is not clear whether the EPP originates from DMN regions. Here, we combined high-density array EEG and an SST to examine response-locked ERPs of error preceding trials in twenty young adult participants. The results showed an EPP in go trials that preceded stop error than stop success trials. Importantly, source modeling identified the origin of the EPP in the pgACC. By employing a bootstrapping procedure, we further confirmed that pgACC rather than the dorsal ACC as the source provides a better fit to the EPP. Together, these results suggest that attentional lapse in association with EPP in the pgACC anticipates failure in response inhibition.

Event-related potentials for post-error and post-conflict slowing.

In a reaction time task, people typically slow down following an error or conflict, each called post-error slowing (PES) and post-conflict slowing (PCS). Despite many studies of the cognitive mechanisms, the neural responses of PES and PCS continue to be debated. In this study, we combined high-density array EEG and a stop-signal task to examine event-related potentials of PES and PCS in sixteen young adult participants. The results showed that the amplitude of N2 is greater during PES but not PCS. In contrast, the peak latency of N2 is longer for PCS but not PES. Furthermore, error-positivity (Pe) but not error-related negativity (ERN) was greater in the stop error trials preceding PES than non-PES trials, suggesting that PES is related to participants' awareness of the error. Together, these findings extend earlier work of cognitive control by specifying the neural correlates of PES and PCS in the stop signal task.

Backward position shift in apparent motion.

We investigated the perceived position of visual targets in apparent motion. A disc moved horizontally through three positions from -10° to +10° in the far periphery (20° above fixation), generating a compelling impression of apparent motion. In the first experiment, observers compared the position of the middle of the three discs to a subsequently presented reference. Unexpectedly, observers judged its position to be shifted backward, in the direction opposite that of the motion. We then tested the middle disc in sequences of 3, 5, and 7 discs, each covering the same spatial and temporal extents (similar speeds). The backwards shift was only found for the three-disc sequence. With the extra discs approaching more continuous motion, the perceived shift was in the same direction as the apparent motion. Finally, using a localization task with constant static references, we measured the position shifts of all the disc locations for two-disc, three-disc and four-disc apparent motion sequences. The backward shift was found for the second location of all sequences. We suggest that the backward shift of the second element along an apparent motion path is due to an attraction effect induced by the initial point of the motion.

Surround modulation of global form perception.

We demonstrate a novel surround modulation of global form perception by using Glass patterns in a center-surround configuration. Glass patterns contain randomly distributed dot pairs, or dipoles, whose orientations are determined by a geometric transform. By integrating across dipoles, an observer can perceive a global structure in the image. We measured the coherence threshold, the minimum proportion of signal dots needed for an observer to detect the global form, at 75% accuracy. The coherence thresholds of the central target Glass patterns were measured either alone or with the presence of various Glass pattern surrounds. Concentric and spiral surrounds increased the coherence threshold for the concentric target compared with that measured for the target alone, while a radial surround had no effect. The coherence threshold for the radial pattern was elevated only by the spiral surround while the spiral and translational targets were not affected by any surrounds. The effect persisted when the center and the surround were segregated by a blank gap and peaked at an optimal gap width. Our results show that global form perception can be modulated by a surround, and this modulation depends on the shapes of the central target and the surround context.