Dissociable online integration processes in visual working memory.

Visual working memory has severe capacity limits, creating a bottleneck for active processing. A key way of mitigating this limitation is by chunking, i.e. compressing several pieces of information into one visual working memory representation. However, despite decades of research, chunking efficiency remains debated because of mixed evidence. We propose that there are actually 2 integration mechanisms: Grouping combines several objects to one representation, and object-unification merges the parts of a single object. Critically, we argue that the fundamental distinction between the 2 processes is their differential use of the pointer system, the indexing process connecting visual working memory representations with perception. In grouping, the objects that are represented together still maintain independent pointers, making integration costly but highly flexible. Conversely, object-unification fuses the pointers as well as the representations, with the single pointer producing highly efficient integration but blocking direct access to individual parts. We manipulated integration cues via task-irrelevant movement, and monitored visual working memory's online electrophysiological marker. Uniquely colored objects were flexibly grouped and ungrouped via independent pointers (experiment 1). If objects turned uniformly black, object-integration could not be undone (experiment 2), requiring visual working memory to reset before re-individuation. This demonstrates 2 integration levels (representational-merging versus pointer-compression) and establishes the dissociation between visual working memory representations and their underlying pointers.

Different features of real-world objects are represented in a dependent manner in long-term memory.

In the present study, we examined how real-world objects are represented in long-term memory. Two contrasting views exist with regard to this question: one argues that real-world objects are represented as a set of independent features, and the other argues that they form bound integrate representations. In 5 experiments, we tested the different predictions of each view, namely whether the different features of real-world items are remembered and forgotten independently from each other, in a feature-based manner, or conversely are stored and lost in an object-based manner, with all features depending upon each other. Across various stimuli, learning tasks (incidental or explicit), experimental setups (within- or between-subjects design), feature-dimensions, and encoding times, we consistently found that information is forgotten in an object-based manner. When an object ceases to be fully remembered, all of its features are lost, instead of only some of the object's features being lost whereas other features are still remembered. Furthermore, we found support for a strong form of dependency among the different features, namely a hierarchical structure. We conclude that visual long-term memory is object-based, challenging previous findings. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Neural evidence for an object-based pointer system underlying working memory.

To accomplish even rudimentary tasks, our cognitive system must update its representation of the changing environment. This process relies on visual working memory (VWM), which can actively modify its representations. We argue that this ability depends on a pointer system, such that each representation is stably and uniquely mapped to a specific stimulus. Without these pointers, VWM representations are inaccessible and therefore unusable. In three Electroencephalogram (EEG) experiments, we examined whether the pointers are allocated in an object-based, featural, or spatial manner: three factors that were confounded in previous studies. We used a feature change-detection task, in which objects moved and could separate into independently-moving parts. Despite the movement and separation being completely task-irrelevant, we found that the separation invalidated the pointers. This happened in a shape task, where the separation changed both the objects and the task-relevant features, but importantly, also in a color task, where the separation destroyed the objects while leaving the task-relevant features intact. Furthermore, even in a color task where all items had identical shapes, object-separation invalidated the pointers. This suggests that objects and not task-relevant features underlie the pointer system. Finally, when each object-part could be individuated already before the separation, the pointers were maintained, suggesting that the pointers are specifically tied to objects rather than locations. These results shed new light on the pointers which underlie VWM performance, demonstrating that the pointer system is object-based regardless of the task requirements.

What can half a million change detection trials tell us about visual working memory?

Visual working memory (VWM) represents the surrounding world in an active and accessible state, but its capacity is severely limited. To better understand VWM and its limits, we collected data from over 3,800 participants in the canonical change detection task. This unique population-level data-set sheds new light on classic debates regarding VWM capacity. First, the result supported a view of VWM as an active process, as manifested by the fact that capacity estimates were not stable across set-sizes, but rather lower for the larger set-size. Another support for this notion came from the tight connection capacity estimates had with a measure of attentional control. Together, the data suggested that individual differences in capacity do not reflect only differences in storage-size, but differences in the efficiency of using this storage. Second, we found a response bias such that subjects are more likely to respond that the probed item changed, and this criterion bias was further shifted as the set-size increased. These findings are naturally explained by a slot-like theory arguing that when load exceeds capacity, certain items remain completely outside of VWM (instead of all items being represented in lower resolution), therefore causing subjects to perceive them as different from VWM contents even when they are unchanged. Additionally, we found that the pattern of d' also confirmed the predictions of a slot-like view of VWM, such that some items are represented with high fixed resolution and others are not represented at all, although this finding is based on two measures with very different underlying assumptions. We also discuss how flexible-resource views can accommodate these results. Moreover, comparing performance between the first and last trials demonstrated no evidence for proactive interference as the driving factor of capacity limitations. We provide further details regarding the distribution of individual capacity, the relations between capacity and demographic variables, and the spatial prioritization of the items.

Delineating resetting and updating in visual working memory based on the object-to-representation correspondence.

When an object we represent in visual working memory (VWM) changes, its representation is modified accordingly. VWM can either access and change the existing representation by an updating process, or it can reset, by encoding the object in its novel status as a new representation. Our goal was to show that the determining factor of updating versus resetting is the availability of a stable correspondence between the object and its VWM representation. Here, we demonstrate that updating relies on the object-to-representation mapping to access and modify the appropriate representation, while losing this mapping triggers a resetting process. We compared very similar situations of object separation that either allowed the mapping to hold, or caused it to be lost. When an object that was mapped to one representation separated, VWM reset, manifested by a sharp drop in the contralateral delay activity (CDA) amplitude (an electrophysiological marker of VWM contents; Experiment 1), and a behavioral cost to detect salient changes that co-occurred with the resetting-triggering event (Experiment 2). When each part was mapped to a different representation, the separation resulted in updating, with a gradual rise in CDA amplitude (Experiment 1), and a reduced behavioral cost (Experiment 2). Thus, while updating and resetting resulted in similar final representations (corresponding to the post-change objects), their dynamics were different, depending on the availability of the mapping. Our results reveal the triggering conditions of resetting and updating, establish methods to study these online processes, and highlight the importance of the object-to-representation correspondence in VWM.

Visual working memory can selectively reset a subset of its representations.

The visual working memory (VWM) resetting process is triggered when the mapping between an object in the environment and its corresponding VWM representation becomes irrelevant. Resetting involves discarding the no longer relevant representations, and encoding novel representations and mappings. We examined how resetting operates on VWM's contents. Specifically, we tested whether losing only part of the encoded mappings led to resetting all of the VWM representations. Subjects monitored moving polygons for an abrupt shape-change. Occasionally, a polygon separated into two halves that continued to move independently, making the original single mapping irrelevant. This loss of mapping triggered a resetting process, producing a performance cost: subjects missed shape-changes when they occurred during resetting, but not when the changes occurred before or after resetting. Critically, the cost was (1) specific to the separated item, (2) larger when more mappings were lost, and (3) unaffected by the set-size. This suggests that resetting is a "local" process: VWM removes only the representations whose mappings are lost.

Neural and Behavioral Evidence for an Online Resetting Process in Visual Working Memory.

Visual working memory (VWM) guides behavior by holding a set of active representations and modifying them according to changes in the environment. This updating process relies on a unique mapping between each VWM representation and an actual object in the environment. Here, we destroyed this mapping by either presenting a coherent object but then breaking it into independent parts or presenting an object but then abruptly replacing it with a different object. This allowed us to introduce the neural marker and behavioral consequence of an online resetting process in humans' VWM. Across seven experiments, we demonstrate that this resetting process involves abandoning the old VWM contents because they no longer correspond to the objects in the environment. Then, VWM encodes the novel information and reestablishes the correspondence between the new representations and the objects. The resetting process was marked by a unique neural signature: a sharp drop in the amplitude of the electrophysiological index of VWM contents (the contralateral delay activity), presumably indicating the loss of the existent object-to-representation mappings. This marker was missing when an updating process occurred. Moreover, when tracking moving items, VWM failed to detect salient changes in the object's shape when these changes occurred during the resetting process. This happened despite the object being fully visible, presumably because the mapping between the object and a VWM representation was lost. Importantly, we show that resetting, its neural marker, and the behavioral cost it entails, are specific to situations that involve a destruction of the objects-to-representations correspondence. SIGNIFICANCE STATEMENT: Visual working memory (VWM) maintains task-relevant information in an online state. Previous studies showed that VWM representations are accessed and modified after changes in the environment. Here, we show that this updating process critically depends on an ongoing mapping between the representations and the objects in the environment. When this mapping breaks, VWM cannot access the old representations and instead resets. The novel resetting process that we introduce removes the existing representations instead of modifying them and this process is accompanied by a unique neural marker. During the resetting process, VWM was blind to salient changes in the object's shape. The resetting process highlights the flexibility of our cognitive system in handling the dynamic environment by abruptly abandoning irrelevant schemas.

Object representations in visual working memory change according to the task context.

This study investigated whether an item's representation in visual working memory (VWM) can be updated according to changes in the global task context. We used a modified change detection paradigm, in which the items moved before the retention interval. In all of the experiments, we presented identical color-color conjunction items that were arranged to provide a common fate Gestalt grouping cue during their movement. Task context was manipulated by adding a condition highlighting either the integrated interpretation of the conjunction items or their individuated interpretation. We monitored the contralateral delay activity (CDA) as an online marker of VWM. Experiment 1 employed only a minimal global context; the conjunction items were integrated during their movement, but then were partially individuated, at a late stage of the retention interval. The same conjunction items were perfectly integrated in an integration context (Experiment 2). An individuation context successfully produced strong individuation, already during the movement, overriding Gestalt grouping cues (Experiment 3). In Experiment 4, a short priming of the individuation context managed to individuate the conjunction items immediately after the Gestalt cue was no longer available. Thus, the representations of identical items changed according to the task context, suggesting that VWM interprets incoming input according to global factors which can override perceptual cues.

The contralateral delay activity as a neural measure of visual working memory.

The contralateral delay activity (CDA) is a negative slow wave sensitive to the number of objects maintained in visual working memory (VWM). In recent years, a growing number of labs started to use the CDA in order to investigate VWM, leading to many fascinating discoveries. Here, we discuss the recent developments and contribution of the CDA in various research fields. Importantly, we report two meta-analyses that unequivocally validate the relationship between the set-size increase in the CDA amplitude and the individual VWM capacity, and between the CDA and filtering efficiency. We further discuss how the CDA was used to study the role of VWM in visual search, multiple object tracking, grouping, binding, and whether VWM capacity allocation is determined by the items' resolution or instead by the number of objects regardless of their complexity. In addition, we report how the CDA has been used to characterize specific VWM deficits in special populations.

Integration of Distinct Objects in Visual Working Memory Depends on Strong Objecthood Cues Even for Different-Dimension Conjunctions.

What makes an integrated object in visual working memory (WM)? Past evidence suggested that WM holds all features of multidimensional objects together, but struggles to integrate color-color conjunctions. This difficulty was previously attributed to a challenge in same-dimension integration, but here we argue that it arises from the integration of 2 distinct objects. To test this, we examined the integration of distinct different-dimension features (a colored square and a tilted bar). We monitored the contralateral delay activity, an event-related potential component sensitive to the number of objects in WM. The results indicated that color and orientation belonging to distinct objects in a shared location were not integrated in WM (Experiment 1), even following a common fate Gestalt cue (Experiment 2). These conjunctions were better integrated in a less demanding task (Experiment 3), and in the original WM task, but with a less individuating version of the original stimuli (Experiment 4). Our results identify the critical factor in WM integration at same- versus separate-objects, rather than at same- versus different-dimensions. Compared with the perfect integration of an object's features, the integration of several objects is demanding, and depends on an interaction between the grouping cues and task demands, among other factors.

The number of objects determines visual working memory capacity allocation for complex items.

The goal of the present study was to examine whether visual working memory (WM) capacity allocation is determined solely by complexity, with the number of objects being redundant, as suggested by flexible resource models. Participants performed the change detection task with random polygons as stimuli, while we monitored the contralateral delay activity (CDA), an electrophysiological marker whose amplitude rises as WM load increases. In Experiment 1, we compared the WM maintenance of one whole polygon to a single half of the polygon, equating the number of items but varying the complexity level. Additionally, we compared the whole polygon to two halves of a polygon, thus roughly equating perceptual complexity but manipulating the number of items. The results suggested that only the number of objects determined WM capacity allocation: the CDA was identical when comparing one whole polygon to one polygon half, even though these conditions differed in complexity. Furthermore, the CDA amplitude was lower in the whole polygon condition relative to the two halves condition, even though both contained roughly the same amount of information. Experiment 2 extended these results by showing that two polygon halves that moved separately but then met and moved together were gradually integrated to consume similar WM capacity as one polygon half. Additionally, in both experiments we found an object benefit in accuracy, corroborating the important role of objects in WM. Our results demonstrate that WM capacity allocation cannot be explained by complexity alone. Instead, it is highly sensitive to objecthood, as suggested by discrete slot models.

How low can you go? Changing the resolution of novel complex objects in visual working memory according to task demands.

In three experiments we manipulated the resolution of novel complex objects in visual working memory (WM) by changing task demands. Previous studies that investigated the trade-off between quantity and resolution in visual WM yielded mixed results for simple familiar stimuli. We used the contralateral delay activity as an electrophysiological marker to directly track the deployment of visual WM resources while participants preformed a change-detection task. Across three experiments we presented the same novel complex items but changed the task demands. In Experiment 1 we induced a medium resolution task by using change trials in which a random polygon changed to a different type of polygon and replicated previous findings showing that novel complex objects are represented with higher resolution relative to simple familiar objects. In Experiment 2 we induced a low resolution task that required distinguishing between polygons and other types of stimulus categories, but we failed in finding a corresponding decrease in the resolution of the represented item. Finally, in Experiment 3 we induced a high resolution task that required discriminating between highly similar polygons with somewhat different contours. This time, we observed an increase in the item's resolution. Our findings indicate that the resolution for novel complex objects can be increased but not decreased according to task demands, suggesting that minimal resolution is required in order to maintain these items in visual WM. These findings support studies claiming that capacity and resolution in visual WM reflect different mechanisms.