The influence of depth on object selection and manipulation in visual working memory within a 3D context.

Recent studies have examined whether the internal selection mechanism functions similarly for perception and visual working memory (VWM). However, the process of how we access and manipulate object representations distributed in a 3D space remains unclear. In this study, we utilized a memory search task to investigate the effect of depth on object selection and manipulation within VWM. The memory display consisted of colored items half positioned at the near depth plane and the other half at the far plane. During memory maintenance, the participants were instructed to search for a target representation and update its color. The results showed that under object-based attention (Experiments 1, 3, and 5), the update time was faster for targets at the near plane than for those at the far plane. This effect was absent in VWM when deploying spatial attention (Experiment 2) and in visual search regardless of the type of attention deployed (Experiment 4). The differential effects of depth on spatial and object-based attention in VWM suggest that spatial attention primarily relied on 2D location information irrespective of depth, whereas object-based attention seemed to prioritize memory representations at the front plane before shifting to the back. Our findings shed light on the interaction between depth perception and the selection mechanisms within VWM in a 3D context, emphasizing the importance of ordinal, rather than metric, spatial information in guiding object-based attention in VWM.

Priming effect of individual similarity and ensemble perception in visual search and working memory.

Perceptual priming is a well-known phenomenon showing that the repetition of an object's feature can facilitate subsequent detection of that item. Although the priming effect has been rigorously studied in visual search, less is known about its effect on working memory and it is unclear whether the repetition of similar features, and furthermore, ensemble perception created by a large set of similar features, can induce priming. In this study, we investigated the priming effects of individual similarity and ensemble perception in visual search and visual working memory (VWM). We replicated the classic perceptual priming effect (Experiment 1a) and found that visual search was enhanced when the current target had a similar color to the previous target (Experiment 1b), but not when the similar color had been shown as a distractor before (Experiment 1c). However, if the target and distractors of similar colors formed ensemble perception, the search efficiency was again promoted even when the current target shared the same color with the previous distractor (Experiment 1d). For VWM, repeating the ensembles of the target- and nontarget-color subsets did not significantly affect the memory capacity, while switching the two harmed the memory fidelity but not capacity (Experiment 2). We suggest different underlying mechanisms for priming in visual search and VWM: in the former, the perception history of individual similarity and stimuli ensemble exert their effects on through the priority map, by forming a gradient distribution of attentional weights that peak at the previous target feature and diminish as stimulus diverges from the previously selected one; while in the latter, perception history of memory ensemble may influence the deployment of existing memory resources across trials, thereby affecting the memory fidelity but not its capacity.

Spatial attention based on 2D location and relative depth order modulates visual working memory in a 3D environment.

The attentional effect on visual working memory (VWM) has been a heated research topic in the past two decades. Studies show that VWM performance for an attended memory item can be improved by cueing its two-dimensional (2D) spatial location during retention. However, few studies have investigated the effect of attentional selection on VWM in a three-dimensional setting, and it remains unknown whether depth information can produce beneficial attentional effects on 2D visual representations similar to 2D spatial information. Here we conducted four experiments, displaying memory items at various stereoscopic depth planes, and examined the retro-cue effects of four types of cues - a cue would either indicate the 2D or depth location of a memory item, and either in the form of physical (directly pointing to a location) or symbolic (numerically mapping onto a location) cues. We found that retro-cue benefits were only observed for cues directly pointing to a 2D location, whereas a null effect was observed for cues directly pointing to a depth location. However, there was a retro-cue effect when cueing the relative depth order, though the effect was weaker than that for cueing the 2D location. The selective effect on VWM based on 2D spatial attention is different from depth-based attention, and the divergence suggests that an object representation is primarily bound with its 2D spatial location, weakly bound with its depth order but not with its metric depth location. This indicates that attentional selection based on memory for depth, particularly metric depth, is ineffective.

Top-down modulation on depth processing: Visual searches for metric and ordinal depth information show a pattern of dissociation.

Depending on the goal, one can selectively process the metric depth or the ordinal depth information in the same scene. It is unknown whether the metric depth and ordinal depth information are processed through a shared or different underlying mechanisms. Here, we investigated the processing of the metric depth and ordinal depth using visual search. Items were presented at multiple depth planes defined by the binocular disparity, with one item per depth plane. In the metric-search task, participants were required to search for the target on a particular depth plane, among one to three distractors. In the ordinal-search task, the target was specified by its depth order indicated by numbers (smaller numbers indicated nearer depth planes). We found that the ordinal search was faster and more accurate than the metric search, and the data showed a pattern of dissociation. Metric search, but not ordinal search, was slowed when the target and distractors were closer in depth, while ordinal search was slower for the middle than the edge positions but metric search was unaffected. These two opposite effects suggest that metric depth and ordinal depth may be processed differently.

The role of ensemble average differs in working memory for depth and planar information.

The representation of individual planar locations and features stored in working memory can be affected by the average representation. However, less is known about how the average representation affects the short-term storage of depth information. To evaluate the possible different roles of the ensemble average in working memory for planar and depth information, we used mathematical models to fit the data collected from one study on working memory for depth and 12 studies on working memory for planar information. The pattern of recalled depth was well captured by models assuming that there was a probability of reporting the average depth instead of the individual depth, compressing the recalled front-back distance of the stimulus ensemble compared to the perceived distance. However, when modeling the recalled planar information, we found that participants tended to report individual nontarget features when the target was not memorized, and the assumption of reporting average information improves the data fitting only in very few studies. These results provide evidence for our hypothesis that average depth information can be used as a substitution for individual depth information stored in working memory, but for planar visual features, the substitution of target with the average works under a constraint that the average of to-be-remembered features is readily accessible.

Contrasting effects of exogenous and endogenous attention on size perception.

Although neuroimaging studies have shown that exogenous and endogenous attention are dissociable, only a few behavioural studies have explored their differential effects on visual sensitivity, and none have directly focused on visual appearance. Here, we show that exogenous and endogenous attention produces contrasting effects on apparent size. Participants performed a spatial pre-cueing comparative judgement task that had been frequently used to test the attentional effects on visual perception. The results showed that a smaller stimulus within the focus of exogenous attention was perceived to be equal in size as a larger unattended stimulus, whereas a larger stimulus within the focus of endogenous attention was perceived to be equal in size as a smaller unattended stimulus. In other words, exogenous attention increased the perceived size while endogenous attention decreased the perceived size. The contrasting effects may be attributed to the mechanism that exogenous attention favours parvocellular processing for which more neurons with smaller receptive fields (RFs) are activated for a given size, whereas endogenous attention favours magnocellular processing for which fewer neurons with larger RFs are activated. This finding shows that exogenous and endogenous attention acts differentially on size perception, and provides supportive evidence for the distinct mechanisms underlying the two types of attentional processing.

The Short-Term Retention of Depth.

We review research on the visual working memory for information portrayed by items arranged in depth (i.e., distance to the observer) within peri-personal space. Most items lose their metric depths within half a second, even though their identities and spatial positions are retained. The paradoxical loss of depth information may arise because visual working memory retains the depth of a single object for the purpose of actions such as pointing or grasping which usually apply to only one thing at a time.

Increasing perceptual separateness affects working memory for depth - re-allocation of attention from boundaries to the fixated center.

For decades, working memory (WM) has been a heated research topic in the field of cognitive psychology. However, most studies on WM presented visual stimuli on a two-dimensional plane, rarely involving depth perception. Several previous studies have investigated how depth information is stored in WM, and found that WM for depth is even more limited in capacity and the memory performance is poor compared to visual WM. In the present study, we used a change detection task to investigate whether dissociating memory items by different visual features, thereby to increase their perceptual separateness, can improve WM performance for depth. Memory items presented at various depth planes were bound with different colors (Experiments 1 and 3) or sizes (Experiment 2). The memory performance for depth locations of visual stimuli with homogeneous and heterogeneous appearances were tested and compared. The results showed a consistent pattern that although separating items with various feature values did not affect the overall memory performance, the manipulation significantly improved memory performance for the middle depth locations but impaired the performance for the boundary locations when observers fixated at the center of the whole depth volume. The memory benefits of feature separation can be attributed to enhanced individuation of memory items, therefore facilitating a more balanced allocation of attention and memory resources.

Effect of attentional selection on working memory for depth in a retro-cueing paradigm.

Recent studies have shown that the temporary storage and manipulation of depth information (working memory for depth; WMd) is largely different from that of visual information in a 2D context (visual working memory; VWM). Although there has been abundant evidence on VWM showing that cueing a memory item during retention could bias attention to its internal representation and thus improves its memory performance (a retro-cue effect), it is unknown whether such an effect differs for WMd that is nested in a 3D context compared with that in a conventional 2D context. Here, we used a change detection task to investigate the effect of attentional selection on WMd by testing several types of retro-cue. The memory array consisted of items positioned at various stereoscopic depth planes, and a cue was presented during retention. Participants needed to make judgments on whether the depth position of target (one memory item) had changed. Our study showed reliable valid retro-cue benefits but no invalid retro-cue cost, indicating that the relational information may be registered in WMd to prevent a strategical removal of the unattended item. There was also a slight improvement in memory performance for cueing depth order compared with that for cueing other feature dimensions or 2D locations. The attentional effect on memory representation in a 3D context is different from that in a 2D context, and the divergence may suggest the distinctive nature of working memory for depth.

Predicting visual working memory with multimodal magnetic resonance imaging.

The indispensability of visual working memory (VWM) in human daily life suggests its importance in higher cognitive functions and neurological diseases. However, despite the extensive research efforts, most findings on the neural basis of VWM are limited to a unimodal context (either structure or function) and have low generalization. To address the above issues, this study proposed the usage of multimodal neuroimaging in combination with machine learning to reveal the neural mechanism of VWM across a large cohort (N = 547). Specifically, multimodal magnetic resonance imaging features extracted from voxel-wise amplitude of low-frequency fluctuations, gray matter volume, and fractional anisotropy were used to build an individual VWM capacity prediction model through a machine learning pipeline, including the steps of feature selection, relevance vector regression, cross-validation, and model fusion. The resulting model exhibited promising predictive performance on VWM (r = .402, p < .001), and identified features within the subcortical-cerebellum network, default mode network, motor network, corpus callosum, anterior corona radiata, and external capsule as significant predictors. The main results were then compared with those obtained on emotional regulation and fluid intelligence using the same pipeline, confirming the specificity of our findings. Moreover, the main results maintained well under different cross-validation regimes and preprocess strategies. These findings, while providing richer evidence for the importance of multimodality in understanding cognitive functions, offer a solid and general foundation for comprehensively understanding the VWM process from the top down.

Training with high perceptual difficulty improves the capacity and fidelity of internal representation in VWM.

It has been shown that the capacity of visual working memory (VWM) is a strong predictor of individual intelligence, and researchers have developed various training protocols to improve VWM capacity. However, it seems that whether the fidelity of internal representation in VWM can also be improved by training is largely overlooked in the past literature. Here, we introduced a new training approach that involved increasing the perceptual difficulty of training materials to enhance VWM, and both memory capacity and the fidelity of representation were examined to assess the training efficacy. Participants with normal vision and cognitive abilities received 3-week training on VWM using a change detection task, and the results showed that both the capacity and the fidelity of memory representations were improved for training with perceptually difficult stimuli, while only the fidelity was improved for training with perceptually normal stimuli. In addition, we found that the training effects on memory precision may be subject to capacity constraints. We suggest that long-term adaptive training with perceptually difficult stimuli may facilitate encoding efficiency through familiarizing trainees with an increased baseline of cognitive workload during the encoding process. The present study offers clear evidence that training with high perceptual difficulty is more effective and the improvements in VWM are more stable than training with perceptually normal materials, and the simple manipulation on training stimuli indicates that the method can be generalized to a wider range of training situations and populations.

Overestimation and contraction biases of depth information stored in working memory depend on spatial configuration.

Depth perception is essential for effective interaction with the environment. Although the accuracy of depth perception has been studied extensively, it is unclear how accurate the depth information is stored in working memory. In this study, we investigated the accuracy and systematic biases of depth representation by a delayed estimation task. The memory array consisted of items presented at various stereoscopic depth positions, and the participants were instructed to estimate the depth position of one target item after a retention interval. We examined the effect of spatial configuration by comparing the memory performance in the whole-display condition where non-target memory items were present during retrieval with that in the single-display condition where non-target memory items were absent. In the single-display condition, we found an overestimation bias that the depth estimates were farther than the corresponding depth positions defined by disparity, and a contraction bias that the stored depth positions near the observer were overestimated and those far from the observer were underestimated. The magnitude of these biases increased with the number of to-be-stored items. However, in the whole-display condition, the overestimation bias was corrected and the contraction bias did not increase with the number of to-be-stored items. Our findings suggested that the number of to-be-stored items could affect the accuracy of depth working memory, and its effect depended crucially on whether the information of spatial configuration of memory display was available at the retrieval stage.

Exogenous spatial attention shortens perceived depth.

Although spatial attention has been found to alter the subjective appearance of visual stimuli in several perceptual dimensions, no research has explored whether exogenous spatial attention can affect depth perception, which is a fundamental dimension in perception that allows us to effectively interact with the environment. Here, we used an experimental paradigm adapted from Gobell and Carrasco (Psychological Science, 16[8], 644-651, 2005) to investigate this question. A peripheral cue preceding two line stimuli was used to direct exogenous attention to either location of the two lines. The two lines were separated by a certain relative disparity, and participants were asked to judge the perceived depth of two lines while attention was manipulated. We found that a farther stereoscopic depth at the attended location was perceived to be equally distant as a nearer depth at the unattended location. No such effect was found in a control experiment that employed a postcue paradigm, suggesting that our findings could not be attributed to response bias. Therefore, our study shows that exogenous spatial attention shortens perceived depth. The apparent change in stereoscopic depth may be regulated by a mechanism involving direct neural enhancement on those tuned to disparity, or be modulated by an attentional effect on apparent contrast. This finding shows that attention can change not only visual appearance but also the perceived spatial relation between an object and an observer.

Relation matters: relative depth order is stored in working memory for depth.

Working memory is considered as a cognitive memory buffer for temporarily holding, processing, and manipulating information. Although working memory for verbal and visual information has been studied extensively in the past literature, few studies have systematically investigated how depth information is stored in working memory. Here, we show that the memory performance for detecting changes in stereoscopic depth is low when there is no change in relative depth order, and the performance is reliably better when depth order is changed. Increasing the magnitude of change only improves memory performance when depth order is kept constant. However, if depth order is changed, the performance remains high, even with a small change magnitude. Our findings suggest that relative depth order is a better indicator for working memory performance than absolute metric depth. The memory representation for individual depth is not independent, but inherently relational, revealing a fundamental organizing principle for depth information in the visual system.

Task-dependent effects of voluntary space-based and involuntary feature-based attention on visual working memory.

Previous research has shown that visual working memory (VWM) can be modulated by space-based or feature-based attentional selection. However, it remains unclear how the two modes of attention operate jointly to affect VWM, and in particular, if involuntary feature-based attention plays a role in VWM. In this study, a pre-cued change detection paradigm was employed to investigate the concurrent effects of space- and feature-based attention on VWM. Space-based attention was manipulated by informative spatial cueing and by varying the proximity between the test item and the cued (fixated) memory item, while feature-based attention was induced in an involuntary manner by having the test item to share the same color or shape with the cued item on a fraction of trials. The results showed that: (1) the memory performance for the cued items was always better than the uncued items, suggesting a beneficial effect of voluntary spatial attention; (2) with a brief duration of the memory array (250 ms), cue-test proximity benefited VWM in the shape judgment task but not in the color judgment task, whereas with a longer duration (1200 ms), no proximity effect was found for either task; (3) VWM was improved for the same-colored items regardless of the task and duration; (4) VWM was improved for the same-shaped items only in the shape judgment task with the longer duration of the memory array. A discrimination task further showed that the proximity effect associated with VWM reflects a perceptual bottleneck in memory encoding for shape but not for color with a brief display. Our results suggest that involuntary feature-based attention could be triggered by spatial cueing to modulate VWM; involuntary color-based attention facilitates VWM independently of task, whereas shape-based facilitation is task-dependent, i.e., confined only to the shape judgment task, presumably reflecting different attention-guiding potencies of the two features.

Working memory for stereoscopic depth is limited and imprecise-evidence from a change detection task.

Most studies on visual working memory (VWM) and spatial working memory (SWM) have employed visual stimuli presented at the fronto-parallel plane and few have involved depth perception. VWM is often considered as a memory buffer for temporarily holding and manipulating visual information that relates to visual features of an object, and SWM for holding and manipulating spatial information that concerns the spatial location of an object. Although previous research has investigated the effect of stereoscopic depth on VWM, the question of how depth positions are stored in working memory has not been systematically investigated, leaving gaps in the existing literature on working memory. Here, we explore working memory for depth by using a change detection task. The memory items were presented at various stereoscopic depth planes perpendicular to the line of sight, with one item per depth plane. Participants were asked to make judgments on whether the depth position of the target (one of the memory items) had changed. The results showed a conservative response bias that observers tended to make 'no change' responses when detecting changes in depth. In addition, we found that similar to VWM, the change detection accuracy degraded with the number of memory items presented, but the accuracy was much lower than that reported for VWM, suggesting that the storage for depth information is severely limited and less precise than that for visual information. The detection sensitivity was higher for the nearest and farthest depths and was better when the probe was presented along with the other items originally in the memory array, indicating that how well the to-be-stored depth can be stored in working memory depends on its relation with the other depth positions.

The transition from feature to object: Storage unit in visual working memory depends on task difficulty.

Visual working memory (VWM) is a cognitive memory buffer for temporarily processing and storing visual information. Previous studies suggest that its capacity is severely limited, and there is an ongoing debate on whether the storage capacity is object-based or feature-based in VWM. In this study, a change-detection task was employed to investigate whether and how task difficulty can affect VWM, specifically, its capacity and the unit of storage. Task difficulty was manipulated through the set size of memory items, memory fidelity required by the resolution of representation and the type of feature tested. We examined two types of stimuli: the single-feature type, where each memory item was composed of a single feature (color or shape), and the conjunctive-feature type, where each item was composed of a conjunction of two features (colored shape). Experiment 1 replicated the previous findings that memory capacity for colors was larger than shapes, and decreased with the resolution demand regardless of the type of stimuli. In Experiment 2, we analyzed and compared the results from single-feature items and conjunctive-feature items in the low- and high-resolution conditions while controlling for the number of to-be-remembered features. By directly matching the estimated capacity based on an object-unit and a feature-unit with the theoretical prediction, the results showed that the unit storage in VWM tended to be feature-based with low task difficulty, and to be object-based with high task difficulty. This suggests that VWM is dynamic and flexible, dependent on the load of the current task.

Saturation and brightness modulate the effect of depth on visual working memory.

Although previous studies show inconsistent results regarding the effect of depth perception on visual working memory (VWM), a recent finding shows that perceptually closer-in-depth items are better remembered than farther items when combining the congruent disparity and relative size cues. In this study, we employed a similar change detection paradigm to investigate the effects of saturation and brightness, alone or in combination with binocular disparity, on VWM. By varying the appearance of the memory items, we aimed to manipulate the visual salience as well as to simulate the aerial perspective cue that induces depth perception. We found that the change detection accuracy was significantly improved for brighter and more saturated items, but not for items solely with higher saturation. Additionally, combining saturation with the congruent disparity cue significantly improved memory performance for perceptually closer items over farther items. Conflicting the disparity cue with saturation eliminated the memory benefit for the closer items. These results indicate that saturation and brightness could modulate the effect of depth on VWM, and both visual salience and depth perception affect VWM possibly through a common underlying mechanism of setting priority for attentional selection.

Evidence for the beneficial effect of perceptual grouping on visual working memory: an empirical study on illusory contour and a meta-analytic study.

The capacity of visual working memory (VWM) is found to be extremely limited. Past research shows that VWM can be facilitated by Gestalt principles of grouping, however, it remains controversial whether factors like the type of Gestalt principles, the characteristics of stimuli and the nature of experimental design could affect the beneficial effect of grouping. In particular, studies have shown that perceptual grouping could improve memory performance for a feature that is relevant for grouping, but it is unclear whether the same improvement exists for a feature that is irrelevant for grouping. In this article, an empirical study and a meta-analytic study were conducted to investigate the effect of perceptual grouping on VWM. In the empirical study, we examined the grouping effect by employing a Kanizsa illusion in which memory items were grouped by illusory contour. We found that the memory performance was improved for the grouped items even though the tested feature was grouping irrelevant, and the improvement was not significantly different from the effect of grouping by physical connectedness or by solid occlusion. In the meta-analytic study, we systematically and quantitatively examined the effect of perceptual grouping on VWM by pulling the results from all eligible studies, and found that the beneficial grouping effect was robust but the magnitude of the effect can be affected by several moderators. Factors like the types of grouping methods, the duration and the layout of the memory display, and the characteristics of the tested feature moderated the grouping effect, whereas whether employing a cue or a verbal suppression task did not. Our study suggests that the underlying mechanism of the grouping benefit may be distinct with regard to grouping relevancy of the to-be-stored feature. The grouping effect on VWM may be independent of attention for a grouping relevant feature, but may rely on attentional prioritization for a grouping irrelevant feature.

Effects of Color and Luminance Contrast on Size Perception-Evidence from a Horizontal Parallel Lines Illusion.

The present study investigated a size illusion composed of two horizontal lines that were vertically separated and parallel to each other. When the two lines were of equal length, the upper line was consistently perceived to be a little longer than the lower line, therefore it was termed as horizontal parallel lines (HPL) illusion. We investigated the effect of color and luminance contrast on the HPL illusion by manipulating the color and luminance of the two lines. Results indicated the following: (1) differences in color between the two lines reduced the illusion; (2) differences in luminance between the two lines reduced the illusion; (3) Effect 1 was greater than Effect 2; (4) the illusory effect could not be affected as long as both of the lines were of the same color or luminance. The results suggest that the color or luminance contrast may contribute to the overall decrease in the illusory effect for lines with different colors/luminances, but generally the illusion decreases as the two lines are less similar to each other. These findings indicate that the similarity or 'sameness' effect dominates the effect of color/luminance contrast on the size illusion over the effect resulted from contrast difference or depth perception.