Time after Time: Preserving Temporal Memories When Experiences Repeat.

Remembering when events occur in time is fundamental to episodic memory. Yet, many experiences repeat over time creating the potential for interference when attempting to recall temporally specific memories. Here, we argue that temporal memories are protected, in part, by reinstatement of temporal context information that is triggered by stimulus repetitions. We motivate this argument by integrating seminal findings across several distinct literatures and methodologies. Specifically, we consider key insights from foundational behavioral studies of temporal memory, recent electrophysiological and neuroimaging approaches to measuring memory reinstatement, and computational models that describe how temporal context representations shape memory processes. We also note several open questions concerning how temporal context reinstatement might influence subsequent temporal memory, including potential mediating effects of event spacing and event boundaries. These ideas and questions have the potential to guide future research and, ultimately, to advance theoretical accounts of how we preserve temporal memories.

Benefits of spaced learning are predicted by re-encoding of past experience in ventromedial prefrontal cortex.

More than a century of research shows that spaced learning improves long-term memory. Yet, there remains debate concerning why. A major limitation to resolving theoretical debates is the lack of evidence for how neural representations change as a function of spacing. Here, leveraging a massive-scale 7T human fMRI dataset, we tracked neural representations and behavioral expressions of memory as participants viewed thousands of natural scene images that repeated at lags ranging from seconds to many months. We show that spaced learning increases the similarity of human ventromedial prefrontal cortex representations across stimulus encounters and, critically, these increases parallel and predict the behavioral benefits of spacing. Additionally, we show that these spacing benefits critically depend on remembering and, in turn, 're-encoding' past experience. Collectively, our findings provide fundamental insight into how spaced learning influences neural representations and why spacing is beneficial.

Adding Meaning to Memories: How Parietal Cortex Combines Semantic Content with Episodic Experience.

Neuroimaging studies of human memory have consistently found that univariate responses in parietal cortex track episodic experience with stimuli (whether stimuli are 'old' or 'new'). More recently, pattern-based fMRI studies have shown that parietal cortex also carries information about the semantic content of remembered experiences. However, it is not well understood how memory-based and content-based signals are integrated within parietal cortex. Here, in humans (males and females), we used voxel-wise encoding models and a recognition memory task to predict the fMRI activity patterns evoked by complex natural scene images based on (1) the episodic history and (2) the semantic content of each image. Models were generated and compared across distinct subregions of parietal cortex and for occipitotemporal cortex. We show that parietal and occipitotemporal regions each encode memory and content information, but they differ in how they combine this information. Among parietal subregions, angular gyrus was characterized by robust and overlapping effects of memory and content. Moreover, subject-specific semantic tuning functions revealed that successful recognition shifted the amplitude of tuning functions in angular gyrus but did not change the selectivity of tuning. In other words, effects of memory and content were additive in angular gyrus. This pattern of data contrasted with occipitotemporal cortex where memory and content effects were interactive: memory effects were preferentially expressed by voxels tuned to the content of a remembered image. Collectively, these findings provide unique insight into how parietal cortex combines information about episodic memory and semantic content.SIGNIFICANCE STATEMENT Neuroimaging studies of human memory have identified multiple brain regions that not only carry information about "whether" a visual stimulus is successfully recognized but also "what" the content of that stimulus includes. However, a fundamental and open question concerns how the brain integrates these two types of information (memory and content). Here, using a powerful combination of fMRI analysis methods, we show that parietal cortex, particularly the angular gyrus, robustly combines memory- and content-related information, but these two forms of information are represented via additive, independent signals. In contrast, memory effects in high-level visual cortex critically depend on (and interact with) content representations. Together, these findings reveal multiple and distinct ways in which the brain combines memory- and content-related information.

Re-expression of CA1 and entorhinal activity patterns preserves temporal context memory at long timescales.

Converging, cross-species evidence indicates that memory for time is supported by hippocampal area CA1 and entorhinal cortex. However, limited evidence characterizes how these regions preserve temporal memories over long timescales (e.g., months). At long timescales, memoranda may be encountered in multiple temporal contexts, potentially creating interference. Here, using 7T fMRI, we measured CA1 and entorhinal activity patterns as human participants viewed thousands of natural scene images distributed, and repeated, across many months. We show that memory for an image's original temporal context was predicted by the degree to which CA1/entorhinal activity patterns from the first encounter with an image were re-expressed during re-encounters occurring minutes to months later. Critically, temporal memory signals were dissociable from predictors of recognition confidence, which were carried by distinct medial temporal lobe expressions. These findings suggest that CA1 and entorhinal cortex preserve temporal memories across long timescales by coding for and reinstating temporal context information.

Perception and memory retrieval states are reflected in distributed patterns of background functional connectivity.

The same visual input can serve as the target of perception or as a trigger for memory retrieval depending on whether cognitive processing is externally oriented (perception) or internally oriented (memory retrieval). While numerous human neuroimaging studies have characterized how visual stimuli are differentially processed during perception versus memory retrieval, perception and memory retrieval may also be associated with distinct neural states that are independent of stimulus-evoked neural activity. Here, we combined human fMRI with full correlation matrix analysis (FCMA) to reveal potential differences in "background" functional connectivity across perception and memory retrieval states. We found that perception and retrieval states could be discriminated with high accuracy based on patterns of connectivity across (1) the control network, (2) the default mode network (DMN), and (3) retrosplenial cortex (RSC). In particular, clusters in the control network increased connectivity with each other during the perception state, whereas clusters in the DMN were more strongly coupled during the retrieval state. Interestingly, RSC switched its coupling between networks as the cognitive state shifted from retrieval to perception. Finally, we show that background connectivity (1) was fully independent from stimulus-related variance in the signal and, further, (2) captured distinct aspects of cognitive states compared to traditional classification of stimulus-evoked responses. Together, our results reveal that perception and memory retrieval are associated with sustained cognitive states that manifest as distinct patterns of connectivity among large-scale brain networks.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory.

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Perception and memory have distinct spatial tuning properties in human visual cortex.

Reactivation of earlier perceptual activity is thought to underlie long-term memory recall. Despite evidence for this view, it is unclear whether mnemonic activity exhibits the same tuning properties as feedforward perceptual activity. Here, we leverage population receptive field models to parameterize fMRI activity in human visual cortex during spatial memory retrieval. Though retinotopic organization is present during both perception and memory, large systematic differences in tuning are also evident. Whereas there is a three-fold decline in spatial precision from early to late visual areas during perception, this pattern is not observed during memory retrieval. This difference cannot be explained by reduced signal-to-noise or poor performance on memory trials. Instead, by simulating top-down activity in a network model of cortex, we demonstrate that this property is well explained by the hierarchical structure of the visual system. Together, modeling and empirical results suggest that computational constraints imposed by visual system architecture limit the fidelity of memory reactivation in sensory cortex.

Long-term memory interference is resolved via repulsion and precision along diagnostic memory dimensions.

When memories share similar features, this can lead to interference, and ultimately forgetting. With experience, however, interference can be resolved. This raises the important question of how memories change, with experience, to minimize interference. Intuitively, interference might be minimized by increasing the precision and accuracy of memories. However, recent evidence suggests a potentially adaptive role for memory distortions. Namely, similarity can trigger exaggerations of subtle differences between memories (repulsion). Here, we tested whether repulsion specifically occurs on feature dimensions along which memories compete and whether repulsion is predictive of reduced memory interference. To test these ideas, we developed synthetic faces in a two-dimensional face space (affect and gender). This allowed us to precisely manipulate similarity between faces and the feature dimension along which faces differed. In three experiments, participants learned to associate faces with unique cue words. Associative memory tests confirmed that when faces were similar (face pairmates), this produced interference. Using a continuous face reconstruction task, we found two changes in face memory that preferentially occurred along the feature dimension that was "diagnostic" of the difference between face pairmates: (1) there was a bias to remember pairmates with exaggerated differences (repulsion) and (2) there was an increase in the precision of feature memory. Critically, repulsion and precision were each associated with reduced associative memory interference, but these were statistically dissociable contributions. Collectively, our findings reveal that similarity between memories triggers dissociable, experience-dependent changes that serve an adaptive role in reducing interference.

Abrupt hippocampal remapping signals resolution of memory interference.

Remapping refers to a decorrelation of hippocampal representations of similar spatial environments. While it has been speculated that remapping may contribute to the resolution of episodic memory interference in humans, direct evidence is surprisingly limited. We tested this idea using high-resolution, pattern-based fMRI analyses. Here we show that activity patterns in human CA3/dentate gyrus exhibit an abrupt, temporally-specific decorrelation of highly similar memory representations that is precisely coupled with behavioral expressions of successful learning. The magnitude of this learning-related decorrelation was predicted by the amount of pattern overlap during initial stages of learning, with greater initial overlap leading to stronger decorrelation. Finally, we show that remapped activity patterns carry relatively more information about learned episodic associations compared to competing associations, further validating the learning-related significance of remapping. Collectively, these findings establish a critical link between hippocampal remapping and episodic memory interference and provide insight into why remapping occurs.

Adaptive Repulsion of Long-Term Memory Representations Is Triggered by Event Similarity.

We tested whether similarity between events triggers adaptive biases in how those events are remembered. We generated pairs of competing objects that were identical except in color and varied the degree of color similarity for the competing objects. Subjects (N = 123 across four experiments) repeatedly studied and were tested on associations between each of these objects and corresponding faces. As expected, high color similarity between competing objects created memory interference for object-face associations. Strikingly, high color similarity also resulted in a systematic bias in how the objects themselves were remembered: Competing objects with highly similar colors were remembered as being further apart (in color space) than they actually were. This repulsion of color memories increased with learning and served a clear adaptive purpose: Greater repulsion was associated with lower associative-memory interference. These findings reveal that similarity between events triggers adaptive-memory distortions that minimize interference.

Adaptive Memory Distortions Are Predicted by Feature Representations in Parietal Cortex.

Similarity between memories is a primary cause of interference and forgetting. Exaggerating subtle differences between memories is therefore a potential mechanism for reducing interference. Here, we report a human fMRI study (n = 29, 19 female) that tested whether behavioral and neural expressions of memories are adaptively distorted to reduce interference. Participants learned and repeatedly retrieved object images, some of which were identical except for subtle color differences. Behavioral measures of color memory revealed exaggeration of differences between similar objects. Importantly, greater memory exaggeration was associated with lower memory interference. fMRI pattern analyses revealed that color information in parietal cortex was stronger during memory recall when color information was critical for discriminating competing memories. Moreover, greater representational distance between competing memories in parietal cortex predicted greater color memory exaggeration and lower memory interference. Together, these findings reveal that competition between memories induces adaptive, feature-specific distortions in parietal representations and corresponding behavioral expressions.SIGNIFICANCE STATEMENT Similarity between memories is a primary cause of interference and forgetting. Here, we show that, when remembering highly similar objects, subtle differences in the features of these objects are exaggerated in memory to reduce interference. These memory distortions are reflected in, and predicted by, overlap of activity patterns in lateral parietal cortex. These findings provide unique insight into how memory interference is resolved and specifically implicate lateral parietal cortex in representing feature-specific memory distortions.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States.

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).1516 Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

When the Memory System Gets Ahead of Itself.

Humans are adept at learning and exploiting statistical regularities to predict future events from current experience. A recent paper by Sherman and Turk-Browne demonstrates that statistical regularities bias the hippocampus toward representing future states over current experience and reduce the degree to which current experience is encoded into memory.

Transforming the Concept of Memory Reactivation.

Reactivation refers to the phenomenon wherein patterns of neural activity expressed during perceptual experience are re-expressed at a later time, a putative neural marker of memory. Reactivation of perceptual content has been observed across many cortical areas and correlates with objective and subjective expressions of memory in humans. However, because reactivation emphasizes similarities between perceptual and memory-based representations, it obscures differences in how perceptual events and memories are represented. Here, we highlight recent evidence of systematic differences in how (and where) perceptual events and memories are represented in the brain. We argue that neural representations of memories are best thought of as spatially transformed versions of perceptual representations. We consider why spatial transformations occur and identify critical questions for future research.

Interference between overlapping memories is predicted by neural states during learning.

One of the primary contributors to forgetting is interference from overlapping memories. Intuitively, this suggests-and prominent theoretical models argue-that memory interference is best avoided by encoding overlapping memories as if they were unrelated. It is therefore surprising that reactivation of older memories during new encoding has been associated with reduced memory interference. Critically, however, prior studies have not directly established why reactivation reduces interference. Here, we first developed a behavioral paradigm that isolates the negative influence that overlapping memories exert during memory retrieval. We then show that reactivating older memories during the encoding of new memories dramatically reduces this interference cost at retrieval. Finally, leveraging multiple fMRI decoding approaches, we show that spontaneous reactivation of older memories during new encoding leads to integration of overlapping memories and, critically, that integration during encoding specifically reduces interference between overlapping, and otherwise competing, memories during retrieval.

Decoding the tradeoff between encoding and retrieval to predict memory for overlapping events.

When new events overlap with past events, there is a natural tradeoff between encoding the new event and retrieving the past event. Given the ubiquity of overlap among memories, this tradeoff between memory encoding and retrieval is of central importance to computational models of episodic memory (O'Reilly & McClelland 1994; Hasselmo 2005). However, prior studies have not directly linked neural markers of encoding/retrieval tradeoffs to behavioral measures of how overlapping events are remembered. Here, by decoding patterns of scalp electroencephalography (EEG) from male and female human subjects, we show that tradeoffs between encoding and retrieval states are reflected in distributed patterns of neural activity and, critically, these neural tradeoffs predict how overlapping events will later be remembered. Namely, new events that overlapped with past events were more likely to be subsequently remembered if neural patterns were biased toward a memory encoding state-or, conversely, away from a retrieval state. Additionally, we show that neural markers of encoding vs. retrieval states are surprisingly independent from previously-described EEG predictors of subsequent memory. Instead, we demonstrate that previously-described EEG predictors of subsequent memory are better explained by task engagement than by memory encoding, per se. Collectively, our findings provide important insight into how the memory system balances memory encoding and retrieval states and, more generally, into the neural mechanisms that support successful memory formation.

Decomposing Parietal Memory Reactivation to Predict Consequences of Remembering.

Memory retrieval can strengthen, but also distort memories. Parietal cortex is a candidate region involved in retrieval-induced memory changes as it reflects retrieval success and represents retrieved content. Here, we conducted an fMRI experiment to test whether different forms of parietal reactivation predict distinct consequences of retrieval. Subjects studied associations between words and pictures of faces, scenes, or objects, and then repeatedly retrieved half of the pictures, reporting the vividness of the retrieved pictures ("retrieval practice"). On the following day, subjects completed a recognition memory test for individual pictures. Critically, the test included lures highly similar to studied pictures. Behaviorally, retrieval practice increased both hit and false alarm (FA) rates to similar lures, confirming a causal influence of retrieval on subsequent memory. Using pattern similarity analyses, we measured two different levels of reactivation during retrieval practice: generic "category-level" reactivation and idiosyncratic "item-level" reactivation. Vivid remembering during retrieval practice was associated with stronger category- and item-level reactivation in parietal cortex. However, these measures differentially predicted subsequent recognition memory performance: whereas higher category-level reactivation tended to predict FAs to lures, item-level reactivation predicted correct rejections. These findings indicate that parietal reactivation can be decomposed to tease apart distinct consequences of memory retrieval.

Parietal Representations of Stimulus Features Are Amplified during Memory Retrieval and Flexibly Aligned with Top-Down Goals.

In studies of human episodic memory, the phenomenon of reactivation has traditionally been observed in regions of occipitotemporal cortex (OTC) involved in visual perception. However, reactivation also occurs in lateral parietal cortex (LPC), and recent evidence suggests that stimulus-specific reactivation may be stronger in LPC than in OTC. These observations raise important questions about the nature of memory representations in LPC and their relationship to representations in OTC. Here, we report two fMRI experiments that quantified stimulus feature information (color and object category) within LPC and OTC, separately during perception and memory retrieval, in male and female human subjects. Across both experiments, we observed a clear dissociation between OTC and LPC: while feature information in OTC was relatively stronger during perception than memory, feature information in LPC was relatively stronger during memory than perception. Thus, while OTC and LPC represented common stimulus features in our experiments, they preferentially represented this information during different stages. In LPC, this bias toward mnemonic information co-occurred with stimulus-level reinstatement during memory retrieval. In Experiment 2, we considered whether mnemonic feature information in LPC was flexibly and dynamically shaped by top-down retrieval goals. Indeed, we found that dorsal LPC preferentially represented retrieved feature information that addressed the current goal. In contrast, ventral LPC represented retrieved features independent of the current goal. Collectively, these findings provide insight into the nature and significance of mnemonic representations in LPC and constitute an important bridge between putative mnemonic and control functions of parietal cortex.SIGNIFICANCE STATEMENT When humans remember an event from the past, patterns of sensory activity that were present during the initial event are thought to be reactivated. Here, we investigated the role of lateral parietal cortex (LPC), a high-level region of association cortex, in representing prior visual experiences. We find that LPC contained stronger information about stimulus features during memory retrieval than during perception. We also found that current task goals influenced the strength of stimulus feature information in LPC during memory. These findings suggest that, in addition to early sensory areas, high-level areas of cortex, such as LPC, represent visual information during memory retrieval, and that these areas may play a special role in flexibly aligning memories with current goals.

Bottom-Up and Top-Down Factors Differentially Influence Stimulus Representations Across Large-Scale Attentional Networks.

Visual attention is thought to be supported by three large-scale frontoparietal networks: the frontoparietal control network (FPCN), the dorsal attention network (DAN), and the ventral attention network (VAN). The traditional view is that these networks support visual attention by biasing and evaluating sensory representations in visual cortical regions. However, recent evidence suggests that frontoparietal regions actively represent perceptual stimuli. Here, we assessed how perceptual stimuli are represented across large-scale frontoparietal and visual networks. Specifically, we tested whether representations of stimulus features across these networks are differentially sensitive to bottom-up and top-down factors. In a pair of pattern-based fMRI studies, male and female human subjects made perceptual decisions about face images that varied along two independent dimensions: gender and affect. Across studies, we interrupted bottom-up visual input using backward masks. Within studies, we manipulated which stimulus features were goal relevant (i.e., whether gender or affect was relevant) and task switching (i.e., whether the goal on the current trial matched the goal on the prior trial). We found that stimulus features could be reliably decoded from all four networks and, importantly, that subregions within each attentional network maintained coherent representations. Critically, the different attentional manipulations (interruption, goal relevance, and task switching) differentially influenced feature representations across networks. Whereas visual interruption had a relatively greater influence on representations in visual regions, goal relevance and task switching had a relatively greater influence on representations in frontoparietal networks. Therefore, large-scale brain networks can be dissociated according to how attention influences the feature representations that they maintain.SIGNIFICANCE STATEMENT Visual attention is supported by multiple frontoparietal attentional networks. However, it remains unclear how stimulus features are represented within these networks and how they are influenced by attention. Here, we assessed feature representations in four large-scale networks using a perceptual decision-making paradigm in which we manipulated top-down and bottom-up factors. We found that top-down manipulations such as goal relevance and task switching modulated feature representations in attentional networks, whereas bottom-up manipulations such as interruption of visual processing had a relatively stronger influence on feature representations in visual regions. Together, these findings indicate that attentional networks actively represent stimulus features and that representations within different large-scale networks are influenced by different forms of attention.

Overlap among Spatial Memories Triggers Repulsion of Hippocampal Representations.

Across the domains of spatial navigation and episodic memory, the hippocampus is thought to play a critical role in disambiguating (pattern separating) representations of overlapping events. However, it is not fully understood how and why hippocampal patterns become separated. Here, we test the idea that event overlap triggers a "repulsion" among hippocampal representations that develops over the course of learning. Using a naturalistic route-learning paradigm and spatiotemporal pattern analysis of human fMRI data, we found that hippocampal representations of overlapping routes gradually diverged with learning to the point that they became less similar than representations of non-overlapping events. In other words, the hippocampus not only disambiguated overlapping events but formed representations that "reversed" the objective similarity among routes. This finding, which was selective to the hippocampus, is not predicted by standard theoretical accounts of pattern separation. Critically, because the overlapping route stimuli that we used ultimately diverged (so that each route contained overlapping and non-overlapping segments), we were able to test whether the reversal effect was selective to the overlapping segments. Indeed, once overlapping routes diverged (eliminating spatial and visual similarity), hippocampal representations paradoxically became relatively more similar. Finally, using a novel analysis approach, we show that the degree to which individual hippocampal voxels were initially shared across route representations was predictive of the magnitude of learning-related separation. Collectively, these findings indicate that event overlap triggers a repulsion of hippocampal representations-a finding that provides critical mechanistic insight into how and why hippocampal representations become separated.