Task imprinting: Another mechanism of representational change?

Research from several areas suggests that mental representations adapt to the specific tasks we carry out in our environment. In this study, we propose a mechanism of adaptive representational change, task imprinting. Thereby, we introduce a computational model, which portrays task imprinting as an adaptation to specific task goals via selective storage of helpful representations in long-term memory. We test the main qualitative prediction of the model in four behavioral experiments using healthy young adults as participants. In each experiment, we assess participants' baseline representations in the beginning of the experiment, then expose participants to one of two tasks intended to shape representations differently according to our model, and finally assess any potential change in representations. Crucially, the tasks used to measure representations differ in the amount that strategic, judgmental processes play a role. The results of Experiments 1 and 2 allow us to exclude the option that representations used in more perceptual tasks become biased categorically. The results of Experiment 4 make it likely that people strategically decide given the specific task context whether they use categorical information or not. One signature of representational change was however observed: category learning practice increased the perceptual sensitivity over and above mere exposure to the same stimuli.

Spontaneous eye movements reflect the representational geometries of conceptual spaces.

Functional neuroimaging studies indicate that the human brain can represent concepts and their relational structure in memory using coding schemes typical of spatial navigation. However, whether we can read out the internal representational geometries of conceptual spaces solely from human behavior remains unclear. Here, we report that the relational structure between concepts in memory might be reflected in spontaneous eye movements during verbal fluency tasks: When we asked participants to randomly generate numbers, their eye movements correlated with distances along the left-to-right one-dimensional geometry of the number space (mental number line), while they scaled with distance along the ring-like two-dimensional geometry of the color space (color wheel) when they randomly generated color names. Moreover, when participants randomly produced animal names, eye movements correlated with low-dimensional similarity in word frequencies. These results suggest that the representational geometries used to internally organize conceptual spaces might be read out from gaze behavior.

Grid-like entorhinal representation of an abstract value space during prospective decision making.

How valuable a choice option is often changes over time, making the prediction of value changes an important challenge for decision making. Prior studies identified a cognitive map in the hippocampal-entorhinal system that encodes relationships between states and enables prediction of future states, but does not inherently convey value during prospective decision making. In this fMRI study, participants predicted changing values of choice options in a sequence, forming a trajectory through an abstract two-dimensional value space. During this task, the entorhinal cortex exhibited a grid-like representation with an orientation aligned to the axis through the value space most informative for choices. A network of brain regions, including ventromedial prefrontal cortex, tracked the prospective value difference between options. These findings suggest that the entorhinal grid system supports the prediction of future values by representing a cognitive map, which might be used to generate lower-dimensional value signals to guide prospective decision making.

Causal role of the angular gyrus in insight-driven memory reconfiguration.

Maintaining an accurate model of the world relies on our ability to update memory representations in light of new information. Previous research on the integration of new information into memory mainly focused on the hippocampus. Here, we hypothesized that the angular gyrus, known to be involved in episodic memory and imagination, plays a pivotal role in the insight-driven reconfiguration of memory representations. To test this hypothesis, participants received continuous theta burst stimulation (cTBS) over the left angular gyrus or sham stimulation before gaining insight into the relationship between previously separate life-like animated events in a narrative-insight task. During this task, participants also underwent EEG recording and their memory for linked and non-linked events was assessed shortly thereafter. Our results show that cTBS to the angular gyrus decreased memory for the linking events and reduced the memory advantage for linked relative to non-linked events. At the neural level, cTBS targeting the angular gyrus reduced centro-temporal coupling with frontal regions and abolished insight-induced neural representational changes for events linked via imagination, indicating impaired memory reconfiguration. Further, the cTBS group showed representational changes for non-linked events that resembled the patterns observed in the sham group for the linked events, suggesting failed pruning of the narrative in memory. Together, our findings demonstrate a causal role of the left angular gyrus in insight-related memory reconfigurations.

Parallel cognitive maps for multiple knowledge structures in the hippocampal formation.

The hippocampal-entorhinal system uses cognitive maps to represent spatial knowledge and other types of relational information. However, objects can often be characterized by different types of relations simultaneously. How does the hippocampal formation handle the embedding of stimuli in multiple relational structures that differ vastly in their mode and timescale of acquisition? Does the hippocampal formation integrate different stimulus dimensions into one conjunctive map or is each dimension represented in a parallel map? Here, we reanalyzed human functional magnetic resonance imaging data from Garvert et al. (2017) that had previously revealed a map in the hippocampal formation coding for a newly learnt transition structure. Using functional magnetic resonance imaging adaptation analysis, we found that the degree of representational similarity in the bilateral hippocampus also decreased as a function of the semantic distance between presented objects. Importantly, while both map-like structures localized to the hippocampal formation, the semantic map was located in more posterior regions of the hippocampal formation than the transition structure and thus anatomically distinct. This finding supports the idea that the hippocampal-entorhinal system forms parallel cognitive maps that reflect the embedding of objects in diverse relational structures.

Mental search of concepts is supported by egocentric vector representations and restructured grid maps.

The human hippocampal-entorhinal system is known to represent both spatial locations and abstract concepts in memory in the form of allocentric cognitive maps. Using fMRI, we show that the human parietal cortex evokes complementary egocentric representations in conceptual spaces during goal-directed mental search, akin to those observable during physical navigation to determine where a goal is located relative to oneself (e.g., to our left or to our right). Concurrently, the strength of the grid-like signal, a neural signature of allocentric cognitive maps in entorhinal, prefrontal, and parietal cortices, is modulated as a function of goal proximity in conceptual space. These brain mechanisms might support flexible and parallel readout of where target conceptual information is stored in memory, capitalizing on complementary reference frames.

Dissociating distinct cortical networks associated with subregions of the human medial temporal lobe using precision neuroimaging.

Tract-tracing studies in primates indicate that different subregions of the medial temporal lobe (MTL) are connected with multiple brain regions. However, no clear framework defining the distributed anatomy associated with the human MTL exists. This gap in knowledge originates in notoriously low MRI data quality in the anterior human MTL and in group-level blurring of idiosyncratic anatomy between adjacent brain regions, such as entorhinal and perirhinal cortices, and parahippocampal areas TH/TF. Using MRI, we intensively scanned four human individuals and collected whole-brain data with unprecedented MTL signal quality. Following detailed exploration of cortical networks associated with MTL subregions within each individual, we discovered three biologically meaningful networks associated with the entorhinal cortex, perirhinal cortex, and parahippocampal area TH, respectively. Our findings define the anatomical constraints within which human mnemonic functions must operate and are insightful for examining the evolutionary trajectory of the MTL connectivity across species.

Hippocampal spatio-predictive cognitive maps adaptively guide reward generalization.

The brain forms cognitive maps of relational knowledge-an organizing principle thought to underlie our ability to generalize and make inferences. However, how can a relevant map be selected in situations where a stimulus is embedded in multiple relational structures? Here, we find that both spatial and predictive cognitive maps influence generalization in a choice task, where spatial location determines reward magnitude. Mirroring behavior, the hippocampus not only builds a map of spatial relationships but also encodes the experienced transition structure. As the task progresses, participants' choices become more influenced by spatial relationships, reflected in a strengthening of the spatial map and a weakening of the predictive map. This change is driven by orbitofrontal cortex, which represents the degree to which an outcome is consistent with the spatial rather than the predictive map and updates hippocampal representations accordingly. Taken together, this demonstrates how hippocampal cognitive maps are used and updated flexibly for inference.

Imagining is not seeing: lower insight-driven memory reconfiguration when imagining the link between separate events.

Gaining insight into the relationship between previously separate events allows us to combine these events into coherent episodes. This insight may occur via observation or imagination. Although much of our reasoning occurs in the absence of direct sensory stimuli, how mnemonic integration is accomplished via imagination has remained completely unknown. Here, we combined fMRI with representational similarity analysis and a real-life-like narrative-insight task (NIT) to elucidate the behavioral and neural effects of insight through imagination (vs. observation). Healthy participants performed the NIT in the MRI scanner and underwent memory testing one week later. Crucially, participants in the observation group gained insight through a video, while participants in the imagination group gained insight through an imagination instruction. Although we show that insight via imagination was weaker than insight via direct observation, the imagination group showed better detail memory. Moreover, the imagination group showed no representational change in the anterior hippocampus or increases in frontal and striatal activity for the linked events, as was the case in the observation group. However, the hippocampus and striatum were more activated during linking via imagination, which might indicate that their increased recruitment during imagination impedes concurrent mnemonic integration but may facilitate long-term memory.

Stress disrupts insight-driven mnemonic reconfiguration in the medial temporal lobe.

Memories are not stored in isolation. Insight into the relationship of initially unrelated events may trigger a flexible reconfiguration of the mnemonic representation of these events. Such representational changes allow the integration of events into coherent episodes and help to build up-to-date-models of the world around us. This process is, however, frequently impaired in stress-related mental disorders resulting in symptoms such as fragmented memories in PTSD. Here, we combined a real life-like narrative-insight task, in which participants learned how initially separate events are linked, with fMRI-based representational similarity analysis to test if and how acute stress interferes with the insight-driven reconfiguration of memories. Our results showed that stress reduced the activity of medial temporal and prefrontal areas when participants gained insight into the link between events. Moreover, stress abolished the insight-related increase in representational dissimilarity for linked events in the anterior part of the hippocampus as well as its association with measures of subsequent memory that we observed in non-stressed controls. However, memory performance, as assessed in a forced-choice recognition test, was even enhanced in the stress group. Our findings suggest that acute stress impedes the neural integration of events into coherent episodes but promotes long-term memory for these integrated narratives and may thus have implications for understanding memory distortions in stress-related mental disorders.

Inducing a mental context for associative memory formation with real-time fMRI neurofeedback.

Memory, one of the hallmarks of human cognition, can be modified when humans voluntarily modulate neural population activity using neurofeedback. However, it is currently unknown whether neurofeedback can influence the integration of memories, and whether memory is facilitated or impaired after such neural perturbation. In this study, participants memorized objects while we provided them with abstract neurofeedback based on their brain activity patterns in the ventral visual stream. This neurofeedback created an implicit face or house context in the brain while memorizing the objects. The results revealed that participants created associations between each memorized object and its implicit context solely due to the neurofeedback manipulation. Our findings shed light onto how memory formation can be influenced by synthetic memory tags with neurofeedback and advance our understanding of mnemonic processing.

The CABB dataset: A multimodal corpus of communicative interactions for behavioural and neural analyses.

We present a dataset of behavioural and fMRI observations acquired in the context of humans involved in multimodal referential communication. The dataset contains audio/video and motion-tracking recordings of face-to-face, task-based communicative interactions in Dutch, as well as behavioural and neural correlates of participants' representations of dialogue referents. Seventy-one pairs of unacquainted participants performed two interleaved interactional tasks in which they described and located 16 novel geometrical objects (i.e., Fribbles) yielding spontaneous interactions of about one hour. We share high-quality video (from three cameras), audio (from head-mounted microphones), and motion-tracking (Kinect) data, as well as speech transcripts of the interactions. Before and after engaging in the face-to-face communicative interactions, participants' individual representations of the 16 Fribbles were estimated. Behaviourally, participants provided a written description (one to three words) for each Fribble and positioned them along 29 independent conceptual dimensions (e.g., rounded, human, audible). Neurally, fMRI signal evoked by each Fribble was measured during a one-back working-memory task. To enable functional hyperalignment across participants, the dataset also includes fMRI measurements obtained during visual presentation of eight animated movies (35 min total). We present analyses for the various types of data demonstrating their quality and consistency with earlier research. Besides high-resolution multimodal interactional data, this dataset includes different correlates of communicative referents, obtained before and after face-to-face dialogue, allowing for novel investigations into the relation between communicative behaviours and the representational space shared by communicators. This unique combination of data can be used for research in neuroscience, psychology, linguistics, and beyond.

Rapid encoding of task regularities in the human hippocampus guides sensorimotor timing.

The brain encodes the statistical regularities of the environment in a task-specific yet flexible and generalizable format. Here, we seek to understand this process by bridging two parallel lines of research, one centered on sensorimotor timing, and the other on cognitive mapping in the hippocampal system. By combining functional magnetic resonance imaging (fMRI) with a fast-paced time-to-contact (TTC) estimation task, we found that the hippocampus signaled behavioral feedback received in each trial as well as performance improvements across trials along with reward-processing regions. Critically, it signaled performance improvements independent from the tested intervals, and its activity accounted for the trial-wise regression-to-the-mean biases in TTC estimation. This is in line with the idea that the hippocampus supports the rapid encoding of temporal context even on short time scales in a behavior-dependent manner. Our results emphasize the central role of the hippocampus in statistical learning and position it at the core of a brain-wide network updating sensorimotor representations in real time for flexible behavior.

Cognitive map formation through tactile map navigation in visually impaired and sighted persons.

The human brain can form cognitive maps of a spatial environment, which can support wayfinding. In this study, we investigated cognitive map formation of an environment presented in the tactile modality, in visually impaired and sighted persons. In addition, we assessed the acquisition of route and survey knowledge. Ten persons with a visual impairment (PVIs) and ten sighted control participants learned a tactile map of a city-like environment. The map included five marked locations associated with different items. Participants subsequently estimated distances between item pairs, performed a direction pointing task, reproduced routes between items and recalled item locations. In addition, we conducted questionnaires to assess general navigational abilities and the use of route or survey strategies. Overall, participants in both groups performed well on the spatial tasks. Our results did not show differences in performance between PVIs and sighted persons, indicating that both groups formed an equally accurate cognitive map. Furthermore, we found that the groups generally used similar navigational strategies, which correlated with performance on some of the tasks, and acquired similar and accurate route and survey knowledge. We therefore suggest that PVIs are able to employ a route as well as survey strategy if they have the opportunity to access route-like as well as map-like information such as on a tactile map.

Cognitive map formation supported by auditory, haptic, and multimodal information in persons with blindness.

For efficient navigation, the brain needs to adequately represent the environment in a cognitive map. In this review, we sought to give an overview of literature about cognitive map formation based on non-visual modalities in persons with blindness (PWBs) and sighted persons. The review is focused on the auditory and haptic modalities, including research that combines multiple modalities and real-world navigation. Furthermore, we addressed implications of route and survey representations. Taking together, PWBs as well as sighted persons can build up cognitive maps based on non-visual modalities, although the accuracy sometime somewhat differs between PWBs and sighted persons. We provide some speculations on how to deploy information from different modalities to support cognitive map formation. Furthermore, PWBs and sighted persons seem to be able to construct route as well as survey representations. PWBs can experience difficulties building up a survey representation, but this is not always the case, and research suggests that they can acquire this ability with sufficient spatial information or training. We discuss possible explanations of these inconsistencies.

Mnemonic construction and representation of temporal structure in the hippocampal formation.

The hippocampal-entorhinal region supports memory for episodic details, such as temporal relations of sequential events, and mnemonic constructions combining experiences for inferential reasoning. However, it is unclear whether hippocampal event memories reflect temporal relations derived from mnemonic constructions, event order, or elapsing time, and whether these sequence representations generalize temporal relations across similar sequences. Here, participants mnemonically constructed times of events from multiple sequences using infrequent cues and their experience of passing time. After learning, event representations in the anterior hippocampus reflected temporal relations based on constructed times. Temporal relations were generalized across sequences, revealing distinct representational formats for events from the same or different sequences. Structural knowledge about time patterns, abstracted from different sequences, biased the construction of specific event times. These findings demonstrate that mnemonic construction and the generalization of relational knowledge combine in the hippocampus, consistent with the simulation of scenarios from episodic details and structural knowledge.

Adaptive cognitive maps for curved surfaces in the 3D world.

Terrains in a 3D world can be undulating. Yet, most prior research has exclusively investigated spatial representations on a flat surface, leaving a 2D cognitive map as the dominant model in the field. Here, we investigated whether humans represent a curved surface by building a dimension-reduced flattened 2D map or a full 3D map. Participants learned the location of objects positioned on a flat and curved surface in a virtual environment by driving on the concave side of the surface (Experiment 1), driving and looking vertically (Experiment 2), or flying (Experiment 3). Subsequently, they were asked to retrieve either the path distance or the 3D Euclidean distance between the objects. Path distance estimation was good overall, but we found a significant underestimation bias for the path distance on the curve, suggesting an influence of potential 3D shortcuts, even though participants were only driving on the surface. Euclidean distance estimation was better when participants were exposed more to the global 3D structure of the environment by looking and flying. These results suggest that the representation of the 2D manifold, embedded in a 3D world, is neither purely 2D nor 3D. Rather, it is flexible and dependent on the behavioral experience and demand.

Lost and foundDark and Magical Places: The Neuroscience of Navigation Christopher Kemp Norton, 2022. 256 pp.

A lyrical meditation on wayfinding offers cultural context and hope for the navigationally challenged.

Magnetic resonance-based eye tracking using deep neural networks.

Viewing behavior provides a window into many central aspects of human cognition and health, and it is an important variable of interest or confound in many functional magnetic resonance imaging (fMRI) studies. To make eye tracking freely and widely available for MRI research, we developed DeepMReye, a convolutional neural network (CNN) that decodes gaze position from the magnetic resonance signal of the eyeballs. It performs cameraless eye tracking at subimaging temporal resolution in held-out participants with little training data and across a broad range of scanning protocols. Critically, it works even in existing datasets and when the eyes are closed. Decoded eye movements explain network-wide brain activity also in regions not associated with oculomotor function. This work emphasizes the importance of eye tracking for the interpretation of fMRI results and provides an open source software solution that is widely applicable in research and clinical settings.