Language models, like humans, show content effects on reasoning tasks.

reasoning is a key ability for an intelligent system. Large language models (LMs) achieve above-chance performance on abstract reasoning tasks but exhibit many imperfections. However, human abstract reasoning is also imperfect. Human reasoning is affected by our real-world knowledge and beliefs, and shows notable "content effects"; humans reason more reliably when the semantic content of a problem supports the correct logical inferences. These content-entangled reasoning patterns are central to debates about the fundamental nature of human intelligence. Here, we investigate whether language models-whose prior expectations capture some aspects of human knowledge-similarly mix content into their answers to logic problems. We explored this question across three logical reasoning tasks: natural language inference, judging the logical validity of syllogisms, and the Wason selection task. We evaluate state of the art LMs, as well as humans, and find that the LMs reflect many of the same qualitative human patterns on these tasks-like humans, models answer more accurately when the semantic content of a task supports the logical inferences. These parallels are reflected in accuracy patterns, and in some lower-level features like the relationship between LM confidence over possible answers and human response times. However, in some cases the humans and models behave differently-particularly on the Wason task, where humans perform much worse than large models, and exhibit a distinct error pattern. Our findings have implications for understanding possible contributors to these human cognitive effects, as well as the factors that influence language model performance.

Backtracking during navigation is correlated with enhanced anterior cingulate activity and suppression of alpha oscillations and the 'default-mode' network.

Successful navigation can require realizing the current path choice was a mistake and the best strategy is to retreat along the recent path: 'back-track'. Despite the wealth of studies on the neural correlates of navigation little is known about backtracking. To explore the neural underpinnings of backtracking we tested humans during functional magnetic resonance imaging on their ability to navigate to a set of goal locations in a virtual desert island riven by lava which constrained the paths that could be taken. We found that on a subset of trials, participants spontaneously chose to backtrack and that the majority of these choices were optimal. During backtracking, activity increased in frontal regions and the dorsal anterior cingulate cortex, while activity was suppressed in regions associated with the core default-mode network. Using the same task, magnetoencephalography and a separate group of participants, we found that power in the alpha band was significantly decreased immediately prior to such backtracking events. These results highlight the importance for navigation of brain networks previously identified in processing internally-generated errors and that such error-detection responses may involve shifting the brain from default-mode states to aid successful spatial orientation.

Correction: An Unexpected Sequence of Events: Mismatch Detection in the Human Hippocampus.

[This corrects the article DOI: 10.1371/journal.pbio.0040424.].

Prefrontal Dynamics Associated with Efficient Detours and Shortcuts: A Combined Functional Magnetic Resonance Imaging and Magnetoencenphalography Study.

Central to the concept of the "cognitive map" is that it confers behavioral flexibility, allowing animals to take efficient detours, exploit shortcuts, and avoid alluring, but unhelpful, paths. The neural underpinnings of such naturalistic and flexible behavior remain unclear. In two neuroimaging experiments, we tested human participants on their ability to navigate to a set of goal locations in a virtual desert island riven by lava, which occasionally spread to block selected paths (necessitating detours) or receded to open new paths (affording real shortcuts or false shortcuts to be avoided). Detours activated a network of frontal regions compared with shortcuts. Activity in the right dorsolateral PFC specifically increased when participants encountered tempting false shortcuts that led along suboptimal paths that needed to be differentiated from real shortcuts. We also report modulation in event-related fields and theta power in these situations, providing insight to the temporal evolution of response to encountering detours and shortcuts. These results help inform current models as to how the brain supports navigation and planning in dynamic environments.

A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.

The game of chess is the longest-studied domain in the history of artificial intelligence. The strongest programs are based on a combination of sophisticated search techniques, domain-specific adaptations, and handcrafted evaluation functions that have been refined by human experts over several decades. By contrast, the AlphaGo Zero program recently achieved superhuman performance in the game of Go by reinforcement learning from self-play. In this paper, we generalize this approach into a single AlphaZero algorithm that can achieve superhuman performance in many challenging games. Starting from random play and given no domain knowledge except the game rules, AlphaZero convincingly defeated a world champion program in the games of chess and shogi (Japanese chess), as well as Go.

Big-Loop Recurrence within the Hippocampal System Supports Integration of Information across Episodes.

Recent evidence challenges the widely held view that the hippocampus is specialized for episodic memory, by demonstrating that it also underpins the integration of information across experiences. Contemporary computational theories propose that these two contrasting functions can be accomplished by big-loop recurrence, whereby the output of the system is recirculated back into the hippocampus. We use ultra-high-resolution fMRI to provide support for this hypothesis, by showing that retrieved information is presented as a new input on the superficial entorhinal cortex-driven by functional connectivity between the deep and superficial entorhinal layers. Further, the magnitude of this laminar connectivity correlated with inferential performance, demonstrating its importance for behavior. Our findings offer a novel perspective on information processing within the hippocampus and support a unifying framework in which the hippocampus captures higher-order structure across experiences, by creating a dynamic memory space from separate episodic codes for individual experiences.

Prefrontal cortex as a meta-reinforcement learning system.

Over the past 20 years, neuroscience research on reward-based learning has converged on a canonical model, under which the neurotransmitter dopamine 'stamps in' associations between situations, actions and rewards by modulating the strength of synaptic connections between neurons. However, a growing number of recent findings have placed this standard model under strain. We now draw on recent advances in artificial intelligence to introduce a new theory of reward-based learning. Here, the dopamine system trains another part of the brain, the prefrontal cortex, to operate as its own free-standing learning system. This new perspective accommodates the findings that motivated the standard model, but also deals gracefully with a wider range of observations, providing a fresh foundation for future research.

Vector-based navigation using grid-like representations in artificial agents.

Deep neural networks have achieved impressive successes in fields ranging from object recognition to complex games such as Go1,2. Navigation, however, remains a substantial challenge for artificial agents, with deep neural networks trained by reinforcement learning3-5 failing to rival the proficiency of mammalian spatial behaviour, which is underpinned by grid cells in the entorhinal cortex 6 . Grid cells are thought to provide a multi-scale periodic representation that functions as a metric for coding space7,8 and is critical for integrating self-motion (path integration)6,7,9 and planning direct trajectories to goals (vector-based navigation)7,10,11. Here we set out to leverage the computational functions of grid cells to develop a deep reinforcement learning agent with mammal-like navigational abilities. We first trained a recurrent network to perform path integration, leading to the emergence of representations resembling grid cells, as well as other entorhinal cell types 12 . We then showed that this representation provided an effective basis for an agent to locate goals in challenging, unfamiliar, and changeable environments-optimizing the primary objective of navigation through deep reinforcement learning. The performance of agents endowed with grid-like representations surpassed that of an expert human and comparison agents, with the metric quantities necessary for vector-based navigation derived from grid-like units within the network. Furthermore, grid-like representations enabled agents to conduct shortcut behaviours reminiscent of those performed by mammals. Our findings show that emergent grid-like representations furnish agents with a Euclidean spatial metric and associated vector operations, providing a foundation for proficient navigation. As such, our results support neuroscientific theories that see grid cells as critical for vector-based navigation7,10,11, demonstrating that the latter can be combined with path-based strategies to support navigation in challenging environments.

Age-related functional changes in domain-specific medial temporal lobe pathways.

There is now converging evidence from studies in animals and humans that the medial temporal lobes (MTLs) harbor anatomically distinct processing pathways for object and scene information. Recent functional magnetic resonance imaging studies in humans suggest that this domain-specific organization may be associated with a functional preference of the anterior-lateral part of the entorhinal cortex (alErC) for objects and the posterior-medial entorhinal cortex (pmErC) for scenes. As MTL subregions are differentially affected by aging and neurodegenerative diseases, the question was raised whether aging may affect the 2 pathways differentially. To address this possibility, we developed a paradigm that allows the investigation of object memory and scene memory in a mnemonic discrimination task. A group of young (n = 43) and healthy older subjects (n = 44) underwent functional magnetic resonance imaging recordings during this novel task, while they were asked to discriminate exact repetitions of object and scene stimuli from novel stimuli that were similar but modified versions of the original stimuli ("lures"). We used structural magnetic resonance images to manually segment anatomical components of the MTL including alErC and pmErC and used these segmented regions to analyze domain specificity of functional activity. Across the entire sample, object processing was associated with activation of the perirhinal cortex (PrC) and alErC, whereas for scene processing, activation was more predominant in the parahippocampal cortex and pmErC. Functional activity related to mnemonic discrimination of object and scene lures from exact repetitions was found to overlap between processing pathways and suggests that while the PrC-alErC pathway was more involved in object discrimination, both pathways were involved in the discrimination of similar scenes. Older adults were behaviorally less accurate than young adults in discriminating similar lures from exact repetitions, but this reduction was equivalent in both domains. However, this was accompanied by significantly reduced domain-specific activity in PrC in older adults compared to what was observed in the young. Furthermore, this reduced domain-specific activity was associated to worse performance in object mnemonic discrimination in older adults. Taken together, we show the fine-grained functional organization of the MTL into domain-specific pathways for objects and scenes and their mnemonic discrimination and further provide evidence that aging might affect these pathways in a differential fashion. Future experiments will elucidate whether the 2 pathways are differentially affected in early stages of Alzheimer's disease in relation to amyloid or tau pathology.

Prior preferences beneficially influence social and non-social learning.

Our personal preferences affect a broad array of social behaviors. This includes the way we learn the preferences of others, an ability that often relies on limited or ambiguous information. Here we report an egocentric influence on this type of social learning that is reflected in both performance and response times. Using computational models that combine inter-trial learning and intra-trial choice, we find transient effects of participants' preferences on the learning process, through the influence of priors, and persistent effects on the choice process. A second experiment shows that these effects generalize to non-social learning, though participants in the social learning experiment appeared to additionally benefit by using their knowledge about the popularity of certain preferences. We further find that the domain-general egocentric influences we identify can yield performance advantages in uncertain environments.People often assume that other people share their preferences, but how exactly this bias manifests itself in learning and decision-making is unclear. Here, authors show that a person's own preferences influence learning in both social and non-social situations, and that this bias improves performance.

Neuroscience-Inspired Artificial Intelligence.

The fields of neuroscience and artificial intelligence (AI) have a long and intertwined history. In more recent times, however, communication and collaboration between the two fields has become less commonplace. In this article, we argue that better understanding biological brains could play a vital role in building intelligent machines. We survey historical interactions between the AI and neuroscience fields and emphasize current advances in AI that have been inspired by the study of neural computation in humans and other animals. We conclude by highlighting shared themes that may be key for advancing future research in both fields.

Overcoming catastrophic forgetting in neural networks.

The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

Computations Underlying Social Hierarchy Learning: Distinct Neural Mechanisms for Updating and Representing Self-Relevant Information.

Knowledge about social hierarchies organizes human behavior, yet we understand little about the underlying computations. Here we show that a Bayesian inference scheme, which tracks the power of individuals, better captures behavioral and neural data compared with a reinforcement learning model inspired by rating systems used in games such as chess. We provide evidence that the medial prefrontal cortex (MPFC) selectively mediates the updating of knowledge about one's own hierarchy, as opposed to that of another individual, a process that underpinned successful performance and involved functional interactions with the amygdala and hippocampus. In contrast, we observed domain-general coding of rank in the amygdala and hippocampus, even when the task did not require it. Our findings reveal the computations underlying a core aspect of social cognition and provide new evidence that self-relevant information may indeed be afforded a unique representational status in the brain.

Semantic representations in the temporal pole predict false memories.

Recent advances in neuroscience have given us unprecedented insight into the neural mechanisms of false memory, showing that artificial memories can be inserted into the memory cells of the hippocampus in a way that is indistinguishable from true memories. However, this alone is not enough to explain how false memories can arise naturally in the course of our daily lives. Cognitive psychology has demonstrated that many instances of false memory, both in the laboratory and the real world, can be attributed to semantic interference. Whereas previous studies have found that a diverse set of regions show some involvement in semantic false memory, none have revealed the nature of the semantic representations underpinning the phenomenon. Here we use fMRI with representational similarity analysis to search for a neural code consistent with semantic false memory. We find clear evidence that false memories emerge from a similarity-based neural code in the temporal pole, a region that has been called the "semantic hub" of the brain. We further show that each individual has a partially unique semantic code within the temporal pole, and this unique code can predict idiosyncratic patterns of memory errors. Finally, we show that the same neural code can also predict variation in true-memory performance, consistent with an adaptive perspective on false memory. Taken together, our findings reveal the underlying structure of neural representations of semantic knowledge, and how this semantic structure can both enhance and distort our memories.

Retrieval-Based Model Accounts for Striking Profile of Episodic Memory and Generalization.

A fundamental theoretical tension exists between the role of the hippocampus in generalizing across a set of related episodes, and in supporting memory for individual episodes. Whilst the former requires an appreciation of the commonalities across episodes, the latter emphasizes the representation of the specifics of individual experiences. We developed a novel version of the hippocampal-dependent paired associate inference (PAI) paradigm, which afforded us the unique opportunity to investigate the relationship between episodic memory and generalization in parallel. Across four experiments, we provide surprising evidence that the overlap between object pairs in the PAI paradigm results in a marked loss of episodic memory. Critically, however, we demonstrate that superior generalization ability was associated with stronger episodic memory. Through computational simulations we show that this striking profile of behavioral findings is best accounted for by a mechanism by which generalization occurs at the point of retrieval, through the recombination of related episodes on the fly. Taken together, our study offers new insights into the intricate relationship between episodic memory and generalization, and constrains theories of the mechanisms by which the hippocampus supports generalization.

Strong Evidence for Pattern Separation in Human Dentate Gyrus.

UNLABELLED: The hippocampus is proposed to be critical in distinguishing between similar experiences by performing pattern separation computations that create orthogonalized representations for related episodes. Previous neuroimaging studies have provided indirect evidence that the dentate gyrus (DG) and CA3 hippocampal subregions support pattern separation by inferring the nature of underlying representations from the observation of novelty signals. Here, we use ultra-high-resolution fMRI at 7 T and multivariate pattern analysis to provide compelling evidence that the DG subregion specifically sustains representations of similar scenes that are less overlapping than in other hippocampal (e.g., CA3) and medial temporal lobe regions (e.g., entorhinal cortex). Further, we provide evidence that novelty signals within the DG are stimulus specific rather than generic in nature. Our study, in providing a mechanistic link between novelty signals and the underlying representations, constitutes the first demonstration that the human DG performs pattern separation. SIGNIFICANCE STATEMENT: A fundamental property of an episodic memory system is the ability to minimize interference between similar episodes. The dentate gyrus (DG) subregion of the hippocampus is widely viewed to realize this function through a computation referred to as pattern separation, which creates distinct nonoverlapping neural codes for individual events. Here, we leveraged 7 T fMRI to test the hypothesis that this region supports pattern separation. Our results demonstrate that the DG supports representations of similar scenes that are less overlapping than those in neighboring subregions. The current study therefore is the first to offer compelling evidence that the human DG supports pattern separation by obtaining critical empirical data at the representational level: the level where this computation is defined.

What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated.

We update complementary learning systems (CLS) theory, which holds that intelligent agents must possess two learning systems, instantiated in mammalians in neocortex and hippocampus. The first gradually acquires structured knowledge representations while the second quickly learns the specifics of individual experiences. We broaden the role of replay of hippocampal memories in the theory, noting that replay allows goal-dependent weighting of experience statistics. We also address recent challenges to the theory and extend it by showing that recurrent activation of hippocampal traces can support some forms of generalization and that neocortical learning can be rapid for information that is consistent with known structure. Finally, we note the relevance of the theory to the design of artificial intelligent agents, highlighting connections between neuroscience and machine learning.

Context-specific activation of hippocampus and SN/VTA by reward is related to enhanced long-term memory for embedded objects.

Animal studies indicate that hippocampal representations of environmental context modulate reward-related processing in the substantia nigra and ventral tegmental area (SN/VTA), a major origin of dopamine in the brain. Using functional magnetic resonance imaging (fMRI) in humans, we investigated the neural specificity of context-reward associations under conditions where the presence of perceptually similar neutral contexts imposed high demands on a putative hippocampal function, pattern separation. The design also allowed us to investigate how contextual reward enhances long-term memory for embedded neutral objects. SN/VTA activity underpinned specific context-reward associations in the face of perceptual similarity. A reward-related enhancement of long-term memory was restricted to the condition where the rewarding and the neutral contexts were perceptually similar, and in turn was linked to co-activation of the hippocampus (subfield DG/CA3) and SN/VTA. Thus, an ability of contextual reward to enhance memory for focal objects is closely linked to context-related engagement of hippocampal-SN/VTA circuitry.

Ventromedial prefrontal cortex drives hippocampal theta oscillations induced by mismatch computations.

Detecting environmental change is fundamental for adaptive behavior in an uncertain world. Previous work indicates the hippocampus supports the generation of novelty signals via implementation of a match-mismatch detector that signals when an incoming sensory input violates expectations based on past experience. While existing work has emphasized the particular contribution of the hippocampus, here we ask which other brain structures also contribute to match-mismatch detection. Furthermore, we leverage the fine-grained temporal resolution of magnetoencephalography (MEG) to investigate whether mismatch computations are spectrally confined to the theta range, based on the prominence of this range of oscillations in models of hippocampal function. By recording MEG activity while human subjects perform a task that incorporates conditions of match-mismatch novelty we show that mismatch signals are confined to the theta band and are expressed in both the hippocampus and ventromedial prefrontal cortex (vmPFC). Effective connectivity analyses (dynamic causal modeling) show that the hippocampus and vmPFC work as a functional circuit during mismatch detection. Surprisingly, our results suggest that the vmPFC drives the hippocampus during the generation and processing of mismatch signals. Our findings provide new evidence that the hippocampal-vmPFC circuit is engaged during novelty processing, which has implications for emerging theories regarding the role of vmPFC in memory.