A rubric for human-like agents and NeuroAI.

Researchers across cognitive, neuro- and computer sciences increasingly reference 'human-like' artificial intelligence and 'neuroAI'. However, the scope and use of the terms are often inconsistent. Contributed research ranges widely from mimicking behaviour, to testing machine learning methods as neurally plausible hypotheses at the cellular or functional levels, or solving engineering problems. However, it cannot be assumed nor expected that progress on one of these three goals will automatically translate to progress in others. Here, a simple rubric is proposed to clarify the scope of individual contributions, grounded in their commitments to human-like behaviour, neural plausibility or benchmark/engineering/computer science goals. This is clarified using examples of weak and strong neuroAI and human-like agents, and discussing the generative, corroborate and corrective ways in which the three dimensions interact with one another. The author maintains that future progress in artificial intelligence will need strong interactions across the disciplines, with iterative feedback loops and meticulous validity tests-leading to both known and yet-unknown advances that may span decades to come. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Representations of Temporal Community Structure in Hippocampus and Precuneus Predict Inductive Reasoning Decisions.

Our understanding of the world is shaped by inferences about underlying structure. For example, at the gym, you might notice that the same people tend to arrive around the same time and infer that they are friends that work out together. Consistent with this idea, after participants are presented with a temporal sequence of objects that follows an underlying community structure, they are biased to infer that objects from the same community share the same properties. Here, we used fMRI to measure neural representations of objects after temporal community structure learning and examine how these representations support inference about object relationships. We found that community structure learning affected inferred object similarity: When asked to spatially group items based on their experience, participants tended to group together objects from the same community. Neural representations in perirhinal cortex predicted individual differences in object grouping, suggesting that high-level object representations are affected by temporal community learning. Furthermore, participants were biased to infer that objects from the same community would share the same properties. Using computational modeling of temporal learning and inference decisions, we found that inductive reasoning is influenced by both detailed knowledge of temporal statistics and abstract knowledge of the temporal communities. The fidelity of temporal community representations in hippocampus and precuneus predicted the degree to which temporal community membership biased reasoning decisions. Our results suggest that temporal knowledge is represented at multiple levels of abstraction, and that perirhinal cortex, hippocampus, and precuneus may support inference based on this knowledge.

Predictive Representations in Hippocampal and Prefrontal Hierarchies.

As we navigate the world, we use learned representations of relational structures to explore and to reach goals. Studies of how relational knowledge enables inference and planning are typically conducted in controlled small-scale settings. It remains unclear, however, how people use stored knowledge in continuously unfolding navigation (e.g., walking long distances in a city). We hypothesized that multiscale predictive representations guide naturalistic navigation in humans, and these scales are organized along posterior-anterior prefrontal and hippocampal hierarchies. We conducted model-based representational similarity analyses of neuroimaging data collected while male and female participants navigated realistically long paths in virtual reality. We tested the pattern similarity of each point, along each path, to a weighted sum of its successor points within predictive horizons of different scales. We found that anterior PFC showed the largest predictive horizons, posterior hippocampus the smallest, with the anterior hippocampus and orbitofrontal regions in between. Our findings offer novel insights into how cognitive maps support hierarchical planning at multiple scales.SIGNIFICANCE STATEMENT Whenever we navigate the world, we represent our journey at multiple horizons: from our immediate surroundings to our distal goal. How are such cognitive maps at different horizons simultaneously represented in the brain? Here, we applied a reinforcement learning-based analysis to neuroimaging data acquired while participants virtually navigated their hometown. We investigated neural patterns in the hippocampus and PFC, key cognitive map regions. We uncovered predictive representations with multiscale horizons in prefrontal and hippocampal gradients, with the longest predictive horizons in anterior PFC and the shortest in posterior hippocampus. These findings provide empirical support for the computational hypothesis that multiscale neural representations guide goal-directed navigation. This advances our understanding of hierarchical planning in everyday navigation of realistic distances.

Collective minds: social network topology shapes collective cognition.

Human cognition is not solitary, it is shaped by collective learning and memory. Unlike swarms or herds, human social networks have diverse topologies, serving diverse modes of collective cognition and behaviour. Here, we review research that combines network structure with psychological and neural experiments and modelling to understand how the topology of social networks shapes collective cognition. First, we review graph-theoretical approaches to behavioural experiments on collective memory, belief propagation and problem solving. These results show that different topologies of communication networks synchronize or integrate knowledge differently, serving diverse collective goals. Second, we discuss neuroimaging studies showing that human brains encode the topology of one's larger social network and show similar neural patterns to neural patterns of our friends and community ties (e.g. when watching movies). Third, we discuss cognitive similarities between learning social and non-social topologies, e.g. in spatial and associative learning, as well as common brain regions involved in processing social and non-social topologies. Finally, we discuss recent machine learning approaches to collective communication and cooperation in multi-agent artificial networks. Combining network science with cognitive, neural and computational approaches empowers investigating how social structures shape collective cognition, which can in turn help design goal-directed social network topologies. This article is part of a discussion meeting issue 'The emergence of collective knowledge and cumulative culture in animals, humans and machines'.

The Learning Salon: Toward a new participatory science.

The Learning Salon is an online weekly forum for discussing points of contention and common ground in biological and artificial learning. Hosting neuroscientists, computer scientists, AI researchers, and philosophers, the Salon promotes short talks and long discussions, committed to an ethos of participation, horizontality, and inclusion.

Rational use of episodic and working memory: A normative account of prospective memory.

Humans often simultaneously pursue multiple plans at different time scales, a capacity known as prospective memory (PM). The successful realization of non-immediate plans (e.g., post package after work) requires keeping track of a future plan while accomplishing other intermediate tasks (e.g., write a paper). Prospective memory capacity requires the integration of noisy evidence from perceptual input with evidence from both short-term working memory (WM) and long-term or episodic memory (LTM/EM). Here we formulate a set of empirical studies of prospective memory, all dual-task problems, as problems of computational rationality. We ask how a rational model should integrate noisy perceptual evidence and memory to maximize payoffs in these PM studies. The model combines reinforcement learning (optimal action selection) with evidence accumulation (optimal inference) in order to derive good decision parameters for optimal task performance (i.e., performing an ongoing task while monitoring for a cue that triggers executing a second prospective task). We compare model behavior to human behavioral evidence of key accuracy and reaction time phenomena in PM. Notably, our normative approach to theorizing and modeling these phenomena makes no assumptions about mechanisms of attention or retrieval. This approach can be extended to study the learning and use of meta-parameters governing the boundedly rational use of memory in planned action in health and disease. A computational psychiatry extension of the model can capture compensatory mnemonic strategies in neuropsychiatric disorders that may be rational responses to disturbances of inference, memory, and action selection.

Anxiety, avoidance, and sequential evaluation.

Anxiety disorders are characterized by a range of aberrations in the processing of and response to threat, but there is little clarity what core pathogenesis might underlie these symptoms. Here we propose that a particular set of unrealistically pessimistic assumptions can distort an agent's behavior and underlie a host of seemingly disparate anxiety symptoms. We formalize this hypothesis in a decision theoretic analysis of maladaptive avoidance and a reinforcement learning model, which shows how a localized bias in beliefs can formally explain a range of phenomena related to anxiety. The core observation, implicit in standard decision theoretic accounts of sequential evaluation, is that the potential for avoidance should be protective: if danger can be avoided later, it poses less threat now. We show how a violation of this assumption - via a pessimistic, false belief that later avoidance will be unsuccessful - leads to a characteristic, excessive propagation of fear and avoidance to situations far antecedent of threat. This single deviation can explain a range of features of anxious behavior, including exaggerated threat appraisals, fear generalization, and persistent avoidance. Simulations of the model reproduce laboratory demonstrations of abnormal decision making in anxiety, including in situations of approach-avoid conflict and planning to avoid losses. The model also ties together a number of other seemingly disjoint phenomena in anxious disorders. For instance, learning under the pessimistic bias captures a hypothesis about the role of anxiety in the later development of depression. The bias itself offers a new formalization of classic insights from the psychiatric literature about the central role of maladaptive beliefs about control and self-efficacy in anxiety. This perspective also extends previous computational accounts of beliefs about control in mood disorders, which neglected the sequential aspects of choice.

Learning Structures: Predictive Representations, Replay, and Generalization.

Memory and planning rely on learning the structure of relationships among experiences. Compact representations of these structures guide flexible behavior in humans and animals. A century after 'latent learning' experiments summarized by Tolman, the larger puzzle of cognitive maps remains elusive: how does the brain learn and generalize relational structures? This review focuses on a reinforcement learning (RL) approach to learning compact representations of the structure of states. We review evidence showing that capturing structures as predictive representations updated via replay offers a neurally plausible account of human behavior and the neural representations of predictive cognitive maps. We highlight multi-scale successor representations, prioritized replay, and policy-dependence. These advances call for new directions in studying the entanglement of learning and memory with prediction and planning.

Bridge ties bind collective memories.

From families to nations, what binds individuals in social groups is, to a large degree, their shared beliefs, norms, and memories. These emergent outcomes are thought to occur because communication among individuals results in community-wide synchronization. Here, we use experimental manipulations in lab-created networks to investigate how the temporal dynamics of conversations shape the formation of collective memories. We show that when individuals that bridge between clusters (i.e., bridge ties) communicate early on in a series of networked interactions, the network reaches higher mnemonic convergence compared to when individuals first interact within clusters (i.e., cluster ties). This effect, we show, is due to the tradeoffs between initial information diversity and accumulated overlap over time. Our approach provides a framework to analyze and design interventions in social networks that optimize information sharing and diminish the likelihood of information bubbles and polarization.

Offline replay supports planning in human reinforcement learning.

Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether 'offline' integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.

Predictive representations can link model-based reinforcement learning to model-free mechanisms.

Humans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence suggest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation.

Mnemonic convergence in social networks: The emergent properties of cognition at a collective level.

The development of shared memories, beliefs, and norms is a fundamental characteristic of human communities. These emergent outcomes are thought to occur owing to a dynamic system of information sharing and memory updating, which fundamentally depends on communication. Here we report results on the formation of collective memories in laboratory-created communities. We manipulated conversational network structure in a series of real-time, computer-mediated interactions in fourteen 10-member communities. The results show that mnemonic convergence, measured as the degree of overlap among community members' memories, is influenced by both individual-level information-processing phenomena and by the conversational social network structure created during conversational recall. By studying laboratory-created social networks, we show how large-scale social phenomena (i.e., collective memory) can emerge out of microlevel local dynamics (i.e., mnemonic reinforcement and suppression effects). The social-interactionist approach proposed herein points to optimal strategies for spreading information in social networks and provides a framework for measuring and forging collective memories in communities of individuals.

The Role of the Parietal Cortex in the Representation of Task-Reward Associations.

Rewards obtained from specific behaviors can and do change across time. To adapt to such conditions, humans need to represent and update associations between behaviors and their outcomes. Much previous work focused on how rewards affect the processing of specific tasks. However, abstract associations between multiple potential behaviors and multiple rewards are an important basis for adaptation as well. In this experiment, we directly investigated which brain areas represent associations between multiple tasks and rewards, using time-resolved multivariate pattern analysis of functional magnetic resonance imaging data. Importantly, we were able to dissociate neural signals reflecting task-reward associations from those related to task preparation and reward expectation processes, variables that were often correlated in previous research. We hypothesized that brain regions involved in processing tasks and/or rewards will be involved in processing associations between them. Candidate areas included the dorsal anterior cingulate cortex, which is involved in associating simple actions and rewards, and the parietal cortex, which has been shown to represent task rules and action values. Our results indicate that local spatial activation patterns in the inferior parietal cortex indeed represent task-reward associations. Interestingly, the parietal cortex flexibly changes its content of representation within trials. It first represents task-reward associations, later switching to process tasks and rewards directly. These findings highlight the importance of the inferior parietal cortex in associating behaviors with their outcomes and further show that it can flexibly reconfigure its function within single trials. Significance statement: Rewards obtained from specific behaviors rarely remain constant over time. To adapt to changing conditions, humans need to continuously update and represent the current association between behavior and its outcomes. However, little is known about the neural representation of behavior-outcome associations. Here, we used multivariate pattern analysis of functional magnetic resonance imaging data to investigate the neural correlates of such associations. Our results demonstrate that the parietal cortex plays a central role in representing associations between multiple behaviors and their outcomes. They further highlight the flexibility of the parietal cortex, because we find it to adapt its function to changing task demands within trials on relatively short timescales.

Encoding of prospective tasks in the human prefrontal cortex under varying task loads.

Successful realization of planned actions requires the brain to encode intentions over delays. Previous research has indicated that several regions in the rostral or anterior prefrontal cortex (PFC) encode delayed intentions. However, different processes may encode the same future task depending on task load during the delay. This difference may depend on the computational resources available when the delay is occupied with an ongoing task and when it is task-free. Here we directly investigated and compared the representation of delayed intentions in the human brain in the presence and absence of ongoing task load during the delay. We acquired fMRI data in combination with an event-based prospective memory design where human subjects remembered to perform the same future tasks over occupied and task-free delays. We used time-resolved multivoxel pattern classification and found that: (1) rostrolateral PFC (BA 46) encoded the delayed intention during both delay types; (2) rostromedial PFC (BA 10) encoded the intentions during occupied delays; whereas (3) a variety of more posterior regions, including the anterior cingulate cortex (BA 24), the supplementary motor area (BA 6), and the precuneus, encoded intentions during task-free delays. Overall, the medial PFC encoded delayed intentions more rostrally in the presence of an ongoing delay task and more caudally in its absence. Thus, rostromedial PFC may play a specialized role in the encoding of prospective memory that depends on higher computational demands (e.g., given higher task load during the delay). In contrast, the rostrolateral PFC is a more general area that encodes future intentions regardless of task load.

Human anterior prefrontal cortex encodes the 'what' and 'when' of future intentions.

On a daily basis we form numerous intentions to perform specific actions. However, we often have to delay the execution of intended actions while engaging in other demanding activities. Previous research has shown that patterns of activity in human prefrontal cortex (PFC) can reveal our current intentions. However, two fundamental questions have remained unresolved: (a) how does the PFC encode information about future tasks while we are busy engaging in other activities, and (b) how does the PFC enable us to commence a stored task at the intended time? Here we investigate how the brain stores and retrieves future intentions during occupied delays, i.e. while a person is busy performing a different task. For this purpose, we conducted a neuroimaging study with a time-based prospective memory paradigm. Using multivariate pattern classification and fMRI we show that during an occupied delay, activity patterns in the anterior PFC encode the content of 'what' subjects intend to do next, and 'when' they intend to do it. Importantly, distinct anterior PFC regions store the 'what' and 'when' components of future intentions during occupied maintenance and self-initiated retrieval. These results show a role for anterior PFC activity patterns in storing future action plans and ensuring their timely retrieval.