Humans adaptively select different computational strategies in different learning environments.

The Rescorla-Wagner rule remains the most popular tool to describe human behavior in reinforcement learning tasks. Nevertheless, it cannot fit human learning in complex environments. Previous work proposed several hierarchical extensions of this learning rule. However, it remains unclear when a flat (nonhierarchical) versus a hierarchical strategy is adaptive, or when it is implemented by humans. To address this question, current work applies a nested modeling approach to evaluate multiple models in multiple reinforcement learning environments both computationally (which approach performs best) and empirically (which approach fits human data best). We consider 10 empirical data sets (N = 407) divided over three reinforcement learning environments. Our results demonstrate that different environments are best solved with different learning strategies; and that humans adaptively select the learning strategy that allows best performance. Specifically, while flat learning fitted best in less complex stable learning environments, humans employed more hierarchically complex models in more complex environments. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

How much data do we need to estimate computational models of decision-making? The COMPASS toolbox.

How much data are needed to obtain useful parameter estimations from a computational model? The standard approach to address this question is to carry out a goodness-of-recovery study. Here, the correlation between individual-participant true and estimated parameter values determines when a sample size is large enough. However, depending on one's research question, this approach may be suboptimal, potentially leading to sample sizes that are either too small (underpowered) or too large (overcostly or unfeasible). In this paper, we formulate a generalized concept of statistical power and use this to propose a novel approach toward determining how much data is needed to obtain useful parameter estimates from a computational model. We describe a Python-based toolbox (COMPASS) that allows one to determine how many participants are needed to fit one specific computational model, namely the Rescorla-Wagner model of learning and decision-making. Simulations revealed that a high number of trials per person (more than the number of persons) are a prerequisite for high-powered studies in this particular setting.

Modulating hierarchical learning by high-definition transcranial alternating current stimulation at theta frequency.

Considerable evidence highlights the dorsolateral prefrontal cortex (DLPFC) as a key region for hierarchical (i.e. multilevel) learning. In a previous electroencephalography (EEG) study, we found that the low-level prediction errors were encoded by frontal theta oscillations (4-7 Hz), centered on right DLPFC (rDLPFC). However, the causal relationship between frontal theta oscillations and hierarchical learning remains poorly understood. To investigate this question, in the current study, participants received theta (6 Hz) and sham high-definition transcranial alternating current stimulation (HD-tACS) over the rDLPFC while performing the probabilistic reversal learning task. Behaviorally, theta tACS induced a significant reduction in accuracy for the stable environment, but not for the volatile environment, relative to the sham condition. Computationally, we implemented a combination of a hierarchical Bayesian learning and a decision model. Theta tACS induced a significant increase in low-level (i.e. probability-level) learning rate and uncertainty of low-level estimation relative to sham condition. Instead, the temperature parameter of the decision model, which represents (inverse) decision noise, was not significantly altered due to theta stimulation. These results indicate that theta frequency may modulate the (low-level) learning rate. Furthermore, environmental features (e.g. its stability) may determine whether learning is optimized as a result.

Theta oscillations shift towards optimal frequency for cognitive control.

Cognitive control allows to flexibly guide behaviour in a complex and ever-changing environment. It is supported by theta band (4-7 Hz) neural oscillations that coordinate distant neural populations. However, little is known about the precise neural mechanisms permitting such flexible control. Most research has focused on theta amplitude, showing that it increases when control is needed, but a second essential aspect of theta oscillations, their peak frequency, has mostly been overlooked. Here, using computational modelling and behavioural and electrophysiological recordings, in three independent datasets, we show that theta oscillations adaptively shift towards optimal frequency depending on task demands. We provide evidence that theta frequency balances reliable set-up of task representation and gating of task-relevant sensory and motor information and that this frequency shift predicts behavioural performance. Our study presents a mechanism supporting flexible control and calls for a reevaluation of the mechanistic role of theta oscillations in adaptive behaviour.

Using top-down modulation to optimally balance shared versus separated task representations.

Human adaptive behavior requires continually learning and performing a wide variety of tasks, often with very little practice. To accomplish this, it is crucial to separate neural representations of different tasks in order to avoid interference. At the same time, sharing neural representations supports generalization and allows faster learning. Therefore, a crucial challenge is to find an optimal balance between shared versus separated representations. Typically, models of human cognition employ top-down modulatory signals to separate task representations, but there exist surprisingly little systematic computational investigations of how such modulation is best implemented. We identify and systematically evaluate two crucial features of modulatory signals. First, top-down input can be processed in an additive or multiplicative manner. Second, the modulatory signals can be adaptive (learned) or non-adaptive (random). We cross these two features, resulting in four modulation networks which are tested on a variety of input datasets and tasks with different degrees of stimulus-action mapping overlap. The multiplicative adaptive modulation network outperforms all other networks in terms of accuracy. Moreover, this network develops hidden units that optimally share representations between tasks. Specifically, different than the binary approach of currently popular latent state models, it exploits partial overlap between tasks.

Neural Synchrony for Adaptive Control.

Cognitive control can be adaptive along several dimensions, including intensity (how intensely do control signals influence bottom-up processing) and selectivity (what information is selected for further processing). Furthermore, control can be exerted along slow or fast time scales. Whereas control on a slow time scale is used to proactively prepare for upcoming challenges, control can also be used on a faster time scale to react to unexpected events that require control. Importantly, a systematic comparison of these dimensions and time scales remains lacking. Moreover, most current models of adaptive control allow predictions only at a behavioral, not neurophysiological, level, thus seriously reducing the range of available empirical restrictions for informing model formulation. The current article addresses this issue by implementing a control loop in an earlier model of neural synchrony. The resulting model is tested on a Stroop task. We observe that only the model that exerts cognitive control on intensity and selectivity dimensions, as well as on two time scales, can account for relevant behavioral and neurophysiological data. Our findings hold important implications for both cognitive control and how computational models can be empirically constrained.

A direct test of the similarity assumption - Focusing on differences as compared with similarities decreases automatic imitation.

Individuals automatically imitate a wide range of different behaviors. Previous research suggests that imitation as a social process depends on the similarity between interaction partners. However, some of the experiments supporting this notion could not be replicated and all of the supporting experiments manipulated not only similarity between actor and observer, but also other features. Thus, the existing evidence leaves open whether similarity as such moderates automatic imitation. To directly test the similarity account, in four experiments, we manipulated participants' focus on similarities or differences while holding the stimulus material constant. In Experiment 1, we presented participants with a hand and let them either focus on similarities, differences, or neutral aspects between their own hand and the other person's hand. The results indicate that focusing on similarities increased perceived similarity between the own and the other person's hand. In Experiments 2 to 4, we tested the hypothesis that focusing on similarities, as compared with differences, increases automatic imitation. Experiment 2 tested the basic effect and found support for our prediction. Experiment 3 and 4 replicated this finding with higher-powered samples. Exploratory investigations further suggest that it is a focus on differences that decreases automatic imitation, and not a focus on similarities that increases automatic imitation. Theoretical implications and future directions are discussed.

Theta and alpha power across fast and slow timescales in cognitive control.

Theta and alpha frequency neural oscillations are important for learning and cognitive control, but their exact role has remained obscure. In particular, it is unknown whether they operate at similar timescales, and whether they support different cognitive processes. We recorded EEG in 30 healthy human participants while they performed a learning task containing both novel (block-unique) and repeating stimuli. We investigated behavior and electrophysiology at both fast (i.e., within blocks) and slow (i.e., between blocks) timescales. Behaviorally, both response time and accuracy improved (respectively decrease and increase) over both fast and slow timescales. However, on the spectral level, theta power significantly decreased along the slow timescale, whereas alpha power significantly increased along the fast timescale. We thus demonstrate that theta and alpha both play a role during learning, but operate at different timescales. This result poses important empirical constraints for theories on learning, cognitive control, and neural oscillations.

Learning to Synchronize: Midfrontal Theta Dynamics during Rule Switching.

In recent years, several hierarchical extensions of well-known learning algorithms have been proposed. For example, when stimulus-action mappings vary across time or context, the brain may learn two or more stimulus-action mappings in separate modules, and additionally (at a hierarchically higher level) learn to appropriately switch between those modules. However, how the brain mechanistically coordinates neural communication to implement such hierarchical learning remains unknown. Therefore, the current study tests a recent computational model that proposed how midfrontal theta oscillations implement such hierarchical learning via the principle of binding by synchrony (Sync model). More specifically, the Sync model uses bursts at theta frequency to flexibly bind appropriate task modules by synchrony. The 64-channel EEG signal was recorded while 27 human subjects (female: 21, male: 6) performed a probabilistic reversal learning task. In line with the Sync model, postfeedback theta power showed a linear relationship with negative prediction errors, but not with positive prediction errors. This relationship was especially pronounced for subjects with better behavioral fit (measured via Akaike information criterion) of the Sync model. Also consistent with Sync model simulations, theta phase-coupling between midfrontal electrodes and temporoparietal electrodes was stronger after negative feedback. Our data suggest that the brain uses theta power and synchronization for flexibly switching between task rule modules, as is useful, for example, when multiple stimulus-action mappings must be retained and used.SIGNIFICANCE STATEMENT Everyday life requires flexibility in switching between several rules. A key question in understanding this ability is how the brain mechanistically coordinates such switches. The current study tests a recent computational framework (Sync model) that proposed how midfrontal theta oscillations coordinate activity in hierarchically lower task-related areas. In line with predictions of this Sync model, midfrontal theta power was stronger when rule switches were most likely (strong negative prediction error), especially in subjects who obtained a better model fit. Additionally, also theta phase connectivity between midfrontal and task-related areas was increased after negative feedback. Thus, the data provided support for the hypothesis that the brain uses theta power and synchronization for flexibly switching between rules.

Time-Based Binding as a Solution to and a Limitation for Flexible Cognition.

Why can't we keep as many items as we want in working memory? It has long been debated whether this resource limitation is a bug (a downside of our fallible biological system) or instead a feature (an optimal response to a computational problem). We propose that the resource limitation is a consequence of a useful feature. Specifically, we propose that flexible cognition requires time-based binding, and time-based binding necessarily limits the number of (bound) memoranda that can be stored simultaneously. Time-based binding is most naturally instantiated via neural oscillations, for which there exists ample experimental evidence. We report simulations that illustrate this theory and that relate it to empirical data. We also compare the theory to several other (feature and bug) resource theories.

The influence of threat on perceived spatial distance to out-group members.

A classic example of discriminatory behavior is keeping spatial distance from an out-group member. To explain this social behavior, the literature offers two alternative theoretical options that we label as the "threat hypothesis" and the "shared-experience hypothesis". The former relies on studies showing that out-group members create a sense of alertness. Consequently, potentially threatening out-group members are represented as spatially close allowing the prevention of costly errors. The latter hypothesis suggests that the observation of out-group members reduces the sharing of somatosensory experiences and, thus, increases the perceived physical distance between oneself and others. In the present paper, we pitted the two hypotheses against each other. In Experiment 1, Caucasian participants expressed multiple implicit "Near/Far" spatial categorization judgments from a Black-African Avatar and a White-Caucasian Avatar located in a 3D environment. Results indicate that the Black-African Avatar was categorized as closer to oneself, as compared with the White-Caucasian Avatar, providing support for "the threat hypothesis". In Experiment 2, we tested to which degree perceived threat contributes to this categorization bias by manipulating the avatar's perceived threat orthogonally to group membership. The results indicate that irrespective of group membership, threatening avatars were categorized as being closer to oneself as compared with no threatening avatars. This suggests that provided information about a person and not the mere group membership influences perceived distance to the person.

Learning to synchronize: How biological agents can couple neural task modules for dealing with the stability-plasticity dilemma.

We provide a novel computational framework on how biological and artificial agents can learn to flexibly couple and decouple neural task modules for cognitive processing. In this way, they can address the stability-plasticity dilemma. For this purpose, we combine two prominent computational neuroscience principles, namely Binding by Synchrony and Reinforcement Learning. The model learns to synchronize task-relevant modules, while also learning to desynchronize currently task-irrelevant modules. As a result, old (but currently task-irrelevant) information is protected from overwriting (stability) while new information can be learned quickly in currently task-relevant modules (plasticity). We combine learning to synchronize with task modules that learn via one of several classical learning algorithms (Rescorla-Wagner, backpropagation, Boltzmann machines). The resulting combined model is tested on a reversal learning paradigm where it must learn to switch between three different task rules. We demonstrate that our combined model has significant computational advantages over the original network without synchrony, in terms of both stability and plasticity. Importantly, the resulting models' processing dynamics are also consistent with empirical data and provide empirically testable hypotheses for future MEG/EEG studies.