Harnessing the flexibility of neural networks to predict dynamic theoretical parameters underlying human choice behavior.

Reinforcement learning (RL) models are used extensively to study human behavior. These rely on normative models of behavior and stress interpretability over predictive capabilities. More recently, neural network models have emerged as a descriptive modeling paradigm that is capable of high predictive power yet with limited interpretability. Here, we seek to augment the expressiveness of theoretical RL models with the high flexibility and predictive power of neural networks. We introduce a novel framework, which we term theoretical-RNN (t-RNN), whereby a recurrent neural network is trained to predict trial-by-trial behavior and to infer theoretical RL parameters using artificial data of RL agents performing a two-armed bandit task. In three studies, we then examined the use of our approach to dynamically predict unseen behavior along with time-varying theoretical RL parameters. We first validate our approach using synthetic data with known RL parameters. Next, as a proof-of-concept, we applied our framework to two independent datasets of humans performing the same task. In the first dataset, we describe differences in theoretical RL parameters dynamic among clinical psychiatric vs. healthy controls. In the second dataset, we show that the exploration strategies of humans varied dynamically in response to task phase and difficulty. For all analyses, we found better performance in the prediction of actions for t-RNN compared to the stationary maximum-likelihood RL method. We discuss the use of neural networks to facilitate the estimation of latent RL parameters underlying choice behavior.

Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder.

Attention-deficit/hyperactivity disorder (ADHD) is characterized by difficulty in acting in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first-stage actions (i.e., initiatory actions), while for actions performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.

Disentangling the contribution of individual and social learning processes in human advice-taking behavior.

The study of social learning examines how individuals learn from others by means of observation, imitation, or compliance with advice. However, it still remains largely unknown whether social learning processes have a distinct contribution to behavior, independent from non-social trial-and-error learning that often occurs simultaneously. 153 participants completed a reinforcement learning task, where they were asked to make choices to gain rewards. Advice from an artificial teacher was presented in 60% of the trials, allowing us to compare choice behavior with and without advice. Results showed a strong and reliable tendency to follow advice (test-retest reliability ~0.73). Computational modeling suggested a unique contribution of three distinct learning strategies: (a) individual learning (i.e., learning the value of actions, independent of advice), (b) informed advice-taking (i.e., learning the value of following advice), and (c) non-informed advice-taking (i.e., a constant bias to follow advice regardless of outcome history). Comparing artificial and empirical data provided specific behavioral regression signatures to both informed and non-informed advice taking processes. We discuss the theoretical implications of integrating internal and external information during the learning process.

Fruit bats adjust their decision-making process according to environmental dynamics.

One of the main functions of behavioral plasticity lies in the ability to contend with dynamic environments. Indeed, while numerous studies have shown that animals adapt their behavior to the environment, how they adapt their latent learning and decision strategies to changes in the environment is less understood. Here, we used a controlled experiment to examine the bats' ability to adjust their decision strategy according to the environmental dynamics. Twenty-five Egyptian fruit bats were placed individually in either a stable or a volatile environment for four consecutive nights. In the stable environment, two feeders offered food, each with a different reward probability (0.2 vs. 0.8) that remained fixed over two nights and were then switched, while in the volatile environment, the positions of the more and the less rewarding feeders were changed every hour. We then fit two alternative commonly used models namely, reinforcement learning and win-stay-lose-shift strategies to the bats' behavior. We found that while the bats adapted their decision-making strategy to the environmental dynamics, they seemed to be limited in their responses based on natural priors. Namely, when the environment had changed slowly, at a rate that is natural for these bats, they seemed to rely on reinforcement learning and their performance was nearly optimal, but when the experimental environment changed much faster than in the natural environment, the bats stopped learning and switched to a random decision-making strategy. Together, these findings exemplify both the bats' decision-making plasticity as well as its natural limitations.

Computational mechanisms underlying latent value updating of unchosen actions.

Current studies suggest that individuals estimate the value of their choices based on observed feedback. Here, we ask whether individuals also update the value of their unchosen actions, even when the associated feedback remains unknown. One hundred seventy-eight individuals completed a multi-armed bandit task, making choices to gain rewards. We found robust evidence suggesting latent value updating of unchosen actions based on the chosen action's outcome. Computational modeling results suggested that this effect is mainly explained by a value updating mechanism whereby individuals integrate the outcome history for choosing an option with that of rejecting the alternative. Properties of the deliberation (i.e., duration/difficulty) did not moderate the latent value updating of unchosen actions, suggesting that memory traces generated during deliberation might take a smaller role in this specific phenomenon than previously thought. We discuss the mechanisms facilitating credit assignment to unchosen actions and their implications for human decision-making.

Distinct reinforcement learning profiles distinguish between language and attentional neurodevelopmental disorders.

BACKGROUND: Theoretical models posit abnormalities in cortico-striatal pathways in two of the most common neurodevelopmental disorders (Developmental dyslexia, DD, and Attention deficit hyperactive disorder, ADHD), but it is still unclear what distinct cortico-striatal dysfunction might distinguish language disorders from others that exhibit very different symptomatology. Although impairments in tasks that depend on the cortico-striatal network, including reinforcement learning (RL), have been implicated in both disorders, there has been little attempt to dissociate between different types of RL or to compare learning processes in these two types of disorders. The present study builds upon prior research indicating the existence of two learning manifestations of RL and evaluates whether these processes can be differentiated in language and attention deficit disorders. We used a two-step RL task shown to dissociate model-based from model-free learning in human learners. RESULTS: Our results show that, relative to neurotypicals, DD individuals showed an impairment in model-free but not in model-based learning, whereas in ADHD the ability to use both model-free and model-based learning strategies was significantly compromised. CONCLUSIONS: Thus, learning impairments in DD may be linked to a selective deficit in the ability to form action-outcome associations based on previous history, whereas in ADHD some learning deficits may be related to an incapacity to pursue rewards based on the tasks' structure. Our results indicate how different patterns of learning deficits may underlie different disorders, and how computation-minded experimental approaches can differentiate between them.

Formation of non-veridical action-outcome associations following exposure to threat-related cues.

Acting in a goal-directed manner requires an ability to accurately predict the outcomes of one's actions. However, not much is known regarding how threat-related cues influence our ability to form action-outcome associations according to the environment's known causal structure. Here, we examined the extent to which threat-related cues influence individuals' tendency to form and act in accordance with action-outcome associations that do not exist in the external environment (i.e., outcome-irrelevant learning). Forty-nine healthy participants completed an online multi-armed reinforcement-learning bandit task in which they were asked to help a child safely cross a street. Outcome-irrelevant learning was estimated as a tendency to assign value to response keys that did not predict an outcome but were used to report participants' choices. We first replicated previous findings showing that individuals tend to form and act in accordance with irrelevant action-outcome associations, across experimental conditions, and despite explicit knowledge regarding the environment's true structure. Importantly, results of a Bayesian regression analysis suggest that the presentation of threat-related images, compared with neutral or no visual cues at trial initiation, increased outcome-irrelevant learning. We discuss outcome-irrelevant learning as a possible theoretical mechanism that may lead to altered learning in the face of perceived threat. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Working memory capacity estimates moderate value learning for outcome-irrelevant features.

To establish accurate action-outcome associations in the environment, individuals must refrain from assigning value to outcome-irrelevant features. However, studies have largely ignored the role of attentional control processes on action value updating. In the current study, we examined the extent to which working memory-a system that can filter and block the processing of irrelevant information in one's mind-also filters outcome-irrelevant information during value-based learning. For this aim, 174 individuals completed a well-established working memory capacity measurement and a reinforcement learning task designed to estimate outcome-irrelevant learning. We replicated previous studies showing a group-level tendency to assign value to tasks' response keys, despite clear instructions and practice suggesting they are irrelevant to the prediction of monetary outcomes. Importantly, individuals with higher working memory capacity were less likely to assign value to the outcome-irrelevant response keys, thus suggesting a significant moderation effect of working memory capacity on outcome-irrelevant learning. We discuss the role of working memory processing on value-based learning through the lens of a cognitive control failure.

Assigning the right credit to the wrong action: compulsivity in the general population is associated with augmented outcome-irrelevant value-based learning.

Compulsive behavior is enacted under a belief that a specific act controls the likelihood of an undesired future event. Compulsive behaviors are widespread in the general population despite having no causal relationship with events they aspire to influence. In the current study, we tested whether there is an increased tendency to assign value to aspects of a task that do not predict an outcome (i.e., outcome-irrelevant learning) among individuals with compulsive tendencies. We studied 514 healthy individuals who completed self-report compulsivity, anxiety, depression, and schizotypal measurements, and a well-established reinforcement-learning task (i.e., the two-step task). As expected, we found a positive relationship between compulsivity and outcome-irrelevant learning. Specifically, individuals who reported having stronger compulsive tendencies (e.g., washing, checking, grooming) also tended to assign value to response keys and stimuli locations that did not predict an outcome. Controlling for overall goal-directed abilities and the co-occurrence of anxious, depressive, or schizotypal tendencies did not impact these associations. These findings indicate that outcome-irrelevant learning processes may contribute to the expression of compulsivity in a general population setting. We highlight the need for future research on the formation of non-veridical action-outcome associations as a factor related to the occurrence and maintenance of compulsive behavior.

Compulsivity is linked to reduced adolescent development of goal-directed control and frontostriatal functional connectivity.

A characteristic of adaptive behavior is its goal-directed nature. An ability to act in a goal-directed manner is progressively refined during development, but this refinement can be impacted by the emergence of psychiatric disorders. Disorders of compulsivity have been framed computationally as a deficit in model-based control, and have been linked also to abnormal frontostriatal connectivity. However, the developmental trajectory of model-based control, including an interplay between its maturation and an emergence of compulsivity, has not been characterized. Availing of a large sample of healthy adolescents (n = 569) aged 14 to 24 y, we show behaviorally that over the course of adolescence there is a within-person increase in model-based control, and this is more pronounced in younger participants. Using a bivariate latent change score model, we provide evidence that the presence of higher compulsivity traits is associated with an atypical profile of this developmental maturation in model-based control. Resting-state fMRI data from a subset of the behaviorally assessed subjects (n = 230) revealed that compulsivity is associated with a less pronounced change of within-subject developmental remodeling of functional connectivity, specifically between the striatum and a frontoparietal network. Thus, in an otherwise clinically healthy population sample, in early development, individual differences in compulsivity are linked to the developmental trajectory of model-based control and a remodeling of frontostriatal connectivity.

Effects of neurofeedback and working memory-combined training on executive functions in healthy young adults.

Given the interest in improving executive functions, the present study examines a promising combination of two training techniques: neurofeedback training (NFT) and working memory training (WMT). NFT targeted increasing the amplitude of individual's upper Alpha frequency band at the parietal midline scalp location (Pz), and WMT consisted of an established computerized protocol with working memory updating and set-shifting components. Healthy participants (n = 140) were randomly allocated to five combinations of training, including visual search training used as an active control training for the WMT; all five groups were compared to a sixth silent control group receiving no training. All groups were evaluated before and after training for resting-state electroencephalogram (EEG) and behavioral executive function measures. The participants in the silent control group were unaware of this procedure, and received one of the training protocols only after study has ended. Results demonstrated significant improvement in the practice tasks in all training groups including non-specific influence of NFT on resting-state EEG spectral topography. There was only a near transfer effect (improvement in working memory task) for WMT, which remained significant in the delayed post-test (after 1 month), in comparison to silent control group but not in comparison to active control training group. The NFT + WMT combined group showed improved mental rotation ability both in the post-training and in the follow-up evaluations. This improvement, however, did not differ significantly from that in the silent control group. We conclude that the current training protocols, including their combination, have very limited influence on the executive functions that were assessed in this study.

Credit assignment to state-independent task representations and its relationship with model-based decision making.

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment's structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.

Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling.

A well-established notion in cognitive neuroscience proposes that multiple brain systems contribute to choice behaviour. These include: (1) a model-free system that uses values cached from the outcome history of alternative actions, and (2) a model-based system that considers action outcomes and the transition structure of the environment. The widespread use of this distinction, across a range of applications, renders it important to index their distinct influences with high reliability. Here we consider the two-stage task, widely considered as a gold standard measure for the contribution of model-based and model-free systems to human choice. We tested the internal/temporal stability of measures from this task, including those estimated via an established computational model, as well as an extended model using drift-diffusion. Drift-diffusion modeling suggested that both choice in the first stage, and RTs in the second stage, are directly affected by a model-based/free trade-off parameter. Both parameter recovery and the stability of model-based estimates were poor but improved substantially when both choice and RT were used (compared to choice only), and when more trials (than conventionally used in research practice) were included in our analysis. The findings have implications for interpretation of past and future studies based on the use of the two-stage task, as well as for characterising the contribution of model-based processes to choice behaviour.

Can we learn to learn? The influence of procedural working-memory training on rapid instructed-task-learning.

Humans have the unique ability to efficiently execute instructions that were never practiced beforehand. In this Rapid Instructed-Task-Learning, not-yet-executed novel rules are presumably held in procedural working-memory (WM), which is assumed to hold stimulus-to-response bindings. In this study, we employed a computerized-cognitive training protocol targeting procedural WM to test this assumption and to examine whether the ability to rapidly learn novel rules can itself be learned. 175 participants were randomly assigned to one of three groups: procedural WM training (involving task-switching and N-back elements, all with novel rules; Shahar and Meiran in PLoS One 10(3):e0119992, 2015), active-control training (adaptive visual-search task), and no-contact control. We examined participants' rapid instructed-task-learning abilities before and after training, by administrating 55 novel choice tasks, and measuring their performance in the first two trials (where participants had no practice). While all participants showed shorter reaction-times in post vs. pretest, only participants in the procedural WM training group did not demonstrate an increased error rate at posttest. Evidence accumulation modelling suggested that this result stems from a reduction in decision threshold (the amount of evidence that needs to be gathered to reach a decision), which was more pronounced in the control groups; possibly accompanied by an increased drift-rate (the rate of evidence accumulation) only for the training group. Implication are discussed.

Formation of abstract task representations: Exploring dosage and mechanisms of working memory training effects.

Working memory is strongly involved in human reasoning, abstract thinking and decision making. Past studies have shown that working memory training generalizes to untrained working memory tasks with similar structure (near-transfer effect). Here, we focused on two questions: First, we ask how much training might be required in order to find a reliable near-transfer effect? Second, we ask which choice- mechanism might underlie training benefits? Participants were allocated to one of three groups: working-memory training (combined set-shifting and N-back task), active-control (visual search) and no-contact control. During pre/post testing, all participants completed tests tapping procedural and declarative working memory as well as reasoning. We found improved performance only in the procedural working-memory transfer tasks, a transfer task that shared a similar structure to that of the training task. Intermediate testing throughout the training period suggest that this effect emerged as soon as after 2 training sessions. We applied evidence accumulation modeling to investigate the choice process responsible for this near-transfer effect and found that trained participants, compared with active-controls had quicker retrieval of the action rules, and more efficient classification of the target. We conclude that participants were able to form abstract representations of the task procedure (i.e., stimulus-response rules) that was then ~applied to novel stimuli and responses.

Computation in Psychotherapy, or How Computational Psychiatry Can Aid Learning-Based Psychological Therapies.

Learning-based therapies, such as cognitive-behavioral therapy, are used worldwide, and their efficacy is endorsed by health and research funding agencies. However, the mechanisms behind both their strengths and their weaknesses are inadequately understood. Here we describe how advances in computational modeling may help formalize and test hypotheses regarding how patients make inferences, which are core postulates of these therapies. Specifically, we highlight the relevance of computations with regard to the development, maintenance, and therapeutic change in psychiatric disorders. A Bayesian approach helps delineate which apparent inferential biases and aberrant beliefs are in fact near-normative, given patients' current concerns, and which are not. As examples, we formalize three hypotheses. First, high-level dysfunctional beliefs should be treated as beliefs over models of the world. There is a need to test how, and whether, people apply these high-level beliefs to guide the formation of lower level beliefs important for real-life decision making, conditional on their experiences. Second, during the genesis of a disorder, maladaptive beliefs grow because more benign alternative schemas are discounted during belief updating. Third, we propose that when patients learn within therapy but fail to benefit in real life, this can be accounted for by a mechanism that we term overaccommodation, similar to that used to explain fear reinstatement. Beyond these specifics, an ambitious collaborative research program between computational psychiatry researchers, therapists, and experts-by-experience needs to form testable predictions out of factors claimed to be important for therapy.

Examining procedural working memory processing in obsessive-compulsive disorder.

Previous research has suggested that a deficit in working memory might underlie the difficulty of obsessive-compulsive disorder (OCD) patients to control their thoughts and actions. However, a recent meta-analyses found only small effect sizes for working memory deficits in OCD. Recently, a distinction has been made between declarative and procedural working memory. Working memory in OCD was tested mostly using declarative measurements. However, OCD symptoms typically concerns actions, making procedural working-memory more relevant. Here, we tested the operation of procedural working memory in OCD. Participants with OCD and healthy controls performed a battery of choice reaction tasks under high and low procedural working memory demands. Reaction-times (RT) were estimated using ex-Gaussian distribution fitting, revealing no group differences in the size of the RT distribution tail (i.e., τ parameter), known to be sensitive to procedural working memory manipulations. Group differences, unrelated to working memory manipulations, were found in the leading-edge of the RT distribution and analyzed using a two-stage evidence accumulation model. Modeling results suggested that perceptual difficulties might underlie the current group differences. In conclusion, our results suggest that procedural working-memory processing is most likely intact in OCD, and raise a novel, yet untested assumption regarding perceptual deficits in OCD.

The role of working memory in rapid instructed task learning and intention-based reflexivity: An individual differences examination.

The ability to efficiently follow novel task instructions (Rapid Instructed Task Learning, RITL) appears late in evolution, is required for successful collaborative teamwork, and appears to involve maintaining instructions in working-memory (WM). RITL is indexed by the efficiency in which the instructions are performed (RITL success) and by whether the instructions operate automatically (intention-based reflexivity). Based on prior normative work employing WM-load manipulations, we predicted that individual differences in WM would positively correlate with these RITL indices. Participants (N=175) performed the NEXT paradigm, which is used to assess RITL, and tests of choice reaction time, intelligence, and WM. Confirmatory factor analyses showed that, contrary to our predictions, successful performance in WM tasks did not predict RITL performance. Tests tapping general-fluid intelligence and reaction time positively correlated with RITL success. However, contrary to our predictions, RITL success positively correlated with little intention-based reflexivity. We suggest that for a RITL paradigm to produce intention-based reflexivity, its WM demand must be low, and, thus, performance does not reflect individual differences in WM.

Increased ongoing neural variability in ADHD.

Attention Deficit Hyperactivity Disorder (ADHD) has been described as a disorder where frequent lapses of attention impair the ability of an individual to focus/attend in a sustained manner, thereby generating abnormally large intra-individual behavioral variability across trials. Indeed, increased reaction time (RT) variability is a fundamental behavioral characteristic of individuals with ADHD found across a large number of cognitive tasks. But what is the underlying neurophysiology that might generate such behavioral instability? Here, we examined trial-by-trial EEG response variability to visual and auditory stimuli while subjects' attention was diverted to an unrelated task at the fixation cross. Comparisons between adult ADHD and control participants revealed that neural response variability was significantly larger in the ADHD group as compared with the control group in both sensory modalities. Importantly, larger trial-by-trial variability in ADHD was apparent before and after stimulus presentation as well as in trials where the stimulus was omitted, suggesting that ongoing (rather than stimulus-evoked) neural activity is continuously more variable (noisier) in ADHD. While the patho-physiological mechanisms causing this increased neural variability remain unknown, they appear to act continuously rather than being tied to a specific sensory or cognitive process.