Flexible control as surrogate reward or dynamic reward maximization.

The utility of a given experience, like interacting with a particular friend or tasting a particular food, fluctuates continually according to homeostatic and hedonic principles. Consequently, to maximize reward, an individual must be able to escape or attain outcomes as preferences change, by switching between actions. Recent work on human and artificial intelligence has defined such flexible instrumental control in information theoretic terms and postulated that it may serve as a reward surrogate. Another possibility, however, is that the adaptability afforded by flexible control is tacitly implemented by planning for dynamic changes in outcome values. In the current study, an expected utility model that computes decision values over a range of possible monetary gains and losses associated with sensory outcomes provided the best fit to behavioral choice data and performed best in terms of earned rewards. Moreover, consistent with previous work on perceived control and personality, individual differences in dimensional schizotypy were correlated with behavioral choice preferences in conditions with the greatest and lowest levels of flexible control. These results contribute to a growing literature on the role of instrumental control in goal-directed choice.

Analytic Causal Knowledge for Constructing Useable Empirical Causal Knowledge: Two Experiments on Pre-schoolers.

The present paper examines a type of abstract domain-general knowledge required for the process of constructing useable domain-specific causal knowledge, the evident goal of causal learning. It tests the hypothesis that analytic knowledge of causal-invariance decomposition functions is essential for this process. Such knowledge specifies the decomposition of an observed outcome into contributions from constituent causes under the default assumption that the empirical knowledge acquired is invariant across contextual/background causes. The paper reports two psychological experiments (and replication studies) with pre-school-age children on generalization across contexts involving binary cause and effect variables. The critical role of causal invariance for constructing useable causal knowledge predicts that even young children should (tacitly) use the causal-invariance decomposition function for such variables rather than a non-causal-invariance decomposition function common in statistical practice in research involving binary outcomes. The findings support the rational shaping of empirical causal knowledge by the causal-invariance constraint, ruling out alternative explanations in terms of non-causal-invariance decomposition functions, heuristics, and biases. For the same causal structure involving candidate causes and outcomes that are binary variables with a "present" value and an "absent" value, the paper argues against the possibility of multiple rational characterizations of the "sameness of causal influence" that justifies generalization across contexts.

Stakes and expertise modulate conformity in economic choice.

The influences of expertise and group size on an individual's tendency to align with a majority opinion have been attributed to informational and normative conformity, respectively: Whereas the former refers to the treatment of others' decisions as proxies for outcomes, the latter involves positive affect elicited by group membership. In this study, using a social gambling task, we pitted alignment with a high- vs. low-expertise majority against a hypothetical monetary reward, thus relating conformity to a broader literature on valuation and choice, and probed the countering influence of a high-expertise minority opinion. We found that the expertise of a countering minority group significantly modulated alignment with a low-expertise majority, but only if such alignment did not come at a cost. Conversely, participants' knowledge of payoff probabilities predicted the degree of majority alignment only when a high-expertise majority endorsed a more costly option. Implications for the relative influences of expertise and stakes on conformity are discussed.

The Rostrolateral Prefrontal Cortex Mediates a Preference for High-Agency Environments.

The ability to exert flexible instrumental control over one's environment is a defining feature of adaptive decision-making. Here, we investigated neural substrates mediating a preference for environments with greater instrumental divergence, the distance between outcome probability distributions associated with alternative actions. A formal index of agency, instrumental divergence allows an organism to flexibly obtain the currently most desired outcome as preferences change. As such, it may have intrinsic utility, guiding decisions toward environments that maximize instrumental power. Consistent with this notion, we found that a measure of expected value that treats instrumental divergence as a reward surrogate provided a better account of male and female human participants' choice preferences than did a conventional model, sensitive only to monetary reward. Using model-based fMRI, we found that activity in the rostrolateral and ventromedial PFC, regions associated with abstract cognitive inferences and subjective value computations, respectively, scaled with the divergence-based account of expected value. Implications for a neural common currency of information theoretic and motivational variables are discussed.SIGNIFICANCE STATEMENT Agency is a central concept in philosophy and psychology. While research thus far has focused on cognitive and perceptual measures of agency, recent work demonstrating a strong preference for high-agency environments indicates a salient motivational dimension. Here, using instrumental divergence, the distance between outcome distributions associated with alternative actions, as a formal index of agency, we found that brain regions associated with directed exploration and subjective value computations, respectively, were selectively modulated by a model that treated agency as a reward surrogate, over models that assigned utility only to monetary payoffs. In a subset of regions, such effects were predicted by the influence of instrumental divergence on economic choice preferences. Our results elucidate neural mechanisms mediating the utility of agency.

Neural Correlates of Causal Confounding.

As scientists, we are keenly aware that if putative causes perfectly covary, the independent influence of neither can be discerned-a "no confounding" constraint on inference, fundamental to philosophical and statistical perspectives on causation. Intriguingly, a substantial behavioral literature suggests that naïve human reasoners, adults and children, are tacitly sensitive to causal confounding. Here, a combination of fMRI and computational cognitive modeling was used to investigate neural substrates mediating such sensitivity. While being scanned, participants observed and judged the influences of various putative causes with confounded or nonconfounded, deterministic or stochastic, influences. During judgments requiring generalization of causal knowledge from a feedback-based learning context to a transfer probe, activity in the dorsomedial pFC was better accounted for by a Bayesian causal model, sensitive to both confounding and stochasticity, than a purely error-driven algorithm, sensitive only to stochasticity. Implications for the detection and estimation of distinct forms of uncertainty, and for a neural mediation of domain-general constraints on causal induction, are discussed.

Mesolimbic dopamine projections mediate cue-motivated reward seeking but not reward retrieval in rats.

Efficient foraging requires an ability to coordinate discrete reward-seeking and reward-retrieval behaviors. We used pathway-specific chemogenetic inhibition to investigate how rats' mesolimbic and mesocortical dopamine circuits contribute to the expression and modulation of reward seeking and retrieval. Inhibiting ventral tegmental area dopamine neurons disrupted the tendency for reward-paired cues to motivate reward seeking, but spared their ability to increase attempts to retrieve reward. Similar effects were produced by inhibiting dopamine inputs to nucleus accumbens, but not medial prefrontal cortex. Inhibiting dopamine neurons spared the suppressive effect of reward devaluation on reward seeking, an assay of goal-directed behavior. Attempts to retrieve reward persisted after devaluation, indicating they were habitually performed as part of a fixed action sequence. Our findings show that complete bouts of reward seeking and retrieval are behaviorally and neurally dissociable from bouts of reward seeking without retrieval. This dichotomy may prove useful for uncovering mechanisms of maladaptive behavior.

The Expression and Transfer of Valence Associated with Social Conformity.

Consensus seeking - abandoning one's own judgment to align with a group majority - is a fundamental feature of human social interaction. Notably, such striving for majority affiliation often occurs in the absence of any apparent economic or social gain, suggesting that achieving consensus might have intrinsic value. Here, using a simple gambling task, in which the decisions of ostensible previous gamblers were indicated below available options on each trial, we examined the affective properties of agreeing with a group majority by assessing the trade-off between social and non-social currencies, and the transfer of social valence to concomitant stimuli. In spite of demonstrating near-perfect knowledge of objective reward probabilities, participant's choices and evaluative judgments reflected a reliable preference for conformity, consistent with the hypothesized value of social alignment.

Instrumental Divergence and the Value of Control.

A critical aspect of flexible choice is that alternative actions yield distinct consequences: Only when available action alternatives produce distinct outcome states does discrimination and selection between actions allow an agent to flexibly obtain the currently most desired outcome. Here, we use instrumental divergence - the degree to which alternative actions differ with respect to their outcome probability distributions - as an index of flexible instrumental control, and assess the influence of this novel decision variable on choice preference. In Experiment 1, when other decision variables, such as expected value and outcome entropy, were held constant, we found a significant preference for high instrumental divergence. In Experiment 2, we used an "auto- vs. self-play" manipulation to eliminate outcome diversity as a source of behavioral preferences, and to contrast flexible instrumental control with the complete absence of voluntary choice. Our results suggest that flexible instrumental control over decision outcomes may have intrinsic value.

How multiple causes combine: independence constraints on causal inference.

According to the causal power view, two core constraints-that causes occur independently (i.e., no confounding) and influence their effects independently-serve as boundary conditions for causal induction. This study investigated how violations of these constraints modulate uncertainty about the existence and strength of a causal relationship. Participants were presented with pairs of candidate causes that were either confounded or not, and that either interacted or exerted their influences independently. Consistent with the causal power view, uncertainty about the existence and strength of causal relationships was greater when causes were confounded or interacted than when unconfounded and acting independently. An elemental Bayesian causal model captured differences in uncertainty due to confounding but not those due to an interaction. Implications of distinct sources of uncertainty for the selection of contingency information and causal generalization are discussed.

Differentiating neural systems mediating the acquisition vs. expression of goal-directed and habitual behavioral control.

Considerable behavioral data indicate that operant actions can become habitual, as demonstrated by insensitivity to changes in the action-outcome contingency and in subjective outcome values. Notably, although several studies have investigated the neural substrates of habits, none has clearly differentiated the areas of the human brain that support habit formation from those that implement habitual control. We scanned participants with functional magnetic resonance imaging as they learned and performed an operant task in which the conditional structure of the environment encouraged either goal-directed encoding of the consequences of actions, or a habit-like mapping of actions to antecedent cues. Participants were also scanned during a subsequent assessment of insensitivity to outcome devaluation. We identified dissociable roles of the cerebellum and ventral striatum, across learning and test performance, in behavioral insensitivity to outcome devaluation. We also showed that the inferior parietal lobule (an area previously implicated in several aspects of goal-directed action selection, including the attribution of intent and awareness of agency) predicted sensitivity to outcome devaluation. Finally, we revealed a potential functional homology between the human subgenual cortex and rodent infralimbic cortex in the implementation of habitual control. In summary, our findings suggested a broad systems division, at the cortical and subcortical levels, between brain areas mediating the encoding and expression of action-outcome and stimulus-response associations.

Characterizing the associative content of brain structures involved in habitual and goal-directed actions in humans: a multivariate FMRI study.

While there is accumulating evidence for the existence of distinct neural systems supporting goal-directed and habitual action selection in the mammalian brain, much less is known about the nature of the information being processed in these different brain regions. Associative learning theory predicts that brain systems involved in habitual control, such as the dorsolateral striatum, should contain stimulus and response information only, but not outcome information, while regions involved in goal-directed action, such as ventromedial and dorsolateral prefrontal cortex and dorsomedial striatum, should be involved in processing information about outcomes as well as stimuli and responses. To test this prediction, human participants underwent fMRI while engaging in a binary choice task designed to enable the separate identification of these different representations with a multivariate classification analysis approach. Consistent with our predictions, the dorsolateral striatum contained information about responses but not outcomes at the time of an initial stimulus, while the regions implicated in goal-directed action selection contained information about both responses and outcomes. These findings suggest that differential contributions of these regions to habitual and goal-directed behavioral control may depend in part on basic differences in the type of information that these regions have access to at the time of decision making.

Anterior insula activity reflects the effects of intentionality on the anticipation of aversive stimulation.

If someone causes you harm, your affective reaction to that person might be profoundly influenced by your inferences about the intentionality of their actions. In the present study, we aimed to understand how affective responses to a biologically salient aversive outcome administered by others are modulated by the extent to which a given individual is judged to have deliberately or inadvertently delivered the outcome. Using fMRI, we examined how neural responses to anticipation and receipt of an aversive stimulus are modulated by this fundamental social judgment. We found that affective evaluations about an individual whose actions led to either noxious or neutral consequences for the subject did indeed depend on the perceived intentions of that individual. At the neural level, activity in the anterior insula correlated with the interaction between perceived intentionality and anticipated outcome valence, suggesting that this region reflects the influence of mental state attribution on aversive expectations.

Neural correlates of the divergence of instrumental probability distributions.

Flexible action selection requires knowledge about how alternative actions impact the environment: a "cognitive map" of instrumental contingencies. Reinforcement learning theories formalize this map as a set of stochastic relationships between actions and states, such that for any given action considered in a current state, a probability distribution is specified over possible outcome states. Here, we show that activity in the human inferior parietal lobule correlates with the divergence of such outcome distributions-a measure that reflects whether discrimination between alternative actions increases the controllability of the future-and, further, that this effect is dissociable from those of other information theoretic and motivational variables, such as outcome entropy, action values, and outcome utilities. Our results suggest that, although ultimately combined with reward estimates to generate action values, outcome probability distributions associated with alternative actions may be contrasted independently of valence computations, to narrow the scope of the action selection problem.

Contributions of the striatum to learning, motivation, and performance: an associative account.

It has long been recognized that the striatum is composed of distinct functional sub-units that are part of multiple cortico-striatal-thalamic circuits. Contemporary research has focused on the contribution of striatal sub-regions to three main phenomena: learning of associations between stimuli, actions and rewards; selection between competing response alternatives; and motivational modulation of motor behavior. Recent proposals have argued for a functional division of the striatum along these lines, attributing, for example, learning to one region and performance to another. Here, we consider empirical data from human and animal studies, as well as theoretical notions from both the psychological and computational literatures, and conclude that striatal sub-regions instead differ most clearly in terms of the associations being encoded in each region.

Dissociable brain systems mediate vicarious learning of stimulus-response and action-outcome contingencies.

Two distinct strategies have been suggested to support action selection in humans and other animals on the basis of experiential learning: a goal-directed strategy that generates decisions based on the value and causal antecedents of action outcomes, and a habitual strategy that relies on the automatic elicitation of actions by environmental stimuli. In the present study, we investigated whether a similar dichotomy exists for actions that are acquired vicariously, through observation of other individuals rather than through direct experience, and assessed whether these strategies are mediated by distinct brain regions. We scanned participants with functional magnetic resonance imaging while they performed an observational learning task designed to encourage either goal-directed encoding of the consequences of observed actions, or a mapping of observed actions to conditional discriminative cues. Activity in different parts of the action observation network discriminated between the two conditions during observational learning and correlated with the degree of insensitivity to outcome devaluation in subsequent performance. Our findings suggest that, in striking parallel to experiential learning, neural systems mediating the observational acquisition of actions may be dissociated into distinct components: a goal-directed, outcome-sensitive component and a less flexible stimulus-response component.

Neural correlates of specific and general Pavlovian-to-Instrumental Transfer within human amygdalar subregions: a high-resolution fMRI study.

It is widely held that the interaction between instrumental and Pavlovian conditioning induces powerful motivational biases. Pavlovian-Instrumental Transfer (PIT) is one of the key paradigms demonstrating this effect, which can further be decomposed into a general and specific component. Although these two forms of PIT have been studied at the level of amygdalar subregions in rodents, it is still unknown whether they involve different areas of the human amygdala. Using a high-resolution fMRI (hr-fMRI) protocol optimized for the amygdala in combination with a novel free operant task designed to elicit effects of both general and specific PIT, we demonstrate that a region of ventral amygdala within the boundaries of the basolateral complex and the ventrolateral putamen are involved in specific PIT, while a region of dorsal amygdala within the boundaries of the centromedial complex is involved in general PIT. These results add to a burgeoning literature indicating different functional contributions for these different amygdalar subregions in reward-processing and motivation.

Anything you can do, you can do better: neural substrates of incentive-based performance enhancement.

Performance-based pay schemes in many organizations share the fundamental assumption that the performance level for a given task will increase as a function of the amount of incentive provided. Consistent with this notion, psychological studies have demonstrated that expectations of reward can improve performance on a plethora of different cognitive and physical tasks, ranging from problem solving to the voluntary regulation of heart rate. However, much less is understood about the neural mechanisms of incentivized performance enhancement. In particular, it is still an open question how brain areas that encode expectations about reward are able to translate incentives into improved performance across fundamentally different cognitive and physical task requirements.

Neural correlates of instrumental contingency learning: differential effects of action-reward conjunction and disjunction.

Contingency theories of goal-directed action propose that experienced disjunctions between an action and its specific consequences, as well as conjunctions between these events, contribute to encoding the action-outcome association. Although considerable behavioral research in rats and humans has provided evidence for this proposal, relatively little is known about the neural processes that contribute to the two components of the contingency calculation. Specifically, while recent findings suggest that the influence of action-outcome conjunctions on goal-directed learning is mediated by a circuit involving ventromedial prefrontal, medial orbitofrontal cortex, and dorsomedial striatum, the neural processes that mediate the influence of experienced disjunctions between these events are unknown. Here we show differential responses to probabilities of conjunctive and disjunctive reward deliveries in the ventromedial prefrontal cortex, the dorsomedial striatum, and the inferior frontal gyrus. Importantly, activity in the inferior parietal lobule and the left middle frontal gyrus varied with a formal integration of the two reward probabilities, ΔP, as did response rates and explicit judgments of the causal efficacy of the action.

Extracting functional equivalence from reversing contingencies.

In 2 experiments assessing acquired equivalence, human participants were initially presented with 4 cues, 2 of which were paired with 1 outcome and 2 of which were paired with a 2nd outcome. These contingencies were then reversed across several training blocks such that, although each cue was paired equally often with each of the two outcomes across blocks, cues A and B always signaled the same outcome within blocks (as did cues C and D). In both experiments, performance on a subsequent transfer discrimination was enhanced when participants were required to generalize between stimuli that had been paired with the same outcome within each block of training. Additional tests did not yield evidence of a bias toward a specific set of cue-outcome contingencies in either experiment. Moreover, interviews conducted at the end of Experiment 2 revealed that performance on the transfer discrimination was enhanced only in participants who discovered the equivalence relationships during initial training. The results challenge simple associative, and attentional, accounts of acquired equivalence and favor the view that this effect is mediated by comparisons of the similarity of adjacent cue-outcome structures.