Distinct basal ganglia contributions to learning from implicit and explicit value signals in perceptual decision-making.

Metacognitive evaluations of confidence provide an estimate of decision accuracy that could guide learning in the absence of explicit feedback. We examine how humans might learn from this implicit feedback in direct comparison with that of explicit feedback, using simultaneous EEG-fMRI. Participants performed a motion direction discrimination task where stimulus difficulty was increased to maintain performance, with intermixed explicit- and no-feedback trials. We isolate single-trial estimates of post-decision confidence using EEG decoding, and find these neural signatures re-emerge at the time of feedback together with separable signatures of explicit feedback. We identified these signatures of implicit versus explicit feedback along a dorsal-ventral gradient in the striatum, a finding uniquely enabled by an EEG-fMRI fusion. These two signals appear to integrate into an aggregate representation in the external globus pallidus, which could broadcast updates to improve cortical decision processing via the thalamus and insular cortex, irrespective of the source of feedback.

Mild exogenous inflammation blunts neural signatures of bounded evidence accumulation and reward prediction error processing in healthy male participants.

BACKGROUND: Altered neural haemodynamic activity during decision making and learning has been linked to the effects of inflammation on mood and motivated behaviours. So far, it has been reported that blunted mesolimbic dopamine reward signals are associated with inflammation-induced anhedonia and apathy. Nonetheless, it is still unclear whether inflammation impacts neural activity underpinning decision dynamics. The process of decision making involves integration of noisy evidence from the environment until a critical threshold of evidence is reached. There is growing empirical evidence that such process, which is usually referred to as bounded accumulation of decision evidence, is affected in the context of mental illness. METHODS: In a randomised, placebo-controlled, crossover study, 19 healthy male participants were allocated to placebo and typhoid vaccination. Three to four hours post-injection, participants performed a probabilistic reversal-learning task during functional magnetic resonance imaging. To capture the hidden neurocognitive operations underpinning decision-making, we devised a hybrid sequential sampling and reinforcement learning computational model. We conducted whole brain analyses informed by the modelling results to investigate the effects of inflammation on the efficiency of decision dynamics and reward learning. RESULTS: We found that during the decision phase of the task, typhoid vaccination attenuated neural signatures of bounded evidence accumulation in the dorsomedial prefrontal cortex, only for decisions requiring short integration time. Consistent with prior work, we showed that, in the outcome phase, mild acute inflammation blunted the reward prediction error in the bilateral ventral striatum and amygdala. CONCLUSIONS: Our study extends current insights into the effects of inflammation on the neural mechanisms of decision making and shows that exogenous inflammation alters neural activity indexing efficiency of evidence integration, as a function of choice discriminability. Moreover, we replicate previous findings that inflammation blunts striatal reward prediction error signals.

Distinct spatiotemporal brainstem pathways of outcome valence during reward- and punishment-based learning.

Learning to seek rewards and avoid punishments, based on positive and negative choice outcomes, is essential for human survival. Yet, the neural underpinnings of outcome valence in the human brainstem and the extent to which they differ in reward and punishment learning contexts remain largely elusive. Here, using simultaneously acquired electroencephalography and functional magnetic resonance imaging data, we show that during reward learning the substantia nigra (SN)/ventral tegmental area (VTA) and locus coeruleus are initially activated following negative outcomes, while the VTA subsequently re-engages exhibiting greater responses for positive than negative outcomes, consistent with an early arousal/avoidance response and a later value-updating process, respectively. During punishment learning, we show that distinct raphe nucleus and SN subregions are activated only by negative outcomes with a sustained post-outcome activity across time, supporting the involvement of these brainstem subregions in avoidance behavior. Finally, we demonstrate that the coupling of these brainstem structures with other subcortical and cortical areas helps to shape participants' serial choice behavior in each context.

Functional Magnetic Resonance Imaging Signatures of Pavlovian and Instrumental Valuation Systems during a Modified Orthogonalized Go/No-go Task.

Motivational (i.e., Pavlovian) values interfere with instrumental responding and can lead to suboptimal decision-making. In humans, task-based neuroimaging studies have only recently started illuminating the functional neuroanatomy of Pavlovian biasing of instrumental control. To provide a mechanistic understanding of the neural dynamics underlying the Pavlovian and instrumental valuation systems, analysis of neuroimaging data has been informed by computational modeling of conditioned behavior. Nonetheless, because of collinearities in Pavlovian and instrumental predictions, previous research failed to tease out hemodynamic activity that is parametrically and dynamically modulated by coexistent Pavlovian and instrumental value expectations. Moreover, neural correlates of Pavlovian to instrumental transfer effects have so far only been identified in extinction (i.e., in the absence of learning). In this study, we devised a modified version of the orthogonalized go/no-go paradigm, which introduced Pavlovian-only catch trials to better disambiguate trial-by-trial Pavlovian and instrumental predictions in both sexes. We found that hemodynamic activity in the ventromedial pFC covaried uniquely with the model-derived Pavlovian value expectations. Notably, modulation of neural activity encoding for instrumental predictions in the supplementary motor cortex was linked to successful action selection in conflict conditions. Furthermore, hemodynamic activity in regions pertaining to the limbic system and medial pFC was correlated with synergistic Pavlovian and instrumental predictions and improved conditioned behavior during congruent trials. Altogether, our results provide new insights into the functional neuroanatomy of decision-making and corroborate the validity of our variant of the orthogonalized go/no-go task as a behavioral assay of the Pavlovian and instrumental valuation systems.

Secondary motor integration as a final arbiter in sensorimotor decision-making.

Sensorimotor decision-making is believed to involve a process of accumulating sensory evidence over time. While current theories posit a single accumulation process prior to planning an overt motor response, here, we propose an active role of motor processes in decision formation via a secondary leaky motor accumulation stage. The motor leak adapts the "memory" with which this secondary accumulator reintegrates the primary accumulated sensory evidence, thus adjusting the temporal smoothing in the motor evidence and, correspondingly, the lag between the primary and motor accumulators. We compare this framework against different single accumulator variants using formal model comparison, fitting choice, and response times in a task where human observers made categorical decisions about a noisy sequence of images, under different speed-accuracy trade-off instructions. We show that, rather than boundary adjustments (controlling the amount of evidence accumulated for decision commitment), adjustment of the leak in the secondary motor accumulator provides the better description of behavior across conditions. Importantly, we derive neural correlates of these 2 integration processes from electroencephalography data recorded during the same task and show that these neural correlates adhere to the neural response profiles predicted by the model. This framework thus provides a neurobiologically plausible description of sensorimotor decision-making that captures emerging evidence of the active role of motor processes in choice behavior.

A Common Neural Account for Social and Nonsocial Decisions.

To date, social and nonsocial decisions have been studied largely in isolation. Consequently, the extent to which social and nonsocial forms of decision uncertainty are integrated using shared neurocomputational resources remains elusive. Here, we address this question using simultaneous electroencephalography (EEG)-functional magnetic resonance imaging (fMRI) in healthy human participants (young adults of both sexes) and a task in which decision evidence in social and nonsocial contexts varies along comparable scales. First, we identify time-resolved build-up of activity in the EEG, akin to a process of evidence accumulation (EA), across both contexts. We then use the endogenous trial-by-trial variability in the slopes of these accumulating signals to construct parametric fMRI predictors. We show that a region of the posterior-medial frontal cortex (pMFC) uniquely explains trial-wise variability in the process of evidence accumulation in both social and nonsocial contexts. We further demonstrate a task-dependent coupling between the pMFC and regions of the human valuation system in dorso-medial and ventro-medial prefrontal cortex across both contexts. Finally, we report domain-specific representations in regions known to encode the early decision evidence for each context. These results are suggestive of a domain-general decision-making architecture, whereupon domain-specific information is likely converted into a "common currency" in medial prefrontal cortex and accumulated for the decision in the pMFC.SIGNIFICANCE STATEMENT Little work has directly compared social-versus-nonsocial decisions to investigate whether they share common neurocomputational origins. Here, using combined electroencephalography (EEG)-functional magnetic resonance imaging (fMRI) and computational modeling, we offer a detailed spatiotemporal account of the neural underpinnings of social and nonsocial decisions. Specifically, we identify a comparable mechanism of temporal evidence integration driving both decisions and localize this integration process in posterior-medial frontal cortex (pMFC). We further demonstrate task-dependent coupling between the pMFC and regions of the human valuation system across both contexts. Finally, we report domain-specific representations in regions encoding the early, domain-specific, decision evidence. These results suggest a domain-general decision-making architecture, whereupon domain-specific information is converted into a common representation in the valuation system and integrated for the decision in the pMFC.

Inferring Macroscale Brain Dynamics via Fusion of Simultaneous EEG-fMRI.

Advances in the instrumentation and signal processing for simultaneously acquired electroencephalography and functional magnetic resonance imaging (EEG-fMRI) have enabled new ways to observe the spatiotemporal neural dynamics of the human brain. Central to the utility of EEG-fMRI neuroimaging systems are the methods for fusing the two data streams, with machine learning playing a key role. These methods can be dichotomized into those that are symmetric and asymmetric in terms of how the two modalities inform the fusion. Studies using these methods have shown that fusion yields new insights into brain function that are not possible when each modality is acquired separately. As technology improves and methods for fusion become more sophisticated, the future of EEG-fMRI for noninvasive measurement of brain dynamics includes mesoscale mapping at ultrahigh magnetic resonance fields, targeted perturbation-based neuroimaging, and using deep learning to uncover nonlinear representations that link the electrophysiological and hemodynamic measurements.

The Leaky Integrating Threshold and its impact on evidence accumulation models of choice response time (RT).

A common assumption in choice response time (RT) modeling is that after evidence accumulation reaches a certain decision threshold, the choice is categorically communicated to the motor system that then executes the response. However, neurophysiological findings suggest that motor preparation partly overlaps with evidence accumulation, and is not independent from stimulus difficulty level. We propose to model this entanglement by changing the nature of the decision criterion from a simple threshold to an actual process. More specifically, we propose a secondary, motor preparation related, leaky accumulation process that takes the accumulated evidence of the original decision process as a continuous input, and triggers the actual response when it reaches its own threshold. We analytically develop this Leaky Integrating Threshold (LIT), applying it to a simple constant drift diffusion model, and show how its parameters can be estimated with the D*M method. Reanalyzing 3 different data sets, the LIT extension is shown to outperform a standard drift diffusion model using multiple statistical approaches. Further, the LIT leak parameter is shown to be better at explaining the speed/accuracy trade-off manipulation than the commonly used boundary separation parameter. These improvements can also be verified using traditional diffusion model analyses, for which the LIT predicts the violation of several common selective parameter influence assumptions. These predictions are consistent with what is found in the data and with what is reported experimentally in the literature. Crucially, this work offers a new benchmark against which to compare neural data to offer neurobiological validation for the proposed processes. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Auditory information enhances post-sensory visual evidence during rapid multisensory decision-making.

Despite recent progress in understanding multisensory decision-making, a conclusive mechanistic account of how the brain translates the relevant evidence into a decision is lacking. Specifically, it remains unclear whether perceptual improvements during rapid multisensory decisions are best explained by sensory (i.e., 'Early') processing benefits or post-sensory (i.e., 'Late') changes in decision dynamics. Here, we employ a well-established visual object categorisation task in which early sensory and post-sensory decision evidence can be dissociated using multivariate pattern analysis of the electroencephalogram (EEG). We capitalize on these distinct neural components to identify when and how complementary auditory information influences the encoding of decision-relevant visual evidence in a multisensory context. We show that it is primarily the post-sensory, rather than the early sensory, EEG component amplitudes that are being amplified during rapid audiovisual decision-making. Using a neurally informed drift diffusion model we demonstrate that a multisensory behavioral improvement in accuracy arises from an enhanced quality of the relevant decision evidence, as captured by the post-sensory EEG component, consistent with the emergence of multisensory evidence in higher-order brain areas.

Neural correlates of weighted reward prediction error during reinforcement learning classify response to cognitive behavioral therapy in depression.

While cognitive behavioral therapy (CBT) is an effective treatment for major depressive disorder, only up to 45% of depressed patients will respond to it. At present, there is no clinically viable neuroimaging predictor of CBT response. Notably, the lack of a mechanistic understanding of treatment response has hindered identification of predictive biomarkers. To obtain mechanistically meaningful fMRI predictors of CBT response, we capitalize on pretreatment neural activity encoding a weighted reward prediction error (RPE), which is implicated in the acquisition and processing of feedback information during probabilistic learning. Using a conventional mass-univariate fMRI analysis, we demonstrate that, at the group level, responders exhibit greater pretreatment neural activity encoding a weighted RPE in the right striatum and right amygdala. Crucially, using multivariate methods, we show that this activity offers significant out-of-sample classification of treatment response. Our findings support the feasibility and validity of neurocomputational approaches to treatment prediction in psychiatry.

Human VMPFC encodes early signatures of confidence in perceptual decisions.

Choice confidence, an individual's internal estimate of judgment accuracy, plays a critical role in adaptive behaviour, yet its neural representations during decision formation remain underexplored. Here, we recorded simultaneous EEG-fMRI while participants performed a direction discrimination task and rated their confidence on each trial. Using multivariate single-trial discriminant analysis of the EEG, we identified a stimulus-independent component encoding confidence, which appeared prior to subjects' explicit choice and confidence report, and was consistent with a confidence measure predicted by an accumulation-to-bound model of decision-making. Importantly, trial-to-trial variability in this electrophysiologically-derived confidence signal was uniquely associated with fMRI responses in the ventromedial prefrontal cortex (VMPFC), a region not typically associated with confidence for perceptual decisions. Furthermore, activity in the VMPFC was functionally coupled with regions of the frontal cortex linked to perceptual decision-making and metacognition. Our results suggest that the VMPFC holds an early confidence representation arising from decision dynamics, preceding and potentially informing metacognitive evaluation.

Dorsal Anterior Cingulate Cortices Differentially Lateralize Prediction Errors and Outcome Valence in a Decision-Making Task.

The dorsal anterior cingulate cortex (dACC) is proposed to facilitate learning by signaling mismatches between the expected outcome of decisions and the actual outcomes in the form of prediction errors. The dACC is also proposed to discriminate outcome valence-whether a result has positive (either expected or desirable) or negative (either unexpected or undesirable) value. However, direct electrophysiological recordings from human dACC to validate these separate, but integrated, dimensions have not been previously performed. We hypothesized that local field potentials (LFPs) would reveal changes in the dACC related to prediction error and valence and used the unique opportunity offered by deep brain stimulation (DBS) surgery in the dACC of three human subjects to test this hypothesis. We used a cognitive task that involved the presentation of object pairs, a motor response, and audiovisual feedback to guide future object selection choices. The dACC displayed distinctly lateralized theta frequency (3-8 Hz) event-related potential responses-the left hemisphere dACC signaled outcome valence and prediction errors while the right hemisphere dACC was involved in prediction formation. Multivariate analyses provided evidence that the human dACC response to decision outcomes reflects two spatiotemporally distinct early and late systems that are consistent with both our lateralized electrophysiological results and the involvement of the theta frequency oscillatory activity in dACC cognitive processing. Further findings suggested that dACC does not respond to other phases of action-outcome-feedback tasks such as the motor response which supports the notion that dACC primarily signals information that is crucial for behavioral monitoring and not for motor control.

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis.

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

Spatiotemporal neural characterization of prediction error valence and surprise during reward learning in humans.

Reward learning depends on accurate reward associations with potential choices. These associations can be attained with reinforcement learning mechanisms using a reward prediction error (RPE) signal (the difference between actual and expected rewards) for updating future reward expectations. Despite an extensive body of literature on the influence of RPE on learning, little has been done to investigate the potentially separate contributions of RPE valence (positive or negative) and surprise (absolute degree of deviation from expectations). Here, we coupled single-trial electroencephalography with simultaneously acquired fMRI, during a probabilistic reversal-learning task, to offer evidence of temporally overlapping but largely distinct spatial representations of RPE valence and surprise. Electrophysiological variability in RPE valence correlated with activity in regions of the human reward network promoting approach or avoidance learning. Electrophysiological variability in RPE surprise correlated primarily with activity in regions of the human attentional network controlling the speed of learning. Crucially, despite the largely separate spatial extend of these representations our EEG-informed fMRI approach uniquely revealed a linear superposition of the two RPE components in a smaller network encompassing visuo-mnemonic and reward areas. Activity in this network was further predictive of stimulus value updating indicating a comparable contribution of both signals to reward learning.

Neural correlates of evidence accumulation during value-based decisions revealed via simultaneous EEG-fMRI.

Current computational accounts posit that, in simple binary choices, humans accumulate evidence in favour of the different alternatives before committing to a decision. Neural correlates of this accumulating activity have been found during perceptual decisions in parietal and prefrontal cortex; however the source of such activity in value-based choices remains unknown. Here we use simultaneous EEG-fMRI and computational modelling to identify EEG signals reflecting an accumulation process and demonstrate that the within- and across-trial variability in these signals explains fMRI responses in posterior-medial frontal cortex. Consistent with its role in integrating the evidence prior to reaching a decision, this region also exhibits task-dependent coupling with the ventromedial prefrontal cortex and the striatum, brain areas known to encode the subjective value of the decision alternatives. These results further endorse the proposition of an evidence accumulation process during value-based decisions in humans and implicate the posterior-medial frontal cortex in this process.

Sounds facilitate visual motion discrimination via the enhancement of late occipital visual representations.

Sensory discriminations, such as judgements about visual motion, often benefit from multisensory evidence. Despite many reports of enhanced brain activity during multisensory conditions, it remains unclear which dynamic processes implement the multisensory benefit for an upcoming decision in the human brain. Specifically, it remains difficult to attribute perceptual benefits to specific processes, such as early sensory encoding, the transformation of sensory representations into a motor response, or to more unspecific processes such as attention. We combined an audio-visual motion discrimination task with the single-trial mapping of dynamic sensory representations in EEG activity to localize when and where multisensory congruency facilitates perceptual accuracy. Our results show that a congruent sound facilitates the encoding of motion direction in occipital sensory - as opposed to parieto-frontal - cortices, and facilitates later - as opposed to early (i.e. below 100ms) - sensory activations. This multisensory enhancement was visible as an earlier rise of motion-sensitive activity in middle-occipital regions about 350ms from stimulus onset, which reflected the better discriminability of motion direction from brain activity and correlated with the perceptual benefit provided by congruent multisensory information. This supports a hierarchical model of multisensory integration in which the enhancement of relevant sensory cortical representations is transformed into a more accurate choice.

Space-by-time decomposition for single-trial decoding of M/EEG activity.

We develop a novel methodology for the single-trial analysis of multichannel time-varying neuroimaging signals. We introduce the space-by-time M/EEG decomposition, based on Non-negative Matrix Factorization (NMF), which describes single-trial M/EEG signals using a set of non-negative spatial and temporal components that are linearly combined with signed scalar activation coefficients. We illustrate the effectiveness of the proposed approach on an EEG dataset recorded during the performance of a visual categorization task. Our method extracts three temporal and two spatial functional components achieving a compact yet full representation of the underlying structure, which validates and summarizes succinctly results from previous studies. Furthermore, we introduce a decoding analysis that allows determining the distinct functional role of each component and relating them to experimental conditions and task parameters. In particular, we demonstrate that the presented stimulus and the task difficulty of each trial can be reliably decoded using specific combinations of components from the identified space-by-time representation. When comparing with a sliding-window linear discriminant algorithm, we show that our approach yields more robust decoding performance across participants. Overall, our findings suggest that the proposed space-by-time decomposition is a meaningful low-dimensional representation that carries the relevant information of single-trial M/EEG signals.

Two spatiotemporally distinct value systems shape reward-based learning in the human brain.

Avoiding repeated mistakes and learning to reinforce rewarding decisions is critical for human survival and adaptive actions. Yet, the neural underpinnings of the value systems that encode different decision-outcomes remain elusive. Here coupling single-trial electroencephalography with simultaneously acquired functional magnetic resonance imaging, we uncover the spatiotemporal dynamics of two separate but interacting value systems encoding decision-outcomes. Consistent with a role in regulating alertness and switching behaviours, an early system is activated only by negative outcomes and engages arousal-related and motor-preparatory brain structures. Consistent with a role in reward-based learning, a later system differentially suppresses or activates regions of the human reward network in response to negative and positive outcomes, respectively. Following negative outcomes, the early system interacts and downregulates the late system, through a thalamic interaction with the ventral striatum. Critically, the strength of this coupling predicts participants' switching behaviour and avoidance learning, directly implicating the thalamostriatal pathway in reward-based learning.

Neural representations of confidence emerge from the process of decision formation during perceptual choices.

Choice confidence represents the degree of belief that one's actions are likely to be correct or rewarding and plays a critical role in optimizing our decisions. Despite progress in understanding the neurobiology of human perceptual decision-making, little is known about the representation of confidence. Importantly, it remains unclear whether confidence forms an integral part of the decision process itself or represents a purely post-decisional signal. To address this issue we employed a paradigm whereby on some trials, prior to indicating their decision, participants could opt-out of the task for a small but certain reward. This manipulation captured participants' confidence on individual trials and allowed us to discriminate between electroencephalographic signals associated with certain-vs.-uncertain trials. Discrimination increased gradually and peaked well before participants indicated their choice. These signals exhibited a temporal profile consistent with a process of evidence accumulation, culminating at time of peak discrimination. Moreover, trial-by-trial fluctuations in the accumulation rate of nominally identical stimuli were predictive of participants' likelihood to opt-out of the task, suggesting that confidence emerges from the decision process itself and is computed continuously as the process unfolds. Correspondingly, source reconstruction placed these signals in regions previously implicated in decision making, within the prefrontal and parietal cortices. Crucially, control analyses ensured that these results could not be explained by stimulus difficulty, lapses in attention or decision accuracy.

Human scalp potentials reflect a mixture of decision-related signals during perceptual choices.

Single-unit animal studies have consistently reported decision-related activity mirroring a process of temporal accumulation of sensory evidence to a fixed internal decision boundary. To date, our understanding of how response patterns seen in single-unit data manifest themselves at the macroscopic level of brain activity obtained from human neuroimaging data remains limited. Here, we use single-trial analysis of human electroencephalography data to show that population responses on the scalp can capture choice-predictive activity that builds up gradually over time with a rate proportional to the amount of sensory evidence, consistent with the properties of a drift-diffusion-like process as characterized by computational modeling. Interestingly, at time of choice, scalp potentials continue to appear parametrically modulated by the amount of sensory evidence rather than converging to a fixed decision boundary as predicted by our model. We show that trial-to-trial fluctuations in these response-locked signals exert independent leverage on behavior compared with the rate of evidence accumulation earlier in the trial. These results suggest that in addition to accumulator signals, population responses on the scalp reflect the influence of other decision-related signals that continue to covary with the amount of evidence at time of choice.