The effect of prediction error on overt attention and learning in humans.

It has been suggested that attention modulates the speed at which cues come to predict contingent outcomes, and that attention changes with the prediction errors generated by cues. Evidence for this interaction in humans is inconsistent, with divergent findings depending on whether attention was measured with eye fixations or learning speed. We included both measures in our experiment. Initially, predictive cues (A and B) were consistently followed by one outcome (o1), while nonpredictive cues (X and Y) were followed by two randomly alternating outcomes (o1 and o2). Consistent with an effect of prediction error, participants' fixated for longer on the nonpredictive cues than on the predictive ones. Then, the cues were combined in three pairs: AX, followed by o1, and AY and BX, followed by o2. Discrimination of AX and AY depended on the previously nonpredictive cues and, given that these received more attention during initial training, it should proceed faster than discrimination of AX and BX, which depended on the previously predictive cues. However, participants learned to predict the outcomes of AY and BX at a similar rate. The fixation times were similar for the previously predictive and previously nonpredictive cues. We discuss reasons that could explain these findings.

Evidence for two attentional mechanisms during learning.

We sought to provide evidence for a combined effect of two attentional mechanisms during associative learning. Participants' eye movements were recorded as they predicted the outcomes following different pairs of cues. Across the trials of an initial stage, a relevant cue in each pair was consistently followed by one of two outcomes, while an irrelevant cue was equally followed by either of them. Thus, the relevant cue should have been associated with small relative prediction errors, compared to the irrelevant cue. In a later stage, each pair came to be followed by one outcome on a random half of the trials and by the other outcome on the remaining half, and thus there should have been a rise in the overall prediction error. Consistent with an attentional mechanism based on relative prediction error, an attentional advantage for the relevant cue was evident in the first stage. However, in accordance with a mechanism linked to overall prediction error, the attention paid to both types of cues increased at the beginning of the second stage. These results showed up in both dwell times and within-trial patterns of fixations, and they were predicted by a hybrid model of attention.

Emergence of complex dynamics of choice due to repeated exposures to extinction learning.

Extinction learning, the process of ceasing an acquired behavior in response to altered reinforcement contingencies, is not only essential for survival in a changing environment, but also plays a fundamental role in the treatment of pathological behaviors. During therapy and other forms of training involving extinction, subjects are typically exposed to several sessions with a similar structure. The effects of this repeated exposure are not well understood. Here, we studied the behavior of pigeons across several sessions of a discrimination-learning task in context A, extinction in context B, and a return to context A to test the context-dependent return of the learned responses (ABA renewal). By focusing on individual learning curves across animals, we uncovered a session-dependent variability of behavior: (1) during extinction, pigeons preferred the unrewarded alternative choice in one-third of the sessions, predominantly during the first one. (2) In later sessions, abrupt transitions of behavior at the onset of context B emerged, and (3) the renewal effect decayed as sessions progressed. We show that the observed results can be parsimoniously accounted for by a computational model based only on associative learning between stimuli and actions. Our work thus demonstrates the critical importance of studying the trial-by-trial dynamics of learning in individual sessions, and the power of "simple" associative learning processes.

Positive affective states can play the role of context to renew extinguished instrumental behavior in rats.

In one experiment with rats, we examined whether positive affective states can serve as contexts in a between-subjects ABA renewal design using appetitive instrumental conditioning. Two groups of rats received training to press a lever for food where each acquisition session was preceded by administration of a tickling procedure (Context A) known to induce positive affective states. Then, lever pressing underwent extinction where rats received a pure handling treatment (Context B) before each session. During a final test session, we found stronger responding when the session was preceded by tickling (Group ABA) compared to handling (Group ABB), indicating an ABA renewal effect. Furthermore, test performance in Group ABB was not different from that in a third group where handling preceded acquisition sessions, and tickling extinction and test sessions (Group BAA), showing that tickling did not elevate instrumental responding during the test if it had been unrelated to initial acquisition. We discuss implications of our results for understanding the role of positive affective states in relapse of problem behavior.

A rise in prediction error increases attention to irrelevant cues.

We investigated whether a sudden rise in prediction error widens an individual's focus of attention by increasing ocular fixations on cues that otherwise tend to be ignored. To this end, we used a discrimination learning task including cues that were either relevant or irrelevant for predicting the outcomes. Half of participants experienced contingency reversal once they had learned to predict the outcomes (reversal group, n = 30). The other half experienced the same contingencies throughout the task (control group, n = 30). As participants' prediction accuracy increased, they showed a decrease in the number of fixations directed to the irrelevant cues. Following contingency reversal, participants in the reversal group showed a drop in accuracy, indicating a rise in prediction error, and fixated on the irrelevant cues more often than participants in the control group. We discuss the results in the context of attentional theories of associative learning.

Reduced emission of alarm 22-kHz ultrasonic vocalizations during fear conditioning in rats lacking the serotonin transporter.

Rats display a rich social behavioral repertoire. An important component of this repertoire is the emission of whistle-like calls in the ultrasonic range, so-called ultrasonic vocalizations (USV). Long low-frequency 22-kHz USV occur in aversive situations, including aggressive interactions, predator exposure, and electric shocks during fear conditioning. They are believed to reflect a negative affective state akin to anxiety and fear. A prominent theory suggests that 22-kHz USV function as alarm calls to warn conspecifics. Serotonin (5-hydroxytryptamine, 5-HT) is strongly implicated in the regulation of affective states, particularly anxiety and fear. A key component of the system is the 5-HT transporter (5-HTT, also known as SERT), regulating 5-HT availability in the synaptic cleft. In the present experiment, we studied the effects of SERT deficiency on overt fear-related behavior and alarm 22-kHz USV during fear conditioning in male and female rats. While overt fear-related behavior was not affected by SERT deficiency and sex, the emission of alarm 22-kHz USV was clearly reduced in homozygous SERT-/- but not heterozygous SERT+/- mutants, as compared to their wildtype SERT+/+ littermate controls. Genotype effects were particularly prominent in females. Females in general emitted fewer alarm 22-kHz USV than males. This supports the view that 22-kHz USV are, at least partly, independently regulated from anxiety or fear and as socially mediated alarm calls do not simply express a negative affective state. Reduced 22-kHz USV emission in rats lacking SERT might be due to social deficits in the use of 22-kHz USV as a socio-affective signal to warn conspecifics about threats.

Adding or removing context components equally disrupts extinction in human predictive learning.

In two human predictive learning experiments, we investigated the impact of adding or removing context components on extinction performance toward a cue. In each experiment, participants initially received repeated pairings of a cue and an outcome in a context composed of two distinctive components (context AB). Initial training was followed by a series of trials in which the cue was no longer followed by the outcome. This extinction treatment was conducted in the presence of a different pair of distinctive context components (context CD). During a final test, we observed that changing the extinction context CD disrupted extinction performance toward the cue regardless of whether the context was changed by adding or removing context components. We discuss implications of our results for theories of associative learning.

The role of common elements in the redundancy effect.

In 2 experiments, participants received a predictive learning task in which the presence of 1 or 2 food items signaled the onset or absence of stomachache in a hypothetical patient. Their task was to identify the cues that signaled the occurrence, or nonoccurrence of this ailment. The 2 groups in Experiment 1 and the single group in Experiment 2 received a blocking treatment, where Cue A and a combination of Cues A and X both signaled stomachache, A+ AX+. These groups also received a simple discrimination where the outcome was signaled by one compound but not another, BY+ CY-. Subsequent test trials revealed the so-called redundancy effect, where X was regarded as a more reliable predictor of the outcome than Y. This result occurred when the trials with A+ preceded those with AX+ (Group E, Experiment 1 and Experiment 2), and when the trials with A+ and AX+ were intermixed (Group C, Experiment 1). The results challenge theories based on the assumption that cues presented together must compete for a limited pool of associative strength. Rather, they are said to support theories that assume changes in attention determine what is learned when two or more cues are presented together. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Test-retest reliability of response recovery after discrimination reversal learning.

In one human predictive learning experiment, we demonstrated that an individual's propensity for response recovery following discrimination reversal learning is stable over time. Participants received four sessions of training with the first three sessions being separated by one week each, while the last session was conducted after a delay of four weeks. During each session, participants initially received discrimination training (E+, F-) in one context, followed by discrimination reversal training (E-, F+) in another context. Sessions each completed with a test, in which the stimuli were presented in the context of initial acquisition. Each test revealed response recovery according to the initially acquired stimulus-outcome contingencies. Furthermore, the strength of response recovery was correlated across sessions that were separated by one week (Sessions 1 and 2), and across sessions separated by four weeks (Sessions 3 and 4). Overall, intra-individual test behavior was stable in 87 % of participants across two sessions, and in 79 % of participants across four sessions. Our results indicate that inter-individual differences in response recovery are a reliable phenomenon, which is a finding that is not accounted by current theories of context-dependent learning.

The impacts of acquisition and extinction cues on ABC renewal of voluntary behaviors.

In two instrumental conditioning experiments with rats, we examined the impacts of acquisition and extinction cues on ABC renewal of instrumental behavior. Animals were reinforced with food for lever pressing in one context, followed by extinction of the response in a second one. Presentations of a brief tone accompanied extinction in Experiment 1 (extinction cue), and acquisition in Experiment 2 (acquisition cue). A final test in a third context revealed that instrumental responding was decreased in the presence of the extinction cue, whereas it was increased in the presence of the acquisition cue. We discuss theoretical and clinical implications of our results.

Extinction of cognitive associations is preserved in patients with cerebellar disease.

In the present study extinction and renewal of cognitive associations were assessed in two experiments in participants with focal and degenerative cerebellar disease. Using a predictive learning task, participants had to learn by trial and error the relationships between food items and the occurrence of stomach trouble in a hypothetical patient. In the first experiment, focus was on renewal effects. Participants with chronic cerebellar stroke (n = 14; mean age 50.9 ± 12 years), participants with degenerative cerebellar disease (n = 16; mean age 58 ± 12 years), age-, sex-, and education matched controls (n = 20; mean age 53.7 ± 10.8 years) and young controls (n = 19; mean age 23.2 ± 2.7 years) were tested. Acquisition and extinction of food-stomach trouble associations took part in two different contexts (represented by restaurants). In a subsequent test phase, food stimuli were presented in both contexts and no feedback was given. This allowed testing for renewal of the initially acquired associations in the acquisition context. Acquisition and extinction learning were not significantly different between groups. Significant renewal effects were present in young controls only. In the second experiment, focus was on extinction. To control for age effects, 19 young participants with chronic surgical lesions of the cerebellum (mean age 25.6 ± 6.1 years), and 24 age-, sex- and education-matched healthy controls were tested. Acquisition and extinction of food-stomach trouble associations took part in the same context. In the extinction phase, the relationship with stomach trouble was reversed in some of the food items. Acquisition and extinction learning were not significantly different between groups. The main finding of the present study was preserved extinction of learned cognitive associations in participants with chronic cerebellar disease. Findings agree with previous observations in the literature that cognitive abnormalities are frequently absent or weak in adults with cerebellar disease. This does not exclude a contribution of the cerebellum to extinction of learned associations. For example, findings may be different in more challenging cognitive tasks, and in participants with acute cerebellar disease with no time for compensation.

How competitive is cue competition?

Cue competition refers to phenomena indicating that learning about the relationship between a cue and an outcome is influenced by learning about the predictive significance of other cues that are concurrently present. In two autoshaping experiments with pigeons, we investigated the strength of competition among cues for predictive value. In each experiment, animals received an overexpectation training (A+, D+ followed by AD+). In addition, the training schedule of each experiment comprised two control conditions-one condition to evaluate the presence of overexpectation (B+ followed by BY+) and a second one to assess the strength of competition among cues (C+ followed by CZ-). Training trials were followed by a test with individual stimuli (A, B, C). Experiment 1 revealed no evidence for cue competition as responding during the test mirrored the individual cue-outcome contingencies. The test results from Experiment 2, which included an outcome additivity training, showed cue competition in form of an overexpectation effect as responding was weaker for Stimulus A than Stimulus B. However, the test results from Experiment 2 also revealed that responding to Stimulus A was stronger than to Stimulus C, which indicates that competition among cues was not as strong as predicted by some influential theories of associative learning.

Transfer of associability and relational structure in human associative learning.

A wealth of recent studies have demonstrated that predictive cues involved in a linearly solvable component discrimination gain associability in subsequent learning relative to nonpredictive cues. In contrast, contradictory findings have been reported about the fate of cues involved in learning biconditional discriminations in which the cues are relevant but none are individually predictive of a specific outcome. In 3 experiments we examined the transfer of learning from component and biconditional discriminations in a within-subjects design. The results show a greater benefit in associability for cues that had previously served as predictive cues in a component discrimination than cues previously used in a biconditional discrimination. Further, new biconditional discriminations were learned faster when they were composed of cues that were previously trained in separate biconditional discriminations. Similarly, new component discriminations were learned faster when they were composed of cues that were previously trained in a separate component discriminations irrespective of whether they were previously predictive or previously nonpredictive. These results provide novel evidence that cue-specific learning of relational structure affects subsequent learning, suggesting changes in cue processing that go beyond simple changes in cue associability based on learned predictiveness. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

The fate of redundant cues in human predictive learning: The outcome ratio effect.

In four experiments, participants were shown a sequence of pairs of pictures of food and asked to predict whether each pair signalled an allergic reaction in a hypothetical patient. The pairs of pictures were used to present two simple discriminations that differed in their outcome ratio. A rich discrimination, 3AX+ BX-, involved three trials in which the compound of two foods, AX, was followed by a reaction, for every trial in which the compound BX was not followed by the outcome. A lean discrimination, CY+ 3DY- was based on the opposite outcome ratio. Upon the completion of this training, participants were asked to rate how likely an individual food would be followed by the allergic reaction. In each experiment, the rating for X was stronger than for Y. This outcome ratio effect poses a challenge for theories of learning that assume changes in associative strength are governed by a common error term, based on the significance of all the cues present on a trial. Instead, the results are consistent with the assumption that changes in associative strength are governed by an individual error term, based on the significance of a single cue.

A test for a difference in the associability of blocked and uninformative cues in human predictive learning.

In human predictive learning, blocking, A+ AB+, and a simple discrimination, UX+ VX-, result in a stronger response to the blocked, B, than the uninformative cue, X (where letters represent cues and + and - represent different outcomes). To assess whether these different treatments result in more attention being paid to blocked than uninformative cues, Stage 1 in each of three experiments generated two blocked cues, B and E, and two uninformative cues, X and Y. In Stage 2, participants received two simple discriminations: either BX+ EX- and BY+ EY-, or BX+ BY- and EX+ EY-. If more attention is paid to blocked than uninformative cues, then the first pair of discriminations will be solved more readily than the second pair. In contrast to this prediction, both discriminations were acquired at the same rate. These results are explained by the theory of Mackintosh, by virtue of the assumption that learning is governed by an individual rather than a common error term.

Attention toward contexts modulates context-specificity of behavior in human predictive learning: Evidence from the n-back task.

According to the attentional theory of context processing (ATCP), learning becomes context specific when acquired under conditions that promote attention toward contextual stimuli regardless of whether attention deployment is guided by learning experience or by other factors unrelated to learning. In one experiment with humans, we investigated whether performance in a predictive learning task can be brought under contextual control by means of a secondary task that was unrelated to predictive learning, but supposed to modulate participants' attention toward contexts. Initially, participants acquired cue-outcome relationships presented in contexts that were each composed of two elements from two dimensions. Acquisition training in the predictive learning task was combined with a one-back task that required participants to match across consecutive trials context elements belonging to one of the two dimensions. During a subsequent test, we observed that acquisition behavior in the predictive learning task was disrupted by changing the acquisition context along the dimension that was relevant for the one-back task, while there was no evidence for context specificity of predictive learning when the acquisition context was changed along the dimension that was irrelevant for the one-back task. Our results support the generality of the principles advocated by ATCP.

Removing but not adding elements of a context affects generalization of instrumental responses.

Three experiments with rats investigated whether adding or removing elements of a context affects generalization of instrumental behavior. Each of the experiments used a free operant procedure. In Experiments 1 and 2, rats were trained to press a lever for food in a distinctive context. Then, transfer of lever pressing was tested in a context created either by adding an element to the context of initial acquisition or by removing one of the acquisition context's elements. In Experiment 3, a similar generalization test was conducted after rats received acquisition and extinction within the same context. For Experiments 1 and 2, we observed that removing elements from the acquisition context disrupted acquisition performance, whereas the addition of elements to the context did not. Experiment 3 revealed that removing elements from but not adding elements to the original context improved extinction performance. Our results are consistent with an elemental view of context representation.

Pupil dilation indicates the coding of past prediction errors: Evidence for attentional learning theory.

The attentional learning theory of Pearce and Hall () predicts more attention to uncertain cues that have caused a high prediction error in the past. We examined how the cue-elicited pupil dilation during associative learning was linked to such error-driven attentional processes. In three experiments, participants were trained to acquire associations between different cues and their appetitive (Experiment 1), motor (Experiment 2), or aversive (Experiment 3) outcomes. All experiments were designed to examine differences in the processing of continuously reinforced cues (consistently followed by the outcome) versus partially reinforced, uncertain cues (randomly followed by the outcome). We measured the pupil dilation elicited by the cues in anticipation of the outcome and analyzed how this conditioned pupil response changed over the course of learning. In all experiments, changes in pupil size complied with the same basic pattern: During early learning, consistently reinforced cues elicited greater pupil dilation than uncertain, randomly reinforced cues, but this effect gradually reversed to yield a greater pupil dilation for uncertain cues toward the end of learning. The pattern of data accords with the changes in prediction error and error-driven attention formalized by the Pearce-Hall theory.

Context modulation of learned attention deployment.

In three experiments, we investigated the contextual control of attention in human discrimination learning. In each experiment, participants initially received discrimination training in which the cues from Dimension A were relevant in Context 1 but irrelevant in Context 2, whereas the cues from Dimension B were irrelevant in Context 1 but relevant in Context 2. In Experiment 1, the same cues from each dimension were used in Contexts 1 and 2, whereas in Experiments 2 and 3, the cues from each dimension were changed across contexts. In each experiment, participants were subsequently shifted to a transfer discrimination involving novel cues from either dimension, to assess the contextual control of attention. In Experiment 1, measures of eye gaze during the transfer discrimination revealed that Dimension A received more attention than Dimension B in Context 1, whereas the reverse occurred in Context 2. Corresponding results indicating the contextual control of attention were found in Experiments 2 and 3, in which we used the speed of learning (associability) as an indirect marker of learned attentional changes. Implications of our results for current theories of learning and attention are discussed.

Fear acquisition and liking of out-group and in-group members: Learning bias or attention?

The present study explores the notion of an out-group fear learning bias that is characterized by facilitated fear acquisition toward harm-doing out-group members. Participants were conditioned with two in-group and two out-group faces as conditioned stimuli. During acquisition, one in-group and one out-group face was paired with an aversive shock whereas the other in-group and out-group face was presented without shock. Psychophysiological measures of fear conditioning (skin conductance and pupil size) and explicit and implicit liking exhibited increased differential responding to out-group faces compared to in-group faces. However, the results did not clearly indicate that harm-doing out-group members were more readily associated with fear than harm-doing in-group members. In contrast, the out-group face not paired with shock decreased conditioned fear and disliking at least to the same extent that the shock-associated out-group face increased these measures. Based on these results, we suggest an account of the out-group fear learning bias that relates to an attentional bias to process in-group information.