Structured, uncertainty-driven exploration in real-world consumer choice.

Making good decisions requires people to appropriately explore their available options and generalize what they have learned. While computational models can explain exploratory behavior in constrained laboratory tasks, it is unclear to what extent these models generalize to real-world choice problems. We investigate the factors guiding exploratory behavior in a dataset consisting of 195,333 customers placing 1,613,967 orders from a large online food delivery service. We find important hallmarks of adaptive exploration and generalization, which we analyze using computational models. In particular, customers seem to engage in uncertainty-directed exploration and use feature-based generalization to guide their exploration. Our results provide evidence that people use sophisticated strategies to explore complex, real-world environments.

Confounds in multivariate pattern analysis: Theory and rule representation case study.

Multivariate pattern analysis (MVPA) is a relatively recent innovation in functional magnetic resonance imaging (fMRI) methods. MVPA is increasingly widely used, as it is apparently more effective than classical general linear model analysis (GLMA) for detecting response patterns or representations that are distributed at a fine spatial scale. However, we demonstrate that widely used approaches to MVPA can systematically admit certain confounds that are appropriately eliminated by GLMA. Thus confounds rather than distributed representations may explain some cases in which MVPA produced positive results but GLMA did not. The issue is that it is common practice in MVPA to conduct group tests on single-subject summary statistics that discard the sign or direction of underlying effects, whereas GLMA group tests are conducted directly on single-subject effects themselves. We describe how this common MVPA practice undermines standard experiment design logic that is intended to control at the group level for certain types of confounds, such as time on task and individual differences. Furthermore, we note that a simple application of linear regression can restore experimental control when using MVPA in many situations. Finally, we present a case study with novel fMRI data in the domain of rule representations, or flexible stimulus-response mappings, which has seen several recent MVPA publications. In our new dataset, as with recent reports, standard MVPA appears to reveal rule representations in prefrontal cortex regions, whereas GLMA produces null results. However, controlling for a variable that is confounded with rule at the individual-subject level but not the group level (reaction time differences across rules) eliminates the MVPA results. This raises the question of whether recently reported results truly reflect rule representations, or rather the effects of confounds such as reaction time, difficulty, or other variables of no interest.

The successor representation and temporal context.

The successor representation was introduced into reinforcement learning by Dayan ( 1993 ) as a means of facilitating generalization between states with similar successors. Although reinforcement learning in general has been used extensively as a model of psychological and neural processes, the psychological validity of the successor representation has yet to be explored. An interesting possibility is that the successor representation can be used not only for reinforcement learning but for episodic learning as well. Our main contribution is to show that a variant of the temporal context model (TCM; Howard & Kahana, 2002 ), an influential model of episodic memory, can be understood as directly estimating the successor representation using the temporal difference learning algorithm (Sutton & Barto, 1998 ). This insight leads to a generalization of TCM and new experimental predictions. In addition to casting a new normative light on TCM, this equivalence suggests a previously unexplored point of contact between different learning systems.