Replies to commentaries on beyond playing 20 questions with nature.

Commentaries on the target article offer diverse perspectives on integrative experiment design. Our responses engage three themes: (1) Disputes of our characterization of the problem, (2) skepticism toward our proposed solution, and (3) endorsement of the solution, with accompanying discussions of its implementation in existing work and its potential for other domains. Collectively, the commentaries enhance our confidence in the promise and viability of integrative experiment design, while highlighting important considerations about how it is used.

The Psychological Science Accelerator's COVID-19 rapid-response dataset.

In response to the COVID-19 pandemic, the Psychological Science Accelerator coordinated three large-scale psychological studies to examine the effects of loss-gain framing, cognitive reappraisals, and autonomy framing manipulations on behavioral intentions and affective measures. The data collected (April to October 2020) included specific measures for each experimental study, a general questionnaire examining health prevention behaviors and COVID-19 experience, geographical and cultural context characterization, and demographic information for each participant. Each participant started the study with the same general questions and then was randomized to complete either one longer experiment or two shorter experiments. Data were provided by 73,223 participants with varying completion rates. Participants completed the survey from 111 geopolitical regions in 44 unique languages/dialects. The anonymized dataset described here is provided in both raw and processed formats to facilitate re-use and further analyses. The dataset offers secondary analytic opportunities to explore coping, framing, and self-determination across a diverse, global sample obtained at the onset of the COVID-19 pandemic, which can be merged with other time-sampled or geographic data.

Beyond Playing 20 Questions with Nature: Integrative Experiment Design in the Social and Behavioral Sciences.

The dominant paradigm of experiments in the social and behavioral sciences views an experiment as a test of a theory, where the theory is assumed to generalize beyond the experiment's specific conditions. According to this view, which Alan Newell once characterized as "playing twenty questions with nature," theory is advanced one experiment at a time, and the integration of disparate findings is assumed to happen via the scientific publishing process. In this article, we argue that the process of integration is at best inefficient, and at worst it does not, in fact, occur. We further show that the challenge of integration cannot be adequately addressed by recently proposed reforms that focus on the reliability and replicability of individual findings, nor simply by conducting more or larger experiments. Rather, the problem arises from the imprecise nature of social and behavioral theories and, consequently, a lack of commensurability across experiments conducted under different conditions. Therefore, researchers must fundamentally rethink how they design experiments and how the experiments relate to theory. We specifically describe an alternative framework, integrative experiment design, which intrinsically promotes commensurability and continuous integration of knowledge. In this paradigm, researchers explicitly map the design space of possible experiments associated with a given research question, embracing many potentially relevant theories rather than focusing on just one. The researchers then iteratively generate theories and test them with experiments explicitly sampled from the design space, allowing results to be integrated across experiments. Given recent methodological and technological developments, we conclude that this approach is feasible and would generate more-reliable, more-cumulative empirical and theoretical knowledge than the current paradigm-and with far greater efficiency.

The experimental evolution of human culture: flexibility, fidelity and environmental instability.

The past 2 Myr have seen both unprecedented environmental instability and the evolution of the human capacity for complex culture. This, along with the observation that cultural evolution occurs faster than genetic evolution, has led to the suggestion that culture is an adaptation to an unstable environment. We test this hypothesis by examining the ability of human social learning to respond to environmental changes. We do this by inserting human participants (n = 4800) into evolutionary simulations with a changing environment while varying the social information available to individuals across five conditions. We find that human social learning shows some signs of adaptation to environmental instability, including critical social learning, the adoption of up-and-coming traits and, unexpectedly, contrariness. However, these are insufficient to avoid significant fitness declines when the environment changes, and many individuals are highly conformist, which exacerbates the fitness effects of environmental change. We conclude that human social learning reflects a compromise between the competing needs for flexibility to accommodate environmental change and fidelity to accurately transmit valuable cultural information.

Shadow banning, astroturfing, catfishing, and other online conflicts where beliefs about group membership diverge.

Drawing from conflicts observed in online communities (e.g., astroturfing and shadow banning), I extend Pietraszewski's theory to accommodate phenomena dependent on the intersubjectivity of groups, where representations of group membership (or beliefs about group membership) diverge. Doing so requires enriching representations to include other agents and their beliefs in a process of recursive mentalizing.

Deep models of superficial face judgments.

The diversity of human faces and the contexts in which they appear gives rise to an expansive stimulus space over which people infer psychological traits (e.g., trustworthiness or alertness) and other attributes (e.g., age or adiposity). Machine learning methods, in particular deep neural networks, provide expressive feature representations of face stimuli, but the correspondence between these representations and various human attribute inferences is difficult to determine because the former are high-dimensional vectors produced via black-box optimization algorithms. Here we combine deep generative image models with over 1 million judgments to model inferences of more than 30 attributes over a comprehensive latent face space. The predictive accuracy of our model approaches human interrater reliability, which simulations suggest would not have been possible with fewer faces, fewer judgments, or lower-dimensional feature representations. Our model can be used to predict and manipulate inferences with respect to arbitrary face photographs or to generate synthetic photorealistic face stimuli that evoke impressions tuned along the modeled attributes.

Memory transmission in small groups and large networks: An empirical study.

When people try to remember information in a group, they often recall less than if they were recalling alone. This finding is called collaborative inhibition, and has been studied primarily in small groups because of the difficulty of bringing large groups into the laboratory. To study the dynamics of collaborative inhibition in large groups (Luhmann & Rajaram, Psychological Science, 26, 1909-1917, 2015) constructed an agent-based model that extrapolated from previous laboratory experiments with small groups. The model predicts that collaborative inhibition should increase with group size. Here, we evaluate this model by recruiting a large number of participants using crowdsourcing, allowing us to replace the artificial agents in the model with people to study collaborative memory at larger scales. Our empirical results did not match the model predictions: there was no evidence for an increase in collaborative inhibition with group size, despite substantial power to detect such an effect. These findings motivate further empirical work to elucidate the mechanisms of collaborative memory.

Serial reproduction reveals the geometry of visuospatial representations.

An essential function of the human visual system is to locate objects in space and navigate the environment. Due to limited resources, the visual system achieves this by combining imperfect sensory information with a belief state about locations in a scene, resulting in systematic distortions and biases. These biases can be captured by a Bayesian model in which internal beliefs are expressed in a prior probability distribution over locations in a scene. We introduce a paradigm that enables us to measure these priors by iterating a simple memory task where the response of one participant becomes the stimulus for the next. This approach reveals an unprecedented richness and level of detail in these priors, suggesting a different way to think about biases in spatial memory. A prior distribution on locations in a visual scene can reflect the selective allocation of coding resources to different visual regions during encoding ("efficient encoding"). This selective allocation predicts that locations in the scene will be encoded with variable precision, in contrast to previous work that has assumed fixed encoding precision regardless of location. We demonstrate that perceptual biases covary with variations in discrimination accuracy, a finding that is aligned with simulations of our efficient encoding model but not the traditional fixed encoding view. This work demonstrates the promise of using nonparametric data-driven approaches that combine crowdsourcing with the careful curation of information transmission within social networks to reveal the hidden structure of shared visual representations.

Experimental evolutionary simulations of learning, memory and life history.

Humans possess an unusual combination of traits, including our cognition, life history, demographics and geographical distribution. Many theories propose that these traits have coevolved. Such hypotheses have been explored both theoretically and empirically, with experiments examining whether human behaviour meets theoretical expectations. However, theory must make assumptions about the human mind, creating a potentially problematic gap between models and reality. Here, we employ a series of 'experimental evolutionary simulations' to reduce this gap and to explore the coevolution of learning, memory and childhood. The approach combines aspects of theory and experiment by inserting human participants as agents within an evolutionary simulation. Across experiments, we find that human behaviour supports the coevolution of learning, memory and childhood, but that this is dampened by rapid environmental change. We conclude by discussing both the implications of these findings for theories of human evolution and the utility of experimental evolutionary simulations more generally. This article is part of the theme issue 'Life history and learning: how childhood, caregiving and old age shape cognition and culture in humans and other animals'.

What the Baldwin Effect affects depends on the nature of plasticity.

In a process known as the Baldwin Effect, developmental plasticity, such as learning, has been argued to accelerate the biological evolution of high-fitness traits, including language and complex intelligence. Here we investigate the evolutionary consequences of developmental plasticity by asking which aspects of a plastic trait are the focus of genetic change. The aspects we consider are: (i) dependencies between elements of a trait, (ii) the importance of each element to fitness, and (iii) the difficulty of acquiring each element through plasticity. We also explore (iv) how cultural inheritance changes the relationship between plasticity and genetic change. We find that evolution by natural selection preferentially fixes elements that are depended upon by others, important to fitness, or difficult to acquire through plasticity, but that cultural inheritance can suppress and even reverse genetic change. We replicate some of these effects in experimental evolutionary simulations with human learners. We conclude that what the Baldwin Effect affects depends upon the mechanism of plasticity, which for behavior and cognition includes the psychology of learning.

Evolution in Mind: Evolutionary Dynamics, Cognitive Processes, and Bayesian Inference.

Evolutionary theory describes the dynamics of population change in settings affected by reproduction, selection, mutation, and drift. In the context of human cognition, evolutionary theory is most often invoked to explain the origins of capacities such as language, metacognition, and spatial reasoning, framing them as functional adaptations to an ancestral environment. However, evolutionary theory is useful for understanding the mind in a second way: as a mathematical framework for describing evolving populations of thoughts, ideas, and memories within a single mind. In fact, deep correspondences exist between the mathematics of evolution and of learning, with perhaps the deepest being an equivalence between certain evolutionary dynamics and Bayesian inference. This equivalence permits reinterpretation of evolutionary processes as algorithms for Bayesian inference and has relevance for understanding diverse cognitive capacities, including memory and creativity.

Looking inward and back: Real-time monitoring of visual working memories.

Confidence in our memories is influenced by many factors, including beliefs about the perceptibility or memorability of certain kinds of objects and events, as well as knowledge about our skill sets, habits, and experiences. Notoriously, our knowledge and beliefs about memory can lead us astray, causing us to be overly confident in eyewitness testimony or to overestimate the frequency of recent experiences. Here, using visual working memory as a case study, we stripped away all these potentially misleading cues, requiring observers to make confidence judgments by directly assessing the quality of their memory representations. We show that individuals can monitor the status of information in working memory as it degrades over time. Our findings suggest that people have access to information reflecting the existence and quality of their working memories, and furthermore, that they can use this information to guide their behavior. (PsycINFO Database Record

Building a Social Network One Choice at a Time.

Newcomers to a social network show preferential attachment, a tendency to befriend those with many friends. Here, we show that preferential attachment is equivalent to a form of 'probability matching' commonly found in studies of decision-making. This equivalence, whereby newcomers probability match to a social signal akin to popularity, marries network science to the study of decision-making and raises new questions about how individual psychology impacts the social structure of groups. We asked people to view a visualization of a social network and to select group members whom they would like to meet and befriend. People varied in how strongly they weighed popularity and this was mildly correlated with aspects of their personality. Individual differences in preferential attachment affect the structure and connectivity of the network that emerges.

Terms of the debate on the format and structure of visual memory.

Our ability to actively maintain information in visual memory is strikingly limited. There is considerable debate about why this is so. As with many questions in psychology, the debate is framed dichotomously: Is visual working memory limited because it is supported by only a small handful of discrete "slots" into which visual representations are placed, or is it because there is an insufficient supply of a "resource" that is flexibly shared among visual representations? Here, we argue that this dichotomous framing obscures a set of at least eight underlying questions. Separately considering each question reveals a rich hypothesis space that will be useful for building a comprehensive model of visual working memory. The questions regard (1) an upper limit on the number of represented items, (2) the quantization of the memory commodity, (3) the relationship between how many items are stored and how well they are stored, (4) whether the number of stored items completely determines the fidelity of a representation (vs. fidelity being stochastic or variable), (5) the flexibility with which the memory commodity can be assigned or reassigned to items, (6) the format of the memory representation, (7) how working memories are formed, and (8) how memory representations are used to make responses in behavioral tasks. We reframe the debate in terms of these eight underlying questions, placing slot and resource models as poles in a more expansive theoretical space.

The crowd is self-aware.

Bentley et al.'s framework assigns phenomena of personal and collective decision-making to regions of a dual-axis map. Here, we propose that understanding the collective dynamics of decision-making requires consideration of factors that guide movement across the map. One such factor is self-awareness, which can lead a group to seek out new knowledge and re-position itself on the map.

Modeling visual working memory with the MemToolbox.

The MemToolbox is a collection of MATLAB functions for modeling visual working memory. In support of its goal to provide a full suite of data analysis tools, the toolbox includes implementations of popular models of visual working memory, real and simulated data sets, Bayesian and maximum likelihood estimation procedures for fitting models to data, visualizations of data and fit, validation routines, model comparison metrics, and experiment scripts. The MemToolbox is released under the permissive BSD license and is available at http://memtoolbox.org.

Learning to detect and combine the features of an object.

To recognize an object, it is widely supposed that we first detect and then combine its features. Familiar objects are recognized effortlessly, but unfamiliar objects--like new faces or foreign-language letters--are hard to distinguish and must be learned through practice. Here, we describe a method that separates detection and combination and reveals how each improves as the observer learns. We dissociate the steps by two independent manipulations: For each step, we do or do not provide a bionic crutch that performs it optimally. Thus, the two steps may be performed solely by the human, solely by the crutches, or cooperatively, when the human takes one step and a crutch takes the other. The crutches reveal a double dissociation between detecting and combining. Relative to the two-step ideal, the human observer's overall efficiency for unconstrained identification equals the product of the efficiencies with which the human performs the steps separately. The two-step strategy is inefficient: Constraining the ideal to take two steps roughly halves its identification efficiency. In contrast, we find that humans constrained to take two steps perform just as well as when unconstrained, which suggests that they normally take two steps. Measuring threshold contrast (the faintness of a barely identifiable letter) as it improves with practice, we find that detection is inefficient and learned slowly. Combining is learned at a rate that is 4× higher and, after 1,000 trials, 7× more efficient. This difference explains much of the diversity of rates reported in perceptual learning studies, including effects of complexity and familiarity.

Variability in the quality of visual working memory.

Working memory is a mental storage system that keeps task-relevant information accessible for a brief span of time, and it is strikingly limited. Its limits differ substantially across people but are assumed to be fixed for a given person. Here we show that there is substantial variability in the quality of working memory representations within an individual. This variability can be explained neither by fluctuations in attention or arousal over time, nor by uneven distribution of a limited mental commodity. Variability of this sort is inconsistent with the assumptions of the standard cognitive models of working memory capacity, including both slot- and resource-based models, and so we propose a new framework for understanding the limitations of working memory: a stochastic process of degradation that plays out independently across memories.

Motion silences awareness of visual change.

Loud bangs, bright flashes, and intense shocks capture attention, but other changes--even those of similar magnitude--can go unnoticed. Demonstrations of change blindness have shown that observers fail to detect substantial alterations to a scene when distracted by an irrelevant flash, or when the alterations happen gradually [1-5]. Here, we show that objects changing in hue, luminance, size, or shape appear to stop changing when they move. This motion-induced failure to detect change, silencing, persists even though the observer attends to the objects, knows that they are changing, and can make veridical judgments about their current state. Silencing demonstrates the tight coupling of motion and object appearance.