A pre-registered, multi-lab non-replication of the action-sentence compatibility effect (ACE).

The Action-sentence Compatibility Effect (ACE) is a well-known demonstration of the role of motor activity in the comprehension of language. Participants are asked to make sensibility judgments on sentences by producing movements toward the body or away from the body. The ACE is the finding that movements are faster when the direction of the movement (e.g., toward) matches the direction of the action in the to-be-judged sentence (e.g., Art gave you the pen describes action toward you). We report on a pre-registered, multi-lab replication of one version of the ACE. The results show that none of the 18 labs involved in the study observed a reliable ACE, and that the meta-analytic estimate of the size of the ACE was essentially zero.

The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.

We investigated the reproducibility of the major statistical conclusions drawn in 46 articles published in 2012 in three APA journals. After having identified 232 key statistical claims, we tried to reproduce, for each claim, the test statistic, its degrees of freedom, and the corresponding p value, starting from the raw data that were provided by the authors and closely following the Method section in the article. Out of the 232 claims, we were able to successfully reproduce 163 (70%), 18 of which only by deviating from the article's analytical description. Thirteen (7%) of the 185 claims deemed significant by the authors are no longer so. The reproduction successes were often the result of cumbersome and time-consuming trial-and-error work, suggesting that APA style reporting in conjunction with raw data makes numerical verification at least hard, if not impossible. This article discusses the types of mistakes we could identify and the tediousness of our reproduction efforts in the light of a newly developed taxonomy for reproducibility. We then link our findings with other findings of empirical research on this topic, give practical recommendations on how to achieve reproducibility, and discuss the challenges of large-scale reproducibility checks as well as promising ideas that could considerably increase the reproducibility of psychological research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Depressive Symptoms and Category Learning: A Preregistered Conceptual Replication Study.

We present a fully preregistered, high-powered conceptual replication of Experiment 1 by Smith, Tracy, and Murray (1993). They observed a cognitive deficit in people with elevated depressive symptoms in a task requiring flexible analytic processing and deliberate hypothesis testing, but no deficit in a task assumed to require more automatic, holistic processing. Specifically, they found that individuals with depressive symptoms showed impaired performance on a criterial-attribute classification task, requiring flexible analysis of the attributes and deliberate hypothesis testing, but not on a family-resemblance classification task, assumed to rely on holistic processing. While deficits in tasks requiring flexible hypothesis testing are commonly observed in people diagnosed with a major depressive disorder, these deficits are much less commonly observed in people with merely elevated depressive symptoms, and therefore Smith et al.'s (1993) finding deserves further scrutiny. We observed no deficit in performance on the criterial-attribute task in people with above average depressive symptoms. Rather, we found a similar difference in performance on the criterial-attribute versus family-resemblance task between people with high and low depressive symptoms. The absence of a deficit in people with elevated depressive symptoms is consistent with previous findings focusing on different tasks.

Using parameter space partitioning to evaluate a model's qualitative fit.

Parameter space partitioning (PSP) is a versatile tool for model analysis that detects the qualitatively distinctive data patterns a model can generate, and partitions a model's parameter space into regions corresponding to these patterns. In this paper, we propose a PSP fit measure that summarizes the outcome of a PSP analysis into a single number, which can be used for model selection. In contrast to traditional model selection methods, PSP-based model selection focuses on qualitative data. We demonstrate PSP-based model selection by use of application examples in the area of category learning. A large-scale model recovery study reveals excellent recovery properties, suggesting that PSP fit is useful for model selection.

Increasing Transparency Through a Multiverse Analysis.

Empirical research inevitably includes constructing a data set by processing raw data into a form ready for statistical analysis. Data processing often involves choices among several reasonable options for excluding, transforming, and coding data. We suggest that instead of performing only one analysis, researchers could perform a multiverse analysis, which involves performing all analyses across the whole set of alternatively processed data sets corresponding to a large set of reasonable scenarios. Using an example focusing on the effect of fertility on religiosity and political attitudes, we show that analyzing a single data set can be misleading and propose a multiverse analysis as an alternative practice. A multiverse analysis offers an idea of how much the conclusions change because of arbitrary choices in data construction and gives pointers as to which choices are most consequential in the fragility of the result.

Corrigendum: Measuring the crowd within again: a pre-registered replication study.

[This corrects the article on p. 786 in vol. 5, PMID: 25120505.].

Outcome probability modulates anticipatory behavior to signals that are equally reliable.

A stimulus is a reliable signal of an outcome when the probability that the outcome occurs in its presence is different from in its absence. Reliable signals of important outcomes are responsible for triggering critical anticipatory or preparatory behavior, which is any form of behavior that prepares the organism to receive a biologically significant event. Previous research has shown that humans and other animals prepare more for outcomes that occur in the presence of highly reliable (i.e., highly contingent) signals, that is, those for which that difference is larger. However, it seems reasonable to expect that, all other things being equal, the probability with which the outcome follows the signal should also affect preparatory behavior. In the present experiment with humans, we used two signals. They were differentially followed by the outcome, but they were equally (and relatively weakly) reliable. The dependent variable was preparatory behavior in a Martians video game. Participants prepared more for the outcome (a Martians' invasion) when the outcome was most probable. These results indicate that the probability of the outcome can bias preparatory behavior to occur with different intensities despite identical outcome signaling.

Measuring the crowd within again: a pre-registered replication study.

According to the crowd within effect, the average of two estimates from one person tends to be more accurate than a single estimate of that person. The effect implies that the well documented wisdom of the crowd effect-the crowd's average estimate tends to be more accurate than the individual estimates-can be obtained within a single individual. In this paper, we performed a high-powered, pre-registered replication study of the original experiment. Our replication results are evaluated with the traditional null hypothesis significance testing approach, as well as with effect sizes and their confidence intervals. We adopted a co-pilot approach, in the sense that all analyses were performed independently by two researchers using different analysis software. Moreover, we report Bayes factors for all tests. We successfully replicated the crowd within effect, both when the second guess was made immediately after the first guess, as well as when it was made 3 weeks later. The experimental protocol, the raw data, the post-processed data and the analysis code are available online.

Belief inhibition in children's reasoning: memory-based evidence.

Adult reasoning has been shown as mediated by the inhibition of intuitive beliefs that are in conflict with logic. The current study introduces a classic procedure from the memory field to investigate belief inhibition in 12- to 17-year-old reasoners. A lexical decision task was used to probe the memory accessibility of beliefs that were cued during thinking on syllogistic reasoning problems. Results indicated an impaired memory access for words associated with misleading beliefs that were cued during reasoning if syllogisms had been solved correctly. This finding supports the claim that even for younger reasoners, correct reasoning is mediated by inhibitory processing as soon as intuitive beliefs conflict with logical considerations.