Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Data from a pre-publication independent replication initiative examining ten moral judgement effects.

We present the data from a crowdsourced project seeking to replicate findings in independent laboratories before (rather than after) they are published. In this Pre-Publication Independent Replication (PPIR) initiative, 25 research groups attempted to replicate 10 moral judgment effects from a single laboratory's research pipeline of unpublished findings. The 10 effects were investigated using online/lab surveys containing psychological manipulations (vignettes) followed by questionnaires. Results revealed a mix of reliable, unreliable, and culturally moderated findings. Unlike any previous replication project, this dataset includes the data from not only the replications but also from the original studies, creating a unique corpus that researchers can use to better understand reproducibility and irreproducibility in science.

Why Do People Tend to Infer "Ought" From "Is"? The Role of Biases in Explanation.

People tend to judge what is typical as also good and appropriate-as what ought to be. What accounts for the prevalence of these judgments, given that their validity is at best uncertain? We hypothesized that the tendency to reason from "is" to "ought" is due in part to a systematic bias in people's (nonmoral) explanations, whereby regularities (e.g., giving roses on Valentine's Day) are explained predominantly via inherent or intrinsic facts (e.g., roses are beautiful). In turn, these inherence-biased explanations lead to value-laden downstream conclusions (e.g., it is good to give roses). Consistent with this proposal, results from five studies (N = 629 children and adults) suggested that, from an early age, the bias toward inherence in explanations fosters inferences that imbue observed reality with value. Given that explanations fundamentally determine how people understand the world, the bias toward inherence in these judgments is likely to exert substantial influence over sociomoral understanding.

Cultural transmission of social essentialism.

Social essentialism entails the belief that certain social categories (e.g., gender, race) mark fundamentally distinct kinds of people. Essentialist beliefs have pernicious consequences, supporting social stereotyping and contributing to prejudice. How does social essentialism develop? In the studies reported here, we tested the hypothesis that generic language facilitates the cultural transmission of social essentialism. Two studies found that hearing generic language about a novel social category diverse for race, ethnicity, age, and sex led 4-y-olds and adults to develop essentialist beliefs about that social category. A third study documented that experimentally inducing parents to hold essentialist beliefs about a novel social category led them to produce more generic language when discussing the category with their children. Thus, generic language facilitates the transmission of essentialist beliefs about social categories from parents to children.