Predicted as observed? How to identify empirically adequate theoretical constructs.

The identification of an empirically adequate theoretical construct requires determining whether a theoretically predicted effect is sufficiently similar to an observed effect. To this end, we propose a simple similarity measure, describe its application in different research designs, and use computer simulations to estimate the necessary sample size for a given observed effect. As our main example, we apply this measure to recent meta-analytical research on precognition. Results suggest that the evidential basis is too weak for a predicted precognition effect of d = 0.20 to be considered empirically adequate. As additional examples, we apply this measure to object-level experimental data from dissonance theory and a recent crowdsourcing hypothesis test, as well as to meta-analytical data on the correlation of personality traits and life outcomes.

A New Empirical Approach to Intercultural Comparisons of Value Preferences Based on Schwartz's Theory.

Empirical tests of Schwartz's theory of culture-level value priorities have predominantly been performed using an averaging approach-as values of the average individual in a culture. However, from a theory of measurement standpoint such an approach seems inadequate. We argue that the averaging approach is an insufficiently accurate methodology in capturing the compatibilities-incompatibilities between values of individuals within cultures. We propose an approach based on the distribution of values of individuals in a given culture-the distribution approach. Using data from two rounds of the European Social Survey, we show how frequencies of specific individual value priorities in a culture can be used toward the description of culture-level value preferences. We recommend a re-conceptualization of Schwartz's culture-level value theory to an orthogonal two-dimensional structure, namely as Alteration vs. Preservation and Amenability vs. Dominance, which we explain based on heterogeneity in socioecological indicators across countries. We conclude that societal challenges may influence the cultural value climate across countries.

Hypothesis-Testing Demands Trustworthy Data-A Simulation Approach to Inferential Statistics Advocating the Research Program Strategy.

In psychology as elsewhere, the main statistical inference strategy to establish empirical effects is null-hypothesis significance testing (NHST). The recent failure to replicate allegedly well-established NHST-results, however, implies that such results lack sufficient statistical power, and thus feature unacceptably high error-rates. Using data-simulation to estimate the error-rates of NHST-results, we advocate the research program strategy (RPS) as a superior methodology. RPS integrates Frequentist with Bayesian inference elements, and leads from a preliminary discovery against a (random) H0-hypothesis to a statistical H1-verification. Not only do RPS-results feature significantly lower error-rates than NHST-results, RPS also addresses key-deficits of a "pure" Frequentist and a standard Bayesian approach. In particular, RPS aggregates underpowered results safely. RPS therefore provides a tool to regain the trust the discipline had lost during the ongoing replicability-crisis.

Data replication matters to an underpowered study, but replicated hypothesis corroboration counts.

Before replication becomes mainstream, the potential for generating theoretical knowledge better be clear. Replicating statistically significant nonrandom data shows that an original study made a discovery; replicating a specified theoretical effect shows that an original study corroborated a theory. Yet only in the latter case is replication a necessary, sound, and worthwhile strategy.

From Discovery to Justification: Outline of an Ideal Research Program in Empirical Psychology.

The gold standard for an empirical science is the replicability of its research results. But the estimated average replicability rate of key-effects that top-tier psychology journals report falls between 36 and 39% (objective vs. subjective rate; Open Science Collaboration, 2015). So the standard mode of applying null-hypothesis significance testing (NHST) fails to adequately separate stable from random effects. Therefore, NHST does not fully convince as a statistical inference strategy. We argue that the replicability crisis is "home-made" because more sophisticated strategies can deliver results the successful replication of which is sufficiently probable. Thus, we can overcome the replicability crisis by integrating empirical results into genuine research programs. Instead of continuing to narrowly evaluate only the stability of data against random fluctuations (discovery context), such programs evaluate rival hypotheses against stable data (justification context).