Comparing the prevalence of statistical reporting inconsistencies in COVID-19 preprints and matched controls: a registered report.

The COVID-19 outbreak has led to an exponential increase of publications and preprints about the virus, its causes, consequences, and possible cures. COVID-19 research has been conducted under high time pressure and has been subject to financial and societal interests. Doing research under such pressure may influence the scrutiny with which researchers perform and write up their studies. Either researchers become more diligent, because of the high-stakes nature of the research, or the time pressure may lead to cutting corners and lower quality output. In this study, we conducted a natural experiment to compare the prevalence of incorrectly reported statistics in a stratified random sample of COVID-19 preprints and a matched sample of non-COVID-19 preprints. Our results show that the overall prevalence of incorrectly reported statistics is 9-10%, but frequentist as well as Bayesian hypothesis tests show no difference in the number of statistical inconsistencies between COVID-19 and non-COVID-19 preprints. In conclusion, the literature suggests that COVID-19 research may on average have more methodological problems than non-COVID-19 research, but our results show that there is no difference in the statistical reporting quality.

Are Speeded Tests Unfair? Modeling the Impact of Time Limits on the Gender Gap in Mathematics.

When cognitive and educational tests are administered under time limits, tests may become speeded and this may affect the reliability and validity of the resulting test scores. Prior research has shown that time limits may create or enlarge gender gaps in cognitive and academic testing. On average, women complete fewer items than men when a test is administered with a strict time limit, whereas gender gaps are frequently reduced when time limits are relaxed. In this study, we propose that gender differences in test strategy might inflate gender gaps favoring men, and relate test strategy to stereotype threat effects under which women underperform due to the pressure of negative stereotypes about their performance. First, we applied a Bayesian two-dimensional item response theory (IRT) model to data obtained from two registered reports that investigated stereotype threat in mathematics, and estimated the latent correlation between underlying test strategy (here, completion factor, a proxy for working speed) and mathematics ability. Second, we tested the gender gap and assessed potential effects of stereotype threat on female test performance. We found a positive correlation between the completion factor and mathematics ability, such that more able participants dropped out later in the test. We did not observe a stereotype threat effect but found larger gender differences on the latent completion factor than on latent mathematical ability, suggesting that test strategies affect the gender gap in timed mathematics performance. We argue that if the effect of time limits on tests is not taken into account, this may lead to test unfairness and biased group comparisons, and urge researchers to consider these effects in either their analyses or study planning.

The potential pitfalls of studying adult sex ratios at aggregate levels in humans.

Human adult sex ratios have been studied extensively across the biological and social sciences. While several studies have examined adult sex ratio effects in a multilevel perspective, many studies have focused on effects at an aggregated level only. In this paper, we review some key issues relating to such analyses. We address not only nation-level analyses, but also aggregation at lower levels, to investigate whether these issues extend to lower levels of aggregation. We illustrate these issues with novel databases covering a broad range of variables. Specifically, we discuss distributional issues with aggregated measures of adult sex ratio, significance testing, and statistical non-independence when using aggregate data. Firstly, we show that there are severe distributional issues with national adult sex ratio, such as extreme cases. Secondly, we demonstrate that many 'meaningless' variables are significantly correlated with adult sex ratio (e.g. the max. elevation level correlates with sex ratio at US state level). Finally, we re-examine associations between adult sex ratios and teenage fertility and find no robust evidence for an association at the aggregate level. Our review highlights the potential issues of using aggregate data on adult sex ratios to test hypotheses from an evolutionary perspective in humans.This article is part of the themed issue 'Adult sex ratios and reproductive decisions: a critical re-examination of sex differences in human and animal societies'.