Forecasting the publication and citation outcomes of COVID-19 preprints.

Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.

Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme.

The Defense Advanced Research Projects Agency (DARPA) programme 'Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.

Use of the visual range of detection to estimate effective sweep width for land search and rescue based on 10 detection experiments in North America.

OBJECTIVE: Standard-of-practice search management requires that the probability of detection (POD) be determined for each search resource after a task. To calculate the POD, a detection index (W) is obtained by field experiments. Because of the complexities of the land environment, search planners need a way to estimate the value of W without conducting formal experiments. We demonstrate a robust empirical correlation between detection range (Rd) and W, and argue that Rd may reliably be used as a quick field estimate for W. METHODS: We obtained the average maximum detection range (AMDR), Rd, and W values from 10 detection experiments conducted throughout North America. We measured the correlation between Rd and W, and tested whether the apparent relationship between W and Rd was statistically significant. RESULTS: On average we found W ≈ 1.645 × Rd with a strong correlation (R(2) = .827). The high-visibility class had W ≈ 1.773 × Rd (also R(2) = .867), the medium-visibility class had W ≈ 1.556 × Rd (R(2) = .560), and the low-visibility had a correction factor of 1.135 (R(2) = .319) for Rd to W. Using analysis of variance and post hoc testing, only the high- and low-visibility classes were significantly different from each other (P < .01). We also found a high correlation between the AMDR and Rd (R(2) = .9974). CONCLUSIONS: Although additional experiments are required for the medium- and low-visibility search objects and in the dry-domain ecoregion, we suggest search planners use the following correction factors to convert field-measured Rd to an estimate of the effective sweep width (W): high-visibility W = 1.8 × Rd; medium-visibility W = 1.6 × Rd; and low-visibility W = 1.1 × Rd.

Expert status and performance.

Expert judgements are essential when time and resources are stretched or we face novel dilemmas requiring fast solutions. Good advice can save lives and large sums of money. Typically, experts are defined by their qualifications, track record and experience. The social expectation hypothesis argues that more highly regarded and more experienced experts will give better advice. We asked experts to predict how they will perform, and how their peers will perform, on sets of questions. The results indicate that the way experts regard each other is consistent, but unfortunately, ranks are a poor guide to actual performance. Expert advice will be more accurate if technical decisions routinely use broadly-defined expert groups, structured question protocols and feedback.

Causation, causal perception, and conservation laws.

We investigated the perception of causation via the ability to detect conservation violations in simple events. We showed that observers were sensitive to energy conservation violations in free-fall events. Furthermore, observers were sensitive to gradually perturbed energy dynamics in such events. However, they were more sensitive to the effect of decreasing gravity than to that of increasing gravity. Displays with decreasing gravity were the only displays in which the energy profile was dominated by (apparent) potential energy, leading to an asymmetric trajectory.