How failure to falsify in high-volume science contributes to the replication crisis.

The number of scientific papers published every year continues to increase, but scientific knowledge is not progressing at the same rate. Here we argue that a greater emphasis on falsification - the direct testing of strong hypotheses - would lead to faster progress by allowing well-specified hypotheses to be eliminated. We describe an example from neuroscience where there has been little work to directly test two prominent but incompatible hypotheses related to traumatic brain injury. Based on this example, we discuss how building strong hypotheses and then setting out to falsify them can bring greater precision to the clinical neurosciences, and argue that this approach could be beneficial to all areas of science.

Investigating the replicability of preclinical cancer biology.

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.

Challenges for assessing replicability in preclinical cancer biology.

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.

Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology.

As part of the Reproducibility Project: Cancer Biology, we published Registered Reports that described how we intended to replicate selected experiments from 29 high-impact preclinical cancer biology papers published between 2010 and 2012. Replication experiments were completed and Replication Studies reporting the results were submitted for 18 papers, of which 17 were accepted and published by eLife with the rejected paper posted as a preprint. Here, we report the status and outcomes obtained for the remaining 11 papers. Four papers initiated experimental work but were stopped without any experimental outcomes. Two papers resulted in incomplete outcomes due to unanticipated challenges when conducting the experiments. For the remaining five papers only some of the experiments were completed with the other experiments incomplete due to mundane technical or unanticipated methodological challenges. The experiments from these papers, along with the other experiments attempted as part of the Reproducibility Project: Cancer Biology, provides evidence about the challenges of repeating preclinical cancer biology experiments and the replicability of the completed experiments.

Initial evidence of research quality of registered reports compared with the standard publishing model.

In registered reports (RRs), initial peer review and in-principle acceptance occur before knowing the research outcomes. This combats publication bias and distinguishes planned from unplanned research. How RRs could improve the credibility of research findings is straightforward, but there is little empirical evidence. Also, there could be unintended costs such as reducing novelty. Here, 353 researchers peer reviewed a pair of papers from 29 published RRs from psychology and neuroscience and 57 non-RR comparison papers. RRs numerically outperformed comparison papers on all 19 criteria (mean difference 0.46, scale range -4 to +4) with effects ranging from RRs being statistically indistinguishable from comparison papers in novelty (0.13, 95% credible interval [-0.24, 0.49]) and creativity (0.22, [-0.14, 0.58]) to sizeable improvements in rigour of methodology (0.99, [0.62, 1.35]) and analysis (0.97, [0.60, 1.34]) and overall paper quality (0.66, [0.30, 1.02]). RRs could improve research quality while reducing publication bias and ultimately improve the credibility of the published literature.

Credibility of preprints: an interdisciplinary survey of researchers.

Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.

What is replication?

Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study's procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect. We propose that replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research. This definition reduces emphasis on operational characteristics of the study and increases emphasis on the interpretation of possible outcomes. The purpose of replication is to advance theory by confronting existing understanding with new evidence. Ironically, the value of replication may be strongest when existing understanding is weakest. Successful replication provides evidence of generalizability across the conditions that inevitably differ from the original study; Unsuccessful replication indicates that the reliability of the finding may be more constrained than recognized previously. Defining replication as a confrontation of current theoretical expectations clarifies its important, exciting, and generative role in scientific progress.

A randomized trial of a lab-embedded discourse intervention to improve research ethics.

We report a randomized trial of a research ethics training intervention designed to enhance ethics communication in university science and engineering laboratories, focusing specifically on authorship and data management. The intervention is a project-based research ethics curriculum that was designed to enhance the ability of science and engineering research laboratory members to engage in reason giving and interpersonal communication necessary for ethical practice. The randomized trial was fielded in active faculty-led laboratories at two US research-intensive institutions. Here, we show that laboratory members perceived improvements in the quality of discourse on research ethics within their laboratories and enhanced awareness of the relevance and reasons for that discourse for their work as measured by a survey administered over 4 mo after the intervention. This training represents a paradigm shift compared with more typical module-based or classroom ethics instruction that is divorced from the everyday workflow and practices within laboratories and is designed to cultivate a campus culture of ethical science and engineering research in the very work settings where laboratory members interact.

Scientific Utopia III: Crowdsourcing Science.

Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.

Making sense of replications.

The first results from the Reproducibility Project: Cancer Biology suggest that there is scope for improving reproducibility in pre-clinical cancer research.

Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency.

Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.

An open investigation of the reproducibility of cancer biology research.

It is widely believed that research that builds upon previously published findings has reproduced the original work. However, it is rare for researchers to perform or publish direct replications of existing results. The Reproducibility Project: Cancer Biology is an open investigation of reproducibility in preclinical cancer biology research. We have identified 50 high impact cancer biology articles published in the period 2010-2012, and plan to replicate a subset of experimental results from each article. A Registered Report detailing the proposed experimental designs and protocols for each subset of experiments will be peer reviewed and published prior to data collection. The results of these experiments will then be published in a Replication Study. The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility.

Depletion of the adaptor protein NCK increases UV-induced p53 phosphorylation and promotes apoptosis.

The cellular response to DNA damage requires the coordination of many proteins involved in diverse molecular processes. Discrete molecular pathways are becoming increasingly well understood, but the interconnectivity and coordination of multiple pathways remains less clear. We now show that NCK, an adapter protein involved in cytoskeletal responses to tyrosine kinase receptor signaling, accumulates in the nucleus in response to DNA damage and this translocation can be blocked by specific inhibition of the ATR protein kinase. Strikingly, HeLa cells depleted of NCK undergo apoptosis shortly after UV irradiation, as monitored by caspase-3 cleavage and PARP cleavage. This rapid, hyperactive apoptosis in NCK depleted cells might be p53 dependent, because loss of NCK also increased UV-induced p53 phosphorylation. Importantly, depletion of SOCS7, which is necessary for NCK nuclear translocation, phenocopies NCK depletion, indicating the nuclear accumulation of NCK is responsible for these molecular events. There are two NCK isoforms that have mostly redundant functions, and although NCK2 appears to have a greater contribution, depletion of NCK1 or NCK2, led to increased p53 phosphorylation and early apoptosis after UV exposure. These data reveal a novel function for NCK in regulating p53 phosphorylation and apoptosis, and provide evidence for interconnectedness of growth factor signaling proteins and the DNA damage response.

Disease-associated human telomerase RNA variants show loss of function for telomere synthesis without dominant-negative interference.

Telomerase adds simple-sequence repeats to chromosome ends to offset the terminal sequence loss inherent in each cycle of genome replication. Inherited mutations in genes encoding subunits of the human telomerase holoenzyme give rise to disease phenotypes including hematopoietic failure and pulmonary fibrosis. Disease-associated variants of the human telomerase RNA are expressed in heterozygous combination with wild-type telomerase RNA. Here, we exploit a sensitized human primary cell assay system to investigate the biological function of disease-linked telomerase RNA variants and their impact on the function of coexpressed wild-type telomerase RNA. We find that telomerase RNA variants discovered in patients with dyskeratosis congenita or aplastic anemia show loss of function without any indication of dominant-negative impact on telomere maintenance by the coexpressed wild-type RNA. To reconcile this result with contradictory findings from reconstitution assays in vitro, we demonstrate that the lack of dominant-negative impact on telomere maintenance correlates with physiological assembly of active human telomerase holoenzyme ribonucleoproteins harboring monomers rather than higher-order multimers of telomerase RNA and telomerase reverse transcriptase. These findings support loss of function of telomerase RNA as a general mechanism of human disease.

Identification of the first specific inhibitor of p90 ribosomal S6 kinase (RSK) reveals an unexpected role for RSK in cancer cell proliferation.

p90 ribosomal S6 kinase (RSK) is an important downstream effector of mitogen-activated protein kinase, but its biological functions are not well understood. We have now identified the first small-molecule, RSK-specific inhibitor, which we isolated from the tropical plant Forsteronia refracta. We have named this novel inhibitor SL0101. SL0101 shows remarkable specificity for RSK. The major determinant of SL0101-binding specificity is the unique ATP-interacting sequence in the amino-terminal kinase domain of RSK. SL0101 inhibits proliferation of the human breast cancer cell line MCF-7, producing a cell cycle block in G(1) phase with an efficacy paralleling its ability to inhibit RSK in intact cells. RNA interference of RSK expression confirmed that RSK regulates MCF-7 proliferation. Interestingly, SL0101 does not alter proliferation of a normal human breast cell line MCF-10A, although SL0101 inhibits RSK in these cells. We show that RSK is overexpressed in approximately 50% of human breast cancer tissue samples, suggesting that regulation of RSK has been compromised. Thus, we show that RSK has an unexpected role in proliferation of transformed cells and may be a useful new target for chemotherapeutic agents. SL0101 will provide a powerful new tool to dissect the molecular functions of RSK in cancer cells.

Extracellular signal-regulated kinase 7, a regulator of hormone-dependent estrogen receptor destruction.

Estrogen receptor alpha (ER alpha) degradation is regulated by ubiquitination, but the signaling pathways that modulate ER alpha turnover are unknown. We found that extracellular signal-regulated kinase 7 (ERK7) preferentially enhances the destruction of ER alpha but not the related androgen receptor. Loss of ERK7 was correlated with breast cancer progression, and all ER alpha-positive breast tumors had decreased ERK7 expression compared to that found in normal breast tissue. In human breast cells, a dominant-negative ERK7 mutant decreased the rate of endogenous ER alpha degradation >4-fold in the presence of hormone and potentiated estrogen responsiveness. ERK7 targets the ER alpha ligand-binding domain for destruction by enhancing its ubiquitination. Thus, ERK7 is a novel regulator of estrogen responsiveness through its control of ER alpha turnover.