Investigating the replicability of preclinical cancer biology.

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.

Challenges for assessing replicability in preclinical cancer biology.

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.

Experiments from unfinished Registered Reports in the Reproducibility Project: Cancer Biology.

As part of the Reproducibility Project: Cancer Biology, we published Registered Reports that described how we intended to replicate selected experiments from 29 high-impact preclinical cancer biology papers published between 2010 and 2012. Replication experiments were completed and Replication Studies reporting the results were submitted for 18 papers, of which 17 were accepted and published by eLife with the rejected paper posted as a preprint. Here, we report the status and outcomes obtained for the remaining 11 papers. Four papers initiated experimental work but were stopped without any experimental outcomes. Two papers resulted in incomplete outcomes due to unanticipated challenges when conducting the experiments. For the remaining five papers only some of the experiments were completed with the other experiments incomplete due to mundane technical or unanticipated methodological challenges. The experiments from these papers, along with the other experiments attempted as part of the Reproducibility Project: Cancer Biology, provides evidence about the challenges of repeating preclinical cancer biology experiments and the replicability of the completed experiments.

Replication study: androgen receptor splice variants determine taxane sensitivity in prostate cancer.

In 2015, as part of the Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative, we published a Registered Report (Shan et al., 2015) that described how we intended to replicate selected experiments from the paper "Androgen Receptor Splice Variants Determine Taxane Sensitivity in Prostate Cancer" (Thadani-Mulero et al., 2014). Here we report the results of those experiments. Growth of tumor xenografts from two prostate cancer xenograft lines, LuCaP 86.2, which expresses wild-type androgen receptor (AR) and AR variant 567, and LuCaP 23.1, which expresses wild-type AR and AR variant 7, were not affected by docetaxel treatment. The LuCaP 23.1 tumor xenografts grew slower than in the original study. This result is different from the original study, which reported significant reduction of tumor growth in the LuCaP 86.2. Furthermore, we were unable to detect ARv7 in the LuCaP 23.1, although we used the antibody as stated in the original study and ensured that it was detecting ARv7 via a known positive control (22rv1, Hörnberg et al., 2011). Finally, we report a meta-analysis of the result.

Registered report: the androgen receptor induces a distinct transcriptional program in castration-resistant prostate cancer in man.

The Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative (PCFMFRI) seeks to address growing concerns about reproducibility in scientific research by conducting replications of recent papers in the field of prostate cancer. This Registered Report describes the proposed replication plan of key experiments from "The Androgen Receptor Induces a Distinct Transcriptional Program in Castration-Resistant Prostate Cancer in Man" by Sharma and colleagues (2013), published in Cancer Cell in 2013. Of thousands of targets for the androgen receptor (AR), the authors elucidated a subset of 16 core genes that were consistently downregulated with castration and re-emerged with castration resistance. These 16 AR binding sites were distinct from those observed in cells in culture. The authors suggested that cellular context can have dramatic effects on downstream transcriptional regulation of AR binding sites. The present study will attempt to replicate Fig. 7C by comparing gene expression of the 16 core genes identified by Sharma and colleagues in xenograft tumor tissue compared to androgen treated LNCaP cells in vitro. The Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative is a collaboration between the Prostate Cancer Foundation, the Movember Initiative, and Science Exchange, and the results of the replications will be published by PeerJ.

Registered report: androgen receptor splice variants determine taxane sensitivity in prostate cancer.

The Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative seeks to address growing concerns about reproducibility in scientific research by conducting replications of recent papers in the field of prostate cancer. This Registered Report describes the proposed replication plan of key experiments from "Androgen Receptor Splice Variants Determine Taxane Sensitivity in Prostate Cancer" by Thadani-Mulero and colleagues (2014) published in Cancer Research in 2014. The experiment that will be replicated is reported in Fig. 6A. Thadani-Mulero and colleagues generated xenografts from two prostate cancer cell lines; LuCaP 86.2, which expresses predominantly the ARv567 splice variant of the androgen receptor (AR), and LuCaP 23.1, which expresses the full length AR as well as the ARv7 variant. Treatment of the tumors with the taxane docetaxel showed that the drug inhibited tumor growth of the LuCaP 86.2 cells but not of the LuCaP 23.1 cells, indicating that expression of splice variants of the AR can affect sensitivity to docetaxel. The Prostate Cancer Foundation-Movember Foundation Reproducibility Initiative is a collaboration between the Prostate Cancer Foundation, the Movember Foundation and Science Exchange, and the results of the replications will be published by PeerJ.

Replication attempt: "Effect of BMAP-28 antimicrobial peptides on Leishmania major promastigote and amastigote growth: role of leishmanolysin in parasite survival".

This study describes an attempt to replicate experiments from the paper "Effect of BMAP-28 Antimicrobial Peptides on Leishmania major Promastigote and Amastigote Growth: Role of Leishmanolysin in Parasite Survival," which was submitted to the Reproducibility Initiative for independent validation. The cathelicidin bovine myeloid antimicrobial peptide 28 (BMAP-28) and its isomers were previously shown to have potent antiparasitic activity against Leishmania major. We tested the effectiveness of L-BMAP-28 and two of its isomers, the D-amino acid form (D-BMAP-28) and the retro-inverso form (RI-BMAP-28), in both unamidated and amidated forms, as anti-leishmanial agents against Leishmania major promastigotes in vitro. We observed that L-BMAP-28, as well as its D and RI isomers, demonstrate anti-leishmanial activity against L. major promastigotes in vitro. The inhibitory effect was lower than what was seen in the original study. At 2 µM of amidated peptides, the viability was 94%, 36%, and 66% with L-, D- and RI-peptides, versus 57%, 6%, and 18% in the original study.

An open investigation of the reproducibility of cancer biology research.

It is widely believed that research that builds upon previously published findings has reproduced the original work. However, it is rare for researchers to perform or publish direct replications of existing results. The Reproducibility Project: Cancer Biology is an open investigation of reproducibility in preclinical cancer biology research. We have identified 50 high impact cancer biology articles published in the period 2010-2012, and plan to replicate a subset of experimental results from each article. A Registered Report detailing the proposed experimental designs and protocols for each subset of experiments will be peer reviewed and published prior to data collection. The results of these experiments will then be published in a Replication Study. The resulting open methodology and dataset will provide evidence about the reproducibility of high-impact results, and an opportunity to identify predictors of reproducibility.

New forms of checks and balances are needed to improve research integrity.

Recent attempts at replicating highly-cited peer-reviewed studies demonstrate that the "reproducibility crisis" is indeed upon us. However, punitive measures against individuals committing research misconduct are neither sufficient nor useful because this is a systemic issue stemming from a lack of positive incentive. As an alternative approach, here we propose a system of checks and balances for the publishing process that involves 1) technical review of methodology by publishers, and 2) incentivizing direct replication of key experimental results. Together, these actions will help restore the self-correcting nature of scientific discovery.

Infiltrating S100A8+ myeloid cells promote metastatic spread of human breast cancer and predict poor clinical outcome.

The mechanisms by which breast cancer (BrC) can successfully metastasize are complex and not yet fully understood. Our goal was to identify tumor-induced stromal changes that influence metastatic cell behavior, and may serve as better targets for therapy. To identify stromal changes in cancer-bearing tissue, dual-species gene expression analysis was performed for three different metastatic BrC xenograft models. Results were confirmed by immunohistochemistry, flow cytometry, and protein knockdown. These results were validated in human clinical samples at the mRNA and protein level by retrospective analysis of cohorts of human BrC specimens. In pre-clinical models of BrC, systemic recruitment of S100A8+ myeloid cells-including myeloid-derived suppressor cells (MDSCs)-was promoted by tumor-derived factors. Recruitment of S100A8+ myeloid cells was diminished by inhibition of tumor-derived factors or depletion of MDSCs, resulting in fewer metastases and smaller primary tumors. Importantly, these MDSCs retain their ability to suppress T cell proliferation upon co-culture. Secretion of macrophage inhibitory factor (MIF) activated the recruitment of S100A8+ myeloid cells systemically. Inhibition of MIF, or depletion of MDSCs resulted in delayed tumor growth and lower metastatic burden. In human BrC specimens, increased mRNA and protein levels of S100A8+ infiltrating cells are highly associated with poor overall survival and shorter metastasis free survival of BrC patients, respectively. Furthermore, analysis of nine different human gene expression datasets confirms the association of increased levels of S100A8 transcripts with an increased risk of death. Recruitment of S100A8+ myeloid cells to primary tumors and secondary sites in xenograft models of BrC enhances cancer progression independent of their suppressive activity on T cells. In clinical samples, infiltrating S100A8+ cells are associated with poor overall survival. Targeting these molecules or associated pathways in cells of the tumor microenvironment may translate into novel therapeutic interventions and benefit patient outcome.

A new mouse model for the study of human breast cancer metastasis.

Breast cancer is the most common cancer in women, and this prevalence has a major impact on health worldwide. Localized breast cancer has an excellent prognosis, with a 5-year relative survival rate of 85%. However, the survival rate drops to only 23% for women with distant metastases. To date, the study of breast cancer metastasis has been hampered by a lack of reliable metastatic models. Here we describe a novel in vivo model using human breast cancer xenografts in NOD scid gamma (NSG) mice; in this model human breast cancer cells reliably metastasize to distant organs from primary tumors grown within the mammary fat pad. This model enables the study of the entire metastatic process from the proper anatomical site, providing an important new approach to examine the mechanisms underlying breast cancer metastasis. We used this model to identify gene expression changes that occur at metastatic sites relative to the primary mammary fat pad tumor. By comparing multiple metastatic sites and independent cell lines, we have identified several gene expression changes that may be important for tumor growth at distant sites.

PI3K independent activation of mTORC1 as a target in lapatinib-resistant ERBB2+ breast cancer cells.

Therapies targeting the ERBB2 receptor, including the kinase inhibitor lapatinib (Tykerb, GlaxoSmithKline), have improved clinical outcome for women with ERBB2-amplified breast cancer. However, acquired resistance to lapatinib remains a significant clinical problem, and the mechanisms governing resistance remain poorly understood. We sought to define molecular alterations that confer an acquired lapatinib resistance phenotype in ER-/ERBB2+ human breast cancer cells. ERBB2-amplified SKBR3 breast cancer cells were rendered resistant to lapatinib via culture in increasing concentrations of the drug, and molecular changes associated with a resistant phenotype were interrogated using a collaborative enzyme-enhanced immunoassay platform and immunoblotting techniques for detection of phosphorylated signaling cascade proteins. Interestingly, despite apparent inactivation of the PI3K/AKT signaling pathway, resistant cells exhibited constitutive activation of mammalian target of rapamycin complex 1 (mTORC1) and were highly sensitive to mTOR inhibition with rapamycin and the dual PI3K/mTOR inhibitor NVP-BEZ235. These data demonstrate a role for downstream activation of mTORC1 in the absence of molecular alterations leading to PI3K/AKT hyperactivation as a potential mechanism of lapatinib resistance in this model of ERBB2+ breast cancer and support the rationale of combination or sequential therapy using ERBB2 and mTOR-targeting molecules to prevent or target resistance to lapatinib. Moreover, our data suggest that assessment of mTOR substrate phosphorylation (i.e., S6) may serve as a more robust biomarker to predict sensitivity to mTOR inhibitors in the context of lapatinib resistance than PI3K mutations, loss of PTEN and p-AKT levels.

Whole genome in vivo RNAi screening identifies the leukemia inhibitory factor receptor as a novel breast tumor suppressor.

Cancer is caused by mutations in oncogenes and tumor suppressor genes, resulting in the deregulation of processes fundamental to the normal behavior of cells. The identification and characterization of oncogenes and tumor suppressors has led to new treatment strategies that have significantly improved cancer outcome. The advent of next generation sequencing has allowed the elucidation of the fine structure of cancer genomes, however, the identification of pathogenic changes is complicated by the inherent genomic instability of cancer cells. Therefore, functional approaches for the identification of novel genes involved in the initiation and development of tumors are critical. Here we report the first whole human genome in vivo RNA interference screen to identify functionally important tumor suppressor genes. Using our novel approach, we identify previously validated tumor suppressor genes including TP53 and MNT, as well as several novel candidate tumor suppressor genes including leukemia inhibitory factor receptor (LIFR). We show that LIFR is a key novel tumor suppressor, whose deregulation may drive the transformation of a significant proportion of human breast cancers. These results demonstrate the power of genome wide in vivo RNAi screens as a method for identifying novel genes regulating tumorigenesis.

Simultaneous analysis of tumor and stromal gene expression profiles from xenograft models.

Identifying the gene expression alterations that occur in both the tumor and stroma is essential to understanding tumor biology. We have developed a dual-species microarray analysis method that allows the dissection of both tumor and stromal gene expression profiles from xenograft models, based on limited interspecies cross-hybridization on Illumina gene expression beadchips. This methodology allows for simultaneous genome-wide analysis of gene expression profiles of both tumor cells and the associated stromal tissue.

Rescue of tropomyosin deficiency in Drosophila and human cancer cells by synaptopodin reveals a role of tropomyosin α in RhoA stabilization.

Tropomyosins are widespread actin-binding proteins that influence numerous cellular functions including actin dynamics, cell migration, tumour suppression, and Drosophila oocyte development. Synaptopodin is another actin-binding protein with a more restricted expression pattern in highly dynamic cell compartments such as kidney podocyte foot processes, where it promotes RhoA signalling by blocking the Smurf1-mediated ubiquitination of RhoA. Here, we show that synaptopodin has a shorter half-life but shares functional properties with the highly stable tropomyosin. Transgenic expression of synaptopodin restores oskar mRNA localization in Drosophila oocytes mutant for TmII, thereby rescuing germline differentiation and fertility. Synaptopodin restores stress fibres in tropomyosin-deficient human MDA-MB 231 breast cancer cells and TPMα-depleted fibroblasts. Gene silencing of TPMα but not TPMß causes loss of stress fibres by promoting Smurf1-mediated ubiquitination and proteasomal degradation of RhoA. Functionally, overexpression of synaptopodin or RhoA(K6,7R) significantly reduces MDA-MB 231 cell migration. Our findings elucidate RhoA stabilization by structurally unrelated actin-binding proteins as a conserved mechanism for regulation of stress fibre dynamics and cell motility in a cell type-specific fashion.

The role of SATB1 in breast cancer pathogenesis.

BACKGROUND: SATB1 has been previously proposed as a key protein that controls the development and progression of breast cancer. We explored the potential of the SATB1 protein as a therapeutic target and prognostic marker for human breast cancer. METHODS: We used aggressive (MDA-MB-231 and BT549) and nonaggressive (SKBR3 and MCF7) breast cancer cell lines to investigate the potential of SATB1 as a therapeutic target. SATB1 mRNA expression was silenced in aggressive cells by use of short hairpin RNAs against SATB1. SATB1 was overexpressed in nonaggressive cells by use of SATB1 expression vectors. We assessed the effect of modifying SATB1 expression on the transformed phenotype by examining anchorage-independent cell proliferation, acinar morphology on matrigel, and migration by wound healing in cultured cells. We examined tumor formation and metastasis, respectively, by use of orthotopic mammary fat pad and tail vein xenograft mouse models (mice were used in groups of six, and in total, 96 mice were used). SATB1 mRNA expression was compared with outcome for patients with primary breast cancer from six previous microarray studies that included a total of 1170 patients. All statistical tests were two-sided. RESULTS: The transformed phenotype was not suppressed by SATB1 silencing in aggressive cells and was not enhanced by ectopic expression of SATB1 in nonaggressive cells. Modifying SATB1 expression did not alter anchorage-independent cell proliferation, invasive acinar morphology, or cell migration in cultured cells and did not affect tumor formation or metastasis in xenograft mouse models. In addition, SATB1 expression was not associated with decreased overall survival of patients with primary breast cancer in six previous independent microarray studies (overall odds ratio = 0.80, 95% confidence interval = 0.62 to 1.03, P = .10). CONCLUSION: In contrast to previous studies, we found that SATB1 expression did not promote breast cancer progression and was not associated with breast cancer outcome.

FGFR1 amplification drives endocrine therapy resistance and is a therapeutic target in breast cancer.

Amplification of fibroblast growth factor receptor 1 (FGFR1) occurs in approximately 10% of breast cancers and is associated with poor prognosis. However, it is uncertain whether overexpression of FGFR1 is causally linked to the poor prognosis of amplified cancers. Here, we show that FGFR1 overexpression is robustly associated with FGFR1 amplification in two independent series of breast cancers. Breast cancer cell lines with FGFR1 overexpression and amplification show enhanced ligand-dependent signaling, with increased activation of the mitogen-activated protein kinase and phosphoinositide 3-kinase-AKT signaling pathways in response to FGF2, but also show basal ligand-independent signaling, and are dependent on FGFR signaling for anchorage-independent growth. FGFR1-amplified cell lines show resistance to 4-hydroxytamoxifen, which is reversed by small interfering RNA silencing of FGFR1, suggesting that FGFR1 overexpression also promotes endocrine therapy resistance. FGFR1 signaling suppresses progesterone receptor (PR) expression in vitro, and likewise, amplified cancers are frequently PR negative, identifying a potential biomarker for FGFR1 activity. Furthermore, we show that amplified cancers have a high proliferative rate assessed by Ki67 staining and that FGFR1 amplification is found in 16% to 27% of luminal B-type breast cancers. Our data suggest that amplification and overexpression of FGFR1 may be a major contributor to poor prognosis in luminal-type breast cancers, driving anchorage-independent proliferation and endocrine therapy resistance.

CRK7 modifies the MAPK pathway and influences the response to endocrine therapy.

Endocrine therapies, which inhibit estrogen receptor (ER)alpha signaling, are the most common and effective treatment for ERalpha-positive breast cancer. However, the use of these agents is limited by the frequent development of resistance. The cyclin-dependent kinase family member CRK7 (aka CRKRS) was identified from an RNA interference screen for modifiers of tamoxifen sensitivity. Here, we demonstrate that silencing of CRK7 not only causes resistance to tamoxifen but also leads to resistance to additional endocrine therapies including ICI 182780 and estrogen deprivation, a model of aromatase inhibition. We show that CRK7 silencing activates the mitogen-activated protein kinase (MAPK)-signaling pathway, which causes a loss of ER dependence, resulting in endocrine therapy resistance. This study identifies a novel role for CRK7 in MAPK regulation and resistance to estrogen signaling inhibitors.