Planning Split-Apheresis Designs for Demonstrating Comparability of Cellular and Gene Therapy Products.

A recent FDA draft guidance discusses statistical considerations for demonstrating comparability of cell and gene therapy products and processes. One experimental study described in the guidance is the split-apheresis design. The FDA draft guidance recommends a paired data analysis for such a design. This paper demonstrates that the paired analysis is under powered for some quality attributes for practical sample sizes of three to five donors unless a significant portion of variability is attributed to donor. Addition of historic lots from the pre-change process can increase the power for these attributes. This paper provides appropriate statistical methods for including this information.

Equivalence Testing Based Fit Index: Standardized Root Mean Squared Residual.

A popular measure of model fit in structural equation modeling (SEM) is the standardized root mean squared residual (SRMR) fit index. Equivalence testing has been used to evaluate model fit in structural equation modeling (SEM) but has yet to be applied to SRMR. Accordingly, the present study proposed equivalence-testing based fit tests for the SRMR (ESRMR). Several variations of ESRMR were introduced, incorporating different equivalence bounds and methods of computing confidence intervals. A Monte Carlo simulation study compared these novel tests with traditional methods for evaluating model fit. The results demonstrated that certain ESRMR tests based on an analytic computation of the confidence interval correctly reject poor-fitting models and are well-powered for detecting good-fitting models. We also present an illustrative example with real data to demonstrate how ESRMR may be incorporated into model fit evaluation and reporting. Our recommendation is that ESRMR tests be presented in addition to descriptive fit indices for model fit reporting in SEM.

Replication of null results: Absence of evidence or evidence of absence?

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a 'replication success.' Here, we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and 'replication success' can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with 'null results' are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.

Statistical Testing for Protein Equivalence Identifies Core Functional Modules Conserved across 360 Cancer Cell Lines and Presents a General Approach to Investigating Biological Systems.

Quantitative proteomics has enhanced our capability to study protein dynamics and their involvement in disease using various techniques, including statistical testing, to discern the significant differences between conditions. While most focus is on what is different between conditions, exploring similarities can provide valuable insights. However, exploring similarities directly from the analyte level, such as proteins, genes, or metabolites, is not a standard practice and is not widely adopted. In this study, we propose a statistical framework called QuEStVar (Quantitative Exploration of Stability and Variability through statistical hypothesis testing), enabling the exploration of quantitative stability and variability of features with a combined statistical framework. QuEStVar utilizes differential and equivalence testing to expand statistical classifications of analytes when comparing conditions. We applied our method to an extensive data set of cancer cell lines and revealed a quantitatively stable core proteome across diverse tissues and cancer subtypes. The functional analysis of this set of proteins highlighted the molecular mechanism of cancer cells to maintain constant conditions of the tumorigenic environment via biological processes, including transcription, translation, and nucleocytoplasmic transport.

[Comparison of DNA Extraction Methods for Processed Foods Containing Soybean or Maize].

Processed foods containing soybean or maize are subject to labeling regulations pertinent to genetically modified (GM) foods in Japan. To confirm the reliability of the labeling procedure of GM foods, the Japanese standard analytical methods (standard methods) using real-time PCR technique have been established. Although certain DNA extraction protocols are stipulated as standard in these methods, the use of other protocols confirmed to be equivalent to the existing ones was permitted. In this study, the equivalence testing of the techniques employed for DNA extraction from processed foods containing soybean or corn was conducted. In this study, the equivalence testing of the techniques employed for DNA extraction from processed foods containing soybean or maize was conducted. The silica membrane-based DNA extraction kits, GM quicker 4 and DNeasy Plant Maxi Kit (Maxi Kit), as an existing method were compared. GM quicker 4 was considered to be equivalent to or better than Maxi Kit.

Evaluating the performance of existing and novel equivalence tests for fit indices in structural equation modelling.

It has been suggested that equivalence testing (otherwise known as negligible effect testing) should be used to evaluate model fit within structural equation modelling (SEM). In this study, we propose novel variations of equivalence tests based on the popular root mean squared error of approximation and comparative fit index fit indices. Using Monte Carlo simulations, we compare the performance of these novel tests to other existing equivalence testing-based fit indices in SEM, as well as to other methods commonly used to evaluate model fit. Results indicate that equivalence tests in SEM have good Type I error control and display considerable power for detecting well-fitting models in medium to large sample sizes. At small sample sizes, relative to traditional fit indices, equivalence tests limit the chance of supporting a poorly fitting model. We also present an illustrative example to demonstrate how equivalence tests may be incorporated in model fit reporting. Equivalence tests in SEM also have unique interpretational advantages compared to other methods of model fit evaluation. We recommend that equivalence tests be utilized in conjunction with descriptive fit indices to provide more evidence when evaluating model fit.

Null regions: a unified conceptual framework for statistical inference.

Ruling out the possibility that there is absolutely no effect or association between variables may be a good first step, but it is rarely the ultimate goal of science. Yet that is the only inference provided by traditional null hypothesis significance testing (NHST), which has been a mainstay of many scientific fields. Reliance on NHST also makes it difficult to define what it means to replicate a finding, and leads to an uncomfortable quandary in which increasing precision in data reduces researchers' ability to perform theory falsification. To solve these problems, in recent years several alternatives to traditional NHST have been proposed. However, each new test is described using its own terminology and practiced in different fields. We describe a simple, unified framework for conceptualizing all these tests so that it is not necessary to learn them separately. Moreover, the framework allows researchers to conduct any of these tests by asking just one question: is the confidence interval entirely outside the null region(s)? This framework may also help researchers choose the test(s) that best answers their research question when simply ruling out 'no effect at all' is not enough.

Improved family-wise error rate control in multiple equivalence testing.

Equivalence testing is an important component of safety assessments, used for example by the European Food Safety Authority, to allow new food or feed products on the market. The aim of such tests is to demonstrate equivalence of characteristics of test and reference crops. Equivalence tests are typically univariate and applied to each measured analyte (characteristic) separately without multiplicity correction. This increases the probability of making false claims of equivalence (type I errors) when evaluating multiple analytes simultaneously. To solve this problem, familywise error rate (FWER) control using Hochberg's method has been proposed. This paper demonstrates that, in the context of equivalence testing, other FWER-controlling methods are more powerful than Hochberg's. Particularly, it is shown that Hommel's method is guaranteed to perform at least as well as Hochberg's and that an "adaptive" version of Bonferroni's method, which uses an estimator of the proportion of non-equivalent characteristics, often substantially outperforms Hommel's method. Adaptive Bonferroni takes better advantage of the particular context of food safety where a large proportion of true equivalences is expected, a situation where other methods are particularly conservative. The different methods are illustrated by their application to two compositional datasets and further assessed and compared using simulated data.

Comparative safety assessment of genetically modified crops: focus on equivalence with reference varieties could contribute to more efficient and effective field trials.

The initial compositional analysis of plants plays an important role within the internationally harmonized comparative safety assessment approach for genetically modified plants. Current EFSA guidance prescribes two types of comparison, namely difference tests with regard to a conventional comparator or control, and equivalence tests with regard to a collection of commercial reference varieties. The experience gained so far shows that most of the statistically significant differences between the test and control can be discounted based on the fact that they are still within equivalence limits of reference varieties with a presumed history of safe use. Inclusion of a test variety and reference varieties into field trial design, and of the statistical equivalence test would already suffice for the purpose of finding relevant parameters that warrant further assessment, hence both the inclusion of a conventional counterpart and the performance of difference testing can be omitted. This would also allow for the inclusion of safety testing regimes into plant variety testing VCU (value for cultivation and use) or other, independent variety trials.

How to Choose between Different Bayesian Posterior Indices for Hypothesis Testing in Practice.

Hypothesis testing is an essential statistical method in experimental psychology and the cognitive sciences. The problems of traditional null hypothesis significance testing (NHST) have been discussed widely, and among the proposed solutions to the replication problems caused by the inappropriate use of significance tests and p-values is a shift toward Bayesian data analysis. However, Bayesian hypothesis testing is concerned with various posterior indices for significance and the size of an effect. This complicates Bayesian hypothesis testing in practice, as the availability of multiple Bayesian alternatives to the traditional p-value causes confusion which one to select and why. In this paper, various Bayesian posterior indices which have been proposed in the literature are compared and their benefits and limitations are discussed. The comparison shows that conceptually not all proposed Bayesian alternatives to NHST and p-values are beneficial, and the usefulness of some indices strongly depends on the study design and research goal. However, the comparison also reveals that there exist at least two candidates among the available Bayesian posterior indices which have appealing theoretical properties and are widely underused in the cognitive sciences.

Measurement Invariant but Non-Normal Treatment Responses in Guided Internet Psychotherapies for Depressive and Generalized Anxiety Disorders.

Assessment of treatment response in psychotherapies can be undermined by lack of longitudinal measurement invariance (LMI) in symptom self-report inventories, by measurement error, and/or by wrong model assumptions. To understand and compare these threats to validity of outcome assessment in psychotherapy research, we studied LMI, sum scores, and Davidian Curve Item Response Theory models in a naturalistic guided internet psychotherapy treatment register of 2,218 generalized anxiety disorder (GAD) patients and 3,922 depressive disorder (DD) patients (aged ≥16 years). Symptoms were repeatedly assessed by Generalized Anxiety Disorder Assessment-7 (GAD-7) or Beck Depression Inventory. The symptom self-reports adhered to LMI under equivalence testing, suggesting sum scores are reasonable proxies for disorder status. However, the standard LMI assumption of normally distributed latent factors did not hold and inflated treatment response estimates by 0.2 to 0.3 standard deviation units compared with sum scores. Further methodological research on non-normally distributed latent constructs holds promise in advancing LMI and mental health assessment.

Equivalence tests before end of follow-up under the class of log transformation model.

One important topic in clinical trials is to show that the effects of new and standard treatments are equivalent in terms of clinical relevance. In literature, many equivalence tests based on the maximal difference between two survival functions for the two treatments over the whole time axis have been proposed. However, since survival times can only be observed until the end of follow-up, an equivalence test should be based on a comparison only in the observed time-window dictated by the end of follow-up. In this article, under the class of log transformation model, we propose an asymptotical α-level equivalence test for the difference between two survival functions that only addresses equivalence until the end of follow-up. We demonstrate that the hypothesis of equivalence of two survival functions before the end of follow-up can be formulated as interval-based hypothesis testing which involves the treatment effect parameter. Simulation results indicate that when sample size is sufficiently large the proposed test controls the type I error effectively and performs well at detecting the equivalence. The proposed test is applied to a dataset from veteran's administration lung cancer trial.

Lead equivalence testing using a custom-built testing kit.

Radiation protective garments should undergo a quality assurance regime comprising of an acceptance test of the lead equivalence before the garment is introduced into clinical service, followed by routine periodic visual and fluoroscopic inspections throughout its remaining clinical lifespan. The IEC 61331-1:2014 [1] is the leading standard outlining the methodology for testing of lead equivalence of these garments and forms the basis of the Australian/New Zealand Standards (1999) [2]. This study outlines the design and development of an IEC compliant broad beam lead equivalence testing setup, using an in-house custom-built testing kit (CBTK). The practicality and robustness of this kit was performance tested using lead equivalence measurements on 97% pure lead sheets. Hospital radiation protective garments are predominantly made of lead-free or lead-composite materials due to their light weight, as such, a set of lead-free (N-Pb) samples was also performance tested. These samples were tested using two different beam qualities; a total filtration of 2.5 mmAl and 0.25 mmCu added filtration, both at 102 kVp. Samples with thicknesses of 0.25 mm, 0.35 mm and 0.50 mm were used. The differential between labelled and measured lead equivalence averaged 3% for both the 'pure-lead' samples and N-Pb samples, with uncertainty of less than 7%. At 102 kVp, the use of Cu or Al filtration has marginal effect on measured lead equivalence for pure lead or N-Pb samples. The efficacy of utilizing the CBTK with a solid-state detector for lead equivalence testing was demonstrated through its ease of use, consistency and precision.

Multivariate equivalence testing for food safety assessment.

Products for food and feed derived from genetically modified (GM) crops are only allowed on the market when they are deemed to be safe for human health and the environment. The European Food Safety Authority (EFSA) performs safety assessment including a comparative approach: the compositional characteristics of a GM genotype are compared to those of reference genotypes that have a history of safe use. Statistical equivalence tests are used to carry out such a comparative assessment. These tests are univariate and therefore only consider one measured variable at a time. Phenotypic data, however, often comprise measurements on multiple variables that must be integrated to arrive at a single decision on acceptance in the regulatory process. The surge of modern molecular phenotyping platforms further challenges this integration, due to the large number of characteristics measured on the plants. This paper presents a new multivariate equivalence test that naturally extends a recently proposed univariate equivalence test and allows to assess equivalence across all variables simultaneously. The proposed test is illustrated on plant compositional data from a field study on maize grain and on untargeted metabolomic data of potato tubers, while its performance is assessed on simulated data.

The effects of U.S. county and state income inequality on self-reported happiness and health are equivalent to zero.

PURPOSE: A popular idea in the social sciences is that contexts with high income inequality undermine people's well-being and health. However, existing studies documenting this phenomenon typically compare a small number of higher-level units (countries/regions). Here, we use local income inequality indicators and temporal designs to provide the most highly powered test to date of the associations between income inequality and self-reported happiness and health in the USA METHOD: We combined county-level income inequality data (county-level Gini coefficients) with the responses from the General Social Survey (GSS) Cross-sectional dataset (13,000 + participants from ≈1000 county-waves) and Panels (3 × 3000 + participants from 3 × ≈500 county-waves); we used the GSS happiness ("not too happy," "pretty happy," or "very happy") and health ("poor," "fair," "good," or "excellent") variables. RESULTS: Multilevel-ordered logistic models and equivalence tests revealed that the within-county effects of income inequality on self-reported happiness and health were systematically equivalent to zero. Additional analyses revealed that the within-state effects were identical, that using alternative measures of state income inequality led to the same conclusions, and that lagged effects (between + 1 and + 12 years) were never significant and always equivalent to zero. CONCLUSION: The present work suggests that-at least in the USA-income inequality is likely neither associated with self-reported happiness nor with self-reported health.

Expression of cystic fibrosis lung disease modifier genes in human airway models.

BACKGROUND: Variation in respiratory response to cystic fibrosis (CF) small molecule therapies is due in part to the contribution of CF lung disease modifier genes. Cultured human bronchial epithelia (HBE) is the gold standard respiratory model for assessing CF therapeutic efficacy but it is hard to access. Cultured human nasal epithelia (HNE) is proposed as a more accessible surrogate model but it is unknown whether the expression profile of the modifier genes are comparable between HNE and HBE which we assess here. METHODS: RNA-sequencing was conducted on paired cultured and fresh HNE and HBE (n = 71 samples) collected from 21 individuals with CF. Genome-wide gene expression was first compared between cultured and fresh cells and then between cultured HNE and HBE based on an equivalence testing procedure we implemented. The co-expression relationships of CFTR and CF lung disease modifier genes were compared between cultured HNE and HBE to determine equivalent interactions. RESULTS: The culturing process had little impact on the expression level of CF lung disease modifier genes. Over 90% of expressed genes showed significant equivalent expression level across cultured HNE and HBE (expression fold-change<2, FDR<0.1), including CFTR and CF lung disease modifier genes. The difference in co-expression relationships among these genes was not significant (p-value=0.99), suggesting their functional interactions are likely to be consistent in the two models. CONCLUSIONS: Cultured HNE recapitulates the expression profile of CF lung disease modifier genes in cultured HBE, suggesting the biological processes involving these genes are likely to be consistent across the two models.

Is "not different" enough to conclude similar cardiovascular responses across sexes?

The number of research studies investigating whether similar or different cardiovascular responses or adaptations exist between males and females is increasing. Traditionally, difference-based statistical methods, e.g., t test, ANOVA, etc., have been implemented to compare cardiovascular function between males and females, with a P value of >0.05 used to denote similarity between sexes. However, an absence of evidence, i.e., large P value, is not evidence of absence, i.e., no sex differences. Equivalence testing determines whether two measures or groups provide statistically equivalent outcomes, in that they differ by less than an "ideally prespecified" smallest effect size of interest. Our perspective discusses the applicability and utility of integrating equivalence testing when conducting sex comparisons in cardiovascular research. An emphasis is placed on how cardiovascular researchers may conduct equivalence testing across multiple study designs, e.g., cross-sectional comparisons, repeated-measures intervention, etc. The strengths and weaknesses of this statistical tool are discussed. Equivalence analyses are relatively simple to conduct, may be used in conjunction with traditional hypothesis testing to interpret findings, and permit the determination of statistically equivalent responses between sexes. We recommend that cardiovascular researchers consider implementing equivalence testing to better our understanding of similar and different cardiovascular processes between sexes.

Smoothed Quantiles for χ2 Type Test Statistics with Applications.

Chi-square type test statistics are widely used in assessing the goodness-of-fit of a theoretical model. The exact distributions of such statistics can be quite different from the nominal chi-square distribution due to violation of conditions encountered with real data. In such instances, the bootstrap or Monte Carlo methodology might be used to approximate the distribution of the statistic. However, the sample quantile may be a poor estimate of the population counterpart when either the sample size is small or the number of different values of the replicated statistic is limited. Using statistical learning, this article develops a method that yields more accurate quantiles for chi-square type test statistics. Formulas for smoothing the quantiles of chi-square type statistics are obtained. Combined with the bootstrap methodology, the smoothed quantiles are further used to conduct equivalence testing in mean and covariance structure analysis. Two real data examples illustrate the applications of the developed formulas in quantifying the size of model misspecification under equivalence testing. The idea developed in the article can also be used to develop formulas for smoothing the quantiles of other types of test statistics or parameter estimates.

External validation of a shortened screening tool using individual participant data meta-analysis: A case study of the Patient Health Questionnaire-Dep-4.

Shortened versions of self-reported questionnaires may be used to reduce respondent burden. When shortened screening tools are used, it is desirable to maintain equivalent diagnostic accuracy to full-length forms. This manuscript presents a case study that illustrates how external data and individual participant data meta-analysis can be used to assess the equivalence in diagnostic accuracy between a shortened and full-length form. This case study compares the Patient Health Questionnaire-9 (PHQ-9) and a 4-item shortened version (PHQ-Dep-4) that was previously developed using optimal test assembly methods. Using a large database of 75 primary studies (34,698 participants, 3,392 major depression cases), we evaluated whether the PHQ-Dep-4 cutoff of ≥ 4 maintained equivalent diagnostic accuracy to a PHQ-9 cutoff of ≥ 10. Using this external validation dataset, a PHQ-Dep-4 cutoff of ≥ 4 maximized the sum of sensitivity and specificity, with a sensitivity of 0.88 (95% CI 0.81, 0.93), 0.68 (95% CI 0.56, 0.78), and 0.80 (95% CI 0.73, 0.85) for the semi-structured, fully structured, and MINI reference standard categories, respectively, and a specificity of 0.79 (95% CI 0.74, 0.83), 0.85 (95% CI 0.78, 0.90), and 0.83 (95% CI 0.80, 0.86) for the semi-structured, fully structured, and MINI reference standard categories, respectively. While equivalence with a PHQ-9 cutoff of ≥ 10 was not established, we found the sensitivity of the PHQ-Dep-4 to be non-inferior to that of the PHQ-9, and the specificity of the PHQ-Dep-4 to be marginally smaller than the PHQ-9.

Equivalence testing of a newly developed interviewer-led telephone script for the EORTC QLQ-C30.

PURPOSE: The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life-Core Questionnaire (QLQ-C30) is a widely used generic self-report measure of health-related quality of life (HRQOL) for cancer patients. However, no validated voice script for interviewer-led telephone administration was previously available. The aim of this study was to develop a voice script for interviewer administration via telephone. METHODS: Following guidelines from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) PRO Mixed Modes Good Research Practices Task Force, a randomised cross-over equivalence study, including cognitive debriefing, was conducted to assess equivalence between paper and telephone administration modes. Assuming an expected intraclass correlation coefficient (ICC) of 0.70 and a minimally acceptable level of 0.50, a sample size of 63 was required. RESULTS: Cognitive interviews with five cancer patients found the voice script to be clear and understandable. Due to a protocol deviation in the first wave of testing, only 26 patients were available for analyses. A second wave of recruitment was conducted, adding 37 patients (n = 63; mean age 55.48; 65.1% female). Total ICCs for mode comparison ranged from 0.72 (nausea and vomiting, 95% CI 0.48-0.86) to 0.90 (global health status/QoL, 95% CI 0.80-0.95; pain, 95% CI 0.79-0.95; constipation, 95% CI 0.80-0.95). For paper first administration, all ICCs were above 0.70, except nausea and vomiting (ICC 0.55; 95% CI 0.24-0.76) and financial difficulties (ICC 0.60; 95% CI 0.31-0.79). For phone first administration, all ICCs were above 0.70. CONCLUSIONS: The equivalence testing results support the voice script's validity for administration of the QLQ-C30 via telephone.