Efficient Multiple Imputation for Sensitivity Analysis of Recurrent Events Data with Informative Censoring.

Missing data are commonly encountered in clinical trials due to dropout or nonadherence to study procedures. In trials in which recurrent events are of interest, the observed count can be an undercount of the events if a patient drops out before the end of the study. In many applications, the data are not necessarily missing at random and it is often not possible to test the missing at random assumption. Consequently, it is critical to conduct sensitivity analysis. We develop a control-based multiple imputation method for recurrent events data, where patients who drop out of the study are assumed to have a similar response profile to those in the control group after dropping out. Specifically, we consider the copy reference approach and the jump to reference approach. We model the recurrent event data using a semiparametric proportional intensity frailty model with the baseline hazard function completely unspecified. We develop nonparametric maximum likelihood estimation and inference procedures. We then impute the missing data based on the large sample distribution of the resulting estimators. The variance estimation is corrected by a bootstrap procedure. Simulation studies demonstrate the proposed method performs well in practical settings. We provide applications to two clinical trials.

Controlling false discovery proportion in identification of drug-related adverse events from multiple system organ classes.

Analyzing safety data from clinical trials to detect safety signals worth further examination involves testing multiple hypotheses, one for each observed adverse event (AE) type. There exists certain hierarchical structure for these hypotheses due to the classification of the AEs into system organ classes, and these AEs are also likely correlated. Many approaches have been proposed to identify safety signals under the multiple testing framework and tried to achieve control of false discovery rate (FDR). The FDR control concerns the expectation of the false discovery proportion (FDP). In practice, the control of the actual random variable FDP could be more relevant and has recently drawn much attention. In this paper, we proposed a two-stage procedure for safety signal detection with direct control of FDP, through a permutation-based approach for screening groups of AEs and a permutation-based approach of constructing simultaneous upper bounds for false discovery proportion. Our simulation studies showed that this new approach has controlled FDP. We demonstrate our approach using data sets derived from a drug clinical trial.

Efficient methods for signal detection from correlated adverse events in clinical trials.

It is an important and yet challenging task to identify true signals from many adverse events that may be reported during the course of a clinical trial. One unique feature of drug safety data from clinical trials, unlike data from post-marketing spontaneous reporting, is that many types of adverse events are reported by only very few patients leading to rare events. Due to the limited study size, the p-values of testing whether the rate is higher in the treatment group across all types of adverse events are in general not uniformly distributed under the null hypothesis that there is no difference between the treatment group and the placebo group. A consequence is that typically fewer than 100α percent of the hypotheses are rejected under the null at the nominal significance level of α . The other challenge is multiplicity control. Adverse events from the same body system may be correlated. There may also be correlations between adverse events from different body systems. To tackle these challenging issues, we develop Monte-Carlo-based methods for the signal identification from patient-reported adverse events in clinical trials. The proposed methodologies account for the rare events and arbitrary correlation structures among adverse events within and/or between body systems. Extensive simulation studies demonstrate that the proposed method can accurately control the family-wise error rate and is more powerful than existing methods under many practical situations. Application to two real examples is provided.

Improved power of familywise error rate procedures for discrete data under dependency.

In many applications where it is necessary to test multiple hypotheses simultaneously, the data encountered are discrete. In such cases, it is important for multiplicity adjustment to take into account the discreteness of the distributions of the p-values, to assure that the procedure is not overly conservative. In this paper, we review some known multiple testing procedures for discrete data that control the familywise error rate, the probability of making any false rejection. Taking advantage of the fact that the exact permutation or exact pairwise permutation distributions of the p-values can often be determined when the sample size is small, we investigate procedures that incorporate the dependence structure through the exact permutation distribution and propose two new procedures that incorporate the exact pairwise permutation distributions. A step-up procedure is also proposed that accounts for the discreteness of the data. The performance of the proposed procedures is investigated through simulation studies and two applications. The results show that by incorporating both discreteness and dependency of p-value distributions, gains in power can be achieved.

Multiple testing with discrete data: Proportion of true null hypotheses and two adaptive FDR procedures.

We consider multiple testing with false discovery rate (FDR) control when p values have discrete and heterogeneous null distributions. We propose a new estimator of the proportion of true null hypotheses and demonstrate that it is less upwardly biased than Storey's estimator and two other estimators. The new estimator induces two adaptive procedures, that is, an adaptive Benjamini-Hochberg (BH) procedure and an adaptive Benjamini-Hochberg-Heyse (BHH) procedure. We prove that the adaptive BH (aBH) procedure is conservative nonasymptotically. Through simulation studies, we show that these procedures are usually more powerful than their nonadaptive counterparts and that the adaptive BHH procedure is usually more powerful than the aBH procedure and a procedure based on randomized p-value. The adaptive procedures are applied to a study of HIV vaccine efficacy, where they identify more differentially polymorphic positions than the BH procedure at the same FDR level.

Periodic benefit-risk assessment using Bayesian stochastic multi-criteria acceptability analysis.

Benefit-risk (BR) assessment is essential to ensure the best decisions are made for a medical product in the clinical development process, regulatory marketing authorization, post-market surveillance, and coverage and reimbursement decisions. One challenge of BR assessment in practice is that the benefit and risk profile may keep evolving while new evidence is accumulating. Regulators and the International Conference on Harmonization (ICH) recommend performing periodic benefit-risk evaluation report (PBRER) through the product's lifecycle. In this paper, we propose a general statistical framework for periodic benefit-risk assessment, in which Bayesian meta-analysis and stochastic multi-criteria acceptability analysis (SMAA) will be combined to synthesize the accumulating evidence. The proposed approach allows us to compare the acceptability of different drugs dynamically and effectively and accounts for the uncertainty of clinical measurements and imprecise or incomplete preference information of decision makers. We apply our approaches to two real examples in a post-hoc way for illustration purpose. The proposed method may easily be modified for other pre and post market settings, and thus be an important complement to the current structured benefit-risk assessment (sBRA) framework to improve the transparent and consistency of the decision-making process.

Control-based imputation for sensitivity analyses in informative censoring for recurrent event data.

In clinical trials, missing data commonly arise through nonadherence to the randomized treatment or to study procedure. For trials in which recurrent event endpoints are of interests, conventional analyses using the proportional intensity model or the count model assume that the data are missing at random, which cannot be tested using the observed data alone. Thus, sensitivity analyses are recommended. We implement the control-based multiple imputation as sensitivity analyses for the recurrent event data. We model the recurrent event using a piecewise exponential proportional intensity model with frailty and sample the parameters from the posterior distribution. We impute the number of events after dropped out and correct the variance estimation using a bootstrap procedure. We apply the method to an application of sitagliptin study.

On inference of control-based imputation for analysis of repeated binary outcomes with missing data.

Missing data are common in longitudinal clinical trials. How to handle missing data is critical for both sponsors and regulatory agencies to assess treatment effect from the trials. Recently, a control-based imputation has been proposed, where the missing data are imputed based on the assumption that patients who discontinued the test drug will have a similar response profile to the patients in the control group. Under control-based imputation, the variance estimation may be biased using Rubin's formula which could produce biased statistical inferences. We evaluate several statistical methods for obtaining appropriate variances under control-based imputation for analysis of repeated binary outcomes with monotone missing data and show that both the analytical method developed by Robins & Wang and the nonparametric bootstrap method provide more appropriate variance estimates under various simulation settings. We use the methods in an application of an antidepressant Phase III clinical trial and give discussion and recommendations on method performance and preference.

Bayesian probability of success for clinical trials using historical data.

Developing sophisticated statistical methods for go/no-go decisions is crucial for clinical trials, as planning phase III or phase IV trials is costly and time consuming. In this paper, we develop a novel Bayesian methodology for determining the probability of success of a treatment regimen on the basis of the current data of a given trial. We introduce a new criterion for calculating the probability of success that allows for inclusion of covariates as well as allowing for historical data based on the treatment regimen, and patient characteristics. A new class of prior distributions and covariate distributions is developed to achieve this goal. The methodology is quite general and can be used with univariate or multivariate continuous or discrete data, and it generalizes Chuang-Stein's work. This methodology will be invaluable for informing the scientist on the likelihood of success of the compound, while including the information of covariates for patient characteristics in the trial population for planning future pre-market or post-market trials.

Medical records-based postmarketing safety evaluation of rare events with uncertain status.

We develop a simple statistic for comparing rates of rare adverse events between treatment groups in postmarketing safety studies where the events have uncertain status. In this setting, the statistic is asymptotically equivalent to the logrank statistic, but the limiting distribution has Poisson and binomial components instead of being Gaussian. We develop two new procedures for computing critical values: a Gaussian approximation and a parametric bootstrap. Both numerical and asymptotic properties of the procedures are studied. The test procedures are demonstrated on a postmarketing safety study of the RotaTeq vaccine. This vaccine was developed to reduce the incidence of severe diarrhea in infants.

Medical records-based postmarketing safety evaluation of rare events with uncertain status.

We develop a simple statistic for comparing rates of rare adverse events between treatment groups in postmarketing safety studies where the events have uncertain status. In this setting, the statistic is asymptotically equivalent to the logrank statistic, but the limiting distribution has Poisson and binomial components instead of being Gaussian. We develop two new procedures for computing critical values, a Gaussian approximation and a parametric bootstrap. Both numerical and asymptotic properties of the procedures are studied. The test procedures are demonstrated on a postmarketing safety study of the RotaTeq vaccine. This vaccine was developed to reduce the incidence of severe diarrhea in infants.

cSSMD: assessing collective activity for addressing off-target effects in genome-scale RNA interference screens.

MOTIVATION: Off-target activity commonly exists in RNA interference (RNAi) screens and often generates false positives. Existing analytic methods for addressing the off-target effects are demonstrably inadequate in RNAi confirmatory screens. RESULTS: Here, we present an analytic method assessing the collective activity of multiple short interfering RNAs (siRNAs) targeting a gene. Using this method, we can not only reduce the impact of off-target activities, but also evaluate the specific effect of an siRNA, thus providing information about potential off-target effects. Using in-house RNAi screens, we demonstrate that our method obtains more reasonable and sensible results than current methods such as the redundant siRNA activity (RSA) method, the RNAi gene enrichment ranking (RIGER) method, the frequency approach and the t-test. CONTACT: xiaohua_zhang@merck.com SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

The use of SSMD-based false discovery and false nondiscovery rates in genome-scale RNAi screens.

In genome-scale RNA interference (RNAi) screens, it is critical to control false positives and false negatives statistically. Traditional statistical methods for controlling false discovery and false nondiscovery rates are inappropriate for hit selection in RNAi screens because the major goal in RNAi screens is to control both the proportion of short interfering RNAs (siRNAs) with a small effect among selected hits and the proportion of siRNAs with a large effect among declared nonhits. An effective method based on strictly standardized mean difference (SSMD) has been proposed for statistically controlling false discovery rate (FDR) and false nondiscovery rate (FNDR) appropriate for RNAi screens. In this article, the authors explore the utility of the SSMD-based method for hit selection in RNAi screens. As demonstrated in 2 genome-scale RNAi screens, the SSMD-based method addresses the unmet need of controlling for the proportion of siRNAs with a small effect among selected hits, as well as controlling for the proportion of siRNAs with a large effect among declared nonhits. Furthermore, the SSMD-based method results in reasonably low FDR and FNDR for selecting inhibition or activation hits. This method works effectively and should have a broad utility for hit selection in RNAi screens with replicates.

Sequential generalized likelihood ratio tests for vaccine safety evaluation.

The evaluation of vaccine safety involves pre-clinical animal studies, pre-licensure randomized clinical trials, and post-licensure safety studies. Sequential design and analysis are of particular interest because they allow early termination of the trial or quick detection that the vaccine exceeds a prescribed bound on the adverse event rate. After a review of the recent developments in this area, we propose a new class of sequential generalized likelihood ratio tests for evaluating adverse event rates in two-armed pre-licensure clinical trials and single-armed post-licensure studies. The proposed approach is illustrated using data from the Rotavirus Efficacy and Safety Trial. Simulation studies of the performance of the proposed approach and other methods are also given.

Age dependence of the risk of intussusception following [corrected] rhesus-human reassortant rotavirus tetravalent vaccine: is it beyond doubt?

The relation between the risk of intussusception and age at the time of receipt of the first dose of rhesus-human reassortant rotavirus tetravalent vaccine (RRV-TV) has been studied extensively on the basis of Centers for Disease Control and Prevention (CDC) matched case-control study data, using various statistical methods, including conditional logistic regression and quadratic smoothing splines. However, different conclusions have been reported in published analyses regarding the dependence of the risk of intussusception on age at first dose. The authors reanalyzed the CDC matched case-control data set using unrestricted and restricted quadratic smoothing spline methods for various exposure windows (i.e., intervals postvaccination). These analyses indicated that the use of different models may lead to different conclusions. The restricted quadratic smoothing spline with appropriately chosen knot locations showed a statistically significant increased risk of intussusception associated with RRV-TV for the exposure window 3-14 days after the first dose at an age as young as 49 days, the youngest age in the data set at which vaccine was administered; this implies an increased risk of intussusception associated with RRV-TV at all ages studied.

Determination of sample size in genome-scale RNAi screens.

MOTIVATION: For genome-scale RNAi research, it is critical to investigate sample size required for the achievement of reasonably low false negative rate (FNR) and false positive rate. RESULTS: The analysis in this article reveals that current design of sample size contributes to the occurrence of low signal-to-noise ratio in genome-scale RNAi projects. The analysis suggests that (i) an arrangement of 16 wells per plate is acceptable and an arrangement of 20-24 wells per plate is preferable for a negative control to be used for hit selection in a primary screen without replicates; (ii) in a confirmatory screen or a primary screen with replicates, a sample size of 3 is not large enough, and there is a large reduction in FNRs when sample size increases from 3 to 4. To search a tradeoff between benefit and cost, any sample size between 4 and 11 is a reasonable choice. If the main focus is the selection of siRNAs with strong effects, a sample size of 4 or 5 is a good choice. If we want to have enough power to detect siRNAs with moderate effects, sample size needs to be 8, 9, 10 or 11. These discoveries about sample size bring insight to the design of a genome-scale RNAi screen experiment.

Analyzing multiple endpoints in clinical trials of pain treatments: IMMPACT recommendations. Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials.

The increasing complexity of randomized clinical trials and the practice of obtaining a wide variety of measurements from study participants have made the consideration of multiple endpoints a critically important issue in the design, analysis, and interpretation of clinical trials. Failure to consider important outcomes can limit the validity and utility of clinical trials; specifying multiple endpoints for the evaluation of treatment efficacy, however, can increase the rate of false positive conclusions about the efficacy of a treatment. We describe the use of multiple endpoints in the design, analysis, and interpretation of pain clinical trials, and review available strategies and methods for addressing multiplicity. To decrease the probability of a Type I error (i.e., the likelihood of obtaining statistically significant results by chance) in pain clinical trials, the use of gatekeeping procedures and other methods that correct for multiple analyses is recommended when a single primary endpoint does not adequately reflect the overall benefits of treatment. We emphasize the importance of specifying in advance the outcomes and clinical decision rule that will serve as the basis for determining that a treatment is efficacious and the methods that will be used to control the overall Type I error rate.

Hit selection with false discovery rate control in genome-scale RNAi screens.

RNA interference (RNAi) is a modality in which small double-stranded RNA molecules (siRNAs) designed to lead to the degradation of specific mRNAs are introduced into cells or organisms. siRNA libraries have been developed in which siRNAs targeting virtually every gene in the human genome are designed, synthesized and are presented for introduction into cells by transfection in a microtiter plate array. These siRNAs can then be transfected into cells using high-throughput screening (HTS) methodologies. The goal of RNAi HTS is to identify a set of siRNAs that inhibit or activate defined cellular phenotypes. The commonly used analysis methods including median +/- kMAD have issues about error rates in multiple hypothesis testing and plate-wise versus experiment-wise analysis. We propose a methodology based on a Bayesian framework to address these issues. Our approach allows for sharing of information across plates in a plate-wise analysis, which obviates the need for choosing either a plate-wise or experimental-wise analysis. The proposed approach incorporates information from reliable controls to achieve a higher power and a balance between the contribution from the samples and control wells. Our approach provides false discovery rate (FDR) control to address multiple testing issues and it is robust to outliers.

Integrating experimental and analytic approaches to improve data quality in genome-wide RNAi screens.

RNA interference (RNAi) not only plays an important role in drug discovery but can also be developed directly into drugs. RNAi high-throughput screening (HTS) biotechnology allows us to conduct genome-wide RNAi research. A central challenge in genome-wide RNAi research is to integrate both experimental and computational approaches to obtain high quality RNAi HTS assays. Based on our daily practice in RNAi HTS experiments, we propose the implementation of 3 experimental and analytic processes to improve the quality of data from RNAi HTS biotechnology: (1) select effective biological controls; (2) adopt appropriate plate designs to display and/or adjust for systematic errors of measurement; and (3) use effective analytic metrics to assess data quality. The applications in 5 real RNAi HTS experiments demonstrate the effectiveness of integrating these processes to improve data quality. Due to the effectiveness in improving data quality in RNAi HTS experiments, the methods and guidelines contained in the 3 experimental and analytic processes are likely to have broad utility in genome-wide RNAi research.

Evaluating the safety of a rotavirus vaccine: the REST of the story.

The Rotavirus Efficacy and Safety Trial (REST) was a blinded, placebo-controlled study of the live pentavalent human-bovine vaccine, RotaTeq (Merck & Co. Inc., West Point, PA). REST was noteworthy because its primary objective was to evaluate the safety of RotaTeq with regard to intussusception, a rare intestinal illness that occurs with a background incidence of approximately 50 cases per 100 000 infant years. The study involved approximately 70 000 infants at over 500 study sites in 11 countries. The study demonstrated that the risk of intussusception was similar in vaccine and placebo recipients and that the vaccine prevented rotavirus gastroenteritis, ameliorated the severity of disease in those who had any disease, and substantially reduced rotavirus-associated hospitalizations and other health care contacts. This report provides an in-depth review of the background, statistical and regulatory considerations, and execution of REST. We describe the rationale and methods used for sample size, continuous safety monitoring, group sequential design, and detailed study execution. The results of the study have been reported elsewhere. The design and conduct of this study may serve as a useful model for planning other future large-scale clinical trials, especially those evaluating uncommon adverse events.