Multiplicity adjustments for the Dunnett procedure under heterokcedasticity.

We give a simulation-based method for computing the multiplicity adjusted p-values and critical constants for the Dunnett procedure for comparing treatments with a control under heteroskedasticity. The Welch-Satterthwaite test statistics used in this procedure do not have a simple multivariate t-distribution because their denominators are mixtures of chi-squares and are correlated because of the common control treatment sample variance present in all denominators. The joint distribution of the denominators of the test statistics is approximated by correlated chi-square variables and is generated using a novel algorithm proposed in this paper. This approximation is used to derive critical constants or adjusted p-values. The familywise error rate (FWER) of the proposed method is compared with some existing methods via simulation under different heteroskedastic scenarios. The results show that our proposed method controls the FWER most accurately, whereas other methods are either too conservative or liberal or control the FWER less accurately. The different methods considered are illustrated on a real data set.

Group sequential Holm and Hochberg procedures.

The problem of testing multiple hypotheses using a group sequential procedure often arises in clinical trials. We review several group sequential Holm (GSHM) type procedures proposed in the literature and clarify the relationships between them. In particular, we show which procedures are equivalent or, if different, which are more powerful and what are their pros and cons. We propose a step-up group sequential Hochberg (GSHC) procedure as a reverse application of a particular step-down GSHM procedure. We conducted an extensive simulation study to evaluate the familywise error rate (FWER) and power properties of that GSHM procedure and the GSHC procedure and found that the GSHC procedure controls FWER more closely and is more powerful. All procedures are illustrated with a common numerical example, the data for which are chosen to bring out the differences between them. A real case study is also presented to illustrate application of these procedures. R programs for applying the proposed procedures, additional simulation results, and the proof of the FWER control of the GSHC procedure in a special case are provided in Supplementary Material.

Advances in p-Value Based Multiple Test Procedures.

In this article we review recent advances in [Formula: see text]-value-based multiple test procedures (MTPs). We begin with a brief review of the basic tests of Bonferroni and Simes. Standard stepwise MTPs derived from them using the closure method of Marcus et al. (1976) are discussed next. They include the well-known MTPs of Holm (1979), Hochberg (1988) and Hommel (1988), and their extensions and improvements. This is followed by stepwise MTPs for a priori ordered hypotheses. Next we present gatekeeping MTPs (Dmitrienko and Tamhane, 2007) for hierarchically ordered families of hypotheses with logical relations among them. Finally, we give a brief review of the graphical approach (Bretz et al., 2009) to constructing and visualizing gatekeeping and other MTPs. Simple numerical examples are given to illustrate the various procedures.

A gatekeeping procedure to test a primary and a secondary endpoint in a group sequential design with multiple interim looks.

Glimm et al. (2010) and Tamhane et al. (2010) studied the problem of testing a primary and a secondary endpoint, subject to a gatekeeping constraint, using a group sequential design (GSD) with K=2 looks. In this article, we greatly extend the previous results to multiple (K>2) looks. If the familywise error rate (FWER) is to be controlled at a preassigned α level then it is clear that the primary boundary must be of level α. We show under what conditions one α-level primary boundary is uniformly more powerful than another. Based on this result, we recommend the choice of the O'Brien and Fleming (1979) boundary over the Pocock (1977) boundary for the primary endpoint. For the secondary endpoint the choice of the boundary is more complicated since under certain conditions the secondary boundary can be refined to have a nominal level α'>α, while still controlling the FWER at level α, thus boosting the secondary power. We carry out secondary power comparisons via simulation between different choices of primary-secondary boundary combinations. The methodology is applied to the data from the RALES study (Pitt et al., 1999; Wittes et al., 2001). An R library package gsrsb to implement the proposed methodology is made available on CRAN.

Allocating recycled significance levels in group sequential procedures for multiple endpoints.

Graphical approaches have been proposed in the literature for testing hypotheses on multiple endpoints by recycling significance levels from rejected hypotheses to unrejected ones. Recently, they have been extended to group sequential procedures (GSPs). Our focus in this paper is on the allocation of recycled significance levels from rejected hypotheses to the stages of the GSPs for unrejected hypotheses. We propose a delayed recycling method that allocates the recycled significance level from Stage r onward, where r is prespecified. We show that r cannot be chosen adaptively to coincide with the random stage at which the hypothesis from which the significance level is recycled is rejected. Such an adaptive GSP does not always control the FWER. One can choose r to minimize the expected sample size for a given power requirement. We illustrate how a simulation approach can be used for this purpose. Several examples, including a clinical trial example, are given to illustrate the proposed procedure.

On generalized Simes critical constants.

We consider the problem treated by Simes of testing the overall null hypothesis formed by the intersection of a set of elementary null hypotheses based on ordered p-values of the associated test statistics. The Simes test uses critical constants that do not need tabulation. Cai and Sarkar gave a method to compute generalized Simes critical constants which improve upon the power of the Simes test when more than a few hypotheses are false. The Simes constants can be viewed as the first order (requiring solution of a linear equation) and the Cai-Sarkar constants as the second order (requiring solution of a quadratic equation) constants. We extend the method to third order (requiring solution of a cubic equation) constants, and also offer an extension to an arbitrary kth order. We show by simulation that the third order constants are more powerful than the second order constants for testing the overall null hypothesis in most cases. However, there are some drawbacks associated with these higher order constants especially for k>3, which limits their practical usefulness.

A general multistage procedure for k-out-of-n gatekeeping.

We generalize a multistage procedure for parallel gatekeeping to what we refer to as k-out-of-n gatekeeping in which at least k out of n hypotheses ( 1 ⩽ k ⩽ n) in a gatekeeper family must be rejected in order to test the hypotheses in the following family. This gatekeeping restriction arises in certain types of clinical trials; for example, in rheumatoid arthritis trials, it is required that efficacy be shown on at least three of the four primary endpoints. We provide a unified theory of multistage procedures for arbitrary k, with k = 1 corresponding to parallel gatekeeping and k = n to serial gatekeeping. The theory provides an insight into the construction of truncated separable multistage procedures using the closure method. Explicit formulae for calculating the adjusted p-values are given. The proposed procedure is simpler to apply for this particular problem using a stepwise algorithm than the mixture procedure and the graphical procedure with memory using entangled graphs.

General theory of mixture procedures for gatekeeping.

The paper introduces a general approach to constructing mixture-based gatekeeping procedures in multiplicity problems with two or more families of hypotheses. Mixture procedures serve as extensions of and overcome limitations of some previous gatekeeping approaches such as parallel gatekeeping and tree-structured gatekeeping. This paper offers a general theory of mixture procedures constructed from nonparametric (p-value based) to parametric (normal theory based) procedures and studies their properties. It is also shown that the mixture procedure for parallel gatekeeping is equivalent to the multistage gatekeeping procedure. A clinical trial example is used to illustrate the mixture approach and the implementation of mixture procedures.

Adaptive extensions of a two-stage group sequential procedure for testing primary and secondary endpoints (I): unknown correlation between the endpoints.

In a previous paper we studied a two-stage group sequential procedure (GSP) for testing primary and secondary endpoints where the primary endpoint serves as a gatekeeper for the secondary endpoint. We assumed a simple setup of a bivariate normal distribution for the two endpoints with the correlation coefficient ρ between them being either an unknown nuisance parameter or a known constant. Under the former assumption, we used the least favorable value of ρ = 1 to compute the critical boundaries of a conservative GSP. Under the latter assumption, we computed the critical boundaries of an exact GSP. However, neither assumption is very practical. The ρ = 1 assumption is too conservative resulting in loss of power, whereas the known ρ assumption is never true in practice. In this part I of a two-part paper on adaptive extensions of this two-stage procedure (part II deals with sample size re-estimation), we propose an intermediate approach that uses the sample correlation coefficient r from the first-stage data to adaptively adjust the secondary boundary after accounting for the sampling error in r via an upper confidence limit on ρ by using a method due to Berger and Boos. We show via simulation that this approach achieves 5-11% absolute secondary power gain for ρ ≤0.5. The preferred boundary combination in terms of high primary as well as secondary power is that of O'Brien and Fleming for the primary and of Pocock for the secondary. The proposed approach using this boundary combination achieves 72-84% relative secondary power gain (with respect to the exact GSP that assumes known ρ). We give a clinical trial example to illustrate the proposed procedure.

Adaptive extensions of a two-stage group sequential procedure for testing primary and secondary endpoints (II): sample size re-estimation.

In this part II of the paper on adaptive extensions of a two-stage group sequential procedure (GSP) for testing primary and secondary endpoints, we focus on the second stage sample size re-estimation based on the first stage data. First, we show that if we use the Cui-Huang-Wang statistics at the second stage, then we can use the same primary and secondary boundaries as for the original procedure (without sample size re-estimation) and still control the type I familywise error rate. This extends their result for the single endpoint case. We further show that the secondary boundary can be sharpened in this case by taking the unknown correlation coefficient ρ between the primary and secondary endpoints into account through the use of the confidence limit method proposed in part I of this paper. If we use the sufficient statistics instead of the CHW statistics, then we need to modify both the primary and secondary boundaries; otherwise, the error rate can get inflated. We show how to modify the boundaries of the original group sequential procedure to control the familywise error rate. We provide power comparisons between competing procedures. We illustrate the procedures with a clinical trial example.

Mixtures of multiple testing procedures for gatekeeping applications in clinical trials.

This paper proposes a general framework for constructing gatekeeping procedures for clinical trials with hierarchical objectives. Such problems frequently exhibit complex structures including multiple families of hypotheses and logical restrictions. The proposed framework is based on combining multiple procedures across families. It enables the construction of powerful and flexible gatekeeping procedures that account for general logical restrictions among the hypotheses of interest. A clinical trial in patients with schizophrenia is used to illustrate the approach for parallel gatekeeping, whereas another clinical trial in patients with hypertension is used to illustrate the approach for gatekeeping with general logical restrictions.

Multistage and mixture parallel gatekeeping procedures in clinical trials.

Gatekeeping procedures have been developed to solve multiplicity problems arising in clinical trials with hierarchical objectives where the null hypotheses that address these objectives are grouped into ordered families. A general method for constructing multistage parallel gatekeeping procedures was proposed by Dmitrienko et al. (2008). The objective of this paper is to study two related classes of parallel gatekeeping procedures. Restricting to two-family hypothesis testing problems, we first use the mixture method developed in Dmitrienko and Tamhane (2011) to define a class of parallel gatekeeping procedures derived using the closure principle that can be more powerful than multistage gatekeeping procedures. Second, we show that power of multistage gatekeeping procedures can also be improved by using α-exhaustive tests for the component procedures. Extensions of these results for multiple families are stated. Illustrative examples from clinical trials are given.

A mixture gatekeeping procedure based on the Hommel test for clinical trial applications.

When conducting clinical trials with hierarchically ordered objectives, it is essential to use multiplicity adjustment methods that control the familywise error rate in the strong sense while taking into account the logical relations among the null hypotheses. This paper proposes a gatekeeping procedure based on the Hommel (1988) test, which offers power advantages compared to other p value-based tests proposed in the literature. A general description of the procedure is given and details are presented on how it can be applied to complex clinical trial designs. Two clinical trial examples are given to illustrate the methodology developed in the paper.

Testing a primary and a secondary endpoint in a group sequential design.

We consider a clinical trial with a primary and a secondary endpoint where the secondary endpoint is tested only if the primary endpoint is significant. The trial uses a group sequential procedure with two stages. The familywise error rate (FWER) of falsely concluding significance on either endpoint is to be controlled at a nominal level α. The type I error rate for the primary endpoint is controlled by choosing any α-level stopping boundary, e.g., the standard O'Brien-Fleming or the Pocock boundary. Given any particular α-level boundary for the primary endpoint, we study the problem of determining the boundary for the secondary endpoint to control the FWER. We study this FWER analytically and numerically and find that it is maximized when the correlation coefficient ρ between the two endpoints equals 1. For the four combinations consisting of O'Brien-Fleming and Pocock boundaries for the primary and secondary endpoints, the critical constants required to control the FWER are computed for different values of ρ. An ad hoc boundary is proposed for the secondary endpoint to address a practical concern that may be at issue in some applications. Numerical studies indicate that the O'Brien-Fleming boundary for the primary endpoint and the Pocock boundary for the secondary endpoint generally gives the best primary as well as secondary power performance. The Pocock boundary may be replaced by the ad hoc boundary for the secondary endpoint with a very little loss of secondary power if the practical concern is at issue. A clinical trial example is given to illustrate the methods.

General multistage gatekeeping procedures.

A general multistage (stepwise) procedure is proposed for dealing with arbitrary gatekeeping problems including parallel and serial gatekeeping. The procedure is very simple to implement since it does not require the application of the closed testing principle and the consequent need to test all nonempty intersections of hypotheses. It is based on the idea of carrying forward the Type I error rate for any rejected hypotheses to test hypotheses in the next ordered family. This requires the use of a so-called separable multiple test procedure (MTP) in the earlier family. The Bonferroni MTP is separable, but other standard MTPs such as Holm, Hochberg, Fallback and Dunnett are not. Their truncated versions are proposed which are separable and more powerful than the Bonferroni MTP. The proposed procedure is illustrated by a clinical trial example.

Superiority inferences on individual endpoints following noninferiority testing in clinical trials.

We consider the problem of drawing superiority inferences on individual endpoints following non-inferiority testing. Röhmel et al. (2006) pointed out this as an important problem which had not been addressed by the previous procedures that only tested for global superiority. Röhmel et al. objected to incorporating the non-inferiority tests in the assessment of the global superiority test by exploiting the relationship between the two, since the results of the latter test then depend on the non-inferiority margins specified for the former test. We argue that this is justified, besides the fact that it enhances the power of the global superiority test. We provide a closed testing formulation which generalizes the three-step procedure proposed by Röhmel et al. for two endpoints. For the global superiority test, Röhmel et al. suggest using the Läuter (1996) test which is modified to make it monotone. The resulting test not only is complicated to use, but the modification does not readily extend to more than two endpoints, and it is less powerful in general than several of its competitors. This is verified in a simulation study. Instead, we suggest applying the one-sided likelihood ratio test used by Perlman and Wu (2004) or the union-intersection t(max) test used by Tamhane and Logan (2004).

A note on tree gatekeeping procedures in clinical trials.

Dmitrienko et al. (Statist. Med. 2007; 26:2465-2478) proposed a tree gatekeeping procedure for testing logically related hypotheses in hierarchically ordered families, which uses weighted Bonferroni tests for all intersection hypotheses in a closure method by Marcus et al. (Biometrika 1976; 63:655-660). An algorithm was given to assign weights to the hypotheses for every intersection. The purpose of this note is to show that any weight assignment algorithm that satisfies a set of sufficient conditions can be used in this procedure to guarantee gatekeeping and independence properties. The algorithm used in Dmitrienko et al. (Statist. Med. 2007; 26:2465-2478) may fail to meet one of the conditions, namely monotonicity of weights, which may cause it to violate the gatekeeping property. An example is given to illustrate this phenomenon. A modification of the algorithm is shown to rectify this problem.

Gatekeeping procedures with clinical trial applications.

The objective of this paper is to give an overview of a relatively new area of multiplicity research that deals with the analysis of hierarchically ordered multiple objectives. Testing procedures for this problem are known as gatekeeping procedures and have found a variety of applications in clinical trials. This paper reviews main classes of these procedures, including serial and parallel gatekeeping procedures, and tree gatekeeping procedures that account for logical restrictions among multiple objectives. We focus on procedures based on marginal p-values; extensions to procedures that exploit the joint distribution of the p-values are also noted. Clinical trial examples are used to illustrate the procedures and their important properties.

Tree-structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives.

This paper discusses a new class of multiple testing procedures, tree-structured gatekeeping procedures, with clinical trial applications. These procedures arise in clinical trials with hierarchically ordered multiple objectives, for example, in the context of multiple dose-control tests with logical restrictions or analysis of multiple endpoints. The proposed approach is based on the principle of closed testing and generalizes the serial and parallel gatekeeping approaches developed by Westfall and Krishen (J. Statist. Planning Infer. 2001; 99:25-41) and Dmitrienko et al. (Statist. Med. 2003; 22:2387-2400). The proposed testing methodology is illustrated using a clinical trial with multiple endpoints (primary, secondary and tertiary) and multiple objectives (superiority and non-inferiority testing) as well as a dose-finding trial with multiple endpoints.