Challenges and strategies for clinical trials in propionic and methylmalonic acidemias.

Clinical trial development in rare diseases poses significant study design and methodology challenges, such as disease heterogeneity and appropriate patient selection, identification and selection of key endpoints, decisions on study duration, choice of control groups, selection of appropriate statistical analyses, and patient recruitment. Therapeutic development in organic acidemias (OAs) shares many challenges with other inborn errors of metabolism, such as incomplete understanding of natural history, heterogenous disease presentations, requirement for sensitive outcome measures and difficulties recruiting a small sample of participants. Here, we review strategies for the successful development of a clinical trial to evaluate treatment response in propionic and methylmalonic acidemias. Specifically, we discuss crucial decisions that may significantly impact success of the study, including patient selection, identification and selection of endpoints, determination of the study duration, consideration of control groups including natural history controls, and selection of appropriate statistical analyses. The significant challenges associated with designing a clinical trial in rare disease can sometimes be successfully met through strategic engagement with experts in the rare disease, seeking regulatory and biostatistical guidance, and early involvement of patients and families.

Study design challenges and strategies in clinical trials for rare diseases: Lessons learned from pantothenate kinase-associated neurodegeneration.

Substantial challenges in study design and methodology exist during clinical trial development to examine treatment response in patients with a rare disease, especially those with predominant central nervous system involvement and heterogeneity in clinical manifestations and natural history. Here we discuss crucial decisions which may significantly impact success of the study, including patient selection and recruitment, identification and selection of endpoints, determination of the study duration, consideration of control groups including natural history controls, and selection of appropriate statistical analyses. We review strategies for the successful development of a clinical trial to evaluate treatment of a rare disease with a focus on inborn errors of metabolism (IEMs) that present with movement disorders. The strategies presented using pantothenate kinase-associated neurodegeneration (PKAN) as the rare disease example can be applied to other rare diseases, particularly IEMs with movement disorders (e.g., other neurodegeneration with brain iron accumulation disorders, lysosomal storage disorders). The significant challenges associated with designing a clinical trial in rare disease can sometimes be successfully met through strategic engagement with experts in the rare disease, seeking regulatory and biostatistical guidance, and early involvement of patients and families. In addition to these strategies, we discuss the urgent need for a paradigm shift within the regulatory processes to help accelerate medical product development and bring new innovations and advances to patients with rare neurodegenerative diseases who need them earlier in disease progression and prior to clinical manifestations.

Sample size formula for a win ratio endpoint.

The win ratio composite endpoint, which organizes the components of the composite hierarchically, is becoming popular in late-stage clinical trials. The method involves comparing data in a pair-wise manner starting with the endpoint highest in priority (eg, cardiovascular death). If the comparison is a tie, the endpoint next highest in priority (eg, hospitalizations for heart failure) is compared, and so on. Its sample size is usually calculated through complex simulations because there does not exist in the literature a simple sample size formula. This article provides a formula that depends on the probability that a randomly selected patient from one group does better than a randomly selected patient from another group, and on the probability of a tie. We compare the published 95% confidence intervals, which require patient-level data, with that calculated from the formula, requiring only summary-level data, for 17 composite or single win ratio endpoints. The two sets of results are similar. Simulations show the sample size formula performs well. The formula provides important insights. It shows when adding an endpoint to the hierarchy can increase power even if the added endpoint has low power by itself. It provides relevant information to modify an on-going blinded trial if necessary. The formula allows a non-specialist to quickly determine the size of the trial with a win ratio endpoint whose use is expected to increase over time.

Improving the Standards of Reporting of Clinical Trial Data.

Although checklists and guidelines for reporting and interpretation of clinical trial results are of immense value there is still room for a biased presentation in journal publications. Two important sources of bias that remain are as follows: (1) The absence of a principle guiding the display of point estimates in abstracts. For example, bias arises, even for a primary endpoint, when the reported point estimate is preferentially selected and does not correspond to the prespecified method of analysis. The benefit of treatment on an endpoint is often communicated through point estimates, and as abstracts contain the main takeaways, establishing ground rules for what to include and what not to include is crucial. (2) A commingling in the body of the publication of results from α-controlled endpoints, non-α-controlled endpoints, and post hoc analyses. The total number of non-α-controlled and post hoc analyses are unknown. Blending a favored selection of these with α-controlled results provides opportunities to overstate or understate findings as desired. Publicly available results provide the grist for the changes proposed to improve reporting standards. Additional changes are recommended as well, including a threshold of significance more stringent than 0.05 for non-α-controlled analyses. For safety, the proposal is to display the data via the mean cumulative function graph for prespecified adverse events of interest. The bottom line is that more objective reporting can be achieved if journals establish standards for reporting of point estimates in abstracts and require a hierarchical display of results in the main body.

Improving the Standards of Reporting of Clinical Trial Data.

Although checklists and guidelines for reporting and interpretation of clinical trial results are of immense value there is still room for a biased presentation in journal publications. Two important sources of bias that remain are as follows: (1) The absence of a principle guiding the display of point estimates in abstracts. For example, bias arises, even for a primary endpoint, when the reported point estimate is preferentially selected and does not correspond to the prespecified method of analysis. The benefit of treatment on an endpoint is often communicated through point estimates, and as abstracts contain the main takeaways, establishing ground rules for what to include and what not to include is crucial. (2) A commingling in the body of the publication of results from α-controlled endpoints, non-α-controlled endpoints, and post hoc analyses. The total number of non-α-controlled and post hoc analyses are unknown. Blending a favored selection of these with α-controlled results provides opportunities to overstate or understate findings as desired. Publicly available results provide the grist for the changes proposed to improve reporting standards. Additional changes are recommended as well, including a threshold of significance more stringent than 0.05 for non-α-controlled analyses. For safety, the proposal is to display the data via the mean cumulative function graph for prespecified adverse events of interest. The bottom line is that more objective reporting can be achieved if journals establish standards for reporting of point estimates in abstracts and require a hierarchical display of results in the main body.

Non-inferiority versus superiority drug claims: the (not so) subtle distinction.

BACKGROUND: Current regulatory guidance and practice of non-inferiority trials are asymmetric in favor of the test treatment (Test) over the reference treatment (Control). These trials are designed to compare the relative efficacy of Test to Control by reference to a clinically important margin, M. MAIN TEXT: Non-inferiority trials allow for the conclusion of: (a) non-inferiority of Test to Control if Test is slightly worse than Control but by no more than M; and (b) superiority if Test is slightly better than Control even if it is by less than M. From Control's perspective, (b) should lead to a conclusion of non-inferiority of Control to Test. The logical interpretation ought to be that, while Test is statistically better, it is not clinically superior to Control (since Control should be able to claim non-inferiority to Test). This article makes a distinction between statistical and clinical significance, providing for symmetry in the interpretation of results. Statistical superiority and clinical superiority are achieved, respectively, when the null and the non-inferiority margins are exceeded. We discuss a similar modification to placebo-controlled trials. CONCLUSION: Rules for interpretation should not favor one treatment over another. Claims of statistical or clinical superiority should depend on whether or not the null margin or the clinically relevant margin is exceeded.

Robust inference for group sequential trials.

For ethical reasons, group sequential trials were introduced to allow trials to stop early in the event of extreme results. Endpoints in such trials are usually mortality or irreversible morbidity. For a given endpoint, the norm is to use a single test statistic and to use that same statistic for each analysis. This approach is risky because the test statistic has to be specified before the study is unblinded, and there is loss in power if the assumptions that ensure optimality for each analysis are not met. To minimize the risk of moderate to substantial loss in power due to a suboptimal choice of a statistic, a robust method was developed for nonsequential trials. The concept is analogous to diversification of financial investments to minimize risk. The method is based on combining P values from multiple test statistics for formal inference while controlling the type I error rate at its designated value.This article evaluates the performance of 2 P value combining methods for group sequential trials. The emphasis is on time to event trials although results from less complex trials are also included. The gain or loss in power with the combination method relative to a single statistic is asymmetric in its favor. Depending on the power of each individual test, the combination method can give more power than any single test or give power that is closer to the test with the most power. The versatility of the method is that it can combine P values from different test statistics for analysis at different times. The robustness of results suggests that inference from group sequential trials can be strengthened with the use of combined tests.

A single test for rejecting the null hypothesis in subgroups and in the overall sample.

In clinical trials, some patient subgroups are likely to demonstrate larger effect sizes than other subgroups. For example, the effect size, or informally the benefit with treatment, is often greater in patients with a moderate condition of a disease than in those with a mild condition. A limitation of the usual method of analysis is that it does not incorporate this ordering of effect size by patient subgroup. We propose a test statistic which supplements the conventional test by including this information and simultaneously tests the null hypothesis in pre-specified subgroups and in the overall sample. It results in more power than the conventional test when the differences in effect sizes across subgroups are at least moderately large; otherwise it loses power. The method involves combining p-values from models fit to pre-specified subgroups and the overall sample in a manner that assigns greater weight to subgroups in which a larger effect size is expected. Results are presented for randomized trials with two and three subgroups.

The potential for increased power from combining P-values testing the same hypothesis.

The conventional approach to hypothesis testing for formal inference is to prespecify a single test statistic thought to be optimal. However, we usually have more than one test statistic in mind for testing the null hypothesis of no treatment effect but we do not know which one is the most powerful. Rather than relying on a single p-value, combining p-values from prespecified multiple test statistics can be used for inference. Combining functions include Fisher's combination test and the minimum p-value. Using randomization-based tests, the increase in power can be remarkable when compared with a single test and Simes's method. The versatility of the method is that it also applies when the number of covariates exceeds the number of observations. The increase in power is large enough to prefer combined p-values over a single p-value. The limitation is that the method does not provide an unbiased estimator of the treatment effect and does not apply to situations when the model includes treatment by covariate interaction.

Improving the operational efficiency of Phase 2 and 3 trials.

The period toward the end of patients' participation in late stage blinded clinical trials is highly resource intensive for the sponsor. Consider first a Phase 3 trial. If the trial is a success, the sponsor has to implement the next steps, which might be filing for approval of the drug with the US Food and Drug Administration (FDA). To shorten the time interval between trial completion and submission of the package to the FDA, sponsors front-load as much work as is possible at risk. The approach is efficient if the trial succeeds but is inefficient if it fails. For a failed trial, the sponsor is unlikely to proceed with the plan that assumed success. Phase 2 trials are also at risk of being inefficient. Many activities, such as planning for drug interaction studies, thorough QT studies, or site selection for Phase 3 trials, are set in motion prior to completion of the Phase 2 trial. The work going on in parallel is wasted if the trial fails. The proposal to improve the efficiency is to let an independent entity provide the sponsor critical information at an earlier time necessary to reevaluate activities ongoing in parallel and external to the trial.

Projecting adverse event incidence rates using empirical Bayes methodology.

Although there is considerable interest in adverse events observed in clinical trials, projecting adverse event incidence rates in an extended period can be of interest when the trial duration is limited compared to clinical practice. A naïve method for making projections might involve modeling the observed rates into the future for each adverse event. However, such an approach overlooks the information that can be borrowed across all the adverse event data. We propose a method that weights each projection using a shrinkage factor; the adverse event-specific shrinkage is a probability, based on empirical Bayes methodology, estimated from all the adverse event data, reflecting evidence in support of the null or non-null hypotheses. Also proposed is a technique to estimate the proportion of true nulls, called the common area under the density curves, which is a critical step in arriving at the shrinkage factor. The performance of the method is evaluated by projecting from interim data and then comparing the projected results with observed results. The method is illustrated on two data sets.

Inference from blinded data in randomized clinical trials.

BACKGROUND: With blinded data, several authors have concluded that there is a negligible chance of inferring a non-null treatment effect. The recent Food and Drug Administration (FDA) draft guidance document on adaptive trials, by encouraging blinded sample size reestimation, implies the same. PURPOSE: We derive methods to investigate whether the probability of inferring a treatment effect is much larger than previously thought, and whether that is of concern. METHODS: A statistic is developed that contributes to improving signal detection. Additionally, trials that are overpowered, for reasons external to powering the primary objective, further strengthen the chance of finding a signal. RESULTS: An example of data from a clinical trial shows how revealing a blinded analysis can be. The ability to infer a non-null effect while a blinded trial is ongoing is a serious matter. LIMITATIONS: The methods apply to superiority trials and are of limited use for non-inferiority or equivalence trials. CONCLUSION: It is important, therefore, that guidance documents include clear language to limit or prevent inference from blinded data to maintain trial integrity. Simple steps are proposed to make inference difficult.

Robust inference from multiple test statistics via permutations: a better alternative to the single test statistic approach for randomized trials.

Formal inference in randomized clinical trials is based on controlling the type I error rate associated with a single pre-specified statistic. The deficiency of using just one method of analysis is that it depends on assumptions that may not be met. For robust inference, we propose pre-specifying multiple test statistics and relying on the minimum p-value for testing the null hypothesis of no treatment effect. The null hypothesis associated with the various test statistics is that the treatment groups are indistinguishable. The critical value for hypothesis testing comes from permutation distributions. Rejection of the null hypothesis when the smallest p-value is less than the critical value controls the type I error rate at its designated value. Even if one of the candidate test statistics has low power, the adverse effect on the power of the minimum p-value statistic is not much. Its use is illustrated with examples. We conclude that it is better to rely on the minimum p-value rather than a single statistic particularly when that single statistic is the logrank test, because of the cost and complexity of many survival trials.

The benefit of stratification in clinical trials revisited.

Stratification is common in clinical trials because it can reduce the variance of the estimated treatment effect. The traditional demonstration of variance reduction relies on the assumption of stratum sizes being fixed quantities. However, in practice, to speed up enrollment, and to obtain a study population with a similar distribution as the overall population, the stratum sizes are allowed to vary. Under the condition that the total sample size is fixed and that the stratum sizes have a multinomial distribution, the criterion changes for achieving a reduction in variance. The relationship between the stratified and unstratified variances is established and shown to be approximately the same for prestratified and post-stratified trials. It is demonstrated why stratification may actually increase the variance compared with no stratification even when the mean square error is reduced on account of stratification. Data from a real clinical trial will be used for illustration. The benefit attributed to stratification needs to be re-examined in light of the findings presented, particularly given its widespread use.

Strengthening the understanding of the relationship between survival designs and test statistics.

In the planning of randomized survival trials, the role of follow-up time of trial participants introduces a level of complexity not encountered in non-survival trials. Of the two commonly used survival designs, one design fixes the follow-up time whereas the other allows it to vary. When the follow-up time is fixed the number of events varies. Conversely, when the number of events is fixed, the follow-up time varies. These two designs influence test statistics in ways that have not been fully explored resulting in a misunderstanding of the design-test statistic relationship. We use examples from the literature to strengthen the understanding of this relationship. Group sequential trials are briefly discussed. When the number of events is fixed, we demonstrate why a two-sample risk difference test statistic reduces to a one-sample test statistic which is nearly equal to the risk ratio test statistic. Some aspects of fixed event designs that need further consideration are also discussed.

Re-estimating the sample size of an on-going blinded trial based on the method of randomization block sums.

We extend a method we had previously described (Statist. Med. 2005) to estimate the within-group variance of a continuous endpoint without breaking the blind in a randomized clinical trial. Specifically, we: (a) explain how the method may be used for a wider set of designs than we had previously indicated; (b) obtain a within-group, covariate-adjusted, blinded variance estimator; (c) illustrate use of the method for sample size re-estimation; and (d) describe a procedure to determine whether or not the blinded variance estimator works well not just on average but for the data set at hand. The proposed method is simple to use and makes no additional assumptions than is made for unblinded analysis. Simulations show that for realistic sample sizes there is virtually no inflation in the Type I error rate. When weighing the burden imposed by interim unblinded re-estimation with the loss in precision with blinded re-estimation, it may be advantageous for some trials to use the blinded method.

Safety and immunogenicity of an intramuscular Helicobacter pylori vaccine in noninfected volunteers: a phase I study.

INTRODUCTION: Helicobacter pylori infection is among the most common human infections and the major risk factor for peptic disease and gastric cancer. Immunization with vaccines containing the H pylori vacuolating cytotoxin A (VacA), cytotoxin-associated antigen (CagA), and neutrophil-activating protein (NAP), alone or in combination, have been shown to prevent experimental infection in animals. AIM: We sought to study the safety and immunogenicity of a vaccine consisting of recombinant VacA, CagA, and NAP given intramuscularly with aluminium hydroxide as an adjuvant to noninfected healthy subjects. METHODS: This controlled, single-blind Phase I study randomized 57 H pylori-negative volunteers into 7 study arms exploring 2 dosages (10 and 25 microg) of each antigen and 3 schedules (0, 1, 2 weeks; 0, 1, 2 months; and 0, 1, 4 months) versus alum controls. All participants were followed for 5 months. Thirty-six subjects received a booster vaccination 18-24 months after the completion of the primary vaccination. RESULTS: Local and systemic adverse reactions were mild and similar in placebo and vaccine recipients on the monthly schedules. All subjects responded to 1 or 2 of the antigens and 86% of all vaccines mounted immunoglobulin G antibody responses to all 3 antigens. Vaccinees exhibited an antigen-specific cellular response. Vaccination 18-24 months later elicited anamnestic antibody and cellular responses. CONCLUSIONS: This intramuscular H pylori vaccine demonstrated satisfactory safety and immunogenicity, produced antigen-specific T-cell memory, and, therefore, warrants further clinical study.