A propensity score-integrated approach for leveraging external data in a randomized controlled trial with time-to-event endpoints.

In a randomized controlled trial with time-to-event endpoint, some commonly used statistical tests to test for various aspects of survival differences, such as survival probability at a fixed time point, survival function up to a specific time point, and restricted mean survival time, may not be directly applicable when external data are leveraged to augment an arm (or both arms) of an RCT. In this paper, we propose a propensity score-integrated approach to extend such tests when external data are leveraged. Simulation studies are conducted to evaluate the operating characteristics of three propensity score-integrated statistical tests, and an illustrative example is given to demonstrate how these proposed procedures can be implemented.

Propensity score-incorporated adaptive design approaches when incorporating real-world data.

The propensity score-integrated composite likelihood (PSCL) method is one method that can be utilized to design and analyze an application when real-world data (RWD) are leveraged to augment a prospectively designed clinical study. In the PSCL, strata are formed based on propensity scores (PS) such that similar subjects in terms of the baseline covariates from both the current study and RWD sources are placed in the same stratum, and then composite likelihood method is applied to down-weight the information from the RWD. While PSCL was originally proposed for a fixed design, it can be extended to be applied under an adaptive design framework with the purpose to either potentially claim an early success or to re-estimate the sample size. In this paper, a general strategy is proposed due to the feature of PSCL. For the possibility of claiming early success, Fisher's combination test is utilized. When the purpose is to re-estimate the sample size, the proposed procedure is based on the test proposed by Cui, Hung, and Wang. The implementation of these two procedures is demonstrated via an example.

Principled leveraging of external data in the evaluation of diagnostic devices via the propensity score-integrated composite likelihood approach.

In the area of diagnostics, it is common practice to leverage external data to augment a traditional study of diagnostic accuracy consisting of prospectively enrolled subjects to potentially reduce the time and/or cost needed for the performance evaluation of an investigational diagnostic device. However, the statistical methods currently being used for such leveraging may not clearly separate study design and outcome data analysis, and they may not adequately address possible bias due to differences in clinically relevant characteristics between the subjects constituting the traditional study and those constituting the external data. This paper is intended to draw attention in the field of diagnostics to the recently developed propensity score-integrated composite likelihood approach, which originally focused on therapeutic medical products. This approach applies the outcome-free principle to separate study design and outcome data analysis and can mitigate bias due to imbalance in covariates, thereby increasing the interpretability of study results. While this approach was conceived as a statistical tool for the design and analysis of clinical studies for therapeutic medical products, here, we will show how it can also be applied to the evaluation of sensitivity and specificity of an investigational diagnostic device leveraging external data. We consider two common scenarios for the design of a traditional diagnostic device study consisting of prospectively enrolled subjects, which is to be augmented by external data. The reader will be taken through the process of implementing this approach step-by-step following the outcome-free principle that preserves study integrity.

Propensity score-based methods for causal inference and external data leveraging in regulatory settings: From basic ideas to implementation.

The breakthrough propensity score methodology was formulated by Rosenbaum and Rubin in the 1980s for the mitigation of confounding bias in non-randomized comparative studies to facilitate causal inference for treatment effects. The methodology had been used mainly in epidemiological and social science studies that may often be exploratory, until it was adopted by FDA/CDRH in 2002 and applied in the evaluation of medical device pre-market confirmatory studies, including those with a control group extracted from a well-designed and executed registry database or historical clinical studies. Around 2013, following the Rubin outcome-free study design principle, the two-stage propensity score design framework was developed for medical device studies to safeguard study integrity and objectivity, thereby strengthening the interpretability of study results. Since 2018, the scope of the propensity score methodology has been broadened so that it can be used for the purpose of leveraging external data to augment a single-arm or randomized traditional clinical study. All these statistical approaches, collectively referred to as propensity score-based methods in this article, have been considered in the design of medical device regulatory studies and stimulated related research, as evidenced by the latest trends in journal publications. We will provide a tutorial on the propensity score-based methods from the basic idea to their implementation in regulatory settings for causal inference and external data leveraging, along with step-by-step descriptions of the procedures of the two-stage outcome-free design through examples, which can be used as templates for real study proposals.

Propensity score-integrated approach to survival analysis: leveraging external evidence in single-arm studies.

External data, referred to as data external to the traditional clinical study being planned, include but are not limited to real-world data (RWD) and data collected from clinical studies being conducted in the past or in other countries. The external data are sometimes leveraged to augment a single-arm, prospectively designed study when appropriate. In such an application, recently developed propensity score-integrated approaches including PSPP and PSCL can be used for study design and data analysis when the clinical outcomes are binary or continuous. In this paper, the propensity score-integrated Kaplan-Meier (PSKM) method is proposed for a similar situation but the outcome of interest is time-to-event. The propensity score methodology is used to select external subjects that are similar to those in the current study in terms of baseline covariates and to stratify the selected subjects from both data sources into more homogeneous strata. The stratum-specific PSKM estimators are obtained based on all subjects in the stratum with the external data being down-weighted, and then these estimators are combined to obtain an overall PSKM estimator. A simulation study is conducted to assess the performance of the PSKM method, and an illustrative example is presented to demonstrate how to implement the proposed method.

Estimands in observational studies: Some considerations beyond ICH E9 (R1).

The document ICH E9 (R1) has brought much attention to the concept of estimand in the clinical trials community. ICH stands for International Conference for Harmonization. In this article, we draw attention to one facet of estimand that is not discussed in that document but is crucial in the context of observational studies, namely weighting for covariate balance. How weighting schemes are connected to estimand, or more specifically to one of its five attributes identified in ICH E9 (R1), the attribute of population, is illustrated using the Rubin Causal Model. Three estimands are examined from both theoretical and practical perspectives. Factors that may be considered in choosing among these estimands are discussed.

Leverage multiple real-world data sources in single-arm medical device clinical studies.

The interest in utilizing real-world data (RWD) has been considerably increasing in medical product development and evaluation. With proper usage and analysis of high-quality real-world data, real-world evidence (RWE) can be generated to inform regulatory and healthcare decision-making. This paper proposes a study design and data analysis approach for a prospective, single-arm clinical study that is supplemented with patients from multiple real-world data sources containing patient-level covariate and outcome data. After the amount of information to be borrowed from each real-world data source is determined, the propensity score-integrated composite likelihood method is applied to obtain an estimate of the parameter of interest based on data from the prospective clinical study and this real-world data source. This method is applied to each real-world data source. The final estimate of the parameter of interest is then obtained by taking a weighted average of all these estimates. The performance of the proposed approach is evaluated via a simulation study. A hypothetical example is presented to illustrate how to implement the proposed approach.

Augmenting Both Arms of a Randomized Controlled Trial Using External Data: An Application of the Propensity Score-Integrated Approaches.

Leveraging external data is a topic that have recently received much attention. The propensity score-integrated approaches are a methodological innovation for this purpose. In this paper we adapt these approaches, originally introduced to augment single-arm studies with external data, for the augmentation of both arms of a randomized controlled trial (RCT) with external data. After recapitulating the basic ideas, we provide a step-by-step tutorial of how to implement the propensity score-integrated approaches, from study design to outcome analysis, in the RCT setting in such a way that the study integrity and objectively are maintained. Both the Bayesian (power prior) approach and the frequentist (composite likelihood) approach are included. Some extensions and variations of these approaches are also outlined at the end of this paper.

Propensity score-integrated power prior approach for augmenting the control arm of a randomized controlled trial by incorporating multiple external data sources.

In this paper, a propensity score-integrated power prior approach is developed to augment the control arm of a two-arm randomized controlled trial (RCT) with subjects from multiple external data sources such as real-world data (RWD) and historical clinical studies containing subject-level outcomes and covariates. The propensity scores for the subjects in the external data sources versus the subjects in the RCT are first estimated, and then subjects are placed in different strata based on their estimated propensity scores. Within each propensity score stratum, a power prior is formulated with the information contributed by the external data sources, and Bayesian inference on the treatment effect is obtained. The proposed approach is implemented under the two-stage study design framework utilizing the outcome-free principle to ensure the integrity of a study. An illustrative example is provided to demonstrate the implementation of the proposed approach.

Leveraging real-world evidence for determining performance goals for medical device studies.

Performance goals are numerical target values pertaining to effectiveness or safety endpoints in single-arm medical device clinical studies. Typically, performance goals are determined at the planning stage of the investigational study under consideration based on summarized outcome information from existing relevant clinical trials. In recent years, there is a growing interest in leveraging real-world evidence in medical product development. In this article, we introduce a new method for proposing performance goals by leveraging real-world evidence. The method applies entropy balancing to address possible patient dissimilarities between the study's target patient population and existing real-world patients, and can take into account operation differences between clinical studies and real-world clinical practice. An illustrative example is provided to demonstrate how to implement the proposed method for performance goal determination while leveraging real-world evidence.

Evaluation of diagnostic tests for low prevalence diseases: a statistical approach for leveraging real-world data to accelerate the study.

The evaluation of diagnostic tests usually involves statistical inference for its sensitivity. As sensitivity is defined as the probability that the test result will be positive when the target condition is present, the key study design consideration of sample size is the determination of the number of subjects with the target condition such that the estimation has adequate precision, or the hypothesis testing has adequate power. Traditionally, one may rely on prospective screening of subjects to obtain the required sample size, which means that if the prevalence of the disease is very low, a large number of subjects would need to be screened, increasing the study duration and cost. In this paper, we consider the possibility of substantially reducing the length and cost of a clinical study by leveraging subjects from a real-world data (RWD) source, focusing specifically on the diagnostic test for the cancer of interest. Using the propensity score methodology, we developed a procedure which ensures that the real-world subjects being leveraged are similar to their prospectively enrolled counterparts, thereby making the leveraging more justified. The procedure allows the down-weighting of the real-world subjects, which can be achieved by either using a Frequentist's method based on the composite likelihood or a Bayesian method based on the power prior. The proposed approach can be applied to the evaluation of any diagnostic test and it is not limited to the current clinical study regarding a cancer diagnostic test. Notably, this paper is in close alignment with a recently released draft framework by the Medical Device Innovation Consortium (MDIC) on real-world clinical evidence and in vitro diagnostics, being a showcase of appropriately leveraging real-world data in diagnostic test evaluation for diseases with low prevalence to support regulatory decision-making.

Propensity score-integrated composite likelihood approach for augmenting the control arm of a randomized controlled trial by incorporating real-world data.

In this paper, a propensity score-integrated composite likelihood (PSCL) approach is developed for cases in which the control arm of a two-arm randomized controlled trial (RCT) (treated vs control) is augmented with patients from real-world data (RWD) containing both clinical outcomes and covariates at the patient-level. RWD patients who were treated with the same therapy as the control arm of the RCT are considered for the augmentation. The PSCL approach first estimates the propensity score for every patient as the probability of the patient being in the RCT rather than the RWD, and then stratifies all patients into strata based on the estimated propensity scores. Within each propensity score stratum, a composite likelihood function is specified and utilized to down-weight the information contributed by the RWD source. Estimates of the stratum-specific parameters are obtained by maximizing the composite likelihood function. These stratum-specific estimates are then combined to obtain an overall population-level estimate of the parameter of interest. The performance of the proposed approach is evaluated via a simulation study. A hypothetical two-arm RCT and a hypothetical RWD source are used to illustrate the implementation of the proposed approach.

Propensity score-integrated composite likelihood approach for incorporating real-world evidence in single-arm clinical studies.

In medical product development, there has been an increased interest in utilizing real-world data which have become abundant with recent advances in biomedical science, information technology, and engineering. High-quality real-world data may be analyzed to generate real-world evidence that can be utilized in the regulatory and healthcare decision-making. In this paper, we consider the case in which a single-arm clinical study, viewed as the primary data source, is supplemented with patients from a real-world data source containing both clinical outcome and covariate data at the patient-level. Propensity score methodology is used to identify real-world data patients that are similar to those in the single-arm study in terms of the baseline characteristics, and to stratify these patients into strata based on the proximity of the propensity scores. In each stratum, a composite likelihood function of a parameter of interest is constructed by down-weighting the information from the real-world data source, and an estimate of the stratum-specific parameter is obtained by maximizing the composite likelihood function. These stratum-specific estimates are then combined to obtain an overall population-level estimate of the parameter of interest. The performance of the proposed approach is evaluated via a simulation study. A hypothetical example based on our experience is provided to illustrate the implementation of the proposed approach.

Homogeneity assessment for pivotal medical device clinical studies.

A question that routinely arises in medical device clinical studies is the homogeneity across demographic subgroups, geographical regions, or investigational sites of the enrolled patients in terms of treatment effects or outcome variables. The main objective of this paper is to discuss statistical concepts and methods for the assessment of such homogeneity and to provide the practitioner a statistical framework and points to consider in conducting homogeneity assessment. Demographic subgroups, geographical regions, and investigational sites are discussed separately as each has its unique issues. Specific considerations are also given to randomized controlled trials, non-randomized comparative studies, and single-arm studies. We point out that judicious use of statistical methods, in conjunction with sound clinical judgment, is essential in handling the issue of homogeneity of treatment effect in medical device clinical studies.

Propensity score-integrated power prior approach for incorporating real-world evidence in single-arm clinical studies.

We are now at an amazing time for medical product development in drugs, biological products and medical devices. As a result of dramatic recent advances in biomedical science, information technology and engineering, ``big data'' from health care in the real-world have become available. Although big data may not necessarily be attuned to provide the preponderance of evidence to a clinical study, high-quality real-world data can be transformed into scientific evidence for regulatory and healthcare decision-making using proven analytical methods and techniques, such as propensity score methodology and Bayesian inference. In this paper, we extend the Bayesian power prior approach for a single-arm study (the current study) to leverage external real-world data. We use propensity score methodology to pre-select a subset of real-world data containing patients that are similar to those in the current study in terms of covariates, and to stratify the selected patients together with those in the current study into more homogeneous strata. The power prior approach is then applied in each stratum to obtain stratum-specific posterior distributions, which are combined to complete the Bayesian inference for the parameters of interest. We evaluate the performance of the proposed method as compared to that of the ordinary power prior approach by simulation and illustrate its implementation using a hypothetical example, based on our regulatory review experience.

Good statistical practice in utilizing real-world data in a comparative study for premarket evaluation of medical devices.

Observational (non-randomized) comparative studies have been adopted in the pre-market safety/effectiveness evaluation of medical devices. There has been an increased interest in utilizing this design with the growing available real-world data. However, in such studies, biases that are introduced in every stage and aspect of study need to be addressed. Otherwise, the objectivity of study design and validity of study results will be compromised. In this paper, challenges and opportunities are discussed from the regulatory perspective. Considerations and good statistical practice to mitigate the potential bias are presented.

Utilizing national and international registries to enhance pre-market medical device regulatory evaluation.

Regulatory decisions are made based on the assessment of risk and benefit of medical devices at the time of pre-market approval and subsequently, when post-market risk-benefit balance needs reevaluation. Such assessments depend on scientific evidence obtained from pre-market studies, post-approval studies, post-market surveillance studies, patient perspective information, as well as other real world data such as national and international registries. Such registries provide real world evidence and are playing a more and more important role in enhancing the safety and effectiveness evaluation of medical devices. While these registries provide large quantities of data reflecting real world practice and can potentially reduce the cost of clinical trials, challenges arise concerning (1) data quality adequate for regulatory decision-making, (2) bias introduced at every stage and aspect of study, (3) scientific validity of study designs, and (4) reliability and interpretability of study results. This article will discuss related statistical and regulatory challenges and opportunities with examples encountered in medical device regulatory reviews.

Statistical innovations in diagnostic device evaluation.

Due to rapid technological development, innovations in diagnostic devices are proceeding at an extremely fast pace. Accordingly, the needs for adopting innovative statistical methods have emerged in the evaluation of diagnostic devices. Statisticians in the Center for Devices and Radiological Health at the Food and Drug Administration have provided leadership in implementing statistical innovations. The innovations discussed in this article include: the adoption of bootstrap and Jackknife methods, the implementation of appropriate multiple reader multiple case study design, the application of robustness analyses for missing data, and the development of study designs and data analyses for companion diagnostics.