Real-world evidence for regulatory decision-making: updated guidance from around the world.

Leveraging the value of real-world evidence (RWE) to make informed regulatory decisions in the field of health care continues to gain momentum. Improving clinical evidence generation by evaluating the outcomes and patient experiences at the point-of-care would help achieve the ultimate aim of ensuring that effective and safe treatments are rapidly approved for patient use. In our previous publication, we assessed the global regulatory landscape with respect to RWE and provided a review of the regional availability of frameworks and guidance through May 2021 on the basis of 3 key regulatory elements: regulatory RWE frameworks, data quality guidance, and study methods guidance. In the current review, we have updated and elaborated upon recent developments in the regulatory RWE environment from a regional perspective under the same 3 regulatory elements stated above. In addition, we have also included a new category on procedural guidance. The review also discusses the perceived gaps and potential opportunities for future development and harmonization in this field to support framework establishment in regions without pre-existing RWE policies. Additionally, the article reviews current developments of health technology assessment (HTA) bodies pertaining to RWE and discusses the status of evidentiary alignment among regulators and HTA agencies.

A novel approach to reducing disparities in health outcomes by enhancing interpretation of cancer clinical trials for underrepresented patient groups.

There has been a growing realization, based on emerging evidence from the point of care, that real-world outcomes of patients with cancer are often inferior to those reported in conventional clinical trials. This phenomenon can be attributed in part to deficits in external validity that are present in many studies. Several factors contribute to external validity deficits, including: narrow eligibility criteria; differences between protocol-specified procedures and routine care; and inadequate access to clinical trial participation among underrepresented and socioeconomically disadvantaged groups. As a result, the current body of clinical evidence derived from conventional clinical trials can be inadequate to inform patient-specific treatment decisions at the point of care. Furthermore, lack of practical guidance on how to evaluate the impact of external validity deficits can impede both the design of more generalizable clinical trials and efforts to personalize treatment decisions for individual patients. In this methodological review, we suggest an approach to aid clinicians in such evaluations, providing visual and quantitative methods for assessing the magnitude of, and adjusting for, the impact of external validity deficits in conventional clinical trials. Our methods and visualizations have broad applicability across important areas of real-world medical decision-making and research, providing opportunities to design clinical studies that are more reflective of the diverse needs of patients with cancer, including those excluded from traditional clinical trials due to narrow eligibility criteria, socioeconomic disadvantages, and other systemic barriers to equitable access to healthcare resources.

The design and evaluation of hybrid controlled trials that leverage external data and randomization.

Patient-level data from completed clinical studies or electronic health records can be used in the design and analysis of clinical trials. However, these external data can bias the evaluation of the experimental treatment when the statistical design does not appropriately account for potential confounders. In this work, we introduce a hybrid clinical trial design that combines the use of external control datasets and randomization to experimental and control arms, with the aim of producing efficient inference on the experimental treatment effects. Our analysis of the hybrid trial design includes scenarios where the distributions of measured and unmeasured prognostic patient characteristics differ across studies. Using simulations and datasets from clinical studies in extensive-stage small cell lung cancer and glioblastoma, we illustrate the potential advantages of hybrid trial designs compared to externally controlled trials and randomized trial designs.

Immune-related adverse events associated with immune checkpoint inhibitors: a call to action for collecting and sharing clinical trial and real-world data.

Immune checkpoint inhibitors (ICIs) have revolutionized the treatment of cancer, improving outcomes in patients with advanced malignancies. The use of ICIs in clinical practice, and the number of ICI clinical trials, are rapidly increasing. The use of ICIs in combination with other forms of cancer therapy, such as chemotherapy, radiotherapy, or targeted therapy, is also expanding. However, immune-related adverse events (irAEs) can be serious in up to a third of patients. Critical questions remain surrounding the characteristics and outcomes of irAEs, and how they may affect the overall risk-benefit relationship for combination therapies. This article proposes a framework for irAE classification and reporting, and identifies limitations in the capture and sharing of data on irAEs from current clinical trial and real-world data. We outline key gaps and suggestions for clinicians, clinical investigators, drug sponsors, patients, and other stakeholders to make these critical data more available to researchers for pooled analysis, to advance contemporary understanding of irAEs, and ultimately improve the efficacy of ICIs.

A digital highway for data fluidity and data equity in precision medicine.

Recent technological advances continue to expand the universe of big data in biomedicine along the four axes of variety, veracity, volume, and velocity, fueling innovations in research and discovery while transforming care delivery. These advances allow quantitative capture of multimodal health, behavioral, social, and environmental data from n-of-all in near real-time to support the development of new therapies and personalization of treatment decisions for the n-of-one. Application of advanced analytical methods, including artificial intelligence and machine learning, to these modern data assets can greatly propel our understanding of health and disease, accelerating the development of safer and more effective anticancer therapies. In this perspective, we rationalize the creation of a universally accessible digital highway system as a foundational infrastructure to enable data fluidity in an equitable manner. An interoperable and integrated digital inter-state highway can facilitate efficient derivation of insights from biomedical big data to improve health outcomes and ensure that the U.S. remains at the leading-edge innovations in technology, advanced analytics, and precision medicine.

Impact of Broadening Trial Eligibility Criteria for Patients with Advanced Non-Small Cell Lung Cancer: Real-World Analysis of Select ASCO-Friends Recommendations.

PURPOSE: Cancer clinical trials often accrue slowly or miss enrollment targets. Strict eligibility criteria are a major reason. Restrictive criteria also limit opportunities for patient participation while compromising external validity of trial results. We examined the impact of broadening select eligibility criteria on characteristics and number of patients eligible for trials, using recommendations of the American Society of Clinical Oncology (ASCO) and Friends of Cancer Research. EXPERIMENTAL DESIGN: A retrospective, observational analysis used electronic health record data from ASCO's CancerLinQ Discovery database. Study cohort included patients with advanced non-small cell lung cancer treated from 2011 to 2018. Patients were grouped by traditional criteria [no brain metastases, no other malignancies, and creatinine clearance (CrCl) ≥ 60 mL/minute] and broadened criteria (including brain metastases, other malignancies, and CrCl ≥ 30 mL/minute). RESULTS: The analysis cohort included 10,500 patients. Median age was 68 years, and 73% of patients were White. Most patients had stage IV disease (65%). A total of 5,005 patients (48%) would be excluded from trial participation using the traditional criteria. The broadened criteria, however, would allow 98% of patients (10,346) to be potential participants. Examination of patients included by traditional criteria (5,495) versus those added (4,851) by broadened criteria showed that the number of women, patients aged 75+ years, and those with stage IV cancer was significantly greater using broadened criteria. CONCLUSIONS: This analysis of real-world data demonstrated that broadening three common eligibility criteria has the potential to double the eligible patient population and include trial participants who are more representative of those encountered in practice.See related commentary by Giantonio, p. 2369.

Pre-clinical animal models are poor predictors of human toxicities in phase 1 oncology clinical trials.

BACKGROUND: Our objective was to determine the correlation between preclinical toxicity found in animal models (mouse, rat, dog and monkey) and clinical toxicity reported in patients participating in Phase 1 oncology clinical trials. METHODS: We obtained from two major early-Phase clinical trial centres, preclinical toxicities from investigational brochures and clinical toxicities from published Phase 1 trials for 108 drugs, including small molecules, biologics and conjugates. Toxicities were categorised according to Common Terminology Criteria for Adverse Events version 4.0. Human toxicities were also categorised based on their reported clinical grade (severity). Positive predictive values (PPV) and negative predictive values (NPV) were calculated to determine the probability that clinical studies would/would not show a particular toxicity category given that it was seen in preclinical toxicology analysis. Statistical analyses also included kappa statistics, and Matthews (MCC) and Spearman correlation coefficients. RESULTS: Overall, animal toxicity did not show strong correlation with human toxicity, with a median PPV of 0.65 and NPV of 0.50. Similar results were obtained based on kappa statistics and MCC. CONCLUSIONS: There is an urgent need to assess more novel approaches to the type and conduct of preclinical toxicity studies in an effort to provide better predictive value for human investigation.

Patient-Centered Approach to Benefit-Risk Characterization Using Number Needed to Benefit and Number Needed to Harm: Advanced Non-Small-Cell Lung Cancer.

This work summarizes the benefit and risk of the results of clinical trials submitted to the US Food and Drug Administration of therapies for the treatment of non-small cell lung cancer (NSCLC) using number needed to benefit (NNB) and number needed to harm (NNH) metrics. NNB and NNH metrics have been reported as potentially being more patient centric and more intuitive to medical practitioners than more common metrics, such as the hazard ratio, and valuable to medical practitioners in complementing other metrics, such as the median time to event. This approach involved the characterization of efficacy and safety results in terms of NNB and NNH of 30 clinical trials in advanced NSCLC supporting US Food and Drug Administration approval decisions from 2003 to 2017. We assessed trends of NNB over time of treatment (eg, for programmed death 1 inhibitors) and variation of NNB across subpopulations (eg, characterized by epidermal growth factor receptor mutation, programmed death ligand 1 expression, Eastern Cooperative Oncology Group performance status, age, and extent of disease progression). Furthermore, the evolution of NNB of treatments for advanced NSCLC was charted from 2003 to 2017. Across subpopulations, NNB, on average, was 4 patients for approved targeted therapies in molecularly enriched populations, 11 patients for approved therapies in nonmolecularly enriched populations, and 23 patients for withdrawn or unapproved therapies. Furthermore, the NNB analysis showed variation for attributes of epidermal growth factor receptor mutations, level of programmed death 1 expression, Eastern Cooperative Oncology Group performance status, etc. When considering the best-case subpopulations and available drugs, the NNB frontier reduced from an estimated value of 7.7 in 2003 to an estimated value of 2.5 in 2017 at the estimated median overall survival-equal to 6 months-of an untreated patient.

Status Update on Data Required to Build a Learning Health System.

Wide adoption of electronic health records (EHRs) has raised the expectation that data obtained during routine clinical care, termed "real-world" data, will be accumulated across health care systems and analyzed on a large scale to produce improvements in patient outcomes and the use of health care resources. To facilitate a learning health system, EHRs must contain clinically meaningful structured data elements that can be readily exchanged, and the data must be of adequate quality to draw valid inferences. At the present time, the majority of EHR content is unstructured and locked into proprietary systems that pose significant challenges to conducting accurate analyses of many clinical outcomes. This article details the current state of data obtained at the point of care and describes the changes necessary to use the EHR to build a learning health system.

Machine-Learning and Stochastic Tumor Growth Models for Predicting Outcomes in Patients With Advanced Non-Small-Cell Lung Cancer.

PURPOSE: The prediction of clinical outcomes for patients with cancer is central to precision medicine and the design of clinical trials. We developed and validated machine-learning models for three important clinical end points in patients with advanced non-small-cell lung cancer (NSCLC)-objective response (OR), progression-free survival (PFS), and overall survival (OS)-using routinely collected patient and disease variables. METHODS: We aggregated patient-level data from 17 randomized clinical trials recently submitted to the US Food and Drug Administration evaluating molecularly targeted therapy and immunotherapy in patients with advanced NSCLC. To our knowledge, this is one of the largest studies of NSCLC to consider biomarker and inhibitor therapy as candidate predictive variables. We developed a stochastic tumor growth model to predict tumor response and explored the performance of a range of machine-learning algorithms and survival models. Models were evaluated on out-of-sample data using the standard area under the receiver operating characteristic curve and concordance index (C-index) performance metrics. RESULTS: Our models achieved promising out-of-sample predictive performances of 0.79 area under the receiver operating characteristic curve (95% CI, 0.77 to 0.81), 0.67 C-index (95% CI, 0.66 to 0.69), and 0.73 C-index (95% CI, 0.72 to 0.74) for OR, PFS, and OS, respectively. The calibration plots for PFS and OS suggested good agreement between actual and predicted survival probabilities. In addition, the Kaplan-Meier survival curves showed that the difference in survival between the low- and high-risk groups was significant (log-rank test P < .001) for both PFS and OS. CONCLUSION: Biomarker status was the strongest predictor of OR, PFS, and OS in patients with advanced NSCLC treated with immune checkpoint inhibitors and targeted therapies. However, single biomarkers have limited predictive value, especially for programmed death-ligand 1 immunotherapy. To advance beyond the results achieved in this study, more comprehensive data on composite multiomic signatures is required.

Characterizing the Feasibility and Performance of Real-World Tumor Progression End Points and Their Association With Overall Survival in a Large Advanced Non-Small-Cell Lung Cancer Data Set.

PURPOSE: Large, generalizable real-world data can enhance traditional clinical trial results. The current study evaluates reliability, clinical relevance, and large-scale feasibility for a previously documented method with which to characterize cancer progression outcomes in advanced non-small-cell lung cancer from electronic health record (EHR) data. METHODS: Patients who were diagnosed with advanced non-small-cell lung cancer between January 1, 2011, and February 28, 2018, with two or more EHR-documented visits and one or more systemic therapy line initiated were identified in Flatiron Health's longitudinal EHR-derived database. After institutional review board approval, we retrospectively characterized real-world progression (rwP) dates, with a random duplicate sample to ascertain interabstractor agreement. We calculated real-world progression-free survival, real-world time to progression, real-world time to next treatment, and overall survival (OS) using the Kaplan-Meier method (index date was the date of first-line therapy initiation), and correlations between OS and other end points were assessed at the patient level (Spearman's ρ). RESULTS: Of 30,276 eligible patients,16,606 (55%) had one or more rwP event. Of these patients, 11,366 (68%) had subsequent death, treatment discontinuation, or new treatment initiation. Correlation of real-world progression-free survival with OS was moderate to high (Spearman's ρ, 0.76; 95% CI, 0.75 to 0.77; evaluable patients, n = 20,020), and for real-world time to progression correlation with OS was lower (Spearman's ρ, 0.69; 95% CI, 0.68 to 0.70; evaluable patients, n = 11,902). Interabstractor agreement on rwP occurrence was 0.94 (duplicate sample, n = 1,065) and on rwP date 0.85 (95% CI, 0.81 to 0.89; evaluable patients n = 358 [patients with two independent event captures within 30 days]). Median rwP abstraction time from individual EHRs was 18.0 minutes (interquartile range, 9.7 to 34.4 minutes). CONCLUSION: We demonstrated that rwP-based end points correlate with OS, and that rwP curation from a large, contemporary EHR data set can be reliable, clinically relevant, and feasible on a large scale.

Real-world progression, treatment, and survival outcomes during rapid adoption of immunotherapy for advanced non-small cell lung cancer.

BACKGROUND: Despite the rapid adoption of immunotherapies in advanced non-small cell lung cancer (advNSCLC), knowledge gaps remain about their real-world (rw) performance. METHODS: This retrospective, observational, multicenter analysis used the Flatiron Health deidentified electronic health record-derived database of rw patients with advNSCLC who received treatment with PD-1 and/or PD-L1 (PD-[L]1) inhibitors before July 1, 2017 (N = 5257) and had ≥6 months of follow-up. The authors investigated PD-(L)1 line of treatment and PD-L1 testing rates and the relationship between overall survival (OS) and rw intermediate endpoints: progression-free survival (rwPFS), rw time to progression (rwTTP), rw time to next treatment (rwTTNT), and rw time to discontinuation (rwTTD). RESULTS: First-line PD-(L)1 inhibitor use increased from 0% (in the third quarter of 2014 [Q3 2014]) to 42% (Q2 2017) over the study period. PD-L1 testing also increased (from 3% in Q3 2015 to 70% in Q2 2017). The estimated median OS was 9.3 months (95% CI, 8.9-9.8 months), and the estimated rwPFS was 3.2 months (95% CI, 3.1-3.3 months). Longer OS and rwPFS were associated with ≥50% PD-L1 percentage staining results. Correlations (⍴) between OS and intermediate endpoints were ⍴ = 0.75 (95% CI, 0.73-0.76) for rwPFS and ⍴ = 0.60 (95% CI, 0.57-0.63) for rwTTP, and, for treatment-based intermediate endpoints, correlations were ⍴ = 0.60 (95% CI, 0.56-0.64) for rwTTNT (N = 856) and ⍴ = 0.81 (95% CI, 0.80-0.82) for rwTTD. CONCLUSIONS: The use of first-line PD-(L)1 inhibitors and PD-L1 testing has substantially increased, with better outcomes for patients who have ≥50% PD-L1 percentage staining. Intermediate rw tumor-dynamics estimates were moderately correlated with OS in patients with advNSCLC who received immunotherapy, highlighting the need for optimizing and standardizing rw endpoints to enhance the understanding of patient outcomes outside clinical trials.

Artificial intelligence and machine learning in clinical development: a translational perspective.

Future of clinical development is on the verge of a major transformation due to convergence of large new digital data sources, computing power to identify clinically meaningful patterns in the data using efficient artificial intelligence and machine-learning algorithms, and regulators embracing this change through new collaborations. This perspective summarizes insights, recent developments, and recommendations for infusing actionable computational evidence into clinical development and health care from academy, biotechnology industry, nonprofit foundations, regulators, and technology corporations. Analysis and learning from publically available biomedical and clinical trial data sets, real-world evidence from sensors, and health records by machine-learning architectures are discussed. Strategies for modernizing the clinical development process by integration of AI- and ML-based digital methods and secure computing technologies through recently announced regulatory pathways at the United States Food and Drug Administration are outlined. We conclude by discussing applications and impact of digital algorithmic evidence to improve medical care for patients.

Erratum: Author Correction: Developing and adopting safe and effective digital biomarkers to improve patient outcomes.

[This corrects the article DOI: 10.1038/s41746-019-0090-4.].

An Exploratory Analysis of Real-World End Points for Assessing Outcomes Among Immunotherapy-Treated Patients With Advanced Non-Small-Cell Lung Cancer.

PURPOSE: This pilot study examined the ability to operationalize the collection of real-world data to explore the potential use of real-world end points extracted from data from diverse health care data organizations and to assess how these relate to similar end points in clinical trials for immunotherapy-treated advanced non-small-cell lung cancer. PATIENTS AND METHODS: Researchers from six organizations followed a common protocol using data from administrative claims and electronic health records to assess real-world end points, including overall survival (rwOS), time to next treatment, time to treatment discontinuation (rwTTD), time to progression, and progression-free survival, among patients with advanced non-small-cell lung cancer treated with programmed death 1/programmed death-ligand 1 inhibitors in real-world settings. Data sets included from 269 to 6,924 patients who were treated between January 2011 and October 2017. Results from contributors were anonymized. RESULTS: Correlations between real-world intermediate end points (rwTTD and time to next treatment) and rwOS were moderate to high (range, 0.6 to 0.9). rwTTD was the most consistent end points as treatment detail was available in all data sets. rwOS at 1 year post-programmed death-ligand 1 initiation ranged from 40% to 57%. In addition, rwOS as assessed via electronic health records and claims data fell within the range of median OS values observed in relevant clinical trials. Data sources had been used extensively for research with ongoing data curation to assure accuracy and practical completeness before the initiation of this research. CONCLUSION: These findings demonstrate that real-world end points are generally consistent with each other and with outcomes observed in randomized clinical trials, which substantiates the potential validity of real-world data to support regulatory and payer decision making. Differences observed likely reflect true differences between real-world and protocol-driven practices.

Generating Real-World Tumor Burden Endpoints from Electronic Health Record Data: Comparison of RECIST, Radiology-Anchored, and Clinician-Anchored Approaches for Abstracting Real-World Progression in Non-Small Cell Lung Cancer.

INTRODUCTION: Real-world evidence derived from electronic health records (EHRs) is increasingly recognized as a supplement to evidence generated from traditional clinical trials. In oncology, tumor-based Response Evaluation Criteria in Solid Tumors (RECIST) endpoints are standard clinical trial metrics. The best approach for collecting similar endpoints from EHRs remains unknown. We evaluated the feasibility of a RECIST-based methodology to assess EHR-derived real-world progression (rwP) and explored non-RECIST-based approaches. METHODS: In this retrospective study, cohorts were randomly selected from Flatiron Health's database of de-identified patient-level EHR data in advanced non-small cell lung cancer. A RECIST-based approach tested for feasibility (N = 26). Three non-RECIST approaches were tested for feasibility, reliability, and validity (N = 200): (1) radiology-anchored, (2) clinician-anchored, and (3) combined. Qualitative and quantitative methods were used. RESULTS: A RECIST-based approach was not feasible: cancer progression could be ascertained for 23% (6/26 patients). Radiology- and clinician-anchored approaches identified at least one rwP event for 87% (173/200 patients). rwP dates matched 90% of the time. In 72% of patients (124/173), the first clinician-anchored rwP event was accompanied by a downstream event (e.g., treatment change); the association was slightly lower for the radiology-anchored approach (67%; 121/180). Median overall survival (OS) was 17 months [95% confidence interval (CI) 14, 19]. Median real-world progression-free survival (rwPFS) was 5.5 months (95% CI 4.6, 6.3) and 4.9 months (95% CI 4.2, 5.6) for clinician-anchored and radiology-anchored approaches, respectively. Correlations between rwPFS and OS were similar across approaches (Spearman's rho 0.65-0.66). Abstractors preferred the clinician-anchored approach as it provided more comprehensive context. CONCLUSIONS: RECIST cannot adequately assess cancer progression in EHR-derived data because of missing data and lack of clarity in radiology reports. We found a clinician-anchored approach supported by radiology report data to be the optimal, and most practical, method for characterizing tumor-based endpoints from EHR-sourced data. FUNDING: Flatiron Health Inc., which is an independent subsidiary of the Roche group.