Toward Real-World Reproducibility: A Systematic Review and Meta-Analysis of Common Rwd/Rwe Protocol Components

ABSTRACT

Objectives: Use of real-world data/real-world evidence (RWD/RWE) in the life sciences is accelerating. The FDA has issued draft guidance for the conduct of real-world data-driven studies in clinical development. However, RWD protocol development standards lag, leading to heterogeneity of findings and consequent unreliability of results. The need to address challenges has become urgent due to increasing importance of reliable evidence generation in the COVID-19 pandemic. We performed a systematic review of published RWD protocols to understand current practices to support improvement in standards frameworks. Methods: We extracted protocols referencing RWD from. We defined essential real-world study concepts and mapped them to standard discrete protocol components. We summarized these components, including but not limited to objectives, operational definitions of endpoints, inclusion/exclusion criteria, patient identification algorithms, schematics, extract/transform/load (ETL) methods, common data model (CDM), safety, analysis, and machine learning (ML)/artificial intelligence (AI). We identified areas of harmonization and disagreement, as well as missing components. Results: The search identified 220 real-world protocols. Despite substantial harmonization in some areas, particularly those components typical to all research studies, there was considerable disagreement regarding the representation of RWD objectives, RWD inclusion/exclusion criteria, data management, ETL, CDM, ML/AI, study design, and statistical analysis. In many cases, studies did not include real-world-specific elements at all. Quantification and statistical attribution of heterogeneity is ongoing. Conclusions: Incorporating best practices and harmonization of protocol development methods and reporting may lead to improved quality, consistency, and reproducibility of studies. The primary limitation of this study was that “real-world” was neither sensitive nor specific as a search term as it is often used imprecisely. Follow-up surveillance studies will quantify and evaluate the impact of improved standards, encompassing all registered observational studies.