Demonstrating Benefit-Risk Profiles of Novel Therapeutic Strategies in Kidney Transplantation: Opportunities and Challenges of Real-World Evidence.

While great progress has been made in transplantation medicine, long-term graft failure and serious side effects still pose a challenge in kidney transplantation. Effective and safe long-term treatments are needed. Therefore, evidence of the lasting benefit-risk of novel therapies is required. Demonstrating superiority of novel therapies is unlikely via conventional randomized controlled trials, as long-term follow-up in large sample sizes pose statistical and operational challenges. Furthermore, endpoints generally accepted in short-term clinical trials need to be translated to real-world (RW) care settings, enabling robust assessments of novel treatments. Hence, there is an evidence gap that calls for innovative clinical trial designs, with RW evidence (RWE) providing an opportunity to facilitate longitudinal transplant research with timely translation to clinical practice. Nonetheless, the current RWE landscape shows considerable heterogeneity, with few registries capturing detailed data to support the establishment of new endpoints. The main recommendations by leading scientists in the field are increased collaboration between registries for data harmonization and leveraging the development of technology innovations for data sharing under high privacy standards. This will aid the development of clinically meaningful endpoints and data models, enabling future long-term research and ultimately establish optimal long-term outcomes for transplant patients.

Pilot and feasibility studies for pragmatic trials have unique considerations and areas of uncertainty.

BACKGROUND AND OBJECTIVE: Feasibility studies are increasingly being used to support the development of, and investigate uncertainties around, future large-scale trials. The future trial can be designed with either a pragmatic or explanatory mindset. Whereas pragmatic trials aim to inform the choice between different care options and thus, are designed to resemble conditions outside of a clinical trial environment, explanatory trials examine the benefit of a treatment under more controlled conditions. There is existing guidance for designing feasibility studies, but none that explicitly considers the goals of pragmatic designs. We aimed to identify unique areas of uncertainty that are relevant to planning a pragmatic trial. RESULTS: We identified ten relevant domains, partly based on the pragmatic-explanatory continuum indicator summary-2 (PRECIS-2) framework, and describe potential questions of uncertainty within each: intervention development, research ethics, participant identification and eligibility, recruitment of individuals, setting, organization, flexibility of delivery, flexibility of adherence, follow-up, and importance of primary outcome to patients and decision-makers. We present examples to illustrate how uncertainty in these domains might be addressed within a feasibility study. CONCLUSION: Researchers planning a feasibility study in advance of a pragmatic trial should consider feasibility objectives specifically relevant to areas of uncertainty for pragmatic trials.

A review of pragmatic trials found a high degree of diversity in design and scope, deficiencies in reporting and trial registry data, and poor indexing.

OBJECTIVE: We established a large database of trials to serve as a resource for future methodological and ethical analyses. Here, we use meta-data to describe the broad landscape of pragmatic trials including research areas, identification as pragmatic, quality of trial registry data and enrolment. STUDY DESIGN AND SETTING: Trials were identified by a validated search filter and included if a primary report of a health-related randomized trial published January 2014-April 2019. Data were collated from MEDLINE, Web of Science, ClinicalTrials.gov, and full text. RESULTS: 4337 eligible trials were identified from 13,065 records, of which 1988 were registered in ClinicalTrials.gov. Research areas were diverse, with the most common being general and internal medicine; public, environmental and occupational health; and health care sciences and services. The term "pragmatic" was seldom used in titles or abstracts. Several domains in ClinicalTrials.gov had questionable data quality. We estimated that one-fifth of trials under-accrued by at least 15%. CONCLUSION: There is a need to improve reporting of pragmatic trials and quality of trial registry data. Under accrual remains a challenge in pragmatic RCTs despite calls for more streamlined recruitment approaches. The diversity of pragmatic trials should be reflected in future ethical analyses.

Streamlining the institutional review board process in pragmatic randomized clinical trials: challenges and lessons learned from the Aspirin Dosing: A Patient-centric Trial Assessing Benefits and Long-Term Effectiveness (ADAPTABLE) trial.

BACKGROUND: New considerations during the ethical review processes may emerge from innovative, yet unfamiliar operational methods enabled in pragmatic randomized controlled trials (RCT), potentially making institutional review board (IRB) evaluation more complex. In this manuscript, key components of the pragmatic "Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long-term Effectiveness (ADAPTABLE)" randomized trial that required a reappraisal of the IRB submission, review, and approval processes are discussed. MAIN TEXT: ADAPTABLE is a pragmatic, multicenter, open-label RCT evaluating the comparative effectiveness of two doses of aspirin widely used for secondary prevention (81 mg and 325 mg) in 15,000 patients with an established history of atherosclerotic cardiovascular disease. The electronic informed consent form is completed online by the participants at the time of enrollment, and endpoint ascertainment is conducted through queries of electronic health records. IRB challenges encountered regarding centralized IRB evaluation, electronic informed consent, patient engagement, and risk determination in ADAPTABLE are described in this manuscript. The experience of ADAPTABLE encapsulates how pragmatic protocol components intended to facilitate the study conduct have been tempered by unexpected, yet justified concerns raised by local IRBs. How the lessons learned can be applied to future similar pragmatic trials is delineated. CONCLUSION: Development of engaging communication channels between IRB and study personnel in pragmatic randomized trials as early as at the time of protocol design allows to reduce issues with IRB approval. Integrations of the lessons learned in ADAPTABLE regarding the IRB process for centralized IRBs, informed consent, patient engagement, and risk determination can be emulated and will be instrumental in future pragmatic studies.

Individual point-of-care trials: a new approach to incorporate patient's preferences into personalized pragmatic clinical trials.

Although Evidence-based medicine (EBM) and Patient-centered medicine (PCM) are often perceived as two conflicting paradigms that speak the language of populations and the language of individuals, respectively, both share the common objective of improving the care of individual patients. As physicians should not practice an EBM that is away from the individual patient nor a PCM that is not based on the best available evidence, it is crucial to connect and combine both movements, promoting the fruitful and natural interaction between research and care. Achieving such interaction requires developing new individual-patient centric research methods. In this commentary, we propose an innovative clinical research design oriented to personalize point-of-care trials-integrating clinical research and medical care-through the incorporation of individual patients' preferences to build personalized research protocols. Building on the framework of N-of-1 studies, in "individual point-of-care trials," each protocol could be personalized for each patient so that the therapeutic objectives, the outcome variables analyzed, and the (operationalization of the) compared interventions would be based not only on the clinical and biological characteristics of each patient but also on their individual preferences, goals, and values. If patient preferences are being progressively integrated into medical practice, it makes sense that they also are incorporated into clinical trials embedded in care delivery. The proposal to perform individual point of care trials may be an optimal way to combine EBM and PCM while preserving their foundational principles, and to ensure the connection between "personalized" and "personal" care.

Use of Real-World Evidence to Support FDA Approval of Oncology Drugs.

OBJECTIVES: Real-world evidence (RWE) has gained increased attention in recent years as a complement to traditional clinical trials. The use of RWE to establish the efficacy of oncology drugs for Food and Drug Administration (FDA) approval has not been described. In this paper, we review 5 recent examples where RWE was submitted in support of the FDA approvals of original or supplementary indications for oncology drugs. METHODS: To identify cases where RWE was used, we reviewed drug approval packages available at Drugs@FDA for oncology drugs approved between 2017 and 2019. Five cases were selected to present a broad overview of different types of RWE, different circumstances under which RWE has been used for regulatory approvals, and how FDA evaluated the data in each case. The type of RWE submitted, the indication, limitations identified by FDA reviewers, and the outcome of the submission are discussed. RESULTS: RWE, particularly historical controls for rare or orphan indications, has been used to support both original and supplementary oncology drug approvals. Types of RWE included data from electronic health records, claims, post-marketing safety reports, retrospective medical record reviews, and expanded access studies. Small sample sizes, data quality, and methodological issues were among concerns cited by FDA reviewers. CONCLUSION: By bridging the gap between the constraints of the trial setting and the realities of clinical practice, RWE can add value to a regulatory submission. These early examples provide insight into how regulators evaluated RWE submitted as evidence of efficacy for oncology drugs.

A Call to Action to Track Generic Drug Quality Using Real-World Data and the FDA's Sentinel Initiative.

DISCLOSURES: The author reports funding from the U.S. Food and Drug Administration to study real-world data approaches to detect generic drug quality issues. There are no further conflicts or disclosures.

Are Novel, Nonrandomized Analytic Methods Fit for Decision Making? The Need for Prospective, Controlled, and Transparent Validation.

Real-world data and patient-level data from completed randomized controlled trials are becoming available for secondary analysis on an unprecedented scale. A range of novel methodologies and study designs have been proposed for their analysis or combination. However, to make novel analytical methods acceptable for regulators and other decision makers will require their testing and validation in broadly the same way one would evaluate a new drug: prospectively, well-controlled, and according to a pre-agreed plan. From a European regulators' perspective, the established methods qualification advice procedure with active participation of patient groups and other decision makers is an efficient and transparent platform for the development and validation of novel study designs.

Building internal capacity in pragmatic trials: a workshop for program scientists at the US National Cancer Institute.

BACKGROUND: Building capacity in research funding organizations to support the conduct of pragmatic clinical trials is an essential component of advancing biomedical and public health research. To date, efforts to increase the ability to design and carry out pragmatic trials have largely focused on training researchers. To complement these efforts, we developed an interactive workshop tailored to meet the roles and responsibilities of program scientists at the National Cancer Institute-the leading cancer research funding agency in the USA. The objectives of the workshop were to improve the understanding of pragmatic trials and enhance the capacity to distinguish between elements that make a trial more pragmatic or more explanatory among key programmatic staff. To our knowledge, this is the first reported description of such a workshop. MAIN BODY: The workshop was developed to meet the needs of program scientists as researchers and stewards of research funds, which often includes promoting scientific initiatives, advising prospective applicants, collaborating with grantees, and creating training programs. The workshop consisted of presentations from researchers with expertise in the design and interpretation of trials across the explanatory-pragmatic continuum. Presentations were followed by interactive, small-group exercises to solidify participants' understanding of the purpose and conduct of these trials, which were tailored to attendees' areas of expertise across the cancer control continuum and designed to reflect their scope of work as program scientists at NCI. A total of 29 program scientists from the Division of Cancer Control and Population Sciences and the Division of Cancer Prevention participated; 19 completed a post-workshop evaluation. Attendees were very enthusiastic about the workshop: they reported improved knowledge, significant relevance of the material to their work, and increased interest in pragmatic trials across the cancer control continuum. CONCLUSION: Training program scientists at major biomedical research agencies who are responsible for developing funding opportunities and advising grantees is essential for increasing the quality and quantity of pragmatic trials. Together with workshops for other target audiences (e.g., academic researchers), this approach has the potential to shape the future of pragmatic trials and continue to generate more and better actionable evidence to guide decisions that are of critical importance to health care practitioners, policymakers, and patients.

Are trials of psychological and psychosocial interventions for schizophrenia and psychosis included in the NICE guidelines pragmatic? A systematic review.

INTRODUCTION: The NICE clinical guidelines on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on the results of randomized controlled trials (RCTs), which may not be studies with a pragmatic design, leading to uncertainty on applicability or recommendations to everyday clinical practice. AIM: To assess the level of pragmatism of the evidence used to develop the NICE guideline for psychosocial interventions in psychoses. MATERIAL AND METHODS: We conducted a systematic and critical appraisal of RCTs used to develop the 'psychological therapy and psychosocial interventions' section of the NICE guideline on the treatment and management of psychosis and schizophrenia in adults, published in 2014. For each study we assessed pragmatism using the pragmatic-explanatory continuum indicator summary-2 (PRECIS-2) and the Cochrane risk of bias tool. The mean score of PRECIS-2, averaging across nine domains, was calculated to describe the level of pragmatism of each individual study. RESULTS: A total of 143 studies were included in the analysis. Based on the PRECIS-2 tool, 16.8% were explanatory, 33.6% pragmatic, and 49.7% were rated in an intermediate category. Compared to explanatory studies, pragmatic studies showed a lower risk of bias. Additionally, pragmatism did not significantly improve over time, and no associations were found between pragmatism and a number of trial characteristics. However, studies with a UK leading investigator had the highest mean score of pragmatism. Cognitive behavioural therapy (CBT), art therapy, family intervention, psychoeducation, and adherence therapy, showed the higher average pragmatism scores. CONCLUSIONS: Two third of studies used to produce NICE recommendations on psychosocial interventions for the treatment of schizophrenia and psychosis in adults are based on studies that did not employ a pragmatic design.

Pragmatic trials and implementation science: grounds for divorce?

BACKGROUND: The paper opens with a brief history of two of the major intellectual components of the recent utilitarian turn in clinical research, namely 'pragmatic trials' and 'implementation science'. The two schools of thought developed independently and the paper scrutinises their mutual compatibilities and incompatibilities, asking: i) what do the leading advocates of pragmatic trials assume about the transfer of research findings to real-world practice and ii) what role pragmatic trials can and should play in the evaluation of implementation science strategies. METHODS: The paper utilises 'explication de texte': i) providing a close reading of the inferential logics contained in major published expositions of the two paradigms, and ii) interrogating the conclusions of a pragmatic trial of an intervention providing guidelines on retinal screening aimed at family practitioners. RESULTS: The paper is in two parts. Part 1 unearths some significant incommensurability - the pragmatic trial literature retains an antiquated view of knowledge transfer and is overly optimistic about the wide applicability the findings of pragmatic trials to 'real world' conditions. Part 2 of the paper outlines an empirical strategy to better penetrate the mechanisms of knowledge transfer and to tackle the issue of the generalisabilty of research findings in implementation science. CONCLUSIONS: Pragmatism, classically, is about problem solving and the melding of perspectives. The core research requirement in implementation science is a fundamental shift from the narrow shoulders of pragmatic trials to a model of explanation building based upon a multi-case, multi-method body of evidence.

The "pragmatic trial": An essentially contested concept?

For over 50 years, clinical research methodology has wrestled with the problem of the lack of correspondence between tests of treatments and applications of treatments. The former comprise of trials featuring scrupulous control of patient eligibility, treatment compliance, clinician expertise, follow-up intensity, and so on. In applying a validated treatment, the practitioner has to confront considerable real-world variation in potential patients and in implementation regimes. The remedy, going by the name of "pragmatic trials," is to conduct clinical trials in conditions corresponding more closely to everyday practice. This solution has proved easier to utter than to execute, and the paper reviews the extensive literature on pragmatic trials, seeking to assess whether it has terminated in clarity or contestation.

Pragmatic trials: ignoring a mediator and adjusting for confounding.

OBJECTIVES: In pragmatic trials, the new treatment is compared with usual care (heterogeneous control arm) that makes the comparison of the new treatment with each treatment within the control arm more difficult. The usual assumption is that we can fully capture the relations between different quantities. In this paper we use simulation to assess the performance of statistical methods that adjust for confounding when the assumed relations are not true. The true relations contain a mediator and heterogeneity with or without confounding, but the assumption is that there is no mediator and that confounding and heterogeneity are fully captured. The statistical methods that are compared include multivariable logistic regression, propensity score, disease risk score, inverse probability weighting, doubly robust inverse probability weighting and standardisation. RESULTS: The misconception that there is no mediator can cause to misleading comparative effectiveness of individual treatments when a method that estimates the conditional causal effect is used. Using a method that estimates the marginal causal effect is a better approach, but not for all scenarios.

Pragmatic clinical trials in the context of regulation of medicines.

The pragmatic clinical trial addresses scientific questions in a setting close to routine clinical practice and sometimes using routinely collected data. From a regulatory perspective, when evaluating a new medicine before approving marketing authorization, there will never be enough patients studied in all subgroups that may potentially be at higher risk for adverse outcomes, or sufficient patients to detect rare adverse events, or sufficient follow-up time to detect late adverse events that require long exposure times to develop. It may therefore be relevant that post-marketing trials sometimes have more pragmatic characteristics, if there is a need for further efficacy and safety information. A pragmatic study design may reflect a situation close to clinical practice, but may also have greater potential methodological concerns, e.g. regarding the validity and completeness of data when using routinely collected information from registries and health records, the handling of intercurrent events, and misclassification of outcomes. In a regulatory evaluation it is important to be able to isolate the effect of a specific product or substance, and to have a defined population that the results can be referred to. A study feature such as having a wide and permissive inclusion of patients might therefore actually hamper the utility of the results for regulatory purposes. Randomization in a registry-based setting addresses confounding that could otherwise complicate a corresponding non-interventional design, but not any other methodological issues. Attention to methodological basics can help generate reliable study results, and is more important than labelling studies as 'pragmatic'.

Developing a framework for the ethical design and conduct of pragmatic trials in healthcare: a mixed methods research protocol.

BACKGROUND: There is a widely recognized need for more pragmatic trials that evaluate interventions in real-world settings to inform decision-making by patients, providers, and health system leaders. Increasing availability of electronic health records, centralized research ethics review, and novel trial designs, combined with support and resources from governments worldwide for patient-centered research, have created an unprecedented opportunity to advance the conduct of pragmatic trials, which can ultimately improve patient health and health system outcomes. Such trials raise ethical issues that have not yet been fully addressed, with existing literature concentrating on regulations in specific jurisdictions rather than arguments grounded in ethical principles. Proposed solutions (e.g. using different regulations in "learning healthcare systems") are speculative with no guarantee of improvement over existing oversight procedures. Most importantly, the literature does not reflect a broad vision of protecting the core liberty and welfare interests of research participants. Novel ethical guidance is required. We have assembled a team of ethicists, trialists, methodologists, social scientists, knowledge users, and community members with the goal of developing guidance for the ethical design and conduct of pragmatic trials. METHODS: Our project will combine empirical and conceptual work and a consensus development process. Empirical work will: (1) identify a comprehensive list of ethical issues through interviews with a small group of key informants (e.g. trialists, ethicists, chairs of research ethics committees); (2) document current practices by reviewing a random sample of pragmatic trials and surveying authors; (3) elicit views of chairs of research ethics committees through surveys in Canada, UK, USA, France, and Australia; and (4) elicit views and experiences of community members and health system leaders through focus groups and surveys. Conceptual work will consist of an ethical analysis of identified issues and the development of new ethical solutions, outlining principles, policy options, and rationales. The consensus development process will involve an independent expert panel to develop a final guidance document. DISCUSSION: Planned output includes manuscripts, educational materials, and tailored guidance documents to inform and support researchers, research ethics committees, journal editors, regulators, and funders in the ethical design and conduct of pragmatic trials.

Applying the pragmatic-explanatory continuum indicator summary to the implementation of a physical activity coaching trial in chronic obstructive pulmonary disease.

BACKGROUND: Observational studies show that physical inactivity is associated with worse outcomes in chronic obstructive pulmonary disease (COPD). Despite practice guidelines recommending regular physical activity (PA), there are no large-scale experimental studies to confirm that patients at high risk for COPD exacerbations can increase their PA and consequently, have improved outcomes. PURPOSE: The purpose of this case study is to describe the use of a widely accepted pragmatic trials framework for the design and implementation of a pragmatic clinical trial (PCT) of PA coaching for COPD in a real-world setting. METHOD: The aim of the trial was to determine the effectiveness of a 12-month PA coaching intervention (Walk On!) compared to standard care for 2,707 patients at high risk for COPD exacerbations from a large integrated health care system. The descriptions of our implementation experiences are anchored within the pragmatic-explanatory continuum indicator summary (PRECIS-2) framework. DISCUSSION: Facilitators of PCT implementation include early and ongoing engagement and support of multiple stakeholders including patients, health system leaders, administrators, physician champions, and frontline clinicians, an organizational/setting that prioritizes positive lifestyle behaviors, and a flexible intervention that allows for individualization. Pragmatic challenges include reliance on electronic data that are not complete or available in real-time for patient identification, timing of outreach may not synchronize with patients' readiness for change, and high turnover of clinical staff drawn from the existing workforce. DISCUSSION: PRECIS-2 is a useful guide for organizing decisions about study designs and implementation approaches to help diverse stakeholders recognize the compromises between internal and external validity with those decisions.