Successful implementation of a stakeholder engagement program for pharmacoepidemiologic research.

PURPOSE: Rigorously conducted pharmacoepidemiologic research requires methodologically complex study designs and analysis yet evaluates problems of high importance to patients and clinicians. Despite this, participation in and mechanisms for stakeholder engagement in pharmacoepidemiologic research are not well-described. Here, we describe our approach and lessons learned from engaging stakeholders, of varying familiarity with research methods, in a rigorous multi-year pharmacoepidemiologic research program evaluating the comparative effectiveness of diabetes medications. METHODS: We recruited 5 patient and 4 clinician stakeholders; each was compensated for their time. Stakeholders received initial formal training in observational research and pharmacoepidemiologic methods sufficient to enable contribution to the research project. After onboarding, stakeholder engagement meetings were held virtually, in the evening, 2-3 times annually. Each was approximately 90 min and focused on 1-2 specific questions about the project, with preparatory materials sent in advance. RESULTS: Stakeholder meeting attendance was high (89%-100%), and all stakeholders engaged with the research project, both during and between meetings. Stakeholders reported positive experiences with meetings, satisfaction, and interest in the research project and its findings, and dedication to the success of the project's goals. They affirmed the value of receiving materials to review in advance and the effectiveness of a virtual platform. Their contributions included prioritizing and suggesting research questions, optimizing written evidence briefs for a lay audience, and guidance on broader topics such as research audience and methods of dissemination. CONCLUSIONS: Stakeholder engagement in pharmacoepidemiologic research using complex study designs and analysis is feasible, acceptable, and positively impacts the research project.

Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials.

Importance: Nonrandomized studies using insurance claims databases can be analyzed to produce real-world evidence on the effectiveness of medical products. Given the lack of baseline randomization and measurement issues, concerns exist about whether such studies produce unbiased treatment effect estimates. Objective: To emulate the design of 30 completed and 2 ongoing randomized clinical trials (RCTs) of medications with database studies using observational analogues of the RCT design parameters (population, intervention, comparator, outcome, time [PICOT]) and to quantify agreement in RCT-database study pairs. Design, Setting, and Participants: New-user cohort studies with propensity score matching using 3 US claims databases (Optum Clinformatics, MarketScan, and Medicare). Inclusion-exclusion criteria for each database study were prespecified to emulate the corresponding RCT. RCTs were explicitly selected based on feasibility, including power, key confounders, and end points more likely to be emulated with real-world data. All 32 protocols were registered on ClinicalTrials.gov before conducting analyses. Emulations were conducted from 2017 through 2022. Exposures: Therapies for multiple clinical conditions were included. Main Outcomes and Measures: Database study emulations focused on the primary outcome of the corresponding RCT. Findings of database studies were compared with RCTs using predefined metrics, including Pearson correlation coefficients and binary metrics based on statistical significance agreement, estimate agreement, and standardized difference. Results: In these highly selected RCTs, the overall observed agreement between the RCT and the database emulation results was a Pearson correlation of 0.82 (95% CI, 0.64-0.91), with 75% meeting statistical significance, 66% estimate agreement, and 75% standardized difference agreement. In a post hoc analysis limited to 16 RCTs with closer emulation of trial design and measurements, concordance was higher (Pearson r, 0.93; 95% CI, 0.79-0.97; 94% meeting statistical significance, 88% estimate agreement, 88% standardized difference agreement). Weaker concordance occurred among 16 RCTs for which close emulation of certain design elements that define the research question (PICOT) with data from insurance claims was not possible (Pearson r, 0.53; 95% CI, 0.00-0.83; 56% meeting statistical significance, 50% estimate agreement, 69% standardized difference agreement). Conclusions and Relevance: Real-world evidence studies can reach similar conclusions as RCTs when design and measurements can be closely emulated, but this may be difficult to achieve. Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.