Variable Selection When Estimating Effects in External Target Populations.

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (i.e., "transport"), some methods require that one account for all effect measure modifiers [EMMs]. However, little is known about how including other variables that are not EMMs (i.e., non-EMMs) in adjustment sets impacts estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing impacts of covariates that A) differ (or not) between the trial and the target, B) are associated with the outcome (or not), and C) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Including variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omitting necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations.

Clarifying the causal contrast: An empirical example applying the prevalent new user study design.

PURPOSE: The prevalent new user design extends the active comparator new user design to include patients switching to a treatment of interest from a comparator. We examined the impact of adding "switchers" to incident new users on the estimated hazard ratio (HR) of hospitalized heart failure. METHODS: Using MarketScan claims data (2000-2014), we estimated HRs of hospitalized heart failure between patients initiating GLP-1 receptor agonists (GLP-1 RA) and sulfonylureas (SU). We considered three estimands: (1) the effect of incident new use; (2) the effect of switching; and (3) the effect of incident new use or switching, combining the two population. We used time-conditional propensity scores (TCPS) and time-stratified standardized morbidity ratio (SMR) weighting to adjust for confounding. RESULTS: We identified 76 179 GLP-1 RA new users, of which 12% were direct switchers (within 30 days) from SU. Among incident new users, GLP-1 RA was protective against heart failure (adjHRSMR = 0.74 [0.69, 0.80]). Among switchers, GLP-1 RA was not protective (adjHRSMR = 0.99 [0.83, 1.18]). Results in the combined population were largely driven by the incident new users, with GLP-1 RA having a protective effect (adjHRSMR = 0.77 [0.72, 0.83]). Results using TCPS were consistent with those estimated using SMR weighting. CONCLUSIONS: When analyses were conducted only among incident new users, GLP-1 RA had a protective effect. However, among switchers from SU to GLP-1 RA, the effect estimates substantially shifted toward the null. Combining patients with varying treatment histories can result in poor confounding control and camouflage important heterogeneity.

Standardizing to Specific Target Populations in Distributed Networks and Multi-Site Pharmacoepidemiologic Studies.

Distributed networks and other multi-site studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multi-site studies may increase interpretability and improve precision. We simulated a four-site study, standardized each site using inverse odds weights (IOW) to resemble the three smallest sites or the smallest site, estimated IOW-weighted RDs, and combined estimates with inverse variance weights (IVW). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of one site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOW reduced differences between estimates and increased precision when targeting the three smallest sites or the smallest site. In the CPRD study, the IOW + IVW estimate was also more precise (smallest region RD and 95% CI: 5.41%, 1.03%-9.79%), IOW+IVW RD and 95% CI: 3.25%, 3.07%-3.43%). When performing pharmacoepidemiologic research in distributed networks or multi-site studies in the presence of EMMs, designating target populations has the potential to improve estimate precision and interpretability.

Visualizing External Validity: Graphical Displays to Inform the Extension of Treatment Effects from Trials to Clinical Practice.

BACKGROUND: In the presence of effect measure modification, estimates of treatment effects from randomized controlled trials may not be valid in clinical practice settings. The development and application of quantitative approaches for extending treatment effects from trials to clinical practice settings is an active area of research. METHODS: In this article, we provide researchers with a practical roadmap and four visualizations to assist in variable selection for models to extend treatment effects observed in trials to clinical practice settings and to assess model specification and performance. We apply this roadmap and visualizations to an example extending the effects of adjuvant chemotherapy (5-fluorouracil vs. plus oxaliplatin) for colon cancer from a trial population to a population of individuals treated in community oncology practices in the United States. RESULTS: The first visualization screens for potential effect measure modifiers to include in models extending trial treatment effects to clinical practice populations. The second visualization displays a measure of covariate overlap between the clinical practice populations and the trial population. The third and fourth visualizations highlight considerations for model specification and influential observations. The conceptual roadmap describes how the output from the visualizations helps interrogate the assumptions required to extend treatment effects from trials to target populations. CONCLUSIONS: The roadmap and visualizations can inform practical decisions required for quantitatively extending treatment effects from trials to clinical practice settings.

External validity in distributed data networks.

PURPOSE: While much has been written about how distributed networks address internal validity, external validity is rarely discussed. We aimed to define key terms related to external validity, discuss how they relate to distributed networks, and identify how three networks (the US Food and Drug Administration's Sentinel System, the Canadian Network for Observational Drug Effect Studies [CNODES], and the National Patient Centered Clinical Research Network [PCORnet]) deal with external validity. METHODS: We define external validity, target populations, target validity, generalizability, and transportability and describe how each relates to distributed networks. We then describe Sentinel, CNODES, and PCORnet and how each approaches these concepts, including a sample case study. RESULTS: Each network approaches external validity differently. As its target population is US citizens and it includes only US data, Sentinel primarily worries about lack of external validity by not including some segments of the population. The fact that CNODES includes Canadian, United States, and United Kingdom data forces them to seriously consider whether the United States and United Kingdom data will be transportable to Canadian citizens when they meta-analyze database-specific estimates. PCORnet, with its focus on study-specific cohorts and pragmatic trials, conducts more case-by-case explorations of external validity for each new analytic data set it generates. CONCLUSIONS: There is no one-size-fits-all approach to external validity within distributed networks. With these networks and comparisons between their findings becoming a key part of pharmacoepidemiology, there is a need to adapt tools for improving external validity to the distributed network setting.

How Choice of Effect Measure Influences Minimally Sufficient Adjustment Sets for External Validity.

Epidemiologic researchers generalizing or transporting effect estimates from a study to a target population must account for effect-measure modifiers (EMMs) on the scale of interest. However, little attention is paid to how the EMMs required may vary depending on the mathematical nuances of each effect measure. We defined 2 types of EMMs: a marginal EMM, where the effect on the scale of interest differs across levels of a variable, and a conditional EMM, where the effect differs conditional on other variables associated with the outcome. These types define 3 classes of variables: class 1 (conditional EMM), class 2 (marginal but not conditional EMM), and class 3 (neither marginal nor conditional EMM). Class 1 variables are necessary to achieve a valid estimate of the RD in a target population, while an RR requires class 1 and class 2 and an OR requires classes 1, 2, and 3 (i.e., all variables associated with the outcome). This does not mean that fewer variables are required for an externally valid RD (because variables may not modify effects on all scales), but it does suggest that researchers should consider the scale of the effect measure when identifying an EMM necessary for an externally valid treatment effect estimate.

Meta-analysis with sample-standardization in multi-site studies.

PURPOSE: To conceptualize a particular target population and estimand for multi-site pharmacoepidemiologic studies within data networks and to analytically examine sample-standardization as a meta-analytic method compared with inverse-variance weighted meta-analyses. METHODS: The target population of interest is all and only all individuals from the data-contributing sites. Standardization, a general conditioning technique frequently employed for confounding control, was adopted to estimate the network-wide causal treatment effect. Specifically, the proposed sample-standardization yields a meta-analysis estimator, that is, a weighted summation of site-specific results, where the weight for a site is the proportion of its size in the entire network. This sample-standardization estimator was evaluated analytically in comparison to estimators from inverse-variance weighted fixed-effect and random-effects meta-analyses in terms of statistical consistency. RESULTS: A proof is reported to justify the consistency of the sample-standardization estimator with and without treatment effect heterogeneity by site. Both inverse-variance weighted fixed-effect and random-effects meta-analyses were found to generally result in inconsistent estimators in the presence of treatment effect heterogeneity by site for this particular target population and estimand. CONCLUSIONS: Sample-standardization is a valid approach to generate causal inference in multi-site studies when the target population comprises all and only all individuals within the network, even in the presence of heterogeneity of treatment effect by site. Multi-site studies should clearly specify the target population and estimand to help select the most appropriate meta-analytic methods.

Comparing Trial and Real-world Adjuvant Oxaliplatin Delivery in Patients With Stage III Colon Cancer Using a Longitudinal Cumulative Dose.

Importance: Delivery of adjuvant chemotherapy can differ substantially between trial and real-world populations. Adherence metrics like relative dose intensity (RDI) cannot capture the timing of modifications and mask differences in the total amount of chemotherapy received. Objective: To compare oxaliplatin delivery between MOSAIC trial participants and patients treated in the US Oncology Network with stage III colon cancer using a longitudinal cumulative dose (LCD). Design, Setting, and Participants: This cohort study used secondary data from the MOSAIC trial, an international randomized clinical trial (concluded in 2004), and electronic health records from US Oncology (2009-2018), a network of community oncology practices in the US. It included participants in MOSAIC with stage III colon cancer who were randomized to receive treatment with oxaliplatin and fluorouracil/leucovorin (n = 663) and US Oncology patients with stage III colon cancer who were treated with a modified FOLFOX-6 regimen (n = 2523). Exposures: Oxaliplatin and fluorouracil/leucovorin. Outcomes and Measures: We evaluated RDI and LCD over time and at the end of treatment in the MOSAIC and US Oncology populations. We used bootstrapping to estimate 95% confidence bands for LCD differences between the populations. Results: The 663 MOSAIC participants (296 women [44.7%]) and 2523 US Oncology patients (1245 women [49.4%]) were generally similar with respect to demographic characteristics. Median RDI was lower in US Oncology (80% in MOSAIC vs 70% in US Oncology). The LCD also suggested differences in the total amount of oxaliplatin received between populations; the final median LCD in US Oncology was 10.2% lower than in MOSAIC, equivalent to receiving 1.2 fewer treatment cycles less of oxaliplatin. This difference only began 133 days into treatment and persisted after accounting for covariates, likely in terms of more frequent oxaliplatin treatment discontinuation in US Oncology patients than their MOSAIC counterparts. Conclusions and Relevance: The study results suggest that real-world patients in community practice in the US treated with modified FOLFOX 6 received less oxaliplatin than their historical counterparts in the MOSAIC trial, with differences manifesting late in the treatment course. The LCD allowed us to identify the amount and extent of these differences, the timing of which was unclear when using RDI alone. Trial Registration: ClinicalTrials.gov identifier: NCT00275210.

A systematic review of whether COVID-19 randomized controlled trials reported on demographic and clinical characteristics.

PURPOSE: We aim to assess the reporting of key patient-level demographic and clinical characteristics among COVID-19 related randomized controlled trials (RCTs). METHODS: We queried English-language articles from PubMed, Web of Science, clinicaltrials.gov, and the CDC library of gray literature databases using keywords of "coronavirus," "covid," "clinical trial" and "randomized controlled trial" from January 2020 to June 2021. From the search, we conducted an initial review to rule-out duplicate entries, identify those that met inclusion criteria (i.e., had results), and exclude those that did not meet the definition of an RCT. Lastly, we abstracted the demographic and clinical characteristics reported on within each RCT. RESULTS: From the initial 43 627 manuscripts, our final eligible manuscripts consisted of 149 RCTs described in 137 articles. Most of the RCTs (113/149) studied potential treatments, while fewer studied vaccines (29), prophylaxis strategies (5), and interventions to prevent transmission among those infected (2). Study populations ranged from 10 to 38 206 participants (median = 100, IQR: 60-300). All 149 RCTs reported on age, 147 on sex, 50 on race, and 110 on the prevalence of at least one comorbidity. No RCTs reported on income, urban versus rural residence, or other indicators of socioeconomic status (SES). CONCLUSIONS: Limited reporting on race and other markers of SES make it difficult to draw conclusions about specific external target populations without making strong assumptions that treatment effects are homogenous. These findings highlight the need for more robust reporting on the clinical and demographic profiles of patients enrolled in COVID-19 related RCTs.

The Timing, the Treatment, the Question: Comparison of Epidemiologic Approaches to Minimize Immortal Time Bias in Real-World Data Using a Surgical Oncology Example.

BACKGROUND: Studies evaluating the effects of cancer treatments are prone to immortal time bias that, if unaddressed, can lead to treatments appearing more beneficial than they are. METHODS: To demonstrate the impact of immortal time bias, we compared results across several analytic approaches (dichotomous exposure, dichotomous exposure excluding immortal time, time-varying exposure, landmark analysis, clone-censor-weight method), using surgical resection among women with metastatic breast cancer as an example. All adult women diagnosed with incident metastatic breast cancer from 2013-2016 in the National Cancer Database were included. To quantify immortal time bias, we also conducted a simulation study where the "true" relationship between surgical resection and mortality was known. RESULTS: 24,329 women (median age 61, IQR 51-71) were included, and 24% underwent surgical resection. The largest association between resection and mortality was observed when using a dichotomized exposure [HR, 0.54; 95% confidence interval (CI), 0.51-0.57], followed by dichotomous with exclusion of immortal time (HR, 0.62; 95% CI, 0.59-0.65). Results from the time-varying exposure, landmark, and clone-censor-weight method analyses were closer to the null (HR, 0.67-0.84). Results from the plasmode simulation found that the time-varying exposure, landmark, and clone-censor-weight method models all produced unbiased HRs (bias -0.003 to 0.016). Both standard dichotomous exposure (HR, 0.84; bias, -0.177) and dichotomous with exclusion of immortal time (HR, 0.93; bias, -0.074) produced meaningfully biased estimates. CONCLUSIONS: Researchers should use time-varying exposures with a treatment assessment window or the clone-censor-weight method when immortal time is present. IMPACT: Using methods that appropriately account for immortal time will improve evidence and decision-making from research using real-world data.

Nondifferential Treatment Misclassification Biases Toward the Null? Not a Safe Bet for Active Comparator Studies.

Active comparator studies are increasingly common, particularly in pharmacoepidemiology. In such studies, the parameter of interest is a contrast (difference or ratio) in the outcome risks between the treatment of interest and the selected active comparator. While it may appear treatment is dichotomous, treatment is actually polytomous as there are at least 3 levels: no treatment, the treatment of interest, and the active comparator. Because misclassification may occur between any of these groups, independent nondifferential treatment misclassification may not be toward the null (as expected with a dichotomous treatment). In this work, we describe bias from independent nondifferential treatment misclassification in active comparator studies with a focus on misclassification that occurs between each active treatment and no treatment. We derive equations for bias in the estimated outcome risks, risk difference, and risk ratio, and we provide bias correction equations that produce unbiased estimates, in expectation. Using data obtained from US insurance claims data, we present a hypothetical comparative safety study of antibiotic treatment to illustrate factors that influence bias and provide an example probabilistic bias analysis using our derived bias correction equations.

Alternative analytic and matching approaches for the prevalent new-user design: A simulation study.

PURPOSE: To describe the creation of prevalent new user (PNU) cohorts and compare the relative bias and computational efficiency of several alternative analytic and matching approaches in PNU studies. METHODS: In a simulated cohort, we estimated the effect of a treatment of interest vs a comparator among those who switched to the treatment of interest using the originally proposed time-conditional propensity score (TCPS) matching, standardized morbidity ratio weighting (SMRW), disease risk scores (DRS), and several alternative propensity score matching approaches. For each analytic method, we compared the average RR (across 2000 replicates) to the known risk ratio (RR) of 1.00. RESULTS: SMRW and DRS yielded unbiased results (RR = 0.998 and 0.997, respectively). TCPS matching with replacement was also unbiased (RR = 0.999). TCPS matching without replacement was unbiased when matches were identified starting with patients with the shortest treatment history as initially proposed (RR = 0.999), but it resulted in very slight bias (RR = 0.983) when starting with patients with the longest treatment history. Similarly, creating a match pool without replacement starting with patients with the shortest treatment history yielded an unbiased estimate (RR = 0.997), but matching with the longest treatment history first resulted in substantial bias (RR = 0.903). The most biased strategy was matching after selecting one random comparator observation per individual that continued on the comparator (RR = 0.802). CONCLUSIONS: Multiple analytic methods can estimate treatment effects without bias in a PNU cohort. Still, researchers should be wary of introducing bias when selecting controls for complex matching strategies beyond the initially proposed TCPS.

Introducing longitudinal cumulative dose to describe chemotherapy patterns over time: Case study of a colon cancer trial.

Adjuvant chemotherapy regimens take months to complete. Despite this, studies evaluate chemotherapy adherence via measures assessed at the end of treatment (eg, number of patients missing any dose, relative dose intensity [RDI]). This approach ignores information like the timing of treatment delays. We propose longitudinal cumulative dose (LCD) to integrate impacts of dose reductions, missed doses and dose delays over time. We obtained data from the 2246 participants in the MOSAIC trial randomized to FOLFOX (all three agents) or 5-FU/LV (only 5-fluorouracil and leucovorin). We evaluated proportions of patients stopping treatment early and reducing, missing or delaying a dose in each arm for each chemotherapy agent at each cycle. We calculated LCD, the fraction of the final standard dose a participant reached by a given day, for each participant and each agent and compared it over time and at 24 weeks between treatment arms. Participants randomized to FOLFOX were more likely to stop treatment, reduce doses, miss doses or delay cycles; these differences increased over time. Median LCD for oxaliplatin in the FOLFOX arm at 24 weeks was 77%. The LCD for 5-fluorouracil differed between arms (FOLFOX arm median: 81%; 5-FU/LV arm median: 96%). Visualizing LCD highlighted the timing of deviations from standard administration in a way RDI could not, with major differences in 5-fluorouracil LCD across treatment arms beginning after the sixth dose. Further evaluation of LCD and its impacts on clinical outcomes may clarify mechanisms for heterogeneous patient outcomes.

Propensity Score Weighting and Trimming Strategies for Reducing Variance and Bias of Treatment Effect Estimates: A Simulation Study.

To extend previous simulations on the performance of propensity score (PS) weighting and trimming methods to settings without and with unmeasured confounding, Poisson outcomes, and various strengths of treatment prediction (PS c statistic), we simulated studies with a binary intended treatment T as a function of 4 measured covariates. We mimicked treatment withheld and last-resort treatment by adding 2 "unmeasured" dichotomous factors that directed treatment to change for some patients in both tails of the PS distribution. The number of outcomes Y was simulated as a Poisson function of T and confounders. We estimated the PS as a function of measured covariates and trimmed the tails of the PS distribution using 3 strategies ("Crump," "Stürmer," and "Walker"). After trimming and reestimation, we used alternative PS weights to estimate the treatment effect (rate ratio): inverse probability of treatment weighting, standardized mortality ratio (SMR)-treated, SMR-untreated, the average treatment effect in the overlap population (ATO), matching, and entropy. With no unmeasured confounding, the ATO (123%) and "Crump" trimming (112%) improved relative efficiency compared with untrimmed inverse probability of treatment weighting. With unmeasured confounding, untrimmed estimates were biased irrespective of weighting method, and only Stürmer and Walker trimming consistently reduced bias. In settings where unmeasured confounding (e.g., frailty) may lead physicians to withhold treatment, Stürmer and Walker trimming should be considered before primary analysis.

Using propensity scores to estimate effects of treatment initiation decisions: State of the science.

Confounding can cause substantial bias in nonexperimental studies that aim to estimate causal effects. Propensity score methods allow researchers to reduce bias from measured confounding by summarizing the distributions of many measured confounders in a single score based on the probability of receiving treatment. This score can then be used to mitigate imbalances in the distributions of these measured confounders between those who received the treatment of interest and those in the comparator population, resulting in less biased treatment effect estimates. This methodology was formalized by Rosenbaum and Rubin in 1983 and, since then, has been used increasingly often across a wide variety of scientific disciplines. In this review article, we provide an overview of propensity scores in the context of real-world evidence generation with a focus on their use in the setting of single treatment decisions, that is, choosing between two therapeutic options. We describe five aspects of propensity score analysis: alignment with the potential outcomes framework, implications for study design, estimation procedures, implementation options, and reporting. We add context to these concepts by highlighting how the types of comparator used, the implementation method, and balance assessment techniques have changed over time. Finally, we discuss evolving applications of propensity scores.

Initiator Types and the Causal Question of the Prevalent New-User Design: A Simulation Study.

New-user designs restricting to treatment initiators have become the preferred design for studying drug comparative safety and effectiveness using nonexperimental data. This design reduces confounding by indication and healthy-adherer bias at the cost of smaller study sizes and reduced external validity, particularly when assessing a newly approved treatment compared with standard treatment. The prevalent new-user design includes adopters of a new treatment who switched from or previously used standard treatment (i.e., the comparator), expanding study sample size and potentially broadening the study population for inference. Previous work has suggested the use of time-conditional propensity-score matching to mitigate prevalent user bias. In this study, we describe 3 "types" of initiators of a treatment: new users, direct switchers, and delayed switchers. Using these initiator types, we articulate the causal questions answered by the prevalent new-user design and compare them with those answered by the new-user design. We then show, using simulation, how conditioning on time since initiating the comparator (rather than full treatment history) can still result in a biased estimate of the treatment effect. When implemented properly, the prevalent new-user design estimates new and important causal effects distinct from the new-user design.

Directed Acyclic Graphs, Effect Measure Modification, and Generalizability.

Directed acyclic graphs (DAGs) have had a major impact on the field of epidemiology by providing straightforward graphical rules for determining when estimates are expected to lack causally interpretable internal validity. Much less attention has been paid, however, to what DAGs can tell researchers about effect measure modification and external validity. In this work, we describe 2 rules based on DAGs related to effect measure modification. Rule 1 states that if a variable, $P$, is conditionally independent of an outcome, $Y$, within levels of a treatment, $X$, then $P$ is not an effect measure modifier for the effect of $X$ on $Y$ on any scale. Rule 2 states that if $P$ is not conditionally independent of $Y$ within levels of $X$, and there are open causal paths from $X$ to $Y$ within levels of $P$, then $P$ is an effect measure modifier for the effect of $X$ on $Y$ on at least 1 scale (given no exact cancelation of associations). We then show how Rule 1 can be used to identify sufficient adjustment sets to generalize nested trials studying the effect of $X$ on $Y$ to the total source population or to those who did not participate in the trial.

Effectiveness of adjuvant FOLFOX vs 5FU/LV in adults over age 65 with stage II and III colon cancer using a novel hybrid approach.

PURPOSE: Estimates of cancer therapy effects can differ in clinical trials and clinical practice, partly due to underrepresentation of certain patient subgroups in trials. We utilize a hybrid approach, combining clinical trial and real-world data, to estimate the comparative effectiveness of two adjuvant chemotherapy regimens for colon cancer. METHODS: We identified patients aged 66 and older enrolled in the Multicenter International Study of Oxaliplatin/5FU-LV in the Adjuvant Treatment of Colon Cancer. Similar patients were identified in the Surveillance, Epidemiology, and End Results (SEER)-Medicare database, initiating adjuvant chemotherapy with either 5-fluorouracil (5FU) alone or in combination with oxaliplatin (FOLFOX). We used logistic regression to estimate the likelihood of trial enrollment as a function of age, sex, and substage. Using inverse odds of sampling weights (IOSW), we compared 5-year mortality in patients randomized to FOLFOX vs 5FU using weighted Cox proportional hazards regression, the Nelson-Aalen estimator for cumulative hazards, and bootstrapping for 95% confidence intervals (CIs). RESULTS: There were 690 trial participants and 3834 SEER-Medicare patients. The SEER-Medicare population was older and had a higher proportion of stage IIIB and IIIC patients than the trial. After controlling for differences between populations, the IOSW 5-year HR was 1.21 (0.89, 1.65), slightly farther from the null than the trial estimate (HR = 1.14, 95%CI: 0.87, 1.49). CONCLUSIONS: This study supports mounting evidence of little to no incremental reduction in 5-year mortality for FOLFOX vs 5FU in older adults with stage II-III colon cancer, emphasizing the importance of combining clinical trial and real-world data to support such conclusions.