Sampling weighting strategies in causal mediation analysis.

BACKGROUND: Causal mediation analysis plays a crucial role in examining causal effects and causal mechanisms. Yet, limited work has taken into consideration the use of sampling weights in causal mediation analysis. In this study, we compared different strategies of incorporating sampling weights into causal mediation analysis. METHODS: We conducted a simulation study to assess 4 different sampling weighting strategies-1) not using sampling weights, 2) incorporating sampling weights into mediation "cross-world" weights, 3) using sampling weights when estimating the outcome model, and 4) using sampling weights in both stages. We generated 8 simulated population scenarios comprising an exposure (A), an outcome (Y), a mediator (M), and six covariates (C), all of which were binary. The data were generated so that the true model of A given C and the true model of A given M and C were both logit models. We crossed these 8 population scenarios with 4 different sampling methods to obtain 32 total simulation conditions. For each simulation condition, we assessed the performance of 4 sampling weighting strategies when calculating sample-based estimates of the total, direct, and indirect effects. We also applied the four sampling weighting strategies to a case study using data from the National Survey on Drug Use and Health (NSDUH). RESULTS: Using sampling weights in both stages (mediation weight estimation and outcome models) had the lowest bias under most simulation conditions examined. Using sampling weights in only one stage led to greater bias for multiple simulation conditions. DISCUSSION: Using sampling weights in both stages is an effective approach to reduce bias in causal mediation analyses under a variety of conditions regarding the structure of the population data and sampling methods.

Estimating generalized propensity scores with survey and attrition weighted data.

Prior work in causal inference has shown that using survey sampling weights in the propensity score estimation stage and the outcome model stage for binary treatments can result in a more robust estimator of the effect of the binary treatment being analyzed. However, to date, extending this work to continuous treatments and exposures has not been explored nor has consideration been given for how to handle attrition weights in the propensity score model. Nonetheless, generalized propensity score (GPS) analyses are being used for estimating continuous treatment effects on outcomes when researchers have observational data, and those data sets often have survey or attrition weights that need to be accounted for in the analysis. Here, we extend prior work and show with analytic results that using survey sampling or attrition weights in the GPS estimation stage and the outcome model stage for continuous treatments can result in a more robust estimator than one that does not. Simulation study results show that, although using weights in both estimation stages is sufficient for robust estimation, it is not necessary and unbiased estimation is possible in some cases under various approaches to using weights in estimation. Analysts do not know if the conditions of our simulation studies hold, so use of weights in both estimation stages might provide insurance for reducing potential bias. We discuss the implications of our results in the context of an empirical example.

A Tutorial for Propensity Score Weighting for Moderation Analysis With Categorical Variables: An Application Examining Smoking Disparities Among Sexual Minority Adults.

OBJECTIVE: To provide step-by-step guidance and STATA and R code for using propensity score (PS) weighting to estimate moderation effects with categorical variables. RESEARCH DESIGN: Tutorial illustrating the key steps for estimating and testing moderation using observational data. Steps include: (1) examining covariate overlap across treatment groups within levels of the moderator; (2) estimating the PS weights; (3) evaluating whether PS weights improved covariate balance; (4) estimating moderated treatment effects; and (5) assessing the sensitivity of findings to unobserved confounding. Our illustrative case study uses data from 41,832 adults from the 2019 National Survey on Drug Use and Health to examine if gender moderates the association between sexual minority status (eg, lesbian, gay, or bisexual [LGB] identity) and adult smoking prevalence. RESULTS: For our case study, there were no noted concerns about covariate overlap, and we were able to successfully estimate the PS weights within each level of the moderator. Moreover, balance criteria indicated that PS weights successfully achieved covariate balance for both moderator groups. PS-weighted results indicated there was significant evidence of moderation for the case study, and sensitivity analyses demonstrated that results were highly robust for one level of the moderator but not the other. CONCLUSIONS: When conducting moderation analyses, covariate imbalances across levels of the moderator can cause biased estimates. As demonstrated in this tutorial, PS weighting within each level of the moderator can improve the estimated moderation effects by minimizing bias from imbalance within the moderator subgroups.

Comparative effectiveness of individual versus family-based substance use treatment on adolescent self-injurious thoughts and behaviors.

INTRODUCTION: Self-injurious thoughts and behaviors (SITB) are of increasing concern among adolescents, especially those who use substances. Some evidence suggests that existing evidence-based substance use treatments (EBTs) could impact not only their intended substance use targets but also SITB. However, which types of substance use treatments may have the greatest impact on youth SITB is not yet clear. Based on prior literature showing that family support and connection may buffer youth from SITB, we initially hypothesized that family-based EBTs would show greater improvement in SITB compared to those receiving individually focused EBTs and that the size of the effects would be small given the comparison between two active, evidence-based interventions, and base rates of SITB. METHODS: In a sample of 2893 youth in substance use treatment, we compared the effectiveness of individually and family-based EBTs in reducing SITBs. The study used entropy balancing and regression modeling to balance the groups on pre-treatment characteristics and examine change in outcomes over a one-year follow-up period. RESULTS: Both groups improved in self-injury and suicide attempts over the one-year study period, but only youth in individual treatment improved in suicidal ideation. However, the study found no significant difference between the changes over time in the two groups for any outcome. As expected, effect sizes were small and power was constrained in this study given the rarity of the outcomes, but effect sizes are similar to those observed with substance use outcomes. CONCLUSIONS: The results provide important exploratory evidence on the potential relative effectiveness of these two treatments for SITBs. This study supports prior findings that EBTs for youth substance use may help to improve SITB and suggests that different treatment formats (individual or family-based) could result in different benefits for SITB outcomes.

Nonparametric Estimation of Population Average Dose-Response Curves using Entropy Balancing Weights for Continuous Exposures.

Weighted estimators are commonly used for estimating exposure effects in observational settings to establish causal relations. These estimators have a long history of development when the exposure of interest is binary and where the weights are typically functions of an estimated propensity score. Recent developments in optimization-based estimators for constructing weights in binary exposure settings, such as those based on entropy balancing, have shown more promise in estimating treatment effects than those methods that focus on the direct estimation of the propensity score using likelihood-based methods. This paper explores recent developments of entropy balancing methods to continuous exposure settings and the estimation of population dose-response curves using nonparametric estimation combined with entropy balancing weights, focusing on factors that would be important to applied researchers in medical or health services research. The methods developed here are applied to data from a study assessing the effect of non-randomized components of an evidence-based substance use treatment program on emotional and substance use clinical outcomes.

Best practices in statistical computing.

The world is becoming increasingly complex, both in terms of the rich sources of data we have access to and the statistical and computational methods we can use on data. These factors create an ever-increasing risk for errors in code and the sensitivity of findings to data preparation and the execution of complex statistical and computing methods. The consequences of coding and data mistakes can be substantial. In this paper, we describe the key steps for implementing a code quality assurance (QA) process that researchers can follow to improve their coding practices throughout a project to assure the quality of the final data, code, analyses, and results. These steps include: (i) adherence to principles for code writing and style that follow best practices; (ii) clear written documentation that describes code, workflow, and key analytic decisions; (iii) careful version control; (iv) good data management; and (v) regular testing and review. Following these steps will greatly improve the ability of a study to assure results are accurate and reproducible. The responsibility for code QA falls not only on individual researchers but institutions, journals, and funding agencies as well.

Switching cluster membership in cluster randomized control trials: Implications for design and analysis.

Randomized control trials (RCTs) often use clustered designs, where intact clusters (such as classroom, schools, or treatment centers) are randomly assigned to treatment and control conditions. Hierarchical linear models (HLMs) are used almost universally to estimate the effects in such experiments. While study designs that utilize intact clusters have many potential advantages, there is little guidance in the literature on how to respond when cluster switching induces noncompliance with the randomization protocol. In the presence of noncompliance the intent-to-treat (ITT) effect becomes the estimand of interest. When fitting the HLM, these individuals who switch clusters can be assigned to either their as-assigned cluster (the cluster they belonged to at the time of randomization) or their as-treated cluster (the cluster they belonged to at the time the outcome was collected). We show analytically and via simulation, that using the as-treated cluster in HLM will bias the estimate of the ITT effect and using the as-assigned cluster will bias the standard error estimates when heterogeneity among clusters is because of heterogeneity in treatment effects. We show that using linear regression with two-way cluster adjusted standard errors can yield unbiased ITT estimates and consistent standard errors regardless of the source of the random effects. We recommend this method replace HLM as the method of choice for testing intervention effects with cluster-randomized trials with noncompliance and cluster switching. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy.

BACKGROUND: Analysis of covariance (ANCOVA) is commonly used to adjust for potential confounders in observational studies of intervention effects. Measurement error in the covariates used in ANCOVA models can lead to inconsistent estimators of intervention effects. While errors-in-variables (EIV) regression can restore consistency, it requires surrogacy assumptions for the error-prone covariates that may be violated in practical settings. OBJECTIVES: The objectives of this article are (1) to derive asymptotic results for ANCOVA using EIV regression when measurement errors may not satisfy the standard surrogacy assumptions and (2) to demonstrate how these results can be used to explore the potential bias from ANCOVA models that either ignore measurement error by using ordinary least squares (OLS) regression or use EIV regression when its required assumptions do not hold. RESULTS: The article derives asymptotic results for ANCOVA with error-prone covariates that cover a variety of cases relevant to applications. It then uses the results in a case study of choosing among ANCOVA model specifications for estimating teacher effects using longitudinal data from a large urban school system. It finds evidence that estimates of teacher effects computed using EIV regression may have smaller bias than estimates computed using OLS regression when the data available for adjusting for students' prior achievement are limited.

Developing Optimized Adaptive Interventions in Education.

Hedges (2018) encourages us to consider asking new scientific questions concerning the optimization of adaptive interventions in education. In this commentary, we have expanded on this (albeit briefly) by providing concrete examples of scientific questions and associated experimental designs to optimize adaptive interventions, and commenting on some of the ways such designs might challenge us to think differently. A great deal of methodological work remains to be done. For example, we have only begun to consider experimental design and analysis methods for developing "cluster-level adaptive interventions" (NeCamp, Kilbourne, & Almirall, 2017), or to extend methods for comparing the marginal mean trajectories between the adaptive interventions embedded in a SMART (Lu et al., 2016) to accommodate random effects. These methodological advances, among others, will propel educational research concerning the construction of more complex, yet meaningful, interventions that are necessary for improving student and teacher outcomes.

Developing adaptive interventions for adolescent substance use treatment settings: protocol of an observational, mixed-methods project.

BACKGROUND: Over 1.6 million adolescents in the United States meet criteria for substance use disorders (SUDs). While there are promising treatments for SUDs, adolescents respond to these treatments differentially in part based on the setting in which treatments are delivered. One way to address such individualized response to treatment is through the development of adaptive interventions (AIs): sequences of decision rules for altering treatment based on an individual's needs. This protocol describes a project with the overarching goal of beginning the development of AIs that provide recommendations for altering the setting of an adolescent's substance use treatment. This project has three discrete aims: (1) explore the views of various stakeholders (parents, providers, policymakers, and researchers) on deciding the setting of substance use treatment for an adolescent based on individualized need, (2) generate hypotheses concerning candidate AIs, and (3) compare the relative effectiveness among candidate AIs and non-adaptive interventions commonly used in everyday practice. METHODS: This project uses a mixed-methods approach. First, we will conduct an iterative stakeholder engagement process, using RAND's ExpertLens online system, to assess the importance of considering specific individual needs and clinical outcomes when deciding the setting for an adolescent's substance use treatment. Second, we will use results from the stakeholder engagement process to analyze an observational longitudinal data set of 15,656 adolescents in substance use treatment, supported by the Substance Abuse and Mental Health Services Administration, using the Global Appraisal of Individual Needs questionnaire. We will utilize methods based on Q-learning regression to generate hypotheses about candidate AIs. Third, we will use robust statistical methods that aim to appropriately handle casemix adjustment on a large number of covariates (marginal structural modeling and inverse probability of treatment weights) to compare the relative effectiveness among candidate AIs and non-adaptive decision rules that are commonly used in everyday practice. DISCUSSION: This project begins filling a major gap in clinical and research efforts for adolescents in substance use treatment. Findings could be used to inform the further development and revision of influential multi-dimensional assessment and treatment planning tools, or lay the foundation for subsequent experiments to further develop or test AIs for treatment planning.

Optimizing Variance-Bias Trade-off in the TWANG Package for Estimation of Propensity Scores.

While propensity score weighting has been shown to reduce bias in treatment effect estimation when selection bias is present, it has also been shown that such weighting can perform poorly if the estimated propensity score weights are highly variable. Various approaches have been proposed which can reduce the variability of the weights and the risk of poor performance, particularly those based on machine learning methods. In this study, we closely examine approaches to fine-tune one machine learning technique (generalized boosted models [GBM]) to select propensity scores that seek to optimize the variance-bias trade-off that is inherent in most propensity score analyses. Specifically, we propose and evaluate three approaches for selecting the optimal number of trees for the GBM in the twang package in R. Normally, the twang package in R iteratively selects the optimal number of trees as that which maximizes balance between the treatment groups being considered. Because the selected number of trees may lead to highly variable propensity score weights, we examine alternative ways to tune the number of trees used in the estimation of propensity score weights such that we sacrifice some balance on the pre-treatment covariates in exchange for less variable weights. We use simulation studies to illustrate these methods and to describe the potential advantages and disadvantages of each method. We apply these methods to two case studies: one examining the effect of dog ownership on the owner's general health using data from a large, population-based survey in California, and a second investigating the relationship between abstinence and a long-term economic outcome among a sample of high-risk youth.

The Right Tool for the Job: Choosing Between Covariate-balancing and Generalized Boosted Model Propensity Scores.

Estimating the causal effect of an exposure (vs. some control) on an outcome using observational data often requires addressing the fact that exposed and control groups differ on pre-exposure characteristics that may be related to the outcome (confounders). Propensity score methods have long been used as a tool for adjusting for observed confounders in order to produce more valid causal effect estimates under the strong ignorability assumption. In this article, we compare two promising propensity score estimation methods (for time-invariant binary exposures) when assessing the average treatment effect on the treated: the generalized boosted models and covariate-balancing propensity scores, with the main objective to provide analysts with some rules-of-thumb when choosing between these two methods. We compare the methods across different dimensions including the presence of extraneous variables, the complexity of the relationship between exposure or outcome and covariates, and the residual variance in outcome and exposure. We found that when noncomplex relationships exist between outcome or exposure and covariates, the covariate-balancing method outperformed the boosted method, but under complex relationships, the boosted method performed better. We lay out criteria for when one method should be expected to outperform the other with no blanket statement on whether one method is always better than the other.

Simulation-Extrapolation with Latent Heteroskedastic Error Variance.

This article considers the application of the simulation-extrapolation (SIMEX) method for measurement error correction when the error variance is a function of the latent variable being measured. Heteroskedasticity of this form arises in educational and psychological applications with ability estimates from item response theory models. We conclude that there is no simple solution for applying SIMEX that generally will yield consistent estimators in this setting. However, we demonstrate that several approximate SIMEX methods can provide useful estimators, leading to recommendations for analysts dealing with this form of error in settings where SIMEX may be the most practical option.

Trends in Classroom Observation Scores.

Observations and ratings of classroom teaching and interactions collected over time are susceptible to trends in both the quality of instruction and rater behavior. These trends have potential implications for inferences about teaching and for study design. We use scores on the Classroom Assessment Scoring System-Secondary (CLASS-S) protocol from 458 middle school teachers over a 2-year period to study changes over time in (a) the average quality of teaching for the population of teachers, (b) the average severity of the population of raters, and (c) the severity of individual raters. To obtain these estimates and assess them in the context of other factors that contribute to the variability in scores, we develop an augmented G study model that is broadly applicable for modeling sources of variability in classroom observation ratings data collected over time. In our data, we found that trends in teaching quality were small. Rater drift was very large during raters' initial days of observation and persisted throughout nearly 2 years of scoring. Raters did not converge to a common level of severity; using our model we estimate that variability among raters actually increases over the course of the study. Variance decompositions based on the model find that trends are a modest source of variance relative to overall rater effects, rater errors on specific lessons, and residual error. The discussion provides possible explanations for trends and rater divergence as well as implications for designs collecting ratings over time.

Provision of mental health services as a quality indicator for adolescent substance abuse treatment facilities.

OBJECTIVE: The study tested whether adolescents receiving substance abuse treatment at facilities offering full (can treat all psychiatric conditions) or partial (cannot treat severe or persistent mental illness) mental health services have better 12-month substance use and mental health outcomes than youths at facilities with no mental health services. METHODS: Data were collected from 3,235 adolescents who were assessed at baseline and at 12 months at one of 50 adolescent treatment facilities. Propensity scores were applied to compare client outcomes from three types of facilities (full, partial, or no mental health services); weighted linear models were estimated to examine outcomes. RESULTS: Youths attending facilities offering full or partial mental health services had better substance abuse treatment outcomes than youths attending facilities offering no such services. There was no evidence of a difference in substance use outcomes between facilities offering full versus partial services, nor was there evidence of differences in mental health outcomes. CONCLUSIONS: These preliminary findings suggest that the availability of mental health services may be a useful quality indicator for adolescent substance abuse treatment facilities. More research is needed to examine specific types of mental health services offered at different facilities.

Do improvements in substance use and mental health symptoms during treatment translate to long-term outcomes in the opposite domain?

Providers who treat adolescents with co-occurring substance use and mental health issues may prioritize treatment of one set of symptoms believing that improvements in one domain will result in improvements of the other. However, limited empirical data for adolescents provide evidence of such "spillover effects." Using data from 2900 youth in an outpatient treatment, we examined whether during-treatment changes in substance use or mental health symptoms predicted 12-month outcomes in the analogous and opposite domains. There was very little evidence of spillover effects, only that youth with no internal distress at 0 and 3 months reported lower levels of substance use problems at 12-months relative to youth with internal distress that stayed the same from 0 to 3 months. These findings suggest that providers treat both sets of substance use and mental health symptoms in an integrated manner given that these symptoms commonly co-occur among youth with either set.

Effectiveness of treatment for adolescent substance use: is biological drug testing sufficient?

OBJECTIVE: The purpose of this study was to compare the relative effectiveness of three treatment modalities for adolescent substance use: biological drug screening (BDS), Motivational Enhancement Therapy-Cognitive Behavioral Therapy (MET/CBT5), and BDS combined with MET/CBT5, relative to no treatment. METHOD: This study comprised 5,186 adolescents (70% male) enrolled in substance use treatment and tracked through the Substance Abuse and Mental Health Services Administration's Center for Substance Abuse Treatment's database (BDS = 1,110; MET/CBT5 = 784; BDS combined with MET/CBT5 = 2,539; no treatment = 753). Outcomes of interest were substance use frequency and severity of substance use problems at 3, 6, and 12 months, as measured by the Global Appraisal of Individual Needs survey. Propensity score weighting was used to adjust for pretreatment covariate imbalances between groups. Weighted generalized linear models were used to estimate the impact of treatment on outcomes at 3, 6, and 12 months. RESULTS: BDS, alone or in combination with MET/CBT5, was associated with improved substance use and substance problems outcomes. Relative to youth reporting no treatment services, the BDS group reported significantly lower substance use at all visits, with the observed difference increasing over time. BDS alone was associated with significantly fewer substance problems than bds combined with met/cbt5 at all visits and significantly lower use at 12 months. CONCLUSIONS: Our results demonstrate significant improvement on substance use outcomes associated with BDS and offer preliminary evidence that BDS, particularly standalone BDS, may be an effective form of drug treatment for adolescents. Further work, including randomized studies, should explore the optimal format of administering BDS to adolescents to achieve maximum effectiveness.

Bayesian hierarchical multivariate formulation with factor analysis for nested ordinal data.

This article devises a Bayesian multivariate formulation for analysis of ordinal data that records teacher classroom performance along multiple dimensions to assess aspects characterizing good instruction. Study designs for scoring teachers seek to measure instructional performance over multiple classroom measurement event sessions at varied occasions using disjoint intervals within each session and employment of multiple ratings on intervals scored by different raters; a design which instantiates a nesting structure with each level contributing a source of variation in recorded scores. We generally possess little a priori knowledge of the existence or form of a sparse generating structure for the multivariate dimensions at any level in the nesting that would permit collapsing over dimensions as is done under univariate modeling. Our approach composes a Bayesian data augmentation scheme that introduces a latent continuous multivariate response linked to the observed ordinal scores with the latent response mean constructed as an additive multivariate decomposition of nested level means that permits the extraction of de-noised continuous teacher-level scores and the associated correlation matrix. A semi-parametric extension facilitates inference for teacher-level dependence among the dimensions of classroom performance under multi-modality induced by sub-groupings of rater perspectives. We next replace an inverse Wishart prior specified for the teacher covariance matrix over dimensions of instruction with a factor analytic structure to allow the simultaneous assessment of an underlying sparse generating structure. Our formulation for Bayesian factor analysis employs parameter expansion with an accompanying post-processing sign re-labeling step of factor loadings that together reduce posterior correlations among sampled parameters to improve parameter mixing in our Markov chain Monte Carlo (MCMC) scheme. We evaluate the performance of our formulation on simulated data and make an application for the assessment of the teacher covariance structure with a dataset derived from a study of middle and high school algebra teachers.

Assessing the generalizability of the CSAT-sponsored GAIN dataset: are the CSAT sites representative of adolescent treatment programs in the U.S.?

The CSAT-sponsored GAIN dataset represents one of the largest longitudinal datasets of adolescent substance use treatment currently available. Understanding the characteristics of the included treatment programs is needed to help inform whether the data are generalizable to adolescent treatment more broadly. Data from a national sample of adolescent treatment programs were compared to the CSAT-funded programs to assess generalizability and understand trends over time in quality service provision. The results indicated that CSAT-funded programs had higher rates of comprehensive mental health assessments, discharge planning, HIV, STD and TB testing, and HIV/AIDS education and support. Conversely, CSAT and non-CSAT-funded programs had similar rates of comprehensive substance use screening and assessment, family and aftercare counseling, drug and alcohol urine screening, case management support, and licensing. The results also showed that service provision has not changed much over the past decade and is in critical need of improvement to reflect expert-informed quality standards.

Time-varying effect moderation using the structural nested mean model: estimation using inverse-weighted regression with residuals.

This article considers the problem of examining time-varying causal effect moderation using observational, longitudinal data in which treatment, candidate moderators, and possible confounders are time varying. The structural nested mean model (SNMM) is used to specify the moderated time-varying causal effects of interest in a conditional mean model for a continuous response given time-varying treatments and moderators. We present an easy-to-use estimator of the SNMM that combines an existing regression-with-residuals (RR) approach with an inverse-probability-of-treatment weighting (IPTW) strategy. The RR approach has been shown to identify the moderated time-varying causal effects if the time-varying moderators are also the sole time-varying confounders. The proposed IPTW+RR approach provides estimators of the moderated time-varying causal effects in the SNMM in the presence of an additional, auxiliary set of known and measured time-varying confounders. We use a small simulation experiment to compare IPTW+RR versus the traditional regression approach and to compare small and large sample properties of asymptotic versus bootstrap estimators of the standard errors for the IPTW+RR approach. This article clarifies the distinction between time-varying moderators and time-varying confounders. We illustrate the methodology in a case study to assess if time-varying substance use moderates treatment effects on future substance use.