A simple Monte Carlo method for estimating power in multilevel designs.

Estimating power for multilevel models is complex because there are many moving parts, several sources of variation to consider, and unique sample sizes at Level 1 and Level 2. Monte Carlo computer simulation is a flexible tool that has received considerable attention in the literature. However, much of the work to date has focused on very simple models with one predictor at each level and one cross-level interaction effect, and approaches that do not share this limitation require users to specify a large set of population parameters. The goal of this tutorial is to describe a flexible Monte Carlo approach that accommodates a broad class of multilevel regression models with continuous outcomes. Our tutorial makes three important contributions. First, it allows any number of within-cluster effects, between-cluster effects, covariate effects at either level, cross-level interactions, and random coefficients. Moreover, we do not assume orthogonal effects, and predictors can correlate at either level. Second, our approach accommodates models with multiple interaction effects, and it does so with exact expressions for the variances and covariances of product random variables. Finally, our strategy for deriving hypothetical population parameters does not require pilot or comparable data. Instead, we use intuitive variance-explained effect size expressions to reverse-engineer solutions for the regression coefficients and variance components. We describe a new R package mlmpower that computes these solutions and automates the process of generating artificial data sets and summarizing the simulation results. The online supplemental materials provide detailed vignettes that annotate the R scripts and resulting output. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A factored regression model for composite scores with item-level missing data.

Composite scores are an exceptionally important psychometric tool for behavioral science research applications. A prototypical example occurs with self-report data, where researchers routinely use questionnaires with multiple items that tap into different features of a target construct. Item-level missing data are endemic to composite score applications. Many studies have investigated this issue, and the near-universal theme is that item-level missing data treatment is superior because it maximizes precision and power. However, item-level missing data handling can be challenging because missing data models become very complex and suffer from the same "curse of dimensionality" problem that plagues the estimation of psychometric models. A good deal of recent missing data literature has focused on advancing factored regression specifications that use a sequence of regression models to represent the multivariate distribution of a set of incomplete variables. The purpose of this paper is to describe and evaluate a factored specification for composite scores with incomplete item responses. We used a series of computer simulations to compare the proposed approach to gold standard multiple imputation and latent variable modeling approaches. Overall, the simulation results suggest that this new approach can be very effective, even under extreme conditions where the number of items is very large (or even exceeds) the sample size. A real data analysis illustrates the application of the method using software available on the internet. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

An Investigation of Factored Regression Missing Data Methods for Multilevel Models with Cross-Level Interactions.

A growing body of literature has focused on missing data methods that factorize the joint distribution into a part representing the analysis model of interest and a part representing the distributions of the incomplete predictors. Relatively little is known about the utility of this method for multilevel models with interactive effects. This study presents a series of Monte Carlo computer simulations that investigates Bayesian and multiple imputation strategies based on factored regressions. When the model's distributional assumptions are satisfied, these methods generally produce nearly unbiased estimates and good coverage, with few exceptions. Severe misspecifications that arise from substantially non-normal distributions can introduce biased estimates and poor coverage. Follow-up simulations suggest that a Yeo-Johnson transformation can mitigate these biases. A real data example illustrates the methodology, and the paper suggests several avenues for future research.

Compatibility in imputation specification.

Missing data such as data missing at random (MAR) are unavoidable in real data and have the potential to undermine the validity of research results. Multiple imputation is one of the most widely used MAR-based methods in education and behavioral science applications. Arbitrarily specifying imputation models can lead to incompatibility and cause biased estimation. Building on the recent developments of model-based imputation and Arnold's compatibility work, this paper systematically summarizes when the traditional fully conditional specification (FCS) is applicable and how to specify a model-based imputation model if needed. We summarize two Compatibility Requirements to help researchers check compatibility more easily and a decision tree to check whether the traditional FCS is applicable in a given scenario. Additionally, we present a clear overview of two types of model-based imputation: the sequential and separate specifications. We illustrate how to specify model-based imputation with examples. Additionally, we provide example code of a free software program, Blimp, for implementing model-based imputation.

A model-based imputation procedure for multilevel regression models with random coefficients, interaction effects, and nonlinear terms.

Despite the broad appeal of missing data handling approaches that assume a missing at random (MAR) mechanism (e.g., multiple imputation and maximum likelihood estimation), some very common analysis models in the behavioral science literature are known to cause bias-inducing problems for these approaches. Regression models with incomplete interactive or polynomial effects are a particularly important example because they are among the most common analyses in behavioral science research applications. In the context of single-level regression, fully Bayesian (model-based) imputation approaches have shown great promise with these popular analysis models. The purpose of this article is to extend model-based imputation to multilevel models with up to 3 levels, including functionality for mixtures of categorical and continuous variables. Computer simulation results suggest that this new approach can be quite effective when applied to multilevel models with random coefficients and interaction effects. In most scenarios that we examined, imputation-based parameter estimates were quite accurate and tracked closely with those of the complete data. The new procedure is available in the Blimp software application for macOS, Windows, and Linux, and the article includes a data analysis example illustrating its use. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

A fully conditional specification approach to multilevel imputation of categorical and continuous variables.

Specialized imputation routines for multilevel data are widely available in software packages, but these methods are generally not equipped to handle a wide range of complexities that are typical of behavioral science data. In particular, existing imputation schemes differ in their ability to handle random slopes, categorical variables, differential relations at Level-1 and Level-2, and incomplete Level-2 variables. Given the limitations of existing imputation tools, the purpose of this manuscript is to describe a flexible imputation approach that can accommodate a diverse set of 2-level analysis problems that includes any of the aforementioned features. The procedure employs a fully conditional specification (also known as chained equations) approach with a latent variable formulation for handling incomplete categorical variables. Computer simulations suggest that the proposed procedure works quite well, with trivial biases in most cases. We provide a software program that implements the imputation strategy, and we use an artificial data set to illustrate its use. (PsycINFO Database Record

Multilevel multiple imputation: A review and evaluation of joint modeling and chained equations imputation.

Although missing data methods have advanced in recent years, methodologists have devoted less attention to multilevel data structures where observations at level-1 are nested within higher-order organizational units at level-2 (e.g., individuals within neighborhoods; repeated measures nested within individuals; students nested within classrooms). Joint modeling and chained equations imputation are the principal imputation frameworks for single-level data, and both have multilevel counterparts. These approaches differ algorithmically and in their functionality; both are appropriate for simple random intercept analyses with normally distributed data, but they differ beyond that. The purpose of this paper is to describe multilevel imputation strategies and evaluate their performance in a variety of common analysis models. Using multiple imputation theory and computer simulations, we derive 4 major conclusions: (a) joint modeling and chained equations imputation are appropriate for random intercept analyses; (b) the joint model is superior for analyses that posit different within- and between-cluster associations (e.g., a multilevel regression model that includes a level-1 predictor and its cluster means, a multilevel structural equation model with different path values at level-1 and level-2); (c) chained equations imputation provides a dramatic improvement over joint modeling in random slope analyses; and (d) a latent variable formulation for categorical variables is quite effective. We use a real data analysis to demonstrate multilevel imputation, and we suggest a number of avenues for future research. (PsycINFO Database Record

Analyzing person, situation and person × situation interaction effects: Latent state-trait models for the combination of random and fixed situations.

Latent state-trait (LST) models (Steyer, Ferring, & Schmitt, 1992) allow separating person-specific (trait) effects from (1) effects of the situation and person × situation interactions, and (2) random measurement error in purely observational studies. Typical LST applications use measurement designs in which all situations are sampled randomly and do not have to be known for any individual. Limitations of conventional LST models for only random situations are that traits are implicitly assumed to generalize perfectly across situations, and that main effects of situations are inseparable from person × situation interaction effects because both are measured by the same latent variable. In this article, we show how these limitations can be overcome by using measurement designs in which two or more random situations are nested within two or more fixed situations that are known for each individual. We present extended LST models for the combination of random and fixed situations (LST-RF approach) and show that the extensions allow (1) examining the extent to which traits are situation-specific and (2) isolating person × situation interactions from situation main effects. We demonstrate that the LST-RF approach can be applied with both homogenous and heterogeneous indicators in either the single- or multilevel structural equation modeling frameworks. Advantages and limitations of the new models as well as their relation to other approaches for studying person × situation interactions are discussed.

Distinguishing state variability from trait change in longitudinal data: the role of measurement (non)invariance in latent state-trait analyses.

Researchers analyzing longitudinal data often want to find out whether the process they study is characterized by (1) short-term state variability, (2) long-term trait change, or (3) a combination of state variability and trait change. Classical latent state-trait (LST) models are designed to measure reversible state variability around a fixed set-point or trait, whereas latent growth curve (LGC) models focus on long-lasting and often irreversible trait changes. In the present article, we contrast LST and LGC models from the perspective of measurement invariance testing. We show that establishing a pure state-variability process requires (1) the inclusion of a mean structure and (2) establishing strong factorial invariance in LST analyses. Analytical derivations and simulations demonstrate that LST models with noninvariant parameters can mask the fact that a trait-change or hybrid process has generated the data. Furthermore, the inappropriate application of LST models to trait change or hybrid data can lead to bias in the estimates of consistency and occasion specificity, which are typically of key interest in LST analyses. Four tips for the proper application of LST models are provided.