Examining the performance of the chi-square difference test when the unrestricted model is slightly misspecified.

Structural equation models are used to model the relationships between latent constructs and observable behaviors such as survey responses. Researchers are often interested in testing nested models to determine whether additional constraints that create a more parsimonious model are also supported by the data. A popular statistical tool for nested model comparison is the chi-square difference test. However, there is some evidence that this test performs suboptimally when the unrestricted model is misspecified. In this paper, we examine the type I error rate of the difference test within the context of single-group confirmatory factor analyses when the less restricted model is misspecified but the constraints imposed by the restricted model are correct. Using empirical simulations and analytic approximations, we find that the chi-square difference test is robust to many but not all forms of realistically sized misspecification in the unrestricted model.

Comparing the Psychometric Properties of a Scale Across Three Likert and Three Alternative Formats: An Application to the Rosenberg Self-Esteem Scale.

Zhang and Savalei proposed an alternative scale format to the Likert format, called the Expanded format. In this format, response options are presented in complete sentences, which can reduce acquiescence bias and method effects. The goal of the current study was to compare the psychometric properties of the Rosenberg Self-Esteem Scale (RSES) in the Expanded format and in two other alternative formats, relative to several versions of the traditional Likert format. We conducted two studies to compare the psychometric properties of the RSES across the different formats. We found that compared with the Likert format, the alternative formats tend to have a unidimensional factor structure, less response inconsistency, and comparable validity. In addition, we found that the Expanded format resulted in the best factor structure among the three alternative formats. Researchers should consider the Expanded format, especially when creating short psychological scales such as the RSES.

Pay Attention to the Ignorable Missing Data Mechanisms! An Exploration of Their Impact on the Efficiency of Regression Coefficients.

The use of modern missing data techniques has become more prevalent with their increasing accessibility in statistical software. These techniques focus on handling data that are missing at random (MAR). Although all MAR mechanisms are routinely treated as the same, they are not equal. The impact of missing data on the efficiency of parameter estimates can differ for different MAR variations, even when the amount of missing data is held constant; yet, in current practice, only the rate of missing data is reported. The impact of MAR on the loss of efficiency can instead be more directly measured by the fraction of missing information (FMI). In this article, we explore this impact using FMIs in regression models with one and two predictors. With the help of a Shiny application, we demonstrate that efficiency loss due to missing data can be highly complex and is not always intuitive. We recommend substantive researchers who work with missing data report estimates of FMIs in addition to the rate of missingness. We also encourage methodologists to examine FMIs when designing simulation studies with missing data, and to explore the behavior of efficiency loss under MAR using FMIs in more complex models.

We need to change how we compute RMSEA for nested model comparisons in structural equation modeling.

Comparison of nested models is common in applications of structural equation modeling (SEM). When two models are nested, model comparison can be done via a chi-square difference test or by comparing indices of approximate fit. The advantage of fit indices is that they permit some amount of misspecification in the additional constraints imposed on the model, which is a more realistic scenario. The most popular index of approximate fit is the root mean square error of approximation (RMSEA). In this article, we argue that the dominant way of comparing RMSEA values for two nested models, which is simply taking their difference, is problematic and will often mask misfit, particularly in model comparisons with large initial degrees of freedom. We instead advocate computing the RMSEA associated with the chi-square difference test, which we call RMSEAD. We are not the first to propose this index, and we review numerous methodological articles that have suggested it. Nonetheless, these articles appear to have had little impact on actual practice. The modification of current practice that we call for may be particularly needed in the context of measurement invariance assessment. We illustrate the difference between the current approach and our advocated approach on three examples, where two involve multiple-group and longitudinal measurement invariance assessment and the third involves comparisons of models with different numbers of factors. We conclude with a discussion of recommendations and future research directions. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

New computations for RMSEA and CFI following FIML and TS estimation with missing data.

The full-information maximum likelihood (FIML) is a popular estimation method for missing data in structural equation modeling (SEM). However, previous research has shown that SEM approximate fit indices (AFIs) such as the root mean square error of approximation (RMSEA) and the comparative fit index (CFI) can be distorted relative to their complete data counterparts when they are computed following the FIML estimation. The main goal of the current paper is to propose and examine an alternative approach for computing AFIs following the FIML estimation, which we refer to as the FIML-corrected or FIML-C approach. The secondary goal of the article is to examine another existing estimation method, the two-stage (TS) approach, for computing AFIs in the presence of missing data. Both FIML-C and TS approaches remove the bias due to missing data, so that the resulting incomplete data AFIs estimate the same population values as their complete data counterparts. For both approaches, we also propose a series of small sample corrections to improve the estimates of AFIs. In two simulation studies, we found that the FIML-C and TS approaches, when implemented with small sample corrections, estimated the population-complete-data AFIs with little bias across a variety of conditions, although the FIML-C approach can fail in a small number of conditions with a high percentage of missing data and a high degree of model misspecification. In contrast, the FIML AFIs as currently computed often performed poorly. We recommend FIML-C and TS approaches for computing AFIs in SEM. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Three Sample Estimates of Fraction of Missing Information From Full Information Maximum Likelihood.

In missing data analysis, the reporting of missing rates is insufficient for the readers to determine the impact of missing data on the efficiency of parameter estimates. A more diagnostic measure, the fraction of missing information (FMI), shows how the standard errors of parameter estimates increase from the information loss due to ignorable missing data. FMI is well-known in the multiple imputation literature (Rubin, 1987), but it has only been more recently developed for full information maximum likelihood (Savalei and Rhemtulla, 2012). Sample FMI estimates using this approach have since then been made accessible as part of the lavaan package (Rosseel, 2012) in the R statistical programming language. However, the properties of FMI estimates at finite sample sizes have not been the subject of comprehensive investigation. In this paper, we present a simulation study on the properties of three sample FMI estimates from FIML in two common models in psychology, regression and two-factor analysis. We summarize the performance of these FMI estimates and make recommendations on their application.

Improving Fit Indices in Structural Equation Modeling with Categorical Data.

Current computations of commonly used fit indices in structural equation modeling (SEM), such as RMSEA and CFI, indicate much better fit when the data are categorical than if the same data had not been categorized. As a result, researchers may be led to accept poorly fitting models with greater frequency when data are categorical. In this article, I first explain why the current computations of categorical fit indices lead to this problematic behavior. I then propose and evaluate alternative ways to compute fit indices with categorical data. The proposed computations approximate what the fit index values would have been had the data not been categorized. The developments in this article are for the DWLS (diagonally weighted least squares) estimator, a popular limited information categorical estimation method. I report on the results of a simulation comparing existing and newly proposed categorical fit indices. The results confirmed the theoretical expectation that the new indices better match the corresponding values with continuous data. The new fit indices performed well across all studied conditions, with the exception of binary data at the smallest studied sample size (N = 200), when all categorical fit indices performed poorly.

Can Researchers' Personal Characteristics Shape Their Statistical Inferences?

Researchers' subjective judgments may affect the statistical results they obtain. This possibility is particularly stark in Bayesian hypothesis testing: To use this increasingly popular approach, researchers specify the effect size they are expecting (the "prior mean"), which is then incorporated into the final statistical results. Because the prior mean represents an expression of confidence that one is studying a large effect, we reasoned that scientists who are more confident in their research skills may be inclined to select larger prior means. Across two preregistered studies with more than 900 active researchers in psychology, we showed that more self-confident researchers selected larger prior means. We also found suggestive but somewhat inconsistent evidence that men may choose larger prior means than women, due in part to gender differences in researcher self-confidence. Our findings provide the first evidence that researchers' personal characteristics might shape the statistical results they obtain with Bayesian hypothesis testing.

Two-stage maximum likelihood approach for item-level missing data in regression.

Psychologists use scales comprised of multiple items to measure underlying constructs. Missing data on such scales often occur at the item level, whereas the model of interest to the researcher is at the composite (scale score) level. Existing analytic approaches cannot easily accommodate item-level missing data when models involve composites. A very common practice in psychology is to average all available items to produce scale scores. This approach, referred to as available-case maximum likelihood (ACML), may produce biased parameter estimates. Another approach researchers use to deal with item-level missing data is scale-level full information maximum likelihood (SL-FIML), which treats the whole scale as missing if any item is missing. SL-FIML is inefficient and it may also exhibit bias. Multiple imputation (MI) produces the correct results using a simulation-based approach. We study a new analytic alternative for item-level missingness, called two-stage maximum likelihood (TSML; Savalei & Rhemtulla, Journal of Educational and Behavioral Statistics, 42(4), 405-431. 2017). The original work showed the method outperforming ACML and SL-FIML in structural equation models with parcels. The current simulation study examined the performance of ACML, SL-FIML, MI, and TSML in the context of univariate regression. We demonstrated performance issues encountered by ACML and SL-FIML when estimating regression coefficients, under both MCAR and MAR conditions. Aside from convergence issues with small sample sizes and high missingness, TSML performed similarly to MI in all conditions, showing negligible bias, high efficiency, and good coverage. This fast analytic approach is therefore recommended whenever it achieves convergence. R code and a Shiny app to perform TSML are provided.

A Multiple Indicators Multiple Causes (MIMIC) model of friendship quality and comorbidities in children with attention-deficit/hyperactivity disorder.

The unique objectives of the current investigation were: (a) to assess the fit of a multiinformant 2-factor measurement model of friendship quality in a clinical sample of children with attention-deficit/hyperactivity disorder (ADHD); and (b) to use a multiple indicators multiple causes approach to evaluate whether comorbid externalizing and internalizing disorders incrementally predict levels of positive and negative friendship quality. Our sample included 165 target children diagnosed with ADHD (33% girls; aged 6-11 years). Target children, their parents, their friends, and the parents of their friends independently completed a self-report measure of friendship quality about the reciprocated friendship between the target child and the friend. Results indicated that a multiinformant 2-factor measurement model with correlated positive friendship quality and negative friendship quality had good fit. The friendships of children with ADHD and a comorbid externalizing disorder were characterized by less positive friendship quality and more negative friendship quality than the friendships of children with ADHD and no externalizing disorder after controlling for the presence of a comorbid internalizing disorder. However, the presence of a comorbid internalizing disorder did not predict positive or negative friendship quality. These findings suggest that soliciting reports from parents in addition to children and friends, and measuring comorbid externalizing disorders, may be valuable evidence-based strategies when assessing friendship quality in ADHD populations. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Improved Properties of the Big Five Inventory and the Rosenberg Self-Esteem Scale in the Expanded Format Relative to the Likert Format.

Previous research by Zhang and Savalei (2015) proposed an alternative scale format to the Likert scale format: the Expanded format. Scale items in the Expanded format present both positively worded and negatively worded sentences as response options for each scale item; therefore, they were less affected by the acquiescence bias and method effects that often occur in the Likert scale items. The major goal of the current study is to further demonstrate the superiority of the Expanded format to the Likert format across different psychological scales. Specifically, we aim to replicate the findings of Zhang and Savalei and to determine whether order effect exists in the Expanded format scales. Six psychological scales were examined in the study, including the five subscales of the big five inventory (BFI) and the Rosenberg self-esteem (RSE) scale. Four versions were created for each psychological scale. One version was the original scale in the Likert format. The other three versions were in different Expanded formats that varied in the order of the response options. For each scale, the participant was randomly assigned to complete one scale version. Across the different versions of each scale, we compared the factor structures and the distributions of the response options. Our results successfully replicated the findings of Zhang and Savalei, and also showed that order effect was generally absent in the Expanded format scales. Based on these promising findings, we encourage researchers to use the Expanded format for these and other scales in their substantive research.

A comparison of several approaches for controlling measurement error in small samples.

It is well known that methods that fail to account for measurement error in observed variables, such as regression and path analysis (PA), can result in poor estimates and incorrect inference. On the other hand, methods that fully account for measurement error, such as structural equation modeling with latent variables and multiple indicators, can produce highly variable estimates in small samples. This article advocates a family of intermediate models for small samples (N < 200), referred to as single indicator (SI) models. In these models, each latent variable has a single composite indicator, with its reliability fixed to a plausible value. A simulation study compared three versions of the SI method with PA and with a multiple-indicator structural equation model (SEM) in small samples (N = 30 to 200). Two of the SI models fixed the reliability of each construct to a value chosen a priori (either .7 or .8). The third SI model (referred to as "SIα") estimated the reliability of each construct from the data via coefficient alpha. The results showed that PA and fixed-reliability SI methods that overestimated reliability slightly resulted in the most accurate estimates as well as in the highest power. Fixed-reliability SI methods also maintained good coverage and Type I error rates. The SIα and SEM methods had intermediate performance. In small samples, use of a fixed-reliability SI method is recommended. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

On the Computation of the RMSEA and CFI from the Mean-And-Variance Corrected Test Statistic with Nonnormal Data in SEM.

A new type of nonnormality correction to the RMSEA has recently been developed, which has several advantages over existing corrections. In particular, the new correction adjusts the sample estimate of the RMSEA for the inflation due to nonnormality, while leaving its population value unchanged, so that established cutoff criteria can still be used to judge the degree of approximate fit. A confidence interval (CI) for the new robust RMSEA based on the mean-corrected ("Satorra-Bentler") test statistic has also been proposed. Follow up work has provided the same type of nonnormality correction for the CFI (Brosseau-Liard & Savalei, 2014). These developments have recently been implemented in lavaan. This note has three goals: a) to show how to compute the new robust RMSEA and CFI from the mean-and-variance corrected test statistic; b) to offer a new CI for the robust RMSEA based on the mean-and-variance corrected test statistic; and c) to caution that the logic of the new nonnormality corrections to RMSEA and CFI is most appropriate for the maximum likelihood (ML) estimator, and cannot easily be generalized to the most commonly used categorical data estimators.

Normal Theory Two-Stage ML Estimator When Data Are Missing at the Item Level.

In many modeling contexts, the variables in the model are linear composites of the raw items measured for each participant; for instance, regression and path analysis models rely on scale scores, and structural equation models often use parcels as indicators of latent constructs. Currently, no analytic estimation method exists to appropriately handle missing data at the item level. Item-level multiple imputation (MI), however, can handle such missing data straightforwardly. In this article, we develop an analytic approach for dealing with item-level missing data-that is, one that obtains a unique set of parameter estimates directly from the incomplete data set and does not require imputations. The proposed approach is a variant of the two-stage maximum likelihood (TSML) methodology, and it is the analytic equivalent of item-level MI. We compare the new TSML approach to three existing alternatives for handling item-level missing data: scale-level full information maximum likelihood, available-case maximum likelihood, and item-level MI. We find that the TSML approach is the best analytic approach, and its performance is similar to item-level MI. We recommend its implementation in popular software and its further study.

Normal Theory GLS Estimator for Missing Data: An Application to Item-Level Missing Data and a Comparison to Two-Stage ML.

Structural equation models (SEMs) can be estimated using a variety of methods. For complete normally distributed data, two asymptotically efficient estimation methods exist: maximum likelihood (ML) and generalized least squares (GLS). With incomplete normally distributed data, an extension of ML called "full information" ML (FIML), is often the estimation method of choice. An extension of GLS to incomplete normally distributed data has never been developed or studied. In this article we define the "full information" GLS estimator for incomplete normally distributed data (FIGLS). We also identify and study an important application of the new GLS approach. In many modeling contexts, the variables in the SEM are linear composites (e.g., sums or averages) of the raw items. For instance, SEMs often use parcels (sums of raw items) as indicators of latent factors. If data are missing at the item level, but the model is at the composite level, FIML is not possible. In this situation, FIGLS may be the only asymptotically efficient estimator available. Results of a simulation study comparing the new FIGLS estimator to the best available analytic alternative, two-stage ML, with item-level missing data are presented.

Type I error rates and power of several versions of scaled chi-square difference tests in investigations of measurement invariance.

A Monte Carlo simulation study was conducted to investigate Type I error rates and power of several corrections for nonnormality to the normal theory chi-square difference test in the context of evaluating measurement invariance via structural equation modeling. Studied statistics include the uncorrected difference test, DML, Satorra and Bentler's (2001) original correction, DSB1, Satorra and Bentler's (2010) strictly positive correction, DSB10, and a hybrid procedure, DSBH (Asparouhov & Muthén, 2013). Multiple-group data were generated from confirmatory factor analytic population models invariant on all parameters, or lacking invariance on residual variances, indicator intercepts, or factor loadings. Conditions varied in terms of the number of indicators associated with each factor in the population model, the location of noninvariance (if any), sample size, sample size ratio in the 2 groups, and nature of nonnormality. Type I error rates and power of corrected statistics were evaluated for a series of 4 nested invariance models. Overall, the strictly positive correction, DSB10, is the best and most consistently performing statistic, as it was found to be much less sensitive than the original correction, DSB1, to model size and sample evenness. (PsycINFO Database Record

Examining the Effect of Reverse Worded Items on the Factor Structure of the Need for Cognition Scale.

Reverse worded (RW) items are often used to reduce or eliminate acquiescence bias, but there is a rising concern about their harmful effects on the covariance structure of the scale. Therefore, results obtained via traditional covariance analyses may be distorted. This study examined the effect of the RW items on the factor structure of the abbreviated 18-item Need for Cognition (NFC) scale using confirmatory factor analysis. We modified the scale to create three revised versions, varying from no RW items to all RW items. We also manipulated the type of the RW items (polar opposite vs. negated). To each of the four scales, we fit four previously developed models. The four models included a 1-factor model, a 2-factor model distinguishing between positively worded (PW) items and RW items, and two 2-factor models, each with one substantive factor and one method factor. Results showed that the number and type of the RW items affected the factor structure of the NFC scale. Consistent with previous research findings, for the original NFC scale, which contains both PW and RW items, the 1-factor model did not have good fit. In contrast, for the revised scales that had no RW items or all RW items, the 1-factor model had reasonably good fit. In addition, for the scale with polar opposite and negated RW items, the factor model with a method factor among the polar opposite items had considerably better fit than the 1-factor model.

Improving the Factor Structure of Psychological Scales: The Expanded Format as an Alternative to the Likert Scale Format.

Many psychological scales written in the Likert format include reverse worded (RW) items in order to control acquiescence bias. However, studies have shown that RW items often contaminate the factor structure of the scale by creating one or more method factors. The present study examines an alternative scale format, called the Expanded format, which replaces each response option in the Likert scale with a full sentence. We hypothesized that this format would result in a cleaner factor structure as compared with the Likert format. We tested this hypothesis on three popular psychological scales: the Rosenberg Self-Esteem scale, the Conscientiousness subscale of the Big Five Inventory, and the Beck Depression Inventory II. Scales in both formats showed comparable reliabilities. However, scales in the Expanded format had better (i.e., lower and more theoretically defensible) dimensionalities than scales in the Likert format, as assessed by both exploratory factor analyses and confirmatory factor analyses. We encourage further study and wider use of the Expanded format, particularly when a scale's dimensionality is of theoretical interest.