Selecting the model for multiple imputation of missing data: Just use an IC!

Multiple imputation and maximum likelihood estimation (via the expectation-maximization algorithm) are two well-known methods readily used for analyzing data with missing values. While these two methods are often considered as being distinct from one another, multiple imputation (when using improper imputation) is actually equivalent to a stochastic expectation-maximization approximation to the likelihood. In this article, we exploit this key result to show that familiar likelihood-based approaches to model selection, such as Akaike's information criterion (AIC) and the Bayesian information criterion (BIC), can be used to choose the imputation model that best fits the observed data. Poor choice of imputation model is known to bias inference, and while sensitivity analysis has often been used to explore the implications of different imputation models, we show that the data can be used to choose an appropriate imputation model via conventional model selection tools. We show that BIC can be consistent for selecting the correct imputation model in the presence of missing data. We verify these results empirically through simulation studies, and demonstrate their practicality on two classical missing data examples. An interesting result we saw in simulations was that not only can parameter estimates be biased by misspecifying the imputation model, but also by overfitting the imputation model. This emphasizes the importance of using model selection not just to choose the appropriate type of imputation model, but also to decide on the appropriate level of imputation model complexity.

Testing multivariate mean collinearity via latent variable modelling.

A procedure for testing mean collinearity in multidimensional spaces is outlined, which is applicable in settings with missing data and can be used when examining group mean differences. The approach is based on non-linear parameter restrictions and is developed within the framework of latent variable modelling. The method provides useful information about the constellation of multiple response centroids in the populations studied, and is illustrated with an example.

A method of bias correction for maximal reliability with dichotomous measures.

This paper is concerned with the reliability of weighted combinations of a given set of dichotomous measures. Maximal reliability for such measures has been discussed in the past, but the pertinent estimator exhibits a considerable bias and mean squared error for moderate sample sizes. We examine this bias, propose a procedure for bias correction, and develop a more accurate asymptotic confidence interval for the resulting estimator. In most empirically relevant cases, the bias correction and mean squared error correction can be performed simultaneously. We propose an approximate (asymptotic) confidence interval for the maximal reliability coefficient, discuss the implementation of this estimator, and investigate the mean squared error of the associated asymptotic approximation. We illustrate the proposed methods using a numerical example.

Estimation of maximal reliability for multiple-component instruments in multilevel designs.

A procedure for point and interval estimation of maximal reliability of multiple-component measuring instruments in multi-level settings is outlined. The approach is applicable to hierarchical designs in which individuals are nested within higher-order units and exhibit possibly related performance on components of a given homogeneous scale. The method is developed within the framework of multi-level factor analysis. The proposed procedure is illustrated with an empirical example.

Maximal reliability and power in covariance structure models.

In covariance structure modelling, the non-centrality parameter of the asymptotic chi-squared distribution is typically used as an indicator of asymptotic power for hypothesis tests. When a latent linear regression is of interest, the contribution to power by the maximal reliability coefficient, which is associated with used latent variable indicators, is examined and this relationship is further explicated in the case of congeneric measures. It is also shown that item parcelling may reduce power of tests of latent regression parameters. Recommendations on weights for parcelling to avoid power loss are provided, which are found to be those of optimal linear composites with maximal reliability.

On the Relationship Between Maximal Reliability and Maximal Validity of Linear Composites.

A linear combination of a set of measures is often sought as an overall score summarizing subject performance. The weights in this composite can be selected to maximize its reliability or to maximize its validity, and the optimal choice of weights is in general not the same for these two optimality criteria. We explore several relationships between the resulting reliability and validity estimates in different situations. Only in the case of congeneric tests are maximal reliability and maximal validity attained with the same weights, and a precise relationship between these two maximality concepts can be derived. A widely and readily applicable procedure for point and interval estimation of maximal validity is also outlined. Several inequalities are established for the case when the measures are not congeneric.

A Direct Method for Obtaining Approximate Standard Error and Confidence Interval of Maximal Reliability for Composites With Congeneric Measures.

Unlike a substantial part of reliability literature in the past, this article is concerned with weighted combinations of a given set of congeneric measures with uncorrelated errors. The relationship between maximal coefficient alpha and maximal reliability for such composites is initially dealt with, and it is shown that the former is a lower bound of the latter. A direct method for obtaining approximate standard error and confidence interval for maximal reliability is then outlined. The procedure is based on a second-order Taylor series approximation and is readily and widely applicable in empirical research via use of covariance structure modeling. The described method is illustrated with a numerical example.

Estimation of reliability for multiple-component measuring instruments in test-retest designs.

A covariance structure modelling method for the estimation of reliability for composites of congeneric measures in test-retest designs is outlined. The approach also allows an approximate standard error and confidence interval for scale reliability in such settings to be obtained. The procedure further permits measurement error components due to possible transient condition influences to be accounted for and evaluated, and is illustrated with a pair of examples.