Using IRTree Models to Promote Selection Validity in the Presence of Extreme Response Styles.

The measurement of psychological constructs is frequently based on self-report tests, which often have Likert-type items rated from "Strongly Disagree" to "Strongly Agree". Recently, a family of item response theory (IRT) models called IRTree models have emerged that can parse out content traits (e.g., personality traits) from noise traits (e.g., response styles). In this study, we compare the selection validity and adverse impact consequences of noise traits on selection when scores are estimated using a generalized partial credit model (GPCM) or an IRTree model. First, we present a simulation which demonstrates that when noise traits do exist, the selection decisions made based on the IRTree model estimated scores have higher accuracy rates and have less instances of adverse impact based on extreme response style group membership when compared to the GPCM. Both models performed similarly when there was no influence of noise traits on the responses. Second, we present an application using data collected from the Open-Source Psychometrics Project Fisher Temperament Inventory dataset. We found that the IRTree model had a better fit, but a high agreement rate between the model decisions resulted in virtually identical impact ratios between the models. We offer considerations for applications of the IRTree model and future directions for research.

The Backup Planning Scale (BUPS): A Brief, Self-Reported Measure of a Person's Tendency to Develop, Reserve, and Use Backup Plans.

To mitigate uncertainty in their goal pursuits, people use backup plans, i.e., alternative means that are developed to potentially replace "Plan A." Several studies have demonstrated that backup plans can introduce unexpected costs into goal pursuits that decrease a person's motivation to continue using their "Plan A," and reduce their chances for achieving their goal. These existing studies used time-intensive experimental and/or observational approaches to assess the effects of backup planning. The present research examines the newly-developed Backup Planning Scale (BUPS) for its measurement invariance, reliability, validity, and other psychometric characteristics across three independent samples with more than 1,500 participants. Consistent with prior theorizing, we found support for a nine-item, three factor structure for the BUPS, indexing latent factors for a person's tendency to develop, reserve, and replace with (or use) backup plans. Furthermore, a novel "IRTree" based statistical technique provided evidence for the validity of the measure, as participants' responses to the BUPS were associated with their actual developing, reserving, and replacing backup planning behaviors in a logic task. We conclude that the freely-available BUPS is a simple, brief, reliable, and valid self-reported instrument for assessing backup planning behaviors across adulthood.

Using a Response Time-Based Expected A Posteriori Estimator to Control for Differential Speededness in Computerized Adaptive Test.

This study investigates using response times (RTs) with item responses in a computerized adaptive test (CAT) setting to enhance item selection and ability estimation and control for differential speededness. Using van der Linden's hierarchical framework, an extended procedure for joint estimation of ability and speed parameters for use in CAT is developed following van der Linden; this is called the joint expected a posteriori estimator (J-EAP). It is shown that the J-EAP estimate of ability and speededness outperforms the standard maximum likelihood estimator (MLE) of ability and speededness in terms of correlation, root mean square error, and bias. It is further shown that under the maximum information per time unit item selection method (MICT)-a method which uses estimates for ability and speededness directly-using the J-EAP further reduces average examinee time spent and variability in test times between examinees above the resulting gains of this selection algorithm with the MLE while maintaining estimation efficiency. Simulated test results are further corroborated with test parameters derived from a real data example.

A Restricted Four-Parameter IRT Model: The Dyad Four-Parameter Normal Ogive (Dyad-4PNO) Model.

Recently, there has been a renewed interest in the four-parameter item response theory model as a way to capture guessing and slipping behaviors in responses. Research has shown, however, that the nested three-parameter model suffers from issues of unidentifiability (San Martín et al. in Psychometrika 80:450-467, 2015), which places concern on the identifiability of the four-parameter model. Borrowing from recent advances in the identification of cognitive diagnostic models, in particular, the DINA model (Gu and Xu in Stat Sin https://doi.org/10.5705/ss.202018.0420 , 2019), a new model is proposed with restrictions inspired by this new literature to help with the identification issue. Specifically, we show conditions under which the four-parameter model is strictly and generically identified. These conditions inform the presentation of a new exploratory model, which we call the dyad four-parameter normal ogive (Dyad-4PNO) model. This model is developed by placing a hierarchical structure on the DINA model and imposing equality constraints on a priori unknown dyads of items. We present a Bayesian formulation of this model, and show that model parameters can be accurately recovered. Finally, we apply the model to a real dataset.

IRTBEMM: An R Package for Estimating IRT Models With Guessing or Slipping Parameters.

A recently released R package IRTBEMM is presented in this article. This package puts together several new estimation algorithms (Bayesian EMM, Bayesian E3M, and their maximum likelihood versions) for the Item Response Theory (IRT) models with guessing and slipping parameters (e.g., 3PL, 4PL, 1PL-G, and 1PL-AG models). IRTBEMM should be of interest to the researchers in IRT estimation and applying IRT models with the guessing and slipping effects to real datasets.

High-Stakes Testing Case Study: A Latent Variable Approach for Assessing Measurement and Prediction Invariance.

The existence of differences in prediction systems involving test scores across demographic groups continues to be a thorny and unresolved scientific, professional, and societal concern. Our case study uses a two-stage least squares (2SLS) estimator to jointly assess measurement invariance and prediction invariance in high-stakes testing. So, we examined differences across groups based on latent as opposed to observed scores with data for 176 colleges and universities from The College Board. Results showed that evidence regarding measurement invariance was rejected for the SAT mathematics (SAT-M) subtest at the 0.01 level for 74.5% and 29.9% of cohorts for Black versus White and Hispanic versus White comparisons, respectively. Also, on average, Black students with the same standing on a common factor had observed SAT-M scores that were nearly a third of a standard deviation lower than for comparable Whites. We also found evidence that group differences in SAT-M measurement intercepts may partly explain the well-known finding of observed differences in prediction intercepts. Additionally, results provided evidence that nearly a quarter of the statistically significant observed intercept differences were not statistically significant at the 0.05 level once predictor measurement error was accounted for using the 2SLS procedure. Our joint measurement and prediction invariance approach based on latent scores opens the door to a new high-stakes testing research agenda whose goal is to not simply assess whether observed group-based differences exist and the size and direction of such differences. Rather, the goal of this research agenda is to assess the causal chain starting with underlying theoretical mechanisms (e.g., contextual factors, differences in latent predictor scores) that affect the size and direction of any observed differences.

The role of multiple-group measurement invariance in family psychology research.

Measurement invariance (MI) is a property of measurement that is often implicitly assumed, but in many cases, not tested. When the assumption of MI is tested, it generally involves determining if the measurement holds longitudinally or cross-culturally. A growing literature shows that other groupings can, and should, be considered as well. Additionally, it is noted that the standard techniques for investigating MI have been focused almost exclusively on the case of 2 groups, with very little work on the case of more than 2 groups, even though the need for such techniques is apparent in many fields of research. This paper introduces and illustrates a model building technique to investigating MI for more than 2 groups. This technique is an extension of the already-existing hierarchy for testing MI introduced by Meredith (1993). An example using data on father involvement in 5 different groups of families of children with and without developmental disabilities from the Early Childhood Longitudinal Study-Birth Cohort dataset will be given. We show that without considering the possible differential functioning of the measurements on multiple developmental groups, the differences present between the groups in terms of the measurements may be obscured. This could lead to incorrect conclusions.

Father involvement and maternal depressive symptoms in families of children with disabilities or delays.

This study examined the longitudinal association between fathers' early involvement in routine caregiving, literacy, play, and responsive caregiving activities at 9 months and maternal depressive symptoms at 4 years. Data for 3,550 children and their biological parents were drawn from the Early Childhood Longitudinal Study-Birth Cohort data set. Analyses in a structural equation modeling framework examined whether the association between father involvement and maternal depressive symptoms differed for families of children with autism spectrum disorder (ASD) and for families of children with other disabilities or delays from families of children who were typically developing. Results indicated that father literacy and responsive caregiving involvement were associated with lower levels of depressive symptoms for mothers of children with ASD. These findings indicate that greater father involvement may benefit families of children with ASD and highlight the need to support and encourage service providers to work with fathers.