Calibrating Items Using an Unfolding Model of Item Response Theory: The Case of the Trait Personality Questionnaire 5 (TPQue5).

Unfolding item response theory (IRT) models are important alternatives to dominance IRT models in describing the response processes on self-report tests. Their usage is common in personality measures, since they indicate potential differentiations in test score interpretation. This paper aims to gain a better insight into the structure of trait personality, by investigating whether the dominance or alternatively the unfolding IRT model are better descriptors of the response processes on a personality measure constructed under the dominance response theorem. For the assessment of the dominant model, the Graded Response Model (GRM) is used; while for the unfolding model, the Generalized Graded Unfolding Model (GGUM) was examined. All analyses are conducted with the freely available R. A sample of 1340 Greek adults, employed in private and public organizations, fulfilled the Trait Personality Questionnaire 5 short-form (TPQue5). Findings contradict previous research on trait personality. In accordance to the construction method employed, the TPQue5 items are best understood by monotonically increasing item response functions (IRFs). Individuals responding to the TPQue5 increase their probability of endorsing its items as their trait level increases; this stands for all personality dimensions, although Openness to Experience exhibited mixed type of item response patterns. Further research directions, implications and limitations are also discussed.

Uncovering Differential Item Functioning effects using MIMIC and mediated MIMIC models.

The aim of this study was twofold: first, to examine the presence of bias across gender in a scholastic achievement test named the Academic Achievement Test (AAT) for the Science Track. Second, to understand the underlying mechanism that causes these bias effects by examining the effect of general cognitive ability as a mediator. The sample consisted of 1,300 Saudi high school students randomly selected from a larger pool of 173,133 participants to reduce the effects of excessive power. To examine both goals, the Multiple Indicators Multiple Causes (MIMIC) approach for detecting Differential Item Functioning (DIF) items was used. The results showed that 13 AAT items exhibited DIF effects for different gender groups. In most of these items, male participants were more likely to answer them correctly than their female counterparts. Next, the mediated MIMIC approach was applied to explore possible underlying mechanisms that explain these DIF effects. The results from this study showed that general cognitive ability (i.e., General Aptitude Test - GAT) seems to be a factor that could explain why an AAT item exhibits DIF across gender. It was found that GAT scores fully explain the DIF effect in two AAT items (full mediation). In most other cases, GAT helps account for only a proportion of the DIF effect (partial mediation). The results from this study will help experts improve the quality of their instruments by identifying DIF items and deciding how to revise them, considering the mediator's effect on participants' responses.

Equidistant Response Options on Likert-Type Instruments: Testing the Interval Scaling Assumption Using Mplus.

The purpose of the present study was to provide the means to evaluate the "interval-scaling" assumption that governs the use of parametric statistics and continuous data estimators in self-report instruments that utilize Likert-type scaling. Using simulated and real data, the methodology to test for this important assumption is evaluated using the popular software Mplus 8.8. Evidence on meeting the assumption is provided using the Wald test and the equidistant index. It is suggested that routine evaluations of self-report instruments engage the present methodology so that the most appropriate estimator will be implemented when testing the construct validity of self-report instruments.

Identifying Ability and Nonability Groups: Incorporating Response Times Using Mixture Modeling.

The goal of the present study was to address the analytical complexity of incorporating responses and response times through applying the Jeon and De Boeck mixture item response theory model in Mplus 8.7. Using both simulated and real data, we attempt to identify subgroups of responders that are rapid guessers or engage knowledge retrieval strategies. When applying the mixture model to a measure of contextual error in linguistics results pointed to the presence of a knowledge retrieval strategy. That is, a participant either knows the content (morphology, grammar rules) and can identify the error, or lacks the requisite knowledge and cannot benefit from spending more time on an item. In contrast, as item difficulty progressed, the high-ability group utilized the additional time to make informed guesses. The methodology is illustrated using annotated code in Mplus 8.7.

Examining academic performance across gender differently: Measurement invariance and latent mean differences using bias-corrected bootstrap confidence intervals.

The aim of this study was threefold: First, to examine the dimensionality of the construct of General Academic Ability (GAA) at the subscale level providing additional insights over and above on the conceptualization of the construct. Second, to explore different degrees of measurement invariance of the GAA across gender using more recent advancements in the examination of Measurement Invariance (i.e., Bias-Corrected bootstrap Confidence Intervals). Third, to examine gender differences across the different facets of the GAA at the latent mean level. The sample consisted of 1,800 high school graduates who applied for higher education in Saudi Arabia. The results from the analysis indicated that the hierarchical model with one higher-order factor (i.e., general academic ability) and four lower-order cognitive factors (i.e., verbal ability, quantitative ability, scholastic aptitude, and GPA) exhibited an excellent fit to the data. In terms of the measurement invariance hypothesis, it was found that the hierarchical model exhibits full configural and metric invariance and partial scalar invariance. Finally, using the Latent Mean Difference procedure, the results showed gender differences in the Verbal and GPA domains. Although significant differences were also found in the Scholastic aptitude domain, this finding is not stable due to several non-invariant items within the domain. In both cases, females scored higher than males. Finally, regarding the higher-order factor (GAA), the results showed that females scored higher than males. There were no significant differences in the Quantitative domain.

Assessing Language Skills Using Diagnostic Classification Models: An Example Using a Language Instrument.

The primary purpose of the present study is to inform and illustrate, using examples, the use of Diagnostic Classification Models (DCMs) for the assessment of skills and competencies in cognition and academic achievement. A secondary purpose is to compare and contrast traditional and contemporary psychometrics for the measurement of skills and competencies. DCMs are described along the lines of other psychometric models within the Confirmatory Factor Analysis tradition such as the bifactor model and the known mixture models that are utilized to classify individuals into subgroups. The inclusion of interaction terms and constraints along with its confirmatory nature enables DCMs to accurately assess the possession of skills and competencies. The above is illustrated using an empirical dataset from Saudi Arabia (n = 2642), in which language skills are evaluated on how they conform to known levels of competency based on the CEFR (Council of Europe, 2001) using the English Proficiency Test (EPT).

El propósito principal del presente estudio fue informar e ilustrar, mediante ejemplos, el uso de Modelos de Clasificación Diagnóstica (DCM) para la evaluación de habilidades y competencias en cognición y rendimiento académico. Un propósito secundario fue comparar y contrastar la psicometría tradicional y contemporánea para la medición de habilidades y competencias. Los DCM se describen siguiendo las líneas de otros modelos psicométricos dentro de la tradición del Análisis Factorial Confirmatorio, como el modelo bifactor y los conocidos modelos mixtos que se utilizan para clasificar a los individuos en subgrupos. La inclusión de términos y restricciones de interacción junto con su naturaleza confirmatoria permite a los DCM evaluar con precisión la posesión de habilidades y competencias. Lo anterior se ilustra utilizando un conjunto de datos empíricos de Arabia Saudita (n = 2642), que evalúan cómo las habilidades lingüísticas se ajustan a los niveles de competencia conocidos, basados en el MCER (Council of Europe, 2001).

Exploring the Dimensions of Smartphone Distraction: Development, Validation, Measurement Invariance, and Latent Mean Differences of the Smartphone Distraction Scale (SDS).

Background: Distraction is a functional emotion regulation strategy utilized to relieve emotional distress. Within the attention economy perspective, distraction is increasingly associated with digital technology use, performance impairments and interference with higher-order cognitive processes. Research on smartphone distraction and its association with problematic smartphone use is still scarce and there is no available psychometric assessment tool to assess this cognitive and emotive process parsimoniously. Method: The present study reports the development and evaluation of the psychometric properties of the Smartphone Distraction Scale (SDS) through exploratory and confirmatory factor analysis, construct validity, gender invariance, and latent mean differences. The study was conducted in a sample of British university students (N = 1,001; M = 21.10 years, SD = 2.77). Results: The 16-item SDS was best conceptualized in a four-factor model solution comprising attention impulsiveness, online vigilance, emotion regulation, and multitasking. Construct validity was established using relevant psychosocial and mental health measures, with SDS scores being moderately associated with deficient self-regulation and problematic social media use. Gender measurement invariance was achieved at the configural, metric, and scalar levels, and latent mean differences indicated that females had significantly higher means than males across all four SDS latent factors. Discussion: The SDS presents with several strengths, including its theoretical grounding, relatively short length, and sound psychometric properties. The SDS enables the assessment of distraction, which appears to be one of the pathways to problematic smartphone use facilitating overuse and overreliance on smartphones for emotion regulation processes. The assessment of distraction in relation to problematic use in vulnerable populations may facilitate interventions that could encourage metacognition and benefit these groups by allowing sustained productivity in an increasingly disrupted work and social environment.

Identifying Student Subgroups as a Function of School Level Attributes: A Multilevel Latent Class Analysis.

The purpose of the present study was to profile high school students' achievement as a function of their demographic characteristics, parent attributes (e.g., education), and school behaviors (e.g., number of absences). Students were nested within schools in the Saudi Arabia Kingdom. Out of a large sample of 500k, participants involved 3 random samples of 2,000 students measured during the years 2016, 2017, and 2018. Randomization was conducted at the student level to ensure that all school units will be represented and at their respective frequency. Students were nested within 50 high schools. We adopted the multilevel latent profile analysis protocol put forth by Schmiege et al. (2018) and Mäkikangas et al. (2018) that account for nested data and tested latent class structure invariance over time. Results pointed to the presence of a 4-profile solution based on BIC, the Bayes factor, and several information criteria put forth by Masyn (2013). Latent profile separation was mostly guided by parents' education and the number of student absences (being positive and negative predictors of high achievement classes, respectively). Two models tested whether the proportions of level 1 profiles to level 2 units are variable and whether level 2 profiles vary as a function of level 1 profiles. Results pointed to the presence of significant variability due to schools.

Accounting for Differential Item Functioning Using Bayesian Approximate Measurement Invariance.

The main thesis of the present study is to use the Bayesian structural equation modeling (BSEM) methodology of establishing approximate measurement invariance (A-MI) using data from a national examination in Saudi Arabia as an alternative to not meeting strong invariance criteria. Instead, we illustrate how to account for the absence of measurement invariance using relative compared to exact criteria. A secondary goal was to compare latent means across groups using invariant parameters only and through utilizing exact and relative evaluative-MI protocol suggested equivalence of the thresholds using prior variances equal to 0.10. Subsequent differences between groups were evaluated using effect size criteria and the prior-posterior predictive p-value (PPPP), which proved to be invaluable in attesting for differences that are beyond zero, some meaningless nonzero estimate, and the three commonly used indices of effect sizes described by Cohen in 1988 (i.e., .20, .50, and .80). Results substantiated the use of the PPPP for evaluating mean differences across groups when utilizing nonexact evaluative criteria.

Measurement Invariance and Differential Item Functioning Across Gender Within a Latent Class Analysis Framework: Evidence From a High-Stakes Test for University Admission in Saudi Arabia.

The main aim of the present study was to investigate the presence of Differential Item Functioning (DIF) using a latent class (LC) analysis approach. Particularly, we examined potential sources of DIF in relation to gender. Data came from 6,265 Saudi Arabia students, who completed a high-stakes standardized admission test for university entrance. The results from a Latent Class Analysis (LCA) revealed a three-class solution (i.e., high, average, and low scorers). Then, to better understand the nature of the emerging classes and the characteristics of the people who comprise them, we applied a new stepwise approach, using the Multiple Indicator Multiple Causes (MIMIC) model. The model identified both uniform and non-uniform DIF effects for several items across all scales of the test, although, for the majority of them, the DIF effect sizes were negligible. Findings from this study have important implications for both measurement quality and interpretation of the results. Particularly, results showed that gender is a potential source of DIF for latent class indicators; thus, it is important to include those direct effects in the latent class regression model, to obtain unbiased estimates not only for the measurement parameters but also of the structural parameters. Ignoring these effects might lead to misspecification of the latent classes in terms of both the size and the characteristics of each class, which in turn, could lead to misinterpretations of the obtained latent class results. Implications of the results for practice are discussed.

An Application of Reliability Estimation in Longitudinal Designs Through Modeling Item-Specific Error Variance.

The purpose of the present study was to apply the methodology developed by Raykov on modeling item-specific variance for the measurement of internal consistency reliability with longitudinal data. Participants were a randomly selected sample of 500 individuals who took on a professional qualifications test in Saudi Arabia over four different occasions. Data were analyzed by use of confirmatory factor analysis, and item error variance was corrected for item specificity. The estimation of reliability involved composite index omega. Results indicated that the initially low and unacceptable levels of internal consistency reliability approached acceptable levels after accounting for item-specific variance. Findings were verified by testing whether the difference estimates of internal consistency reliability deviated from a zero-mean distribution using 10,000 replicated samples assuming a known (symmetric) or unknown (asymmetric) population distribution of the difference reliability coefficients. Percentage improvement reliability estimates indices were also estimated along with their 95% confidence intervals. Two appendices provide annotated Mplus syntax files for future use.

Using Confidence Intervals of the Item and Test Information Functions to Test Differential Item and Test Functioning: Visual and Statistical Analyses.

The purpose of the present paper was twofold: (a) to use 95% confidence intervals of the item and test information functions as a means of visualizing differences between groups on the information provided at the item and test levels, and, (b) to statistically compare item and test information functions as a method for evaluating differential item and differential test functioning. Participants were 2,305 high school students who took a Mathematics National entrance examination in Saudi Arabia. Item and test information functions, conditional standard errors of measurement and reliability were estimated for both males and females. Differences between groups became evident when plotting 95% confidence intervals of the item and test information functions and the visual findings were confirmed using population-based Z-tests of point estimates using a Monte-Carlo simulation. It was concluded that differential group behavior at the item and test levels can be evidenced using information functions and inferential tests of significance can be constructed using the bootstrap distribution. The current procedure involves both item difficulties and discrimination indices and provides increased sensitivity over the traditional methods relying on item difficulties only.

An IRT-Multiple Indicators Multiple Causes (MIMIC) Approach as a Method of Examining Item Response Latency.

The analysis of response time has received increasing attention during the last decades, since evidence from several studies supported the argument that there is a direct relationship between item response time and test performance. The aim of this study was to investigate whether item response latency affects person's ability parameters, in that it represents an adaptive or maladaptive practice. To examine the above research question data from 8,475 individuals completing the computerized version of the Postgraduate General Aptitude Test (PAGAT) were analyzed. To determine the extent to which response latency affects person's ability, we used a Multiple Indicators Multiple Causes (MIMIC) model, in which every item in a scale was linked to its corresponding covariate (i.e., item response latency). We ran the MIMIC model within the Item Response Theory (IRT) framework (2-PL model). The results supported the hypothesis that item response latency could provide valuable information for getting more accurate estimations for persons' ability levels. Results indicated that for individuals who invest more time on easy items, their likelihood of success does not improve, most likely because slow and fast responders have significantly different levels of ability (fast responders are of higher ability compared to slow responders). Consequently, investing more time for low ability individuals does not prove to be adaptive. The opposite was found for difficult items: individuals spending more time on difficult items increase their likelihood of success, more likely because they are high achievers (in difficult items individuals who spent more time were of significantly higher ability compared to fast responders). Thus, it appears that there is an interaction between the difficulty of the item and person abilities that explain the effects of response time on likelihood of success. We concluded that accommodating item response latency in a computerized assessment model, can inform test quality and test takers' behavior, and in that way, enhance score measurement accuracy.

Assessing Construct Validity in Math Achievement: An Application of Multilevel Structural Equation Modeling (MSEM).

The purpose of the present study was to model math achievement at both the person and university levels of the analyses in order to understand the optimal factor structure of math competency. Data involved 2,881 students who took a national mathematics examination as part of their entry at the university public system in Saudi Arabia. Four factors from the National math examination comprised the math achievement measure, namely, numbers and operations, algebra and analysis, geometry and measurement, and, statistics and probabilities. Data were analyzed using the aggregate method and by use of Multilevel Structural Equation Modeling (MSEM). Results indicated that both a unidimensional and a 4-factor correlated model fitted the data equally well using aggregate data, where for reasons of parsimony the unidimensional model was the preferred choice with these data. When modeling data including clustering, results pointed to alternative factor structures at the person and university levels. Thus, a unidimensional model provided the best fit at the University level, whereas a four-factor correlated model was most descriptive for person level data. The optimal simple structure was evaluated using the Ryu and West (2009) methodology for partially saturating the MSEM model and also met criteria for discriminant validation as described in Gorsuch (1983). Furthermore, a university level variable, namely the year of establishment, pointed to the superiority of older institutions with regard to math achievement. It is concluded that ignoring a multilevel structure in the data may result in erroneous conclusions with regard to the optimal factor structure and the tests of structural models following that.

The Network Structure of Schizotypal Personality Traits.

Elucidating schizotypal traits is important if we are to understand the various manifestations of psychosis spectrum liability and to reliably identify individuals at high risk for psychosis. The present study examined the network structures of (1) 9 schizotypal personality domains and (2) 74 individual schizotypal items, and (3) explored whether networks differed across gender and culture (North America vs China). The study was conducted in a sample of 27001 participants from 12 countries and 21 sites (M age = 22.12; SD = 6.28; 37.5% males). The Schizotypal Personality Questionnaire (SPQ) was used to assess 74 self-report items aggregated in 9 domains. We used network models to estimate conditional dependence relations among variables. In the domain-level network, schizotypal traits were strongly interconnected. Predictability (explained variance of each node) ranged from 31% (odd/magical beliefs) to 55% (constricted affect), with a mean of 43.7%. In the item-level network, variables showed relations both within and across domains, although within-domain associations were generally stronger. The average predictability of SPQ items was 27.8%. The network structures of men and women were similar (r = .74), node centrality was similar across networks (r = .90), as was connectivity (195.59 and 199.70, respectively). North American and Chinese participants networks showed lower similarity in terms of structure (r = 0.44), node centrality (r = 0.56), and connectivity (180.35 and 153.97, respectively). In sum, the present article points to the value of conceptualizing schizotypal personality as a complex system of interacting cognitive, emotional, and affective characteristics.

Comparisons of schizotypal traits across 12 countries: Results from the International Consortium for Schizotypy Research.

BACKGROUND: Schizotypal traits are expressions of underlying vulnerability to psychotic disorders which have a potential impact on mental health status, neurocognition, quality of life, and daily functioning. To date, little research has examined epidemiologic landscape of schizotypal traits at the cross-national level. Our aim was to study the expression of schizotypal traits by sex, age, and country in a combined sample gathered from 12 countries. METHODS: A total of 27,001 participants completed the Schizotypal Personality Questionnaire (SPQ). The mean age of participants was 22.12 (SD=6.28); 37.5% (n=10,126) were males. RESULTS: Schizotypal traits varied according to sex, age, and country. Females scored higher than males in the positive dimension, whereas males scored higher in the disorganization dimension. By age, a significant decrease in the positive schizotypal traits was observed. Epidemiological expression of schizotypal traits varied by country. Moreover, several interactions by sex, age, and country were found. CONCLUSIONS: This pattern is similar to those found in patients with psychosis and psychotic-like experiences. These findings provide new insights and the opportunity to explore the phenotypic expression of schizotypal traits at cross-national level.

Brief assessment of schizotypal traits: A multinational study.

The Schizotypal Personality Questionnaire-Brief (SPQ-B) was developed with the aim of examining variations in healthy trait schizotypy, as well as latent vulnerability to psychotic-spectrum disorders. No previous study has studied the cross-cultural validity of the SPQ-B in a large cross-national sample. The main goal of the present study was to analyze the reliability and the internal structure of SPQ-B scores in a multinational sample of 28,426 participants recruited from 14 countries. The mean age was 22.63years (SD=7.08; range 16-68years), 37.7% (n=10,711) were men. The omega coefficients were high, ranging from 0.86 to 0.92 for the total sample. Confirmatory factor analysis revealed that SPQ-B items were grouped either in a theoretical structure of three first-order factors (Cognitive-Perceptual, Interpersonal, and Disorganized) or in a bifactor model (three first-order factors plus a general factor of schizotypal personality). In addition, the results supported configural but not strong measurement invariance of SPQ-B scores across samples. These findings provide new information about the factor structure of schizotypal personality, and support the validity and utility of the SPQ-B, a brief and easy tool for assessing self-reported schizotypal traits, in cross-national research. Theoretical and clinical implications for diagnostic systems, psychosis models, and cross-national mental health strategies are derived from these results.

Improving Measures via Examining the Behavior of Distractors in Multiple-Choice Tests: Assessment and Remediation.

The purpose of the present article was to illustrate, using an example from a national assessment, the value from analyzing the behavior of distractors in measures that engage the multiple-choice format. A secondary purpose of the present article was to illustrate four remedial actions that can potentially improve the measurement of the construct(s) under study. Participants were 2,248 individuals who took a national examination of chemistry. The behavior of the distractors was analyzed by modeling their behavior within the Rasch model. Potentially informative distractors were (a) further modeled using the partial credit model, (b) split onto separate items and retested for model fit and parsimony, (c) combined to form a "super" item or testlet, and (d) reexamined after deleting low-ability individuals who likely guessed on those informative, albeit erroneous, distractors. Results indicated that all but the item split strategies were associated with better model fit compared with the original model. The best fitted model, however, involved modeling and crediting informative distractors via the partial credit model or eliminating the responses of low-ability individuals who likely guessed on informative distractors. The implications, advantages, and disadvantages of modeling informative distractors for measurement purposes are discussed.

The Impact of Non-attempted and Dually-Attempted Items on Person Abilities Using Item Response Theory.

The purpose of the present study was to relate response strategy with person ability estimates. Two behavioral strategies were examined: (a) the strategy to skip items in order to save time on timed tests, and, (b) the strategy to select two responses on an item, with the hope that one of them may be considered correct. Participants were 4,422 individuals who were administered a standardized achievement measure related to math, biology, chemistry, and physics. In the present evaluation, only the physics subscale was employed. Two analyses were conducted: (a) a person-based one to identify differences between groups and potential correlates of those differences, and, (b) a measure-based analysis in order to identify the parts of the measure that were responsible for potential group differentiation. For (a) person abilities the 2-PL model was employed and later the 3-PL and 4-PL models in order to estimate upper and lower asymptotes of person abilities. For (b) differential item functioning, differential test functioning, and differential distractor functioning were investigated. Results indicated that there were significant differences between groups with completers having the highest ability compared to both non-attempters and dual responders. There were no significant differences between no-attempters and dual responders. The present findings have implications for response strategy efficacy and measure evaluation, revision, and construction.

Associations of differential schizotypal dimensions with executive working memory: A moderated-mediation analysis.

BACKGROUND: Increased schizotypal traits are observed in a percentage of the general population and in the schizophrenia-spectrum and have been associated with impairments in working memory. In this study we examined the effects of four schizotypal dimensions [Negative (NegS), Paranoid (ParS), Cognitive-Perceptual (CPS), Disorganized (DiS)] on executive working memory (EWM), as mediated by set-shifting, planning and control inhibition. We also examined whether these associations are moderated by family-history of psychosis. METHODS: Our sample consisted of 110 unaffected first-degree relatives of schizophrenia-spectrum patients and 120 control individuals. Schizotypy was assessed with the Schizotypal Personality Questionnaire. Participants were also tested with the Letter-Number Sequencing, Wisconsin Card Sorting, Stroop Color-Word and Stockings of Cambridge tasks. The effects of set-shifting, control inhibition and planning on the relationship between schizotypy and EWM were examined with mediation analyses. Moderated-mediation analyses examined potential moderating effects of group membership (unaffected relative/community participant). RESULTS: All mediators were significant in the relationship between NegS and EWM. The effects of ParS were mediated only by set-shifting and planning. Planning and control inhibition were the only significant mediators on the effects of CPS and DiS on EWM, respectively. The moderated-mediation analyses revealed that these findings apply only in the community group. CONCLUSIONS: We found that the effects of different schizotypal dimensions on EWM are mediated by other cognitive processes in individuals without personal/family history of psychosis. This is probably due to either more severe impairments in the cognitive processes of the relatives or restrictions in our sample and study-design.