Overfactoring in rating scale data: A comparison between factor analysis and item response theory.

Educational and psychological measurement is typically based on dichotomous variables or rating scales comprising a few ordered categories. When the mean of the observed responses approaches the upper or the lower bound of the scale, the distribution of the data becomes skewed and, if a categorical factor model holds in the population, the Pearson correlation between variables is attenuated. The consequence of this correlation attenuation is that the traditional linear factor model renders an excessive number of factors. This article presents the results of a simulation study investigating the problem of overfactoring and some solutions. We compare five widely known approaches: (1) The maximum-likelihood factor analysis (FA) model for normal data, (2) the categorical factor analysis (FAC) model based on polychoric correlations and maximum likelihood (ML) estimation, (3) the FAC model estimated using a weighted least squares algorithm, (4) the mean corrected chi-square statistic by Satorra-Bentler to handle the lack of normality, and (5) the Samejima's graded response model (GRM) from item response theory (IRT). Likelihood-ratio chi-square, parallel analysis (PA), and categorical parallel analysis (CPA) are used as goodness-of-fit criteria to estimate the number of factors in the simulation study. Our results indicate that the maximum-likelihood estimation led to overfactoring in the presence of skewed variables both for the linear and categorical factor model. The Satorra-Bentler and GRM constitute the most reliable alternatives to estimate the number of factors.

What are the consequences of ignoring cross-loadings in bifactor models? A simulation study assessing parameter recovery and sensitivity of goodness-of-fit indices.

Bifactor latent models have gained popularity and are widely used to model construct multidimensionality. When adopting a confirmatory approach, a common practice is to assume that all cross-loadings take zero values. This article presents the results of a simulation study exploring the impact of ignoring non-zero cross-loadings on the performance of confirmatory bifactor analysis. The present work contributes to previous research by including study conditions that had not been examined before. For instance, a wider range of values of the factor loadings both for the group factors and the cross-loadings is considered. Parameter recovery is analyzed, but the focus of the study is on assessing the sensitivity of goodness-of-fit indices to detect the model misspecification that involves ignoring non-zero cross-loadings. Several commonly used SEM fit indices are examined: both biased estimators of the fit index (CFI, GFI, and SRMR) and unbiased estimators (RMSEA and SRMR). Results indicated that parameter recovery worsens when ignoring moderate and large cross-loading values and using small sample sizes, and that commonly used SEM fit indices are not useful to detect such model misspecifications. We recommend the use of the unbiased SRMR index with a cutoff value adjusted by the communality level (R 2), as it is the only fit index sensitive to the model misspecification due to ignoring non-zero cross-loadings in the bifactor model. The results of the present study provide insights into modeling cross-loadings in confirmatory bifactor models but also practical recommendations to researchers.

Nominal Factor Analysis of Situational Judgment Tests: Evaluation of Latent Dimensionality and Factorial Invariance.

Situational judgment tests have gained popularity in educational and psychological measurement and are widely used in personnel assessment. A situational judgment item presents a hypothetical scenario and a list of actions, and the individuals are asked to select their most likely action for that scenario. Because actions have no explicit order, the item generates nominal responses consisting of the actions selected by the individuals. This article shows how to factor-analyze the nominal responses originated from such a test, including the estimation of the number of latent factors and a factor invariance analysis in a multiple group design. The method consists of applying the MNCM, a multidimensional extension of the nominal categories model by Bock. The article includes the results of two studies: (1) a simulation study about Type-I error rate, statistical power, and recovery of the parameters in a multigroup factorial invariance design and (2) a real data example using responses to a situational judgment test measuring gender stereotypes to illustrate the approach. Results suggest the use of the Akaike information criterion, Bayesian information criterion, and corrected Bayesian information criterion indices to guide the selection of the number of factors with nominal responses. All the analyses are conducted using the computer program Mplus. The code is included as Supplemental Material (available online) for the readers so that they can adapt it to their own purposes.

Validation of a Questionnaire for Personality Profiling using Cognitive Diagnostic Modeling.

This article describes the development of the ENCUIST (Extroversion, Neuroticism, Callous-Unemotional, Instability, Short-Test) questionnaire, which has been created to provide a personality profiling method based on a cognitive diagnostic modeling framework. The ENCUIST measures the attributes of extroversion, neuroticism, callous unemotionality and overt expressions of anger that are relevant in a forensic context. The scores provided by the ENCUIST are binary classifications of the individuals (high/low) in these attributes. The ENCUIST was developed using a sample of 516 subjects to study its validation through psychometric procedures, including factor analysis, cognitive diagnostic modeling and structural equation modeling. The results supported a four-factor structure. Linear regressions were used to evaluate the predictive validity of the scores provided by ENCUIST with respect to two external criteria that are relevant in the forensic context, namely behavioral activation and behavioral inhibition. The results showed that the extroversion dimension is positively related to behavioral activation, although the effect size is modest and the proportion of explained variance is only 11%. Moreover, the dimensions of neuroticism and anger expression are positively related to behavioral inhibition, with 7% of the variance explained. Together, these results suggest that cognitive diagnostic models are useful tools for the elaboration of personality profiles based on classifying subjects along binary attributes.

Validation of a questionnaire for personality profiling using cognitive diagnostic modeling

This article describes the development of the ENCUIST (Extroversion, Neuroticism, Callous-Unemotional, Instability, Short-Test) questionnaire, which has been created to provide a personality profiling method based on a cognitive diagnostic modeling framework. The ENCUIST measures the attributes of extroversion, neuroticism, callous unemotionality and overt expressions of anger that are relevant in a forensic context. The scores provided by the ENCUIST are binary classifications of the individuals (high/low) in these attributes. The ENCUIST was developed using a sample of 516 subjects to study its validation through psychometric procedures, including factor analysis, cognitive diagnostic modeling and structural equation modeling. The results supported a four-factor structure. Linear regressions were used to evaluate the predictive validity of the scores provided by ENCUIST with respect to two external criteria that are relevant in the forensic context, namely behavioral activation and behavioral inhibition. The results showed that the extroversion dimension is positively related to behavioral activation, although the effect size is modest and the proportion of explained variance is only 11%. Moreover, the dimensions of neuroticism and anger expression are positively related to behavioral inhibition, with 7% of the variance explained. Together, these results suggest that cognitive diagnostic models are useful tools for the elaboration of personality profiles based on classifying subjects along binary attributes

No disponible

Bayesian Dimensionality Assessment for the Multidimensional Nominal Response Model.

This article introduces Bayesian estimation and evaluation procedures for the multidimensional nominal response model. The utility of this model is to perform a nominal factor analysis of items that consist of a finite number of unordered response categories. The key aspect of the model, in comparison with traditional factorial model, is that there is a slope for each response category on the latent dimensions, instead of having slopes associated to the items. The extended parameterization of the multidimensional nominal response model requires large samples for estimation. When sample size is of a moderate or small size, some of these parameters may be weakly empirically identifiable and the estimation algorithm may run into difficulties. We propose a Bayesian MCMC inferential algorithm to estimate the parameters and the number of dimensions underlying the multidimensional nominal response model. Two Bayesian approaches to model evaluation were compared: discrepancy statistics (DIC, WAICC, and LOO) that provide an indication of the relative merit of different models, and the standardized generalized discrepancy measure that requires resampling data and is computationally more involved. A simulation study was conducted to compare these two approaches, and the results show that the standardized generalized discrepancy measure can be used to reliably estimate the dimensionality of the model whereas the discrepancy statistics are questionable. The paper also includes an example with real data in the context of learning styles, in which the model is used to conduct an exploratory factor analysis of nominal data.

Turn-amplitude analysis as a diagnostic test for myofascial syndrome in patients with chronic pelvic pain.

BACKGROUND: Myofascial pain syndrome of the pelvic floor (MPSPF) is a common disease in the context of chronic pelvic pain (CPP); however, there is currently no gold-standard test to diagnose it. OBJECTIVE: To validate the turns-amplitude analysis (TAA) as a diagnostic test for MPSPF in patients with CPP. METHODS: A case-control study was performed, and patients were consecutively sampled within a specified period of time. A total of 128 patients were included: 64 patients with CPP (32 men and 32 women) and 64 control patients (32 men and 32 women). The same operator conducted all tests. Electromyography of the TAA is based on the collection of motor unit potentials that measure the number of changes in the signal and the mean amplitude of the changes. The electromyogram transfers the data to a graphical point cloud, which enables the patient's results to be compared with the results of the healthy subjects. RESULTS: In patients and control subjects, the sensitivity and specificity of the proposed diagnostic test showed a marked clinical significance: the sensitivity was 83%, and the specificity was 100%. A positive predictive value of 1 (95% CI 1 to 1) and a negative predictive value of 0.85 (95% CI 0.77 to 0.93) were observed. CONCLUSION: TAA is a reliable diagnostic test to detect MPSPF. Further studies are needed to reproduce these results.

Recovery of Weak Factor Loadings When Adding the Mean Structure in Confirmatory Factor Analysis: A Simulation Study.

This article extends previous research on the recovery of weak factor loadings in confirmatory factor analysis (CFA) by exploring the effects of adding the mean structure. This issue has not been examined in previous research. This study is based on the framework of Yung and Bentler (1999) and aims to examine the conditions that affect the recovery of weak factor loadings when the model includes the mean structure, compared to analyzing the covariance structure alone. A simulation study was conducted in which several constraints were defined for one-, two-, and three-factor models. Results show that adding the mean structure improves the recovery of weak factor loadings and reduces the asymptotic variances for the factor loadings, particularly for the models with a smaller number of factors and a small sample size. Therefore, under certain circumstances, modeling the means should be seriously considered for covariance models containing weak factor loadings.

[Anal stretching device for patients with chronic prostatitis and chronic pelvic pain syndrome].

OBJECTIVES: Chronic pelvic pain syndrome (CPPS) is a poorly understood and ill-treated condition. It is accompanied by the shortening and increase in tone of the pelvic floor muscles and is closely related to myofascial pain syndrome (MPS). This study aims to evaluate the utility of an anal stretching device (ASD) for improving the pain manifestations of chronic prostatitis (CP) and CPPS. METHODS: Thirty-one men(38.6 years ± 8.2) were consecutively recruited with an average monitoring period of 14.4 months (± 8.2). The treatment duration was between six months and three years. A clinical history was compiled along with a physical examination and neurophysiological tests. To evaluate pain, the Visual Analogue Scale (VAS) was used before and after treatment; at the final visit, the Clinical Global Impression of Improvement scale (CGI-I) was administered. The ASD is a device that is commercially available in different diameters and lengths. RESULT: Patients were diagnosed with MPS using neurophysiological tests. Significant differences were found before and after the treatment when evaluating the intensity of the pain using the VAS (6.1±2.1 vs. 1.9±1.3; p < .001). The CGI-I showed a total of 21 patients (70%) whose symptoms were improved or very much improved. Only one patient was worse after the treatment. CONCLUSIONS: ASD appears to be a safe and useful tool to treat the pain manifestations of CPPS without notable side effects.

Dispositivo anal para estiramientos: una herramientoa útil para pacientes con prostatitis crónica/SDPC / Anal stretching device for patients with chronic prostatitis and chronic pelvic syndrome

OBJETIVO: El síndrome de dolor pélvico crónico (SDPC) es una entidad pobremente definida, entendida y tratada, siempre va acompañada de un acortamiento y un aumento del tono de los músculos del suelo pélvico y está estrechamente relacionada con el síndrome de dolor miofascial (SDM). Este estudio trata de valorar si el dispositivo anal (DAE) para estiramientos es útil para mejorar el dolor en la prostatitis crónica/SDPC. MÉTODOS: 31 hombres (38,6 años ± 8,2) se reclutaron de forma consecutiva y con un seguimiento medio de 14, 4 meses (± 8,2). La duración del tratamiento fue entre seis meses y tres años. Se realizó historia clínica, examen físico y tests neurofisiológicos. En la evaluación del dolor se utilizó la escala visual analógica antes y después del tratamiento y en la última visita se completo con la escala de impresión de mejoría global del paciente (CGI-I). El DAE es un dispositivo disponible en el mercado en diferentes anchos y longitudes. RESULTADO: los pacientes fueron diagnosticados de SDM después de realizarles los tests neurofisiológicos. Se encontraron diferencias significativas antes y después con la EVA al evaluar la intensidad del dolor (6, 1±2.1 vs. 1, 9±1.3; p < 0.001) y el CGI-I mostró un total de 21 pacientes (70%) estar mucho mejor o muchísimo mejor. Solo un paciente empeoro durante el tratamiento. CONCLUSIONES: El DAE perece una herramienta segura y útil para mejorar el dolor del SDPC, sin efectos secundarios reseñables(AU)

OBJECTIVES: Chronic pelvic pain syndrome (CPPS) is a poorly understood and ill-treated condition. It is accompanied by the shortening and increase in tone of the pelvic floor muscles and is closely related to myofascial pain syndrome (MPS). This study aims to evaluate the utility of an anal stretching device (ASD) for improving the pain manifestations of chronic prostatitis (CP) and CPPS. METHODS: Thirty-one men (38.6 years ± 8.2) were consecutively recruited with an average monitoring period of 14.4 months (± 8.2). The treatment duration was between six months and three years. A clinical history was compiled along with a physical examination and neurophysiological tests. To evaluate pain, the Visual Analogue Scale (VAS) was used before and after treatment; at the final visit, the Clinical Global Impression of Improvement scale (CGI-I) was administered. The ASD is a device that is commercially available in different diameters and lengths. RESULT: Patients were diagnosed with MPS using neurophysiological tests. Significant differences were found before and after the treatment when evaluating the intensity of the pain using the VAS (6.1±2.1 vs. 1.9±1.3; p <0 .001). The CGI-I showed a total of 21 patients (70%) whose symptoms were improved or very much improved. Only one patient was worse after the treatment. CONCLUSIONS: ASD appears to be a safe and useful tool to treat the pain manifestations of CPPS without notable side effects(AU)

Factorial invariance in a repeated measures design: an application to the study of person-organization fit.

An important methodological concern of any research based on a person-environment (P-E) fit approach is the operationalization of the fit, which imposes some measurement requirements that are rarely empirically tested with statistical methods. Among them, the assessment of the P and E components along commensurate dimensions is possibly the most cited one. This paper proposes to test the equivalence across the P and E measures by analyzing the measurement invariance of a multi-group confirmatory factor analysis model. From a methodological point of view, the distinct aspect of this approach within the context of P-E fit research is that measurement invariance is assessed in a repeated measures design. An example illustrating the procedure in a person-organization (P-O) fit dataset is provided. Measurement invariance was tested at five different hierarchical levels: (1) configural, (2) first-order factor loadings, (3) second-order factor loadings, (4) residual variances of observed variables, and (5) disturbances of first-order factors. The results supported the measurement invariance across the P and O measures at the third level. The implications of these findings for P-E fit studies are discussed.

Factorial invariance in a repeated measures design: an application to the study of person-organization fit

An important methodological concern of any research based on a person-environment (P-E) fit approach is the operationalization of the fit, which imposes some measurement requirements that are rarely empirically tested with statistical methods. Among them, the assessment of the P and E components along commensurate dimensions is possibly the most cited one. This paper proposes to test the equivalence across the P and E measures by analyzing the measurement invariance of a multi-group confirmatory factor analysis model. From a methodological point of view, the distinct aspect of this approach within the context of P-E fit research is that measurement invariance is assessed in a repeated measures design. An example illustrating the procedure in a person-organization (P-O) fit dataset is provided. Measurement invariance was tested at five different hierarchical levels: (1) configural, (2) first-order factor loadings, (3) second-order factor loadings, (4) residual variances of observed variables, and (5) disturbances of first-order factors. The results supported the measurement invariance across the P and O measures at the third level. The implications of these findings for P-E fit studies are discussed (AU)

Un importante aspecto metodológico de las investigaciones basadas en el enfoque del ajuste persona-ambiente (P-A) se refiere a la operacionalización del ajuste P-A, que impone una serie de supuestos que raramente se evalúan utilizando métodos estadísticos. Entre ellos, la evaluación de los componentes P y A en dimensiones conmensurables o equivalentes, es posiblemente uno de los más citados. En este artículo se propone que la equivalencia entre las medidas de P y A puede evaluarse mediante el análisis de la invarianza factorial de un modelo de análisis factorial confirmatorio multi-grupo. Desde un punto de vista metodológico, la aportación del enfoque propuesto a la investigación del ajuste P-A es que la invarianza en la medida se evalúa en un diseño de medidas repetidas. El artículo también presenta un ejemplo que ilustra el procedimiento propuesto con datos para una medida de ajuste persona-organización (P-O). La invarianza en la medida se evalúa en cinco niveles jerárquicos: (1) configural, (2) saturaciones de primer orden, (3) saturaciones de segundo orden, (4) varianzas error de las variables observadas y (5) errores típicos asociados a los factores de primer orden. Los resultados dan apoyo a la invarianza factorial en las medidas de P y O en el tercer nivel. Por último, se discuten las implicaciones del enfoque propuesto para los estudios de ajuste P-A (AU)

Recovery of weak factor loadings in confirmatory factor analysis under conditions of model misspecification.

This article presents the results of two Monte Carlo simulation studies of the recovery of weak factor loadings, in the context of confirmatory factor analysis, for models that do not exactly hold in the population. This issue has not been examined in previous research. Model error was introduced using a procedure that allows for specifying a covariance structure with a specified discrepancy in the population. The effects of sample size, estimation method (maximum likelihood vs. unweighted least squares), and factor correlation were also considered. The first simulation study examined recovery for models correctly specified with the known number of factors, and the second investigated recovery for models incorrectly specified by underfactoring. The results showed that recovery was not affected by model discrepancy for the correctly specified models but was affected for the incorrectly specified models. Recovery improved in both studies when factors were correlated, and unweighted least squares performed better than maximum likelihood in recovering the weak factor loadings.

Análisis de un test de aritmética mediante el modelo logístico lineal de rasgo latente / Analisys of an arithmetic test by means of a logistic linear trait model

En este trabajo se presenta un ejemplo sobre cómo integrar la psicología cognitiva con la psicometría para determinar las fuentes de dificultad de los ítems de un Test de Aritmética (TA),construido por los autores, por medio de su análisis con el modelo LLTM. Para alcanzar este propósito se han propuesto un grupo de operaciones necesarias para resolver los ítems del test, se ha evaluado su dimensionalidad y se ha estudiado el ajuste de los ítems tanto al modelo de Rasch como al LLTM. Los resultados obtenidos mediante el análisis del TA han permitidoilustrar las ventajas y limitaciones del LLTM en la medición componencial, así como presentar las principales técnicas empleadas en el proceso.

This paper presents an illustration of the integration of cognitive psychology and psychometric models to determine sources of item difficulty in an Arithmetic Test (AT), constructed by theauthors, by means of its analysis with the LLTM. In order to reach this aim, a group of operations required to solve the items of the test were proposed, the dimensionality was evaluated, and thegoodness of fit of items to both the Rasch and the LLTM models was studied. Results obtained were used to illustrate the advantages and limitations of the LLTM for the componential measurement; the main techniques used in the analysis were also introduced.

Extending the CLAST sequential rule to one-way ANOVA under group sampling.

Several studies have demonstrated that the fixed-sample stopping rule (FSR), in which the sample size is determined in advance, is less practical and efficient than are sequential-stopping rules. The composite limited adaptive sequential test (CLAST) is one such sequential-stopping rule. Previous research has shown that CLAST is more efficient in terms of sample size and power than are the FSR and other sequential rules and that it reflects more realistically the practice of experimental psychology researchers. The CLAST rule has been applied only to the t test of mean differences with two matched samples and to the chi-square independence test for twofold contingency tables. The present work extends previous research on the efficiency of CLAST to multiple group statistical tests. Simulation studies were conducted to test the efficiency of the CLAST rule for the one-way ANOVA for fixed effects models. The ANOVA general test and two linear contrasts of multiple comparisons among treatment means are considered. The article also introduces four rules for allocating N observations to J groups under the general null hypothesis and three allocation rules for the linear contrasts. Results show that the CLAST rule is generally more efficient than the FSR in terms of sample size and power for one-way ANOVA tests. However, the allocation rules vary in their optimality and have a differential impact on sample size and power. Thus, selecting an allocation rule depends on the cost of sampling and the intended precision.

Validación de la estructura cognitiva del test de signos mediante modelos de ecuaciones estructurales / Validation of the cognitive structure of the test of signs by structural equation modeling

El objetivo de este trabajo es realizar un estudio de validación de los procesos cognitivos implicados en la resolución de ítems de un test de operaciones aritméticas básicas entre números enteros. La validación de la estructura cognitiva propuesta para la tarea se lleva a cabo mediante modelos de ecuaciones estructurales y triangulación. Los resultados muestran relaciones de subordinación fuertes y positivas entre los ítems en algunas de las rutas, apoyando solo parcialmente la estructura propuesta. Sin embargo, la triangulación aporta una mayor evidencia de validez

The present work is aimed to carry out a validation study of the cognitive operations required for the correct solution of items of a math test which includes basic arithmetic operations between integer numbers. The validation of the hypothesized cognitive structure is made by means of structural equation modeling and triangulation methods. Results show strong and positive cognitive subordination relationships between some items but the structural equation model fit only provides a partial support for the proposed structure. However, the triangulation procedure provides further evidence of validity

Optimization of sample size in controlled experiments: the CLAST rule.

Sequential rules are explored in the context of null hypothesis significance testing. Several studies have demonstrated that the fixed-sample stopping rule, in which the sample size used by researchers is determined in advance, is less practical and less efficient than sequential stopping rules. It is proposed that a sequential stopping rule called CLAST (composite limited adaptive sequential test) is a superior variant of COAST (composite open adaptive sequential test), a sequential rule proposed by Frick (1998). Simulation studies are conducted to test the efficiency of the proposed rule in terms of sample size and power. Two statistical tests are used: the one-tailed t test of mean differences with two matched samples, and the chi-square independence test for twofold contingency tables. The results show that the CLAST rule is more efficient than the COAST rule and reflects more realistically the practice of experimental psychology researchers.

[Validation of the cognitive structure of the test of signs by structural equation modeling]. / Validación de la estructura cognitiva del test de signos mediante modelos de ecuaciones estructurales.

Validation of the cognitive structure of the test of signs by structural equation modeling. The present work is aimed to carry out a validation study of the cognitive operations required for the correct solution of items of a math test which includes basic arithmetic operations between integer numbers. The validation of the hypothesized cognitive structure is made by means of structural equation modeling and triangulation methods. Results show strong and positive cognitive subordination relationships between some items but the structural equation model fit only provides a partial support for the proposed structure. However, the triangulation procedure provides further evidence of validity.