Analyzing and Comparing Univariate, Multivariate, and Bifactor Generalizability Theory Designs for Hierarchically Structured Personality Traits.

We demonstrate how to use structural equation models to represent generalizability theory-based univariate, multivariate, and bifactor model designs. Analyses encompassed multi-occasion data obtained from the recently expanded form of the Big Five Inventory (BFI-2) that measures the broad personality domain constructs Agreeableness, Conscientiousness, Extraversion, Negative Emotionality, and Open-Mindedness along with three nested subdomain facets within each global domain. Results overall highlighted the importance of taking both item and occasion effects into account but underscored additional benefits of the multivariate and bifactor designs in providing more appropriate indices of generalizability for composite scores and effective ways to gauge subscale added value. Bifactor models further extended partitioning of universe score variance to separate general and group factor effects at both composite and subscale levels, expanded score consistency indices to distinguish or combine such effects, and allowed for further evaluation of score dimensionality and subscale viability. We provide guidelines, formulas, and code in R for analyzing all illustrated designs within the article and extended online Supplemental Material.

Applying multivariate generalizability theory to psychological assessments.

Multivariate generalizability theory (GT) represents a comprehensive framework for quantifying score consistency, separating multiple sources contributing to measurement error, correcting correlation coefficients for such error, assessing subscale viability, and determining the best ways to change measurement procedures at different levels of score aggregation. Despite such desirable attributes, multivariate GT has rarely been applied when measuring psychological constructs and far less often than univariate techniques that are subsumed within that framework. Our purpose in this tutorial is to describe multivariate GT in a simple way and illustrate how it expands and complements univariate procedures. We begin with a review of univariate GT designs and illustrate how such designs serve as subcomponents of corresponding multivariate designs. Our empirical examples focus primarily on subscale and composite scores for objectively scored measures, but guidelines are provided for applying the same techniques to subjectively scored performance and clinical assessments. We also compare multivariate GT indices of score consistency and measurement error to those obtained using alternative GT-based procedures and across different software packages for analyzing multivariate GT designs. Our online supplemental materials include instruction, code, and output for common multivariate GT designs analyzed using mGENOVA and the gtheory, glmmTMB, lavaan, and related packages in R. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A three-dimensional taxonomy of achievement emotions.

We present a three-dimensional taxonomy of achievement emotions that considers valence, arousal, and object focus as core features of these emotions. By distinguishing between positive and negative emotions (valence), activating and deactivating emotions (arousal), and activity emotions, prospective outcome emotions, and retrospective outcome emotions (object focus), the taxonomy has a 2 × 2 × 3 structure representing 12 groups of achievement emotions. In four studies across different countries (N = 330, 235, 323, and 269 participants in Canada, the United States, Germany, and the U.K., respectively), we investigated the empirical robustness of the taxonomy in educational (Studies 1-3) and work settings (Study 4). An expanded version of the Achievement Emotions Questionnaire was used to assess 12 key emotions representing the taxonomy. Consistently across the four studies, findings from multilevel facet analysis and structural equation modeling documented the importance of the three dimensions for explaining achievement emotions. In addition, based on hypotheses about relations with external variables, the findings show clear links of the emotions with important antecedents and outcomes. The Big Five personality traits, appraisals of control and value, and context perceptions were predictors of the emotions. The 12 emotions, in turn, were related to participants' use of strategies, cognitive performance, and self-reported health problems. Taken together, the findings provide robust evidence for the unique positions of different achievement emotions in the proposed taxonomy, as well as unique patterns of relations with external variables. Directions for future research and implications for policy and practice are discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Expanding bifactor models of psychological traits to account for multiple sources of measurement error.

Over the last decade, applications of bifactor modeling within clinical settings have increased markedly but typically rely on data collected on single occasions. A shortcoming of such research is that reliability coefficients are likely inflated because key sources of measurement error are inadequately modeled and/or confounded with construct variance. We address these problems using three variations of multi-occasion bifactor models with Bayesian-derived parameter estimates to separate systematic variance into general and group factor effects and measurement error into three subcomponents (transient, specific-factor, and random-response). Collectively, these models produce indices of reliability and validity aligned with both standard confirmatory factor models and generalizability designs that extend interpretations of results to the broader domains from which items and occasions are sampled. We demonstrate how these techniques can provide new insights into psychometric properties of scores using Negative Emotionality domain and facet scales from the newly updated Big Five Inventory (BFI-2; Soto & John, 2017). Overall, the two-occasion congeneric bifactor model provided the best fit to the data and most informative indices for revising measures, examining dimensionality of composite and subscale scores, and evaluating the viability of those scores. We include code in R for analyzing all models in our extended online Supplemental Material. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Using Parallel Splits with Self-Report and Other Measures to Enhance Precision in Generalizability Theory Analyses.

Although generalizability theory (G-theory) provides indices of reliability that take multiple sources of measurement error into account, those indices are typically conservative in nature because they reflect random rather than classical parallelism. One way to address these shortcomings is to use parallel splits rather than items as the unit of analysis in G-theory designs. In this article, we provide the most extensive treatment to date in how to effectively integrate parallel splits into an extended set of G-theory designs using data from the newly developed version of the Big Five Inventory (BFI-2; Soto & John). Results revealed that properly designed splits approximated classical parallelism while improving overall score consistency and reducing key components of measurement error. Variance components within appropriately chosen G-theory designs also provided effective means to evaluate the quality of splits and determine the best ways to improve score consistency and reduce specific sources of measurement error. To help readers in applying these techniques, we provide a comprehensive instructional supplement with code in R for creating parallel splits, analyzing all illustrated designs, and modifying those designs for other objectively or subjectively scored measures.

Interrelationships between latent state-trait theory and generalizability theory within a structural equation modeling framework.

Over recent years, latent state-trait theory (LST) and generalizability theory (GT) have been applied to a wide variety of situations in numerous disciplines to enhance understanding of the reliability and validity of assessment data. Both methodologies involve partitioning of observed score variation into systematic and measurement error components. LST theory is focused on separating state, trait, error, and sometimes method effects, whereas generalizability theory is concerned with distinguishing universe score effects from multiple sources of measurement error. Despite these fundamental differences in focus, LST and GT share much in common. In this article, we use data from a widely used personality measure to illustrate similarities and differences between these two frameworks and show how the same data can be readily interpreted from both perspectives. We also provide comprehensive instructional online supplemental materials to demonstrate how to analyze data using the R package for all LST models and GT designs discussed. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Expanding G-Theory Models to Incorporate Congeneric Relationships: Illustrations Using the Big Five Inventory.

We used structural equation modeling techniques to expand traditional generalizability theory (G-theory) models to allow for congeneric relationships among item responses while accounting for the primary sources of measurement error that affect results from objectively scored, self-report measures. Data came from 919 respondents who completed the Agreeableness, Conscientiousness, Extraversion, Neuroticism, and Openness subscales of the Big Five Inventory (BFI; John et al., 1991) on two occasions. When compared to traditional and factor-based essential tau-equivalent G-theory models, congeneric models on average yielded superior fit statistics, higher estimates of reliability, and lower estimates of transient and specific-factor measurement error. Essential tau-equivalent and congeneric factor models also were configured to allow for simultaneous partitioning of systematic and measurement error variance at both total score and individual item levels. We provide detailed guidelines, examples, and computer code in R for all models discussed in an extended online supplement to enable readers to apply the demonstrated techniques.

Interchangeability of Results From Computerized and Traditional Administration of the BIDR: Convenience Can Match Reality.

Although interchangeability of results across computer and paper modes of administration is commonly assumed, recent meta-analyses and individual studies continue to reveal mean differences in scores for measures of socially desirable responding (SDR). Results from these studies have also failed to include new methods of scoring and crucial aspects of scaling, reliability, validity, and administration emphasized in professional standards for assessment that are essential in establishing equivalence. We addressed these shortcomings in a comprehensive, repeated measures investigation for 6 ways of scoring the Balanced Inventory of Desirable Responding (BIDR), one of the most frequently administered companion measures of SDR in research and practice. Results for many previously unexamined, standards-driven aspects of scaling, reliability, and validity strongly supported the interchangeability of scores across modes of administration. Computer questionnaires also took considerably less time to complete and were overwhelmingly favored by respondents in relation to physical characteristics of the measures, appraisals of the the assessment experience, and perceived quality of information obtained. Collectively, these results highlight the importance of following professional standards when constructing and administering computerized assessments and the evolution of computer technology in providing viable, effective, and accepted platforms for administering and scoring the BIDR in numerous ways.

Using generalizability theory with continuous latent response variables.

In this article, we illustrate ways in which generalizability theory (G-theory) can be used with continuous latent response variables (CLRVs) to address problems of scale coarseness resulting from categorization errors caused by representing ranges of continuous variables by discrete data points and transformation errors caused by unequal interval widths between those data points. The mechanism to address these problems is applying structural equation modeling (SEM) as a tool in deriving variance components needed to estimate indices of score consistency and validity. Illustrations include quantification of multiple sources of measurement error, use of non-nested and nested designs, derivation of indices of consistency for norm- and criterion-referenced interpretation of scores, estimation of effects when changing measurement procedures and designs, and disattenuation of correlation coefficients for measurement error. These illustrations underscore the effectiveness of G-theory with continuous latent response variables in providing stable indices of reliability and validity that are reasonably independent of the number of original scale points used, unevenness of scale intervals, and average degree of item skewness. We discuss general distinctions in reliability estimation within G-theory, SEM, and classical test theory; make specific recommendations for using G-theory on raw score and CLRV metrics; and provide computer code in an online supplement for doing all key analyses demonstrated in the article using R and Mplus. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Using Generalizability Theory to Disattenuate Correlation Coefficients for Multiple Sources of Measurement Error.

Over the years, research in the social sciences has been dominated by reporting of reliability coefficients that fail to account for key sources of measurement error. Use of these coefficients, in turn, to correct for measurement error can hinder scientific progress by misrepresenting true relationships among the underlying constructs being investigated. In the research reported here, we addressed these issues using generalizability theory (G-theory) in both traditional and new ways to account for the three key sources of measurement error (random-response, specific-factor, and transient) that affect scores from objectively scored measures. Results from 20 widely used measures of personality, self-concept, and socially desirable responding showed that conventional indices consistently misrepresented reliability and relationships among psychological constructs by failing to account for key sources of measurement error and correlated transient errors within occasions. The results further revealed that G-theory served as an effective framework for remedying these problems. We discuss possible extensions in future research and provide code from the computer package R in an online supplement to enable readers to apply the procedures we demonstrate to their own research.

Applications of generalizability theory and their relations to classical test theory and structural equation modeling.

Although widely recognized as a comprehensive framework for representing score reliability, generalizability theory (G-theory), despite its potential benefits, has been used sparingly in reporting of results for measures of individual differences. In this article, we highlight many valuable ways that G-theory can be used to quantify, evaluate, and improve psychometric properties of scores. Our illustrations encompass assessment of overall reliability, percentages of score variation accounted for by individual sources of measurement error, dependability of cut-scores for decision making, estimation of reliability and dependability for changes made to measurement procedures, disattenuation of validity coefficients for measurement error, and linkages of G-theory with classical test theory and structural equation modeling. We also identify computer packages for performing G-theory analyses, most of which can be obtained free of charge, and describe how they compare with regard to data input requirements, ease of use, complexity of designs supported, and output produced. (PsycINFO Database Record

Practical Applications of Generalizability Theory for Designing, Evaluating, and Improving Psychological Assessments.

In this article, we illustrate how generalizability theory (G-theory) can extend traditional assessment methods for designing, improving, and evaluating results from both objectively and subjectively scored measures of individual differences. Our illustrations include quantification of multiple sources of measurement error, derivation of unique indexes of consistency for norm- and criterion-referenced interpretations of scores, estimation of score consistency when changing a measurement procedure, and disattenuation of correlation coefficients for measurement error. We also expand G-theory analyses beyond the item level to include parcels and split measures and highlight linkages among G-theory, classical test theory, and structural equation modeling. Computer code and sample data are provided in online supplements to help readers apply the demonstrated techniques to their own assessments.

Using G-Theory to Enhance Evidence of Reliability and Validity for Common Uses of the Paulhus Deception Scales.

We applied a new approach to Generalizability theory (G-theory) involving parallel splits and repeated measures to evaluate common uses of the Paulhus Deception Scales based on polytomous and four types of dichotomous scoring. G-theory indices of reliability and validity accounting for specific-factor, transient, and random-response measurement error supported use of polytomous over dichotomous scores as contamination checks; as control, explanatory, and outcome variables; as aspects of construct validation; and as indexes of environmental effects on socially desirable responding. Polytomous scoring also provided results for flagging faking as dependable as those when using dichotomous scoring methods. These findings argue strongly against the nearly exclusive use of dichotomous scoring for the Paulhus Deception Scales in practice and underscore the value of G-theory in demonstrating this. We provide guidelines for applying our G-theory techniques to other objectively scored clinical assessments, for using G-theory to estimate how changes to a measure might improve reliability, and for obtaining software to conduct G-theory analyses free of charge.

Development of the Classism Attitudinal Profile (CAP).

Despite increasing interest in social class issues within psychology, there are a limited number of theoretically rooted instruments to measure subjective social class, particularly related to classism. The purpose of this project was to create a brief, psychometrically sound, and theoretically grounded instrument, called the Classism Attitudinal Profile (CAP), designed to measure 2 aspects of classism (downward and upward) defined in Liu's (2011) Social Class World View Model Revised (SCWM-R). Data from 2 independent samples (n = 608, n = 199) provided evidence in support of the consistency (alpha and test-retest coefficients), anticipated factor structure, and convergent/discriminant validity of CAP subscale scores. Downward and upward classism scores were only modestly correlated with each other and differentially correlated with other measures, thereby demonstrating that CAP subscales measure distinct constructs rather than opposite ends of the same continuum. Validity of CAP scores was further supported by logically consistent patterns of relationships with measures of subjective social status, materialistic values, Protestant work ethic, life satisfaction, racism, sexism, and key demographic variables. (PsycINFO Database Record

Psychometric properties for the Balanced Inventory of Desirable Responding: dichotomous versus polytomous conventional and IRT scoring.

[Correction Notice: An Erratum for this article was reported in Vol 26(3) of Psychological Assessment (see record 2014-16017-001). The mean, standard deviation and alpha coefficient originally reported in Table 1 should be 74.317, 10.214 and .802, respectively. The validity coefficients in the last column of Table 4 are affected as well. Correcting this error did not change the substantive interpretations of the results, but did increase the mean, standard deviation, alpha coefficient, and validity coefficients reported for the Honesty subscale in the text and in Tables 1 and 4. The corrected versions of Tables 1 and Table 4 are shown in the erratum.] Item response theory (IRT) models were applied to dichotomous and polytomous scoring of the Self-Deceptive Enhancement and Impression Management subscales of the Balanced Inventory of Desirable Responding (Paulhus, 1991, 1999). Two dichotomous scoring methods reflecting exaggerated endorsement and exaggerated denial of socially desirable behaviors were examined. The 1- and 2-parameter logistic models (1PLM, 2PLM, respectively) were applied to dichotomous responses, and the partial credit model (PCM) and graded response model (GRM) were applied to polytomous responses. For both subscales, the 2PLM fit dichotomous responses better than did the 1PLM, and the GRM fit polytomous responses better than did the PCM. Polytomous GRM and raw scores for both subscales yielded higher test-retest and convergent validity coefficients than did PCM, 1PLM, 2PLM, and dichotomous raw scores. Information plots showed that the GRM provided consistently high measurement precision that was superior to that of all other IRT models over the full range of both construct continuums. Dichotomous scores reflecting exaggerated endorsement of socially desirable behaviors provided noticeably weak precision at low levels of the construct continuums, calling into question the use of such scores for detecting instances of "faking bad." Dichotomous models reflecting exaggerated denial of the same behaviors yielded much better precision at low levels of the constructs, but it was still less precision than that of the GRM. These results support polytomous over dichotomous scoring in general, alternative dichotomous scoring for detecting faking bad, and extension of GRM scoring to situations in which IRT offers additional practical advantages over classical test theory (adaptive testing, equating, linking, scaling, detecting differential item functioning, and so forth).

A generalizability analysis of score consistency for the Balanced Inventory of Desirable Responding.

Our goal in this investigation was to evaluate the reliability of scores from the Balanced Inventory of Desirable Responding (BIDR) more comprehensively than in prior research using a generalizability-theory framework based on both dichotomous and polytomous scoring of items. Generalizability coefficients accounting for specific-factor, transient, and random-response error ranged from .64 to .75 for the BIDR's Self-Deception Enhancement (SDE) and Impression Management (IM) subscale scores, and these values were systematically lower than corresponding alpha (.66 to .83) and 1-week test-retest (.78 to .86) coefficients. Polytomous scoring provided higher reliability than dichotomous scoring on nearly all indexes reported. Random-response (8%-17%) and specific factor error (11%-17%) exceeded transient error (3%-6%) for both subscales and scoring methods. Doubling the number of items on a single occasion provided greater improvements in generalizability (.76-.83) than aggregating scores across 2 administrations (.72-.81). Both scoring methods provided reasonably high indexes of consistency (φ coefficients≥.91) at cut scores on the IM scale for detecting faked responses when all sources of error were taken into account. Implications of these results for common uses of the BIDR are discussed.

Computer versus paper-and-pencil assessment of educational development: a comparison of psychometric features and examinee preferences.

The purposes of this study were to assess the comparability of scores obtained from computer and paper-and-pencil versions of the Iowa Tests of Educational Development and to evaluate examinees' attitudes about multiple aspects of test administration in the two modes. Findings supported the comparability of scores across administration modes with regard to scaling (means and standard deviations), internal consistency, and criterion- and construct-related validity. Overall, examinees preferred taking the computerized tests and valued many operational features of those tests. The least favorable attitudes were reported for the literary skills tests and their scrollable reading passages in particular.

Model assessment and model building in fMRI.

Model quality is rarely assessed in fMRI data analyses and less often reported. This may have contributed to several shortcomings in the current fMRI data analyses, including: (1) Model mis-specification, leading to incorrect inference about the activation-maps, SPM[t] and SPM[F]; (2) Improper model selection based on the number of activated voxels, rather than on model quality; (3) Under-utilization of systematic model building, resulting in the common but suboptimal practice of using only a single, pre-specified, usually over-simplified model; (4) Spatially homogenous modeling, neglecting the spatial heterogeneity of fMRI signal fluctuations; and (5) Lack of standards for formal model comparison, contributing to the high variability of fMRI results across studies and centers. To overcome these shortcomings, it is essential to assess and report the quality of the models used in the analysis. In this study, we applied images of the Durbin-Watson statistic (DW-map) and the coefficient of multiple determination (R(2)-map) as complementary tools to assess the validity as well as goodness of fit, i.e., quality, of models in fMRI data analysis. Higher quality models were built upon reduced models using classic model building. While inclusion of an appropriate variable in the model improved the quality of the model, inclusion of an inappropriate variable, i.e., model mis-specification, adversely affected it. Higher quality models, however, occasionally decreased the number of activated voxels, whereas lower quality or inappropriate models occasionally increased the number of activated voxels, indicating that the conventional approach to fMRI data analysis may yield sub-optimal or incorrect results. We propose that model quality maps become part of a broader package of maps for quality assessment in fMRI, facilitating validation, optimization, and standardization of fMRI result across studies and centers. Hum. Brain Mapping 20:227-238, 2003.