Online cognitive assessment in the era of COVID-19: Examining the validity of the MEZURE.

Developed more than 2 decades ago, the MEZURE (Assessment Technologies, 1995-2020; https://www.mezure.com/) has received increased attention as a result of the COVID-19 pandemic. It is the first individualized test of cognitive ability created to use an online (local or remote) assessment modality. The MEZURE claims to be aligned both with the extended Gf-Gc theory and the Cattell-Horn-Carroll model of abilities. Whereas the test publisher claims it used exploratory factor analysis to investigate the instrument's factor structure, only the subtest factor loadings on the Gf-Gc factors were furnished. No other structural validity information was provided, suggesting that users of the instrument should interpret the scores produced by the MEZURE with caution. Accordingly, the present study used exploratory and confirmatory factor analysis to more fully investigate the structural validity of the MEZURE. The results revealed that the MEZURE contains a combined perceptual reasoning (i.e., [Gf/Gv]/working memory [Gwm]) group factor, a verbal ability group factor, and a relatively weak general factor that is dominated by perceptual reasoning. The finding of a paltry general factor that is weakly loaded by verbal subtests is inconsistent with the broader research on traditional cognitive ability assessment and could be related to the online administration format of the test. Future research is required to better understand this finding. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Bootstrap Exploratory Graph Analysis of the WISC-V with a Clinical Sample.

One important aspect of construct validity is structural validity. Structural validity refers to the degree to which scores of a psychological test are a reflection of the dimensionality of the construct being measured. A factor analysis, which assumes that unobserved latent variables are responsible for the covariation among observed test scores, has traditionally been employed to provide structural validity evidence. Factor analytic studies have variously suggested either four or five dimensions for the WISC-V and it is unlikely that any new factor analytic study will resolve this dimensional dilemma. Unlike a factor analysis, an exploratory graph analysis (EGA) does not assume a common latent cause of covariances between test scores. Rather, an EGA identifies dimensions by locating strongly connected sets of scores that form coherent sub-networks within the overall network. Accordingly, the present study employed a bootstrap EGA technique to investigate the structure of the 10 WISC-V primary subtests using a large clinical sample (N = 7149) with a mean age of 10.7 years and a standard deviation of 2.8 years. The resulting structure was composed of four sub-networks that paralleled the first-order factor structure reported in many studies where the fluid reasoning and visual-spatial dimensions merged into a single dimension. These results suggest that discrepant construct and scoring structures exist for the WISC-V that potentially raise serious concerns about the test interpretations of psychologists who employ the test structure preferred by the publisher.

Long-term stability of Wechsler Intelligence Scale for Children-fifth edition scores in a clinical sample.

This study investigated the stability of Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V) scores for 225 children and adolescents from an outpatient neuropsychological clinic across, on average, a 2.6 year test-retest interval. WISC-V mean scores were relatively constant but subtest stability score coefficients were all below 0.80 (M = 0.66) and only the Verbal Comprehension Index (VCI), Visual Spatial Index (VSI), and omnibus Full Scale IQ (FSIQ) stability coefficients exceeded 0.80. Neither intraindividual subtest difference scores nor intraindividual composite difference scores were stable across time (M = 0.26 and 0.36, respectively). Rare and unusual subtest and composite score differences as well as subtest and index scatter at initial testing were unlikely to be repeated at retest (kappa = 0.03 to 0.49). It was concluded that VCI, VSI, and FSIQ scores might be sufficiently stable to support normative comparisons but that none of the intraindividual (i.e. idiographic, ipsative, or person-relative) measures were stable enough for confident clinical decision making.

Construct validity of the BASC-3 teacher rating scales: Independent hierarchical exploratory factor analyses with the normative sample.

The Behavior Assessment System for Children-Third Edition (BASC-3) is the most recent edition and the Teacher Rating Scales (TRS) was reported to be the most frequently used test in school psychology practice. Despite its popularity, there is a lack of independent empirical research regarding psychometric properties. The BASC-3 Manual, while quite detailed in many respects, lacks important details in reporting TRS item- and scale-level factor analyses limiting confidence in construct validity based on internal structure. The present study examined the latent factor structure of the BASC-3 TRS Preschool, Child, and Adolescent Clinical and Adaptive scales using best practices in exploratory factor analysis (EFA). EFA was conducted with the Clinical and Adaptive scales jointly, and with the Clinical scales separately, to aid interpretive clarity. Results indicated theoretically consistent alignment of the BASC-3 TRS Clinical scales to their specified factors (Externalizing, Internalizing, and School Problems) and an additional factor (Social Disengagement) was identified, suggesting a possible new latent construct for a composite scale score containing the Withdrawal and Atypicality scales. Variance partitioning applied to second-order EFA and model-based validity statistics, however, indicated that the composite scales (Externalizing, Internalizing, School Problems, and Social Disengagement) appear to lack sufficient unique variance for confident clinical interpretation in isolation. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Monte Carlo Modeling of Contemporary Intelligence Test (IQ) Factor Structure: Implications for IQ Assessment, Interpretation, and Theory.

Researchers continue to debate the constructs measured by commercial ability tests. Factor analytic investigations of these measures have been used to develop and refine widely adopted psychometric theories of intelligence particularly the Cattell-Horn-Carroll (CHC) model. Even so, this linkage may be problematic as many of these investigations examine a particular instrument in isolation and CHC model specification across tests and research teams has not been consistent. To address these concerns, the present study used Monte Carlo resampling to investigate the latent structure of four of the most widely used intelligence tests for children and adolescents. The results located the approximate existence of the publisher posited CHC theoretical group factors in the Differential Abilities Scales-Second edition and the Kaufman Assessment Battery for Children-Second edition but not in the Wechsler Intelligence Scale for Children-Fifth edition or the Woodcock-Johnson IV Tests of Cognitive Abilities. Instead, the results supported alternative conceptualizations from independent factor analytic research. Additionally, whereas a bifactor model produced superior fit indices in two instruments (Wechsler Intelligence Scale for Children-Fifth edition and Woodcock-Johnson IV Tests of Cognitive Abilities), a higher order structure was found to be superior in the Kaufman Assessment Battery for Children-Second edition and the Differential Abilities Scales-Second edition. Regardless of the model employed, the general factor captured a significant portion of each instrument's variance. Implications for IQ test assessment, interpretation, and theory are discussed.

Conducting Psychoeducational Assessments During the COVID-19 Crisis: the Danger of Good Intentions.

Decision-makers in school psychology are presently engaged in the process of determining how to, if possible, move forward with conducting mandated psychoeducational evaluations of students in schools during the pandemic. Whereas prominent organizations within the profession (e.g., American Psychological Association, National Association of School Psychologists) have issued guidance and encouraged practitioners to delay testing, it is not clear whether that is a viable option in every jurisdiction. Accordingly, professionals are now considering the potential use of telehealth platforms to conduct assessments, in some form, as we move forward and deal with this crisis. The goal of this brief commentary is to raise some provisional limitations associated with the use of telehealth to conduct psychological assessments that we believe will have to be considered as use of these platforms is debated. Recommendations for professional practice are also provided.

Construct Validity of the WISC-V in Clinical Cases: Exploratory and Confirmatory Factor Analyses of the 10 Primary Subtests.

Independent exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) research with the Wechsler Intelligence Scale for Children-Fifth Edition (WISC-V) standardization sample has failed to provide support for the five group factors proposed by the publisher, but there have been no independent examinations of the WISC-V structure among clinical samples. The present study examined the latent structure of the 10 WISC-V primary subtests with a large (N = 2,512), bifurcated clinical sample (EFA, n = 1,256; CFA, n = 1,256). EFA did not support five factors as there were no salient subtest factor pattern coefficients on the fifth extracted factor. EFA indicated a four-factor model resembling the WISC-IV with a dominant general factor. A bifactor model with four group factors was supported by CFA as suggested by EFA. Variance estimates from both EFA and CFA found that the general intelligence factor dominated subtest variance and omega-hierarchical coefficients supported interpretation of the general intelligence factor. In both EFA and CFA, group factors explained small portions of common variance and produced low omega-hierarchical subscale coefficients, indicating that the group factors were of poor interpretive value.

Construct validity of the Wechsler Intelligence Scale For Children - Fifth UK Edition: Exploratory and confirmatory factor analyses of the 16 primary and secondary subtests.

BACKGROUND: There is inadequate information regarding the factor structure of the Wechsler Intelligence Scale for Children - Fifth UK Edition (WISC-VUK ; Wechsler, 2016a, Wechsler Intelligence Scale for Children-Fifth UK Edition, Harcourt Assessment, London, UK) to guide interpretation. AIMS AND METHODS: The WISC-VUK was examined using complementary exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) for all models proposed by Wechsler (2016b, Wechsler Intelligence Scale for Children-Fifth UK Edition: Administration and scoring manual, Harcourt Assessment, London, UK) as well as rival bifactor models. SAMPLE: The WISC-VUK standardization sample (N = 415) correlation matrix was used in analyses due to denial of standardization sample raw data. RESULTS: EFA did not support a theoretically posited fifth factor because only one subtest (Matrix Reasoning) had a salient pattern coefficient on the fifth factor. A model with four group factors and a general intelligence factor resembling the Wechsler Intelligence Scale for Children - Fourth Edition (WISC-IV; Wechsler, 2003, Wechsler Intelligence Scale for Children-Fourth Edition, Psychological Corporation, San Antonio, TX, USA) was supported by both EFA and CFA. General intelligence (g) was the dominant source of subtest variance and large omega-hierarchical coefficients supported interpretation of the Full Scale IQ (FSIQ) score. In contrast, the four group factors accounted for small portions of subtest variance and low omega-hierarchical subscale coefficients indicated that the four-factor index scores were of questionable interpretive value independent of g. Present results replicated independent assessments of the Canadian, Spanish, French, and US versions of the WISC-V (Canivez, Watkins, & Dombrowski, 2016, Psychological Assessment, 28, 975; 2017, Psychological Assessment, 29, 458; Fennollar-Cortés & Watkins, 2018, International Journal of School & Educational Psychology; Lecerf & Canivez, 2018, Psychological Assessment; Watkins, Dombrowski, & Canivez, 2018, International Journal of School and Educational Psychology). CONCLUSION: Primary interpretation of the WISC-VUK should be of the FSIQ as an estimate of general intelligence.

Cognitive profile analysis in school psychology: History, issues, and continued concerns.

Intelligence testing remains a fixture in school psychology training and practice. Despite their popularity, the use of IQ tests is not without controversy and researchers have long debated how these measures should be interpreted with children and adolescents. A controversial aspect of this debate relates to the utility of cognitive profile analysis, a class of interpretive methods that encourage practitioners to make diagnostic decisions and/or treatment recommendations based on the strengths and weaknesses observed in ability score profiles. Whereas numerous empirical studies and reviews have challenged long-standing assumptions about the utility of these methods, much of this literature is nearly two decades old and new profile analysis methods (e.g., XBA, PSW) have been proffered. To help update the field's understanding of these issues, the present review traces the historical development of cognitive profile analysis and (re)introduces readers to a body of research evidence suggesting new and continued concerns with the use of these methods in school psychology practice. It is believed that this review will serve as a useful resource to practitioners and trainers for understanding and promoting a countering view on these matters.

Revisiting Carroll's survey of factor-analytic studies: Implications for the clinical assessment of intelligence.

[Correction Notice: An Erratum for this article was reported in Vol 30(8) of Psychological Assessment (see record 2018-37729-003). In the article "Revisiting Carroll's Survey of Factor-Analytic Studies: Implications for the Clinical Assessment of Intelligence," by Nicholas F. Benson, A. Alexander Beaujean, Ryan J. McGill, and Stefan C. Dombrowski (Psychological Assessment, Advance online publication, May 24, 2018, http://dx.doi.org/10.1037/pas0000556), the majority of values in the ωH and ωHS columns of Table 4 were incorrect and have been amended. These revisions required text in the fourth paragraph of the Results section to be changed from "Moreover, the ωHS value for Gs is relatively high and very close to the and ωH values for g" to "Moreover, the ωHS values for Gs and Gv are relatively high, exceeding the ω and ωH values for g." All versions of this article have been corrected.] John Carroll's three-stratum theory (and the decades of research behind its development) is foundational to the contemporary practice of intellectual assessment. The present study addresses some limitations of Carroll's work: specification, reproducibility with more modern methods, and interpretive relevance. We reanalyzed select data sets from Carroll's survey of factor analytic studies using confirmatory factor analysis as well as modern indices of interpretive relevance. For the majority of data sets, we found that Carroll likely extracted too many factors representing Stratum II abilities. Moreover, almost all factors representing Stratum II abilities had little-to-no interpretive relevance above and beyond that of general intelligence. We conclude by discussing the implications of this research with respect to the interpretive relevance and clinical utility of scores reflecting cognitive abilities at all strata of the three-stratum theory and offer some directions for future research. (PsycINFO Database Record

What Does the WRAML2 Core Battery Measure? Utilizing Exploratory and Confirmatory Techniques to Disclose Higher Order Structure.

The present study examined the factor structure of the Wide Range Assessment of Memory and Learning-Second Edition (WRAML2) core battery with participants from the normative sample aged 9 to 90 years ( n = 880) using higher order exploratory and confirmatory factor analytic techniques that were not reported in the in the WRAML2 Administration and Technical Manual. Exploratory factor analysis results suggested only one factor, whereas confirmatory factor analysis results favored the three factors posited by the test authors. Although model fit statistics were equivalent for the oblique, indirect hierarchical, and direct hierarchical measurement models, it was determined that the bifactor model best disclosed the influence of latent dimensions on WRAML2 manifest variables. In the three-factor bifactor model, the general factor accounted for 31% of the total variance and 69% of the common variance, whereas the three first-order factors combined accounted for 41% of the total variance and 31% of the common variance. Latent factor reliability coefficients (as estimated by ωh) indicated that only the general factor was measured with enough precision to warrant confident clinical interpretation. Implications for clinical interpretation of WRAML2 scores and the procedures utilized in the development of related measures are discussed.

Hierarchical exploratory factor analyses of the Woodcock-Johnson IV Full Test Battery: Implications for CHC application in school psychology.

The Woodcock-Johnson (fourth edition; WJ IV; Schrank, McGrew, & Mather, 2014a) was recently redeveloped and retains its linkage to Cattell-Horn-Carroll theory (CHC). Independent reviews (e.g., Canivez, 2017) and investigations (Dombrowski, McGill, & Canivez, 2017) of the structure of the WJ IV full test battery and WJ IV Cognitive have suggested the need for additional factor analytic exploration. Accordingly, the present study used principal axis factoring (PAF) followed by the Schmid and Leiman (SL; Schmid & Leiman, 1957) procedure with the 2 school-aged correlation matrices from the normative sample to determine the degree to which the WJ IV total battery structure could be replicated. Although 7 factors emerged across the 9 to 19 age range, the pattern of subtests loadings did not fully cohere with the structure presented in the Technical Manual, most notably for the academic fluency subtests. Also, the Fluid Reasoning (Gf) and Quantitative Reasoning (Gq) subtests coalesced to form a combined factor rather than 2 separate factors and the Long Term Retrieval (Gltr) subtests aligned with a variety of different factors. The results of this study indicated that the general intelligence factor variance far exceeded the variance attributed to the lower-order CHC factors. The combination of subtest migration and nominal total/common variance of the CHC lower-order factors suggests caution when interpreting the myriad CHC-related indices when making high stakes decisions. Implications for clinical practice are discussed. (PsycINFO Database Record

A Misuse of IQ Scores: Using the Dual Discrepancy/Consistency Model for Identifying Specific Learning Disabilities.

The purpose of this article is to describe the origins of patterns of strengths and weaknesses (PSW) methods for identifying specific learning disabilities (SLD) and to provide a comprehensive review of the assumptions and evidence supporting the most commonly-used PSW method in the United States: Dual Discrepancy/Consistency (DD/C). Given their use in determining whether students have access to special education and related services, it is important that any method used to identify SLD have supporting evidence. A review of the DD/C evidence indicates it cannot currently be classified as an evidence-based method for identifying individuals with a SLD. We show that the DD/C method is unsound for three major reasons: (a) it requires test scores have properties that they fundamentally lack, (b) lack of experimental utility evidence supporting its use, and (c) evidence supporting the inability of the method to identify SLD accurately.

Exploratory and hierarchical factor analysis of the WJ-IV Cognitive at school age.

Exploratory and confirmatory factor analytic studies were not reported in the Technical Manual for the Woodcock-Johnson, 4th ed. Cognitive (WJ IV Cognitive; Schrank, McGrew, & Mather, 2014b) Instead, the internal structure of the WJ IV Cognitive was extrapolated from analyses based on the full WJ IV test battery (Schrank, McGrew, & Mather, 2014b). Even if the veracity of extrapolating from the WJ IV full battery were accepted, there were shortcomings in the choices of analyses used and only limited information regarding those analyses was presented in the WJ IV Technical Manual (McGrew, Laforte, & Shrank, 2014). The present study examined the structure of the WJ IV Cognitive using exploratory factor analysis procedures (principal axis factoring with oblique [promax] rotation followed by application of the Schmid-Leiman, 1957, procedure) applied to standardization sample correlation matrices for 2 school age groups (ages 9-13; 14-19). Four factors emerged for both the 9-13 and 14-19 age groups in contrast to the publisher's proposed 7 factors. Results of these analyses indicated a robust manifestation of general intelligence (g) that exceeded the variance attributed to the lower-order factors. Model-based reliability estimates supported interpretation of the higher-order factor (i.e., g). Additional analyses were conducted by forcing extraction of the 7 theoretically posited factors; however, the resulting solution was only partially aligned (i.e., Gs, Gwm) with the theoretical structure promoted in the Technical Manual and suggested the preeminence of the higher-order factor. Results challenge the hypothesized structure of the WJ IV Cognitive and raise concerns about its alignment with Cattell-Horn-Carroll theory. (PsycINFO Database Record

Exploratory Higher Order Analysis of the Luria Interpretive Model on the Kaufman Assessment Battery for Children-Second Edition (KABC-II) School-Age Battery.

Higher order factor structure of the Luria interpretive scheme on the Kaufman Assessment Battery for Children-Second Edition (KABC-II) for the 7- to 12-year and the 13- to 18-year age groups in the KABC-II normative sample ( N = 2,025) is reported. Using exploratory factor analysis, multiple factor extraction criteria, and hierarchical exploratory factor analysis not included in the KABC-II manual, two-, three-, and four-factor extractions were analyzed to assess the hierarchical factor structure by sequentially partitioning variance appropriately to higher order and lower order dimensions as recommended by Carroll. No evidence for a four-factor solution was found. Results showed that the largest portions of total and common variance were accounted for by the second-order general factor and that interpretation should focus primarily, if not exclusively, at that level of measurement.

Factor structure of the Differential Ability Scales-Second Edition: Exploratory and hierarchical factor analyses with the core subtests.

The present study examined the factor structure of the Differential Ability Scales-Second Edition (DAS-II; Elliott, 2007a) standardization sample using exploratory factor analyses, multiple factor extraction criteria, and hierarchical exploratory factor analyses (Schmid & Leiman, 1957) not included in the DAS-II Introductory and Technical Handbook (Elliott, 2007b). Exploratory factor analyses with multiple factor extraction criteria and hierarchical analyses with the Schmid and Leiman (1957) procedure were conducted with the 3 DAS-II standardization samples (Lower Early Years [Ages 2:6-3:5], Upper Early Years [Ages 3:6-6:11], School-Age [Ages 7:0-17:11]). All factor extraction criteria suggested 1 factor despite the author and publisher recommended and promoted 2 (Ages 2:6-3:5) or 3 (Ages 3:6-6:11, Ages 7:0-17:11) factors. Results indicated that most DAS-II subtests were properly associated with the theoretically proposed first-order factors. Hierarchical exploratory analyses with the Schmid and Leiman procedure, however, found that the hierarchical g factor accounted for large portions of total and common variance, while the 2 or 3 first-order factors accounted for small portions of total and common variance. It was concluded that the DAS-II provides strong measurement of general intelligence but clinical interpretation should be primarily at that level. (PsycINFO Database Record

Interpretation of KABC-II scores: An evaluation of the incremental validity of Cattell-Horn-Carroll (CHC) factor scores in predicting achievement.

This study is an examination of the incremental validity of Cattell-Horn-Carroll (CHC) factor scores from the Kaufman Assessment Battery for Children-second edition (KABC-II) for predicting scores on the Kaufman Test of Educational Achievement-second edition (KTEA-II). The participants were children and adolescents, ages 7-18, (N = 2,025) drawn from the KABC-II standardization sample. The sample was nationally stratified and proportional to U.S. census estimates for sex, ethnicity, geographic region, and parent education level. Hierarchical multiple regression analyses were used to assess for factor-level effects after controlling for the variance accounted for by the full scale Fluid-Crystallized Index (FCI) score. The results were interpreted using the R2/ΔR2 statistic as effect size indices. Consistent with similar incremental validity studies, the FCI accounted for statistically and clinically significant portions of KTEA-II score variance, with R2 values ranging from .30 to .65. KABC-II CHC factor scores collectively provided statistically significant incremental variance beyond the FCI in all of the regression models, although the effect size estimates were consistently negligible to small (Average ÄRCHC2 = .03). Individually, the KABC-II factor scores accounted for mostly small portions of achievement variance across the prediction models, with none of the individual CHC factors accounting for clinically significant incremental prediction beyond the FCI. Additionally, most of the unique first-order predictive variance was captured by the Crystallized Ability factor alone. The potential clinical and theoretical implications of these results are discussed.

Incremental validity of the WJ III COG: Limited predictive effects beyond the GIA-E.

This study is an examination of the incremental validity of Cattell-Horn-Carroll (CHC) broad clusters from the Woodcock-Johnson III Tests of Cognitive Abilities (WJ III COG) for predicting scores on the Woodcock-Johnson III Tests of Achievement (WJ III ACH). The participants were children and adolescents, ages 6-18 (n = 4,722), drawn from the WJ III standardization sample. The sample was nationally stratified and proportional to U.S. census estimates for race/ethnicity, parent education level, and geographic region. Hierarchical multiple regression analyses were used to assess for cluster-level effects after controlling for the variance accounted for by the General Intellectual Ability-Extended (GIA-E) composite score. The results were interpreted using the R²/ΔR² statistic as the effect size indicator. Consistent with previous studies, the GIA-E accounted for statistically and clinically significant portions of WJ III ACH cluster score variance, with R2 values ranging from .29 to .56. WJ III COG CHC cluster scores collectively provided statistically significant incremental variance beyond the GIA-E in all of the regression models, although the effect sizes were consistently negligible to small (Average ΔR2(CHC) = .06), with significant effects observed only in the Oral Expression model (ΔR²(CHC) = .23). Individually, the WJ III COG cluster scores accounted for mostly small portions of achievement variance across the prediction models, with a large effect found for the Comprehension-Knowledge cluster in the Oral Expression model (ΔR²(Gc) = .23). The potential clinical and theoretical implications of these results are discussed.