Computerized adaptive test strategies for the matrix reasoning subtest of the Wechsler Adult Intelligence Scale, 4th Edition (WAIS-IV).

OBJECTIVE: Most neuropsychological tests were developed without the benefit of modern psychometric theory. We used item response theory (IRT) methods to determine whether a widely used test - the 26-item Matrix Reasoning subtest of the WAIS-IV - might be used more efficiently if it were administered using computerized adaptive testing (CAT). METHOD: Data on the Matrix Reasoning subtest from 2197 participants enrolled in the National Neuropsychology Network (NNN) were analyzed using a two-parameter logistic (2PL) IRT model. Simulated CAT results were generated to examine optimal short forms using fixed-length CATs of 3, 6, and 12 items and scores were compared to the original full subtest score. CAT models further explored how many items were needed to achieve a selected precision of measurement (standard error ≤ .40). RESULTS: The fixed-length CATs of 3, 6, and 12 items correlated well with full-length test results (with r = .90, .97 and .99, respectively). To achieve a standard error of .40 (approximate reliability = .84) only 3-7 items had to be administered for a large percentage of individuals. CONCLUSIONS: This proof-of-concept investigation suggests that the widely used Matrix Reasoning subtest of the WAIS-IV might be shortened by more than 70% in most examinees while maintaining acceptable measurement precision. If similar savings could be realized in other tests, the accessibility of neuropsychological assessment might be markedly enhanced, and more efficient time use could lead to broader subdomain assessment.

Estimating individual health-related quality of life changes in low back pain patients.

BACKGROUND: There is a need to evaluate different options for estimating individual change in health-related quality of life for patients with low back pain. METHODS: Secondary analysis of data collected at baseline and 6 weeks later in a randomized trial of 749 adults with low back pain receiving usual medical care (UMC) or UMC plus chiropractic care at a small hospital at a military training site or two large military medical centers. The mean age was 31; 76% were male and 67% were White. The study participants completed the Patient-Reported Outcomes Measurement Information System (PROMIS®)-29 v 1.0 physical function, pain interference, pain intensity, fatigue, sleep disturbance, depression, anxiety, satisfaction with participation in social roles, physical summary, and mental health summary scores (T-scored with mean = 50 and standard deviation (SD) = 10 in the U.S. general population). RESULTS: Reliability estimates at the baseline ranged from 0.700 to 0.969. Six-week test-retest intraclass correlation estimates were substantially lower than these estimates: the median test-retest intraclass correlation for the two-way mixed-effects model was 0. 532. Restricting the test-retest reliability estimates to the subset who reported they were about the same as at baseline on a retrospective rating of change item increased the median test-retest reliability to 0.686. The amount of individual change that was statistically significant varied by how reliability was estimated, and which SD was used. The smallest change needed was found when internal consistency reliability and the SD at baseline were used. When these values were used, the amount of change needed to be statistically significant (p < .05) at the individual level ranged from 3.33 (mental health summary scale) to 12.30 (pain intensity item) T-score points. CONCLUSIONS: We recommend that in research studies estimates of the magnitude of individual change needed for statistical significance be provided for multiple reliability and standard deviation estimates. Whenever possible, patients should be classified based on whether they 1) improved significantly and perceived they got better, 2) improved significantly but did not perceive they were better, 3) did not improve significantly but felt they got better, or 4) did not improve significantly or report getting better.

Clinical Validity of the PROMIS Healthcare Engagement 8-Item Short Form.

BACKGROUND: Healthcare engagement is a key measurement target for value-based healthcare, but a reliable and valid patient-reported measure has not yet been widely adopted. OBJECTIVE: To assess the validity of a newly developed patient-reported measure of healthcare engagement, the 8-item PROMIS Healthcare Engagement (PHE-8a). DESIGN: Prospective cohort study of the association between healthcare engagement and quality of care over 1 year. We fit mixed effects models of quality indicators as a function of engagement scores, adjusting for age, race/ethnicity, rural residence, and risk scores. PARTICIPANTS: National stratified random sample of 9552 Veterans receiving Veterans Health Administration care for chronic conditions (hypertension, diabetes) or mental health conditions (depression, post-traumatic stress disorder). MAIN MEASURES: Patient experience: Consumer Assessment of Health Plans and Systems communication and self-management support composites; no-show rates for primary care and mental health appointments; use of patient portal My HealtheVet; and Healthcare Effectiveness Data and Information Set electronic quality measures: HbA1c poor control, controlling high blood pressure, and hyperlipidemia therapy adherence. KEY RESULTS: Higher engagement scores were associated with better healthcare quality across all outcomes, with each 5-point increase (1/2 standard deviation) in engagement scores associated with statistically significant and clinically meaningful gains in quality. Across the continuum of low to high engagement scores, we observed a concomitant reduction in primary care no-show rates of 37% and 24% for mental health clinics; an increased likelihood of My HealtheVet use of 15.4%; and a decreased likelihood of poor diabetes control of 44%. CONCLUSIONS: The PHE-8a is a brief, reliable, and valid patient-reported measure of healthcare engagement. These results confirm previously untested hypotheses that patient engagement can promote healthcare quality.

Response Category Functioning on the Health Care Engagement Measure Using the Nominal Response Model.

As part of a scale development project, we fit a nominal response item response theory model to responses to the Health Care Engagement Measure (HEM). When using the original 5-point response format, categories were not ordered as intended for six of the 23 items. For the remaining, the category boundary discrimination between Categories 0 (not at all true) and 1 (a little bit true) was only weakly discriminating, suggesting uninformative categories. When the lowest two categories were collapsed, psychometric properties improved greatly. Category boundary discriminations within items, however, varied significantly. Specifically, higher response category distinctions, such as responding 3 (very true) versus 2 (mostly true) were considerably more discriminating than lower response category distinctions. Implications for HEM scoring and for improving measurement precision at lower levels of the construct are presented as is the unique role of the nominal response model in category analysis.

Construct identification in the neuropsychological battery: What are we measuring?

OBJECTIVE: Major obstacles to data harmonization in neuropsychology include lack of consensus about what constructs and tests are most important and invariant across healthy and clinical populations. This study addressed these challenges using data from the National Neuropsychology Network (NNN). METHOD: Data were obtained from 5,000 NNN participants and Pearson standardization samples. Analyses included variables from four instruments: Wechsler Adult Intelligence Scale, 4th Edition (WAIS-IV); Wechsler Memory Scale, 4th Edition (WMS-IV); California Verbal Learning Test, 3rd Edition (CVLT3); and Delis-Kaplan Executive Function System (D-KEFS). We used confirmatory factor analysis to evaluate models suggested by prior work and examined fit statistics and measurement invariance across samples. We examined relations of factor scores to demographic and clinical characteristics. RESULTS: For each instrument, we identified four first-order and one second-order factor. Optimal models in patients generally paralleled the best-fitting models in the standardization samples, including task-specific factors. Analysis of the NNN data prompted specification of a Recognition-Familiarity factor on the WMS-IV and an Inhibition-Switching factor on the D-KEFS. Analyses showed strong to strict factorial invariance across samples with expected differences in factor means and variances. The Recognition-Familiarity factor correlated with age more strongly in NNN than in the standardization sample. CONCLUSIONS: Factor models derived from healthy groups generally fit well in patients. NNN data helped identify novel Recognition-Familiarity and Inhibition-Switching factors that were also invariant across samples and may be clinically useful. The findings support efforts to identify evidence-based and optimally efficient measurements of neuropsychological constructs that are valid across groups. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Rationale and Design of the National Neuropsychology Network.

OBJECTIVE: The National Neuropsychology Network (NNN) is a multicenter clinical research initiative funded by the National Institute of Mental Health (NIMH; R01 MH118514) to facilitate neuropsychology's transition to contemporary psychometric assessment methods with resultant improvement in test validation and assessment efficiency. METHOD: The NNN includes four clinical research sites (Emory University; Medical College of Wisconsin; University of California, Los Angeles (UCLA); University of Florida) and Pearson Clinical Assessment. Pearson Q-interactive (Q-i) is used for data capture for Pearson published tests; web-based data capture tools programmed by UCLA, which serves as the Coordinating Center, are employed for remaining measures. RESULTS: NNN is acquiring item-level data from 500-10,000 patients across 47 widely used Neuropsychology (NP) tests and sharing these data via the NIMH Data Archive. Modern psychometric methods (e.g., item response theory) will specify the constructs measured by different tests and determine their positive/negative predictive power regarding diagnostic outcomes and relationships to other clinical, historical, and demographic factors. The Structured History Protocol for NP (SHiP-NP) helps standardize acquisition of relevant history and self-report data. CONCLUSIONS: NNN is a proof-of-principle collaboration: by addressing logistical challenges, NNN aims to engage other clinics to create a national and ultimately an international network. The mature NNN will provide mechanisms for data aggregation enabling shared analysis and collaborative research. NNN promises ultimately to enable robust diagnostic inferences about neuropsychological test patterns and to promote the validation of novel adaptive assessment strategies that will be more efficient, more precise, and more sensitive to clinical contexts and individual/cultural differences.

Friendship Network Satisfaction: A multifaceted construct scored as a unidimensional scale.

Although satisfying friendships are crucial for well-being throughout adulthood, measures of friendship satisfaction have been limited by: (1) item content relevant to children only, (2) a focus on single relationships rather than the friendship network, and (3) disagreement about the number of dimensions necessary to capture the construct. To overcome these limitations, we assembled an item pool from a number of existing measures, created additional items drawn from research on friendships, and then examined the structure and psychometric properties of those items in two online surveys of over 2000 respondents each. Factor analyses consistently identified two correlated factors-closeness and socializing-but bi-factor modeling revealed that scores on both subscales load strongly on a general factor, suggesting that the multifaceted content can be scored efficiently as a unidimensional composite. Analyses using item response theory (IRT) supported the creation of a reliable 14-item instrument that demonstrated adequate convergent and predictive validity. Thus, the Friendship Network Satisfaction (FNS) Scale is a psychometrically sound tool to advance research on friendships across the lifespan.

Matching IRT Models to Patient-Reported Outcomes Constructs: The Graded Response and Log-Logistic Models for Scaling Depression.

Item response theory (IRT) model applications extend well beyond cognitive ability testing, and various patient-reported outcomes (PRO) measures are among the more prominent examples. PRO (and like) constructs differ from cognitive ability constructs in many ways, and these differences have model fitting implications. With a few notable exceptions, however, most IRT applications to PRO constructs rely on traditional IRT models, such as the graded response model. We review some notable differences between cognitive and PRO constructs and how these differences can present challenges for traditional IRT model applications. We then apply two models (the traditional graded response model and an alternative log-logistic model) to depression measure data drawn from the Patient-Reported Outcomes Measurement Information System project. We do not claim that one model is "a better fit" or more "valid" than the other; rather, we show that the log-logistic model may be more consistent with the construct of depression as a unipolar phenomenon. Clearly, the graded response and log-logistic models can lead to different conclusions about the psychometrics of an instrument and the scaling of individual differences. We underscore, too, that, in general, explorations of which model may be more appropriate cannot be decided only by fit index comparisons; these decisions may require the integration of psychometrics with theory and research findings on the construct of interest.

Using Item Response Theory to Identify Responders to Treatment: Examples with the Patient-Reported Outcomes Measurement Information System (PROMIS®) Physical Function Scale and Emotional Distress Composite.

The reliable change index has been used to evaluate the significance of individual change in health-related quality of life. We estimate reliable change for two measures (physical function and emotional distress) in the Patient-Reported Outcomes Measurement Information System (PROMIS®) 29-item health-related quality of life measure (PROMIS-29 v2.1). Using two waves of data collected 3 months apart in a longitudinal observational study of chronic low back pain and chronic neck pain patients receiving chiropractic care, and simulations, we compare estimates of reliable change from classical test theory fixed standard errors with item response theory standard errors from the graded response model. We find that unless true change in the PROMIS physical function and emotional distress scales is substantial, classical test theory estimates of significant individual change are much more optimistic than estimates of change based on item response theory.

Impact of stress resilience and susceptibility on fear learning, anxiety, and alcohol intake.

Post-traumatic stress disorder (PTSD) can develop after exposure to traumatic events and severely impacts the quality of life. PTSD is frequently comorbid with substance use disorders, with alcoholism being particularly common. However, not everyone who experiences trauma develops PTSD and the factors that render individuals susceptible or resilient to the effects of stress are unknown although gender appears to play an important role. Rodent models of stress exposure such as stress-enhanced fear learning (SEFL) recapitulate some aspects of PTSD symptomology, making them an invaluable tool for studying this disorder. This study examined whether exposure to a modified version of the SEFL procedure (4 footshocks instead of the standard 15 over 90 min) would reveal both susceptible and resilient subjects. Following stress exposure, distinct susceptible and resilient groups emerged that differed in fear learning and anxiety-related behavior as well as voluntary alcohol intake. Some aspects of stress susceptibility manifested differently in males compared to females, with susceptibility associated with increased alcohol intake in males and increased baseline anxiety in females.

Psychometric evaluation of a patient-reported item bank for healthcare engagement.

PURPOSE: Healthcare engagement is a core measurement target for efforts to improve healthcare systems. This construct is broadly defined as the extent to which healthcare services represent collaborative partnerships with patients. Previous qualitative work operationalized healthcare engagement as generalized self-efficacy in four related subdomains: self-management, collaborative communication, health information use, and healthcare navigation. Building on this work, our objective was to establish a healthcare engagement instrument that is sufficiently unidimensional to yield a single score. METHOD: We conducted cognitive interviews followed by a nation-wide mail survey of US Veteran Administration (VA) healthcare users. Data were collected on 49 candidate healthcare engagement items, as well as measures of self-efficacy for managing symptoms, provider communication, and perceived access. Items were subjected to exploratory bifactor, statistical learning, and IRT analyses. RESULTS: Cognitive interviews were completed by 56 patients and 9552 VA healthcare users with chronic conditions completed the mail survey. Participants were mostly white and male but with sizable minority participation. Psychometric analyses and content considerations reduced the item pool to 23 items, which demonstrated a strong general factor (OmegaH of .89). IRT analyses revealed a high level of reliability across the trait range and little DIF across groups. Most health information use items were removed during analyses, suggesting a more independent role for this domain. CONCLUSION: We provide quantitative evidence for a relatively unidimensional measure of healthcare engagement. Despite developed with VA healthcare users, the measure is intended for general use. Future work includes short-form development and validation with other patient groups.

Extensions of Multiple-Group Item Response Theory Alignment: Application to Psychiatric Phenotypes in an International Genomics Consortium.

Large-scale studies spanning diverse project sites, populations, languages, and measurements are increasingly important to relate psychological to biological variables. National and international consortia already are collecting and executing mega-analyses on aggregated data from individuals, with different measures on each person. In this research, we show that Asparouhov and Muthén's alignment method can be adapted to align data from disparate item sets and response formats. We argue that with these adaptations, the alignment method is well suited for combining data across multiple sites even when they use different measurement instruments. The approach is illustrated using data from the Whole Genome Sequencing in Psychiatric Disorders consortium and a real-data-based simulation is used to verify accurate parameter recovery. Factor alignment appears to increase precision of measurement and validity of scores with respect to external criteria. The resulting parameter estimates may further inform development of more effective and efficient methods to assess the same constructs in prospectively designed studies.

Ecological validity of a quantitative classification system for mental illness in treatment-seeking adults.

Quantitative models of mental illness, such as the Hierarchical Taxonomy of Psychopathology (HiTOP), aim to optimize clinical assessment, which conventionally follows categorical diagnostic rubrics. The evidence base for these models is robust, but also uniform; available data come mostly from structured diagnostic interviews in nationally representative samples. It remains to be seen whether HiTOP adequately reflects mental illness as evaluated in routine clinical care, where diagnosis is often unsystematic and incomplete, relative to controlled research conditions. To test the generalizability of a quantitative nosology to real-world assessment contexts, we fit the HiTOP model to diagnoses in a large sample (N = 25,002) of treatment-seeking university students who were seen by health professionals in everyday practice. We then examined the criterion validity of model components in relation to clinically relevant outcomes (i.e., suicide attempts, self-injury, and binge drinking). Three related structures fit the data well: a correlated-factor model with internalizing, externalizing, and eating pathology dimensions; a higher-order model that added a general factor of psychopathology that spanned these 3 first-order factors; and a bifactor model that partitioned diagnostic (co)variance across a general factor and 3 orthogonal group factors. The first-order factors had expected patterns of criterion validity, and the general factor was a strong predictor of suicidality and self-injury, paralleling past research. Bifactor model group factors had interpretative problems, however. Across models, categorical diagnoses consistently offered minimal incremental validity relative to the transdiagnostic factors. We conclude that HiTOP is ecologically valid-explaining comorbidity patterns among diagnoses assigned "in the field"-and is poised to enhance clinical assessment and decision-making in routine care. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Neuropsychological tests of the future: How do we get there from here?

OBJECTIVE: This article reviews current approaches to neuropsychological assessment, identifies opportunities for development of new methods using modern psychometric theory and advances in technology, and suggests a transition path that promotes application of novel methods without sacrificing validity. METHODS: Theoretical/state-of-the-art review. CONCLUSIONS: Clinical neuropsychological assessment today does not reflect advances in neuroscience, modern psychometrics, or technology. Major opportunities for improving practice include both psychometric and technological strategies. Modern psychometric approaches including item response theory (IRT) enable linking procedures that can place different measures on common scales; adaptive testing algorithms that can dramatically increase efficiency of assessment; examination of differential item functioning (DIF) to detect measures that behave differently in different groups; and person fit statistics to detect aberrant patterns of responding of high value for performance validity testing. Opportunities to introduce novel technologies include computerized adaptive testing, Web-based assessment, healthcare- and bio-informatics strategies, mobile platforms, wearables, and the 'internet-of-things'. To overcome inertia in current practices, new methods must satisfy requirements for back-compatibility with legacy instrumentation, enabling us to leverage the wealth of validity data already accrued for classic procedures. A path to achieve these goals involves creation of a global network to aggregate item-level data into a shared repository that will enable modern psychometric analyses to refine existing methods, and serve as a platform to evolve novel assessment strategies, which over time can revolutionize neuropsychological assessment practices world-wide.

Development and public release of a computerized adaptive (CAT) version of the Schizotypal Personality Questionnaire.

One of the most widely used measures of psychosis-related symptoms and characteristics is the 74-item Schizotypal Personality Questionnaire (SPQ). Using multidimensional Item Response Theory (bifactor 2-parameter model), we calibrated SPQ items in a sample of 375 youths aged 9-24 years and constructed a fully functional computerized adaptive form of the SPQ on an open-source platform for public use. To assess validity, we used the above parameters to simulate CAT sessions in a separate validation sample (N = 100) using three test-length-based stopping rules: 8 items, 16 items, and 32 items. Those scores were then compared to full-form and SPQ-Brief scores on their abilities to predict psychosis or clinical risk status. Areas under the receiver operating characteristic curves indicated mediocre predictive ability, but did not differ among any of the forms, even when only eight adaptive items were administered. The Youden index for the 16-item adaptive version was higher than that for the 22-item SPQ-Brief. Classification accuracy for the full SPQ was 73% compared to 66% for the both the SPQ-Brief and adaptive versions (average of three stopping rules). The SPQ-CAT shows promise as a much shorter but valid assessment of schizotypy which can save time with minimal loss of information.

Bifactor Modeling of the Positive and Negative Syndrome Scale: Generalized Psychosis Spans Schizoaffective, Bipolar, and Schizophrenia Diagnoses.

Objective: Common genetic variation spans schizophrenia, schizoaffective and bipolar disorders, but historically, these syndromes have been distinguished categorically. A symptom dimension shared across these syndromes, if such a general factor exists, might provide a clearer target for understanding and treating mental illnesses that share core biological bases. Method: We tested the hypothesis that a bifactor model of the Positive and Negative Syndrome Scale (PANSS), containing 1 general factor and 5 specific factors (positive, negative, disorganized, excited, anxiety), explains the cross-diagnostic structure of symptoms better than the traditional 5-factor model, and examined the extent to which a general factor reflects the overall severity of symptoms spanning diagnoses in 5094 total patients with a diagnosis of schizophrenia, schizoaffective, and bipolar disorder. Results: The bifactor model provided superior fit across diagnoses, and was closer to the "true" model, compared to the traditional 5-factor model (Vuong test; P < .001). The general factor included high loadings on 28 of the 30 PANSS items, omitting symptoms associated with the excitement and anxiety/depression domains. The general factor had highest total loadings on symptoms that are often associated with the positive and disorganization syndromes, but there were also substantial loadings on the negative syndrome thus leading to the interpretation of this factor as reflecting generalized psychosis. Conclusions: A bifactor model derived from the PANSS can provide a stronger framework for measuring cross-diagnostic psychopathology than a 5-factor model, and includes a generalized psychosis dimension shared at least across schizophrenia, schizoaffective, and bipolar disorder.

The global therapist competence scale for youth psychosocial treatment: Development and initial validation.

OBJECTIVE: We describe the development and initial psychometric properties of the observer-rated Global Therapist Competence Scale for Youth Psychosocial Treatment (G-COMP) in the context of cognitive-behavioral treatment (CBT) for youth anxiety disorders. METHOD: Independent coders rated 744 sessions from a sample of 68 youth (mean age = 10.56 years) using the G-COMP and the instruments of alliance, involvement, CBT adherence, CBT competence. RESULTS: Inter-rater reliability coefficients, ICC(2,2), were greater than .60 for the 5 G-COMP domain scores. G-COMP scores yielded small to medium correlations with instruments of alliance (rs = .17-.44) and youth involvement in treatment (rs = .08-.53), and medium to large correlations with instruments of CBT competence and adherence (rs = .26-.63). Therapists in the research setting were rated higher compared to newly trained therapists in community clinics. CONCLUSION: Preliminary reliability and validity of the G-COMP are promising, but future research is needed with non-CBT samples.

A 10-minute measure of global cognition: Validation of the Brief Cognitive Assessment Tool for Schizophrenia (B-CATS).

INTRODUCTION: Schizophrenia is marked by a global cognitive impairment that contributes significantly to chronic disability and unemployment. As new treatments are developed for cognition in schizophrenia, clinicians require easily administered instruments to assess cognition. We previously developed a very brief cognitive battery (Bell et al., 2005). The Brief Cognitive Assessment Tool for Schizophrenia (B-CATS) was developed specifically to provide clinicians with a way to assess cognition in their patients with schizophrenia. Here, we report the results of a validity study comparing B-CATS to a larger neurocognitive battery, the Measurement and Treatment Research to Improve Cognition in Schizophrenia (MATRICS) battery. METHODS: Outpatients with schizophrenia (N=91) were administered the B-CATS and the non-overlapping tests of the MATRICS battery at two time points separated by 1month. They were also administered the UCSD Performance-based Skills Assessment-Brief (UPSA-B), a measure of functional capacity. RESULT: The B-CATS has an administration time of approximately 10min. It demonstrates good test-retest reliability and internal consistency. It correlates 0.76 (p<0.01) with the MATRICS battery. The shorter B-CATS and the MATRICS battery correlate with the UPSA-B at 0.50 and 0.58 respectively. CONCLUSION: A 10-minute version of the B-CATS correlates highly with the "gold standard" neurocognitive battery that has an administration time of over 60min. Both measures correlate moderately with a measure of functional capacity. This brief battery was designed to allow clinicians to monitor cognitive change and better inform treatment decisions.

Alternative Approaches to Addressing Non-Normal Distributions in the Application of IRT Models to Personality Measures.

It is generally assumed that the latent trait is normally distributed in the population when estimating logistic item response theory (IRT) model parameters. This assumption requires that the latent trait be fully continuous and the population homogenous (i.e., not a mixture). When this normality assumption is violated, models are misspecified, and item and person parameter estimates are inaccurate. When normality cannot be assumed, it might be appropriate to consider alternative modeling approaches: (a) a zero-inflated mixture, (b) a log-logistic, (c) a Ramsay curve, or (d) a heteroskedastic-skew model. The first 2 models were developed to address modeling problems associated with so-called quasi-continuous or unipolar constructs, which apply only to a subset of the population, or are meaningful at one end of the continuum only. The second 2 models were developed to address non-normal latent trait distributions and violations of homogeneity of error variance, respectively. To introduce these alternative IRT models and illustrate their strengths and weaknesses, we performed real data application comparing results to those from a graded response model. We review both statistical and theoretical challenges in applying these models and choosing among them. Future applications of these and other alternative models (e.g., unfolding, diffusion) are needed to advance understanding about model choice in particular situations.

Finding Pure Sub-Models for Improved Differentiation of Bi-Factor and Second-Order Models.

Several studies have indicated that bi-factor models fit a broad range of psychometric data better than alternative multidimensional models such as second-order models, e.g Rodriguez, Reise and Haviland (2016), Gignac (2016), and Carnivez (2016). Murray and Johnson (2013) and Gignac (2016) argue that this phenomenon is partially due to un-modeled complexities (e.g. un-modeled cross-factor loadings) that induce a bias in standard statistical measures that favors bi-factor models over second-order models. We extend the Murray and Johnson simulation studies to show how the ability to distinguish second-order and bi-factor models diminishes as the amount of un-modeled complexity increases. By using theorems about rank constraints on the covariance matrix to find sub-models of measurement models that have less un-modeled complexity, we are able to reduce the statistical bias in favor of bi-factor models; this allows researchers to reliably distinguish between bi-factor and second-order models.