Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes.

OBJECTIVE: Performance validity test (PVT) research studies commonly utilize a known-groups design, but the criterion grouping approaches within the design vary greatly from one study to another. At the present time, it is unclear as to what degree different criterion grouping approaches might impact PVT classification accuracy statistics. METHOD: To analyze this, the authors used three different criterion grouping approaches to examine how classification accuracy statistics of a PVT (Word Choice Test; WCT) would differ. The three criterion grouping approaches included: (1) failure of 2+ PVTs versus failure of 0 PVTs, (2) failure of 2+ PVTs versus failure of 0-1 PVT, and (3) failure of a stand-alone PVT versus passing of a stand-alone PVT (Test of Memory Malingering). RESULTS: When setting specificity at ≥.90, WCT cutoff scores ranged from 41 to 44 and associated sensitivity values ranged from .64 to .88, depending on the criterion grouping approach that was utilized. CONCLUSIONS: When using a stand-alone PVT to define criterion group status, classification accuracy rates of the WCT were higher than expected, likely due to strong correlations between the reference PVT and the WCT. This held true even when considering evidence that this grouping approach results in higher rates of criterion group misclassification. Conversely, when using criterion grouping approaches that utilized failure of 2+ PVTs, accuracy rates were more consistent with expectations. These findings demonstrate that criterion grouping approaches can impact PVT classification accuracy rates and resultant cutoff scores. Strengths, weaknesses, and practical implications of each of the criterion grouping approaches are discussed.

Malignant Self-Regard in clinical outpatient samples.

While the malignant self-regard construct and a corresponding questionnaire, the Malignant Self-Regard Questionnaire (MSRQ; Huprich, 2011) have been empirically evaluated in several non-clinical samples, it has yet to be evaluated in a clinical population. In this study, 139 outpatients in the Departments of Psychiatry and Behavioral Sciences (n = 83) and Internal Medicine (n = 57) were administered the MSRQ and a number other measures to assess its construct and incremental validity. The MSRQ was internally consistent in both samples, and was positively correlated with four dimensions of levels of personality functioning, DSM-5 personality trait domains, overdependence, detachment, insecure attachment, and general perceptions of physical and mental well-being. Furthermore, the MSRQ incrementally predicated variance in levels of personality functioning, interpersonal dependency, and general perceptions of physical and mental well-being above and beyond DSM-5 traits and (where appropriate) levels of personality functioning. These findings converge with studies of the MSRQ in nonclinical samples and support the clinical utility and validity of the MSRQ for further use.

Rates of Abnormally Low TOPF Word Reading Scores in Individuals Failing Versus Passing Performance Validity Testing.

The present study examined the impact of performance validity test (PVT) failure on the Test of Premorbid Functioning (TOPF) in a sample of 252 neuropsychological patients. Word reading performance differed significantly according to PVT failure status, and number of PVTs failed accounted for 7.4% of the variance in word reading performance, even after controlling for education. Furthermore, individuals failing ≥2 PVTs were twice as likely as individuals passing all PVTs (33% vs. 16%) to have abnormally low obtained word reading scores relative to demographically predicted scores when using a normative base rate of 10% to define abnormality. When compared with standardization study clinical groups, those failing ≥2 PVTs were twice as likely as patients with moderate to severe traumatic brain injury and as likely as patients with Alzheimer's dementia to obtain abnormally low TOPF word reading scores. Findings indicate that TOPF word reading based estimates of premorbid functioning should not be interpreted in individuals invalidating cognitive testing.

Convergent and Discriminant Validity and Utility of the DSM-5 Levels of Personality Functioning Questionnaire (DLOPFQ): Associations with Medical Health Care Provider Ratings and Measures of Physical Health.

The ability to evaluate patients' level of personality functioning in assessing personality disorders has become increasingly important since the DSM-5 Section III hybrid system of personality disorder assessment was released. One measure developed to assess this criterion is the DSM-5 Levels of Personality Functioning Questionnaire (DLOPFQ; Huprich et al., 2017 ), which assesses individuals' self and other representations in four domains-self-direction, identity, empathy, and intimacy-across two contexts-work or school and relationships. A sample of 140 psychiatric and internal medicine outpatients were administered several questionnaires, including the DLOPFQ. Provider ratings also were obtained for level of functioning and DSM-5 pathological personality traits. Several of the DLOPFQ scales were significantly correlated with self-reported and provider-reported measures of DSM-5 trait domains and levels of functioning, along with self-reported measures of effortful control, overall physical and mental health, and well-being. Certain DLOPFQ scales and subscales were associated with provider ratings of likeability and patient contact with the providers. However, relatively modest validity coefficients, as well as poor discriminant validity of domain scales, indicate further research and measure refinement might be needed. It is concluded that, although further research is necessary, the DLOPFQ could be useful for understanding patients' personality pathology in clinical settings.

Introduction of the DSM-5 levels of Personality Functioning Questionnaire.

With the introduction in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) of a hybrid system of personality disorder assessment, the ability to assess patients' traits, as well as their level of personality functioning, has become increasingly important. To assess this criterion, the DSM-5 Levels of Personality Functioning Questionnaire (DLOPFQ) was developed. The DLOPFQ assesses individuals' self-impairments and other impairments in several domains (self-direction, identity, empathy, and intimacy) and across 2 contexts (work/school and relationships). A sample of 140 psychiatric and medical outpatients was administered the DLOPFQ and several other measures to assess its reliability and construct, incremental, and discriminant validity. The internal consistency and convergence with validation measures yielded generally meaningful and expected results. Several DLOPFQ scales and subscales were significantly correlated with measures of DSM-5 trait domains and levels of personality functioning. DLOPFQ scales also correlated with self-reported ratings of overdependence, detachment, healthy dependency, and overall mental health and well-being. The DLOPFQ also predicted interpersonal and general functioning beyond DSM-5 trait domains. These results support the reliability and validity of the DLOPFQ, which appears to be suitable for clinical use and warrants ongoing study. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

A tale of two norms: the impact of normative sample selection criteria on standardized scores in older adults.

OBJECTIVE: Normative sample composition can vary notably as a function of exclusionary criteria in older adult samples given that typical medical conditions of exclusion become more prevalent with age. Older adult norms may consist of typically aging or healthy aging individuals according to the stringency of the exclusion criteria, a difference illustrated when comparing two normative samples for the RBANS: the original standardization sample and the Oklahoma sample. METHODS: The current study examined the clinical impact of norm selection in older adults by analyzing impairment classification rates for 105 older adult behavioral health inpatients. Rates of impaired performance (scaled score ≤4 or its normalized equivalent) were compared across RBANS subtests using standardization sample vs. Oklahoma sample normative data in patients aged 70-79 (n = 57) and 80-89 (n = 48). RESULTS: Across most RBANS subtests, utilization of the original standardization sample vs. Oklahoma sample norms resulted in a greater proportion of individuals classified as impaired. These findings occurred across both age groups and persevered when holding constant the sample normalization process. Overall, use of the standardization sample vs. Oklahoma sample normative data were 1-11 times more likely to indicate impairment, with one of the more notable differences seen on the Coding subtest in the 80-89 sample (65% vs. 8% labeled as impaired). CONCLUSIONS: Sample composition and exclusionary criteria require consideration when selecting an appropriate normative sample against which to compare older adult test performance. Salient differences between typically aging vs. healthy aging older adult norms are discussed.

Comparative prediction of nonepileptic events using MMPI-2 clinical scales, Harris Lingoes subscales, and restructured clinical scales.

The Minnesota Multiphasic Personality Inventory-2 (MMPI-2) is a psychological testing tool used to measure psychological and personality constructs. The MMPI-2 has proven helpful in identifying individuals with nonepileptic events/nonepileptic seizures. However, the MMPI-2 has had some updates that enhanced its original scales. The aim of this article was to test the utility of updated MMPI-2 scales in predicting the likelihood of non-epileptic seizures in individuals admitted to an EEG video monitoring unit. We compared sensitivity, specificity, and likelihood ratios of traditional MMPI-2 Clinical Scales against more homogenous MMPI-2 Harris-Lingoes subscales and the newer Restructured Clinical (RC) scales. Our results showed that the Restructured Scales did not show significant improvement over the original Clinical scales. However, one Harris-Lingoes subscale (HL4 of Clinical Scale 3) did show improved predictive utility over the original Clinical scales as well as over the newer Restructured Clinical scales. Our study suggests that the predictive utility of the MMPI-2 can be improved using already existing scales. This is particularly useful for those practitioners who are not invested in switching over to the newly developed MMPI-2 Restructured Form (MMPI-2 RF).

Wernicke-Korsakoff Syndrome as a Consequence of Delusional Food Refusal: A Case Study.

Wernicke-Korsakoff syndrome is caused by thiamine (vitamin B1) deficiency, typically resulting from malnutrition secondary to chronic alcohol abuse. Less often, other conditions can lead to malnutrition and Wernicke-Korsakoff syndrome. We describe a 35-year-old man who developed Wernicke-Korsakoff syndrome with a typical neurologic and neuropsychological presentation after somatic delusions led him to refuse to eat. Cases like his serve to heighten awareness of the interplay between psychiatric and neurologic conditions, their sometimes atypical pathogenesis, and the value to primary care providers of consulting with psychiatrists, neurologists, and neuropsychologists when managing patients with possible Wernicke-Korsakoff syndrome.

Does True Neurocognitive Dysfunction Contribute to Minnesota Multiphasic Personality Inventory-2nd Edition-Restructured Form Cognitive Validity Scale Scores?

Previous research has demonstrated RBS and FBS-r to identify non-credible reporters of cognitive symptoms, but the extent that these scales might be influenced by true neurocognitive dysfunction has not been previously studied. The present study examined the relationship between these cognitive validity scales and neurocognitive performance across seven domains of cognitive functioning, both before and after controlling for PVT status in 120 individuals referred for neuropsychological evaluations. Variance in RBS, but not FBS-r, was significantly accounted for by neurocognitive test performance across most cognitive domains. After controlling for PVT status, however, relationships between neurocognitive test performance and validity scales were no longer significant for RBS, and remained non-significant for FBS-r. Additionally, PVT failure accounted for a significant proportion of the variance in both RBS and FBS-r. Results support both the convergent and discriminant validity of RBS and FBS-r. As neither scale was impacted by true neurocognitive dysfunction, these findings provide further support for the use of RBS and FBS-r in neuropsychological evaluations.

An examination of the frequency of invalid forgetting on the Test of Memory Malingering.

The Test of Memory Malingering (TOMM) is the most used performance validity test in neuropsychology, but does not measure response consistency, which is central in the measurement of credible presentation. Gunner, Miele, Lynch, and McCaffrey (2012) developed the Albany Consistency Index (ACI) to address this need. The ACI consistency measurement, however, may penalize examinees, resulting in suboptimal accuracy. The Invalid Forgetting Frequency Index (IFFI), created for the present study, utilizes an algorithm to identify and differentiate learning and inconsistent response patterns across TOMM trials. The purpose of this study was to assess the diagnostic accuracy of the ACI and IFFI against a reference test (Malingered Neurocognitive Dysfunction criteria), and to compare both to the standard TOMM indexes. This retrospective case-control study used 59 forensic cases from an outpatient clinic in Southern Kansas. Results indicated that sensitivity, negative predictive value, and overall accuracy of the IFFI were superior to both the TOMM indexes and ACI. Logistic regression odds ratios were similar for TOMM Trial 2, Retention, and IFFI (1.25, 1.24, 1.25, respectively), with the ACI somewhat lower (1.18). The IFFI had the highest rate of group membership predictions (79.7%). Implications and limitations of the present study are discussed.

Frontotemporal dementia mimicking bipolar disorder.

Frontotemporal dementia is a cause of behavioral disturbance that usually appears in individuals between 45 and 65 years of age. The authors present the case of a 65-year-old patient that illustrates how frontotemporal dementia can be misdiagnosed based on a behavioral pattern that suggests the presence of a primary mood disorder. Early accurate diagnosis of frontotemporal dementia and subsequent supportive measures can allow patients and families to make important decisions about business and legal affairs and how to spend remaining leisure time in the most meaningful and enjoyable way possible.

The Coin-in-the-Hand Test and dementia: more evidence for a screening test for neurocognitive symptom exaggeration.

BACKGROUND: The Coin-in-the-Hand Test was developed to help clinicians distinguish patients who are neurocognitively impaired from patients who are exaggerating or feigning memory complaints. Previous findings have shown that participants asked to feign memory problems and patients suspected of malingering performed worse on the test than patients with genuine neurocognitive dysfunction. OBJECTIVE: We reviewed the literature on the Coin-in-the-Hand Test and evaluated test performance by 45 hospitalized patients who had dementia with moderately to severely impaired cognition. METHODS: We analyzed Coin-in-the-Hand Test scores, neuropsychological findings, and other data to determine whether demographic or neurocognitive variables affected Coin-in-the-Hand Test scores. We also calculated base rates of these scores and provided cutoff ranges for clinical use. RESULTS: Coin-in-the-Hand Test scores were independent of neurocognitive functioning, age, education level, and type of dementia. Base rates of scores suggest that a low cutoff can help differentiate between patients with true neurocognitive impairments and those exaggerating or feigning memory complaints. CONCLUSIONS: Both the literature and our findings show the Coin-in-the-Hand Test to have potential as a quick and easy screening tool to detect neurocognitive symptom exaggeration. This test could effectively supplement commonly used neurocognitive screens such as the Mini-Mental State Examination, the Saint Louis University Mental Status Examination, and the Montreal Cognitive Assessment.

Validation of MMPI-2-RF validity scales in criterion group neuropsychological samples.

This study utilized multiple criterion group neuropsychological samples to evaluate the "over-reporting" and "under-reporting" MMPI-2-RF validity scales. The five criterion groups included in this study were (1) litigating traumatic brain injury patients who failed Slick et al. criteria for probable malingering, (2) litigating traumatic brain injury patients who passed Slick et al. criteria, (3) mixed neuropsychological outpatients who passed SVTs and were diagnosed with primary neurological conditions, (4) mixed neuropsychological outpatients who passed SVTs and were diagnosed with primary psychiatric conditions, and (5) epileptic seizure disorder inpatients who were diagnosed via video-EEG. Using the data from these groups, cumulative percentages for all possible T-scores and sensitivity and specificity rates for optimal cutoff scores were determined. When specificity rates were set at 90% across all non-malingering neurological condition groups, sensitivity rates ranged from 48% (FBS-r) to 10% (K-r).

Reliable Digit Span: a systematic review and cross-validation study.

Reliable Digit Span (RDS) is a heavily researched symptom validity test with a recent literature review yielding more than 20 studies ranging in dates from 1994 to 2011. Unfortunately, limitations within some of the research minimize clinical generalizability. This systematic review and cross-validation study was conducted to address these limitations, thus increasing the measure's clinical utility. Sensitivity and specificity rates were calculated for the ≤6 and ≤7 cutoffs when data were globally combined and divided by clinical groups. The cross-validation of specific diagnostic groups was consistent with the data reported in the literature. Overall, caution should be used when utilizing the ≤7 cutoff in all clinical groups and when utilizing the ≤6 cutoff in the following groups: cerebrovascular accident, severe memory disorders, mental retardation, borderline intellectual functioning, and English as a second language. Additional limitations and cautions are provided.

Bifactor and item response theory analyses of interviewer report scales of cognitive impairment in schizophrenia.

A psychometric analysis of 2 interview-based measures of cognitive deficits was conducted: the 21-item Clinical Global Impression of Cognition in Schizophrenia (CGI-CogS; Ventura et al., 2008), and the 20-item Schizophrenia Cognition Rating Scale (SCoRS; Keefe et al., 2006), which were administered on 2 occasions to a sample of people with schizophrenia. Traditional psychometrics, bifactor analysis, and item response theory methods were used to explore item functioning and dimensionality and to compare instruments. Despite containing similar item content, responses to the CGI-CogS demonstrated superior psychometric properties (e.g., higher item intercorrelations, better spread of ratings across response categories) relative to the SCoRS. The authors argue that these differences arise mainly from the differential use of prompts and how the items are phrased and scored. Bifactor analysis demonstrated that although both measures capture a broad range of cognitive functioning (e.g., working memory, social cognition), the common variance on each is overwhelmingly explained by a single general factor. Item response theory analyses of the combined pool of 41 items showed that measurement precision is peaked in the mild to moderate range of cognitive impairment. Finally, simulated adaptive testing revealed that only about 10 to 12 items are necessary to achieve latent trait level estimates with reasonably small standard errors for most individuals. This suggests that these interview-based measures of cognitive deficits could be shortened without loss of measurement precision.

The MCCB impairment profile for schizophrenia outpatients: results from the MATRICS psychometric and standardization study.

The MATRICS Psychometric and Standardization Study was conducted as a final stage in the development of the MATRICS Consensus Cognitive Battery (MCCB). The study included 176 persons with schizophrenia or schizoaffective disorder and 300 community residents. Data were analyzed to examine the cognitive profile of clinically stable schizophrenia patients on the MCCB. Secondarily, the data were analyzed to identify which combination of cognitive domains and corresponding cut-off scores best discriminated patients from community residents, and patients competitively employed vs. those not. Raw scores on the ten MCCB tests were entered into the MCCB scoring program which provided age- and gender-corrected T-scores on seven cognitive domains. To test for between-group differences, we conducted a 2 (group)×7 (cognitive domain) MANOVA with follow-up independent t-tests on the individual domains. Classification and regression trees (CART) were used for the discrimination analyses. Examination of patient T-scores across the seven cognitive domains revealed a relatively compact profile with T-scores ranging from 33.4 for speed of processing to 39.3 for reasoning and problem-solving. Speed of processing and social cognition best distinguished individuals with schizophrenia from community residents; speed of processing along with visual learning and attention/vigilance optimally distinguished patients competitively employed from those who were not. The cognitive profile findings provide a standard to which future studies can compare results from other schizophrenia samples and related disorders; the classification results point to specific areas and levels of cognitive impairment that may advance work rehabilitation efforts.

The Cognitive Assessment Interview (CAI): development and validation of an empirically derived, brief interview-based measure of cognition.

BACKGROUND: Practical, reliable "real world" measures of cognition are needed to supplement neurocognitive performance data to evaluate possible efficacy of new drugs targeting cognitive deficits associated with schizophrenia. Because interview-based measures of cognition offer one possible approach, data from the MATRICS initiative (n=176) were used to examine the psychometric properties of the Schizophrenia Cognition Rating Scale (SCoRS) and the Clinical Global Impression of Cognition in Schizophrenia (CGI-CogS). METHOD: We used classical test theory methods and item response theory to derive the 10-item Cognitive Assessment Interview (CAI) from the SCoRS and CGI-CogS ("parent instruments"). Sources of information for CAI ratings included the patient and an informant. Validity analyses examined the relationship between the CAI and objective measures of cognitive functioning, intermediate measures of cognition, and functional outcome. RESULTS: The rater's score from the newly derived CAI (10 items) correlate highly (r=.87) with those from the combined set of the SCoRS and CGI-CogS (41 items). Both the patient (r=.82) and the informant (r=.95) data were highly correlated with the rater's score. The CAI was modestly correlated with objectively measured neurocognition (r=-.32), functional capacity (r=-.44), and functional outcome (r=-.32), which was comparable to the parent instruments. CONCLUSIONS: The CAI allows for expert judgment in evaluating a patient's cognitive functioning and was modestly correlated with neurocognitive functioning, functional capacity, and functional outcome. The CAI is a brief, repeatable, and potentially valuable tool for rating cognition in schizophrenia patients who are participating in clinical trials.

Can hypnosis differentiate epileptic from nonepileptic events in the video/EEG monitoring unit? Data from a pilot study.

OBJECTIVE: An estimated 24% of patients referred to epilepsy clinics actually have nonepileptic seizures. Various procedures have been used to precipitate nonepileptic events. The goal of this study was to use hypnosis in seizure provocation and differentiation between epileptic and nonepileptic seizure events. METHODS: Fifty study participants were enrolled from the Via Christi Comprehensive Epilepsy Center's video/electroencephalography unit. Patients underwent the Hypnotic Induction Profile (HIP) to assess susceptibility to hypnosis. After completion of the HIP, participants underwent hypnosis by a physician trained to do so. They received a hypnotic suggestion to have a seizure. All seizure-like events were classified as either an epileptic, nonepileptic, or undetermined event based on whether or not the patient had abnormal EEG activity during the event. RESULTS: Of the 50 participants enrolled, 3 withdrew consent, resulting in 47 participants. Seven (15%) participants failed to have an event of any type and were classified as undetermined. Sixteen (34%) participants were classified as having epileptic seizure events, and 24 (51%) participants had nonepileptic events. Most participants were Caucasian (87%), female (57%), and unemployed (55%). HIP scores ranged from 0 to 10. Participants classified with nonepileptic scores had higher mean HIP scores (8.08, SD 2.483) than those diagnosed with epileptic seizures [5.94, SD 3.492, t(25)=2.126, P=0.044]. The sensitivity of eliciting a nonepileptic event during hypnosis was only 0.46, but the specificity was 0.88. CONCLUSION: Hypnosis may be considered as a method of seizure provocation. Events provoked by hypnotic suggestion were more likely than not to be nonepileptic events. However, the current study has moderate specificity and poor sensitivity. Seizures could not be induced in patients who did not also have spontaneous seizures. Additional methodologies for seizure provocation need to be explored.

Normative data on and psychometric properties of Verbal and Visual Indexes of the RBANS in older adults.

The original structure of the RBANS includes five Indexes; however, recent factor analyses support a two-factor solution (Duff et al., 2006; Wilde, 2006). Unfortunately, normative and psychometric data do not currently exist on these two new Indexes. Building on prior work, the current study provides age- and education-corrected normative data to calculate the factor-derived Verbal and Visual RBANS Indexes (Duff et al., 2006) in a large cohort (n = 718) of older adult primary care patients. Psychometric data (e.g., discrepancy scores, internal consistency, retest reliability) on these new Indexes are also presented. These data might allow clinicians and researchers to better assess laterality effects of brain dysfunction when using the RBANS, although clinical validation is needed.

Functional co-primary measures for clinical trials in schizophrenia: results from the MATRICS Psychometric and Standardization Study.

OBJECTIVE: During the consensus meetings of the National Institute of Mental Health Measurement and Treatment Research to Improve Cognition in Schizophrenia (NIMH-MATRICS) Initiative, the U.S. Food and Drug Administration took the position that a drug for this purpose should show changes on 1) an accepted consensus cognitive performance measure and 2) an additional measure (i.e., a co-primary) that is considered functionally meaningful. The goal of the current study was to describe steps to evaluate four potential co-primary measures for psychometric properties and validity. METHOD: As part of the five-site MATRICS Psychometric and Standardization Study (PASS), two measures of functional capacity and two interview-based measures of cognition were evaluated in 176 patients with schizophrenia (167 of these patients were retested 4 weeks later). RESULTS: Data are presented for each co-primary measure for test-retest reliability, utility as a repeated measure, relationship to cognitive performance, relationship to functioning, tolerability/practicality, and number of missing data. CONCLUSIONS: Psychometric properties of all of the measures were considered acceptable, and the measures were generally comparable across the various criteria, except that the functional capacity measures had stronger relationships to cognitive performance and fewer missing data. The development and evaluation of potential co-primary measures is still at an early stage, and it was decided not to endorse a single measure for clinical trials at this point. The current findings offer the initial steps to identify functionally meaningful co-primary measures in this area and will help to guide further evaluation of such measures.