Threats to the integrity of psychological assessment: The misuse of test raw data and materials.

In the practice of psychological assessment there have been warnings for decades by the American Psychological Association (APA), the National Academy of Neuropsychology (NAN), other associations, and test vendors, against the disclosure of test raw data and test materials. Psychological assessment occurs across several different practice environments, and test raw data is a particularly sensitive aspect of practice considering what it implicitly represents about a client/patient, and this concept is further developed in this paper. Many times, test materials are intellectual property protected by copyrights and user agreements. It follows that improper management of the release of test raw data and test materials threatens the scientific integrity of psychological assessment. Here the matters of test raw data, test materials, and different practice environments are addressed to highlight the challenges involved with improper releases and to offer guidance concerning good-faith efforts to preserve the integrity of psychological assessment and legal agreements. The unique demands of forensic practice are also discussed, including attorneys' needs for cross-examination and discovery, which may place psychologists (and other duly vetted evaluators) in conflict with their commitment to professional ethical codes and legal agreements. To this end, important threats to the proper use of test raw data and test materials include uninformed professionals and compromised evaluators. In this paper, the mishandling of test raw data and materials by both psychologists and other evaluators is reviewed, representative case examples, including those from the literature, are provided, pertinent case law is discussed, and practical stepwise conflict resolutions are offered.

Ecological validity of the Meyers Neuropsychological Battery.

Neuropsychologists are increasingly being asked to apply neuropsychological test results to real world functioning; however, neuropsychological tests are not usually constructed to do so, but instead are more concerned with diagnostic accuracy than with prediction of daily functioning. Using samples of 5,460 patients that did self-ratings and 2791 patients that had family ratings plus the Meyers Neuropsychological Battery (MNB), it was found that the family ratings were better predicted by neuropsychological test data than were self-ratings on the 38 item Patient Competency Rating Scale (PCRS). The R values for family ratings on the 36 regression equations ranged from .236 to .763. The results show that the ratings given patients by family members could be predicted by the neuropsychological test results. These findings can help the clinician to make broad statements regarding likely real-life functioning and also support the ecological validity of the tests that make up the MNB.

Objective methods for matching neuropsychological patterns: Formulas and comparisons.

INTRODUCTION: Objective neuropsychology test score pattern matching methods can help to identify data similarities and differences with comparison groups which can help the clinician in diagnosis and in identifying treatment options. MATERIALS AND METHODS: The current study examines five methods of matching a data set: Correlation, Configuration, Kullback-Leibler (KL) Divergence, Pooled Effect Size (Cohen's d), and a new method called MNB (Meyers Neuropsychological Battery) Code. Thirty data sets diagnosed with Traumatic Brain Injury (TBI) were compared with four Comparison Group data sets consisting of TBI, Depression, Anxiety and Attention Deficit/Hyperactivity Disorder. RESULTS: The Correlation Method was correct 90% (27/30) and Configuration was correct 86% (26/30). The KL Divergence was correct 76% (23/30) and the MNB Code was correct 73% (22/30). The Effect Size Method was correct 70% (21/30). When using a simple majority of all the matching methods, the classification rate was 90+ percent. CONCLUSIONS: The results of this study demonstrate that there are statistical methods that can identify patterns of cognitive strengths and weaknesses. Multiple matching methods and a simple majority of agreement between the different comparisons suggests the best matching profile for diagnosis. In some cases, more than one pattern may be present.

Physician norms for the Automated Neuropsychological Assessment Metrics (ANAM).

INTRODUCTION: The development of population specific norms for the Automated Neuropsychological Assessment Metrics (ANAM) is to help expand the utility for use with other population groups. Currently, normative data are available for general military, special operations, athletic, and community populations. The current study provides normative reference values for use in assessing physicians. MATERIALS AND METHODS: Retrospective data from a sample of 2,288 physicians who completed the ANAM were examined. The data were gathered from practicing civilian physicians serving in the National Guard or Reserves who completed the assessment as part of the mandated Department of Defense clinical testing policy. The effects of age and sex on performance were examined for all subtests. Normative tables were stratified by age (23-29, 30-34, 35-39, 40-44, 45-49, 50-54, 55-59, 60-64, 65-79) and sex. Base rates of low scores among the physician population were also examined. RESULTS: Older age was associated with lower performance on all subtests with small-to-medium effect sizes. A mixed pattern of performance was observed as a factor of sex with females scoring better on one subtest and males scoring better on three subtests. However, effects sizes associated with sex were small and likely of minimal clinical significance. A total of 11.3% of the physician sample scored below average on two or more tests in the battery. CONCLUSIONS: This study provides normative data for practicing physicians that can be used to help inform clinical decision-making.

A Validity Measure for the Automated Neuropsychological Assessment Metrics.

The Automated Neuropsychological Assessment Metrics (ANAM) is one of the most widely used and validated neuropsychological instruments for assessing cognition. The ANAM Test System includes a reporting tool, the ANAM Validity Indicator Report that generates scores for the embedded effort measure, the ANAM Performance Validity Index (APVI). The current study seeks to develop a proxy for the APVI, using raw subtest summary test scores. This would be useful for situations where the APVI score is unavailable (e.g., validity report not generated at the time of the assessment) or when the item level data needed to generate this score are inaccessible. ANAM scores from a large data set of 1,000,000+ observations were used for this retrospective analysis. Results of linear regression analysis suggest that the APVI can be reasonably estimated from the raw subtest summary test scores that are presented on the ANAM Performance Report. Clinically, this means that an important step in the interpretation process, checking the validity of test data, can still be performed even when the APVI is not available.

Premorbid estimates of neuropsychological functioning for diverse groups.

One of the basic tasks performed by a neuropsychologist is to identify the difference between current performance and the premorbid expected performance. Baseline expected performance for Intellectually Impaired (n = 21), Developmentally Delayed (n = 40), Attention Deficit Disorder (n = 98), Learning Disability (n = 42), and "Normal" groups (n = 75) were developed along with a demographically corrected prediction of premorbid functioning and a word reading based prediction of premorbid functioning. We utilized a subset of this data pool for development (n = 107) and validation (n = 108) of premorbid functioning estimates. Findings show that a combination of three methods (baseline, demographic, and reading) were superior to any individual method. The effect size (Cohen's d) calculations show that differences in the prediction of domain level performances were small and likely not clinically meaningful, indicating that the premorbid estimates would be usable as a prediction of expected performance at the domain level. However, the motor domains were not well predicted.

Time estimation: Close your eyes and tell me when a minute goes by.

The 1-minute estimation test is a relatively new task. This article is an initial offering showing normative data for children from age six up through adulthood (age =76). The task is quite simple, the individual was simply asked to estimate the passage of time for 1 minute. The normative data indicate that children's data needs to be stratified by age, whereas adult data did not need to be stratified by age. Demographic variables such as handedness, gender and ethnicity were not significantly related to 1-minute estimation performance. In looking at a comparison between Traumatic Brain Injured (TBI) patients and Normal Controls, there was a significant effect suggesting that the 1-minute estimation is sensitive to cognitive impairment. Further investigation is needed; however, investigation must begin with the availability of adequate norms so that comparisons can be made. This article provides the needed normative data.

Long-Term Test-Retest Stability of ANAM in a Large Military Sample.

OBJECTIVE: This paper examines the long-term stability of the Automated Neuropsychological Assessment Metrics (ANAM) over 1-year, 3-year, and 5-year time periods. METHODS: The sample was drawn from active duty military personnel. All individuals were administered the ANAM and were re-administered the ANAM a second time, either 1 year, 3 years, or 5 years later. All individuals had no reported cognitive problems or concussion or any traumatic brain injury between the first and second assessments and had no intervening contact with the ANAM. All were assessed using the military standard approach with trained proctors. RESULTS: The results for (n = 19,997) individuals who were tested 1 year apart showed an intra-class correlation (ICC) of .6 for simple reaction time and simple reaction time repeat and .7 and above for all other scales. For the sample with 3 years between testings (n = 9,166), the ICC was similar to the 1-year sample. When the 5 year between testings data was examined (n = 2,270), the ICCs for all scales except the simple reaction time and simple reaction time repeat showed ICCs of .7 and.8. The simple reaction time and simple reaction time repeat showed ICCs of .596. CONCLUSIONS: These results indicate that the use of the ANAN baseline for comparison can be done even if the baseline is up to 5 years old.

Use of multiple performance and symptom validity measures: Determining the optimal per test cutoff for determination of invalidity, analysis of skew, and inter-test correlations in valid and invalid performance groups.

Objective: Discrimination of patients passing vs. failing the Word Memory Test (WMT) by performance on 11 performance and symptom validity tests (PVTs, SVTs) from the Meyers Neuropsychological Battery (MNB) at per-test false positive cutoffs ranging from 0 to 15%. PVT and SVT intercorrelation in subgroups passing and failing the WMT, as well as the degree of skew of the individual PVTs and SVT in the pass/fail subgroups, were also analyzed. Method: In 255 clinical and forensic cases, 100 failed and 155 passed the WMT, at a base-rate of invalid performance of 39.2%. Performance was contrasted on 10 PVTs and 1 SVT from the MNB, using per-test false positive rates of 0.0%, 3.3%, 5.0%, 10.0%, and 15.0% in discriminating WMT pass and WMT fail groups. These two WMT groups were also contrasted using the 10 PVTs and 1 SVT as continuous variables in a logistic regression. Results: The per-PVT false positive rate of 10% yielded the highest WMT pass/fail classification, and more closely approximated the classification obtained by logistic regression than other cut scores. PVT and SVT correlations were higher in cases failing the WMT, and data were more highly skewed in those passing the WMT. Conclusions: The optimal per-PVT and SVT cutoff is at a false positive rate of 10%, with failure of ≥3 PVTs/SVTs out of 11 yielding sensitivity of 61.0% and specificity of 90.3%. PVTs with the best classification had the greatest degree of skew in the WMT pass subgroup.

Presidential Address National Academy of Neuropsychology Conference Boston 2017.

This presidential address attempts to predict the future directions of neuropsychology. Predicting the future is always a difficult thing. By examining population trends such as aging and demographics, a clearer picture becomes visible. The population is getting older and more ethnically diverse. Also, examination of the spending trends in health care indicates that neuropsychology needs to be able to adapt to working with larger population-based patient care as well as individual patient care. Shifts in the demographics of neuropsychology, in that the profession previously was 70% male dominate and now is >70% female dominant are also discussed. Trends in NAN's speaker and leader demographics are examined as well as the need to stay current in the trends and latest neuropsychological research lest we become dinosaurs in the next 5-10 years. Recommendations for new neuropsychologists and post-doctoral fellows are also presented.

WAIS-IV Seven-Subtest Short Form: Validity and Clinical Use in Schizophrenia.

OBJECTIVE: This study assesses the psychometric properties of Ward's seven-subtest short form (SF) for WAIS-IV in a sample of adults with schizophrenia (SZ) and schizoaffective disorder. METHOD: Seventy patients diagnosed with schizophrenia or schizoaffective disorder were administered the full version of the WAIS-IV. Four different versions of the Ward's SF were then calculated. The subtests used were: Similarities, Digit Span, Arithmetic, Information, Coding, Picture Completion, and Block Design (BD version) or Matrix Reasoning (MR version). Prorated and regression-based formulae were assessed for each version. RESULTS: The actual and estimated factorial indexes reflected the typical pattern observed in schizophrenia. The four SFs correlated significantly with their full-version counterparts, but the Perceptual Reasoning Index (PRI) correlated below the acceptance threshold for all four versions. The regression-derived estimates showed larger differences compared to the full form. The four forms revealed comparable but generally low clinical category agreement rates for factor indexes. All SFs showed an acceptable reliability, but they were not correlated with clinical outcomes. CONCLUSIONS: The WAIS-IV SF offers a good estimate of WAIS-IV intelligence quotient, which is consistent with previous results. Although the overall scores are comparable between the four versions, the prorated forms provided a better estimation of almost all indexes. MR can be used as an alternative for BD without substantially changing the psychometric properties of the SF. However, we recommend a cautious use of these abbreviated forms when it is necessary to estimate the factor index scores, especially PRI, and Processing Speed Index.

Embedded Measures of Performance Validity in the Rey Complex Figure Test in a Clinical Sample of Veterans.

The purpose of this study was to determine how well scores from the Rey Complex Figure Test (RCFT) could serve as embedded measures of performance validity in a large, heterogeneous clinical sample at an urban-based Veterans' Affairs hospital. Participants were divided into credible performance (n = 244) and noncredible performance (n = 87) groups based on common performance validity tests during their respective clinical evaluations. We evaluated how well preselected RCFT scores could discriminate between the 2 groups using cut scores from single indexes as well as multivariate logistic regression prediction models. Additionally, we evaluated how well memory error patterns (MEPs) could discriminate between the 2 groups. Optimal discrimination occurred when indexes from the Copy and Recognition trials were simultaneous predictors in logistic regression models, with 91% specificity and at least 53% sensitivity. Logistic regression yielded superior discrimination compared with individual indexes and compared with the use of MEPs. Specific scores on the RCFT, including the Copy and Recognition trials, can serve as adequate indexes of performance validity, when using both cut scores and logistic regression prediction models. We provide logistic regression equations that can be applied in similar clinical settings to assist in determining performance validity.

Normative Data for the Neurobehavioral Symptom Inventory.

The demographically diverse populations served by large health care systems (Veterans Affairs, Department of Defense, Medicare, Medicaid) are routinely screened with the Neurobehavioral Symptom Inventory (NSI). The extent to which a patient's report of symptoms either initially and/or across time is affected by demographic variables-gender, ethnicity, age, or education-has not been investigated despite widespread use of the NSI. In practice, the effectiveness of this tool might be improved with demographically based norms. A large data set of normal community-dwelling individuals was collected using the NSI. Emphasis was made to collect data from individuals of diverse ethnic backgrounds. It was hypothesized that ethnic/cultural backgrounds would have an impact on NSI scores. The results provide normative data for the NSI applicable to a wide variety of individuals of various ages and ethnic backgrounds. An analysis of variance indicated there was no significant difference in NSI responses based on ethnic/cultural background; however, age and gender were found to contribute significantly to the variance associated with symptom endorsement. The NSI appears to be a reliable measure of self-report postconcussive symptoms. Age is a variable associated with differential symptom endorsement on the NSI. Follow-up studies are needed to provide a measure of the sensitivity and specificity of this measure.

Accuracy of the Modified Somatic Perception Questionnaire and Pain Disability Index in the detection of malingered pain-related disability in chronic pain.

The Modified Somatic Perception Questionnaire (MSPQ) and the Pain Disability Index (PDI) are both popular clinical screening instruments in general orthopedic, rheumatologic, and neurosurgical clinics and are useful for identifying pain patients whose physical symptom presentations and disability may be non-organic. Previous studies found both to accurately detect malingered pain presentations; however, the generalizability of these results is not clear. This study used a criterion groups validation design (retrospective cohort of patients with chronic pain, n = 328) with a simulator group (college students, n = 98) to determine the accuracy of the MSPQ and PDI in detecting Malingered Pain Related Disability. Patients were grouped based on independent psychometric evidence of MPRD. Results showed that MSPQ and PDI scores were not associated with objective medical pathology. However, they accurately differentiated Not-MPRD from MPRD cases. Diagnostic statistics associated with a range of scores are presented for application to individual cases. Data from this study can inform the clinical management of chronic pain patients by screening for psychological overlay and malingering, thus alerting clinicians to the possible presence of psychosocial obstacles to effective treatment and triggering further psychological assessment and/or treatment.

Emotional distress affects attention and concentration: the difference between mountains and valleys.

The current study tests the hypothesis that the "mountains and valleys pattern" (MVP) observed within the Attention and Concentration domain of the Meyers Neuropsychological Battery reflects the interference of emotional distress/anxiety on the patient's cognitive test performance. First, the MVP was objectively quantified using a formula that took into account both increased and decreased scores, rather than canceling them out through averaging. Using a total sample of 787 subjects, the Minnesota Multiphasic Personality Inventory-Second Edition Restructured Form (MMPI-2-RF) profile scores of cases with and without this pattern were then compared using an extensive database followed by a smaller, matched-groups design. The presence of the MVP was related to MMPI-2-RF test performance. In particular, this pattern was related to emotional distress/anxiety scales but was not related to scales reflecting neurological or cognitive complaints. The degree of emotional distress experienced may affect attention and concentration test performance in a way that sometimes heightens focus and at other times disrupts focus. The MVP may be used to assess the effects of emotional distress on the consistency of an individual patient's attention and concentration test performance.

Using pattern analysis matching to differentiate TBI and PTSD in a military sample.

Distinguishing between traumatic brain injury (TBI) residuals and the effects of posttraumatic stress disorder (PTSD) during neuropsychological evaluation can be difficult because of significant overlap of symptom presentation. Using a standardized battery of tests, an artificial neural network was used to create an algorithm to perform pattern analysis matching (PAM) functions that can be used to assist with diagnosis. PAM analyzes a patient's neuropsychological data and provides a best fit classification, according to one of four groups: TBI, PTSD, malingering/invalid data, or "other" (depressed/anxious/postconcussion syndrome/normal). The original PAM was modeled on civilian data; the current study was undertaken using a database of 100 active-duty army service personnel who were referred for neuropsychological assessment in a military TBI clinic. The PAM classifications showed 90% overall accuracy when compared with clinicians' diagnoses. The PAM function is able to classify detailed neuropsychological profiles from a military population with a high degree of accuracy and is able to distinguish between TBI, PTSD, malingering/invalid data, or "other." PAM is a useful tool to help with clinical decision-making.

An adaptation of the MMPI-2 Meyers Index for the MMPI-2-RF.

Using an overall sample of 278 individuals who had taken the Minnesota Multiphasic Personality Inventory-Second Edition (MMPI-2) and who had clear diagnostic information available in their medical records, the Meyers Index (MI) for the MMPI-2 (Meyers, Millis, & Volkert, 2002 ) was calculated for each individual, and a new version of the MI created for the MMPI-2 Restructured Form (MMPI-2-RF) was calculated. The MI is a method of combining multiple MMPI-2 validity scales into a single weighted index to assess exaggerated self-report on the MMPI-2. The new index is intended to provide the same type of global assessment of validity but for the MMPI-2-RF (MI-r). The MI and the MI-r were compared at both individual and group levels and were found to correlate well (r = .87). Diagnostic groups of litigants and nonlitigants of traumatic brain injury, chronic pain, and posttraumatic stress disorder were also examined; and the performance of the MI and the MI-r were similar. Similarly, the pass and fail agreement rate for the two scales was 93%. The results indicate that the MI and MI-r perform very similarly and are good methods of assessing overall validity of MMPI-2 and MMPI-2-RF test performance.

Finger Tapping Test performance as a measure of performance validity.

The Finger Tapping Test (FTT) has been presented as an embedded measure of performance validity in most standard neuropsychological evaluations. The present study evaluated the utility of three different scoring systems intended to detect invalid performance based on FTT. The scoring systems were evaluated in neuropsychology cases from clinical and independent practices, in which credible performance was determined based on passing all performance validity measures or failing two or more validity indices. Each FTT scoring method presented with specificity rates at approximately 90% and sensitivity of slightly more than 40%. When suboptimal performance was based on the failure of any of the three scoring methods, specificity was unchanged and sensitivity improved to 50%. The results are discussed in terms of the utility of combining multiple scoring measures for the same test as well as benefits of embedded measures administered over the duration of the evaluation.

Using likelihood ratios to detect invalid performance with performance validity measures.

Larrabee (2008) applied chained likelihood ratios to selected performance validity measures (PVMs) to identify non-valid performances on neuropsychological tests. He presented a method of combining different PVMs with different sensitivities and specificities into an overall probability of non-validity. We applied his methodology to a set of 11 PVMs using a sample of 255 subjects. The results of the study show that in various combinations of two or three PVMs, a high reliability of invalidity can be determined using the chained likelihood ratio method. This study advances the ability of clinicians to chain various PVMs together and calculate the probability that a set of data is invalid.

Cognitive intra-individual variability has a positive association with traumatic brain injury severity and suboptimal effort.

This study examined intra-individual variability in a large sample (n = 629) of individuals with a history of mild traumatic brain injury (mTBI) or TBI referred for neuropsychological evaluation. Variability was assessed using the overall test battery mean standard deviation (OTBM SD). We found a negative linear relation between OTBM and OTBM SD (r = -.672) in this sample with a history of neurologic pathology, indicating that the variability is inversely related to cognitive performance and contrary to what is observed in most normative data. Analyses revealed main effects for OTBM and OTBM SD across three TBI severity groups: loss of consciousness (LOC) <1 h, LOC 1 h-6 days, and LOC >6 days. These effects were found for both a valid performance group (no failed embedded validity measures; n = 504) and an invalid performance group (failed one or more embedded validity measures; n = 125). These findings support that cognitive intra-individual variability is increased uniquely by both neuropathology and suboptimal effort, there is a dose-response relationship between neuropathology and cognitive variability, and intra-individual variability may have utility as a clinical index of both.