Performance and Symptom Validity Testing in Pediatric Assessment: A Review of the Literature.

There has been an increased recognition that validity testing is an integral component of evaluations conducted with youth. The incorporation of validity testing provides an objective basis for placing confidence in the test data as an accurate assessment of the child's or adolescent's current ability level and/or an accurate indication of the child's or adolescent's current symptoms or behavioral functioning. The use of objective performance validity and symptom validity is consistent with the current emphasis of data-driven decision-making. This paper provides a review of the literature on performance and symptom validity tests for children and adolescents. The strengths and limitations of the available validity tests are discussed, and recommendations for use in pediatric assessments are provided.

A Generalizable, Data-Driven Approach to Predict Daily Risk of Clostridium difficile Infection at Two Large Academic Health Centers.

OBJECTIVE An estimated 293,300 healthcare-associated cases of Clostridium difficile infection (CDI) occur annually in the United States. To date, research has focused on developing risk prediction models for CDI that work well across institutions. However, this one-size-fits-all approach ignores important hospital-specific factors. We focus on a generalizable method for building facility-specific models. We demonstrate the applicability of the approach using electronic health records (EHR) from the University of Michigan Hospitals (UM) and the Massachusetts General Hospital (MGH). METHODS We utilized EHR data from 191,014 adult admissions to UM and 65,718 adult admissions to MGH. We extracted patient demographics, admission details, patient history, and daily hospitalization details, resulting in 4,836 features from patients at UM and 1,837 from patients at MGH. We used L2 regularized logistic regression to learn the models, and we measured the discriminative performance of the models on held-out data from each hospital. RESULTS Using the UM and MGH test data, the models achieved area under the receiver operating characteristic curve (AUROC) values of 0.82 (95% confidence interval [CI], 0.80-0.84) and 0.75 ( 95% CI, 0.73-0.78), respectively. Some predictive factors were shared between the 2 models, but many of the top predictive factors differed between facilities. CONCLUSION A data-driven approach to building models for estimating daily patient risk for CDI was used to build institution-specific models at 2 large hospitals with different patient populations and EHR systems. In contrast to traditional approaches that focus on developing models that apply across hospitals, our generalizable approach yields risk-stratification models tailored to an institution. These hospital-specific models allow for earlier and more accurate identification of high-risk patients and better targeting of infection prevention strategies. Infect Control Hosp Epidemiol 2018;39:425-433.

Perfluoroalkyl substances, thyroid hormones, and neuropsychological status in older adults.

Minimal data exist regarding the neurotoxicity of perfluoroalkyl substances (PFASs) in aging populations and the possible mediating effects of thyroid hormones (THs). Hence, the aims of this study were to: (i) assess associations between PFASs and neuropsychological function, and (ii) determine if such associations are mediated by changes in circulating THs in an aging population. We measured perfluorooctanoic acid (PFOA), perfluorooctane sulfonate (PFOS), total thyroxine (T4) and free thyroxine (fT4) in serum and performed neuropsychological tests in 126 men and women aged 55-74 years and living in upper Hudson River communities. Multivariable linear regressions were conducted to assess associations between PFASs and neuropsychological test scores. Mediation analyses were performed in a subset of 87 participants for whom information was available on both PFASs and THs. We calculated TH-mediated, non-TH mediated, and total effects of PFASs on neuropsychological test scores. Higher PFOA was associated with better performance in tasks of the California Verbal Learning Test and the Wisconsin Card Sorting Test. Higher PFOS was associated with improved performance in a Wechsler Memory Scale subtest and Block Design Subtest (BDT) total scores. There was no evidence of mediation by THs for PFOA-neuropsychological function associations. However, T4 and fT4 partially mediated the protective effect of PFOS on BDT total scores. Our findings do not suggest that PFASs are associated with poor neuropsychological function. There was some evidence of mediation for the association between PFASs and neuropsychological functions by THs, although some other modes of action also appear likely.

Not all performance validity tests are created equal: The role of recollection and familiarity in the Test of Memory Malingering and Word Memory Test.

INTRODUCTION: The Test of Memory Malingering (TOMM) and the Word Memory Test (WMT) are both performance validity tests (PVTs) that use a two-alternative forced-choice (2AFC) recognition memory format. Several studies have reported that these tests are susceptible to cognitive impairment and that the WMT is more susceptible than the TOMM. The current study explored components of recognition memory (i.e., conscious recollection and familiarity) underlying the TOMM and WMT to identify factors that make them susceptible and resilient to cognitive impairment. METHOD: Fifty-four nonclinical undergraduate research participants were administered the TOMM and WMT while providing introspective judgments about their recognition memory using the remember/know/guess procedure. In addition, half of participants were administered dual-task interference, a manipulation intended to reduce recollection, during these tests, while the other half completed these tests without interference. Standard cutoffs on the TOMM and WMT were explored, as well as alternative cutoffs based on TOMM Trial 1 scores. RESULTS: The WMT was more impacted by dual-task interference than standard TOMM cutoff trials, while alternative TOMM cutoff trials were equally impacted by dual-task interference relative to the WMT. Dual-task interference reduced recollection on these tests, but spared familiarity. Standard TOMM trials and the WMT were relatively comparable on levels of recollection, but familiarity contributed more to the TOMM than to the WMT. Alternative TOMM trials possessed lower familiarity and recollection than standard TOMM trials and lower recollection than the WMT. CONCLUSIONS: Reduced recollection places examinees at risk of failing the TOMM and WMT, while familiarity contributes to the relative resilience of the standard TOMM. Future development of 2AFC recognition memory PVTs should attempt to maximize the contribution of familiarity to their completion.

Thyroid function and neuropsychological status in older adults.

Overt thyroid dysfunction is recognized as a risk factor for neuropsychological deficits in aging populations, yet evidence for how changes in levels of circulatory thyroid hormones impact specific neuropsychological domains is limited. Here we report cross-sectional associations between serum thyroid hormone concentrations and several neuropsychological function domains among men and women aged 55-74years. We administered neuropsychological tests to assess memory, learning, executive function, measures of attention, visuospatial function, affective state, and motor function. Multivariable linear regression analyses were performed adjusting for age, sex, education, and cigarette smoking. Effects were reported as differences in test scores per one interquartile range (IQR) increase in hormone concentration. Higher total thyroxine (T4) and free thyroxine (fT4) were associated with improved visuospatial function, as measured by Block Design Subtest total scores; associated increments per IQR differences in T4 and fT4 were 15% and 19%, respectively (false discovery rate q-values <0.05). We also detected statistical interactions between age and fT4 for effects in tasks of memory and learning. Concurrent increases in age and fT4 were associated with deficits in memory and learning as measured by California Verbal Learning Test subtests (10% and 16% deficits in t-score and short delay free recall score, respectively). Our findings suggest that changes in thyroid hormones may have important implications for neuropsychological function in aging populations. Further large-scale studies with comprehensive thyroid function and neuropsychological outcome assessments are warranted to confirm these results.

Validation of Victoria Symptom Validity Test Cutoff Scores among Mild Traumatic Brain Injury Litigants Using a Known-Groups Design.

The Victoria Symptom Validity Test (VSVT) is one of the most accurate performance validity tests. Previous research has recommended several cutoffs for performance invalidity classification on the VSVT. However, only one of these studies used a known groups design and no study has investigated these cutoffs in an exclusively mild traumatic brain injury (mTBI) medico-legal sample. The current study used a known groups design to validate VSVT cutoffs among mild traumatic brain injury litigants and explored the best approach for using the multiple recommended cutoffs for this test. Cutoffs of <18 Hard items correct, <41 Total items correct, an Easy - Hard items correct difference >6, and <5 items correct on any block yielded the strongest classification accuracy. Using multiple cutoffs in conjunction reduced classification accuracy. Given convergence across studies, a cutoff of <18 Hard items correct is the most appropriate for use with mTBI litigants.

Evaluation of the Generalizability of the Number of Abnormal Scores and the Overall Test Battery Mean as Measures of Performance Validity to a Different Test Battery.

Davis, Axelrod, McHugh, Hanks, and Millis (2013) documented that in a battery of 25 tests, producing 15, 10, and 5 abnormal scores at 1, 1.5, and 2 standard deviations below the norm-referenced mean, respectively, and an overall test battery mean (OTBM) of T ≤ 38 accurately identifies performance invalidity. However, generalizability of these findings to other samples and test batteries remains unclear. This study evaluated the use of abnormal scores and the OTBM as performance validity measures in a different sample that was administered a 25-test battery that minimally overlapped with Davis et al.'s test battery. Archival analysis of 48 examinees with mild traumatic brain injury seen for medico-legal purposes was conducted. Producing 18 or more, 7 or more, and 5 or more abnormal scores at 1, 1.5, and 2 standard deviations below the norm-referenced mean, respectively, and an OTBM of T ≤ 40 most accurately classified examinees; however, using Davis et al.'s proposed cutoffs in the current sample maintained specificity at or near acceptable levels. Due to convergence across studies, producing ≥5 abnormal scores at 2 standard deviations below the norm-referenced mean is the most appropriate cutoff for clinical implementation; however, for batteries consisting of a different quantity of tests than 25, an OTBM of T ≤ 38 is more appropriate.

Rates of False-Positive Classification Resulting From the Analysis of Additional Embedded Performance Validity Measures.

Several studies have documented improvements in the classification accuracy of performance validity tests (PVTs) when they are combined to form aggregated models. Fewer studies have evaluated the impact of aggregating additional PVTs and changing the classification threshold within these models. A recent Monte Carlo simulation demonstrated that to maintain a false-positive rate (FPR) of ≤.10, only 1, 4, 8, 10, and 15 PVTs should be analyzed at classification thresholds of failing at least 1, at least 2, at least 3, at least 4, and at least 5 PVTs, respectively. The current study sought to evaluate these findings with embedded PVTs in a sample of real-life litigants and to highlight a potential danger in analytic flexibility with embedded PVTs. Results demonstrated that to maintain an FPR of ≤.10, only 3, 7, 10, 14, and 15 PVTs should be analyzed at classification thresholds of failing at least 1, at least 2, at least 3, at least 4, and at least 5 PVTs, respectively. Analyzing more than these numbers of PVTs resulted in a dramatic increase in the FPR. In addition, in the most extreme case, flexibility in analyzing and reporting embedded PVTs increased the FPR by 67%. Given these findings, a more objective approach to analyzing and reporting embedded PVTs should be introduced.

Performance Validity Classification Accuracy of Single-, Pairwise-, and Triple-Failure Models Using the Halstead-Reitan Neuropsychological Battery for Adults.

Embedded performance validity tests (PVTs) have been criticized for their poor specificity and sensitivity. Aggregated models of embedded PVTs have been proposed to improve their classification accuracy; however, limitations to aggregation-based improvement of PVTs have yet to be explored. The current study evaluated the classification accuracy of 3 types of models of embedded PVTs in the Halstead-Reitan Neuropsychological Battery for Adults (HRNB): a single-, a pairwise-, and a triple-failure model. In addition, this study evaluated the impact of aggregating between 1 and 6 embedded PVTs in each of these 3 types of models. Analyzing only the 2, 4, and 6 most discriminating embedded PVTs in the single-, pairwise-, and triple-failure models maximized classification accuracy, respectively. Comparisons across these models indicated that the single-failure model including only the two most discriminating embedded PVTs had the best classification accuracy; however, classification accuracy was only minimally improved in this model relative to analyzing just Reliable Digit Span. These results suggest that aggregation of embedded PVTs from the HRNB does not substantially improve their classification accuracy and that the benefits of aggregating PVTs may only emerge when the PVTs entered into the aggregated models have sufficient classification accuracy on their own.

Earthquake and tsunami forecasts: relation of slow slip events to subsequent earthquake rupture.

The 5 September 2012 M(w) 7.6 earthquake on the Costa Rica subduction plate boundary followed a 62-y interseismic period. High-precision GPS recorded numerous slow slip events (SSEs) in the decade leading up to the earthquake, both up-dip and down-dip of seismic rupture. Deeper SSEs were larger than shallower ones and, if characteristic of the interseismic period, release most locking down-dip of the earthquake, limiting down-dip rupture and earthquake magnitude. Shallower SSEs were smaller, accounting for some but not all interseismic locking. One SSE occurred several months before the earthquake, but changes in Mohr-Coulomb failure stress were probably too small to trigger the earthquake. Because many SSEs have occurred without subsequent rupture, their individual predictive value is limited, but taken together they released a significant amount of accumulated interseismic strain before the earthquake, effectively defining the area of subsequent seismic rupture (rupture did not occur where slow slip was common). Because earthquake magnitude depends on rupture area, this has important implications for earthquake hazard assessment. Specifically, if this behavior is representative of future earthquake cycles and other subduction zones, it implies that monitoring SSEs, including shallow up-dip events that lie offshore, could lead to accurate forecasts of earthquake magnitude and tsunami potential.

Base rate of performance invalidity among non-clinical undergraduate research participants.

Neuropsychological research frequently uses non-clinical undergraduate participants to evaluate neuropsychological tests. However, a recent study by An and colleagues (2012, Archives of Clinical Neuropsychology, 27, 849-857) called into question that the extent to which the interpretation of these participants' performance on neuropsychological tests is valid. This study found that in a sample of 36 participants, 55.6% exhibited performance invalidity at an initial session and 30.8% exhibited performance invalidity at a follow-up session. The current study attempted to replicate these findings in a larger, more representative sample using a more rigorous methodology. Archival data from 133 non-clinical undergraduate research participants were analyzed. Participants were classified as performance invalid if they failed any one PVT. In the current sample, only 2.26% of participants exhibited performance invalidity. Thus, concerns regarding insufficient effort and performance invalidity when using undergraduate research participants appear to be overstated.

A comparison of the standard category test with a new computer version.

The Halstead Category Test is a popular measure of abstraction, concept formation, and logical analysis skills. Due to its large apparatus, however, ease of administration of the standard Category Test is limited. For this reason, a number of computer versions of the Category Test have been developed to facilitate its administration. The current study evaluated the equivalence of a new computer version to the standard Category Test in a sample of undergraduate students. Analyses revealed that the two versions did not differ significantly on subtest error scores, total error scores, or Neuropsychological Deficit Scale scores. Results of the current study support the equivalence of this new computer version to the standard version of the Category Test.

Tibial bone lead, but not serum polychlorinated biphenyl, concentrations are associated with neurocognitive deficits in former capacitor workers.

OBJECTIVE: To determine the relationships between tibial bone lead and serum polychlorinated biphenyl (PCB) concentrations and neurocognitive function. METHODS: The study population consisted of men and women former capacitor workers had been employed by the General Electric Corporation between 1946 and 1977. Regression analyses evaluated the association between neurocognitive function and lipid-adjusted serum PCB and tibia lead concentrations. RESULTS: Tibia lead, but not serum PCBs, was significantly correlated with deficits in neurocognitive function. Women showed more associations between tibia lead and neurocognitive function than men, especially regarding executive function. CONCLUSIONS: These results demonstrate that low levels of tibia lead, but not serum PCBs, are associated with neurocognitive deficits and that postmenopausal women show a greater number of deficits in executive function than men.

Can the Sensitivity and Specificity of the TOMM Be Increased with Differential Cutoff Scores?

Accurate determination of performance validity is paramount in any neuropsychological assessment. Numerous freestanding symptom validity tests, like the Test of Memory Malingering (TOMM), have been developed to assist in this process; however, research and clinical experiences have suggested that each may not function with the same classification accuracy. In an effort to increase the TOMM's ability to accurately classify performance validity, recent research has investigated the use of nonstandard cutoff scores. The purpose of this study was to potentially validate the use of two, nonstandard cutoff scores (<49 on Trial 2 or the Retention Trial or ≤39 on Trial 1) applied to the TOMM in a medicolegal sample of mild traumatic brain injury litigants. Both descriptive and inferential statistics found that the cutoff of <49 on Trial 2 or the Retention Trial was the most sensitive to performance validity as compared with both the standard TOMM criteria and the cutoff of ≤39. These findings support the use of nonstandard cutoffs to increase the TOMM's classification accuracy.

The degree of conation on neuropsychological tests does not account for performance invalidity among litigants.

Conation has been defined as the ability to focus and maintain intellectual energy over time. Prior research has shown that conation contributes to the magnitude of differences in test scores among brain-damaged and non-brain-damaged examinees. The purpose of the current investigation was to determine if conation might similarly account for differences in test scores among performance valid and performance invalid examinees. An archival analysis was therefore carried out on 52 examinees administered the Halstead-Reitan Neuropsychological Battery (HRNB) and several performance validity tests in a medico-legal context. Analyses revealed that conation had no impact on the magnitude of test score differences between groups and that performance invalid examinees scored worse than performance valid examinees on all but one test of the HRNB. These results support the idea that the identification of performance invalidity calls into question the reliability and the validity of all test score interpretations in an evaluation, even those with less conative load.

Performance of non-neurological older adults on the Wisconsin Card Sorting Test and the Stroop Color-Word Test: normal variability or cognitive impairment?

There is currently no standard criterion for determining abnormal test scores in neuropsychology; thus, a number of different criteria are commonly used. We investigated base rates of abnormal scores in healthy older adults using raw and T-scores from indices of the Wisconsin Card Sorting Test and Stroop Color-Word Test. Abnormal scores were examined cumulatively at seven cutoffs including >1.0, >1.5, >2.0, >2.5, and >3.0 standard deviations (SD) from the mean as well as those below the 10th and 5th percentiles. In addition, the number of abnormal scores at each of the seven cutoffs was also examined. Results showed when considering raw scores, ∼15% of individuals obtained scores>1.0 SD from the mean, around 10% were less than the 10th percentile, and 5% fell >1.5 SD or <5th percentile from the mean. Using T-scores, approximately 15%-20% and 5%-10% of scores were >1.0 and >1.5 SD from the mean, respectively. Roughly 15% and 5% fell at the <10th and <5th percentiles, respectively. Both raw and T-scores>2.0 SD from the mean were infrequent. Although the presence of a single abnormal score at 1.0 and 1.5 SD from the mean or at the 10th and 5th percentiles was not unusual, the presence of ≥2 abnormal scores using any criteria was uncommon. Consideration of base rate data regarding the percentage of healthy individuals scoring in the abnormal range should help avoid classifying normal variability as neuropsychological impairment.

The Albany Consistency Index for the Test of Memory Malingering.

The determination of examinee effort is an important component of a neuropsychological evaluation and relies heavily on the use of symptom validity tests (SVTs) such as the Test of Memory Malingering (TOMM) and the Word Memory Test (WMT). Diagnostic utility of SVTs varies. The sensitivity of traditional TOMM criteria to suboptimal effort is low. An index of response consistency across three trials of the TOMM was developed, denoted the Albany Consistency Index (ACI). This index identified a large proportion of examinees classified as optimal effort using traditional TOMM interpretive guidelines but suboptimal effort using the WMT profile analysis. In addition, previous research was extended, demonstrating a relationship between examinee performance on SVTs and neuropsychological tests. Effort classification using the ACI predicted the performance on the Global Memory Index from the Memory Assessment Scales. In conclusion, the ACI was a more sensitive indicator of suboptimal effort than traditional TOMM interpretive guidelines.

Are embedded validity indices equivalent to free-standing symptom validity tests?

Symptom validity assessment is an important part of neuropsychological evaluation. There are currently several free-standing symptom validity tests (SVTs), as well as a number of empirically derived embedded validity indices, that have been developed to assess that an examinee is putting forth an optimal level of effort during testing. The use of embedded validity indices is attractive since they do not increase overall testing time and may also be less vulnerable to coaching. In addition, there are some instances where embedded validity indices are the only tool available to the neuropsychological practitioner for assessing an examinee's level of effort. As with free-standing measures, the sensitivity and specificity of embedded validity indices to suboptimal effort varies. The present study evaluated the diagnostic validity of 17 embedded validity indices by comparing performance on these indices to performance on combinations of free-standing SVTs. Results from the current medico-legal sample revealed that of the embedded validity indices, Reliable Digit Span had the best classification accuracy; however, the findings do not support the use of this embedded validity index in the absence of free-standing SVTs.

Polybrominated diphenyl ethers (PBDEs), polychlorinated biphenyls (PCBs) and neuropsychological status among older adults in New York.

Polybrominated diphenyl ethers (PBDEs) are emerging environmental contaminants, but little is known about their possible human health effects. The objective of this study is to evaluate the association between exposure to PBDEs and neuropsychological function among older adults and the possibility of effect modification with polychlorinated biphenyls (PCBs). Serum samples were analyzed for concentrations of 9 PBDE and 30 PCB congeners and 34 tests of cognitive and motor function, affective state, and olfactory function were assessed among 144 men and women of 55-74 years of age. After adjustment for relevant confounders, no overall associations were observed between the sum of the PBDE congener concentrations in serum (∑ PBDE) and scores on the neuropsychological tests. However, statistically significant interactions were found between PBDEs and PCBs for some measures of verbal learning and memory. Among persons with ∑ PCB concentrations at or above the median of 467ppb (lipid basis), an increase in ∑ PBDE concentrations from the 25th to 75th percentile was associated with decreases between 7% and 12% on scores for certain subscales of the California Verbal Learning Test. In contrast, no statistically significant associations were observed for PBDEs among persons with ∑ PCB levels below the median. The results suggest that PBDEs and PCBs may interact to affect verbal memory and learning among persons 55-74 years old. This is the first study to evaluate the neuropsychological effects of PBDEs in adults and the possibility of synergy with PCBs in humans.