Mobile toolbox (MTB) remote measures of executive function and processing speed: development and validation.

OBJECTIVE: The ability to remotely monitor cognitive skills is increasing with the ubiquity of smartphones. The Mobile Toolbox (MTB) is a new measurement system that includes measures assessing Executive Functioning (EF) and Processing Speed (PS): Arrow Matching, Shape-Color Sorting, and Number-Symbol Match. The purpose of this study was to assess their psychometric properties. METHOD: MTB measures were developed for smartphone administration based on constructs measured in the NIH Toolbox® (NIHTB). Psychometric properties of the resulting measures were evaluated in three studies with participants ages 18 to 90. In Study 1 (N = 92), participants completed MTB measures in the lab and were administered both equivalent NIH TB measures and other external measures of similar cognitive constructs. In Study 2 (N = 1,021), participants completed the equivalent NIHTB measures in the lab and then took the MTB measures on their own, remotely. In Study 3 (N = 168), participants completed MTB measures twice remotely, two weeks apart. RESULTS: All three measures exhibited very high internal consistency and strong test-retest reliability, as well as moderately high correlations with comparable NIHTB tests and moderate correlations with external measures of similar constructs. Phone operating system (iOS vs. Android) had a significant impact on performance for Arrow Matching and Shape-Color Sorting, but no impact on either validity or reliability. CONCLUSIONS: Results support the reliability and convergent validity of MTB EF and PS measures for use across the adult lifespan in remote, self-administered designs.

Development and validation of an episodic memory measure in the Mobile Toolbox (MTB): Arranging Pictures.

INTRODUCTION: Arranging Pictures is a new episodic memory test based on the NIH Toolbox (NIHTB) Picture Sequence Memory measure and optimized for self-administration on a personal smartphone within the Mobile Toolbox (MTB). We describe evidence from three distinct validation studies. METHOD: In Study 1, 92 participants self-administered Arranging Pictures on study-provided smartphones in the lab and were administered external measures of similar and dissimilar constructs by trained examiners to assess validity under controlled circumstances. In Study 2, 1,021 participants completed the external measures in the lab and self-administered Arranging Pictures remotely on their personal smartphones to assess validity in real-world contexts. In Study 3, 141 participants self-administered Arranging Pictures remotely twice with a two-week delay on personal iOS smartphones to assess test-retest reliability and practice effects. RESULTS: Internal consistency was good across samples (ρxx = .80 to .85, p < .001). Test-retest reliability was marginal (ICC = .49, p < .001) and there were significant practice effects after a two-week delay (ΔM = 3.21 (95% CI [2.56, 3.88]). As expected, correlations with convergent measures were significant and moderate to large in magnitude (ρ = .44 to .76, p < .001), while correlations with discriminant measures were small (ρ = .23 to .27, p < .05) or nonsignificant. Scores demonstrated significant negative correlations with age (ρ = -.32 to -.21, p < .001). Mean performance was slightly higher in the iOS compared to the Android group (MiOS = 18.80, NiOS = 635; MAndroid = 17.11, NAndroid = 386; t(757.73) = 4.17, p < .001), but device type did not significantly influence the psychometric properties of the measure. Indicators of potential cheating were mixed; average scores were significantly higher in the remote samples (F(2, 850) = 11.415, p < .001), but there were not significantly more perfect scores. CONCLUSION: The MTB Arranging Pictures measure demonstrated evidence of reliability and validity when self-administered on personal device. Future research should examine the potential for cheating in remote settings and the properties of the measure in clinical samples.

Development and Validation of a Vocabulary Measure in the Mobile Toolbox.

OBJECTIVE: We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies. METHOD: Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones. RESULTS: The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test-retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67-0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001). CONCLUSION: The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.

Exploring symptom clusters in mild cognitive impairment and dementia with the NIH Toolbox.

OBJECTIVE: Symptom clustering research provides a unique opportunity for understanding complex medical conditions. The objective of this study was to apply a variable-centered analytic approach to understand how symptoms may cluster together, within and across domains of functioning in mild cognitive impairment (MCI) and dementia, to better understand these conditions and potential etiological, prevention, and intervention considerations. METHOD: Cognitive, motor, sensory, emotional, and social measures from the NIH Toolbox were analyzed using exploratory factor analysis (EFA) from a dataset of 165 individuals with a research diagnosis of either amnestic MCI or dementia of the Alzheimer's type. RESULTS: The six-factor EFA solution described here primarily replicated the intended structure of the NIH Toolbox with a few deviations, notably sensory and motor scores loading onto factors with measures of cognition, emotional, and social health. These findings suggest the presence of cross-domain symptom clusters in these populations. In particular, negative affect, stress, loneliness, and pain formed one unique symptom cluster that bridged the NIH Toolbox domains of physical, social, and emotional health. Olfaction and dexterity formed a second unique cluster with measures of executive functioning, working memory, episodic memory, and processing speed. A third novel cluster was detected for mobility, strength, and vision, which was considered to reflect a physical functioning factor. Somewhat unexpectedly, the hearing test included did not load strongly onto any factor. CONCLUSION: This research presents a preliminary effort to detect symptom clusters in amnestic MCI and dementia using an existing dataset of outcome measures from the NIH Toolbox.

Identifying Health-Related Quality of Life Domains After Upper Extremity Transplantation.

OBJECTIVE: To identify the most important health-related quality of life (HRQOL) domains and patient-reported outcomes after upper extremity transplantation (UET) in individuals with upper extremity amputation. DESIGN: Verbatim audio-recordings of individual interviews and focus groups were analyzed using qualitative, grounded theory-based methods to identify important domains of HRQOL and provide guidance for outcomes measurement after UET. SETTING: Individual interviews were conducted by phone. Focus groups were conducted at 5 upper extremity vascularized composite allotransplantation (VCA) centers in the US and at an international conference of VCA experts. PARTICIPANTS: Individual phone interviews were conducted with 5 individuals with lived experience of UET. Thirteen focus groups were conducted with a total of 59 clinical professionals involved in UET. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Not applicable. RESULTS: Twenty-eight key HRQOL domains were identified, including physical functioning and medical complications, positive and negative emotional functioning, and social participation, relations, and independence. We identified key constructs for use in evaluation of the potentially substantial physical, medical, social, and emotional effects of UET. CONCLUSIONS: This study provides an overview of the most important issues affecting HRQOL after UET, including several topics that are unique to individuals with UET. This information will be used to establish systematic, comprehensive, and longitudinal measurement of post-UET HRQOL outcomes.

Validation of a tablet-based assessment of auditory sensitivity for researchers.

Quantifying hearing acuity is increasingly important across a wide range of research areas in the behavioral and neurosciences. Scientists have relied on either self-reported hearing status or the availability of diagnostic hearing assessment in past studies. There remains a need for a valid and reliable assessment of auditory sensitivity that can provide estimates of the magnitude of hearing loss, if present, without requirements for professional audiologists, facilities, and equipment that are needed to conduct a diagnostic hearing assessment. The goal of this experiment was to validate the NIH Toolbox® Hearing Threshold Test (HTT), a tablet-based hearing assessment available via iPad application that uses consumer-grade headphones, on a clinical sample of children and adults with varying degrees of hearing acuity. Electroacoustic analysis of the hearing assessment application and headphones demonstrated acoustic outputs within established conformity standards for hearing assessment. Twenty-seven children and 63 adults participated in a standard diagnostic hearing assessment and the experimental tablet-based assessment. The results showed that thresholds from the tablet-based assessment were highly correlated with thresholds from the clinical hearing assessment (r = .83-.93) for children and adults for all frequencies and across a range of levels of hearing acuity. The HTT also met clinical test-retest reliability standards (Cronbach's α > .86). The tablet-based hearing assessment provides acceptable estimates of hearing levels for children and adults when diagnostic audiometric assessment capabilities are not available.

Designing a Visit Schedule for Longitudinal Studies in Pediatric Research.

A challenge for longitudinal studies is combining individual assessments into visits that are scientifically logical, not burdensome for participants, well-choreographed, and operationally feasible. The visits then need to be sequenced and spaced to address the scientific goals and generate a data archive that is sufficiently robust and well-documented to support subsequent analyses. This paper summarizes comprehensive multi-disciplinary activities that were coordinated to design the content, format, and structure of the National Children's Study and concurrently serve as a model and resource for other studies.

ARMADA: Assessing reliable measurement in Alzheimer's disease and cognitive aging project methods.

INTRODUCTION: Early detection of cognitive decline in older adults is a public health priority. Advancing Reliable Measurement in Alzheimer's Disease and Cognitive Aging (ARMADA), a multisite study, is validating cognition, emotion, motor, and sensory modules of the National Institutes of Health Toolbox for Assessment of Neurological and Behavioral Function (NIHTB) in the aging spectrum from cognitively normal to dementia of the Alzheimer's type (DAT). METHODS: Participants 65 to 85 years old, in demographic groups racially proportional to the general US population, are recruited in one of three groups to validate the NIHTB: cognitively normal, amnestic mild cognitive impairment (aMCI), or mild DAT. Additional special emphasis cohorts include (1) Blacks in the three clinical groups; (2) Spanish-speakers in the three clinical groups; (3) cognitively normal, population-proportional, over age 85. DISCUSSION: Longitudinal study will determine whether NIHTB can predict cognitive decline and is associated with Alzheimer's disease biomarkers. Here, we detail the methods for the ARMADA study.

Assessment of quality of life after upper extremity transplantation: Framework for patient-reported outcome scale domains.

Upper extremity transplantation offers the promise of restored function and regained quality of life (QOL) for individuals who have sustained hand or arm amputation. However, a major challenge for this procedure becoming an accessible treatment option for patients is the lack of standard measures to document benefits to QOL. Patient-reported outcomes (PRO) measures are well-suited for this kind of intervention, where the perspective of the patient is central to defining treatment success. To date, qualitative work with experts, clinicians, and patients has been used to identify the most important domains of QOL for PRO item development. Specifically, our group's qualitative work has identified several domains of QOL that are unique to individuals who have received upper extremity transplants, which are distinct from topics covered by existing PRO measures. These include emotional and social aspects of upper extremity transplant, such as Expectations and Perceived Outcomes, Integration and Assimilation of Transplant, Fitting in, and Post-Surgical Challenges and Complications. The broad topic of Satisfaction with Transplant was subdivided into three subtopics: Function, Sensation, and Aesthetics. Satisfaction with Sensation was also identified as a unique domain not evaluated by existing PRO measures. This report operationalizes these eight QOL domains by presenting scoping definitions. This manuscript describes the work that has been completed for domain characterization as an early step toward developing standardized PRO measures to evaluate these important outcomes specific to upper extremity transplantation.

Measurement of Cognition for the National Children's Study.

The National Children's Study Cognitive Health Domain Team developed detailed plans for assessing cognition longitudinally from infancy to early adulthood. These plans identify high-priority aspects of cognition that can be measured efficiently and effectively, and we believe they can serve as a model for future large-scale longitudinal research. For infancy and toddlerhood, we proposed several paradigms that collectively allowed us to assess six broad cognitive constructs: (1) executive function skills, (2) episodic memory, (3) language, (4) processing speed, (5) spatial and numerical processing, and (6) social cognition. In some cases, different trial sequences within a paradigm allow for the simultaneous assessment of multiple cognitive skills (e.g., executive function skills and processing speed). We define each construct, summarize its significance for understanding developmental outcomes, discuss the feasibility of its assessment throughout development, and present our plan for measuring specific skills at different ages. Given the need for well-validated, direct behavioral measures of cognition that can be used in large-scale longitudinal studies, especially from birth to age 3 years, we also initiated three projects focused on the development of new measures.

NIH Toolbox executive function measures with developmental extensions: Reliability and validity with preschoolers in emergency housing.

Research has shown that executive function (EF) skills are associated with resilience in preschoolers experiencing risk and adversity, but these studies have typically relied on large batteries of tasks to measure children's EF skills. There is a need for brief, reliable EF assessments that can be used in the field with diverse young children. The current study assessed the validity and test-retest reliability of two tablet-based EF tasks from the NIH Toolbox: The Dimensional Change Card Sort (DCCS) and the Flanker Inhibitory Control and Attention Test, each with a developmental extension (Dext) that is triggered when a child struggles with the standardized versions. Dext versions include easier levels intended to improve task accessibility for younger and disadvantaged children. Eighty-six preschoolers residing in emergency housing participated in two study sessions about one week apart, completing tablet-based DCCS-Dext and Flanker-Dext tasks, along with a table-top EF task (Peg-Tapping) and measures of vocabulary and numeracy. The majority of participants triggered the Dext portion of the DCCS and almost half triggered the Dext portion of the Flanker, underscoring the need for extensions of the Toolbox EF tasks to lower the floor of these measures. The Dext EF measures were positively associated with Peg-Tapping, after controlling for age and vocabulary, indicating construct validity. They were also correlated with math achievement, suggesting criterion validity. DCCS-Dext and Flanker-Dext showed moderate test-retest reliability after one week. Together, these findings demonstrate the value of developmental extensions for assessing EF skills among children experiencing risk and adversity.

Reliability and Validity of the Spanish-Language Version of the NIH Toolbox.

The psychometric properties of the English-language NIH Toolbox for Assessment of Neurological and Behavioral Function (NIH Toolbox) have been examined in numerous populations. This study evaluated the reliability and validity of the Spanish-language NIH Toolbox. Participants were children aged 3 to 7 years and adults aged 18 to 85 years who took part in the NIH Toolbox norming study in Spanish. Results supported the internal consistency reliability of included measures. Test-retest reliability was strong for most tests, though it was weaker for the test of olfaction among children and the test of locomotion among adults. Spearman's correlations and general linear models showed Spanish tests were often associated with age, sex, and education. Convergent validity for the two language measures that underwent more intensive development, evaluated via Spearman's correlations with legacy measures, was strong. Results support using the Spanish-language NIH Toolbox to measure neurological and behavioral functioning among Spanish-speaking individuals in the United States.

The NIH Toolbox: Overview of Development for Use with Hispanic Populations.

OBJECTIVE: Hispanics/Latinos are the largest and fastest-growing minority population in the United States. To facilitate appropriate outcome assessment of this expanding population, the NIH Toolbox for Assessment of Neurological and Behavioral Function® (NIH Toolbox®) was developed with particular attention paid to the cultural and linguistic needs of English- and Spanish-speaking Hispanics/Latinos. METHODS: A Cultural Working Group ensured that all included measures were appropriate for use with Hispanics/Latinos in both English and Spanish. In addition, a Spanish Language Working Group assessed all English-language NIH Toolbox measures for translatability. RESULTS: Measures were translated following the Functional Assessment of Chronic Illness Therapy (FACIT) translation methodology for instances where language interpretation could impact scores, or a modified version thereof for more simplified translations. The Spanish versions of the NIH Toolbox Cognition Battery language measures (i.e., Picture Vocabulary Test, Oral Reading Recognition Test) were developed independently of their English counterparts. CONCLUSIONS: The Spanish-language version of the NIH Toolbox provides a much-needed set of tools that can be selected as appropriate to complement existing protocols being conducted with the growing Hispanic/Latino population in the United States.

Nih toolbox premorbid ability adjustments: Application in a traumatic brain injury sample.

PURPOSE/OBJECTIVE: Metrics to estimate premorbid cognitive ability, such as word reading tests, are important for clinical determination of cognitive changes following brain injury. In the present study, reading adjusted scores for the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) fluid tests were developed and validated with a sample of individuals with traumatic brain injury (TBI), to evaluate the clinical utility of reading-adjusted scores. Research Method/Design: The development sample included 843 adult participants, ages 20-85, from the NIHTB-CB standardization sample. A sample of 158 participants with complicated mild or moderate TBI (n = 74) or severe TBI (n = 84) were administered the NIHTB-CB, and comprised the validation sample. Scores were derived for the five fluid tests using four adjustment models: age-only, demographic-only, age-and-reading, and demographic-and-reading referenced scores. RESULTS: Estimated premorbid ability varies depending on the reference model. Scores from each of the four reference models differentiated the comparison and TBI samples at the group level. However, performance varied by premorbid ability. CONCLUSIONS/IMPLICATIONS: Premorbid ability affects identification of cognitive difficulties after TBI. Reading referenced scores provide an individualized estimate of the effects of premorbid ability than demographic characteristics alone. Each model identified a similar number of individuals as having cognitive difficulties; however, the models differed on which individuals had cognitive difficulties. The models had higher disagreement rates in the clinical compared with the comparison sample, particularly for individuals with lower premorbid ability. Clinical use and caveats are discussed. (PsycINFO Database Record

Interpreting Patterns of Low Scores on the NIH Toolbox Cognition Battery.

INTRODUCTION: The National Institutes of Health Toolbox for Assessment of Neurological and Behavioral Function Cognition Battery is comprised of seven cognitive tests, including two tests measuring crystallized cognitive ability (i.e., vocabulary and reading) and five tests measuring fluid cognitive functioning (i.e., working memory, memory, speed of processing, and executive functioning). This study presents comprehensive base rate tables for the frequency of low scores in adults and older adults from the normative sample. METHODS: Participants were 843 adults, ages 20-85, from the NIH Toolbox standardization sample who completed all seven cognition tests. Rates of low scores were derived for standard age-adjusted and fully-demographically-adjusted scores at multiple cut-scores. Base rates were stratified by education, crystallized intellectual ability, and cognitive domain. RESULTS: Using the five demographically-adjusted fluid cognitive test scores, 45.9% of adults obtained one or more scores at or below the 16th percentile, and 16.8% obtained one or more score at or below the 5th percentile, which is consistent with findings from other neurocognitive test batteries. DISCUSSION: Based on the study findings, nearly 50% of adults in the general population would meet psychometric criteria for a diagnosis of the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5) Mild Neurocognitive Disorder (MND). We developed new psychometric criteria for identifying MND using the NIH Toolbox Cognition Battery that reduce the false positive rate. Knowing these multivariate normative base rates will help researchers and clinicians interpret NIH Toolbox scores in people with neurodevelopmental, psychiatric, medical, neurological, and neurodegenerative disorders that affect cognitive functioning.

Demographically Corrected Normative Standards for the Spanish Language Version of the NIH Toolbox Cognition Battery.

Hispanics are the fastest growing ethnicity in the United States, yet there are limited well-validated neuropsychological tools in Spanish, and an even greater paucity of normative standards representing this population. The Spanish NIH Toolbox Cognition Battery (NIHTB-CB) is a novel neurocognitive screener; however, the original norms were developed combining Spanish- and English-versions of the battery. We developed normative standards for the Spanish NIHTB-CB, fully adjusting for demographic variables and based entirely on a Spanish-speaking sample. A total of 408 Spanish-speaking neurologically healthy adults (ages 18-85 years) and 496 children (ages 3-7 years) completed the NIH Toolbox norming project. We developed three types of scores: uncorrected based on the entire Spanish-speaking cohort, age-corrected, and fully demographically corrected (age, education, sex) scores for each of the seven NIHTB-CB tests and three composites (Fluid, Crystallized, Total Composites). Corrected scores were developed using polynomial regression models. Demographic factors demonstrated medium-to-large effects on uncorrected NIHTB-CB scores in a pattern that differed from that observed on the English NIHTB-CB. For example, in Spanish-speaking adults, education was more strongly associated with Fluid scores, but showed the strongest association with Crystallized scores among English-speaking adults. Demographic factors were no longer associated with fully corrected scores. The original norms were not successful in eliminating demographic effects, overestimating children's performances, and underestimating adults' performances on the Spanish NIHTB-CB. The disparate pattern of demographic associations on the Spanish versus English NIHTB-CB supports the need for distinct normative standards developed separately for each population. Fully adjusted scores presented here will aid in more accurately characterizing acquired brain dysfunction among U.S. Spanish-speakers.

Demographically Corrected Normative Standards for the English Version of the NIH Toolbox Cognition Battery.

Demographic factors impact neuropsychological test performances and accounting for them may help to better elucidate current brain functioning. The NIH Toolbox Cognition Battery (NIHTB-CB) is a novel neuropsychological tool, yet the original norms developed for the battery did not adequately account for important demographic/cultural factors known to impact test performances. We developed norms fully adjusting for all demographic variables within each language group (English and Spanish) separately. The current study describes the standards for individuals tested in English. Neurologically healthy adults (n=1038) and children (n=2917) who completed the NIH Toolbox norming project in English were included. We created uncorrected scores weighted to the 2010 Census demographics, and applied polynomial regression models to develop age-corrected and fully demographically adjusted (age, education, sex, race/ethnicity) scores for each NIHTB-CB test and composite (i.e., Fluid, Crystallized, and Total Composites). On uncorrected NIHTB-CB scores, age and education demonstrated significant, medium-to-large associations, while sex showed smaller, but statistically significant effects. In terms of race/ethnicity, a significant stair-step effect on uncorrected NIHTB-CB scores was observed (African American

Normative Scores for the NIH Toolbox Dynamic Visual Acuity Test from 3 to 85 Years.

As part of the National Institutes of Health Toolbox initiative, a computerized test of dynamic visual acuity (cDVA) was developed and validated as an easy-to-administer, cost- and time-efficient test of vestibular and visual function. To establish normative reference values, 3,992 individuals, aged 3-85 years, without vestibular pathology underwent cDVA testing at multiple clinical research testing facilities across the United States. Test scores were stratified by sociodemographic characteristics. cDVA was worse in males (p < 0.001) and those subjects 50 years or older, while there was no difference in dynamic visual acuity across age groups binned from 3 to 49 years. Furthermore, we used these normative cDVA data as a criterion reference to compare both the long (validated) and short versions of the test. Both versions can distinguish between those with and without vestibular pathology (p = 0.0002 long; p = 0.0025 short). The intraclass correlation coefficient between long- and short-cDVA tests was 0.86.

The cognition battery of the NIH toolbox for assessment of neurological and behavioral function: validation in an adult sample.

This study introduces a special series on validity studies of the Cognition Battery (CB) from the U.S. National Institutes of Health Toolbox for the Assessment of Neurological and Behavioral Function (NIHTB) (Gershon, Wagster et al., 2013) in an adult sample. This first study in the series describes the sample, each of the seven instruments in the NIHTB-CB briefly, and the general approach to data analysis. Data are provided on test-retest reliability and practice effects, and raw scores (mean, standard deviation, range) are presented for each instrument and the gold standard instruments used to measure construct validity. Accompanying papers provide details on each instrument, including information about instrument development, psychometric properties, age and education effects on performance, and convergent and discriminant construct validity. One study in the series is devoted to a factor analysis of the NIHTB-CB in adults and another describes the psychometric properties of three composite scores derived from the individual measures representing fluid and crystallized abilities and their combination. The NIHTB-CB is designed to provide a brief, comprehensive, common set of measures to allow comparisons among disparate studies and to improve scientific communication.