Mobile toolbox (MTB) remote measures of executive function and processing speed: development and validation.

OBJECTIVE: The ability to remotely monitor cognitive skills is increasing with the ubiquity of smartphones. The Mobile Toolbox (MTB) is a new measurement system that includes measures assessing Executive Functioning (EF) and Processing Speed (PS): Arrow Matching, Shape-Color Sorting, and Number-Symbol Match. The purpose of this study was to assess their psychometric properties. METHOD: MTB measures were developed for smartphone administration based on constructs measured in the NIH Toolbox® (NIHTB). Psychometric properties of the resulting measures were evaluated in three studies with participants ages 18 to 90. In Study 1 (N = 92), participants completed MTB measures in the lab and were administered both equivalent NIH TB measures and other external measures of similar cognitive constructs. In Study 2 (N = 1,021), participants completed the equivalent NIHTB measures in the lab and then took the MTB measures on their own, remotely. In Study 3 (N = 168), participants completed MTB measures twice remotely, two weeks apart. RESULTS: All three measures exhibited very high internal consistency and strong test-retest reliability, as well as moderately high correlations with comparable NIHTB tests and moderate correlations with external measures of similar constructs. Phone operating system (iOS vs. Android) had a significant impact on performance for Arrow Matching and Shape-Color Sorting, but no impact on either validity or reliability. CONCLUSIONS: Results support the reliability and convergent validity of MTB EF and PS measures for use across the adult lifespan in remote, self-administered designs.

Looking Ahead: Advancing Measurement and Analysis of the Block Design Test Using Technology and Artificial Intelligence.

The block design test (BDT) has been used for over a century in research and clinical contexts as a measure of spatial cognition, both as a singular ability and as part of more comprehensive intelligence assessment. Traditionally, the BDT has been scored using methods that do not reflect the full potential of individual differences that could be measured by the test. Recent advancements in technology, including eye-tracking, embedded sensor systems, and artificial intelligence, have provided new opportunities to measure and analyze data from the BDT. In this methodological review, we outline the information that BDT can assess, review several recent advancements in measurement and analytic methods, discuss potential future uses of these methods, and advocate for further research using these methods.

Development and validation of an episodic memory measure in the Mobile Toolbox (MTB): Arranging Pictures.

INTRODUCTION: Arranging Pictures is a new episodic memory test based on the NIH Toolbox (NIHTB) Picture Sequence Memory measure and optimized for self-administration on a personal smartphone within the Mobile Toolbox (MTB). We describe evidence from three distinct validation studies. METHOD: In Study 1, 92 participants self-administered Arranging Pictures on study-provided smartphones in the lab and were administered external measures of similar and dissimilar constructs by trained examiners to assess validity under controlled circumstances. In Study 2, 1,021 participants completed the external measures in the lab and self-administered Arranging Pictures remotely on their personal smartphones to assess validity in real-world contexts. In Study 3, 141 participants self-administered Arranging Pictures remotely twice with a two-week delay on personal iOS smartphones to assess test-retest reliability and practice effects. RESULTS: Internal consistency was good across samples (ρxx = .80 to .85, p < .001). Test-retest reliability was marginal (ICC = .49, p < .001) and there were significant practice effects after a two-week delay (ΔM = 3.21 (95% CI [2.56, 3.88]). As expected, correlations with convergent measures were significant and moderate to large in magnitude (ρ = .44 to .76, p < .001), while correlations with discriminant measures were small (ρ = .23 to .27, p < .05) or nonsignificant. Scores demonstrated significant negative correlations with age (ρ = -.32 to -.21, p < .001). Mean performance was slightly higher in the iOS compared to the Android group (MiOS = 18.80, NiOS = 635; MAndroid = 17.11, NAndroid = 386; t(757.73) = 4.17, p < .001), but device type did not significantly influence the psychometric properties of the measure. Indicators of potential cheating were mixed; average scores were significantly higher in the remote samples (F(2, 850) = 11.415, p < .001), but there were not significantly more perfect scores. CONCLUSION: The MTB Arranging Pictures measure demonstrated evidence of reliability and validity when self-administered on personal device. Future research should examine the potential for cheating in remote settings and the properties of the measure in clinical samples.

Protocol for a construct and clinical validation study of MyCog Mobile: a remote smartphone-based cognitive screener for older adults.

INTRODUCTION: Annual cognitive screening in older adults is essential for early detection of cognitive impairment, yet primary care settings face time constraints that present barriers to routine screening. A remote cognitive screener completed on a patient's personal smartphone before a visit has the potential to save primary care clinics time, encourage broader screening practices and increase early detection of cognitive decline. MyCog Mobile is a promising new remote smartphone-based cognitive screening app for primary care settings. We propose a combined construct and clinical validation study of MyCog Mobile. METHODS AND ANALYSIS: We will recruit a total sample of 300 adult participants aged 65 years and older. A subsample of 200 healthy adult participants and a subsample of 100 adults with a cognitive impairment diagnosis (ie, dementia, mild cognitive impairment, cognitive deficits or other memory loss) will be recruited from the general population and specialty memory care centres, respectively. To evaluate the construct validity of MyCog Mobile, the healthy control sample will self-administer MyCog Mobile on study-provided smartphones and be administered a battery of gold-standard neuropsychological assessments. We will compare correlations between performance on MyCog Mobile and measures of similar and dissimilar constructs to evaluate convergent and discriminant validity. To assess clinical validity, participants in the clinical sample will self-administer MyCog Mobile on a smartphone and be administered a Mini-Cog screener and these data will be combined with the healthy control sample. We will then apply several supervised model types to determine the best predictors of cognitive impairment within the sample. Area under the receiver operating characteristic curve, accuracy, sensitivity and specificity will be the primary performance metrics for clinical validity. ETHICS AND DISSEMINATION: The Institutional Review Board at Northwestern University (STU00214921) approved this study protocol. Results will be published in peer-reviewed journals and summaries provided to the study's funders.

Development and Validation of a Vocabulary Measure in the Mobile Toolbox.

OBJECTIVE: We describe the development of a new computer adaptive vocabulary test, Mobile Toolbox (MTB) Word Meaning, and validity evidence from 3 studies. METHOD: Word Meaning was designed to be a multiple-choice synonym test optimized for self-administration on a personal smartphone. The items were first calibrated online in a sample of 7,525 participants to create the computer-adaptive test algorithm for the Word Meaning measure within the MTB app. In Study 1, 92 participants self-administered Word Meaning on study-provided smartphones in the lab and were administered external measures by trained examiners. In Study 2, 1,021 participants completed the external measures in the lab and Word Meaning was self-administered remotely on their personal smartphones. In Study 3, 141 participants self-administered Word Meaning remotely twice with a 2-week delay on personal iPhones. RESULTS: The final bank included 1363 items. Internal consistency was adequate to good across samples (ρxx = 0.78 to 0.81, p < .001). Test-retest reliability was good (ICC = 0.65, p < .001), and the mean theta score was not significantly different upon the second administration. Correlations were moderate to large with measures of similar constructs (ρ = 0.67-0.75, p < .001) and non-significant with measures of dissimilar constructs. Scores demonstrated small to moderate correlations with age (ρ = 0.35 to 0.45, p < .001) and education (ρ = 0.26, p < .001). CONCLUSION: The MTB Word Meaning measure demonstrated evidence of reliability and validity in three samples. Further validation studies in clinical samples are necessary.

Remote Self-Administration of Cognitive Screeners for Older Adults Prior to a Primary Care Visit: Pilot Cross-Sectional Study of the Reliability and Usability of the MyCog Mobile Screening App.

BACKGROUND: Routine cognitive screening is essential in the early detection of dementia, but time constraints in primary care settings often limit clinicians' ability to conduct screenings. MyCog Mobile is a newly developed cognitive screening system that patients can self-administer on their smartphones before a primary care visit, which can help save clinics' time, encourage broader screening practices, and increase early detection of cognitive decline. OBJECTIVE: The goal of this pilot study was to examine the feasibility, acceptability, and initial psychometric properties of MyCog Mobile. Research questions included (1) Can older adults complete MyCog Mobile remotely without staff support? (2) Are the internal consistency and test-retest reliability of the measures acceptable? and (3) How do participants rate the user experience of MyCog Mobile? METHODS: A sample of adults aged 65 years and older (N=51) self-administered the MyCog Mobile measures remotely on their smartphones twice within a 2- to 3-week interval. The pilot version of MyCog Mobile includes 4 activities: MyFaces measures facial memory, MySorting measures executive functioning, MySequences measures working memory, and MyPictures measures episodic memory. After their first administration, participants also completed a modified version of the Simplified System Usability Scale (S-SUS) and 2 custom survey items. RESULTS: All participants in the sample passed the practice items and completed each measure. Findings indicate that the Mobile Toolbox assessments measure the constructs well (internal consistency 0.73 to 0.91) and are stable over an approximately 2-week delay (test-retest reliability 0.61 to 0.71). Participants' rating of the user experience (mean S-SUS score 73.17, SD 19.27) indicated that older adults found the usability of MyCog Mobile to be above average. On free-response feedback items, most participants provided positive feedback or no feedback at all, but some indicated a need for clarity in certain task instructions, concerns about participants' abilities, desire to be able to contact a support person or use in-app technical support, and desire for additional practice items. CONCLUSIONS: Pilot evidence suggests that the MyCog Mobile cognitive screener can be reliably self-administered by older adults on their smartphones. Participants in our study generally provided positive feedback about the MyCog Mobile experience and rated the usability of the app highly. Based on participant feedback, we will conduct further usability research to improve support functionality, optimize task instructions and practice opportunities, and ensure that patients feel comfortable using MyCog Mobile. The next steps include a clinical validation study that compares MyCog Mobile to gold-standard assessments and tests the sensitivity and specificity of the measures for identifying dementia.

Using the Mobile Toolbox in child and adolescent samples: A feasibility study.

Cognitive research with developmental samples requires improved methods that support large-scale, diverse, and open science. This paper offers initial evidence to support the Mobile Toolbox (MTB), a self-administered remote smartphone-based cognitive battery, in youth populations, from a pilot sample of 99 children (Mage = 11.79 years; 36% female; 53% White, 33% Black or African American, 9% Asian, and 15% Hispanic). Completion rates (95%-99%), practice performance (96%-100%), internal consistency (0.60-0.98), and correlations with similar NIHTB measures (0.55-0.77) provide the first evidence to support the MTB in a youth sample, although there were some inconsistencies across measures. Preliminary findings provide promising evidence of the MTB in developmental populations, and further studies are encouraged.

Remote Cognitive Screening Of Healthy Older Adults for Primary Care With the MyCog Mobile App: Iterative Design and Usability Evaluation.

BACKGROUND: Annual cognitive screening in adults aged >65 years can improve early detection of cognitive impairment, yet less than half of all cases are identified in primary care. Time constraints in primary care settings present a major barrier to routine screening. A remote cognitive screener completed on a patient's own smartphone before a visit has the potential to save primary care clinics time, encourage broader screening practices, and increase early detection of cognitive decline. OBJECTIVE: We described the iterative design and proposed the implementation of a remote cognitive screening app, MyCog Mobile, to be completed on a patient's smartphone before an annual wellness visit. The research questions were as follows: What would motivate primary care clinicians and clinic administrators to implement a remote cognitive screening process? How might we design a remote cognitive screener to fit well with existing primary care workflows? What would motivate an older adult patient to complete a cognitive screener on a smartphone before a primary care visit? How might we optimize the user experience of completing a remote cognitive screener on a smartphone for older adults? METHODS: To address research questions 1 and 2, we conducted individual interviews with clinicians (n=5) and clinic administrators (n=3). We also collaborated with clinic administrators to create user journey maps of their existing and proposed MyCog Mobile workflows. To address research questions 3 and 4, we conducted individual semistructured interviews with cognitively healthy older adults (n=5) and solicited feedback from a community stakeholder panel (n=11). We also tested and refined high-fidelity prototypes of the MyCog Mobile app with the older adult interview participants, who rated the usability on the Simplified System Usability Scale and After-Scenario Questionnaire. RESULTS: Clinicians and clinic administrators were motivated to adopt a remote cognitive screening process if it saved time in their workflows. Findings from interviews and user journey mapping informed the proposed implementation and core functionality of MyCog Mobile. Older adult participants were motivated to complete cognitive screeners to ensure that they were cognitively healthy and saw additional benefits to remote screening, such as saving time during their visit and privacy. Older adults also identified potential challenges to remote smartphone screening, which informed the user experience design of the MyCog Mobile app. The average rating across prototype versions was 91 (SD 5.18) on the Simplified System Usability Scale and 6.13 (SD 8.40) on the After-Scenario Questionnaire, indicating above-average usability. CONCLUSIONS: Through an iterative, human-centered design process, we developed a viable remote cognitive screening app and proposed an implementation strategy for primary care settings that was optimized for multiple stakeholders. The next steps include validating the cognitive screener in clinical and healthy populations and piloting the finalized app in a community primary care clinic.

Format Effects of iPad Administration of Wechsler Adult Intelligence Scale-Fourth Edition: Cross-Sectional Evidence for Score Equivalency in Routine Clinical Practice.

OBJECTIVE: The literature lacks independent investigations of the influence of tablet administration of cognitive assessments in applied clinical settings. The present study examined the influence of iPad administration on (Wechsler Adult Intelligence Scale-Fourth Edition) WAIS-IV core subtest scores in a university-based clinic. METHOD: Record review was conducted for a convenience sample (N = 66) of university students who were administered the WAIS-IV via iPad or traditional format. Bayesian difference testing was used to evaluate the strength of the evidence for subtest score equivalence across groups. RESULTS: Evidence supported score equivalency for the 10 core subtests across administration groups (BF > 3). The one exception was digit span-forward condition, for which equivalence was supported (BF = 2.44), but did not meet cut-off criteria. CONCLUSIONS: iPad administration of WAIS-IV is unlikely to influence subtest scores in routine clinical practice with healthy young adults. Further independent research in diverse clinical populations is recommended.