Using virtual reality-based neurocognitive testing and eye tracking to study naturalistic cognitive-motor performance.

Natural human behavior arises from continuous interactions between the cognitive and motor domains. However, assessments of cognitive abilities are typically conducted using pen and paper tests, i.e., in isolation from "real life" cognitive-motor behavior and in artificial contexts. In the current study, we aimed to assess cognitive-motor task performance in a more naturalistic setting while recording multiple motor and eye tracking signals. Specifically, we aimed to (i) delineate the contribution of cognitive and motor components to overall task performance and (ii) probe for a link between cognitive-motor performance and pupil size. To that end, we used a virtual reality (VR) adaptation of a well-established neurocognitive test for executive functions, the 'Color Trails Test' (CTT). The VR-CTT involves performing 3D reaching movements to follow a trail of numbered targets. To tease apart the cognitive and motor components of task performance, we included two additional conditions: a condition where participants only used their eyes to perform the CTT task (using an eye tracking device), incurring reduced motor demands, and a condition where participants manually tracked visually-cued targets without numbers on them, incurring reduced cognitive demands. Our results from a group of 30 older adults (>65) showed that reducing cognitive demands shortened completion times more extensively than reducing motor demands. Conditions with higher cognitive demands had longer target search time, as well as decreased movement execution velocity and head-hand coordination. We found larger pupil sizes in the more cognitively demanding conditions, and an inverse correlation between pupil size and completion times across individuals in all task conditions. Lastly, we found a possible link between VR-CTT performance measures and clinical signatures of participants (fallers versus non-fallers). In summary, performance and pupil parameters were mainly dependent on task cognitive load, while maintaining systematic interindividual differences. We suggest that this paradigm opens the possibility for more detailed profiling of individual cognitive-motor performance capabilities in older adults and other at-risk populations.

Higher cognitive load interferes with head-hand coordination: virtual reality-based study.

Daily life activities often involve decision-based reaching movements in different contexts and circumstances. These activities span a wide array of cognitive load types we face while executing motor functions. Here we use a virtual reality-based neurocognitive testing platform to assess cognitive-induced changes in motor behavior as reflected by modulations in head-hand coordination. Our paradigm is based on the Color Trails Test (CTT), which is designed to assess two types of cognitive functions: Trails A-sustained visual attention (SVA), and Trails B-divided attention (DA). The virtual reality CTT adaptation (VR-CTT) requires execution of large multi-directional hand movements and head rotations. We employed a cross-correlation analysis on hand and head kinematics data collected from 122 healthy participants (ages: 20-90 years; divided as follows: young, middle-aged, and older adults) who completed the VR-CTT. The level of spatial coherence of head-hand movements was found to be high (R ≥ 0.76) in both Trails A and B, in all age groups. However, assessing head-hand phase shifts revealed longer time lags (i.e., in which head leads hand) in Trails B versus Trails A, in all age groups. We conclude that allocating cognitive resources to DA task reduces head-hand synchrony as compared to SVA conditions.

Studying cognitive-motor interactions using a tablet-based application of the Color Trails Test.

The Color Trails Test (CTT) is a pen and paper (P&P) test designed to measure cognitive function. The test consists of two parts that evaluate primarily sustained visual attention (Trails A) and divided attention (Trails B). Based on clinical interest in converting neuropsychological testing from P&P to computerized testing, we developed a digital version of the CTT ('Tablet-CTT'). Twenty-four young, healthy participants performed Trails A and B of both the original P&P and the Tablet-CTT. Hand kinematics were calculated using the continuous location of an electronic pen on the tablet touch screen. To compare motor control aspects, we differentiated for each test session the 'movement planning' and 'movement execution' times (accumulated across all single target-to-target trajectories). Concurrent validity was demonstrated by the high correlation between completion times of the P&P and Tablet-CTT, in both Trails A (r = 0.6; p < 0.005) and Trails B (r = 0.8; p < 0.001). Trails B yielded significantly longer completion times in both formats (p < 0.001). Examining hand kinematics in Tablet-CTT revealed that the difference in durations was mostly due to prolonged planning time, but also due to significantly lower execution velocity in Trails B (p < 0.001). Lastly, we found increased hand velocity during the planning phase in Trails B compared to Trails A (p < 0.001). This study demonstrates how transforming the CTT to a digital platform could be useful for studying cognitive-motor interactions in healthy individuals. Moreover, it could potentially serve as a diagnosis tool by introducing a more comprehensive testing method that incorporates online recordings of hand movements.

Virtual reality-based sensorimotor adaptation shapes subsequent spontaneous and naturalistic stimulus-driven brain activity.

Our everyday life summons numerous novel sensorimotor experiences, to which our brain needs to adapt in order to function properly. However, tracking plasticity of naturalistic behavior and associated brain modulations is challenging. Here, we tackled this question implementing a prism adaptation-like training in virtual reality (VRPA) in combination with functional neuroimaging. Three groups of healthy participants (N = 45) underwent VRPA (with a shift either to the left/right side, or with no shift), and performed functional magnetic resonance imaging (fMRI) sessions before and after training. To capture modulations in free-flowing, task-free brain activity, the fMRI sessions included resting-state and free-viewing of naturalistic videos. We found significant decreases in spontaneous functional connectivity between attentional and default mode (DMN)/fronto-parietal networks, only for the adaptation groups, more pronouncedly in the hemisphere contralateral to the induced shift. In addition, VRPA was found to bias visual responses to naturalistic videos: Following rightward adaptation, we found upregulation of visual response in an area in the parieto-occipital sulcus (POS) only in the right hemisphere. Notably, the extent of POS upregulation correlated with the size of the VRPA-induced after-effect measured in behavioral tests. This study demonstrates that a brief VRPA exposure can change large-scale cortical connectivity and correspondingly bias visual responses to naturalistic sensory inputs.

Developing multiple shortened forms of virtual reality-based color trails test.

The Color Trails Test (CTT) is a pencil-and-paper (P&P) neuropsychological test. The CTT is divided into two parts that assess sustained visual attention (Trails A) and divided attention (Trails B). The CTT can also be performed in a virtual reality setting (VR-CTT) introducing a wider spatial range of targets. In cases of multiple assessments, repeating the same CTT configuration can bias the results due to fatigue and learning effects. The aim of this study is to create five different short versions of the VR-CTT. The different forms were created by rotating or flipping the original targets spatial layout on one of the axes and by ending it at ball #13. Healthy young participants (N = 15) performed the shortened VR-CTT forms (in a counterbalanced order), the P&P CTT and the original VR-CTT. We found no difference between the completion times of the five forms (p > 0.2), and a significant difference between Trails A and B across all forms (p < 0.04). Additionally, there was no evidence of a learning effect between trials (p > 0.4). Moreover, the shortened VR-CTT forms showed correlations with the P&P CTT (p < 0.05) and with the original VR-CTT (p < 0.06). These findings suggest that all five forms have an equal level of difficulty and that the different forms managed to mitigate the learning effects reported for repeated testing of the same spatial layout. This opens the possibility of applying the shortened VR-CTT forms for research settings and sets the basis for developing it into a clinical diagnostics tool.

The trail less traveled: Analytical approach for creating shortened versions for virtual reality-based color trails test.

The Color Trails Test ("CTT") is among the most popular neuropsychological assessment tests of executive function, targeting sustained visual attention (Trails A), and divided attention (Trails B). During the pen-and-paper (P&P) test, the participant traces 25 consecutive numbered targets marked on a page, and the completion time is recorded. In many cases, multiple assessments are performed on the same individual, either under varying experimental conditions or at several timepoints. However, repeated testing often results in learning and fatigue effects, which confound test outcomes. To mitigate these effects, we set the grounds for developing shorter versions of the CTT (<25 targets), using virtual reality (VR) based CTT (VR-CTT). Our aim was to discover the minimal set of targets that is sufficient for maintaining concurrent validity with the CTT including differentiation between age groups, and the difference between Trails A and B. To this aim, healthy participants in three age groups (total N = 165; young, middle-aged, or older adults) performed both the P&P CTT, and one type of VR-CTT (immersive head-mounted-device VR, large-scale 3D VR, or tablet). A subset of 13 targets was highly correlated with overall task completion times in all age groups and platforms (r > 0.8). We tested construct validity and found that the shortened-CTT preserved differences between Trails A and B (p < 0.001), showed concurrent validity relative to the P&P scores (r > 0.5; p < 0.05), and differentiated between age groups (p < 0.05). These findings open the possibility for shortened "CTT-versions", to be used in repeated-measures experiments or longitudinal studies, with potential implications for shortening neurocognitive assessment protocols.

How do the blind 'see'? The role of spontaneous brain activity in self-generated perception.

Spontaneous activity of the human brain has been well documented, but little is known about the functional role of this ubiquitous neural phenomenon. It has previously been hypothesized that spontaneous brain activity underlies unprompted (internally generated) behaviour. We tested whether spontaneous brain activity might underlie internally-generated vision by studying the cortical visual system of five blind/visually-impaired individuals who experience vivid visual hallucinations (Charles Bonnet syndrome). Neural populations in the visual system of these individuals are deprived of external input, which may lead to their hyper-sensitization to spontaneous activity fluctuations. To test whether these spontaneous fluctuations can subserve visual hallucinations, the functional MRI brain activity of participants with Charles Bonnet syndrome obtained while they reported their hallucinations (spontaneous internally-generated vision) was compared to the: (i) brain activity evoked by veridical vision (externally-triggered vision) in sighted controls who were presented with a visual simulation of the hallucinatory streams; and (ii) brain activity of non-hallucinating blind controls during visual imagery (cued internally-generated vision). All conditions showed activity spanning large portions of the visual system. However, only the hallucination condition in the Charles Bonnet syndrome participants demonstrated unique temporal dynamics, characterized by a slow build-up of neural activity prior to the reported onset of hallucinations. This build-up was most pronounced in early visual cortex and then decayed along the visual hierarchy. These results suggest that, in the absence of external visual input, a build-up of spontaneous fluctuations in early visual cortex may activate the visual hierarchy, thereby triggering the experience of vision.

Combined virtual reality and haptic robotics induce space and movement invariant sensorimotor adaptation.

Prism adaptation is a method for studying visuomotor plasticity in healthy individuals, as well as for rehabilitating patients suffering spatial neglect. We developed a new set-up based on virtual-reality (VR) and haptic-robotics allowing us to induce sensorimotor adaptation and to reproduce the effect of prism adaptation in a more ecologically valid, yet experimentally controlled context. Participants were exposed to an immersive VR environment while controlling a virtual hand via a robotic-haptic device to reach virtual objects. During training, a rotational shift was induced between the position of the participant's real hand and that of the virtual hand in order to trigger sensorimotor recalibration. The use of VR and haptic-robotics allowed us to simulate and test multiple components of sensorimotor adaptation: training either peripersonal or extrapersonal space and testing generalization for the non-trained sector of space, and using active versus robot-guided reaching movements. Results from 60 neurologically intact participants show that participants exposed to the virtual shift were able to quickly adapt their reaching movements to aim correctly at the target objects. When the shift was removed, participants showed a systematic deviation of their movements during open-loop tasks in the direction opposite to that of the shift, which generalized to un-trained portions of space and occurred also when their movements were robotically-guided during the adaptation. Interestingly, follow-up questionnaires revealed that when the adaptation training was robotically-guided, participants were largely unaware of the mismatch between their hand and the virtual hand's position. The stability of the aftereffects, despite the changing experimental parameters, suggests that the induced sensory-motor adaptation does not rely on low-level processing of sensory stimuli during the training, but taps into high-level representations of space. Importantly, the flexibility of the trained space and the option of robotically-guided movements open novel possibilities of fine-tuning the training to patients' level of spatial and motor impairment, thus possibly resulting in a better outcome.

Prism adaptation enhances decoupling between the default mode network and the attentional networks.

Prism adaptation (PA) is a procedure used for studying visuomotor plasticity in healthy individuals, as well as for alleviating spatial neglect in patients. The adaptation is achieved by performing goal-directed movements while wearing prismatic lenses that induce a lateral displacement of visual information. This results in an initial movement error that is compensated by a recalibration of sensory-motor coordinates; consequently, a lateral bias in both motor and perceptual measurements occurs after prism removal, i.e., after effects. Neuroimaging studies have shown that a brief exposure to a rightward-shifting prism changes the activations in the inferior parietal lobule (IPL) and modulates interhemispheric balance during attention tasks. However, it is yet unknown how PA changes global interplay between cortical networks as evident from task-free resting state connectivity. Thus we compared resting state functional connectivity patterns before ('Pre') and after ('Post') participants performed a session of pointing movements with a rightward-shifting prism (N = 14) or with neutral lenses (as a control condition; N = 12). Global connectivity analysis revealed significant decreases in functional connectivity following PA in two nodes of the Default Mode Network (DMN), and in the left anterior insula. Further analyses of these regions showed specific connectivity decrease between either of the DMN nodes and areas within the attentional networks, including the inferior frontal gyrus, the anterior insula and the right superior temporal sulcus. On the other hand, the anterior insula decreased its connectivity to a large set of areas, all within the boundaries of the DMN. These results demonstrate that a brief exposure to PA enhances the decoupling between the DMN and the attention networks. The change in interplay between those pre-existing networks might be the basis of the rapid and wide-ranged behavioural changes induce by PA in healthy individuals.

Resting-State Activity in High-Order Visual Areas as a Window into Natural Human Brain Activations.

A major limitation of conventional human brain research has been its basis in highly artificial laboratory experiments. Due to technical constraints, little is known about the nature of cortical activations during ecological real life. We have previously proposed the "spontaneous trait reactivation (STR)" hypothesis arguing that resting-state patterns, which emerge spontaneously in the absence of external stimulus, reflect the statistics of habitual cortical activations during real life. Therefore, these patterns can serve as a window into daily life cortical activity. A straightforward prediction of this hypothesis is that spontaneous patterns should preferentially correlate to patterns generated by naturalistic stimuli compared with artificial ones. Here we targeted high-level category-selective visual areas and tested this prediction by comparing BOLD functional connectivity patterns formed during rest to patterns formed in response to naturalistic stimuli, as well as to more artificial category-selective, dynamic stimuli. Our results revealed a significant correlation between the resting-state patterns and functional connectivity patterns generated by naturalistic stimuli. Furthermore, the correlations to naturalistic stimuli were significantly higher than those found between resting-state patterns and those generated by artificial control stimuli. These findings provide evidence of a stringent link between spontaneous patterns and the activation patterns during natural vision.

Spontaneously Emerging Patterns in Human Visual Cortex Reflect Responses to Naturalistic Sensory Stimuli.

In the absence of stimulus or task, the cortex spontaneously generates rich and consistent functional connectivity patterns (termed resting state networks) which are evident even within individual cortical areas. We and others have previously hypothesized that habitual cortical network activations during daily life contribute to the shaping of these connectivity patterns. Here we tested this hypothesis by comparing, using blood oxygen level-dependent-functional magnetic resonance imaging, the connectivity patterns that spontaneously emerge during rest in retinotopic visual areas to the patterns generated by naturalistic visual stimuli (repeated movie segments). These were then compared with connectivity patterns produced by more standard retinotopic mapping stimuli (polar and eccentricity mapping). Our results reveal that the movie-driven patterns were significantly more similar to the spontaneously emerging patterns, compared with the connectivity patterns of either eccentricity or polar mapping stimuli. Intentional visual imagery of naturalistic stimuli was unlikely to underlie these results, since they were duplicated when participants were engaged in an auditory task. Our results suggest that the connectivity patterns that appear during rest better reflect naturalistic activations rather than controlled, artificially designed stimuli. The results are compatible with the hypothesis that the spontaneous connectivity patterns in human retinotopic areas reflect the statistics of cortical coactivations during natural vision.

Diminished Auditory Responses during NREM Sleep Correlate with the Hierarchy of Language Processing.

Natural sleep provides a powerful model system for studying the neuronal correlates of awareness and state changes in the human brain. To quantitatively map the nature of sleep-induced modulations in sensory responses we presented participants with auditory stimuli possessing different levels of linguistic complexity. Ten participants were scanned using functional magnetic resonance imaging (fMRI) during the waking state and after falling asleep. Sleep staging was based on heart rate measures validated independently on 20 participants using concurrent EEG and heart rate measurements and the results were confirmed using permutation analysis. Participants were exposed to three types of auditory stimuli: scrambled sounds, meaningless word sentences and comprehensible sentences. During non-rapid eye movement (NREM) sleep, we found diminishing brain activation along the hierarchy of language processing, more pronounced in higher processing regions. Specifically, the auditory thalamus showed similar activation levels during sleep and waking states, primary auditory cortex remained activated but showed a significant reduction in auditory responses during sleep, and the high order language-related representation in inferior frontal gyrus (IFG) cortex showed a complete abolishment of responses during NREM sleep. In addition to an overall activation decrease in language processing regions in superior temporal gyrus and IFG, those areas manifested a loss of semantic selectivity during NREM sleep. Our results suggest that the decreased awareness to linguistic auditory stimuli during NREM sleep is linked to diminished activity in high order processing stations.

Repetitive speech elicits widespread deactivation in the human cortex: the "Mantra" effect?

BACKGROUND: Mantra (prolonged repetitive verbal utterance) is one of the most universal mental practices in human culture. However, the underlying neuronal mechanisms that may explain its powerful emotional and cognitive impact are unknown. In order to try to isolate the effect of silent repetitive speech, which is used in most commonly practiced Mantra meditative practices, on brain activity, we studied the neuronal correlates of simple repetitive speech in nonmeditators - that is, silent repetitive speech devoid of the wider context and spiritual orientations of commonly practiced meditation practices. METHODS: We compared, using blood oxygenated level-dependent (BOLD) functional magnetic resonance imaging (fMRI), a simple task of covertly repeating a single word to resting state activity, in 23 subjects, none of which practiced meditation before. RESULTS: We demonstrate that the repetitive speech was sufficient to induce a widespread reduction in BOLD signal compared to resting baseline. The reduction was centered mainly on the default mode network, associated with intrinsic, self-related processes. Importantly, contrary to most cognitive tasks, where cortical-reduced activation in one set of networks is typically complemented by positive BOLD activity of similar magnitude in other cortical networks, the repetitive speech practice resulted in unidirectional negative activity without significant concomitant positive BOLD. A subsequent behavioral study showed a significant reduction in intrinsic thought processes during the repetitive speech condition compared to rest. CONCLUSIONS: Our results are compatible with a global gating model that can exert a widespread induction of negative BOLD in the absence of a corresponding positive activation. The triggering of a global inhibition by the minimally demanding repetitive speech may account for the long-established psychological calming effect associated with commonly practiced Mantra-related meditative practices.

Dissociating between object affordances and spatial compatibility effects using early response components.

PERCEPTION AND ACTION ARE TIGHTLY LINKED: objects may be perceived not only in terms of visual features, but also in terms of possibilities for action. Previous studies showed that when a centrally located object has a salient graspable feature (e.g., a handle), it facilitates motor responses corresponding with the feature's position. However, such so-called affordance effects have been criticized as resulting from spatial compatibility effects, due to the visual asymmetry created by the graspable feature, irrespective of any affordances. In order to dissociate between affordance and spatial compatibility effects, we asked participants to perform a simple reaction-time task to typically graspable and non-graspable objects with similar visual features (e.g., lollipop and stop sign). Responses were measured using either electromyography (EMG) on proximal arm muscles during reaching-like movements, or with finger key-presses. In both EMG and button press measurements, participants responded faster when the object was either presented in the same location as the responding hand, or was affordable, resulting in significant and independent spatial compatibility and affordance effects, but no interaction. Furthermore, while the spatial compatibility effect was present from the earliest stages of movement preparation and throughout the different stages of movement execution, the affordance effect was restricted to the early stages of movement execution. Finally, we tested a small group of unilateral arm amputees using EMG, and found residual spatial compatibility but no affordance, suggesting that spatial compatibility effects do not necessarily rely on individuals' available affordances. Our results show dissociation between affordance and spatial compatibility effects, and suggest that rather than evoking the specific motor action most suitable for interaction with the viewed object, graspable objects prompt the motor system in a general, body-part independent fashion.

Combining classification with fMRI-derived complex network measures for potential neurodiagnostics.

Complex network analysis (CNA), a subset of graph theory, is an emerging approach to the analysis of functional connectivity in the brain, allowing quantitative assessment of network properties such as functional segregation, integration, resilience, and centrality. Here, we show how a classification framework complements complex network analysis by providing an efficient and objective means of selecting the best network model characterizing given functional connectivity data. We describe a novel kernel-sum learning approach, block diagonal optimization (BDopt), which can be applied to CNA features to single out graph-theoretic characteristics and/or anatomical regions of interest underlying discrimination, while mitigating problems of multiple comparisons. As a proof of concept for the method's applicability to future neurodiagnostics, we apply BDopt classification to two resting state fMRI data sets: a trait (between-subjects) classification of patients with schizophrenia vs. controls, and a state (within-subjects) classification of wake vs. sleep, demonstrating powerful discriminant accuracy for the proposed framework.

Coupling between spontaneous (resting state) fMRI fluctuations and human oculo-motor activity.

The recent discovery of incessant spontaneous fluctuations in human brain activity (also termed resting state fMRI) has been a focus of intense research in brain imaging. The spontaneous BOLD activity shows organized anatomical specialization as well as disruption in a number of brain pathologies. The link between the spontaneous fMRI fluctuations and human behavior is therefore of acute interest and importance. Here we report that a highly significant correlation exists between spontaneous BOLD fluctuations and eye movements which occur subliminally and spontaneously in the absence of any visual stimulation. Of the various eye movement parameters tested, we found robust and anatomically consistent correlations with both the amplitude and velocity of spontaneous eye movements. Control experiments ruled out a contribution of spatial and visual attention as well as smooth pursuit eye movements to the effect. The consistent anatomical specificity of the correlation patterns and their tight temporal link at the proper hemodynamic delay argues against a non-neuronal explanation of the effect, such as cardiac or respiratory cycles. Our results thus demonstrate a link between resting state and spontaneously emerging subconscious oculo-motor behavior.