On the Relative Contribution of Deep Convolutional Neural Networks for SSVEP-Based Bio-Signal Decoding in BCI Speller Applications.

Brain-computer interfaces (BCI) harnessing steady state visual evoked potentials (SSVEPs) manipulate the frequency and phase of visual stimuli to generate predictable oscillations in neural activity. For BCI spellers, oscillations are matched with alphanumeric characters allowing users to select target numbers and letters. Advances in BCI spellers can, in part, be accredited to subject-specific optimization, including; 1) custom electrode arrangements; 2) filter sub-band assessments; and 3) stimulus parameter tuning. Here, we apply deep convolutional neural networks (DCNNs) demonstrating cross-subject functionality for the classification of frequency and phase encoded SSVEP. Electroencephalogram (EEG) data are collected and classified using the same parameters across subjects. Subjects fixate forty randomly cued flickering characters ( 5 ×8 keyboard array) during concurrent wet-EEG acquisition. These data are provided by an open source SSVEP dataset. Our proposed DCNN, PodNet, achieves 86% and 77% offline accuracy of classification across-subjects for two data capture periods, respectively, 6-seconds (information transfer rate = 40 bpm) and 2-seconds (information transfer rate = 101 bpm). Subjects demonstrating sub-optimal (<70%) performance are classified to similar levels after a short subject-specific training period. PodNet outperforms filter-bank canonical correlation analysis for a low volume (3-channel) clinically feasible occipital electrode configuration. The networks defined in this study achieve functional performance for the largest number of SSVEP classes decoded via DCNN to date. Our results demonstrate PodNet achieves cross-subject, calibrationless classification and adaptability to sub-optimal subject data, and low-volume EEG electrode arrangements.

Human neuroimaging reveals the subcomponents of grasping, reaching and pointing actions.

Although the neural underpinnings of visually guided grasping and reaching have been well delineated within lateral and medial fronto-parietal networks (respectively), the contributions of subcomponents of visuomotor actions have not been explored in detail. Using careful subtraction logic, here we investigated which aspects of grasping, reaching, and pointing movements drive activation across key areas within visuomotor networks implicated in hand actions. For grasping tasks, we find activation differences based on the precision required (fine > coarse grip: anterior intraparietal sulcus, aIPS), the requirement to lift the object (grip + lift > grip: aIPS; dorsal premotor cortex, PMd; and supplementary motor area, SMA), and the number of digits employed (3-/5- vs. 2-digit grasps: ventral premotor cortex, PMv; motor cortex, M1, and somatosensory cortex, S1). For reaching/pointing tasks, we find activation differences based on whether the task required arm transport ((reach-to-point with index finger and reach-to-touch with knuckles) vs. point-without-reach; anterior superior parietal lobule, aSPL) and whether it required pointing to the object centre ((point-without-reach and reach-to-point) vs. reach-to-touch: anterior superior parieto-occipital cortex, aSPOC). For point-without-reach, in which the index finger is oriented towards the object centre but from a distance (point-without-reach > (reach-to-point and reach-to-touch)), we find activation differences that may be related to the communicative nature of the task (temporo-parietal junction, TPJ) and the need to precisely locate the target (lateral occipito-temporal cortex, LOTC). The present findings elucidate the different subcomponents of hand actions and the roles of specific brain regions in their computation.

Gender differences in non-standard mapping tasks: A kinematic study using pantomimed reach-to-grasp actions.

Comparison between real and pantomimed actions is used in neuroscience to dissociate stimulus-driven (real) as compared to internally driven (pantomimed) visuomotor transformations, with the goal of testing models of vision (Milner & Goodale, 1995) and diagnosing neuropsychological deficits (apraxia syndrome). Real actions refer to an overt movement directed toward a visible target whereas pantomimed actions refer to an overt movement directed either toward an object that is no longer available. Although similar, real and pantomimed actions differ in their kinematic parameters and in their neural substrates. Pantomimed-reach-to-grasp-actions show reduced reaching velocities, higher wrist movements, and reduced grip apertures. In addition, seminal neuropsychological studies and recent neuroimaging findings confirmed that real and pantomimed actions are underpinned by separate brain networks. Although previous literature suggests differences in the praxis system between males and females, no research to date has investigated whether or not gender differences exist in the context of real versus pantomimed reach-to-grasp actions. We asked ten male and ten female participants to perform real and pantomimed reach-to-grasp actions toward objects of different sizes, either with or without visual feedback. During pantomimed actions participants were required to pick up an imaginary object slightly offset relative to the location of the real one (which was in turn the target of the real reach-to-grasp actions). Results demonstrate a significant difference between the kinematic parameters recorded in male and female participants performing pantomimed, but not real reach-to-grasp tasks, depending on the availability of visual feedback. With no feedback both males and females showed smaller grip aperture, slower movement velocity and lower reach height. Crucially, these same differences were abolished when visual feedback was available in male, but not in female participants. Our results suggest that male and female participants should be evaluated separately in the clinical environment and in future research in the field.

Representational content of occipitotemporal and parietal tool areas.

It is now established that the perception of tools engages a left-lateralized network of frontoparietal and occipitotemporal cortical regions. Nevertheless, the precise computational role played by these areas is not yet well understood. To address this question, we used functional MRI to investigate the distribution of responses to pictures of tools and hands relative to other object categories in the so-called "tool" areas. Although hands and tools are visually not alike and belong to different object categories, these are both functionally linked when considering the common role of hands and tools in object manipulation. This distinction can provide insight into the differential functional role of areas within the "tool" network. Results demonstrated that images of hands and tools activate a common network of brain areas in the left intraparietal sulcus (IPS), left lateral occipitotemporal cortex (LOTC) and ventral occipitotemporal cortex (VOTC). Importantly, multivoxel pattern analysis revealed that the distribution of hand and tool response patterns in these regions differs. These observations provide support for the idea that the left IPS, left LOTC and VOTC might have distinct computational roles with regard to tool use. Specifically, these results suggest that while left IPS supports tool action-related computations and VOTC primarily encodes category specific aspects of objects, left LOTC bridges ventro occipitotemporal perception-related and parietal action-related representations by encoding both types of object information.

Coding of attention across the human intraparietal sulcus.

There has been concentrated debate over four decades as to whether or not the nonhuman primate parietal cortex codes for intention or attention. In nonhuman primates, certain studies report results consistent with an intentional role, whereas others provide support for coding of visual-spatial attention. Until now, no one has yet directly contrasted an established motor "intention" paradigm with a verified "attention" paradigm within the same protocol. This debate has continued in both the nonhuman primate and healthy human brain and is subsequently timely. We incorporated both paradigms across two distinct temporal epochs within a whole-parietal slow event-related human functional magnetic resonance imaging experiment. This enabled us to examine whether or not one paradigm proves more effective at driving the neural response across three intraparietal areas. As participants performed saccadic eye and/or pointing tasks, discrete event-related components with dissociable responses were elicited in distinct sub-regions of human parietal cortex. Critically, the posterior intraparietal area showed robust activity consistent with attention (no intention planning). The most contentious area in the literature, the middle intraparietal area produced activation patterns that further reinforce attention coding in human parietal cortex. Finally, the anterior intraparietal area showed the same pattern. Therefore, distributed coding of attention is relatively more pronounced across the two computations within human parietal cortex.

Non-obstructing 3D depth cues influence reach-to-grasp kinematics.

It has been demonstrated that both visual feedback and the presence of certain types of non-target objects in the workspace can affect kinematic measures and the trajectory path of the moving hand during reach-to-grasp movements. Yet no study to date has examined the possible effect of providing non-obstructing three-dimensional (3D) depth cues within the workspace and with consistent retinal inputs and whether or not these alter manual prehension movements. Participants performed a series of reach-to-grasp movements in both open- (without visual feedback) and closed-loop (with visual feedback) conditions in the presence of one of three possible 3D depth cues. Here, it is reported that preventing online visual feedback (or not) and the presence of a particular depth cue had a profound effect on kinematic measures for both the reaching and grasping components of manual prehension-despite the fact that the 3D depth cues did not act as a physical obstruction at any point. The depth cues modulated the trajectory of the reaching hand when the target block was located on the left side of the workspace but not on the right. These results are discussed in relation to previous reports and implications for brain-computer interface decoding algorithms are provided.

Gaze-dependent topography in human posterior parietal cortex.

The brain must convert retinal coordinates into those required for directing an effector. One prominent theory holds that, through a combination of visual and motor/proprioceptive information, head-/body-centered representations are computed within the posterior parietal cortex (PPC). An alternative theory, supported by recent visual and saccade functional magnetic resonance imaging (fMRI) topographic mapping studies, suggests that PPC neurons provide a retinal/eye-centered coordinate system, in which the coding of a visual stimulus location and/or intended saccade endpoints should remain unaffected by changes in gaze position. To distinguish between a retinal/eye-centered and a head-/body-centered coordinate system, we measured how gaze direction affected the representation of visual space in the parietal cortex using fMRI. Subjects performed memory-guided saccades from a central starting point to locations "around the clock." Starting points varied between left, central, and right gaze relative to the head-/body midline. We found that memory-guided saccadotopic maps throughout the PPC showed spatial reorganization with very subtle changes in starting gaze position, despite constant retinal input and eye movement metrics. Such a systematic shift is inconsistent with models arguing for a retinal/eye-centered coordinate system in the PPC, but it is consistent with head-/body-centered coordinate representations.

Optic ataxia as a model to investigate the role of the posterior parietal cortex in visually guided action: evidence from studies of patient M.H.

Optic ataxia is a neuropsychological disorder that affects the ability to interact with objects presented in the visual modality following either unilateral or bilateral lesions of the posterior parietal cortex (PPC). Patients with optic ataxia fail to reach accurately for objects, particularly when they are presented in peripheral vision. The present review will focus on a series of experiments performed on patient M.H. Following a lesion restricted largely to the left PPC, he developed mis-reaching behavior when using his contralesional right arm for movements directed toward the contralesional (right) visual half-field. Given the clear-cut specificity of this patient's deficit, whereby reaching actions are essentially spared when executed toward his ipsilateral space or when using his left arm, M.H. provides a valuable "experiment of nature" for investigating the role of the PPC in performing different visually guided actions. In order to address this, we used kinematic measurement techniques to investigate M.H.'s reaching and grasping behavior in various tasks. Our experiments support the idea that optic ataxia is highly function-specific: it affects a specific sub-category of visually guided actions (reaching but not grasping), regardless of their specific end goal (both reaching toward an object and reaching to avoid an obstacle); and finally, is independent of the limb used to perform the action (whether the arm or the leg). Critically, these results are congruent with recent functional MRI experiments in neurologically intact subjects which suggest that the PPC is organized in a function-specific, rather than effector-specific, manner with different sub-portions of its mantle devoted to guiding actions according to their specific end-goal (reaching, grasping, or looking), rather than according to the effector used to perform them (leg, arm, hand, or eyes).

Parallel feedback pathways in visual cortex of cats revealed through a modified rabies virus.

The visual cortex of cats is highly evolved. Analogously to the brains of primates, large numbers of visual areas are arranged hierarchically and can be parsed into separate dorsal and ventral streams for object recognition and visuospatial representation. Within early primate visual areas, V1 and V2, and to a lesser extent V3, the two streams are relatively segregated and relayed in parallel to higher order cortex, although there is some evidence suggesting an alignment of V2 and V3 to one stream over the other. For cats, there is no evidence of anatomical segregation in areas 18 and 19, the analogs to V2 and V3. However, previous work was only qualitative in nature. Here we re-examined the feedback connectivity patterns of areas 18/19 in quantitative detail. To accomplish this, we used a genetically modified rabies virus that acts as a retrograde tracer and fills neurons with fluorescent protein. After injections into area 19, many more neurons were labeled in putative ventral stream area 21a than in putative dorsal stream region posterolateral suprasylvian complex of areas (PLS), and the dendrites of neurons in 21a were significantly more complex. Conversely, area 18 injections labeled more neurons in PLS, and these were more complex than neurons in 21a. We infer from our results that area 19 in cat is more aligned to the ventral stream and area 18 to the dorsal stream. Based on the success of our approach, we suggest that this method could be applied to resolve similar issues related to primate V3.

The case for primate V3.

The visual system in primates is represented by a remarkably large expanse of the cerebral cortex. While more precise investigative studies that can be performed in non-human primates contribute towards understanding the organization of the human brain, there are several issues of visual cortex organization in monkey species that remain unresolved. In all, more than 20 areas comprise the primate visual cortex, yet there is little agreement as to the exact number, size and visual field representation of all but three. A case in point is the third visual area, V3. It is found relatively early in the visual system hierarchy, yet over the last 40 years its organization and even its very existence have been a matter of debate among prominent neuroscientists. In this review, we discuss a large body of recent work that provides straightforward evidence for the existence of V3. In light of this, we then re-examine results from several seminal reports and provide parsimonious re-interpretations in favour of V3. We conclude with analysis of human and monkey functional magnetic resonance imaging literature to make the case that a complete V3 is an organizational feature of all primate species and may play a greater role in the dorsal stream of visual processing.

Saccade preparation signals in the human frontal and parietal cortices.

Our ability to prepare an action in advance allows us to respond to our environment quickly, accurately, and flexibly. Here, we used event-related functional MRI to measure human brain activity while subjects maintained an active state of preparedness. At the beginning of each trial, subjects were instructed to prepare a pro- or antisaccade to a visual cue that was continually present during a long and variable preparation interval, but to defer the saccade's execution until a go signal. The deferred saccade task eliminated the mnemonic component inherent in memory-guided saccade tasks and placed the emphasis entirely on advance motor preparation. During the delay while subjects were in an active state of motor preparedness, the blood oxygen level-dependent signal in the frontal cortex showed 1) a sustained elevation throughout the preparation interval; 2) a linear increase with increasing delay length; 3) a bias for contra- rather than ipsiversive movements; 4) greater activity when the specific metrics of the planned saccade were known compared with when they were not; and 5) increased activity when the saccade was directed toward an internal versus an external representation (i.e., anticue location). These findings support the hypothesis that both the human frontal and parietal cortices are involved in the spatial selection and preparation of saccades.

Effector-specific fields for motor preparation in the human frontal cortex.

We investigated the neural correlates of advance motor preparation in two experiments that required a movement in response to a peripheral visual stimulus. In one experiment (the memory delay paradigm), subjects knew the target location during a preparatory 'memory delay' interval; in the other experiment they did not know the target location during a 'gap period' (the gap paradigm). In both experiments we further varied the effector that was instructed, either the eye or the forelimb. An area that codes motor preparation should exhibit increases during the memory delay and gap period and such increases should predict some attribute of performance (planning to use the eye or the forelimb). We first identified the frontoparietal visuomotor areas using standard fMRI block designs. Subjects were then scanned using event-related fMRI. With the exception of primary motor cortex (M1), all areas (putative lateral intraparietal area (putLIP), dorsal premotor cortex (PMd), frontal eye field (FEF), ventral frontal eye field (FEFv), supplementary motor area (SMA)) showed gap and memory delay activation for both saccades and pointing. Gap activity in the frontal areas was higher than in the parietal area(s) investigated. The observation that 'memory delay' activity was equivalent or less than gap activity in all areas suggests that what is commonly considered to be memory-related responses largely represents advance motor preparation. Certain areas showed increased activation during the gap or memory delay intervals for pointing (PMd, FEF, FEFv) or saccades (SMA, putLIP). These observations suggest an important role of the frontal cortex in advance motor preparation.

fMRI activation in the human frontal eye field is correlated with saccadic reaction time.

Variation in response latency to identical sensory stimuli has been attributed to variation in neural activity mediating preparatory set. Here we report evidence for a relationship between saccadic reaction time (SRT) and set-related brain activity measured with event-related functional magnetic resonance imaging. We measured hemodynamic activation time-courses during a preparatory "gap" period, during which no visual stimulus was present and no saccades were made. The subjects merely anticipated appearance of the target. Saccade direction and latency were recorded during scanning, and trials were sorted according to SRT. Both the frontal (FEF) and supplementary eye fields showed pre-target preparatory activity, but only in the FEF was this activity correlated with SRT. Activation in the intraparietal sulcus did not show any preparatory activity. These data provide evidence that the human FEF plays a central role in saccade initiation; pre-target activity in this region predicts both the type of eye movement (whether the subject will look toward or away from the target) and when a future saccade will occur.

FMRI evidence for a 'parietal reach region' in the human brain.

Event-related functional magnetic resonance imaging was used to examine activation in the posterior parietal cortex when subjects made pointing movements or saccades to the same spatial location. One region, well positioned to be homologous to the monkey parietal reach region (PRR), responded preferentially during memory-delay trials in which the subject planned to point to a specific location as compared to trials in which the subject planned to make a saccade to that same location. We therefore conclude that activation in this region is related to specific motor intent; i.e. it encodes information related to the subject's intention to make a specific movement to a particular spatial location.

Human fMRI evidence for the neural correlates of preparatory set.

We used functional magnetic resonance imaging (fMRI) to study readiness and intention signals in frontal and parietal areas that have been implicated in planning saccadic eye movements-the frontal eye fields (FEF) and intraparietal sulcus (IPS). To track fMRI signal changes correlated with readiness to act, we used an event-related design with variable gap periods between disappearance of the fixation point and appearance of the target. To track changes associated with intention, subjects were instructed before the gap period to make either a pro-saccade (look at target) or an anti-saccade (look away from target). FEF activation increased during the gap period and was higher for anti- than for pro-saccade trials. No signal increases were observed during the gap period in the IPS. Our findings suggest that within the frontoparietal networks that control saccade generation, the human FEF, but not the IPS, is critically involved in preparatory set, coding both the readiness and intention to perform a particular movement.