Task modulation of brain responses in visual word recognition as studied using EEG/MEG and fMRI.

Do task demands change the way we extract information from a stimulus, or only how we use this information for decision making? In order to answer this question for visual word recognition, we used EEG/MEG as well as fMRI to determine the latency ranges and spatial areas in which brain activation to words is modulated by task demands. We presented letter strings in three tasks (lexical decision, semantic decision, silent reading), and measured combined EEG/MEG as well as fMRI responses in two separate experiments. EEG/MEG sensor statistics revealed the earliest reliable task effects at around 150 ms, which were localized, using minimum norm estimates (MNE), to left inferior temporal, right anterior temporal and left precentral gyri. Later task effects (250 and 480 ms) occurred in left middle and inferior temporal gyri. Our fMRI data showed task effects in left inferior frontal, posterior superior temporal and precentral cortices. Although there was some correspondence between fMRI and EEG/MEG localizations, discrepancies predominated. We suggest that fMRI may be less sensitive to the early short-lived processes revealed in our EEG/MEG data. Our results indicate that task-specific processes start to penetrate word recognition already at 150 ms, suggesting that early word processing is flexible and intertwined with decision making.

Multimodal communication in animals, humans and robots: an introduction to perspectives in brain-inspired informatics.

Recent years have seen convergence in research on brain mechanisms and neurocomputational approaches, culminating in the creation of a new generation of robots whose artificial "brains" respect neuroscience principles and whose "cognitive" systems venture into higher cognitive domains such as planning and action sequencing, complex object and concept processing, and language. The present article gives an overview of selected projects in this general multidisciplinary field. The work reviewed centres on research funded by the EU in the context of the New and Emergent Science and Technology, NEST, funding scheme highlighting the topic "What it means to be human". Examples of such projects include learning by imitation (Edici project), examining the origin of human rule-based reasoning (Far), studying the neural origins of language (Neurocom), exploring the evolutionary origins of the human mind (Pkb140404), researching into verbal and non-verbal communication (Refcom), using and interpreting signs (Sedsu), characterising human language by structural complexity (Chlasc), and representing abstract concepts (Abstract). Each of the communication-centred research projects revealed individual insights; however, there had been little overall analysis of results and hypotheses. In the Specific Support Action Nestcom, we proposed to analyse some NEST projects focusing on the central question "What it means to communicate" and to review, understand and integrate the results of previous communication-related research, in order to develop and communicate multimodal experimental hypotheses for investigation by future projects. The present special issue includes a range of papers on the interplay between neuroinformatics, brain science and robotics in the general area of higher cognitive functions and multimodal communication. These papers extend talks given at the NESTCOM workshops, at ICANN (http://www.his.sunderland.ac.uk/nestcom/workshop/icann.html) in Porto and at the first meeting of the Federation of the European Societies of Neuropsychology in Edinburgh in 2008 (http://www.his.sunderland.ac.uk/nestcom/workshop/esn.html). We hope that the collection will give a vivid insight into current trends in the field.

Can I have a quick word? Early electrophysiological manifestations of psycholinguistic processes revealed by event-related regression analysis of the EEG.

We applied multiple linear regression analysis to event-related electrophysiological responses to words and pseudowords in a visual lexical decision task, yielding event-related regression coefficients (ERRCs) instead of the traditional event-related potential (ERP) measure. Our main goal was to disentangle the earliest ERP effects of the length of letter strings ("word length") and orthographic neighbourhood size (Coltheart's "N"). With respect to N, existing evidence is still ambiguous with respect to whether effects of N reflect early access to lexico-semantic information, or whether they occur at later decision or verification stages. In the present study, we found distinct neurophysiological manifestations of both N and word length around 100ms after word onset. Importantly, the effect of N distinguished between words and pseudowords, while the effect of word length did not. Minimum norm source estimation revealed the most dominant sources for word length in bilateral posterior brain areas for both words and pseudowords. For N, these sources were more left-lateralised and consistent with perisylvian brain areas, with activation peaks in temporal areas being more anterior for words compared to pseudowords. Our results support evidence for an effect of N at early and elementary stages of word recognition. We discuss the implications of these results for the time line of word recognition processes, and emphasise the value of ERRCs in combination with source analysis in psycholinguistic and cognitive brain research.

The time course of action and action-word comprehension in the human brain as revealed by neurophysiology.

Numerous previous neuroimaging studies suggest an involvement of cortical motor areas not only in action execution but also in action recognition and understanding. Motor areas of the human brain have also been found to activate during the processing of written and spoken action-related words and sentences. Even more strikingly, stimuli referring to different bodily effectors produced specific somatotopic activation patterns in the motor areas. However, metabolic neuroimaging results can be ambiguous with respect to the processing stage they reflect. This is a serious limitation when hypotheses concerning linguistic processes are tested, since in this case it is usually crucial to distinguish early lexico-semantic processing from strategic effects or mental imagery that may follow lexico-semantic information access. Timing information is therefore pivotal to determine the functional significance of motor areas in action recognition and action-word comprehension. Here, we review attempts to reveal the time course of these processes using neurophysiological methods (EEG, MEG and TMS), in visual and auditory domains. We will highlight the importance of the choice of appropriate paradigms in combination with the corresponding method for the extraction of timing information. The findings will be discussed in the general context of putative brain mechanisms of word and object recognition.

How the camel lost its hump: the impact of object typicality on event-related potential signals in object decision.

Using an object decision task, event-related potentials (ERPs), and minimum norm current source estimates, we investigated early spatiotemporal aspects of cortical activation elicited by line drawings that were manipulated on two dimensions: authenticity and typicality. Authentic objects were those that match real-world experience, whereas nonauthentic objects were "doctored" by deletion or addition of features (e.g., a camel with its hump removed, a hammer with two handles). The main manipulation of interest for both authentic and nonauthentic objects was the degree of typicality in the object's structure: typical items are composed of parts that have tended to co-occur across many different objects in the perceiver's experience. The ERP pattern revealed a significant typicality effect at 116 msec after stimulus onset. Both atypical authentic objects (e.g., a camel with its hump) and atypical nonauthentic objects (e.g., a jackal with a hump) elicited stronger brain activation than did objects with typical structure. A significant effect of authenticity was observed at 480 msec, with stronger activation for the nonauthentic objects. The factors of typicality and authenticity interacted at 160 and 330 msec. The most prominent source of the typicality effect was the bilateral occipitotemporal cortex, whereas the interaction and the authenticity effects were mainly observed in the more anterior bilateral temporal cortex. These findings support the hypothesis that within the first few hundred milliseconds after stimulus presentation onset, visual-form-related perceptual and conceptual processes represent distinct but interacting stages in object recognition.

[Q:] When would you prefer a SOSSAGE to a SAUSAGE? [A:] At about 100 msec. ERP correlates of orthographic typicality and lexicality in written word recognition.

Using a speeded lexical decision task, event-related potentials (ERPs), and minimum norm current source estimates, we investigated early spatiotemporal aspects of cortical activation elicited by words and pseudo-words that varied in their orthographic typicality, that is, in the frequency of their component letter pairs (bi-grams) and triplets (tri-grams). At around 100 msec after stimulus onset, the ERP pattern revealed a significant typicality effect, where words and pseudo-words with atypical orthography (e.g., yacht, cacht) elicited stronger brain activation than items characterized by typical spelling patterns (cart, yart). At approximately 200 msec, the ERP pattern revealed a significant lexicality effect, with pseudo-words eliciting stronger brain activity than words. The two main factors interacted significantly at around 160 msec, where words showed a typicality effect but pseudo-words did not. The principal cortical sources of the effects of both typicality and lexicality were localized in the inferior temporal cortex. Around 160 msec, atypical words elicited the stronger source currents in the left anterior inferior temporal cortex, whereas the left perisylvian cortex was the site of greater activation to typical words. Our data support distinct but interactive processing stages in word recognition, with surface features of the stimulus being processed before the word as a meaningful lexical entry. The interaction of typicality and lexicality can be explained by integration of information from the early form-based system and lexicosemantic processes.

The sound of actions as reflected by mismatch negativity: rapid activation of cortical sensory-motor networks by sounds associated with finger and tongue movements.

In order to explore the activation dynamics of the human action recognition system, we investigated electrophysiological distinctions between the brain responses to sounds produced by human finger and tongue movements. Of special interest were the questions of how early these differences may occur, and whether the neural activation at the early stages of processing involves cortical motor representations related to the generation of these sounds. For this purpose we employed a high-density EEG set-up and recorded mismatch negativity (MMN) using a recently developed novel multideviant paradigm which allows acquisition of a high number of trials within a given time period. Deviant stimuli were naturally recorded finger and tongue clicks, as well as control stimuli with similar physical features but without the clear action associations (this was tested in a separate behavioural experiment). Both natural stimuli produced larger MMNs than their respective control stimuli at approximately 100 ms, indicating activation of memory traces for familiar action-related sounds. Furthermore, MMN topography at this latency differed between the brain responses to the natural finger and natural tongue sounds. Source estimation revealed the strongest sources for finger sounds in centrolateral areas of the left hemisphere, suggesting that hearing a sound related to finger actions evokes activity in motor areas associated with the dominant hand. Furthermore, tongue sounds produced activation in more inferior brain areas. Our data suggest that motor areas in the human brain are part of neural systems subserving the early automatic recognition of action-related sounds.

The time course of visual word recognition as revealed by linear regression analysis of ERP data.

EEG correlates of a range of psycholinguistic word properties were used to investigate the time course of access to psycholinguistic information during visual word recognition. Neurophysiological responses recorded in a visual lexical decision task were submitted to linear regression analysis. First, 10 psycholinguistic features of each of 300 stimulus words were submitted to a principal component analysis, which yielded four orthogonal variables likely to reflect separable processes in visual word recognition: Word length, Letter n-gram frequency, Lexical frequency and Semantic coherence of a word's morphological family. Since the lexical decision task required subjects to distinguish between words and pseudowords, the binary variable Lexicality was also investigated using a factorial design. Word-pseudoword differences in the event-related potential first appeared at 160 ms after word onset. However, regression analysis of EEG data documented a much earlier effect of both Word length and Letter n-gram frequency around 90 ms. Lexical frequency showed its earliest effect slightly later, at 110 ms, and Semantic coherence significantly correlated with neurophysiological measures around 160 ms, simultaneously with the lexicality effect. Source estimates indicated parieto-temporo-occipital generators for the factors Length, Letter n-gram frequency and Word frequency, but widespread activation with foci in left anterior temporal lobe and inferior frontal cortex related to Semantic coherence. At later stages (>200 ms), all variables exhibited simultaneous EEG correlates. These results indicate that information about surface form and meaning of a lexical item is first accessed at different times in different brain systems and then processed simultaneously, thus supporting cascaded interactive processing models.

Effects of word length and frequency on the human event-related potential.

OBJECTIVE: We investigated the influence of the length and frequency of printed words on the amplitude and peak latencies of event-related potentials (ERPs). This served two goals, namely (I) to clarify their possible effects as confounds in ERP experiments employing word-stimuli, and (II) to determine the point in time of lexical access in visual word recognition. METHODS: EEG was recorded from 64 scalp sites while subjects (n=12) performed a lexical decision task. Word length and frequency were orthogonally varied between stimulus groups, whereas variables including regularity of spelling and orthographic tri-gram frequency were kept constant. RESULTS: Long words produced the strongest brain response early on (approximately 100 ms after stimulus onset), whereas those to short words became strongest later (150-360 ms). Lower ERP amplitudes were elicited by words with high frequency compared with low frequency words in the latency ranges 150-190 ms and 320-360 ms. However, we did not find evidence for a robust alteration of peak latencies with word frequency. CONCLUSIONS: Length and frequency of word stimuli have independent and additive effects on the amplitude of the ERP. Studies on the precise time course of cognitive processes should consider their potentially confounding character. Our data support the view that lexical access takes place as early as 150 ms after onset of written word stimuli.

Functional asymmetry in schizophrenic patients during auditory speech processing.

In two experiments, functional laterality and interhemispheric transfer was investigated in schizophrenic patients (n=14) and healthy controls (n=17). In Experiment 1, words and pseudowords were presented either to the left or right ear (monaural condition) or simultaneously to both ears (binaural condition). In Experiment 2, subjects had to discriminate two tones differing in frequency during monaural and binaural stimulation. Healthy controls showed a right ear advantage (REA) for word stimuli, indicating left-hemispheric superiority for word processing. The same lateralization pattern was found in schizophrenic patients, indicating unimpaired functional lateralization of auditory language processing. In both groups, no REA was found for pseudowords resulting in significant WordnessxEar interactions. When presented binaurally, auditory processing of words and pseudowords did not differ significantly from any of the two monaural conditions. Tone discrimination did not lead to any ear asymmetry. The results show normal patterns of functional asymmetry during auditory language processing and tone discrimination in schizophrenic patients.

The right hemisphere's role in action word processing: a double case study.

Word category-specific deficits were investigated in two patients with right hemispheric lesions and hemiparesis affecting the left extremities. Words from three categories, action verbs, nouns with strong visual associations and nouns with both strong action and visual associations, were presented in a lexical decision task. The stimulus categories were matched for word length and frequency. In both patients, responses to action verbs were slowed and/or less accurate compared with the other word categories. This was so even in the patient with a minor lesion in the motor, pre-motor and somatosensory areas of the hand representation. Control subjects did not show category differences when tested with the same stimulus materials. These results are consistent with the view that the cortical areas involved in the programming of body movements, even those in the hemisphere not dominant for language, specifically contribute to and are necessary for the processing of words referring to such movements. As an alternative, the affected brain areas may be of particular relevance for the processing of words from the lexical category of verbs. The results are consistent with a brain model of language based on Hebb's cell assembly concept.

Memory traces for words as revealed by the mismatch negativity.

Brain responses to the same spoken syllable completing a Finnish word or a pseudo-word were studied. Native Finnish-speaking subjects were instructed to ignore the sound stimuli and watch a silent movie while the mismatch negativity (MMN), an automatic index of experience-dependent auditory memory traces, was recorded. The MMN to each syllable was larger when it completed a word than when it completed a pseudo-word. This enhancement, reaching its maximum amplitude at about 150 ms after the word's recognition point, did not occur in foreign subjects who did not know any Finnish. These results provide the first demonstration of the presence of memory traces for individual spoken words in the human brain. Using whole-head magnetoencephalography, the major intracranial source of this word-related MMN was found in the left superior temporal lobe.

Walking or talking? Behavioral and neurophysiological correlates of action verb processing.

Brain activity elicited by visually presented words was investigated using behavioral measures and current source densities calculated from high-resolution EEG recordings. Verbs referring to actions usually performed with different body parts were compared. Behavioral data indicated faster processing of verbs referring to actions performed with the face muscles and articulators (face-related words) compared to verbs referring to movements involving the lower half of the body (leg-related words). Significant topographical differences in brain activity elicited by verb types were found starting approximately 250 ms after word onset. Differences were seen at recording sites located over the motor strip and adjacent frontal cortex. At the vertex, close to the cortical representation of the leg, leg-related verbs (for example, to walk) produced strongest in-going currents, whereas for face-related verbs (for example, to talk) the most in-going activity was seen at more lateral electrodes placed over the left Sylvian fissure, close to the representation of the articulators. Thus, action words caused differential activation along the motor strip, with strongest in-going activity occurring close to the cortical representation of the body parts primarily used for carrying out the actions the verbs refer to. Topographically specific physiological signs of word processing started earlier for face-related words and lasted longer for verbs referring to leg movements. We conclude that verb types can differ in their processing speed and can elicit neurophysiological activity with different cortical topographies. These behavioral and physiological differences can be related to cognitive processes, in particular to lexical semantic access. Our results are consistent with associative theories postulating that words are organized in the brain as distributed cell assemblies whose cortical distributions reflect the words' meanings.

Syntactic and semantic processing in the healthy and aphasic human brain.

A syntactic and a semantic task were performed by German-speaking healthy subjects and aphasics with lesions in the dominant left hemisphere. In both tasks, pictures of objects were presented that had to be classified by pressing buttons. The classification was into grammatical gender in the syntactic task (masculine or feminine gender?) and into semantic category in the semantic task (man- or nature made?). Behavioral data revealed a significant Group by Task interaction, with aphasics showing most pronounced problems with syntax. Brain event-related potentials 300-600 ms following picture onset showed different task-dependent laterality patterns in the two groups. In controls, the syntax task induced a left-lateralized negative ERP, whereas the semantic task produced more symmetric responses over the hemispheres. The opposite was the case in the patients, where, paradoxically, stronger laterality of physiological brain responses emerged in the semantic task than in the syntactic task. We interpret these data based on neuro-psycholinguistic models of word processing and current theories about the roles of the hemispheres in language recovery.

Constraint-induced therapy of chronic aphasia after stroke.

Patients with chronic aphasia were assigned randomly to a group to receive either conventional aphasia therapy or constraint-induced (CI) aphasia therapy, a new therapeutic technique requiring intense practice over a relatively short period of consecutive days. CI aphasia therapy is realized in a communicative therapeutic environment constraining patients to practice systematically speech acts with which they have difficulty. Patients in both groups received the same amount of treatment (30 to 35 hours) as 10 days of massed-practice language exercises for the CI aphasia therapy group (3 hours per day minimum; 10 patients) or over a longer period of approximately 4 weeks for the conventional therapy group (7 patients). CI aphasia therapy led to significant and pronounced improvements on several standard clinical tests, on self-ratings, and on blinded-observer ratings of the patients' communicative effectiveness in everyday life. Patients who received the control intervention failed to achieve comparable improvements. Data suggest that the language skills of patients with chronic aphasia can be improved in a short period by use of an appropriate massed-practice technique that focuses on the patients' communicative needs.

Investigation of brain dynamics in Parkinson's disease by methods derived from nonlinear dynamics.

EEGs were recorded from patients in early stages of Parkinson's disease (17 patients, 9 females) and healthy controls (12 subjects, 8 females) during rest and during execution/imagining of a complex motor task. The prediction that Parkinson's disease patients compared to controls would show more complex brain dynamics during performance of a complex motor task and imagination of the movements was confirmed by methods derived from nonlinear dynamics. In the resting state, analysis of correlation dimension of EEG time series revealed only slight topographical differences between the groups. During performance of a complex motor task, however, data from Parkinson's disease patients showed higher dimensionality than data from controls, indicating more complex EEG time series. The same difference was found when subjects did not perform any motor movements but imagined the complex movements they had just performed. The data are consistent with the hypothesis that the disturbances in Parkinson's disease result in the recruitment of superfluous cortical networks due to failed inhibition of alternative motor programs in the striatum and thus increase the complexity of cortical representation in motor conditions.

Neuromagnetic evidence for early access to cognitive representations.

How do physical and cognitive properties of stimulus words influence the neuromagnetic response of the human brain? Are the physiological correlates of these properties dissociable and at which latencies can they be observed? Short and long words, as well as rare and common words, were repeatedly presented in a memory task while neuromagnetic brain responses were recorded using MEG. Word length and frequency were reflected by brain responses at overlapping but distinct intervals. The influence of the physical factor, length, started at approximately 100 ms after onset of written words, immediately followed by a physiological manifestation of the non-physical cognitive stimulus property, word frequency, which was first apparent at 120-160 ms. There was a differential frequency effect: neurophysiological correlates of short words showed the frequency influence much earlier than did longer words. These data indicate that non-physical cognitive aspects of word stimuli can be reflected in early neuromagnetic responses, and that the latency of these physiological correlates of cognitive stimulus properties may depend on the physical stimulus make-up.

Neuromagnetic evidence for early semantic access in word recognition.

Magnetic brain responses recorded in the human magnetoencephalogram (MEG) distinguished between words with different semantics but carefully matched for frequency and length. Multiple recordings from a single subject showed that 100 ms following stimulus onset, significantly stronger neuromagnetic responses were elicited by words with strong multimodal semantic associations than by other word material. At this early processing step, there was a highly significant correlation (0.80) between the magnitude of brain responses to individual words recorded over parieto-occipital areas and their semantic association strengths. Subsequent to this early difference related to word meaning, additional differences in MEG responses emerged for words from different grammatical categories. Together, these results suggest that word meaning can be reflected by early neuromagnetic brain responses and before the grammatical information about the word is encoded.

Interhemispheric cooperation during word processing: evidence for callosal transfer dysfunction in schizophrenic patients.

Functional lateralization and interhemispheric interaction during word processing were investigated in schizophrenic patients (n=12) and matched healthy controls (n=18). Words and phonologically regular pseudowords were presented tachistoscopically either in the left or right visual field (unilateral conditions), or simultaneously in both visual hemifields (bilateral condition). Consistent with earlier findings, healthy controls showed a right visual field advantage (RVFA), indicating left-hemispheric dominance for language. The patients showed a RVFA similar to that of controls, consistent with normal left-hemispheric language dominance. Importantly, controls performed much better on words presented in the bilateral condition, when two copies of the same word appeared twice, compared to stimulation in only one of the visual hemifields. This bilateral advantage, which has been interpreted as evidence for cooperation between the hemispheres, was absent in schizophrenics. These data show that schizophrenic patients can exhibit similar lateralization patterns as healthy controls. Their specific functional deficit may be a lack of cooperation between the hemispheres.

Neurophysiological distinction of verb categories.

Neurophysiological brain responses to subcategories of action verbs were recorded using high resolution EEG. Starting 240 ms after word onset, topographies of event-related potentials distinguished between verbs referring to different action types. Current source density estimates revealed that verbs referring to actions executed with the legs (to kick) produced an activity focus close to the vertex, above motor areas involved in the programming of leg movements, whereas face-related verbs (to speak) produced a focus at left-lateral recordings, above perisylvian areas and the cortical representation of the face and articulators. This is evidence that brain mechanisms involved in word processing can differ even between fine-grained lexico-semantic subcategories and already within the first quarter of a second after word onset. We offer an explanation of the data in neurobiological terms.