Implicit relevance feedback from electroencephalography and eye tracking in image search.

OBJECTIVE: Methods from brain-computer interfacing (BCI) open a direct access to the mental processes of computer users, which offers particular benefits in comparison to standard methods for inferring user-related information. The signals can be recorded unobtrusively in the background, which circumvents the time-consuming and distracting need for the users to give explicit feedback to questions concerning the individual interest. The obtained implicit information makes it possible to create dynamic user interest profiles in real-time, that can be taken into account by novel types of adaptive, personalised software. In the present study, the potential of implicit relevance feedback from electroencephalography (EEG) and eye tracking was explored with a demonstrator application that simulated an image search engine. APPROACH: The participants of the study queried for ambiguous search terms, having in mind one of the two possible interpretations of the respective term. Subsequently, they viewed different images arranged in a grid that were related to the query. The ambiguity of the underspecified search term was resolved with implicit information present in the recorded signals. For this purpose, feature vectors were extracted from the signals and used by multivariate classifiers that estimated the intended interpretation of the ambiguous query. MAIN RESULT: The intended interpretation was inferred correctly from a combination of EEG and eye tracking signals in 86% of the cases on average. Information provided by the two measurement modalities turned out to be complementary. SIGNIFICANCE: It was demonstrated that BCI methods can extract implicit user-related information in a setting of human-computer interaction. Novelties of the study are the implicit online feedback from EEG and eye tracking, the approximation to a realistic use case in a simulation, and the presentation of a large set of photographies that had to be interpreted with respect to the content.

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control.

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

Is Neural Activity Detected by ERP-Based Brain-Computer Interfaces Task Specific?

OBJECTIVE: Brain-computer interfaces (BCIs) that are based on event-related potentials (ERPs) can estimate to which stimulus a user pays particular attention. In typical BCIs, the user silently counts the selected stimulus (which is repeatedly presented among other stimuli) in order to focus the attention. The stimulus of interest is then inferred from the electroencephalogram (EEG). Detecting attention allocation implicitly could be also beneficial for human-computer interaction (HCI), because it would allow software to adapt to the user's interest. However, a counting task would be inappropriate for the envisaged implicit application in HCI. Therefore, the question was addressed if the detectable neural activity is specific for silent counting, or if it can be evoked also by other tasks that direct the attention to certain stimuli. APPROACH: Thirteen people performed a silent counting, an arithmetic and a memory task. The tasks required the subjects to pay particular attention to target stimuli of a random color. The stimulus presentation was the same in all three tasks, which allowed a direct comparison of the experimental conditions. RESULTS: Classifiers that were trained to detect the targets in one task, according to patterns present in the EEG signal, could detect targets in all other tasks (irrespective of some task-related differences in the EEG). SIGNIFICANCE: The neural activity detected by the classifiers is not strictly task specific but can be generalized over tasks and is presumably a result of the attention allocation or of the augmented workload. The results may hold promise for the transfer of classification algorithms from BCI research to implicit relevance detection in HCI.

Classification of Eye Fixation Related Potentials for Variable Stimulus Saliency.

OBJECTIVE: Electroencephalography (EEG) and eye tracking can possibly provide information about which items displayed on the screen are relevant for a person. Exploiting this implicit information promises to enhance various software applications. The specific problem addressed by the present study is that items shown in real applications are typically diverse. Accordingly, the saliency of information, which allows to discriminate between relevant and irrelevant items, varies. As a consequence, recognition can happen in foveal or in peripheral vision, i.e., either before or after the saccade to the item. Accordingly, neural processes related to recognition are expected to occur with a variable latency with respect to the eye movements. The aim was to investigate if relevance estimation based on EEG and eye tracking data is possible despite of the aforementioned variability. APPROACH: Sixteen subjects performed a search task where the target saliency was varied while the EEG was recorded and the unrestrained eye movements were tracked. Based on the acquired data, it was estimated which of the items displayed were targets and which were distractors in the search task. RESULTS: Target prediction was possible also when the stimulus saliencies were mixed. Information contained in EEG and eye tracking data was found to be complementary and neural signals were captured despite of the unrestricted eye movements. The classification algorithm was able to cope with the experimentally induced variable timing of neural activity related to target recognition. SIGNIFICANCE: It was demonstrated how EEG and eye tracking data can provide implicit information about the relevance of items on the screen for potential use in online applications.

EEG-based usability assessment of 3D shutter glasses.

OBJECTIVE: Neurotechnology can contribute to the usability assessment of products by providing objective measures of neural workload and can uncover usability impediments that are not consciously perceived by test persons. In this study, the neural processing effort imposed on the viewer of 3D television by shutter glasses was quantified as a function of shutter frequency. In particular, we sought to determine the critical shutter frequency at which the 'neural flicker' vanishes, such that visual fatigue due to this additional neural effort can be prevented by increasing the frequency of the system. APPROACH: Twenty-three participants viewed an image through 3D shutter glasses, while multichannel electroencephalogram (EEG) was recorded. In total ten shutter frequencies were employed, selected individually for each participant to cover the range below, at and above the threshold of flicker perception. The source of the neural flicker correlate was extracted using independent component analysis and the flicker impact on the visual cortex was quantified by decoding the state of the shutter from the EEG. MAIN RESULT: Effects of the shutter glasses were traced in the EEG up to around 67 Hz-about 20 Hz over the flicker perception threshold-and vanished at the subsequent frequency level of 77 Hz. SIGNIFICANCE: The impact of the shutter glasses on the visual cortex can be detected by neurotechnology even when a flicker is not reported by the participants. Potential impact. Increasing the shutter frequency from the usual 50 Hz or 60 Hz to 77 Hz reduces the risk of visual fatigue and thus improves shutter-glass-based 3D usability.

Visuomotor functional network topology predicts upcoming tasks.

It is a vital ability of humans to flexibly adapt their behavior to different environmental situations. Constantly, the rules for our sensory-to-motor mappings need to be adapted to the current task demands. For example, the same sensory input might require two different motor responses depending on the actual situation. How does the brain prepare for such different responses? It has been suggested that the functional connections within cortex are biased according to the present rule to guide the flow of information in accordance with the required sensory-to-motor mapping. Here, we investigated with fMRI whether task settings might indeed change the functional connectivity structure in a large-scale brain network. Subjects performed a visuomotor response task that required an interaction between visual and motor cortex: either within each hemisphere or across the two hemispheres of the brain depending on the task condition. A multivariate analysis on the functional connectivity graph of a cortical visuomotor network revealed that the functional integration, i.e., the connectivity structure, is altered according to the task condition already during a preparatory period before the visual cue and the actual movement. Our results show that the topology of connection weights within a single network changes according to and thus predicts the upcoming task. This suggests that the human brain prepares to respond in different conditions by altering its large scale functional connectivity structure even before an action is required.