Fuzzy Decision-Making Fuser (FDMF) for Integrating Human-Machine Autonomous (HMA) Systems with Adaptive Evidence Sources.

A brain-computer interface (BCI) creates a direct communication pathway between the human brain and an external device or system. In contrast to patient-oriented BCIs, which are intended to restore inoperative or malfunctioning aspects of the nervous system, a growing number of BCI studies focus on designing auxiliary systems that are intended for everyday use. The goal of building these BCIs is to provide capabilities that augment existing intact physical and mental capabilities. However, a key challenge to BCI research is human variability; factors such as fatigue, inattention, and stress vary both across different individuals and for the same individual over time. If these issues are addressed, autonomous systems may provide additional benefits that enhance system performance and prevent problems introduced by individual human variability. This study proposes a human-machine autonomous (HMA) system that simultaneously aggregates human and machine knowledge to recognize targets in a rapid serial visual presentation (RSVP) task. The HMA focuses on integrating an RSVP BCI with computer vision techniques in an image-labeling domain. A fuzzy decision-making fuser (FDMF) is then applied in the HMA system to provide a natural adaptive framework for evidence-based inference by incorporating an integrated summary of the available evidence (i.e., human and machine decisions) and associated uncertainty. Consequently, the HMA system dynamically aggregates decisions involving uncertainties from both human and autonomous agents. The collaborative decisions made by an HMA system can achieve and maintain superior performance more efficiently than either the human or autonomous agents can achieve independently. The experimental results shown in this study suggest that the proposed HMA system with the FDMF can effectively fuse decisions from human brain activities and the computer vision techniques to improve overall performance on the RSVP recognition task. This conclusion demonstrates the potential benefits of integrating autonomous systems with BCI systems.

A testbed for optimizing electrodes embedded in the skull or in artificial skull replacement pieces used after injury.

BACKGROUND: Custom-fitted skull replacement pieces are often used after a head injury or surgery to replace damaged bone. Chronic brain recordings are beneficial after injury/surgery for monitoring brain health and seizure development. Embedding electrodes directly in these artificial skull replacement pieces would be a novel, low-risk way to perform chronic brain monitoring in these patients. Similarly, embedding electrodes directly in healthy skull would be a viable minimally-invasive option for many other neuroscience and neurotechnology applications requiring chronic brain recordings. NEW METHOD: We demonstrate a preclinical testbed that can be used for refining electrode designs embedded in artificial skull replacement pieces or for embedding directly into the skull itself. Options are explored to increase the surface area of the contacts without increasing recording contact diameter to maximize recording resolution. RESULTS: Embedding electrodes in real or artificial skull allows one to lower electrode impedance without increasing the recording contact diameter by making use of conductive channels that extend into the skull. The higher density of small contacts embedded in the artificial skull in this testbed enables one to optimize electrode spacing for use in real bone. COMPARISON WITH EXISTING METHODS: For brain monitoring applications, skull-embedded electrodes fill a gap between electroencephalograms recorded on the scalp surface and the more invasive epidural or subdural electrode sheets. CONCLUSIONS: Embedding electrodes into the skull or in skull replacement pieces may provide a safe, convenient, minimally-invasive alternative for chronic brain monitoring. The manufacturing methods described here will facilitate further testing of skull-embedded electrodes in animal models.

From Trust in Automation to Decision Neuroscience: Applying Cognitive Neuroscience Methods to Understand and Improve Interaction Decisions Involved in Human Automation Interaction.

Human automation interaction (HAI) systems have thus far failed to live up to expectations mainly because human users do not always interact with the automation appropriately. Trust in automation (TiA) has been considered a central influence on the way a human user interacts with an automation; if TiA is too high there will be overuse, if TiA is too low there will be disuse. However, even though extensive research into TiA has identified specific HAI behaviors, or trust outcomes, a unique mapping between trust states and trust outcomes has yet to be clearly identified. Interaction behaviors have been intensely studied in the domain of HAI and TiA and this has led to a reframing of the issues of problems with HAI in terms of reliance and compliance. We find the behaviorally defined terms reliance and compliance to be useful in their functionality for application in real-world situations. However, we note that once an inappropriate interaction behavior has occurred it is too late to mitigate it. We therefore take a step back and look at the interaction decision that precedes the behavior. We note that the decision neuroscience community has revealed that decisions are fairly stereotyped processes accompanied by measurable psychophysiological correlates. Two literatures were therefore reviewed. TiA literature was extensively reviewed in order to understand the relationship between TiA and trust outcomes, as well as to identify gaps in current knowledge. We note that an interaction decision precedes an interaction behavior and believe that we can leverage knowledge of the psychophysiological correlates of decisions to improve joint system performance. As we believe that understanding the interaction decision will be critical to the eventual mitigation of inappropriate interaction behavior, we reviewed the decision making literature and provide a synopsis of the state of the art understanding of the decision process from a decision neuroscience perspective. We forward hypotheses based on this understanding that could shape a research path toward the ability to mitigate interaction behavior in the real world.

A Permutation Test for Unbalanced Paired Comparisons of Global Field Power.

Global field power is a valuable summary of multi-channel electroencephalography data. However, global field power is biased by the noise typical of electroencephalography experiments, so comparisons of global field power on data with unequal noise are invalid. Here, we demonstrate the relationship between the number of trials that contribute to a global field power measure and the expected value of that global field power measure. We also introduce a statistical testing procedure that can be used for multi-subject, repeated-measures (also called within-subjects) comparisons of global field power when the number of trials per condition is unequal across conditions. Simulations demonstrate the effect of unequal trial numbers on global field power comparisons and show the validity of the proposed test in contrast to conventional approaches. Finally, the proposed test and two alternative tests are applied to data collected in a rapid serial visual presentation target detection experiment. The results show that the proposed test finds global field power differences in the classical P3 range; the other tests find differences in that range but also at other times including at times before stimulus onset. These results are interpreted as showing that the proposed test is valid and sensitive to real within-subject differences in global field power in multi-subject unbalanced data.

A regression method for estimating performance in a rapid serial visual presentation target detection task.

BACKGROUND: Estimating target detection performance in the rapid serial visual presentation (RSVP) target detection paradigm can be challenging when the inter-stimulus interval is small relative to the variability in human response time. The challenge arises because assigning a particular response to the correct image cannot be done with certainty. Existing solutions to this challenge establish a heuristic for assigning responses to images and thereby determining which responses are hits and which are false alarms. NEW METHOD: We developed a regression-based method for estimating hit rate and false alarm rate that corrects for expected errors in a likelihood-based assignment of responses to stimuli. RESULTS: Simulations show that this regression method results in an unbiased and accurate estimate of target detection performance. COMPARISON WITH EXISTING METHODS: The regression method had lower estimation error compared to three existing methods, and in contrast to the existing methods, the errors made by the regression method do not depend strongly on the true values of hit rate and false alarm rate. The most commonly used existing method performed well when simulated performance was nearly perfect, but not when behavioral error rates increased. CONCLUSIONS: Based on its better estimation of hit rate and false alarm rate, the regression method proposed here would seem the best choice when estimating the hit rate and false alarm rate is the primary interest.

Improved Neural Signal Classification in a Rapid Serial Visual Presentation Task Using Active Learning.

The application space for brain-computer interface (BCI) technologies is rapidly expanding with improvements in technology. However, most real-time BCIs require extensive individualized calibration prior to use, and systems often have to be recalibrated to account for changes in the neural signals due to a variety of factors including changes in human state, the surrounding environment, and task conditions. Novel approaches to reduce calibration time or effort will dramatically improve the usability of BCI systems. Active Learning (AL) is an iterative semi-supervised learning technique for learning in situations in which data may be abundant, but labels for the data are difficult or expensive to obtain. In this paper, we apply AL to a simulated BCI system for target identification using data from a rapid serial visual presentation (RSVP) paradigm to minimize the amount of training samples needed to initially calibrate a neural classifier. Our results show AL can produce similar overall classification accuracy with significantly less labeled data (in some cases less than 20%) when compared to alternative calibration approaches. In fact, AL classification performance matches performance of 10-fold cross-validation (CV) in over 70% of subjects when training with less than 50% of the data. To our knowledge, this is the first work to demonstrate the use of AL for offline electroencephalography (EEG) calibration in a simulated BCI paradigm. While AL itself is not often amenable for use in real-time systems, this work opens the door to alternative AL-like systems that are more amenable for BCI applications and thus enables future efforts for developing highly adaptive BCI systems.

The effect of target and non-target similarity on neural classification performance: a boost from confidence.

Brain computer interaction (BCI) technologies have proven effective in utilizing single-trial classification algorithms to detect target images in rapid serial visualization presentation tasks. While many factors contribute to the accuracy of these algorithms, a critical aspect that is often overlooked concerns the feature similarity between target and non-target images. In most real-world environments there are likely to be many shared features between targets and non-targets resulting in similar neural activity between the two classes. It is unknown how current neural-based target classification algorithms perform when qualitatively similar target and non-target images are presented. This study address this question by comparing behavioral and neural classification performance across two conditions: first, when targets were the only infrequent stimulus presented amongst frequent background distracters; and second when targets were presented together with infrequent non-targets containing similar visual features to the targets. The resulting findings show that behavior is slower and less accurate when targets are presented together with similar non-targets; moreover, single-trial classification yielded high levels of misclassification when infrequent non-targets are included. Furthermore, we present an approach to mitigate the image misclassification. We use confidence measures to assess the quality of single-trial classification, and demonstrate that a system in which low confidence trials are reclassified through a secondary process can result in improved performance.

Optimization of Single-Trial Detection of Event-Related Potentials Through Artificial Trials.

GOAL: Many brain-computer interface (BCI) classification techniques rely on a large number of labeled brain responses to create efficient classifiers. A large database representing all of the possible variability in the signal is impossible to obtain in a short period of time, and prolonged calibration times prevent efficient BCI use. We propose to improve BCIs based on the detection of event-related potentials (ERPs) in two ways. METHODS: First, we increase the size of the training database by considering additional deformed trials. The creation of the additional deformed trials is based on the addition of Gaussian noise, and on the variability of the ERP latencies. Second, we exploit the variability of the ERP latencies by combining decisions across multiple deformed trials. These new methods are evaluated on data from 16 healthy subjects participating in a rapid serial visual presentation task. RESULTS: The results show a significant increase in the performance of single-trial detection with the addition of artificial trials, and the combination of decisions obtained from altered trials. When the number of trials to train a classifier is low, the proposed approach allows us improve performance from an AUC of 0.533±0.080 to 0.905±0.053. This improvement represents approximately an 80% reduction in classification error. CONCLUSION: These results demonstrate that artificially increasing the training dataset leads to improved single-trial detection. SIGNIFICANCE: Calibration sessions can be shortened for BCIs based on ERP detection.

Sliding HDCA: single-trial EEG classification to overcome and quantify temporal variability.

Patterns of neural data obtained from electroencephalography (EEG) can be classified by machine learning techniques to increase human-system performance. In controlled laboratory settings this classification approach works well; however, transitioning these approaches into more dynamic, unconstrained environments will present several significant challenges. One such challenge is an increase in temporal variability in measured behavioral and neural responses, which often results in suboptimal classification performance. Previously, we reported a novel classification method designed to account for temporal variability in the neural response in order to improve classification performance by using sliding windows in hierarchical discriminant component analysis (HDCA), and demonstrated a decrease in classification error by over 50% when compared to the standard HDCA method (Marathe et al., 2013). Here, we expand upon this approach and show that embedded within this new method is a novel signal transformation that, when applied to EEG signals, significantly improves the signal-to-noise ratio and thereby enables more accurate single-trial analysis. The results presented here have significant implications for both brain-computer interaction technologies and basic science research into neural processes.

Stereoelectroencephalography for continuous two-dimensional cursor control in a brain-machine interface.

Stereoelectroencephalography (SEEG) is becoming more prevalent as a planning tool for surgical treatment of intractable epilepsy. Stereoelectroencephalography uses long, thin, cylindrical "depth" electrodes containing multiple recording contacts along each electrode's length. Each lead is inserted into the brain percutaneously. The advantage of SEEG is that the electrodes can easily target deeper brain structures that are inaccessible with subdural grid electrodes, and SEEG does not require a craniotomy. Brain-machine interface (BMI) research is also becoming more common in the Epilepsy Monitoring Unit. A brain-machine interface decodes a person's desired movement or action from the recorded brain activity and then uses the decoded brain activity to control an assistive device in real time. Although BMIs are primarily being developed for use by severely paralyzed individuals, epilepsy patients undergoing invasive brain monitoring provide an opportunity to test the effectiveness of different invasive recording electrodes for use in BMI systems. This study investigated the ability to use SEEG electrodes for control of 2D cursor velocity in a BMI. Two patients who were undergoing SEEG for intractable epilepsy participated in this study. Participants were instructed to wiggle or rest the hand contralateral to their SEEG electrodes to control the horizontal velocity of a cursor on a screen. Simultaneously they were instructed to wiggle or rest their feet to control the vertical component of cursor velocity. The BMI system was designed to detect power spectral changes associated with hand and foot activity and translate those spectral changes into horizontal and vertical cursor movements in real time. During testing, participants used their decoded SEEG signals to move the brain-controlled cursor to radial targets that appeared on the screen. Although power spectral information from 28 to 32 electrode contacts were used for cursor control during the experiment, post hoc analysis indicated that better control may have been possible using only a single SEEG depth electrode containing multiple recording contacts in both hand and foot cortical areas. These results suggest that the advantages of using SEEG for epilepsy monitoring may also apply to using SEEG electrodes in BMI systems. Specifically, SEEG electrodes can target deeper brain structures, such as foot motor cortex, and both hand and foot areas can be targeted with a single SEEG electrode implanted percutaneously. Therefore, SEEG electrodes may be an attractive option for simple BMI systems that use power spectral modulation in hand and foot cortex for independent control of 2 degrees of freedom.

Virtual reality hardware and graphic display options for brain-machine interfaces.

Virtual reality hardware and graphic displays are reviewed here as a development environment for brain-machine interfaces (BMIs). Two desktop stereoscopic monitors and one 2D monitor were compared in a visual depth discrimination task and in a 3D target-matching task where able-bodied individuals used actual hand movements to match a virtual hand to different target hands. Three graphic representations of the hand were compared: a plain sphere, a sphere attached to the fingertip of a realistic hand and arm, and a stylized pacman-like hand. Several subjects had great difficulty using either stereo monitor for depth perception when perspective size cues were removed. A mismatch in stereo and size cues generated inappropriate depth illusions. This phenomenon has implications for choosing target and virtual hand sizes in BMI experiments. Target-matching accuracy was about as good with the 2D monitor as with either 3D monitor. However, users achieved this accuracy by exploring the boundaries of the hand in the target with carefully controlled movements. This method of determining relative depth may not be possible in BMI experiments if movement control is more limited. Intuitive depth cues, such as including a virtual arm, can significantly improve depth perception accuracy with or without stereo viewing.