Affective response to volitional input perturbations in obstacle avoidance and target tracking games.

We present the use of two game-like tasks, Catnip and Dinorun, to explore affective responses to volitional control perturbations. We analyze behavioral and physiological measures with the self-assessment manikin (SAM), pupillometry, and electroencephalography (EEG) responses to provide intratrial emotional state as well as inter-trial correlates with selfreported survey responses. We find that subject gameplay characteristics significantly correlate with valence and dominance scores for both games, and that perturbations to the games produce a measurable decrease in response scores for Dinorun. During perturbation events, pupillometry analysis reveals considerable SAM-agnostic dilation, with stronger responses in more rigid trialized event structures. Furthermore, analyses of neural activity from central and parietal regions demonstrate significant measurable evoked responses to perturbed events across the majority of subjects for both games. By introducing perturbations, this set of experiments and analyses inform and enable further studies of affective responses to the loss of volitional control during engaging, game-like tasks.

A Wearable Multi-Modal Bio-Sensing System Towards Real-World Applications.

Multi-modal bio-sensing has recently been used as effective research tools in affective computing, autism, clinical disorders, and virtual reality among other areas. However, none of the existing bio-sensing systems support multi-modality in a wearable manner outside well-controlled laboratory environments with research-grade measurements. This paper attempts to bridge this gap by developing a wearable multi-modal bio-sensing system capable of collecting, synchronizing, recording, and transmitting data from multiple bio-sensors: PPG, EEG, eye-gaze headset, body motion capture, GSR, etc., while also providing task modulation features including visual-stimulus tagging. This study describes the development and integration of various components of our system. We evaluate the developed sensors by comparing their measurements to those obtained by a standard research-grade bio-sensors. We first evaluate different sensor modalities of our headset, namely, earlobe-based PPG module with motion-noise canceling for ECG during heart-beat calculation. We also compare the steady-state visually evoked potentials measured by our shielded dry EEG sensors with the potentials obtained by commercially available dry EEG sensors. We also investigate the effect of head movements on the accuracy and precision of our wearable eye-gaze system. Furthermore, we carry out two practical tasks to demonstrate the applications of using multiple sensor modalities for exploring previously unanswerable questions in bio-sensing. Specifically, utilizing bio-sensing, we show which strategy works best for playing "Where is Waldo?" visual-search game, changes in EEG corresponding to true vs. false target fixations in this game, and predicting the loss/draw/win states through bio-sensing modalities while learning their limitations in a "Rock-Paper-Scissors" game.

Mental State Assessment and Validation Using Personalized Physiological Biometrics.

Mental state monitoring is a critical component of current and future human-machine interfaces, including semi-autonomous driving and flying, air traffic control, decision aids, training systems, and will soon be integrated into ubiquitous products like cell phones and laptops. Current mental state assessment approaches supply quantitative measures, but their only frame of reference is generic population-level ranges. What is needed are physiological biometrics that are validated in the context of task performance of individuals. Using curated intake experiments, we are able to generate personalized models of three key biometrics as useful indicators of mental state; namely, mental fatigue, stress, and attention. We demonstrate improvements to existing approaches through the introduction of new features. Furthermore, addressing the current limitations in assessing the efficacy of biometrics for individual subjects, we propose and employ a multi-level validation scheme for the biometric models by means of k-fold cross-validation for discrete classification and regression testing for continuous prediction. The paper not only provides a unified pipeline for extracting a comprehensive mental state evaluation from a parsimonious set of sensors (only EEG and ECG), but also demonstrates the use of validation techniques in the absence of empirical data. Furthermore, as an example of the application of these models to novel situations, we evaluate the significance of correlations of personalized biometrics to the dynamic fluctuations of accuracy and reaction time on an unrelated threat detection task using a permutation test. Our results provide a path toward integrating biometrics into augmented human-machine interfaces in a judicious way that can help to maximize task performance.