EEG correlates to perceived urgency elicited by vibration stimulation of the upper body.

Conveying information effectively while minimizing user distraction is critical to human-computer interaction. As the proliferation of audio-visual communication pushes human information processing capabilities to the limit, researchers are turning their attention to haptic interfaces. Haptic feedback has the potential to create a desirable sense of urgency that allows users to selectively focus on events/tasks or process presented information with minimal distraction or annoyance. There is a growing interest in understanding the neural mechanisms associated with haptic stimulation. In this study, we aim to investigate the EEG correlates associated with the perceived urgency elicited by vibration stimuli on the upper body using a haptic vest. A total of 31 participants enrolled in this experiment and were exposed to three conditions: no vibration pattern (NVP), urgent vibration pattern (UVP), and very urgent vibration pattern (VUVP). Through self-reporting, participants confirmed that the vibration patterns elicited significantly different levels of perceived urgency (Friedman test, Holm-Bonferroni correction, p < 0.01). Furthermore, neural analysis revealed that the power spectral density of the delta, theta, and alpha frequency bands in the middle central area (C1, Cz, and C2) significantly increased for the UVP and VUVP conditions as compared to the NVP condition (One-way ANOVA test, Holm-Bonferroni correction, p < 0.01). While the perceptual experience of haptic-induced urgency is well studied with self-reporting and behavioral evidence, this is the first effort to evaluate the neural correlates to haptic-induced urgency using EEG. Further research is warranted to identify unique correlates to the cognitive processes associated with urgency from sensory feedback correlates.

Individual-finger motor imagery classification: a data-driven approach with Shapley-informed augmentation.

Objective. Classifying motor imagery (MI) tasks that involve fine motor control of the individual five fingers presents unique challenges when utilizing electroencephalography (EEG) data. In this paper, we systematically assess the classification of MI functions for the individual five fingers using single-trial time-domain EEG signals. This assessment encompasses both within-subject and cross-subject scenarios, supported by data-driven analysis that provides statistical validation of the neural correlate that could potentially discriminate between the five fingers.Approach. We present Shapley-informed augmentation, an informed approach to enhance within-subject classification accuracy. This method is rooted in insights gained from our data-driven analysis, which revealed inconsistent temporal features encoding the five fingers MI across sessions of the same subject. To evaluate its impact, we compare within-subject classification performance both before and after implementing this augmentation technique.Main results. Both the data-driven approach and the model explainability analysis revealed that the parietal cortex contains neural information that helps discriminate the individual five fingers' MI apart. Shapley-informed augmentation successfully improved classification accuracy in sessions severely affected by inconsistent temporal features. The accuracy for sessions impacted by inconsistency in their temporal features increased by an average of26.3%±6.70, thereby enabling a broader range of subjects to benefit from brain-computer interaction (BCI) applications involving five-fingers MI classification. Conversely, non-impacted sessions experienced only a negligible average accuracy decrease of2.01±5.44%. The average classification accuracy achieved is around 60.0% (within-session), 50.0% (within-subject) and 40.0% (leave-one-subject-out).Significance. This research offers data-driven evidence of neural correlates that could discriminate between the individual five fingers MI and introduces a novel Shapley-informed augmentation method to address temporal variability of features, ultimately contributing to the development of personalized systems.

Assessing handwriting task difficulty levels through kinematic features: a deep-learning approach.

Introduction: Handwriting is a complex task that requires coordination of motor, sensory, cognitive, memory, and linguistic skills to master. The extent these processes are involved depends on the complexity of the handwriting task. Evaluating the difficulty of a handwriting task is a challenging problem since it relies on subjective judgment of experts. Methods: In this paper, we propose a machine learning approach for evaluating the difficulty level of handwriting tasks. We propose two convolutional neural network (CNN) models for single- and multilabel classification where single-label classification is based on the mean of expert evaluation while the multilabel classification predicts the distribution of experts' assessment. The models are trained with a dataset containing 117 spatio-temporal features from the stylus and hand kinematics, which are recorded for all letters of the Arabic alphabet. Results: While single- and multilabel classification models achieve decent accuracy (96% and 88% respectively) using all features, the hand kinematics features do not significantly influence the performance of the models. Discussion: The proposed models are capable of extracting meaningful features from the handwriting samples and predicting their difficulty levels accurately. The proposed approach has the potential to be used to personalize handwriting learning tools and provide automatic evaluation of the quality of handwriting.

An ensemble deep-learning approach for single-trial EEG classification of vibration intensity.

Objective. Single-trial electroencephalography (EEG) classification is a promising approach to evaluate the cognitive experience associated with haptic feedback. Convolutional neural networks (CNNs), which are among the most widely used deep learning techniques, have demonstrated their effectiveness in extracting EEG features for the classification of different cognitive functions, including the perception of vibration intensity that is often experienced during human-computer interaction. This paper proposes a novel CNN ensemble model to classify the vibration-intensity from a single trial EEG data that outperforms the state-of-the-art EEG models.Approach. The proposed ensemble model, named SE NexFusion, builds upon the observed complementary learning behaviors of the EEGNex and TCNet Fusion models, exhibited in learning personal as well generic neural features associated with vibration intensity. The proposed ensemble employs multi-branch feature encoders corroborated with squeeze-and-excitation units that enables rich-feature encoding while at the same time recalibrating the weightage of the obtained feature maps based on their discriminative power. The model takes in a single trial of raw EEG as an input and does not require complex EEG signal-preprocessing.Main results. The proposed model outperforms several state-of-the-art bench-marked EEG models by achieving an average accuracy of 60.7% and 61.6% under leave-one-subject-out and within-subject cross-validation (three-classes), respectively. We further validate the robustness of the model through Shapley values explainability method, where the most influential spatio-temporal features of the model are counter-checked with the neural correlates that encode vibration intensity.Significance. Results show that SE NexFusion outperforms other benchmarked EEG models in classifying the vibration intensity. Additionally, explainability analysis confirms the robustness of the model in attending to features associated with the neural correlates of vibration intensity.

EEG-Based Machine Learning Models to Evaluate Haptic Delay: Should We Label Data Based on Self-Reporting or Physical Stimulation?

Reliable haptic interfaces augment human-computer interaction via simulated tactile and kinesthetic feedback. As haptic technologies advance, user experience evaluation becomes more crucial. Conventionally, self-reporting is used to evaluate haptic experiences; however, it could be inconsistent or imprecise due to human error. A promising alternative is using neurocognitive methods with machine or deep learning models to evaluate the human haptic experience. Machine and deep learning models can be trained on Electroencephalography (EEG) data labeled based on self-report or actual physical stimulation. As the literature lacks a systematic study on which approach is more robust, we develop a visuo-haptic task to answer this question by examining an important haptic experience, namely, haptic delay. EEG is recorded during the experiment, and participants report whether they detected a delay in the haptic modality through self-report. Four machine/deep learning models were trained twice on the EEG data using the two labeling methods. Models trained with labels from the physical stimuli significantly outperformed those trained with self-reporting labels. Although this finding holds true for one particular haptic experience (haptic delay), it cannot be extrapolated to others; rather, it suggests that EEG data labeling plays a prominent role in evaluating the haptic experience through neurocognitive methods.

Assessment of EEG-based functional connectivity in response to haptic delay.

Haptic technologies enable users to physically interact with remote or virtual environments by applying force, vibration, or motion via haptic interfaces. However, the delivery of timely haptic feedback remains a challenge due to the stringent computation and communication requirements associated with haptic data transfer. Haptic delay disrupts the realism of the user experience and interferes with the quality of interaction. Research efforts have been devoted to studying the neural correlates of delayed sensory stimulation to better understand and thus mitigate the impact of delay. However, little is known about the functional neural networks that process haptic delay. This paper investigates the underlying neural networks associated with processing haptic delay in passive and active haptic interactions. Nineteen participants completed a visuo-haptic task using a computer screen and a haptic device while electroencephalography (EEG) data were being recorded. A combined approach based on phase locking value (PLV) functional connectivity and graph theory was used. To assay the effects of haptic delay on functional connectivity, we evaluate a global connectivity property through the small-worldness index and a local connectivity property through the nodal strength index. Results suggest that the brain exhibits significantly different network characteristics when a haptic delay is introduced. Haptic delay caused an increased manifestation of the small-worldness index in the delta and theta bands as well as an increased nodal strength index in the middle central region. Inter-regional connectivity analysis showed that the middle central region was significantly connected to the parietal and occipital regions as a result of haptic delay. These results are expected to indicate the detection of conflicting visuo-haptic information at the middle central region and their respective resolution and integration at the parietal and occipital regions.

An ensemble deep learning approach to evaluate haptic delay from a single trial EEG data.

Haptic technologies are becoming increasingly valuable in Human-Computer interaction systems as they provide means of physical interaction with a remote or virtual environment. One of the persistent challenges in tele-haptic systems, communicating haptic information over a computer network, is the synchrony of the delivered haptic information with the rest of the sensory modalities. Delayed haptic feedback can have serious implications on the user performance and overall experience. Limited research efforts have been devoted to studying the implication of haptic delay on the human neural response and relating it to the overall haptic experience. Deep learning could offer autonomous brain activity interpretation in response to a haptic experience such as haptic delay. In this work, we propose an ensemble of 2D CNN and transformer models that is capable of detecting the presence and redseverity of haptic delay from a single-trial Electroencephalography data. Two EEG-based experiments involving visuo-haptic interaction tasks are proposed. The first experiment aims to collect data for detecting the presence of haptic delay during discrete force feedback using a bouncing ball on a racket simulation, while the second aims to collect data for detecting the severity level (none, mild, moderate, severe) of the haptic delay during continuous force feedback via grasping/releasing of an object in a bucket. The ensemble model showed a promising performance with an accuracy of 0.9142 ± 0.0157 for detecting haptic delay during discrete force feedback and 0.6625 ± 0.0067 for classifying the severity of haptic delay during continuous force feedback (4 levels). These results were obtained based on training the model with raw EEG data as well as their wavelet transform using several wavelet kernels. This study is a step forward towards developing cognitive evaluation of the user experience while interaction with haptic interfaces.

Midfrontal theta power encodes the value of haptic delay.

The use of haptic technologies in modern life scenarios is becoming the new normal particularly in rehabilitation, medical training, and entertainment applications. An evident challenge in haptic telepresence systems is the delay in haptic information. How humans perceive delayed visual and audio information has been extensively studied, however, the same for haptically delayed environments remains largely unknown. Here, we develop a visuo-haptic experimental setting that simulates pick and place task and involves continuous haptic feedback stimulation with four possible haptic delay levels. The setting is built using a haptic device and a computer screen. We use electroencephalography (EEG) to study the neural correlates that could be used to identify the amount of the experienced haptic delay. EEG data were collected from 34 participants. Results revealed that midfrontal theta oscillation plays a pivotal role in quantifying the amount of haptic delay while parietal alpha showed a significant modulation that encodes the presence of haptic delay. Based on the available literature, these results suggest that the amount of haptic delay is proportional to the neural activation that is associated with conflict detection and resolution as well as for multi-sensory divided attention.

Midfrontal theta oscillation encodes haptic delay.

Haptic technologies aim to simulate tactile or kinesthetic interactions with a physical or virtual environment in order to enhance user experience and/or performance. However, due to stringent communication and computational needs, the user experience is influenced by delayed haptic feedback. While delayed feedback is well understood in the visual and auditory modalities, little research has systematically examined the neural correlates associated with delayed haptic feedback. In this paper, we used electroencephalography (EEG) to study sensory and cognitive neural correlates caused by haptic delay during passive and active tasks performed using a haptic device and a computer screen. Results revealed that theta power oscillation was significantly higher at the midfrontal cortex under the presence of haptic delay. Sensory correlates represented by beta rebound were found to be similar in the passive task and different in the active task under the delayed and synchronous conditions. Additionally, the event related potential (ERP) P200 component is modulated under the haptic delay condition during the passive task. The P200 amplitude significantly reduced in the last 20% of trials during the passive task and in the absence of haptic delay. Results suggest that haptic delay could be associated with increased cognitive control processes including multi-sensory divided attention followed by conflict detection and resolution with an earlier detection during the active task. Additionally, haptic delay tends to generate greater perceptual attention that does not significantly decay across trials during the passive task.

FaceGuard: A Wearable System To Avoid Face Touching.

Most people touch their faces unconsciously, for instance to scratch an itch or to rest one's chin in their hands. To reduce the spread of the novel coronavirus (COVID-19), public health officials recommend against touching one's face, as the virus is transmitted through mucous membranes in the mouth, nose and eyes. Students, office workers, medical personnel and people on trains were found to touch their faces between 9 and 23 times per hour. This paper introduces FaceGuard, a system that utilizes deep learning to predict hand movements that result in touching the face, and provides sensory feedback to stop the user from touching the face. The system utilizes an inertial measurement unit (IMU) to obtain features that characterize hand movement involving face touching. Time-series data can be efficiently classified using 1D-Convolutional Neural Network (CNN) with minimal feature engineering; 1D-CNN filters automatically extract temporal features in IMU data. Thus, a 1D-CNN based prediction model is developed and trained with data from 4,800 trials recorded from 40 participants. Training data are collected for hand movements involving face touching during various everyday activities such as sitting, standing, or walking. Results showed that while the average time needed to touch the face is 1,200 ms, a prediction accuracy of more than 92% is achieved with less than 550 ms of IMU data. As for the sensory response, the paper presents a psychophysical experiment to compare the response time for three sensory feedback modalities, namely visual, auditory, and vibrotactile. Results demonstrate that the response time is significantly smaller for vibrotactile feedback (427.3 ms) compared to visual (561.70 ms) and auditory (520.97 ms). Furthermore, the success rate (to avoid face touching) is also statistically higher for vibrotactile and auditory feedback compared to visual feedback. These results demonstrate the feasibility of predicting a hand movement and providing timely sensory feedback within less than a second in order to avoid face touching.

Screen Printed Passives and Interconnects on Bio-Degradable Medical Hydrocolloid Dressing for Wearable Sensors.

The healthcare system is undergoing a noticeable transformation from a reactive, post-disease treatment to a preventive, predictive continuous healthcare. The key enabler for such a system is a pervasive wearable platform. Several technologies have been suggested and implemented as a wearable platform, but these technologies either lack reliability, manufacturing practicability or pervasiveness. We propose a screen-printed circuit board on bio-degradable hydrocolloid dressings, which are medically used and approved, as a platform for wearable biomedical sensors to overcome the aforementioned problems. We experimentally characterize and prepare the surface of the hydrocolloid and demonstrate high-quality screen-printed passive elements and interconnects on its surface using conductive silver paste. We also propose appropriate models of the thick-film screen-printed passives, validated through measurements and FEM simulations. We further elucidate on the usage of the hydrocolloid dressing by prototyping a Wireless Power Transfer (WPT) sensor and a humidity sensor using printed spiral inductors and interdigital capacitors, respectively.