Neurophysiological mental fatigue assessment for developing user-centered Artificial Intelligence as a solution for autonomous driving.

The human factor plays a key role in the automotive field since most accidents are due to drivers' unsafe and risky behaviors. The industry is now pursuing two main solutions to deal with this concern: in the short term, there is the development of systems monitoring drivers' psychophysical states, such as inattention and fatigue, and in the medium-long term, there is the development of fully autonomous driving. This second solution is promoted by recent technological progress in terms of Artificial Intelligence and sensing systems aimed at making vehicles more and more accurately aware of their "surroundings." However, even with an autonomous vehicle, the driver should be able to take control of the vehicle when needed, especially during the current transition from the lower (SAE < 3) to the highest level (SAE = 5) of autonomous driving. In this scenario, the vehicle has to be aware not only of its "surroundings" but also of the driver's psychophysical state, i.e., a user-centered Artificial Intelligence. The neurophysiological approach is one the most effective in detecting improper mental states. This is particularly true if considering that the more automatic the driving will be, the less available the vehicular data related to the driver's driving style. The present study aimed at employing a holistic approach, considering simultaneously several neurophysiological parameters, in particular, electroencephalographic, electrooculographic, photopletismographic, and electrodermal activity data to assess the driver's mental fatigue in real time and to detect the onset of fatigue increasing. This would ideally work as an information/trigger channel for the vehicle AI. In all, 26 professional drivers were engaged in a 45-min-lasting realistic driving task in simulated conditions, during which the previously listed biosignals were recorded. Behavioral (reaction times) and subjective measures were also collected to validate the experimental design and to support the neurophysiological results discussion. Results showed that the most sensitive and timely parameters were those related to brain activity. To a lesser extent, those related to ocular parameters were also sensitive to the onset of mental fatigue, but with a delayed effect. The other investigated parameters did not significantly change during the experimental session.

Vision-Based Driver's Cognitive Load Classification Considering Eye Movement Using Machine Learning and Deep Learning.

Due to the advancement of science and technology, modern cars are highly technical, more activity occurs inside the car and driving is faster; however, statistics show that the number of road fatalities have increased in recent years because of drivers' unsafe behaviors. Therefore, to make the traffic environment safe it is important to keep the driver alert and awake both in human and autonomous driving cars. A driver's cognitive load is considered a good indication of alertness, but determining cognitive load is challenging and the acceptance of wire sensor solutions are not preferred in real-world driving scenarios. The recent development of a non-contact approach through image processing and decreasing hardware prices enables new solutions and there are several interesting features related to the driver's eyes that are currently explored in research. This paper presents a vision-based method to extract useful parameters from a driver's eye movement signals and manual feature extraction based on domain knowledge, as well as automatic feature extraction using deep learning architectures. Five machine learning models and three deep learning architectures are developed to classify a driver's cognitive load. The results show that the highest classification accuracy achieved is 92% by the support vector machine model with linear kernel function and 91% by the convolutional neural networks model. This non-contact technology can be a potential contributor in advanced driver assistive systems.

Study on human subjects - influence of stress and alcohol in simulated traffic situations.

This report presents a research study plan on human subjects - the influence of stress and alcohol in simulated traffic situations under an H2020 project named SIMUSAFE. This research study focuses on road-users', i.e., car drivers, motorcyclists, bicyclists and pedestrians, behaviour in relation to retrospective studies, where interaction between the users are considered. Here, the study includes sample size, inclusion/exclusion criteria, detailed study plan, protocols, potential test scenarios and all related ethical issues. The study plan has been included in a national ethics application and received approval for implementation.

A Novel Mutual Information Based Feature Set for Drivers' Mental Workload Evaluation Using Machine Learning.

Analysis of physiological signals, electroencephalography more specifically, is considered a very promising technique to obtain objective measures for mental workload evaluation, however, it requires a complex apparatus to record, and thus, with poor usability in monitoring in-vehicle drivers' mental workload. This study proposes a methodology of constructing a novel mutual information-based feature set from the fusion of electroencephalography and vehicular signals acquired through a real driving experiment and deployed in evaluating drivers' mental workload. Mutual information of electroencephalography and vehicular signals were used as the prime factor for the fusion of features. In order to assess the reliability of the developed feature set mental workload score prediction, classification and event classification tasks were performed using different machine learning models. Moreover, features extracted from electroencephalography were used to compare the performance. In the prediction of mental workload score, expert-defined scores were used as the target values. For classification tasks, true labels were set from contextual information of the experiment. An extensive evaluation of every prediction tasks was carried out using different validation methods. In predicting the mental workload score from the proposed feature set lowest mean absolute error was 0.09 and for classifying mental workload highest accuracy was 94%. According to the outcome of the study, it can be stated that the novel mutual information based features developed through the proposed approach can be employed to classify and monitor in-vehicle drivers' mental workload.

Towards Intelligent Data Analytics: A Case Study in Driver Cognitive Load Classification.

One debatable issue in traffic safety research is that the cognitive load by secondary tasks reduces primary task performance, i.e., driving. In this paper, the study adopted a version of the n-back task as a cognitively loading secondary task on the primary task, i.e., driving; where drivers drove in three different simulated driving scenarios. This paper has taken a multimodal approach to perform 'intelligent multivariate data analytics' based on machine learning (ML). Here, the k-nearest neighbour (k-NN), support vector machine (SVM), and random forest (RF) are used for driver cognitive load classification. Moreover, physiological measures have proven to be sophisticated in cognitive load identification, yet it suffers from confounding factors and noise. Therefore, this work uses multi-component signals, i.e., physiological measures and vehicular features to overcome that problem. Both multiclass and binary classifications have been performed to distinguish normal driving from cognitive load tasks. To identify the optimal feature set, two feature selection algorithms, i.e., sequential forward floating selection (SFFS) and random forest have been applied where out of 323 features, a subset of 42 features has been selected as the best feature subset. For the classification, RF has shown better performance with F1-score of 0.75 and 0.80 than two other algorithms. Moreover, the result shows that using multicomponent features classifiers could classify better than using features from a single source.

Open-Source Data Collection and Data Sets for Activity Recognition in Smart Homes.

As research in smart homes and activity recognition is increasing, it is of ever increasing importance to have benchmarks systems and data upon which researchers can compare methods. While synthetic data can be useful for certain method developments, real data sets that are open and shared are equally as important. This paper presents the E-care@home system, its installation in a real home setting, and a series of data sets that were collected using the E-care@home system. Our first contribution, the E-care@home system, is a collection of software modules for data collection, labeling, and various reasoning tasks such as activity recognition, person counting, and configuration planning. It supports a heterogeneous set of sensors that can be extended easily and connects collected sensor data to higher-level Artificial Intelligence (AI) reasoning modules. Our second contribution is a series of open data sets which can be used to recognize activities of daily living. In addition to these data sets, we describe the technical infrastructure that we have developed to collect the data and the physical environment. Each data set is annotated with ground-truth information, making it relevant for researchers interested in benchmarking different algorithms for activity recognition.

Non-Contact Physiological Parameters Extraction Using Facial Video Considering Illumination, Motion, Movement and Vibration.

OBJECTIVE: In this paper, four physiological parameters, i.e., heart rate (HR), inter-beat-interval (IBI), heart rate variability (HRV), and oxygen saturation (SpO2), are extracted from facial video recordings. METHODS: Facial videos were recorded for 10 min each in 30 test subjects while driving a simulator. Four regions of interest (ROIs) are automatically selected in each facial image frame based on 66 facial landmarks. Red-green-blue color signals are extracted from the ROIs and four physiological parameters are extracted from the color signals. For the evaluation, physiological parameters are also recorded simultaneously using a traditional sensor "cStress," which is attached to hands and fingers of test subjects. RESULTS: The Bland Altman plots show 95% agreement between the camera system and "cStress" with the highest correlation coefficient R = 0.96 for both HR and SpO2. The quality index is estimated for IBI considering 100 ms R-peak error; the accumulated percentage achieved is 97.5%. HRV features in both time and frequency domains are compared and the highest correlation coefficient achieved is 0.93. One-way analysis of variance test shows that there are no statistically significant differences between the measurements by camera and reference sensors. CONCLUSION: These results present high degrees of accuracy of HR, IBI, HRV, and SpO2 extraction from facial image sequences. SIGNIFICANCE: The proposed non-contact approach could broaden the dimensionality of physiological parameters extraction using cameras. This proposed method could be applied for driver monitoring application under realistic conditions, i.e., illumination, motion, movement, and vibration.

Automated EEG Artifact Handling With Application in Driver Monitoring.

Automated analyses of electroencephalographic (EEG) signals acquired in naturalistic environments are becoming increasingly important in areas such as brain-computer interfaces and behavior science. However, the recorded EEG in such environments is often heavily contaminated by motion artifacts and eye movements. This poses new requirements on artifact handling. The objective of this paper is to present an automated EEG artifacts handling algorithm, which will be used as a preprocessing step in a driver monitoring application. The algorithm, named Automated aRTifacts handling in EEG (ARTE), is based on wavelets, independent component analysis, and hierarchical clustering. The algorithm is tested on a dataset obtained from a driver sleepiness study including 30 drivers and 540 30-min 30-channel EEG recordings. The algorithm is evaluated by a clinical neurophysiologist, by quantitative criteria (signal quality index, mean square error, relative error, and mean absolute error), and by demonstrating its usefulness as a preprocessing step in driver monitoring, here exemplified with driver sleepiness classification. All results are compared with a state-of-the-art algorithm called FORCe. The quantitative and expert evaluation results show that the two algorithms are comparable, and that both algorithms significantly reduce the impact of artifacts in recorded EEG signals. When artifact handling is used as a preprocessing step in driver sleepiness classification, the classification accuracy increased by 5% when using ARTE and by 2% when using FORCe. The advantage with ARTE is that it is data driven and does not rely on additional reference signals or manually defined thresholds, making it well suited for use in dynamic settings where unforeseen and rare artifacts are commonly encountered.

Quality Index Analysis on Camera-Based R-Peak Identification Considering Movements and Light Illumination.

This paper presents a quality index (QI) analysis on R-peak extracted by a camera system considering movements and light illumination. Here, the proposed camera system is compared with a reference system named Shimmer PPG sensor. The study considers five test subjects with a 15 minutes measurement protocol, where the protocol consists of several conditions. The conditions are: normal sittings, head movements i.e., up/down/left/right/forward/backword, with light on/off and with moving flash on/off. A percentage of corrected R-peaks are calculated based on time difference in milliseconds (MS) between the R-peaks extracted both from camera-based and sensor-based systems. A comparison results between normal, movements, and lighting condition is presented as individual and group wise. Furthermore, the comparison is extended considering gender and origin of the subjects. According to the results, more than 90% R-peaks are correctly identified by the camera system with ±200 MS time differences, however, it decreases with while there is no light than when it is on. At the same time, the camera system shows more 95% accuracy for European than Asian men.

Classifying Drivers' Cognitive Load Using EEG Signals.

A growing traffic safety issue is the effect of cognitive loading activities on traffic safety and driving performance. To monitor drivers' mental state, understanding cognitive load is important since while driving, performing cognitively loading secondary tasks, for example talking on the phone, can affect the performance in the primary task, i.e. driving. Electroencephalography (EEG) is one of the reliable measures of cognitive load that can detect the changes in instantaneous load and effect of cognitively loading secondary task. In this driving simulator study, 1-back task is carried out while the driver performs three different simulated driving scenarios. This paper presents an EEG based approach to classify a drivers' level of cognitive load using Case-Based Reasoning (CBR). The results show that for each individual scenario as well as using data combined from the different scenarios, CBR based system achieved approximately over 70% of classification accuracy.

Non-Contact Heart Rate Monitoring Using Lab Color Space.

Research progressing during the last decade focuses more on non-contact based systems to monitor Heart Rate (HR) which are simple, low-cost and comfortable to use. Most of the non-contact based systems are using RGB videos which is suitable for lab environment. However, it needs to progress considerably before they can be applied in real life applications. As luminance (light) has significance contribution on RGB videos HR monitoring using RGB videos are not efficient enough in real life applications in outdoor environment. This paper presents a HR monitoring method using Lab color facial video captured by a webcam of a laptop computer. Lab color space is device independent and HR can be extracted through facial skin color variation caused by blood circulation considering variable environmental light. Here, three different signal processing methods i.e., Fast Fourier Transform (FFT), Independent Component Analysis (ICA) and Principal Component Analysis (PCA) have been applied on the color channels in video recordings and blood volume pulse (BVP) has been extracted from the facial regions. In this study, HR is subsequently quantified and compare with a reference measurement. The result shows that high degrees of accuracy have been achieved compared to the reference measurements. Thus, this technology has significant potential for advancing personal health care, telemedicine and many real life applications such as driver monitoring.

Supervised machine learning algorithms to diagnose stress for vehicle drivers based on physiological sensor signals.

Machine learning algorithms play an important role in computer science research. Recent advancement in sensor data collection in clinical sciences lead to a complex, heterogeneous data processing, and analysis for patient diagnosis and prognosis. Diagnosis and treatment of patients based on manual analysis of these sensor data are difficult and time consuming. Therefore, development of Knowledge-based systems to support clinicians in decision-making is important. However, it is necessary to perform experimental work to compare performances of different machine learning methods to help to select appropriate method for a specific characteristic of data sets. This paper compares classification performance of three popular machine learning methods i.e., case-based reasoning, neutral networks and support vector machine to diagnose stress of vehicle drivers using finger temperature and heart rate variability. The experimental results show that case-based reasoning outperforms other two methods in terms of classification accuracy. Case-based reasoning has achieved 80% and 86% accuracy to classify stress using finger temperature and heart rate variability. On contrary, both neural network and support vector machine have achieved less than 80% accuracy by using both physiological signals.

A personalized health-monitoring system for elderly by combining rules and case-based reasoning.

Health-monitoring system for elderly in home environment is a promising solution to provide efficient medical services that increasingly interest by the researchers within this area. It is often more challenging when the system is self-served and functioning as personalized provision. This paper proposed a personalized self-served health-monitoring system for elderly in home environment by combining general rules with a case-based reasoning approach. Here, the system generates feedback, recommendation and alarm in a personalized manner based on elderly's medical information and health parameters such as blood pressure, blood glucose, weight, activity, pulse, etc. A set of general rules has used to classify individual health parameters. The case-based reasoning approach is used to combine all different health parameters, which generates an overall classification of health condition. According to the evaluation result considering 323 cases and k=2 i.e., top 2 most similar retrieved cases, the sensitivity, specificity and overall accuracy are achieved as 90%, 97% and 96% respectively. The preliminary result of the system is acceptable since the feedback; recommendation and alarm messages are personalized and differ from the general messages. Thus, this approach could be possibly adapted for other situations in personalized elderly monitoring.

A Generic System-Level Framework for Self-Serve Health Monitoring System through Internet of Things (IoT).

Sensor data are traveling from sensors to a remote server, data is analyzed remotely in a distributed manner, and health status of a user is presented in real-time. This paper presents a generic system-level framework for a self-served health monitoring system through the Internet of Things (IoT) to facilities an efficient sensor data management.

Physiological sensor signals classification for healthcare using sensor data fusion and case-based reasoning.

Today, clinicians often do diagnosis and classification of diseases based on information collected from several physiological sensor signals. However, sensor signal could easily be vulnerable to uncertain noises or interferences and due to large individual variations sensitivity to different physiological sensors could also vary. Therefore, multiple sensor signal fusion is valuable to provide more robust and reliable decision. This paper demonstrates a physiological sensor signal classification approach using sensor signal fusion and case-based reasoning. The proposed approach has been evaluated to classify Stressed or Relaxed individuals using sensor data fusion. Physiological sensor signals i.e., Heart Rate (HR), Finger Temperature (FT), Respiration Rate (RR), Carbon dioxide (CO2) and Oxygen Saturation (SpO2) are collected during the data collection phase. Here, sensor fusion has been done in two different ways: (i) decision-level fusion using features extracted through traditional approaches; and (ii) data-level fusion using features extracted by means of Multivariate Multiscale Entropy (MMSE). Case-Based Reasoning (CBR) is applied for the classification of the signals. The experimental result shows that the proposed system could classify Stressed or Relaxed individual 87.5% accurately compare to an expert in the domain. So, it shows promising result in the psychophysiological domain and could be possible to adapt this approach to other relevant healthcare systems.

Data mining for wearable sensors in health monitoring systems: a review of recent trends and challenges.

The past few years have witnessed an increase in the development of wearable sensors for health monitoring systems. This increase has been due to several factors such as development in sensor technology as well as directed efforts on political and stakeholder levels to promote projects which address the need for providing new methods for care given increasing challenges with an aging population. An important aspect of study in such system is how the data is treated and processed. This paper provides a recent review of the latest methods and algorithms used to analyze data from wearable sensors used for physiological monitoring of vital signs in healthcare services. In particular, the paper outlines the more common data mining tasks that have been applied such as anomaly detection, prediction and decision making when considering in particular continuous time series measurements. Moreover, the paper further details the suitability of particular data mining and machine learning methods used to process the physiological data and provides an overview of the properties of the data sets used in experimental validation. Finally, based on this literature review, a number of key challenges have been outlined for data mining methods in health monitoring systems.

Physical activity classification for elderly based on pulse rate.

Physical activity is one of the key components for elderly in order to be actively ageing. However, it is difficult to differentiate and identify the body movement and actual physical activity using only accelerometer measurements. Therefore, this paper presents an application of a case-based retrieval classification scheme to classify the physical activity of elderly based on pulse rate measurements. Here, a case-based retrieval approach used the features extracted from both time and frequency domain. The evaluation result shows the best accuracy performance while considering the combination of time and frequency domain features. According to the evaluation result while considering the control measurements, the sensitivity, specificity and overall accuracy are achieved as 95%, 96% and 96%, respectively. Considering the test dataset, the system succeeded to identify 13 physical activities out of 16, i.e,. the percentage of the correctness was 81%.

A multi-module case-based biofeedback system for stress treatment.

OBJECTIVE: Biofeedback is today a recognized treatment method for a number of physical and psychological problems. Experienced clinicians often achieve good results in these areas and their success largely builds on many years of experience and often thousands of treated patients. Unfortunately many of the areas where biofeedback is used are very complex, e.g. diagnosis and treatment of stress. Less experienced clinicians may even have difficulties to initially classify the patient correctly. Often there are only a few experts available to assist less experienced clinicians. To reduce this problem we propose a computer-assisted biofeedback system helping in classification, parameter setting and biofeedback training. METHODS: The decision support system (DSS) analysis finger temperature in time series signal where the derivative of temperature in time is calculated to extract the features. The case-based reasoning (CBR) is used in three modules to classify a patient, estimate parameters and biofeedback. In each and every module the CBR approach retrieves most similar cases by comparing a new finger temperature measurement with previously solved measurements. Three different methods are used to calculate similarity between features, they are: modified distance function, similarity matrix and fuzzy similarity. RESULTS AND CONCLUSION: We explore how such a DSS can be designed and validated the approach in the area of stress where the system assists in the classification, parameter setting and finally in the training. In this case study we show that the case based biofeedback system outperforms trainee clinicians based on a case library of cases authorized by an expert.