Sensing technologies and machine learning methods for emotion recognition in autism: Systematic review.

BACKGROUND: Human Emotion Recognition (HER) has been a popular field of study in the past years. Despite the great progresses made so far, relatively little attention has been paid to the use of HER in autism. People with autism are known to face problems with daily social communication and the prototypical interpretation of emotional responses, which are most frequently exerted via facial expressions. This poses significant practical challenges to the application of regular HER systems, which are normally developed for and by neurotypical people. OBJECTIVE: This study reviews the literature on the use of HER systems in autism, particularly with respect to sensing technologies and machine learning methods, as to identify existing barriers and possible future directions. METHODS: We conducted a systematic review of articles published between January 2011 and June 2023 according to the 2020 PRISMA guidelines. Manuscripts were identified through searching Web of Science and Scopus databases. Manuscripts were included when related to emotion recognition, used sensors and machine learning techniques, and involved children with autism, young, or adults. RESULTS: The search yielded 346 articles. A total of 65 publications met the eligibility criteria and were included in the review. CONCLUSIONS: Studies predominantly used facial expression techniques as the emotion recognition method. Consequently, video cameras were the most widely used devices across studies, although a growing trend in the use of physiological sensors was observed lately. Happiness, sadness, anger, fear, disgust, and surprise were most frequently addressed. Classical supervised machine learning techniques were primarily used at the expense of unsupervised approaches or more recent deep learning models. Studies focused on autism in a broad sense but limited efforts have been directed towards more specific disorders of the spectrum. Privacy or security issues were seldom addressed, and if so, at a rather insufficient level of detail.

T1DiabetesGranada: a longitudinal multi-modal dataset of type 1 diabetes mellitus.

Type 1 diabetes mellitus (T1D) patients face daily difficulties in keeping their blood glucose levels within appropriate ranges. Several techniques and devices, such as flash glucose meters, have been developed to help T1D patients improve their quality of life. Most recently, the data collected via these devices is being used to train advanced artificial intelligence models to characterize the evolution of the disease and support its management. Data scarcity is the main challenge for generating these models, as most works use private or artificially generated datasets. For this reason, this work presents T1DiabetesGranada, an open under specific permission longitudinal dataset that not only provides continuous glucose levels, but also patient demographic and clinical information. The dataset includes 257 780 days of measurements spanning four years from 736 T1D patients from the province of Granada, Spain. This dataset advances beyond the state of the art as one the longest and largest open datasets of continuous glucose measurements, thus boosting the development of new artificial intelligence models for glucose level characterization and prediction.

COVID-BEHAVE dataset: measuring human behaviour during the COVID-19 pandemic.

Aiming to illuminate the effects of enforced confinements on people's lives, this paper presents a novel dataset that measures human behaviour holistically and longitudinally during the COVID-19 outbreak. In particular, we conducted a study during the first wave of the lockdown, where 21 healthy subjects from the Netherlands and Greece participated, collecting multimodal raw and processed data from smartphone sensors, activity trackers, and users' responses to digital questionnaires. The study lasted more than two months, although the duration of the data collection varies per participant. The data are publicly available and can be used to model human behaviour in a broad sense as the dataset explores physical, social, emotional, and cognitive domains. The dataset offers an exemplary perspective on a given group of people that could be considered to build new models for investigating behaviour changes as a consequence of the lockdown. Importantly, to our knowledge, this is the first dataset combining passive sensing, experience sampling, and virtual assistants to study human behaviour dynamics in a prolonged lockdown situation.

HIGEA: An Intelligent Conversational Agent to Detect Caregiver Burden.

Mental health disorders increasingly affect people worldwide. As a consequence, more families and relatives find themselves acting as caregivers. Most often, these are untrained people who experience loneliness, abandonment, and often develop signs of depression (i.e., caregiver burden syndrome). In this work, we present HIGEA, a digital system based on a conversational agent to help to detect caregiver burden. The conversational agent naturally embeds psychological test questions into informal conversations, which aim at increasing the adherence of use and avoiding user bias. A proof-of-concept is developed based on the popular Zarit Test, which is widely used to assess caregiver burden. Preliminary results show the system is useful and effective.

Mobile and Wearable Technology for the Monitoring of Diabetes-Related Parameters: Systematic Review.

BACKGROUND: Diabetes mellitus is a metabolic disorder that affects hundreds of millions of people worldwide and causes several million deaths every year. Such a dramatic scenario puts some pressure on administrations, care services, and the scientific community to seek novel solutions that may help control and deal effectively with this condition and its consequences. OBJECTIVE: This study aims to review the literature on the use of modern mobile and wearable technology for monitoring parameters that condition the development or evolution of diabetes mellitus. METHODS: A systematic review of articles published between January 2010 and July 2020 was performed according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Manuscripts were identified through searching the databases Web of Science, Scopus, and PubMed as well as through hand searching. Manuscripts were included if they involved the measurement of diabetes-related parameters such as blood glucose level, performed physical activity, or feet condition via wearable or mobile devices. The quality of the included studies was assessed using the Newcastle-Ottawa Scale. RESULTS: The search yielded 1981 articles. A total of 26 publications met the eligibility criteria and were included in the review. Studies predominantly used wearable devices to monitor diabetes-related parameters. The accelerometer was by far the most used sensor, followed by the glucose monitor and heart rate monitor. Most studies applied some type of processing to the collected data, mainly consisting of statistical analysis or machine learning for activity recognition, finding associations among health outcomes, and diagnosing conditions related to diabetes. Few studies have focused on type 2 diabetes, even when this is the most prevalent type and the only preventable one. None of the studies focused on common diabetes complications. Clinical trials were fairly limited or nonexistent in most of the studies, with a common lack of detail about cohorts and case selection, comparability, and outcomes. Explicit endorsement by ethics committees or review boards was missing in most studies. Privacy or security issues were seldom addressed, and even if they were addressed, they were addressed at a rather insufficient level. CONCLUSIONS: The use of mobile and wearable devices for the monitoring of diabetes-related parameters shows early promise. Its development can benefit patients with diabetes, health care professionals, and researchers. However, this field is still in its early stages. Future work must pay special attention to privacy and security issues, the use of new emerging sensor technologies, the combination of mobile and clinical data, and the development of validated clinical trials.

Internet of Things for Mental Health: Open Issues in Data Acquisition, Self-Organization, Service Level Agreement, and Identity Management.

The increase of mental illness cases around the world can be described as an urgent and serious global health threat. Around 500 million people suffer from mental disorders, among which depression, schizophrenia, and dementia are the most prevalent. Revolutionary technological paradigms such as the Internet of Things (IoT) provide us with new capabilities to detect, assess, and care for patients early. This paper comprehensively survey works done at the intersection between IoT and mental health disorders. We evaluate multiple computational platforms, methods and devices, as well as study results and potential open issues for the effective use of IoT systems in mental health. We particularly elaborate on relevant open challenges in the use of existing IoT solutions for mental health care, which can be relevant given the potential impairments in some mental health patients such as data acquisition issues, lack of self-organization of devices and service level agreement, and security, privacy and consent issues, among others. We aim at opening the conversation for future research in this rather emerging area by outlining possible new paths based on the results and conclusions of this work.

Validation of mDurance, A Wearable Surface Electromyography System for Muscle Activity Assessment.

The mDurance® system is an innovative digital tool that combines wearable surface electromyography (sEMG), mobile computing and cloud analysis to streamline and automatize the assessment of muscle activity. The tool is particularly devised to support clinicians and sport professionals in their daily routines, as an assessment tool in the prevention, monitoring rehabilitation and training field. This study aimed at determining the validity of the mDurance system for measuring muscle activity by comparing sEMG output with a reference sEMG system, the Delsys® system. Fifteen participants were tested during isokinetic knee extensions at three different speeds (60, 180, and 300 deg/s), for two muscles (rectus femoris [RF] and vastus lateralis [VL]) and two different electrodes locations (proximal and distal placement). The maximum voluntary isometric contraction was carried out for the normalization of the signal, followed by dynamic isokinetic knee extensions for each speed. The sEMG output for both systems was obtained from the raw sEMG signal following mDurance's processing and filtering. Mean, median, first quartile, third quartile and 90th percentile was calculated from the sEMG amplitude signals for each system. The results show an almost perfect ICC relationship for the VL (ICC > 0.81) and substantial to almost perfect for the RF (ICC > 0.762) for all variables and speeds. The Bland-Altman plots revealed heteroscedasticity of error for mean, quartile 3 and 90th percentile (60 and 300 deg/s) for RF and at mean and 90th percentile for VL (300 deg/s). In conclusion, the results indicate that the mDurance® sEMG system is a valid tool to measure muscle activity during dynamic contractions over a range of speeds. This innovative system provides more time for clinicians (e.g., interpretation patients' pathologies) and sport trainers (e.g., advising athletes), thanks to automatic processing and filtering of the raw sEMG signal and generation of muscle activity reports in real-time.

Sensor Technology for Smart Homes.

As advances in technology continue relentlessly, intriguing possibilities for smart home services have emerged [...].

Composition Classification of Ultra-High Energy Cosmic Rays.

The study of cosmic rays remains as one of the most challenging research fields in Physics. From the many questions still open in this area, knowledge of the type of primary for each event remains as one of the most important issues. All of the cosmic rays observatories have been trying to solve this question for at least six decades, but have not yet succeeded. The main obstacle is the impossibility of directly detecting high energy primary events, being necessary to use Monte Carlo models and simulations to characterize generated particles cascades. This work presents the results attained using a simulated dataset that was provided by the Monte Carlo code CORSIKA, which is a simulator of high energy particles interactions with the atmosphere, resulting in a cascade of secondary particles extending for a few kilometers (in diameter) at ground level. Using this simulated data, a set of machine learning classifiers have been designed and trained, and their computational cost and effectiveness compared, when classifying the type of primary under ideal measuring conditions. Additionally, a feature selection algorithm has allowed for identifying the relevance of the considered features. The results confirm the importance of the electromagnetic-muonic component separation from signal data measured for the problem. The obtained results are quite encouraging and open new work lines for future more restrictive simulations.

CoVidAffect, real-time monitoring of mood variations following the COVID-19 outbreak in Spain.

The COVID-19 outbreak and the ensuing confinement measures are expected to bear a significant psychological impact on the affected populations. To date, all available studies designed to investigate the psychological effects of this unprecedented global crisis are based on cross-sectional surveys that do not capture emotional variations over time. Here, we present the data from CoVidAffect, a nationwide citizen science project aimed to provide longitudinal data of mood changes following the COVID-19 outbreak in the spanish territory. Spain is among the most affected countries by the pandemic, with one of the most restrictive and prolonged lockdowns worldwide. The project also collected a baseline of demographic and socioeconomic data. These data can be further analyzed to quantify emotional responses to specific measures and policies, and to understand the effect of context variables on psychological resilience. Importantly, to our knowledge this is the first dataset that offers the opportunity to study the behavior of emotion dynamics in a prolonged lockdown situation.

Smartphone-Based Platform for Affect Monitoring through Flexibly Managed Experience Sampling Methods.

The identification of daily life events that trigger significant changes on our affective state has become a fundamental task in emotional research. To achieve it, the affective states must be assessed in real-time, along with situational information that could contextualize the affective data acquired. However, the objective monitoring of the affective states and the context is still in an early stage. Mobile technologies can help to achieve this task providing immediate and objective data of the users' context and facilitating the assessment of their affective states. Previous works have developed mobile apps for monitoring affective states and context, but they use a fixed methodology which does not allow for making changes based on the progress of the study. This work presents a multimodal platform which leverages the potential of the smartphone sensors and the Experience Sampling Methods (ESM) to provide a continuous monitoring of the affective states and the context in an ubiquitous way. The platform integrates several elements aimed to expedite the real-time management of the ESM questionnaires. In order to show the potential of the platform, and evaluate its usability and its suitability for real-time assessment of affective states, a pilot study has been conducted. The results demonstrate an excellent usability level and a good acceptance from the users and the specialists that conducted the study, and lead to some suggestions for improving the data quality of mobile context-aware ESM-based systems.

Adaptive Data Boosting Technique for Robust Personalized Speech Emotion in Emotionally-Imbalanced Small-Sample Environments.

Personalized emotion recognition provides an individual training model for each target user in order to mitigate the accuracy problem when using general training models collected from multiple users. Existing personalized speech emotion recognition research has a cold-start problem that requires a large amount of emotionally-balanced data samples from the target user when creating the personalized training model. Such research is difficult to apply in real environments due to the difficulty of collecting numerous target user speech data with emotionally-balanced label samples. Therefore, we propose the Robust Personalized Emotion Recognition Framework with the Adaptive Data Boosting Algorithm to solve the cold-start problem. The proposed framework incrementally provides a customized training model for the target user by reinforcing the dataset by combining the acquired target user speech with speech from other users, followed by applying SMOTE (Synthetic Minority Over-sampling Technique)-based data augmentation. The proposed method proved to be adaptive across a small number of target user datasets and emotionally-imbalanced data environments through iterative experiments using the IEMOCAP (Interactive Emotional Dyadic Motion Capture) database.

Smart Sensing Technologies for Personalised e-Coaching.

People living in both developed and developing countries face serious health challenges related to sedentary lifestyles. It is therefore essential to find new ways to improve health so that people can live longer and age well. With an ever-growing number of smart sensing systems developed and deployed across the globe, experts are primed to help coach people to have healthier behaviors. The increasing accountability associated with app- and device-based behavior tracking not only provides timely and personalized information and support, but also gives us an incentive to set goals and do more. This paper outlines some of the recent efforts made towards automatic and autonomous identification and coaching of troublesome behaviors to procure lasting, beneficial behavioral changes.

Smartphone Location-Independent Physical Activity Recognition Based on Transportation Natural Vibration Analysis.

Activity recognition through smartphones has been proposed for a variety of applications. The orientation of the smartphone has a significant effect on the recognition accuracy; thus, researchers generally propose using features invariant to orientation or displacement to achieve this goal. However, those features reduce the capability of the recognition system to differentiate among some specific commuting activities (e.g., bus and subway) that normally involve similar postures. In this work, we recognize those activities by analyzing the vibrations of the vehicle in which the user is traveling. We extract natural vibration features of buses and subways to distinguish between them and address the confusion that can arise because the activities are both static in terms of user movement. We use the gyroscope to fix the accelerometer to the direction of gravity to achieve an orientation-free use of the sensor. We also propose a correction algorithm to increase the accuracy when used in free living conditions and a battery saving algorithm to consume less power without reducing performance. Our experimental results show that the proposed system can adequately recognize each activity, yielding better accuracy in the detection of bus and subway activities than existing methods.

Ontology-Based High-Level Context Inference for Human Behavior Identification.

Recent years have witnessed a huge progress in the automatic identification of individual primitives of human behavior, such as activities or locations. However, the complex nature of human behavior demands more abstract contextual information for its analysis. This work presents an ontology-based method that combines low-level primitives of behavior, namely activity, locations and emotions, unprecedented to date, to intelligently derive more meaningful high-level context information. The paper contributes with a new open ontology describing both low-level and high-level context information, as well as their relationships. Furthermore, a framework building on the developed ontology and reasoning models is presented and evaluated. The proposed method proves to be robust while identifying high-level contexts even in the event of erroneously-detected low-level contexts. Despite reasonable inference times being obtained for a relevant set of users and instances, additional work is required to scale to long-term scenarios with a large number of users.

Human Behavior Analysis by Means of Multimodal Context Mining.

There is sufficient evidence proving the impact that negative lifestyle choices have on people's health and wellness. Changing unhealthy behaviours requires raising people's self-awareness and also providing healthcare experts with a thorough and continuous description of the user's conduct. Several monitoring techniques have been proposed in the past to track users' behaviour; however, these approaches are either subjective and prone to misreporting, such as questionnaires, or only focus on a specific component of context, such as activity counters. This work presents an innovative multimodal context mining framework to inspect and infer human behaviour in a more holistic fashion. The proposed approach extends beyond the state-of-the-art, since it not only explores a sole type of context, but also combines diverse levels of context in an integral manner. Namely, low-level contexts, including activities, emotions and locations, are identified from heterogeneous sensory data through machine learning techniques. Low-level contexts are combined using ontological mechanisms to derive a more abstract representation of the user's context, here referred to as high-level context. An initial implementation of the proposed framework supporting real-time context identification is also presented. The developed system is evaluated for various realistic scenarios making use of a novel multimodal context open dataset and data on-the-go, demonstrating prominent context-aware capabilities at both low and high levels.

The Mining Minds digital health and wellness framework.

BACKGROUND: The provision of health and wellness care is undergoing an enormous transformation. A key element of this revolution consists in prioritizing prevention and proactivity based on the analysis of people's conducts and the empowerment of individuals in their self-management. Digital technologies are unquestionably destined to be the main engine of this change, with an increasing number of domain-specific applications and devices commercialized every year; however, there is an apparent lack of frameworks capable of orchestrating and intelligently leveraging, all the data, information and knowledge generated through these systems. METHODS: This work presents Mining Minds, a novel framework that builds on the core ideas of the digital health and wellness paradigms to enable the provision of personalized support. Mining Minds embraces some of the most prominent digital technologies, ranging from Big Data and Cloud Computing to Wearables and Internet of Things, as well as modern concepts and methods, such as context-awareness, knowledge bases or analytics, to holistically and continuously investigate on people's lifestyles and provide a variety of smart coaching and support services. RESULTS: This paper comprehensively describes the efficient and rational combination and interoperation of these technologies and methods through Mining Minds, while meeting the essential requirements posed by a framework for personalized health and wellness support. Moreover, this work presents a realization of the key architectural components of Mining Minds, as well as various exemplary user applications and expert tools to illustrate some of the potential services supported by the proposed framework. CONCLUSIONS: Mining Minds constitutes an innovative holistic means to inspect human behavior and provide personalized health and wellness support. The principles behind this framework uncover new research ideas and may serve as a reference for similar initiatives.

On Curating Multimodal Sensory Data for Health and Wellness Platforms.

In recent years, the focus of healthcare and wellness technologies has shown a significant shift towards personal vital signs devices. The technology has evolved from smartphone-based wellness applications to fitness bands and smartwatches. The novelty of these devices is the accumulation of activity data as their users go about their daily life routine. However, these implementations are device specific and lack the ability to incorporate multimodal data sources. Data accumulated in their usage does not offer rich contextual information that is adequate for providing a holistic view of a user's lifelog. As a result, making decisions and generating recommendations based on this data are single dimensional. In this paper, we present our Data Curation Framework (DCF) which is device independent and accumulates a user's sensory data from multimodal data sources in real time. DCF curates the context of this accumulated data over the user's lifelog. DCF provides rule-based anomaly detection over this context-rich lifelog in real time. To provide computation and persistence over the large volume of sensory data, DCF utilizes the distributed and ubiquitous environment of the cloud platform. DCF has been evaluated for its performance, correctness, ability to detect complex anomalies, and management support for a large volume of sensory data.

Design, implementation and validation of a novel open framework for agile development of mobile health applications.

The delivery of healthcare services has experienced tremendous changes during the last years. Mobile health or mHealth is a key engine of advance in the forefront of this revolution. Although there exists a growing development of mobile health applications, there is a lack of tools specifically devised for their implementation. This work presents mHealthDroid, an open source Android implementation of a mHealth Framework designed to facilitate the rapid and easy development of mHealth and biomedical apps. The framework is particularly planned to leverage the potential of mobile devices such as smartphones or tablets, wearable sensors and portable biomedical systems. These devices are increasingly used for the monitoring and delivery of personal health care and wellbeing. The framework implements several functionalities to support resource and communication abstraction, biomedical data acquisition, health knowledge extraction, persistent data storage, adaptive visualization, system management and value-added services such as intelligent alerts, recommendations and guidelines. An exemplary application is also presented along this work to demonstrate the potential of mHealthDroid. This app is used to investigate on the analysis of human behavior, which is considered to be one of the most prominent areas in mHealth. An accurate activity recognition model is developed and successfully validated in both offline and online conditions.

Traffic Behavior Recognition Using the Pachinko Allocation Model.

CCTV-based behavior recognition systems have gained considerable attention in recent years in the transportation surveillance domain for identifying unusual patterns, such as traffic jams, accidents, dangerous driving and other abnormal behaviors. In this paper, a novel approach for traffic behavior modeling is presented for video-based road surveillance. The proposed system combines the pachinko allocation model (PAM) and support vector machine (SVM) for a hierarchical representation and identification of traffic behavior. A background subtraction technique using Gaussian mixture models (GMMs) and an object tracking mechanism based on Kalman filters are utilized to firstly construct the object trajectories. Then, the sparse features comprising the locations and directions of the moving objects are modeled by PAMinto traffic topics, namely activities and behaviors. As a key innovation, PAM captures not only the correlation among the activities, but also among the behaviors based on the arbitrary directed acyclic graph (DAG). The SVM classifier is then utilized on top to train and recognize the traffic activity and behavior. The proposed model shows more flexibility and greater expressive power than the commonly-used latent Dirichlet allocation (LDA) approach, leading to a higher recognition accuracy in the behavior classification.