MM-HAR: Multi-Modal Human Activity Recognition Using Consumer Smartwatch and Earbuds.

Human Activity Recognition (HAR) is one of the important applications of digital health that helps to track fitness or to avoid sedentary behavior by monitoring daily activities. Due to the growing popularity of consumer wearable devices, smartwatches, and earbuds are being widely adopted for HAR applications. However, using just one of the devices may not be sufficient to track all activities properly. This paper proposes a multi-modal approach to HAR by using both buds and watch. Using a large dataset of 44 subjects collected from both in-lab and in-home environments, we demonstrate the limitations of using a single modality as well as the importance of a multi-modal approach. Moreover, we also train and evaluate the performance of five different machine learning classifiers for various combinations of devices such as buds only, watch only, and both. We believe the detailed analyses presented in this paper may serve as a benchmark for the research community to explore and build upon in the future.

Power Optimized Smartwatch-Earbuds Multimodal System for Monitoring Activities of Daily Living.

Activities of daily living is an important entity to monitor for promoting healthy lifestyle for chronic disease patients, children and the healthy population. This paper presents a smartwatch and earbuds inertial sensors based multi-modal power efficient end-to-end mobile system for continuous, passive and accurate detection of broad daily activity classes. We collected various posture, stationary and moving activity data from 40 diverse subjects using earbuds and smartwatch and develop the novel power optimized end-to-end operational system consisting of i) optimized device sampling rates and Bluetooth packet transfer rates, ii) data buffering mechanism, iii) background services, and iv) optimized model size, and demonstrating 93% macro recall score in detecting various activities. Our power optimized solution uses 80%, 40% and 33.33% less battery power for the smartphone, smartwatch, and earbuds respectively, compared to a power agnostic system with an estimated continuous no-charging run time of 50 hours, 16.67 hours, and 25 hours for the smartphone, smartwatch, and earbuds respectively.Clinical relevance- The end-to-end power optimized activity detection system presented in this paper will assist practicing clinicians toward treatment of various chronic disease patients (e.g. diabetes, hypertension, heart disease and obesity) by long-term, continuous monitoring of their lifestyle and sedentary behavior.

RRMonitor: A Resource-Aware End-to-End System for Continuous Monitoring of Respiration Rate Using Earbuds.

Respiration rate is considered as a critical vital sign, and daily monitoring of respiration rate could provide helpful information about any acute condition in the human body. While researchers have been exploring mobile devices for respiration rate monitoring, passive and continuous monitoring is still not feasible due to many usability challenges (e.g., active participation) in existing approaches. This paper presents an end-to-end system called RRMonitor that leverages the movement sensors from commodity earbuds to continuously monitor the respiration rate in near real-time. While developing the systems, we extensively explored some key parameters, algorithms, and approaches from existing literature that are better suited for continuous and passive respiration rate monitoring. RRMonitor can passively track the respiration rate with a mean absolute error as low as 1.64 cycles per minute without requiring active participation from the user.

SpeechSpiro: Lung Function Assessment from Speech Pattern as an Alternative to Spirometry for Mobile Health Tracking.

Respiratory illnesses are common in the United States and globally; people deal with these illnesses in various forms, such as asthma, chronic obstructive pulmonary diseases, or infectious respiratory diseases (e.g., coronavirus). The lung function of subjects affected by these illnesses degrades due to infection or inflammation in their respiratory airways. Typically, lung function is assessed using in-clinic medical equipment, and quite recently, via portable spirometry devices. Research has shown that the obstruction and restriction in the respiratory airways affect individuals' voice characteristics. Hence, audio features could play a role in predicting the lung function and severity of the obstruction. In this paper, we go beyond well-known voice audio features and create a hybrid deep learning model using CNN-LSTM to discover spatiotemporal patterns in speech and predict the lung function parameters with accuracy comparable to conventional devices. We validate the performance and generalizability of our method using the data collected from 201 subjects enrolled in two studies internally and in collaboration with a pulmonary hospital. SpeechSpiro measures lung function parameters (e.g., forced vital capacity) with a mean normalized RMSE of 12% and R2 score of up to 76% using 60-second phone audio recordings of individuals reading a passage.Clinical relevance - Speech-based spirometry has the potential to eliminate the need for an additional device to carry out the lung function assessment outside clinical settings; hence, it can enable continuous and mobile track of the individual's condition, healthy or with a respiratory illness, using a smartphone.

A Novel Multi-Centroid Template Matching Algorithm and Its Application to Cough Detection.

Cough is a major symptom of respiratory-related diseases. There exists a tremendous amount of work in detecting coughs from audio but there has been no effort to identify coughs from solely inertial measurement unit (IMU). Coughing causes motion across the whole body and especially on the neck and head. Therefore, head motion data during coughing captured by a head-worn IMU sensor could be leveraged to detect coughs using a template matching algorithm. In time series template matching problems, K-Nearest Neighbors (KNN) combined with elastic distance measurement (esp. Dynamic Time Warping (DTW)) achieves outstanding performance. However, it is often regarded as prohibitively time-consuming. Nearest Centroid Classifier is thereafter proposed. But the accuracy is comprised of only one centroid obtained for each class. Centroid-based Classifier performs clustering and averaging for each cluster, but requires manually setting the number of clusters. We propose a novel self-tuning multi-centroid template-matching algorithm, which can automatically adjust the number of clusters to balance accuracy and inference time. Through experiments conducted on synthetic datasets and a real-world earbud-based cough dataset, we demonstrate the superiority of our proposed algorithm and present the result of cough detection with a single accelerometer sensor on the earbuds platform.Clinical relevance- Coughing is a ubiquitous symptom of pulmonary disease, especially for patients with COPD and asthma. This work explores the possibility and and presents the result of cough detection using an IMU sensor embedded in earables.

A Comprehensive Approach for Classification of the Cough Type.

Identifying the presence of sputum in the lung is essential in detection of diseases such as lung infection, pneumonia and cancer. Cough type classification (dry/wet) is an effective way of examining presence of lung sputum. This is traditionally done through physical exam in a clinical visit which is subjective and inaccurate. This work proposes an objective approach relying on the acoustic features of the cough sound. A total number of 5971 coughs (5242 dry and 729 wet) were collected from 131 subjects using Smartphone. The data was reviewed and annotated by a novel multi-layer labeling platform. The annotation kappa inter-rater agreement score is measured to be 0.81 and 0.37 for 1st and 2nd layer respectively. Sensitivity and specificity values of 88% and 86% are measured for classification between wet and dry coughs (highest across the literature).

Estimation of the Lung Function Using Acoustic Features of the Voluntary Cough.

Spirometry test, a measure of the patient's lung function, is the gold standard for diagnosis and monitoring of chronic pulmonary diseases. Spirometry is currently being done in hospital settings by having the patients blow the air out of their lungs forcefully and into the spirometer's tubes under the supervision and constant guidance of clinicians. This test is expensive, cumbersome and not easily applicable to every-day monitoring of these patients. The lung mechanism when performing a cough is very similar to when spirometry test is done. That includes a big inhalation, air compression and forceful exhalation. Therefore, it is reasonable to assume that obstruction of lung airways should have a similar effect on both cough features and spirometry measures. This paper explores the estimation of lung obstruction using cough acoustic features. A total number of 3695 coughs were collected from patients from 4 different conditions and 4 different severity categories along with their lung function measures in a clinical setting using a smartphone's microphone and a hospital-grade spirometry lab. After feature-set optimization and model hyperparameter tuning, the lung obstruction was estimated with MAE (Mean Absolute Error) of 8% for COPD and 9% for asthma populations. In addition to lung obstruction estimation, we were able to classify patients' disease state with 91% accuracy and patients' severity within each disease state with 95% accuracy.Clinical Relevance- This enables effort-independent estimation of lung function spirometry parameters which could potentially lead to passive monitoring of pulmonary patients.

CoughGAN: Generating Synthetic Coughs that Improve Respiratory Disease Classification.

Despite the prevalence of respiratory diseases, their diagnosis by clinicians is challenging. Accurately assessing airway sounds requires extensive clinical training and equipment that may not be easily available. Current methods that automate this diagnosis are hindered by their use of features that require pulmonary function tests. We leverage the audio characteristics of coughs to create classifiers that can distinguish common respiratory diseases in adults. Moreover, we build on recent advances in generative adversarial networks to augment our dataset with cleverly engineered synthetic cough samples for each class of major respiratory disease, to balance and increase our dataset size. We experimented on cough samples collected with a smartphone from 45 subjects in a clinic. Our CoughGAN-improved Support Vector Machine and Random Forest models show up to 76% test accuracy and 83% F1 score in classifying subjects' conditions between healthy and three major respiratory diseases. Adding our synthetic coughs improves the performance we can obtain from a relatively small unbalanced healthcare dataset by boosting the accuracy over 30%. Our data augmentation reduces overfitting and discourages the prediction of a single, dominant class. These results highlight the feasibility of automatic, cough-based respiratory disease diagnosis using smartphones or wearables in the wild.

CoughMatch - Subject Verification Using Cough for Personal Passive Health Monitoring.

Automatic cough detection using audio has advanced passive health monitoring on devices such as smart phones and wearables; it enables capturing longitudinal health data by eliminating user interaction and effort. One major issue arises when coughs from surrounding people are also detected; capturing false coughs leads to significant false alarms, excessive cough frequency, and thereby misdiagnosis of user condition. To address this limitation, in this paper, a method is proposed that creates a personal cough model of the primary subject using limited number of cough samples; the model is used by the automatic cough detection to verify whether the identified coughs match the personal pattern and belong to the primary subject. A Gaussian mixture model is trained using audio features from cough to implement the subject verification method; novel cough embeddings are learned using neural networks and integrated into the model to further improve the prediction accuracy. We analyze the performance of the method using our cough dataset collected by a smart phone in a clinical study. Population in the dataset involves subjects categorized of healthy or patients with COPD or Asthma, with the purpose of covering a wider range of pulmonary conditions. Cross-subject validation on a diverse dataset shows that the method achieves an average error rate of less than 10%, using a personal cough model generated by only 5 coughs from the primary subject.

Adapting to Noise in Speech Obfuscation by Audio Profiling Using Generative Models for Passive Health Monitoring.

Passive health monitoring has been introduced as a solution for continuous diagnosis and tracking of subjects' condition with minimal effort. This is partially achieved by the technology of passive audio recording although it poses major audio privacy issues for subjects. Existing methods are limited to controlled recording environments and their prediction is significantly influenced by background noises. Meanwhile, they are too compute-intensive to be continuously running on smart phones. In this paper, we implement an efficient and robust audio privacy preserving method that profiles the background audio to focus only on audio activities detected during recording for performance improvement, and to adapt to the noise for more accurate speech segmentation. We analyze the performance of our method using audio data collected by a smart watch in lab noisy settings. Our obfuscation results show a low false positive rate of 20% with a 92% true positive rate by adapting to the recording noise level. We also reduced model memory footprint and execution time of the method on a smart phone by 75% and 62% to enable continuous speech obfuscation.

Toward Early Severity Assessment of Obstructive Lung Disease Using Multi-Modal Wearable Sensor Data Fusion During Walking.

Early detection of chronic diseases helps to minimize the disease impact on patient's health and reduce the economic burden. Continuous monitoring of such diseases helps in the evaluation of rehabilitation program effectiveness as well as in the detection of exacerbation. The use of everyday wearables i.e. chest band, smartwatch and smart band equipped with good quality sensor and light weight machine learning algorithm for the early detection of diseases is very promising and holds tremendous potential as they are widely used. In this study, we have investigated the use of acceleration, electrocardiogram, and respiration sensor data from a chest band for the evaluation of obstructive lung disease severity. Recursive feature elimination technique has been used to identity top 15 features from a set of 62 features including gait characteristics, respiration pattern and heart rate variability. A precision of 0.93, recall of 0.91 and F-1 score of 0.92 have been achieved with a support vector machine for the classification of severe patients from the non-severe patients in a data set of 60 patients. In addition, the selected features showed significant correlation with the percentage of predicted FEV1.Clinical Relevance- The study result indicates that wearable sensor data collected during natural walk can be used in the early evaluation of pulmonary patients thus enabling them to seek medical attention and avoid exacerbation. In addition, it may serve as a complementary tool for pulmonary patient evaluation during a 6-minute walk test.

Opportunistic Environmental Sensing with Smartphones: a Critical Review of Current Literature and Applications.

PURPOSE OF REVIEW: This review sought to summarize recent literature and applications of passive, or opportunistic, mobile sensing in the fields of exposure science in built environment settings; highlight innovative opportunistic sensing systems; and analyze their functionality, significant features, and limitations. RECENT FINDINGS: Fifty-two papers related to opportunistic environmental sensing from 2009 or later were related to this review, of which 27 were included. An array of applications have emerged in environmental monitoring, employing anywhere from one to six of the phone's on-board sensors. The viability of an application is determined by several key factors: the number and quality of sensors on-board the smartphone; power and processing demand; algorithm complexity; data security; mobile network coverage; reliance on external data sources; minimum number of users required; and degree of user burden when using the application. Some factors are universal, while others are more context-specific. Future research should assess sensing applications based on these factors.

A dynamic data source selection system for smartwatch platform.

A novel data source selection algorithm is proposed for ambulatory activity tracking of elderly people. The algorithm introduces the concept of dynamic switching between the data collection modules (a smartwatch and a smartphone) to improve accuracy and battery life using contextual information. We show that by making offloading decisions as a function of activity, the proposed algorithm improves power consumption and accuracy of the previous work by 7 hours and 5% respectively compared to the baseline.

Can smartwatches replace smartphones for posture tracking?

This paper introduces a human posture tracking platform to identify the human postures of sitting, standing or lying down, based on a smartwatch. This work develops such a system as a proof-of-concept study to investigate a smartwatch's ability to be used in future remote health monitoring systems and applications. This work validates the smartwatches' ability to track the posture of users accurately in a laboratory setting while reducing the sampling rate to potentially improve battery life, the first steps in verifying that such a system would work in future clinical settings. The algorithm developed classifies the transitions between three posture states of sitting, standing and lying down, by identifying these transition movements, as well as other movements that might be mistaken for these transitions. The system is trained and developed on a Samsung Galaxy Gear smartwatch, and the algorithm was validated through a leave-one-subject-out cross-validation of 20 subjects. The system can identify the appropriate transitions at only 10 Hz with an F-score of 0.930, indicating its ability to effectively replace smart phones, if needed.