A multimodal dataset of real world mobility activities in Parkinson's disease.

Parkinson's disease (PD) is a neurodegenerative disorder characterised by motor symptoms such as gait dysfunction and postural instability. Technological tools to continuously monitor outcomes could capture the hour-by-hour symptom fluctuations of PD. Development of such tools is hampered by the lack of labelled datasets from home settings. To this end, we propose REMAP (REal-world Mobility Activities in Parkinson's disease), a human rater-labelled dataset collected in a home-like setting. It includes people with and without PD doing sit-to-stand transitions and turns in gait. These discrete activities are captured from periods of free-living (unobserved, unstructured) and during clinical assessments. The PD participants withheld their dopaminergic medications for a time (causing increased symptoms), so their activities are labelled as being "on" or "off" medications. Accelerometry from wrist-worn wearables and skeleton pose video data is included. We present an open dataset, where the data is coarsened to reduce re-identifiability, and a controlled dataset available on application which contains more refined data. A use-case for the data to estimate sit-to-stand speed and duration is illustrated.

Automated Real-World Video Analysis of Sit-to-Stand Transitions Predicts Parkinson's Disease Severity.

Introduction: Technology holds the potential to track disease progression and response to neuroprotective therapies in Parkinson's disease (PD). The sit-to-stand (STS) transition is a frequently occurring event which is important to people with PD. The aim of this study was to demonstrate an automatic approach to quantify STS duration and speed using a real-world free-living dataset and look at clinical correlations of the outcomes, including whether STS parameters change when someone withholds PD medications. Methods: Eighty-five hours of video data were collected from 24 participants staying in pairs for 5-day periods in a naturalistic setting. Skeleton joints were extracted from the video data; the head trajectory was estimated and used to estimate the STS parameters of duration and speed. Results: 3.14 STS transitions were seen per hour per person on average. Significant correlations were seen between automatic and manual STS duration (Pearson rho - 0.419, p = 0.042) and between automatic STS speed and manual STS duration (Pearson rho - 0.780, p < 0.001). Significant and strong correlations were seen between the gold-standard clinical rating scale scores and both STS duration and STS speed; these correlations were not seen in the STS transitions when the participants were carrying something in their hand(s). Significant differences were seen at the cohort level between control and PD participants' ON medications' STS duration (U = 6,263, p = 0.018) and speed (U = 9,965, p < 0.001). At an individual level, only two participants with PD became significantly slower to STS when they were OFF medications; withholding medications did not significantly change STS duration at an individual level in any participant. Conclusion: We demonstrate a novel approach to automatically quantify and ecologically validate two STS parameters which correlate with gold-standard clinical tools measuring disease severity in PD.

Toward Enhanced Clinical Decision Support for Patients Undergoing a Hip or Knee Replacement: Focus Group and Interview Study With Surgeons.

BACKGROUND: The current assessment of recovery after total hip or knee replacement is largely based on the measurement of health outcomes through self-report and clinical observations at follow-up appointments in clinical settings. Home activity-based monitoring may improve assessment of recovery by enabling the collection of more holistic information on a continuous basis. OBJECTIVE: This study aimed to introduce orthopedic surgeons to time-series analyses of patient activity data generated from a platform of sensors deployed in the homes of patients who have undergone primary total hip or knee replacement and understand the potential role of these data in postoperative clinical decision-making. METHODS: Orthopedic surgeons and registrars were recruited through a combination of convenience and snowball sampling. Inclusion criteria were a minimum required experience in total joint replacement surgery specific to the hip or knee or familiarity with postoperative recovery assessment. Exclusion criteria included a lack of specific experience in the field. Of the 9 approached participants, 6 (67%) orthopedic surgeons and 3 (33%) registrars took part in either 1 of 3 focus groups or 1 of 2 interviews. Data were collected using an action-based approach in which stimulus materials (mock data visualizations) provided imaginative and creative interactions with the data. The data were analyzed using a thematic analysis approach. RESULTS: Each data visualization was presented sequentially followed by a discussion of key illustrative commentary from participants, ending with a summary of key themes emerging across the focus group and interview data set. CONCLUSIONS: The limitations of the evidence are as follows. The data presented are from 1 English hospital. However, all data reflect the views of surgeons following standard national approaches and training. Although convenience sampling was used, participants' background, skills, and experience were considered heterogeneous. Passively collected home monitoring data offered a real opportunity to more objectively characterize patients' recovery from surgery. However, orthopedic surgeons highlighted the considerable difficulty in navigating large amounts of complex data within short medical consultations with patients. Orthopedic surgeons thought that a proposed dashboard presenting information and decision support alerts would fit best with existing clinical workflows. From this, the following guidelines for system design were developed: minimize the risk of misinterpreting data, express a level of confidence in the data, support clinicians in developing relevant skills as time-series data are often unfamiliar, and consider the impact of patient engagement with data in the future. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.1136/bmjopen-2018-021862.

Personalized Energy Expenditure Estimation: Visual Sensing Approach With Deep Learning.

BACKGROUND: Calorimetry is both expensive and obtrusive but provides the only way to accurately measure energy expenditure in daily living activities of any specific person, as different people can use different amounts of energy despite performing the same actions in the same manner. Deep learning video analysis techniques have traditionally required a lot of data to train; however, recent advances in few-shot learning, where only a few training examples are necessary, have made developing personalized models without a calorimeter a possibility. OBJECTIVE: The primary aim of this study is to determine which activities are most well suited to calibrate a vision-based personalized deep learning calorie estimation system for daily living activities. METHODS: The SPHERE (Sensor Platform for Healthcare in a Residential Environment) Calorie data set is used, which features 10 participants performing 11 daily living activities totaling 4.5 hours of footage. Calorimeter and video data are available for all recordings. A deep learning method is used to regress calorie predictions from video. RESULTS: Models are personalized with 32 seconds from all 11 actions in the data set, and mean square error (MSE) is taken against a calorimeter ground truth. The best single action for calibration is wipe (1.40 MSE). The best pair of actions are sweep and sit (1.09 MSE). This compares favorably to using a whole 30-minute sequence containing 11 actions to calibrate (1.06 MSE). CONCLUSIONS: A vision-based deep learning energy expenditure estimation system for a wide range of daily living activities can be calibrated to a specific person with footage and calorimeter data from 32 seconds of sweeping and 32 seconds of sitting.

Multimodal Classification of Parkinson's Disease in Home Environments with Resiliency to Missing Modalities.

Parkinson's disease (PD) is a chronic neurodegenerative condition that affects a patient's everyday life. Authors have proposed that a machine learning and sensor-based approach that continuously monitors patients in naturalistic settings can provide constant evaluation of PD and objectively analyse its progression. In this paper, we make progress toward such PD evaluation by presenting a multimodal deep learning approach for discriminating between people with PD and without PD. Specifically, our proposed architecture, named MCPD-Net, uses two data modalities, acquired from vision and accelerometer sensors in a home environment to train variational autoencoder (VAE) models. These are modality-specific VAEs that predict effective representations of human movements to be fused and given to a classification module. During our end-to-end training, we minimise the difference between the latent spaces corresponding to the two data modalities. This makes our method capable of dealing with missing modalities during inference. We show that our proposed multimodal method outperforms unimodal and other multimodal approaches by an average increase in F1-score of 0.25 and 0.09, respectively, on a data set with real patients. We also show that our method still outperforms other approaches by an average increase in F1-score of 0.17 when a modality is missing during inference, demonstrating the benefit of training on multiple modalities.

Person Re-ID by Fusion of Video Silhouettes and Wearable Signals for Home Monitoring Applications.

The use of visual sensors for monitoring people in their living environments is critical in processing more accurate health measurements, but their use is undermined by the issue of privacy. Silhouettes, generated from RGB video, can help towards alleviating the issue of privacy to some considerable degree. However, the use of silhouettes would make it rather complex to discriminate between different subjects, preventing a subject-tailored analysis of the data within a free-living, multi-occupancy home. This limitation can be overcome with a strategic fusion of sensors that involves wearable accelerometer devices, which can be used in conjunction with the silhouette video data, to match video clips to a specific patient being monitored. The proposed method simultaneously solves the problem of Person ReID using silhouettes and enables home monitoring systems to employ sensor fusion techniques for data analysis. We develop a multimodal deep-learning detection framework that maps short video clips and accelerations into a latent space where the Euclidean distance can be measured to match video and acceleration streams. We train our method on the SPHERE Calorie Dataset, for which we show an average area under the ROC curve of 76.3% and an assignment accuracy of 77.4%. In addition, we propose a novel triplet loss for which we demonstrate improving performances and convergence speed.