Improving Dysarthric Speech Segmentation With Emulated and Synthetic Augmentation.

Acoustic features extracted from speech can help with the diagnosis of neurological diseases and monitoring of symptoms over time. Temporal segmentation of audio signals into individual words is an important pre-processing step needed prior to extracting acoustic features. Machine learning techniques could be used to automate speech segmentation via automatic speech recognition (ASR) and sequence to sequence alignment. While state-of-the-art ASR models achieve good performance on healthy speech, their performance significantly drops when evaluated on dysarthric speech. Fine-tuning ASR models on impaired speech can improve performance in dysarthric individuals, but it requires representative clinical data, which is difficult to collect and may raise privacy concerns. This study explores the feasibility of using two augmentation methods to increase ASR performance on dysarthric speech: 1) healthy individuals varying their speaking rate and loudness (as is often used in assessments of pathological speech); 2) synthetic speech with variations in speaking rate and accent (to ensure more diverse vocal representations and fairness). Experimental evaluations showed that fine-tuning a pre-trained ASR model with data from these two sources outperformed a model fine-tuned only on real clinical data and matched the performance of a model fine-tuned on the combination of real clinical data and synthetic speech. When evaluated on held-out acoustic data from 24 individuals with various neurological diseases, the best performing model achieved an average word error rate of 5.7% and a mean correct count accuracy of 94.4%. In segmenting the data into individual words, a mean intersection-over-union of 89.2% was obtained against manual parsing (ground truth). It can be concluded that emulated and synthetic augmentations can significantly reduce the need for real clinical data of dysarthric speech when fine-tuning ASR models and, in turn, for speech segmentation.

Automating sedation state assessments using natural language processing.

INTRODUCTION: Common goals for procedural sedation are to control pain and ensure the patient is not moving to an extent that is impeding safe progress or completion of the procedure. Clinicians perform regular assessments of the adequacy of procedural sedation in accordance with these goals to inform their decision-making around sedation titration and also for documentation of the care provided. Natural language processing could be applied to real-time transcriptions of audio recordings made during procedures in order to classify sedation states that involve movement and pain, which could then be integrated into clinical documentation systems. The aim of this study was to determine whether natural language processing algorithms will work with sufficient accuracy to detect sedation states during procedural sedation. DESIGN: A prospective observational study was conducted. METHODS: Audio recordings from consenting participants undergoing elective procedures performed in the interventional radiology suite at a large academic hospital were transcribed using an automated speech recognition model. Sentences of transcribed text were used to train and evaluate several different NLP pipelines for a text classification task. The NLP pipelines we evaluated included a simple Bag-of-Words (BOW) model, an ensemble architecture combining a linear BOW model and a "token-to-vector" (Tok2Vec) component, and a transformer-based architecture using the RoBERTa pre-trained model. RESULTS: A total of 15,936 sentences from transcriptions of 82 procedures was included in the analysis. The RoBERTa model achieved the highest performance among the three models with an area under the ROC curve (AUC-ROC) of 0.97, an F1 score of 0.87, a precision of 0.86, and a recall of 0.89. The Ensemble model showed a similarly high AUC-ROC of 0.96, but lower F1 score of 0.79, precision of 0.83, and recall of 0.77. The BOW approach achieved an AUC-ROC of 0.97 and the F1 score was 0.7, precision was 0.83 and recall was 0.66. CONCLUSION: The transformer-based architecture using the RoBERTa pre-trained model achieved the best classification performance. Further research is required to confirm the that this natural language processing pipeline can accurately perform text classifications with real-time audio data to allow for automated sedation state assessments. CLINICAL RELEVANCE: Automating sedation state assessments using natural language processing pipelines would allow for more timely documentation of the care received by sedated patients, and, at the same time, decrease documentation burden for clinicians. Downstream applications can also be generated from the classifications, including for example real-time visualizations of sedation state, which may facilitate improved communication of the adequacy of the sedation between clinicians, who may be performing supervision remotely. Also, accumulation of sedation state assessments from multiple procedures may reveal insights into the efficacy of particular sedative medications or identify procedures where the current approach for sedation and analgesia is not optimal (i.e. a significant amount of time spent in "pain" or "movement" sedation states).

Automated, Vision-Based Goniometry and Range of Motion Calculation in Individuals With Suspected Ehlers-Danlos Syndromes/Generalized Hypermobility Spectrum Disorders: A Comparison of Pose-Estimation Libraries to Goniometric Measurements.

Generalized joint hypermobility (GJH) often leads clinicians to suspect a diagnosis of Ehlers Danlos Syndrome (EDS), but it can be difficult to objectively assess. Video-based goniometry has been proposed to objectively estimate joint range of motion in hyperextended joints. As part of an exam of joint hypermobility at a specialized EDS clinic, a mobile phone was used to record short videos of 97 adults (89 female, 35.0 ± 9.9 years old) undergoing assessment of the elbows, knees, shoulders, ankles, and fifth fingers. Five body keypoint pose-estimation libraries (AlphaPose, Detectron, MediaPipe-Body, MoveNet - Thunder, OpenPose) and two hand keypoint pose-estimation libraries (AlphaPose, MediaPipe-Hands) were used to geometrically calculate the maximum angle of hyperextension or hyperflexion of each joint. A custom domain-specific model with a MobileNet-v2 backbone finetuned on data collected as part of this study was also evaluated for the fifth finger movement. Spearman's correlation was used to analyze the angles calculated from the tracked joint positions, the angles calculated from manually annotated keypoints, and the angles measured using a goniometer. Moderate correlations between the angles estimated using pose-tracked keypoints and the goniometer measurements were identified for the elbow (rho =.722; Detectron), knee (rho =.608; MoveNet - Thunder), shoulder (rho =.632; MoveNet - Thunder), and fifth finger (rho =.786; custom model) movements. The angles estimated from keypoints predicted by open-source libraries at the ankles were not significantly correlated with the goniometer measurements. Manually annotated angles at the elbows, knees, shoulders, and fifth fingers were moderately to strongly correlated to goniometer measurements but were weakly correlated for the ankles. There was not one pose-estimation library which performed best across all joints, so the library of choice must be selected separately for each joint of interest. This work evaluates several pose-estimation models as part of a vision-based system for estimating joint angles in individuals with suspected joint hypermobility. Future applications of the proposed system could facilitate objective assessment and screening of individuals referred to specialized EDS clinics.

Evaluating the ability of a predictive vision-based machine learning model to measure changes in gait in response to medication and DBS within individuals with Parkinson's disease.

INTRODUCTION: Gait impairments in Parkinson's disease (PD) are treated with dopaminergic medication or deep-brain stimulation (DBS), although the magnitude of the response is variable between individuals. Computer vision-based approaches have previously been evaluated for measuring the severity of parkinsonian gait in videos, but have not been evaluated for their ability to identify changes within individuals in response to treatment. This pilot study examines whether a vision-based model, trained on videos of parkinsonism, is able to detect improvement in parkinsonian gait in people with PD in response to medication and DBS use. METHODS: A spatial-temporal graph convolutional model was trained to predict MDS-UPDRS-gait scores in 362 videos from 14 older adults with drug-induced parkinsonism. This model was then used to predict MDS-UPDRS-gait scores on a different dataset of 42 paired videos from 13 individuals with PD, recorded while ON and OFF medication and DBS treatment during the same clinical visit. Statistical methods were used to assess whether the model was responsive to changes in gait in the ON and OFF states. RESULTS: The MDS-UPDRS-gait scores predicted by the model were lower on average (representing improved gait; p = 0.017, Cohen's d = 0.495) during the ON medication and DBS treatment conditions. The magnitude of the differences between ON and OFF state was significantly correlated between model predictions and clinician annotations (p = 0.004). The predicted scores were significantly correlated with the clinician scores (Kendall's tau-b = 0.301, p = 0.010), but were distributed in a smaller range as compared to the clinician scores. CONCLUSION: A vision-based model trained on parkinsonian gait did not accurately predict MDS-UPDRS-gait scores in a different PD cohort, but detected weak, but statistically significant proportional changes in response to medication and DBS use. Large, clinically validated datasets of videos captured in many different settings and treatment conditions are required to develop accurate vision-based models of parkinsonian gait.

A pilot observational study of gait changes over time before and after an unplanned hospital visit in long-term care residents with dementia.

BACKGROUND: Older adults with dementia living in long-term care (LTC) have high rates of hospitalization. Two common causes of unplanned hospital visits for LTC residents are deterioration in health status and falls. Early detection of health deterioration or increasing falls risk may present an opportunity to intervene and prevent hospitalization. There is some evidence that impairments in older adults' gait, such as reduced gait speed, increased variability, and poor balance may be associated with hospitalization. However, it is not clear whether changes in gait are observable and measurable before an unplanned hospital visit and whether these changes persist after the acute medical issue has been resolved. The objective of this study was to examine gait changes before and after an unplanned acute care hospital visit in people with dementia. METHODS: We performed a secondary analysis of quantitative gait measures extracted from videos of natural gait captured over time on a dementia care unit and collected information about unplanned hospitalization from health records. RESULTS: Gait changes in study participants before hospital visits were characterized by decreasing stability and step length, and increasing step variability, although these changes were also observed in participants without hospital visits. In an age and sex-adjusted mixed effects model, gait speed and step length declined more quickly in those with a hospital visit compared to those without. CONCLUSIONS: These results provide preliminary evidence that clinically meaningful longitudinal gait changes may be captured by repeated non-invasive gait monitoring, although a larger study is needed to identify changes specific to future medical events.

Vision-based detection and quantification of maternal sleeping position in the third trimester of pregnancy in the home setting-Building the dataset and model.

In 2021, the National Guideline Alliance for the Royal College of Obstetricians and Gynaecologists reviewed the body of evidence, including two meta-analyses, implicating supine sleeping position as a risk factor for growth restriction and stillbirth. While they concluded that pregnant people should be advised to avoid going to sleep on their back after 28 weeks' gestation, their main critique of the evidence was that, to date, all studies were retrospective and sleeping position was not objectively measured. As such, the Alliance noted that it would not be possible to prospectively study the associations between sleeping position and adverse pregnancy outcomes. Our aim was to demonstrate the feasibility of building a vision-based model for automated and accurate detection and quantification of sleeping position throughout the third trimester-a model with the eventual goal to be developed further and used by researchers as a tool to enable them to either confirm or disprove the aforementioned associations. We completed a Canada-wide, cross-sectional study in 24 participants in the third trimester. Infrared videos of eleven simulated sleeping positions unique to pregnancy and a sitting position both with and without bed sheets covering the body were prospectively collected. We extracted 152,618 images from 48 videos, semi-randomly down-sampled and annotated 5,970 of them, and fed them into a deep learning algorithm, which trained and validated six models via six-fold cross-validation. The performance of the models was evaluated using an unseen testing set. The models detected the twelve positions, with and without bed sheets covering the body, achieving an average precision of 0.72 and 0.83, respectively, and an average recall ("sensitivity") of 0.67 and 0.76, respectively. For the supine class with and without bed sheets covering the body, the models achieved an average precision of 0.61 and 0.75, respectively, and an average recall of 0.74 and 0.81, respectively.

Analytical Validation of a Webcam-Based Assessment of Speech Kinematics: Digital Biomarker Evaluation following the V3 Framework.

Introduction: Kinematic analyses have recently revealed a strong potential to contribute to the assessment of neurological diseases. However, the validation of home-based kinematic assessments using consumer-grade video technology has yet to be performed. In line with best practices for digital biomarker development, we sought to validate webcam-based kinematic assessment against established, laboratory-based recording gold standards. We hypothesized that webcam-based kinematics would possess psychometric properties comparable to those obtained using the laboratory-based gold standards. Methods: We collected data from 21 healthy participants who repeated the phrase "buy Bobby a puppy" (BBP) at four different combinations of speaking rate and volume: Slow, Normal, Loud, and Fast. We recorded these samples twice back-to-back, simultaneously using (1) an electromagnetic articulography ("EMA"; NDI Wave) system, (2) a 3D camera (Intel RealSense), and (3) a 2D webcam for video recording via an in-house developed app. We focused on the extraction of kinematic features in this study, given their demonstrated value in detecting neurological impairments. We specifically extracted measures of speed/acceleration, range of motion (ROM), variability, and symmetry using the movements of the center of the lower lip during these tasks. Using these kinematic features, we derived measures of (1) agreement between recording methods, (2) test-retest reliability of each method, and (3) the validity of webcam recordings to capture expected changes in kinematics as a result of different speech conditions. Results: Kinematics measured using the webcam demonstrated good agreement with both the RealSense and EMA (ICC-A values often ≥0.70). Test-retest reliability, measured using the absolute agreement (2,1) formulation of the intraclass correlation coefficient (i.e., ICC-A), was often "moderate" to "strong" (i.e., ≥0.70) and similar between the webcam and EMA-based kinematic features. Finally, the webcam kinematics were typically as sensitive to differences in speech tasks as EMA and the 3D camera gold standards. Discussion and Conclusions: Our results suggested that webcam recordings display good psychometric properties, comparable to laboratory-based gold standards. This work paves the way for a large-scale clinical validation to continue the development of these promising technologies for the assessment of neurological diseases via home-based methods.

Ambient Monitoring of Gait and Machine Learning Models for Dynamic and Short-Term Falls Risk Assessment in People With Dementia.

Falls are a leading cause of morbidity and mortality in older adults with dementia residing in long-term care. Having access to a frequently updated and accurate estimate of the likelihood of a fall over a short time frame for each resident will enable care staff to provide targeted interventions to prevent falls and resulting injuries. To this end, machine learning models to estimate and frequently update the risk of a fall within the next 4 weeks were trained on longitudinal data from 54 older adult participants with dementia. Data from each participant included baseline clinical assessments of gait, mobility, and fall risk at the time of admission, daily medication intake in three medication categories, and frequent assessments of gait performed via a computer vision-based ambient monitoring system. Systematic ablations investigated the effects of various hyperparameters and feature sets and experimentally identified differential contributions from baseline clinical assessments, ambient gait analysis, and daily medication intake. In leave-one-subject-out cross-validation, the best performing model predicts the likelihood of a fall over the next 4 weeks with a sensitivity and specificity of 72.8 and 73.2, respectively, and achieved an area under the receiver operating characteristic curve (AUROC) of 76.2. By contrast, the best model excluding ambient gait features achieved an AUROC of 56.2 with a sensitivity and specificity of 51.9 and 54.0, respectively. Future research will focus on externally validating these findings to prepare for the implementation of this technology to reduce fall and fall-related injuries in long-term care.

Validity of non-contact methods for diagnosis of Obstructive Sleep Apnea: a systematic review and meta-analysis.

STUDY OBJECTIVE: Obstructive Sleep Apnea (OSA) is associated with increased perioperative cardiac, respiratory and neurological complications. Pre-operative OSA risk assessment is currently done through screening questionnaires with high sensitivity but poor specificity. The objective of this study was to evaluate the validity and diagnostic accuracy of portable, non-contact devices in the diagnosis of OSA as compared with polysomnography. DESIGN: This study is a systematic review of English observational cohort studies with meta-analysis and risk of bias assessment. SETTING: Pre-operative, including in the hospital and clinic setting. PATIENTS: Adult patients undergoing sleep apnea assessment using polysomnography and an experimental non-contact tool. INTERVENTIONS: A novel non-contact device, which does not utilize any monitor that makes direct contact with the patient's body, in conjunction with polysomnography. MEASUREMENTS: Primary outcomes included pooled sensitivity and specificity of the experimental device in the diagnosis of obstructive sleep apnea, in comparison to gold-standard polysomnography. RESULTS: Twenty-eight of 4929 screened studies were included in the meta-analysis. A total of 2653 patients were included with the majority being patients referred to a sleep clinic (88.8%). Average age was 49.7(SD±6.1) years, female sex (31%), average body mass index of 29.5(SD±3.2) kg/m2, average apnea-hypopnea index (AHI) of 24.7(SD±5.6) events/h, and pooled OSA prevalence of 72%. Non-contact technology used was mainly video, sound, or bio-motion analysis. Pooled sensitivity and specificity of non-contact methods in moderate to severe OSA diagnosis (AHI > 15) was 0.871 (95% CI 0.841,0.896, I2 0%) and 0.8 (95% CI 0.719,0.862), respectively (AUC 0.902). Risk of bias assessment showed an overall low risk of bias across all domains except for applicability concerns (none were conducted in the perioperative setting). CONCLUSION: Available data indicate contactless methods have high pooled sensitivity and specificity for OSA diagnosis with moderate to high level of evidence. Future research is needed to evaluate these tools in the perioperative setting.

3D Video Tracking Technology in the Assessment of Orofacial Impairments in Neurological Disease: Clinical Validation.

PURPOSE: This study sought to determine whether clinically interpretable kinematic features extracted automatically from three-dimensional (3D) videos were correlated with corresponding perceptual clinical orofacial ratings in individuals with orofacial impairments due to neurological disorders. METHOD: 45 participants (19 diagnosed with motor neuron diseases [MNDs] and 26 poststroke) performed two nonspeech tasks (mouth opening and lip spreading) and one speech task (repetition of a sentence "Buy Bobby a Puppy") while being video-recorded in a standardized lab setting. The color video recordings of participants were assessed by an expert clinician-a speech language pathologist-on the severity of three orofacial measures: symmetry, range of motion (ROM), and speed. Clinically interpretable 3D kinematic features, linked to symmetry, ROM, and speed, were automatically extracted from video recordings, using a deep facial landmark detection and tracking algorithm for each of the three tasks. Spearman correlations were used to identify features that were significantly correlated (p value < .05) with their corresponding clinical scores. Clinically significant kinematic features were then used in the subsequent multivariate regression models to predict the overall orofacial impairment severity score. RESULTS: Several kinematic features extracted from 3D video recordings were associated with their corresponding perceptual clinical scores, indicating clinical validity of these automatically derived measures. Different patterns of significant features were observed between MND and poststroke groups; these differences were aligned with clinical expectations in both cases. CONCLUSIONS: The results show that kinematic features extracted automatically from simple clinical tasks can capture characteristics used by clinicians during assessments. These findings support the clinical validity of video-based automatic extraction of kinematic features.

Feasibility of video-based joint hypermobility assessment in individuals with suspected Ehlers-Danlos syndromes/generalised hypermobility spectrum disorders: a single-site observational study protocol.

INTRODUCTION: Ehlers-Danlos syndromes (EDS)/generalised hypermobility spectrum disorders (G-HSD) affect the connective tissue of the body and present with a heterogeneous set of symptoms that pose a challenge for diagnosis. One of the main diagnostic criteria of EDS/G-HSD is generalised joint hypermobility, which is currently assessed by clinicians during a physical exam. However, the practice for measuring joint hypermobility is inconsistent between clinicians, leading to high inter-rater variability. Often patients are misdiagnosed with EDS/G-HSD based on an incorrect hypermobility assessment, leading to increased referral rates and resource utilisation at specialised EDS clinics that results in unnecessary emotional distress for patients. An objective, validated and scalable method for assessing hypermobility might mitigate these issues and result in improved EDS/G-HSD patient care. METHODS AND ANALYSIS: This study will examine the use of videos obtained using a smartphone camera to assess the range of motion (ROM) and hypermobility of the joints assessed in Beighton score and more (spine, shoulders, elbows, knees, ankles, thumbs and fifth fingers) in individuals with suspected EDS/G-HSD. Short videos of participants will be captured as they undergo a formal assessment of joint hypermobility at the GoodHope EDS Clinic at Toronto General Hospital. Clinicians will measure the ROM at each joint using a clinical-grade goniometer to establish ground truth measurements. Open-source human pose-estimation libraries will be used to extract the locations of key joints from the videos. Deterministic and machine learning systems will be developed and evaluated for estimating the ROM at each joint. Results will be analysed separately for each joint and human pose-estimation library. ETHICS AND DISSEMINATION: This study was approved by the Research Ethics Board of the University Health Network in Toronto on 26 April 2022. Participants will provide written informed consent. Findings from this study will be published in peer-reviewed journals and presented at conferences. TRIAL REGISTRATION NUMBER: NCT05366114.

Video-Based Facial Movement Analysis in the Assessment of Bulbar Amyotrophic Lateral Sclerosis: Clinical Validation.

PURPOSE: Facial movement analysis during facial gestures and speech provides clinically useful information for assessing bulbar amyotrophic lateral sclerosis (ALS). However, current kinematic methods have limited clinical application due to the equipment costs. Recent advancements in consumer-grade hardware and machine/deep learning made it possible to estimate facial movements from videos. This study aimed to establish the clinical validity of a video-based facial analysis for disease staging classification and estimation of clinical scores. METHOD: Fifteen individuals with ALS and 11 controls participated in this study. Participants with ALS were stratified into early and late bulbar ALS groups based on their speaking rate. Participants were recorded with a three-dimensional (3D) camera (color + depth) while repeating a simple sentence 10 times. The lips and jaw movements were estimated, and features related to sentence duration and facial movements were used to train a machine learning model for multiclass classification and to predict the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R) bulbar subscore and speaking rate. RESULTS: The classification model successfully separated healthy controls, the early ALS group, and the late ALS group with an overall accuracy of 96.1%. Video-based features demonstrated a high ability to estimate the speaking rate (adjusted R 2 = .82) and a moderate ability to predict the ALSFRS-R bulbar subscore (adjusted R 2 = .55). CONCLUSIONS: The proposed approach based on a 3D camera and machine learning algorithms represents an easy-to-use and inexpensive system that can be included as part of a clinical assessment of bulbar ALS to integrate facial movement analysis with other clinical data seamlessly.

Reliability of Automatic Computer Vision-Based Assessment of Orofacial Kinematics for Telehealth Applications.

Introduction: Telehealth/remote assessment using readily available 2D mobile cameras and deep learning-based analyses is rapidly becoming a viable option for detecting orofacial and speech impairments associated with neurological and neurodegenerative disease during telehealth practice. However, the psychometric properties (e.g., internal consistency and reliability) of kinematics obtained from these systems have not been established, which is a crucial next step before their clinical usability is established. Methods: Participants were assessed in lab using a 3 dimensional (3D)-capable camera and at home using a readily-available 2D camera in a tablet. Orofacial kinematics was estimated from videos using a deep facial landmark tracking model. Kinematic features quantified the clinically relevant constructs of velocity, range of motion, and lateralization. In lab, all participants performed the same oromotor task. At home, participants were split into two groups that each performed a variant of the in-lab task. We quantified within-assessment consistency (Cronbach's α), reliability (intraclass correlation coefficient [ICC]), and fitted linear mixed-effects models to at-home data to capture individual-/task-dependent longitudinal trajectories. Results: Both in lab and at home, Cronbach's α was typically high (>0.80) and ICCs were often good (>0.70). The linear mixed-effect models that best fit the longitudinal data were those that accounted for individual- or task-dependent effects. Discussion: Remotely gathered orofacial kinematics were as internally consistent and reliable as those gathered in a controlled laboratory setting using a high-performance 3D-capable camera and could additionally capture individual- or task-dependent changes over time. These results highlight the potential of remote assessment tools as digital biomarkers of disease status and progression and demonstrate their suitability for novel telehealth applications.

Concurrent Validity of Zeno Instrumented Walkway and Video-Based Gait Features in Adults With Parkinson's Disease.

BACKGROUND: Parkinson's disease (PD) presents with motor symptoms such as bradykinesia, rigidity, and tremor that can affect gait. To monitor changes associated with disease progression or medication use, quantitative gait assessment is often performed during clinical visits. Conversely, vision-based solutions have been proposed for monitoring gait quality in non-clinical settings. METHODS: We use three 2D human pose-estimation libraries (AlphaPose, Detectron, OpenPose) and one 3D library (ROMP) to calculate gait features from color video, and correlate them with those extracted by a Zeno instrumented walkway in older adults with PD. We calculate video-based gait features using a manual and automated heel-strike detection algorithm, and compare the correlations when the participants walk towards and away from the camera separately. RESULTS: Based on analysis of 67 bidirectional walking bouts from 25 adults with PD, moderate to strong positive correlations were identified between the number of steps, cadence, as well as the mean and coefficient of variation of step width calculated from Zeno and video using 2D pose-estimation libraries. We noted that our automated heel-strike annotation method struggled to identify short steps. CONCLUSION: Gait features calculated from 2D joint trajectories are more strongly correlated with the Zeno than analogous gait features calculated from ROMP. Based on our analysis, videos processed with 2D pose-estimation libraries can be used for longitudinal gait monitoring in individuals with PD. Future work will seek to improve the prediction of gait features using a comprehensive machine learning model to predict gait features directly from color video without relying on intermediate extraction of joint trajectories.

The Toronto older adults gait archive: video and 3D inertial motion capture data of older adults' walking.

We introduce the Toronto Older Adults Gait Archive, a gait dataset of 14 older adults containing 2D video recordings, and 2D (video pose tracking algorithms) and 3D (inertial motion capture) joint locations of the lower body. Participants walked for 60 seconds. We also collected participants' scores on four clinical assessments of gait and balance, namely the Tinneti performance-oriented mobility assessment (POMA-gait and -balance), the Berg balance scale (BBS), and the timed-up-and-go (TUG). Three human pose tracking models (Alphapose, OpenPose, and Detectron) were used to detect body joint positions in 2D video frames and a number of gait parameters were computed using 2D video-based and 3D motion capture data. To show an example usage of our datasets, we performed a correlation analysis between the gait variables and the clinical scores. Our findings revealed that the temporal but not the spatial or variability gait variables from both systems had high correlations to clinical scores. This dataset can be used to evaluate, or to enhance vision-based pose-tracking models to the specifics of older adults' walking.

Detecting and predicting visually induced motion sickness with physiological measures in combination with machine learning techniques.

Visually induced motion sickness (VIMS) is a common sensation when using visual displays such as smartphones or Virtual Reality. In the present study, we investigated whether Machine Learning (ML) techniques in combination with physiological measures (ECG, EDA, EGG, respiration, body and skin temperature, and body movements) could be used to detect and predict the severity of VIMS in real-time, minute-by-minute. A total of 43 healthy younger adults (25 female) were exposed to a 15-minute VIMS-inducing video. VIMS severity was subjectively measured during the video using the Fast Motion Sickness Scale (FMS) as well as before and after the video using the Simulator Sickness Questionnaire (SSQ). Thirty-one participants (72%) experienced VIMS in the present study. Results showed that changes in facial skin temperature and body movement had the strongest relationship with VIMS. On a minute-by-minute basis, ML models revealed a medium correlation between the physiological measures and the FMS scores. An acceptable classification score distinguishing between sick and non-sick participants was found. Our findings suggest that physiological measures may be useful for measuring VIMS, but they are not a reliable standalone method to detect or predict VIMS severity in real-time.

Estimating Parkinsonism Severity in Natural Gait Videos of Older Adults With Dementia.

Drug-induced parkinsonism affects many older adults with dementia, often causing gait disturbances. New advances in vision-based human pose- estimation have opened possibilities for frequent and unobtrusive analysis of gait in long-term care settings. This work leverages spatial-temporal graph convolutional network (ST-GCN) architectures and training procedures to predict clinical scores of parkinsonism in gait from video of individuals with dementia. We propose a two-stage training approach consisting of a self-supervised pretraining stage that encourages the ST-GCN model to learn about gait patterns before predicting clinical scores in the finetuning stage. The proposed ST-GCN models are evaluated on joint trajectories extracted from video and are compared against traditional (ordinal, linear, random forest) regression models and temporal convolutional network baselines. Three 2D human pose-estimation libraries (OpenPose, Detectron, AlphaPose) and the Microsoft Kinect (2D and 3D) are used to extract joint trajectories of 4787 natural walking bouts from 53 older adults with dementia. A subset of 399 walks from 14 participants is annotated with scores of parkinsonism severity on the gait criteria of the Unified Parkinson's Disease Rating Scale (UPDRS) and the Simpson-Angus Scale (SAS). Our results demonstrate that ST-GCN models operating on 3D joint trajectories extracted from the Kinect consistently outperform all other models and feature sets. Prediction of parkinsonism scores in natural walking bouts of unseen participants remains a challenging task, with the best models achieving macro-averaged F1-scores of 0.53 ± 0.03 and 0.40 ± 0.02 for UPDRS-gait and SAS-gait, respectively. Pre-trained model and demo code for this work is available.1.

Prediction of Parkinsonian Gait in Older Adults with Dementia using Joint Trajectories and Gait Features from 2D Video.

Older adults with dementia have a high risk of developing drug-induced parkinsonism; however, formal clinical gait assessments are too infrequent to capture fluctuations in their gait. Camera-based human pose estimation and tracking provides a means to frequently monitor gait in nonclinical settings. In this study, 2160 walking bouts from 49 participants were recorded using a ceiling-mounted camera. Recorded color videos were processed using AlphaPose to obtain 2D joint trajectories of the participant as they were walking down a hallway of the unit. A subset of 324 walking bouts from 14 participants were annotated with clinical scores of parkinsonism on the Unified Parkinson's Disease Rating Scale (UPDRS)-gait scale. Linear, random forest, and ordinal logistic regression models were evaluated for regression to UPDRS-gait scores using engineered 2D gait features calculated from the AlphaPose joint trajectories. Additionally, spatial temporal graph convolutional networks (ST-GCNs) were trained to predict UPDRS-gait scores from joint trajectories and gait features using a two-stage training scheme (self-supervised pretraining stage on all walks followed by a finetuning stage on labelled walks). All models were trained using leave-one-subject-out cross-validation to simulate testing on previously unseen participants. The macro-averaged F1-score was 0.333 for the best model operating on only gait features and 0.372 for the top ST-GCN model that used both joint trajectories and gait features as input. When accepting predicted scores that were only off by at most 1 point on the UPDRS-gait scale, the accuracy of the model that only used gait features was 82.8%, while the model that also used joint trajectories had an accuracy of 94.2%.Clinical Relevance- The combination of gait features and joint trajectories capture parkinsonian qualities in gait better than either group of data individually.

Virtual Hand Illusion in younger and older adults.

INTRODUCTION: Embodiment involves experiencing ownership over our body and localizing it in space and is informed by multiple senses (visual, proprioceptive and tactile). Evidence suggests that embodiment and multisensory integration may change with older age. The Virtual Hand Illusion (VHI) has been used to investigate multisensory contributions to embodiment, but has never been evaluated in older adults. Spatio-temporal factors unique to virtual environments may differentially affect the embodied perceptions of older and younger adults. METHODS: Twenty-one younger (18-35 years) and 19 older (65+ years) adults completed the VHI paradigm. Body localization was measured at baseline and again, with subjective ownership ratings, following synchronous and asynchronous visual-tactile interactions. RESULTS: Higher ownership ratings were observed in the synchronous relative to the asynchronous condition, but no effects on localization/drift were found. No age differences were observed. Localization accuracy was biased in both age groups when the virtual hand was aligned with the real hand, indicating a visual mislocalization of the virtual hand. CONCLUSIONS: No age-related differences in the VHI were observed. Mislocalization of the hand in VR occurred for both groups, even when congruent and aligned; however, tactile feedback reduced localization biases. Our results expand the current understanding of age-related changes in multisensory embodiment within virtual environments.