Facial and Vocal Markers of Schizophrenia Measured Using Remote Smartphone Assessments: Observational Study.

BACKGROUND: Machine learning-based facial and vocal measurements have demonstrated relationships with schizophrenia diagnosis and severity. Demonstrating utility and validity of remote and automated assessments conducted outside of controlled experimental or clinical settings can facilitate scaling such measurement tools to aid in risk assessment and tracking of treatment response in populations that are difficult to engage. OBJECTIVE: This study aimed to determine the accuracy of machine learning-based facial and vocal measurements acquired through automated assessments conducted remotely through smartphones. METHODS: Measurements of facial and vocal characteristics including facial expressivity, vocal acoustics, and speech prevalence were assessed in 20 patients with schizophrenia over the course of 2 weeks in response to two classes of prompts previously utilized in experimental laboratory assessments: evoked prompts, where subjects are guided to produce specific facial expressions and speech; and spontaneous prompts, where subjects are presented stimuli in the form of emotionally evocative imagery and asked to freely respond. Facial and vocal measurements were assessed in relation to schizophrenia symptom severity using the Positive and Negative Syndrome Scale. RESULTS: Vocal markers including speech prevalence, vocal jitter, fundamental frequency, and vocal intensity demonstrated specificity as markers of negative symptom severity, while measurement of facial expressivity demonstrated itself as a robust marker of overall schizophrenia symptom severity. CONCLUSIONS: Established facial and vocal measurements, collected remotely in schizophrenia patients via smartphones in response to automated task prompts, demonstrated accuracy as markers of schizophrenia symptom severity. Clinical implications are discussed.

Remote Digital Measurement of Facial and Vocal Markers of Major Depressive Disorder Severity and Treatment Response: A Pilot Study.

Objectives: Multiple machine learning-based visual and auditory digital markers have demonstrated associations between major depressive disorder (MDD) status and severity. The current study examines if such measurements can quantify response to antidepressant treatment (ADT) with selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine uptake inhibitors (SNRIs). Methods: Visual and auditory markers were acquired through an automated smartphone task that measures facial, vocal, and head movement characteristics across 4 weeks of treatment (with time points at baseline, 2 weeks, and 4 weeks) on ADT (n = 18). MDD diagnosis was confirmed using the Mini-International Neuropsychiatric Interview (MINI), and the Montgomery-Åsberg Depression Rating Scale (MADRS) was collected concordantly to assess changes in MDD severity. Results: Patient responses to ADT demonstrated clinically and statistically significant changes in the MADRS [F (2, 34) = 51.62, p < 0.0001]. Additionally, patients demonstrated significant increases in multiple digital markers including facial expressivity, head movement, and amount of speech. Finally, patients demonstrated significantly decreased frequency of fear and anger facial expressions. Conclusion: Digital markers associated with MDD demonstrate validity as measures of treatment response.

Technology for Measuring and Monitoring Treatment Compliance Remotely.

Medication non-adherence during clinical trials is an ongoing challenge that can result in insufficient safety and efficacy data. For patients with Parkinson's disease and other neurological disorders, symptomatology such as forgetfulness compounds traditional obstacles to adherence. Today, sponsors and clinical study sites can call upon various technology tools that improve adherence by monitoring and confirming dosage in near real-time. These tools have the potential to improve the quality of data gleaned from these studies.

Validation of Visual and Auditory Digital Markers of Suicidality in Acutely Suicidal Psychiatric Inpatients: Proof-of-Concept Study.

BACKGROUND: Multiple symptoms of suicide risk have been assessed based on visual and auditory information, including flattened affect, reduced movement, and slowed speech. Objective quantification of such symptomatology from novel data sources can increase the sensitivity, scalability, and timeliness of suicide risk assessment. OBJECTIVE: We aimed to examine measurements extracted from video interviews using open-source deep learning algorithms to quantify facial, vocal, and movement behaviors in relation to suicide risk severity in recently admitted patients following a suicide attempt. METHODS: We utilized video to quantify facial, vocal, and movement markers associated with mood, emotion, and motor functioning from a structured clinical conversation in 20 patients admitted to a psychiatric hospital following a suicide risk attempt. Measures were calculated using open-source deep learning algorithms for processing facial expressivity, head movement, and vocal characteristics. Derived digital measures of flattened affect, reduced movement, and slowed speech were compared to suicide risk with the Beck Scale for Suicide Ideation controlling for age and sex, using multiple linear regression. RESULTS: Suicide severity was associated with multiple visual and auditory markers, including speech prevalence (ß=-0.68, P=.02, r2=0.40), overall expressivity (ß=-0.46, P=.10, r2=0.27), and head movement measured as head pitch variability (ß=-1.24, P=.006, r2=0.48) and head yaw variability (ß=-0.54, P=.06, r2=0.32). CONCLUSIONS: Digital measurements of facial affect, movement, and speech prevalence demonstrated strong effect sizes and linear associations with the severity of suicidal ideation.

Computer Vision-Based Assessment of Motor Functioning in Schizophrenia: Use of Smartphones for Remote Measurement of Schizophrenia Symptomatology.

INTRODUCTION: Motor abnormalities have been shown to be a distinct component of schizophrenia symptomatology. However, objective and scalable methods for assessment of motor functioning in schizophrenia are lacking. Advancements in machine learning-based digital tools have allowed for automated and remote "digital phenotyping" of disease symptomatology. Here, we assess the performance of a computer vision-based assessment of motor functioning as a characteristic of schizophrenia using video data collected remotely through smartphones. METHODS: Eighteen patients with schizophrenia and 9 healthy controls were asked to remotely participate in smartphone-based assessments daily for 14 days. Video recorded from the smartphone front-facing camera during these assessments was used to quantify the Euclidean distance of head movement between frames through a pretrained computer vision model. The ability of head movement measurements to distinguish between patients and healthy controls as well as their relationship to schizophrenia symptom severity as measured through traditional clinical scores was assessed. RESULTS: The rate of head movement in participants with schizophrenia (1.48 mm/frame) and those without differed significantly (2.50 mm/frame; p = 0.01), and a logistic regression demonstrated that head movement was a significant predictor of schizophrenia diagnosis (p = 0.02). Linear regression between head movement and clinical scores of schizophrenia showed that head movement has a negative relationship with schizophrenia symptom severity (p = 0.04), primarily with negative symptoms of schizophrenia. CONCLUSIONS: Remote, smartphone-based assessments were able to capture meaningful visual behavior for computer vision-based objective measurement of head movement. The measurements of head movement acquired were able to accurately classify schizophrenia diagnosis and quantify symptom severity in patients with schizophrenia.

Accuracy of machine learning-based prediction of medication adherence in clinical research.

Medication non-adherence represents a significant barrier to treatment efficacy. Remote, real-time measurement of medication dosing can facilitate dynamic prediction of risk for medication non-adherence, which in-turn allows for proactive clinical intervention to optimize health outcomes. We examine the accuracy of dynamic prediction of non-adherence using data from remote real-time measurements of medication dosing. Participants across a large set of clinical trials (n = 4,182) were observed via a smartphone application that video records patients taking their prescribed medication. The patients' primary diagnosis, demographics, and prior indication of observed adherence/non-adherence were utilized to predict (1) adherence rates ≥ 80% across the clinical trial, (2) adherence ≥ 80% for the subsequent week, and (3) adherence the subsequent day using machine learning-based classification models. Empirically observed adherence was demonstrated to be the strongest predictor of future adherence/non-adherence. Collectively, the classification models accurately predicted adherence across the trial (AUC = 0.83), the subsequent week (AUC = 0.87) and the subsequent day (AUC = 0.87). Real-time measurement of dosing can be utilized to dynamically predict medication adherence with high accuracy.