Multimodal Mental Health Digital Biomarker Analysis From Remote Interviews Using Facial, Vocal, Linguistic, and Cardiovascular Patterns.

OBJECTIVE: Psychiatric evaluation suffers from subjectivity and bias, and is hard to scale due to intensive professional training requirements. In this work, we investigated whether behavioral and physiological signals, extracted from tele-video interviews, differ in individuals with psychiatric disorders. METHODS: Temporal variations in facial expression, vocal expression, linguistic expression, and cardiovascular modulation were extracted from simultaneously recorded audio and video of remote interviews. Averages, standard deviations, and Markovian process-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination. RESULTS: Statistically significant feature differences were found between psychiatric and control subjects. Correlations were found between features and self-rated depression and anxiety scores. Heart rate dynamics provided the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities provided AUROCs of 0.72-0.82. CONCLUSION: Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status. SIGNIFICANCE: The proposed multimodal approach has the potential to facilitate scalable, remote, and low-cost assessment for low-burden automated mental health services.

Disentangling Visual Exploration Differences in Cognitive Impairment.

OBJECTIVE: Individuals with cognitive impairment (CI) exhibit different oculomotor functions and viewing behaviors. In this work we aimed to quantify the differences in these functions with CI severity, and assess general CI and specific cognitive functions related to visual exploration behaviors. METHODS: A validated passive viewing memory test with eyetracking was administered to 348 healthy controls and CI individuals. Spatiotemporal properties of the scanpath, the semantic category of the viewed regions, and other composite features were extracted from the estimated eyegaze locations on the corresponding pictures displayed during the test. These features were then used to characterize viewing patterns, classify cognitive impairment, and estimate scores in various neuropsychological tests using machine learning. RESULTS: Statistically significant differences in spatial, spatiotemporal, and semantic features were found between healthy controls and individuals with CI. The CI group spent more time gazing at the center of the image, looked at more regions of interest (ROI), transitioned less often between ROI yet in a more unpredictable manner, and exhibited different semantic preferences. A combination of these features achieved an area under the receiver-operator curve of 0.78 in differentiating CI individuals from controls. Statistically significant correlations were identified between actual and estimated CI scores and other neuropsychological tests. CONCLUSION: Evaluating visual exploration behaviors provided quantitative and systematic evidence of differences in CI individuals, leading to an improved approach for passive cognitive impairment screening. SIGNIFICANCE: The proposed passive, accessible, and scalable approach could help with earlier detection and a better understanding of cognitive impairment.

Using HIPAA (Health Insurance Portability and Accountability Act)-Compliant Transcription Services for Virtual Psychiatric Interviews: Pilot Comparison Study.

BACKGROUND: Automatic speech recognition (ASR) technology is increasingly being used for transcription in clinical contexts. Although there are numerous transcription services using ASR, few studies have compared the word error rate (WER) between different transcription services among different diagnostic groups in a mental health setting. There has also been little research into the types of words ASR transcriptions mistakenly generate or omit. OBJECTIVE: This study compared the WER of 3 ASR transcription services (Amazon Transcribe [Amazon.com, Inc], Zoom-Otter AI [Zoom Video Communications, Inc], and Whisper [OpenAI Inc]) in interviews across 2 different clinical categories (controls and participants experiencing a variety of mental health conditions). These ASR transcription services were also compared with a commercial human transcription service, Rev (Rev.Com, Inc). Words that were either included or excluded by the error in the transcripts were systematically analyzed by their Linguistic Inquiry and Word Count categories. METHODS: Participants completed a 1-time research psychiatric interview, which was recorded on a secure server. Transcriptions created by the research team were used as the gold standard from which WER was calculated. The interviewees were categorized into either the control group (n=18) or the mental health condition group (n=47) using the Mini-International Neuropsychiatric Interview. The total sample included 65 participants. Brunner-Munzel tests were used for comparing independent sets, such as the diagnostic groupings, and Wilcoxon signed rank tests were used for correlated samples when comparing the total sample between different transcription services. RESULTS: There were significant differences between each ASR transcription service's WER (P<.001). Amazon Transcribe's output exhibited significantly lower WERs compared with the Zoom-Otter AI's and Whisper's ASR. ASR performances did not significantly differ across the 2 different clinical categories within each service (P>.05). A comparison between the human transcription service output from Rev and the best-performing ASR (Amazon Transcribe) demonstrated a significant difference (P<.001), with Rev having a slightly lower median WER (7.6%, IQR 5.4%-11.35 vs 8.9%, IQR 6.9%-11.6%). Heat maps and spider plots were used to visualize the most common errors in Linguistic Inquiry and Word Count categories, which were found to be within 3 overarching categories: Conversation, Cognition, and Function. CONCLUSIONS: Overall, consistent with previous literature, our results suggest that the WER between manual and automated transcription services may be narrowing as ASR services advance. These advances, coupled with decreased cost and time in receiving transcriptions, may make ASR transcriptions a more viable option within health care settings. However, more research is required to determine if errors in specific types of words impact the analysis and usability of these transcriptions, particularly for specific applications and in a variety of populations in terms of clinical diagnosis, literacy level, accent, and cultural origin.

Multimodal mental health assessment with remote interviews using facial, vocal, linguistic, and cardiovascular patterns.

Objective: The current clinical practice of psychiatric evaluation suffers from subjectivity and bias, and requires highly skilled professionals that are often unavailable or unaffordable. Objective digital biomarkers have shown the potential to address these issues. In this work, we investigated whether behavioral and physiological signals, extracted from remote interviews, provided complimentary information for assessing psychiatric disorders. Methods: Time series of multimodal features were derived from four conceptual modes: facial expression, vocal expression, linguistic expression, and cardiovascular modulation. The features were extracted from simultaneously recorded audio and video of remote interviews using task-specific and foundation models. Averages, standard deviations, and hidden Markov model-derived statistics of these features were computed from 73 subjects. Four binary classification tasks were defined: detecting 1) any clinically-diagnosed psychiatric disorder, 2) major depressive disorder, 3) self-rated depression, and 4) self-rated anxiety. Each modality was evaluated individually and in combination. Results: Statistically significant feature differences were found between controls and subjects with mental health conditions. Correlations were found between features and self-rated depression and anxiety scores. Visual heart rate dynamics achieved the best unimodal performance with areas under the receiver-operator curve (AUROCs) of 0.68-0.75 (depending on the classification task). Combining multiple modalities achieved AUROCs of 0.72-0.82. Features from task-specific models outperformed features from foundation models. Conclusion: Multimodal features extracted from remote interviews revealed informative characteristics of clinically diagnosed and self-rated mental health status. Significance: The proposed multimodal approach has the potential to facilitate objective, remote, and low-cost assessment for low-burden automated mental health services.

Disentangling visual exploration differences in cognitive impairment.

Objective: Compared to individuals without cognitive impairment (CI), those with CI exhibit differences in both basic oculomotor functions and complex viewing behaviors. However, the characteristics of the differences and how those differences relate to various cognitive functions have not been widely explored. In this work we aimed to quantify those differences and assess general cognitive impairment and specific cognitive functions. Methods: A validated passive viewing memory test with eyetracking was administered to 348 healthy controls and CI individuals. Spatial, temporal, semantic, and other composite features were extracted from the estimated eye-gaze locations on the corresponding pictures displayed during the test. These features were then used to characterize viewing patterns, classify cognitive impairment, and estimate scores in various neuropsychological tests using machine learning. Results: Statistically significant differences in spatial, spatiotemporal, and semantic features were found between healthy controls and individuals with CI. CI group spent more time gazing at the center of the image, looked at more regions of interest (ROI), transitioned less often between ROI yet in a more unpredictable manner, and had different semantic preferences. A combination of these features achieved an area under the receiver-operator curve of 0.78 in differentiating CI individuals from controls. Statistically significant correlations were identified between actual and estimated MoCA scores and other neuropsychological tests. Conclusion: Evaluating visual exploration behaviors provided quantitative and systematic evidence of differences in CI individuals, leading to an improved approach for passive cognitive impairment screening. Significance: The proposed passive, accessible, and scalable approach could help with earlier detection and a better understanding of cognitive impairment.

An investigation of privacy preservation in deep learning-based eye-tracking.

BACKGROUND: The expanding usage of complex machine learning methods such as deep learning has led to an explosion in human activity recognition, particularly applied to health. However, complex models which handle private and sometimes protected data, raise concerns about the potential leak of identifiable data. In this work, we focus on the case of a deep network model trained on images of individual faces. MATERIALS AND METHODS: A previously published deep learning model, trained to estimate the gaze from full-face image sequences was stress tested for personal information leakage by a white box inference attack. Full-face video recordings taken from 493 individuals undergoing an eye-tracking- based evaluation of neurological function were used. Outputs, gradients, intermediate layer outputs, loss, and labels were used as inputs for a deep network with an added support vector machine emission layer to recognize membership in the training data. RESULTS: The inference attack method and associated mathematical analysis indicate that there is a low likelihood of unintended memorization of facial features in the deep learning model. CONCLUSIONS: In this study, it is showed that the named model preserves the integrity of training data with reasonable confidence. The same process can be implemented in similar conditions for different models.

Issues in the automated classification of multilead ecgs using heterogeneous labels and populations.

Objective.The standard twelve-lead electrocardiogram (ECG) is a widely used tool for monitoring cardiac function and diagnosing cardiac disorders. The development of smaller, lower-cost, and easier-to-use ECG devices may improve access to cardiac care in lower-resource environments, but the diagnostic potential of these devices is unclear. This work explores these issues through a public competition: the 2021 PhysioNet Challenge. In addition, we explore the potential for performance boosting through a meta-learning approach.Approach.We sourced 131,149 twelve-lead ECG recordings from ten international sources. We posted 88,253 annotated recordings as public training data and withheld the remaining recordings as hidden validation and test data. We challenged teams to submit containerized, open-source algorithms for diagnosing cardiac abnormalities using various ECG lead combinations, including the code for training their algorithms. We designed and scored the algorithms using an evaluation metric that captures the risks of different misdiagnoses for 30 conditions. After the Challenge, we implemented a semi-consensus voting model on all working algorithms.Main results.A total of 68 teams submitted 1,056 algorithms during the Challenge, providing a variety of automated approaches from both academia and industry. The performance differences across the different lead combinations were smaller than the performance differences across the different test databases, showing that generalizability posed a larger challenge to the algorithms than the choice of ECG leads. A voting model improved performance by 3.5%.Significance.The use of different ECG lead combinations allowed us to assess the diagnostic potential of reduced-lead ECG recordings, and the use of different data sources allowed us to assess the generalizability of the algorithms to diverse institutions and populations. The submission of working, open-source code for both training and testing and the use of a novel evaluation metric improved the reproducibility, generalizability, and applicability of the research conducted during the Challenge.

Multimodal Assessment of Schizophrenia and Depression Utilizing Video, Acoustic, Locomotor, Electroencephalographic, and Heart Rate Technology: Protocol for an Observational Study.

BACKGROUND: Current standards of psychiatric assessment and diagnostic evaluation rely primarily on the clinical subjective interpretation of a patient's outward manifestations of their internal state. While psychometric tools can help to evaluate these behaviors more systematically, the tools still rely on the clinician's interpretation of what are frequently nuanced speech and behavior patterns. With advances in computing power, increased availability of clinical data, and improving resolution of recording and sensor hardware (including acoustic, video, accelerometer, infrared, and other modalities), researchers have begun to demonstrate the feasibility of cutting-edge technologies in aiding the assessment of psychiatric disorders. OBJECTIVE: We present a research protocol that utilizes facial expression, eye gaze, voice and speech, locomotor, heart rate, and electroencephalography monitoring to assess schizophrenia symptoms and to distinguish patients with schizophrenia from those with other psychiatric disorders and control subjects. METHODS: We plan to recruit three outpatient groups: (1) 50 patients with schizophrenia, (2) 50 patients with unipolar major depressive disorder, and (3) 50 individuals with no psychiatric history. Using an internally developed semistructured interview, psychometrically validated clinical outcome measures, and a multimodal sensing system utilizing video, acoustic, actigraphic, heart rate, and electroencephalographic sensors, we aim to evaluate the system's capacity in classifying subjects (schizophrenia, depression, or control), to evaluate the system's sensitivity to within-group symptom severity, and to determine if such a system can further classify variations in disorder subtypes. RESULTS: Data collection began in July 2020 and is expected to continue through December 2022. CONCLUSIONS: If successful, this study will help advance current progress in developing state-of-the-art technology to aid clinical psychiatric assessment and treatment. If our findings suggest that these technologies are capable of resolving diagnoses and symptoms to the level of current psychometric testing and clinician judgment, we would be among the first to develop a system that can eventually be used by clinicians to more objectively diagnose and assess schizophrenia and depression with the possibility of less risk of bias. Such a tool has the potential to improve accessibility to care; to aid clinicians in objectively evaluating diagnoses, severity of symptoms, and treatment efficacy through time; and to reduce treatment-related morbidity. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/36417.

Utilizing computer vision for facial behavior analysis in schizophrenia studies: A systematic review.

BACKGROUND: Schizophrenia is a severe psychiatric disorder that causes significant social and functional impairment. Currently, the diagnosis of schizophrenia is based on information gleaned from the patient's self-report, what the clinician observes directly, and what the clinician gathers from collateral informants, but these elements are prone to subjectivity. Utilizing computer vision to measure facial expressions is a promising approach to adding more objectivity in the evaluation and diagnosis of schizophrenia. METHOD: We conducted a systematic review using PubMed and Google Scholar. Relevant publications published before (including) December 2021 were identified and evaluated for inclusion. The objective was to conduct a systematic review of computer vision for facial behavior analysis in schizophrenia studies, the clinical findings, and the corresponding data processing and machine learning methods. RESULTS: Seventeen studies published between 2007 to 2021 were included, with an increasing trend in the number of publications over time. Only 14 articles used interviews to collect data, of which different combinations of passive to evoked, unstructured to structured interviews were used. Various types of hardware were adopted and different types of visual data were collected. Commercial, open-access, and in-house developed models were used to recognize facial behaviors, where frame-level and subject-level features were extracted. Statistical tests and evaluation metrics varied across studies. The number of subjects ranged from 2-120, with an average of 38. Overall, facial behaviors appear to have a role in estimating diagnosis of schizophrenia and psychotic symptoms. When studies were evaluated with a quality assessment checklist, most had a low reporting quality. CONCLUSION: Despite the rapid development of computer vision techniques, there are relatively few studies that have applied this technology to schizophrenia research. There was considerable variation in the clinical paradigm and analytic techniques used. Further research is needed to identify and develop standardized practices, which will help to promote further advances in the field.

Automated analysis of facial emotions in subjects with cognitive impairment.

Differences in expressing facial emotions are broadly observed in people with cognitive impairment. However, these differences have been difficult to objectively quantify and systematically evaluate among people with cognitive impairment across disease etiologies and severity. Therefore, a computer vision-based deep learning model for facial emotion recognition trained on 400.000 faces was utilized to analyze facial emotions expressed during a passive viewing memory test. In addition, this study was conducted on a large number of individuals (n = 493), including healthy controls and individuals with cognitive impairment due to diverse underlying etiologies and across different disease stages. Diagnoses included subjective cognitive impairment, Mild Cognitive Impairment (MCI) due to AD, MCI due to other etiologies, dementia due to Alzheimer's diseases (AD), and dementia due to other etiologies (e.g., Vascular Dementia, Frontotemporal Dementia, Lewy Body Dementia, etc.). The Montreal Cognitive Assessment (MoCA) was used to evaluate cognitive performance across all participants. A participant with a score of less than or equal to 24 was considered cognitively impaired (CI). Compared to cognitively unimpaired (CU) participants, CI participants expressed significantly less positive emotions, more negative emotions, and higher facial expressiveness during the test. In addition, classification analysis revealed that facial emotions expressed during the test allowed effective differentiation of CI from CU participants, largely independent of sex, race, age, education level, mood, and eye movements (derived from an eye-tracking-based digital biomarker for cognitive impairment). No screening methods reliably differentiated the underlying etiology of the cognitive impairment. The findings provide quantitative and comprehensive evidence that the expression of facial emotions is significantly different in people with cognitive impairment, and suggests this may be a useful tool for passive screening of cognitive impairment.

Classification of 12-lead ECGs: the PhysioNet/Computing in Cardiology Challenge 2020.

OBJECTIVE: Vast 12-lead ECGs repositories provide opportunities to develop new machine learning approaches for creating accurate and automatic diagnostic systems for cardiac abnormalities. However, most 12-lead ECG classification studies are trained, tested, or developed in single, small, or relatively homogeneous datasets. In addition, most algorithms focus on identifying small numbers of cardiac arrhythmias that do not represent the complexity and difficulty of ECG interpretation. This work addresses these issues by providing a standard, multi-institutional database and a novel scoring metric through a public competition: the PhysioNet/Computing in Cardiology Challenge 2020. APPROACH: A total of 66 361 12-lead ECG recordings were sourced from six hospital systems from four countries across three continents; 43 101 recordings were posted publicly with a focus on 27 diagnoses. For the first time in a public competition, we required teams to publish open-source code for both training and testing their algorithms, ensuring full scientific reproducibility. MAIN RESULTS: A total of 217 teams submitted 1395 algorithms during the Challenge, representing a diversity of approaches for identifying cardiac abnormalities from both academia and industry. As with previous Challenges, high-performing algorithms exhibited significant drops ([Formula: see text]10%) in performance on the hidden test data. SIGNIFICANCE: Data from diverse institutions allowed us to assess algorithmic generalizability. A novel evaluation metric considered different misclassification errors for different cardiac abnormalities, capturing the outcomes and risks of different diagnoses. Requiring both trained models and code for training models improved the generalizability of submissions, setting a new bar in reproducibility for public data science competitions.

Screening of Coulomb interactions in liquid dielectrics.

The interaction of charges in dielectric materials is screened by the dielectric constant of the bulk dielectric. In dielectric theories, screening is assigned to surface charges appearing from preferential orientations of dipoles along the local field in the interface. For liquid dielectrics, such interfacial orientations are affected by the interfacial structure characterized by a separate interfacial dipolar susceptibility. We argue that dielectric properties of polar liquids should be characterized by two distinct susceptibilities responsible for local response (solvation) and long-range response (dielectric screening). A microscopic model for dipolar screening in polar liquids is developed here. It shows that the standard bulk dielectric constant is responsible for screening at large distances. The potential of mean force between ions in polar liquids becomes oscillatory at short distances. Oscillations arise from the coupling of collective longitudinal excitations (dipolarons) of the polar liquid with its interfacial density profile. Interfacial electrostatics demonstrates a crossover beyond the solute radius of ∼1 nm from the scaling laws roughly consistent with continuum expectations for small solutes to new scaling trends characterizing the much softer nano-scale solute-solvent interface. This crossover also marks the transition to a continuum-type electrostatic screening between ions, when short-distance oscillation of the potential of mean force become strongly damped. Screening oscillations are enhanced for more structured interfaces.

Protein Dielectrophoresis in Solution.

Proteins experience either pulling or repelling force from the gradient of an external electric field due to the effect known as dielectrophoresis (DEP). The susceptibility to the field gradient is traditionally calculated from the solution of the electrostatic boundary-value problem, which requires assigning a dielectric constant to the protein. This assignment is essential since the DEP susceptibility is proportional, in dielectric theories, to the Clausius-Mossotti factor, the sign of which is controlled by whether the protein dielectric constant is below (repelling) or above (pulling) the dielectric constant of water. The dielectric constant is not uniquely or even well-defined for a particle of molecular size and the Clausius-Mossotti factor is shown here to be inadequate for describing the dipolar response of the protein and hydration water. An alternative theory is developed from the standpoint of molecular properties of the protein solute and water solvent. The effective polarity of the protein molecule enters the theory in terms of the variance of its molecular dipole moment and its refractive index. Molecular dynamics (MD) simulations of the protein cytochrome c in solution are performed to calculate the dipolar susceptibilities entering the theory. We find that tumbling of the protein on the nanosecond time scale results in a positive DEP (pulling). The DEP susceptibility for cytochrome c acquired from MD simulations is 103-104 times higher than predicted by the Clausius-Mossotti factor. Nevertheless, this high DEP susceptibility is fully consistent with empirically confirmed Oncley's equation connecting the protein dipole to dielectric increments of protein solutions. For cytochrome c, high DEP susceptibilities calculated from MD are consistent with experimental dielectric data. We provide a general relation connecting the DEP susceptibility to the dielectric increment of solution.

Termination of Biological Function at Low Temperatures: Glass or Structural Transition?

Energy of life is produced by electron transfer in energy chains of respiration or photosynthesis. A small input of free energy available to biology puts significant restrictions on how much free energy can be lost in each electron-transfer reaction. We advocate the view that breaking ergodicity, leading to violation of the fluctuation-dissipation theorem (FDT), is how proteins achieve high reaction rates without sacrificing the reaction free energy. Here we show that a significant level of nonergodicity, represented by a large extent of the configurational temperature over the kinetic temperature, is maintained in the entire physiological range for the cytochrome c electron transfer protein. The protein returns to the state consistent with the FDT below the crossover temperature close to the temperature of the protein glass transition. This crossover leads to a sharp increase in the activation barrier of electron transfer and is displayed by a kink in the Arrhenius plot for the reaction rate constant.

Ergodicity breaking of iron displacement in heme proteins.

We present a model of the dynamical transition of atomic displacements in proteins. Increased mean-square displacement at higher temperatures is caused by the softening of the force constant for atomic/molecular displacements by electrostatic and van der Waals forces from the protein-water thermal bath. Displacement softening passes through a nonergodic dynamical transition when the relaxation time of the force-force correlation function enters, with increasing temperature, the instrumental observation window. Two crossover temperatures are identified. The lower crossover, presently connected to the glass transition, is related to the dynamical unfreezing of rotations of water molecules within nanodomains polarized by charged surface residues of the protein. The higher crossover temperature, usually assigned to the dynamical transition, marks the onset of water translations. All crossovers are ergodicity breaking transitions depending on the corresponding observation windows. Allowing stretched exponential relaxation of the protein-water thermal bath significantly improves the theory-experiment agreement when applied to solid protein samples studied by Mössbauer spectroscopy.

Theory and Electrochemistry of Cytochrome c.

Extensive simulations of cytochrome c in solution are performed to address the apparent contradiction between large reorganization energies of protein electron transfer typically reported by atomistic simulations and much smaller values produced by protein electrochemistry. The two sets of data are reconciled by deriving the activation barrier for electrochemical reaction in terms of an effective reorganization energy composed of half the Stokes shift (characterizing the medium polarization in response to electron transfer) and the variance reorganization energy (characterizing the breadth of electrostatic fluctuations). This effective reorganization energy is much smaller than each of the two components contributing to it and is fully consistent with electrochemical measurements. Calculations in the range of temperatures between 280 and 360 K combine long, classical molecular dynamics simulations with quantum calculations of the protein active site. The results agree with the Arrhenius plots for the reaction rates and with cyclic voltammetry of cytochrome c immobilized on self-assembled monolayers. Small effective reorganization energy, and the resulting small activation barrier, is a general phenomenology of protein electron transfer allowing fast electron transport within biological energy chains.

Dynamical and orientational structural crossovers in low-temperature glycerol.

Mean-square displacements of hydrogen atoms in glass-forming materials and proteins, as reported by incoherent elastic neutron scattering, show kinks in their temperature dependence. This crossover, known as the dynamical transition, connects two approximately linear regimes. It is often assigned to the dynamical freezing of subsets of molecular modes at the point of equality between their corresponding relaxation times and the instrumental observation window. The origin of the dynamical transition in glass-forming glycerol is studied here by extensive molecular dynamics simulations. We find the dynamical transition to occur for both the center-of-mass translations and the molecular rotations at the same temperature, insensitive to changes of the observation window. Both the translational and rotational dynamics of glycerol show a dynamic crossover from the structural to a secondary relaxation at the temperature of the dynamical transition. A significant and discontinuous increase in the orientational Kirkwood factor and in the dielectric constant is observed in the same range of temperatures. No indication is found of a true thermodynamic transition to an ordered low-temperature phase. We therefore suggest that all observed crossovers are dynamic in character. The increase in the dielectric constant is related to the dynamic freezing of dipolar domains on the time scale of simulations.