Automatic detection of expressed emotion from Five-Minute Speech Samples: Challenges and opportunities.

Research into clinical applications of speech-based emotion recognition (SER) technologies has been steadily increasing over the past few years. One such potential application is the automatic recognition of expressed emotion (EE) components within family environments. The identification of EE is highly important as they have been linked with a range of adverse life events. Manual coding of these events requires time-consuming specialist training, amplifying the need for automated approaches. Herein we describe an automated machine learning approach for determining the degree of warmth, a key component of EE, from acoustic and text natural language features. Our dataset of 52 recorded interviews is taken from recordings, collected over 20 years ago, from a nationally representative birth cohort of British twin children, and was manually coded for EE by two researchers (inter-rater reliability 0.84-0.90). We demonstrate that the degree of warmth can be predicted with an F1-score of 64.7% despite working with audio recordings of highly variable quality. Our highly promising results suggest that machine learning may be able to assist in the coding of EE in the near future.

Speech patterns in responses to questions asked by an intelligent virtual agent can help to distinguish between people with early stage neurodegenerative disorders and healthy controls.

Previous research has provided strong evidence that speech patterns can help to distinguish between people with early stage neurodegenerative disorders (ND) and healthy controls. This study examined speech patterns in responses to questions asked by an intelligent virtual agent (IVA): a talking head on a computer which asks pre-recorded questions. The study investigated whether measures of response length, speech rate and pausing in responses to questions asked by an IVA help to distinguish between healthy control participants and people diagnosed with Mild Cognitive Impairment (MCI) or Alzheimer's disease (AD). The study also considered whether those measures can further help to distinguish between people with MCI, people with AD, and healthy control participants (HC). There were 38 people with ND (31 people with MCI, 7 people with AD) and 26 HC. All interactions took place in English. People with MCI spoke fewer words compared to HC, and people with AD and people with MCI spoke for less time than HC. People with AD spoke at a slower rate than people with MCI and HC. There were significant differences across all three groups for the proportion of time spent pausing and the average pause duration: silent pauses make up the greatest proportion of responses from people with AD, who also have the longest average silent pause duration, followed by people with MCI then HC. Therefore, the study demonstrates the potential of an IVA as a method for collecting data showing patterns which can help to distinguish between diagnostic groups.

Characterising spoken responses to an intelligent virtual agent by persons with mild cognitive impairment.

The diagnosis of Mild Cognitive Impairment (MCI) characterises patients at risk of dementia and may provide an opportunity for disease-modifying interventions. Identifying persons with MCI (PwMCI) from adults of a similar age without cognitive complaints is a significant challenge. The main aims of this study were to determine whether generic speech differences were evident between PwMCI and healthy controls (HC), whether such differences were identifiable in responses to recent or remote memory questions, and to determine which speech variables showed the clearest between-group differences. This study analysed recordings of 8 PwMCI (5 females, 3 males) and 14 HC of a similar age (8 females, 6 males). Participants were recorded interacting with an intelligent virtual agent: a computer-generated talking head on a computer screen which asks pre-recorded questions when prompted by the interviewee through pressing the next key on a computer keyboard. Responses to recent and remote memory questions were analysed. Mann-Whitney U tests were used to test for statistically significant differences between PwMCI and HC on each of 12 speech variables, relating to temporal characteristics, number of words produced and pitch. It was found that compared to HC, PwMCI produce speech for less time and in shorter chunks, they pause more often and for longer, take longer to begin speaking and produce fewer words in their answers. It was also found that the PwMCI and HC were more alike when responding to remote memory questions than when responding to recent memory questions. These findings show great promise and suggest that detailed speech analysis can make an important contribution to diagnostic and stratification systems in patients with memory complaints.

Fully automated cognitive screening tool based on assessment of speech and language.

INTRODUCTION: Recent years have seen an almost sevenfold rise in referrals to specialist memory clinics. This has been associated with an increased proportion of patients referred with functional cognitive disorder (FCD), that is, non-progressive cognitive complaints. These patients are likely to benefit from a range of interventions (eg, psychotherapy) distinct from the requirements of patients with neurodegenerative cognitive disorders. We have developed a fully automated system, 'CognoSpeak', which enables risk stratification at the primary-secondary care interface and ongoing monitoring of patients with memory concerns. METHODS: We recruited 15 participants to each of four groups: Alzheimer's disease (AD), mild cognitive impairment (MCI), FCD and healthy controls. Participants responded to 12 questions posed by a computer-presented talking head. Automatic analysis of the audio and speech data involved speaker segmentation, automatic speech recognition and machine learning classification. RESULTS: CognoSpeak could distinguish between participants in the AD or MCI groups and those in the FCD or healthy control groups with a sensitivity of 86.7%. Patients with MCI were identified with a sensitivity of 80%. DISCUSSION: Our fully automated system achieved levels of accuracy comparable to currently available, manually administered assessments. Greater accuracy should be achievable through further system training with a greater number of users, the inclusion of verbal fluency tasks and repeat assessments. The current data supports CognoSpeak's promise as a screening and monitoring tool for patients with MCI. Pending confirmation of these findings, it may allow clinicians to offer patients at low risk of dementia earlier reassurance and relieve pressures on specialist memory services.

A Virtual Agent to Support Individuals Living With Physical and Mental Comorbidities: Co-Design and Acceptability Testing.

BACKGROUND: Individuals living with long-term physical health conditions frequently experience co-occurring mental health problems. This comorbidity has a significant impact on an individual's levels of emotional distress, health outcomes, and associated health care utilization. As health care services struggle to meet demand and care increasingly moves to the community, digital tools are being promoted to support patients to self-manage their health. One such technology is the autonomous virtual agent (chatbot, conversational agent), which uses artificial intelligence (AI) to process the user's written or spoken natural language and then to select or construct the corresponding appropriate responses. OBJECTIVE: This study aimed to co-design the content, functionality, and interface modalities of an autonomous virtual agent to support self-management for patients with an exemplar long-term condition (LTC; chronic pulmonary obstructive disease [COPD]) and then to assess the acceptability and system content. METHODS: We conducted 2 co-design workshops and a proof-of-concept implementation of an autonomous virtual agent with natural language processing capabilities. This implementation formed the basis for video-based scenario testing of acceptability with adults with a diagnosis of COPD and health professionals involved in their care. RESULTS: Adults (n=6) with a diagnosis of COPD and health professionals (n=5) specified 4 priority self-management scenarios for which they would like to receive support: at the time of diagnosis (information provision), during acute exacerbations (crisis support), during periods of low mood (emotional support), and for general self-management (motivation). From the scenario testing, 12 additional adults with COPD felt the system to be both acceptable and engaging, particularly with regard to internet-of-things capabilities. They felt the system would be particularly useful for individuals living alone. CONCLUSIONS: Patients did not explicitly separate mental and physical health needs, although the content they developed for the virtual agent had a clear psychological approach. Supported self-management delivered via an autonomous virtual agent was acceptable to the participants. A co-design process has allowed the research team to identify key design principles, content, and functionality to underpin an autonomous agent for delivering self-management support to older adults living with COPD and potentially other LTCs.

A new diagnostic approach for the identification of patients with neurodegenerative cognitive complaints.

Neurodegenerative diseases causing dementia are known to affect a person's speech and language. Part of the expert assessment in memory clinics therefore routinely focuses on detecting such features. The current outpatient procedures examining patients' verbal and interactional abilities mainly focus on verbal recall, word fluency, and comprehension. By capturing neurodegeneration-associated characteristics in a person's voice, the incorporation of novel methods based on the automatic analysis of speech signals may give us more information about a person's ability to interact which could contribute to the diagnostic process. In this proof-of-principle study, we demonstrate that purely acoustic features, extracted from recordings of patients' answers to a neurologist's questions in a specialist memory clinic can support the initial distinction between patients presenting with cognitive concerns attributable to progressive neurodegenerative disorders (ND) or Functional Memory Disorder (FMD, i.e., subjective memory concerns unassociated with objective cognitive deficits or a risk of progression). The study involved 15 FMD and 15 ND patients where a total of 51 acoustic features were extracted from the recordings. Feature selection was used to identify the most discriminating features which were then used to train five different machine learning classifiers to differentiate between the FMD/ND classes, achieving a mean classification accuracy of 96.2%. The discriminative power of purely acoustic approaches could be integrated into diagnostic pathways for patients presenting with memory concerns and are computationally less demanding than methods focusing on linguistic elements of speech and language that require automatic speech recognition and understanding.

Toward the Automation of Diagnostic Conversation Analysis in Patients with Memory Complaints.

BACKGROUND: The early diagnosis of dementia is of great clinical and social importance. A recent study using the qualitative methodology of conversation analysis (CA) demonstrated that language and communication problems are evident during interactions between patients and neurologists, and that interactional observations can be used to differentiate between cognitive difficulties due to neurodegenerative disorders (ND) or functional memory disorders (FMD). OBJECTIVE: This study explores whether the differential diagnostic analysis of doctor-patient interactions in a memory clinic can be automated. METHODS: Verbatim transcripts of conversations between neurologists and patients initially presenting with memory problems to a specialist clinic were produced manually (15 with FMD, and 15 with ND). A range of automatically detectable features focusing on acoustic, lexical, semantic, and visual information contained in the transcripts were defined aiming to replicate the diagnostic qualitative observations. The features were used to train a set of five machine learning classifiers to distinguish between ND and FMD. RESULTS: The mean rate of correct classification between ND and FMD was 93% ranging from 97% by the Perceptron classifier to 90% by the Random Forest classifier.Using only the ten best features, the mean correct classification score increased to 95%. CONCLUSION: This pilot study provides proof-of-principle that a machine learning approach to analyzing transcripts of interactions between neurologists and patients describing memory problems can distinguish people with neurodegenerative dementia from people with FMD.