Promoting emotional and behavioral interventions in ASD treatment: Evidence from EPIGRAM, A naturalistic, prospective and longitudinal study.

BACKGROUND: Prognostic factors from naturalistic treatment studies of children with Autism Spectrum Disorder (ASD) remain largely unknown. We aimed to identify baseline and treatment-related prognostic predictors at 1-year follow-up after Integrative Care Practices (ICPs). METHODS: Eighty-nine preschool children with severe ASD were given ICP combining nine therapeutic workshops based on children's needs. Participants were assessed at baseline and during 12 months follow-up with the Psycho-educational Profile-3-R, Children Autism Rating Scale, Parental Global Impression, and the Autistic Behaviors Scale. We assessed prognostic predictors using multivariable regression models and explored treatment ingredients influencing outcome using Classification and Regression Trees (CART). RESULTS: Multivariable models showed that being a child from first generation immigrant parents predicted increased maladaptive behaviors, whereas play activities had an opposite effect; severity of ASD symptoms and impaired cognitive functions predicted worse autism severity at follow-up; and lower play activities predicted worse parent impression. Regarding treatment effects, more emotion/behavioral interventions predicted better outcomes, and more communication interventions predicted lower autism severity, whereas more education and cognitive interventions had an opposite effect. CART confirmed that more hours of intervention in the emotion/behavioral domain helped classifying cases with better outcomes. More parental support was associated with decreased maladaptive behaviors. Sensorimotor and education interventions also significantly contributed to classifying cases according to outcomes but defined subgroups with opposite prognosis. CONCLUSION: Children who exhibited the best prognosis following ICPs had less autism severity, better cognition, and non-immigrant parents at baseline. Emotion/behavior interventions appeared key across all outcomes and should be promoted.

Two ways to make your robot proactive: Reasoning about human intentions or reasoning about possible futures.

Robots sharing their space with humans need to be proactive to be helpful. Proactive robots can act on their own initiatives in an anticipatory way to benefit humans. In this work, we investigate two ways to make robots proactive. One way is to recognize human intentions and to act to fulfill them, like opening the door that you are about to cross. The other way is to reason about possible future threats or opportunities and to act to prevent or to foster them, like recommending you to take an umbrella since rain has been forecast. In this article, we present approaches to realize these two types of proactive behavior. We then present an integrated system that can generate proactive robot behavior by reasoning on both factors: intentions and predictions. We illustrate our system on a sample use case including a domestic robot and a human. We first run this use case with the two separate proactive systems, intention-based and prediction-based, and then run it with our integrated system. The results show that the integrated system is able to consider a broader variety of aspects that are required for proactivity.

Does the Goal Matter? Emotion Recognition Tasks Can Change the Social Value of Facial Mimicry Towards Artificial Agents.

In this paper, we present a study aimed at understanding whether the embodiment and humanlikeness of an artificial agent can affect people's spontaneous and instructed mimicry of its facial expressions. The study followed a mixed experimental design and revolved around an emotion recognition task. Participants were randomly assigned to one level of humanlikeness (between-subject variable: humanlike, characterlike, or morph facial texture of the artificial agents) and observed the facial expressions displayed by three artificial agents differing in embodiment (within-subject variable: video-recorded robot, physical robot, and virtual agent) and a human (control). To study both spontaneous and instructed facial mimicry, we divided the experimental sessions into two phases. In the first phase, we asked participants to observe and recognize the emotions displayed by the agents. In the second phase, we asked them to look at the agents' facial expressions, replicate their dynamics as closely as possible, and then identify the observed emotions. In both cases, we assessed participants' facial expressions with an automated Action Unit (AU) intensity detector. Contrary to our hypotheses, our results disclose that the agent that was perceived as the least uncanny, and most anthropomorphic, likable, and co-present, was the one spontaneously mimicked the least. Moreover, they show that instructed facial mimicry negatively predicts spontaneous facial mimicry. Further exploratory analyses revealed that spontaneous facial mimicry appeared when participants were less certain of the emotion they recognized. Hence, we postulate that an emotion recognition goal can flip the social value of facial mimicry as it transforms a likable artificial agent into a distractor. Further work is needed to corroborate this hypothesis. Nevertheless, our findings shed light on the functioning of human-agent and human-robot mimicry in emotion recognition tasks and help us to unravel the relationship between facial mimicry, liking, and rapport.

Exploring Behavioral Creativity of a Proactive Robot.

Creativity, in one sense, can be seen as an effort or action to bring novelty. Following this, we explore how a robot can be creative by bringing novelty in a human-robot interaction (HRI) scenario. Studies suggest that proactivity is closely linked with creativity. Proactivity can be defined as acting or interacting by anticipating future needs or actions. This study aims to explore the effect of proactive behavior and the relation of such behaviors to the two aspects of creativity: 1) the perceived creativity observed by the user in the robot's proactive behavior and 2) creativity of the user by assessing how creativity in HRI can be shaped or influenced by proactivity. We do so by conducting an experimental study, where the robot tries to support the user on the completion of the task regardless of the end result being novel or not and does so by exhibiting anticipatory proactive behaviors. In our study, the robot instantiates a set of verbal communications as proactive robot behavior. To our knowledge, the study is among the first to establish and investigate the relationship between creativity and proactivity in the HRI context, based on user studies. The initial results have indicated a relationship between observed proactivity, creativity, and task achievement. It also provides valuable pointers for further investigation in this domain.

Reinforcement Learning With Human Advice: A Survey.

In this paper, we provide an overview of the existing methods for integrating human advice into a reinforcement learning process. We first propose a taxonomy of the different forms of advice that can be provided to a learning agent. We then describe the methods that can be used for interpreting advice when its meaning is not determined beforehand. Finally, we review different approaches for integrating advice into the learning process.

"Motherese" Prosody in Fetal-Directed Speech: An Exploratory Study Using Automatic Social Signal Processing.

Introduction: Motherese, or emotional infant directed speech (IDS), is the specific form of speech used by parents to address their infants. The prosody of IDS has affective properties, expresses caregiver involvement, is a marker of caregiver-infant interaction quality. IDS prosodic characteristics can be detected with automatic analysis. We aimed to explore whether pregnant women "speak" to their unborn baby, whether they use motherese while speaking and whether anxio-depressive or obstetrical status impacts speaking to the fetus. Participants and Methods: We conducted an observational study of pregnant women with gestational ages from 26 to 38 weeks. Women were recruited in a university hospital department of obstetrics. Thirty-five women agreed to participate in the study, and 26 audio records were exploitable. We collected obstetrical and sociodemographic data, pregnancy outcomes, anxiety and depressive status using the Covy and Raskin Scales, and life events using the Sensations During Pregnancy and Life Event Questionnaire. Each participant was left alone with an audio recorder with a recommendation to feel free to speak to her fetus as she would have done at home. The recording was stopped after 3 min. Audio recordings were analyzed by two methods: psycholinguist experts' annotation and computational objective automatic analyses. Results: Most mothers (89%) reported speaking to their fetuses. We found a correlation between maternal first perceptions of fetal movements and the start of mother's speaking to fetus. Motherese prosody was detected with both annotation and automatic analysis with a significant correlation between the two methods. In this exploratory study, motherese use was not associated with maternal anxiodepressive or obstetrical status. However, the more future mothers were depressed, the less they spoke with their fetuses during the recording. Conclusion: Fetal directed speech (FDS) can be detected during pregnancy, and it contains a period of prosody that shares the same characteristics of motherese that can be described as prenatal motherese or emotional fetal-directed speech (e-FDS). This means that pregnant women start using motherese much earlier than expected. FDS seems to be correlated with maternal first perceptions of fetal movements and depression scores. However, more research is needed to confirm these exploratory results.

"It Is Not the Robot Who Learns, It Is Me." Treating Severe Dysgraphia Using Child-Robot Interaction.

Writing disorders are frequent and impairing. However, social robots may help to improve children's motivation and to propose enjoyable and tailored activities. Here, we have used the Co-writer scenario in which a child is asked to teach a robot how to write via demonstration on a tablet, combined with a series of games we developed to train specifically pressure, tilt, speed, and letter liaison controls. This setup was proposed to a 10-year-old boy with a complex neurodevelopmental disorder combining phonological disorder, attention deficit/hyperactivity disorder, dyslexia, and developmental coordination disorder with severe dysgraphia. Writing impairments were severe and limited his participation in classroom activities despite 2 years of specific support in school and professional speech and motor remediation. We implemented the setup during his occupational therapy for 20 consecutive weekly sessions. We found that his motivation was restored; avoidance behaviors disappeared both during sessions and at school; handwriting quality and posture improved dramatically. In conclusion, treating dysgraphia using child-robot interaction is feasible and improves writing. Larger clinical studies are required to confirm that children with dysgraphia could benefit from this setup.

Adolescents with borderline personality disorder show a higher response to stress but a lack of self-perception: Evidence through affective computing.

Stress reactivity is a complex phenomenon associated with multiple and multimodal expressions and functions. Herein, we hypothesized that compared with healthy controls (HCs), adolescents with borderline personality disorder (BPD) would exhibit a stronger response to stressors and a deficit in self-perception of stress due to their lack of insight. Twenty adolescents with BPD and 20 matched HCs performed a socially evaluated mental arithmetic test to induce stress. We assessed self- and heteroperception using both human ratings and affective computing-based methods for the automatic extraction of 39 behavioral features (2D + 3D video recording) and 62 physiological features (Nexus-10 recording). Predictions were made using machine learning. In addition, salivary cortisol was measured. Human ratings showed that adolescents with BPD experienced more stress than HCs. Human ratings and automated machine learning indicated opposite results regarding self- and heteroperceived stress in adolescents with BPD compared to HCs. Adolescents with BPD had higher levels of heteroperceived stress than self-perceived stress. Similarly, affective computing achieved better classification for heteroperceived stress. HCs had an opposite profile; they had higher levels of self-perceived stress, and affective computing reached a better classification for self-perceived stress. We conclude that adolescents with BPD are more sensitive to stress and show a lack of self-perception (or insight). In terms of clinical implications, our affective computing measures may help distinguish hetero- vs. self-perceptions of stress in natural settings and may offer external feedback during therapeutic interaction.

Age-related changes in gaze behaviour during social interaction: An eye-tracking study with an embodied conversational agent.

Previous research has highlighted age-related differences in social perception, in particular emotional expression processing. To date, such studies have largely focused on approaches that use static emotional stimuli that the participant has to identify passively without the possibility of any interaction. In this study, we propose an interactive virtual environment to better address age-related variations in social and emotional perception. A group of 22 young (18-30 years) and 20 older (60-80 years) adults were engaged in a face-to-face conversation with an embodied conversational agent. Participants were invited to interact naturally with the agent and to identify his facial expression. Their gaze behaviour was captured by an eye-tracking device throughout the interaction. We also explored whether the Big Five personality traits (particularly extraversion) and anxiety modulated gaze during the social interaction. Findings suggested that age-related differences in gaze behaviour were only apparent when decoding social signals (i.e., listening to a partner's question, identifying facial expressions) and not when communicating social information (i.e., when speaking). Furthermore, higher extraversion levels consistently led to a shorter amount of time gazing towards the eyes, whereas higher anxiety levels led to slight modulations of gaze only when participants were listening to questions. Face-to-face conversation with virtual agents can provide a more naturalistic framework for the assessment of online socio-emotional interaction in older adults, which is not easily observable in classical offline paradigms. This study provides novel and important insights into the specific circumstances in which older adults may experience difficulties in social interactions.

Behavior and interaction imaging at 9 months of age predict autism/intellectual disability in high-risk infants with West syndrome.

Automated behavior analysis are promising tools to overcome current assessment limitations in psychiatry. At 9 months of age, we recorded 32 infants with West syndrome (WS) and 19 typically developing (TD) controls during a standardized mother-infant interaction. We computed infant hand movements (HM), speech turn taking of both partners (vocalization, pause, silences, overlap) and motherese. Then, we assessed whether multimodal social signals and interactional synchrony at 9 months could predict outcomes (autism spectrum disorder (ASD) and intellectual disability (ID)) of infants with WS at 4 years. At follow-up, 10 infants developed ASD/ID (WS+). The best machine learning reached 76.47% accuracy classifying WS vs. TD and 81.25% accuracy classifying WS+ vs. WS-. The 10 best features to distinguish WS+ and WS- included a combination of infant vocalizations and HM features combined with synchrony vocalization features. These data indicate that behavioral and interaction imaging was able to predict ASD/ID in high-risk children with WS.

Engagement in Human-Agent Interaction: An Overview.

Engagement is a concept of the utmost importance in human-computer interaction, not only for informing the design and implementation of interfaces, but also for enabling more sophisticated interfaces capable of adapting to users. While the notion of engagement is actively being studied in a diverse set of domains, the term has been used to refer to a number of related, but different concepts. In fact it has been referred to across different disciplines under different names and with different connotations in mind. Therefore, it can be quite difficult to understand what the meaning of engagement is and how one study relates to another one accordingly. Engagement has been studied not only in human-human, but also in human-agent interactions i.e., interactions with physical robots and embodied virtual agents. In this overview article we focus on different factors involved in engagement studies, distinguishing especially between those studies that address task and social engagement, involve children and adults, are conducted in a lab or aimed for long term interaction. We also present models for detecting engagement and for generating multimodal behaviors to show engagement.

The social Simon effect in the tactile sensory modality: a negative finding.

This study seeks to investigate whether users activate cognitive representations of their partner's action when they are involved in tactile collaborative tasks. The social Simon effect is a spatial stimulus-response interference induced by the mere presence of a partner in a go/nogo task. It has been extensively studied in the visual and auditory sensory modalities, but never before in the tactile modality. We compared the performances of 28 participants in three tasks: (1) a standard Simon task where participants responded to two different tactile stimuli applied to their fingertips with either their left or right foot, (2) an individual go/nogo task where participants responded to only one stimulus and (3) a social go/nogo task where they again responded to only one stimulus, but were partnered with another person who responded to the complementary stimulus. The interference effect due to spatial incongruence between the side where participants received the stimulus and the foot used to answer increased significantly in the standard Simon task compared to the social go/nogo task. Such a difference was not observed between the social and individual go/nogo tasks. Performances were nevertheless enhanced in the social go/nogo task, but irrespectively of the stimulus-response congruency. This study is the first to report a negative result for the social Simon effect in the tactile modality. Results suggest that cognitive representation of the co-actor is weaker in this modality.

Interpersonal Synchronization, Motor Coordination, and Control Are Impaired During a Dynamic Imitation Task in Children With Autism Spectrum Disorder.

Background: Impairments in imitation abilities have been commonly described in children with autism spectrum disorder (ASD). How motricity in interpersonal coordination impacts imitation, during long lasting semi-ecological conditions, has not been carefully investigated. Methods: Eighty-five children and adolescents (39 controls with typical development, TD; 29 patients with ASD; 17 patients with developmental coordination disorder, DCD), aged 6 to 20 years, participated to a behavioral paradigm in which participants, standing and moving, interacted with a virtual tightrope walker standing and moving as well. During the protocol, we measured automatically and continuously bodily postures and movements from RGB sensor recording to assess participants' behavioral imitation. Results: We show that (1) interpersonal synchronization (as evidenced by the synchrony between the participant's and the tightrope walker's bars) and (2) motor coordination (as evidenced by the synchrony between the participant's bar and its own head axis) increased with age and were more impaired in patients with ASD. Also, motor control as evidenced by the movement angle standard deviations of participants' bar and head were significantly impaired in ASD compared to TD or DCD. Conclusion: Interpersonal synchronization and motor coordination during ecological interaction show both subtle impairment in children with ASD as compared to children with TD or DCD. These results questioned how motricity mature in terms of motor control and proprioception in children with ASD.

ICT and autism care: state of the art.

PURPOSE OF REVIEW: Over the past 10 years, the use of information and communication technologies (ICTs) has increased in regard to the treatment of individuals with autism spectrum disorders (ASDs). ICT support mechanisms (e.g. computers, laptops, robots) are particularly attractive and are adapted to children with ASD. In addition, ICT algorithms can offer new perspectives for clinicians, outside direct apps or gaming proposals. Here, we will focus on the use of serious games and robots because of their attractiveness and their value in working on social skills. RECENT FINDINGS: The latest knowledge regarding the use of ICT in the forms of serious games and robotics applied to individuals with ASD shows that the field of serious games has already achieved interesting and promising results, although the clinical validations are not always complete. In the field of robotics, there are still many limitations on the use of ICT (e.g. most interaction are similar to the wizard of Oz), and questions remain concerning their eventual effectiveness. SUMMARY: To describe the implications of the findings for clinical practice or research, we describe two large projects, namely, JEMImE and Michelangelo, as examples of current studies that are aimed at enhancing social skills in children with ASD by including novel algorithms with clinical insights in robots or serious games.

Behavioral Own-Body-Transformations in Children and Adolescents With Typical Development, Autism Spectrum Disorder, and Developmental Coordination Disorder.

Background: In motor imitation, taking a partner's perspective often involves a mental body transformation from an embodied, ego-centered viewpoint to a disembodied, hetero-centered viewpoint. Impairments of both own-body-transformation (OBT) and abnormalities in visual-spatial processing have been reported in patients with neurodevelopmental disorders including autism spectrum disorder (ASD). In the context of a visual-motor interactive task, studying OBT impairments while disentangling the contribution of visual-spatial impairments associated with motor coordination problems has not been investigated. Methods: 85 children and adolescents (39 controls with typical development, TD; 29 patients with ASD; 17 patients with developmental coordination disorder, DCD), aged 6-19 years, participated in a behavioral paradigm in which participants interacted with a virtual tightrope walker (TW) standing and moving with him. The protocol enables to distinguish ego-centered and hetero-centered perspectives. Results: We show that (1) OBT was possible but difficult for children with neurodevelopmental disorders, as well as for TD children, when the task required the participant to perform a mental rotation in order to adopt a hetero-centered perspective. (2) Using multivariate models, hetero-centered perspective score was significantly associated with age, TW orientation, latency, and diagnosis. ASD and TD groups' performances were close and significantly correlated with age. However, it was not the case for DCD, since this group was specifically handicapped by visual-spatial impairments. (3) ASD and DCD did not perform similarly: motor performance as shown by movement amplitude was better in DCD than ASD. ASD motor response was more ambiguous and hardly readable. Conclusion: Changing perspective in a spatial environment is possible for patients with ASD although delayed compared with TD children. In patients with DCD, their visual-spatial impairments negatively modulated their performances in the experiment.

Children Facial Expression Production: Influence of Age, Gender, Emotion Subtype, Elicitation Condition and Culture.

The production of facial expressions (FEs) is an important skill that allows children to share and adapt emotions with their relatives and peers during social interactions. These skills are impaired in children with Autism Spectrum Disorder. However, the way in which typical children develop and master their production of FEs has still not been clearly assessed. This study aimed to explore factors that could influence the production of FEs in childhood such as age, gender, emotion subtype (sadness, anger, joy, and neutral), elicitation task (on request, imitation), area of recruitment (French Riviera and Parisian) and emotion multimodality. A total of one hundred fifty-seven children aged 6-11 years were enrolled in Nice and Paris, France. We asked them to produce FEs in two different tasks: imitation with an avatar model and production on request without a model. Results from a multivariate analysis revealed that: (1) children performed better with age. (2) Positive emotions were easier to produce than negative emotions. (3) Children produced better FE on request (as opposed to imitation); and (4) Riviera children performed better than Parisian children suggesting regional influences on emotion production. We conclude that facial emotion production is a complex developmental process influenced by several factors that needs to be acknowledged in future research.

Pre-linguistic infants employ complex communicative loops to engage mothers in social exchanges and repair interaction ruptures.

Language has long been identified as a powerful communicative tool among humans. Yet, pre-linguistic communication, which is common in many species, is also used by human infants prior to the acquisition of language. The potential communicational value of pre-linguistic vocal interactions between human infants and mothers has been studied in the past decades. With 120 dyads (mothers and three- or six-month-old infants), we used the classical Still Face Paradigm (SFP) in which mothers interact freely with their infants, then refrain from communication (Still Face, SF), and finally resume play. We employed innovative automated techniques to measure infant and maternal vocalization and pause, and dyadic parameters (infant response to mother, joint silence and overlap) and the emotional component of Infant Directed Speech (e-IDS) throughout the interaction. We showed that: (i) during the initial free play mothers use longer vocalizations and more e-IDS when they interact with older infants and (ii) infant boys exhibit longer vocalizations and shorter pauses than girls. (iii) During the SF and reunion phases, infants show marked and sustained changes in vocalizations but their mothers do not and (iv) mother-infant dyadic parameters increase in the reunion phase. Our quantitative results show that infants, from the age of three months, actively participate to restore the interactive loop after communicative ruptures long before vocalizations show clear linguistic meaning. Thus, auditory signals provide from early in life a channel by which infants co-create interactions, enhancing the mother-infant bond.

Distinguish self- and hetero-perceived stress through behavioral imaging and physiological features.

Stress reactivity is a complex phenomenon associated to multiple and multimodal expressions. Response to stressors has an obvious survival function and may be seen as an internal regulation to adapt to threat or danger. The intensity of this internal response can be assessed as the self-perception of the stress response. In species with social organization, this response also serves a communicative function, so-called hetero-perception. Our study presents multimodal stress detection assessment - a new methodology combining behavioral imaging and physiological monitoring for analyzing stress from these two perspectives. The system is based on automatic extraction of 39 behavioral (2D+3D video recording) and 62 physiological (Nexus-10 recording) features during a socially evaluated mental arithmetic test. The analysis with machine learning techniques for automatic classification using Support Vector Machine (SVM) show that self-perception and hetero-perception of social stress are both close but different phenomena: self-perception was significantly correlated with hetero-perception but significantly differed from it. Also, assessing stress with SVM through multimodality gave excellent classification results (F1 score values: 0.9±0.012 for hetero-perception and 0.87±0.021 for self-perception). In the best selected feature subsets, we found some common behavioral and physiological features that allow classification of both self- and hetero-perceived stress. However, we also found the contributing features for automatic classifications had opposite distributions: self-perception classification was mainly based on physiological features and hetero-perception was mainly based on behavioral features.

Gaze Behavior Consistency among Older and Younger Adults When Looking at Emotional Faces.

The identification of non-verbal emotional signals, and especially of facial expressions, is essential for successful social communication among humans. Previous research has reported an age-related decline in facial emotion identification, and argued for socio-emotional or aging-brain model explanations. However, more perceptual differences in the gaze strategies that accompany facial emotional processing with advancing age have been under-explored yet. In this study, 22 young (22.2 years) and 22 older (70.4 years) adults were instructed to look at basic facial expressions while their gaze movements were recorded by an eye-tracker. Participants were then asked to identify each emotion, and the unbiased hit rate was applied as performance measure. Gaze data were first analyzed using traditional measures of fixations over two preferential regions of the face (upper and lower areas) for each emotion. Then, to better capture core gaze changes with advancing age, spatio-temporal gaze behaviors were deeper examined using data-driven analysis (dimension reduction, clustering). Results first confirmed that older adults performed worse than younger adults at identifying facial expressions, except for "joy" and "disgust," and this was accompanied by a gaze preference toward the lower-face. Interestingly, this phenomenon was maintained during the whole time course of stimulus presentation. More importantly, trials corresponding to older adults were more tightly clustered, suggesting that the gaze behavior patterns of older adults are more consistent than those of younger adults. This study demonstrates that, confronted to emotional faces, younger and older adults do not prioritize or ignore the same facial areas. Older adults mainly adopted a focused-gaze strategy, consisting in focusing only on the lower part of the face throughout the whole stimuli display time. This consistency may constitute a robust and distinctive "social signature" of emotional identification in aging. Younger adults, however, were more dispersed in terms of gaze behavior and used a more exploratory-gaze strategy, consisting in repeatedly visiting both facial areas.