Multimodal job interview simulator for training of autistic individuals.

Autistic individuals face difficulties in finding and maintaining employment, and studies have shown that the job interview is often a significant barrier to obtaining employment. Prior computer-based job interview training interventions for autistic individuals have been associated with better interview outcomes. These previous interventions, however, do not leverage the use of multimodal data that could give insight into the emotional underpinnings of autistic individuals' challenges in job interviews. In this article, the authors present the design of a novel multimodal job interview training platform called CIRVR that simulates job interviews through spoken interaction and collects eye gaze, facial expressions, and physiological responses of the participants to understand their stress response and their affective state. Results from a feasibility study with 23 autistic participants who interacted with CIRVR are presented. In addition, qualitative feedback was gathered from stakeholders on visualizations of data on CIRVR's visualization tool called the Dashboard. The data gathered indicate the potential of CIRVR along with the Dashboard to be used in the creation of individualized job interview training of autistic individuals.

Semi-Supervised Behavior Labeling Using Multimodal Data during Virtual Teamwork-Based Collaborative Activities.

Adaptive human-computer systems require the recognition of human behavior states to provide real-time feedback to scaffold skill learning. These systems are being researched extensively for intervention and training in individuals with autism spectrum disorder (ASD). Autistic individuals are prone to social communication and behavioral differences that contribute to their high rate of unemployment. Teamwork training, which is beneficial for all people, can be a pivotal step in securing employment for these individuals. To broaden the reach of the training, virtual reality is a good option. However, adaptive virtual reality systems require real-time detection of behavior. Manual labeling of data is time-consuming and resource-intensive, making automated data annotation essential. In this paper, we propose a semi-supervised machine learning method to supplement manual data labeling of multimodal data in a collaborative virtual environment (CVE) used to train teamwork skills. With as little as 2.5% of the data manually labeled, the proposed semi-supervised learning model predicted labels for the remaining unlabeled data with an average accuracy of 81.3%, validating the use of semi-supervised learning to predict human behavior.

Design of a Desktop Virtual Reality-Based Collaborative Activities Simulator (ViRCAS) to Support Teamwork in Workplace Settings for Autistic Adults.

Autistic adults possess many skills sought by employers, but may be at a disadvantage in the workplace if social-communication differences negatively impact teamwork. We present a novel collaborative virtual reality (VR)-based activities simulator, called ViRCAS, that allows autistic and neurotypical adults to work together in a shared virtual space, offering the chance to practice teamwork and assess progress. ViRCAS has three main contributions: 1) a new collaborative teamwork skills practice platform; 2) a stakeholder-driven collaborative task set with embedded collaboration strategies; and 3) a framework for multimodal data analysis to assess skills. Our feasibility study with 12 participant pairs showed preliminary acceptance of ViRCAS, a positive impact of the collaborative tasks on supported teamwork skills practice for autistic and neurotypical individuals, and promising potential to quantitatively assess collaboration through multimodal data analysis. The current work paves the way for longitudinal studies that will assess whether the collaborative teamwork skill practice that ViRCAS provides also contributes towards improved task performance.