Active Inference Through Energy Minimization in Multimodal Affective Human-Robot Interaction.

During communication, humans express their emotional states using various modalities (e.g., facial expressions and gestures), and they estimate the emotional states of others by paying attention to multimodal signals. To ensure that a communication robot with limited resources can pay attention to such multimodal signals, the main challenge involves selecting the most effective modalities among those expressed. In this study, we propose an active perception method that involves selecting the most informative modalities using a criterion based on energy minimization. This energy-based model can learn the probability of the network state using energy values, whereby a lower energy value represents a higher probability of the state. A multimodal deep belief network, which is an energy-based model, was employed to represent the relationships between the emotional states and multimodal sensory signals. Compared to other active perception methods, the proposed approach demonstrated improved accuracy using limited information in several contexts associated with affective human-robot interaction. We present the differences and advantages of our method compared to other methods through mathematical formulations using, for example, information gain as a criterion. Further, we evaluate performance of our method, as pertains to active inference, which is based on the free energy principle. Consequently, we establish that our method demonstrated superior performance in tasks associated with mutually correlated multimodal information.

An Information Theoretic Approach to Reveal the Formation of Shared Representations.

Modality-invariant categorical representations, i.e., shared representation, is thought to play a key role in learning to categorize multi-modal information. We have investigated how a bimodal autoencoder can form a shared representation in an unsupervised manner with multi-modal data. We explored whether altering the depth of the network and mixing the multi-modal inputs at the input layer affect the development of the shared representations. Based on the activation of units in the hidden layers, we classified them into four different types: visual cells, auditory cells, inconsistent visual and auditory cells, and consistent visual and auditory cells. Our results show that the number and quality of the last type (i.e., shared representation) significantly differ depending on the depth of the network and are enhanced when the network receives mixed inputs as opposed to separate inputs for each modality, as occurs in typical two-stage frameworks. In the present work, we present a way to utilize information theory to understand the abstract representations formed in the hidden layers of the network. We believe that such an information theoretic approach could potentially provide insights into the development of more efficient and cost-effective ways to train neural networks using qualitative measures of the representations that cannot be captured by analyzing only the final outputs of the networks.

Measurement of Three-Dimensional Force Applied to Elastic Suture Training Pads for Laparoscopic Suturing.

Laparoscopic surgery requires highly trained skills to manipulate the laparoscopic instruments. The effectiveness and efficiency of manipulation training are expected to increase by quantitatively evaluating how a series of trainee's manipulations are different from those conducted by skilled surgeons and providing feedback. However, such detailed feed- back is not available because adequate measurement systems have not been proposed to measure the precise forces applied to suture training pads without disturbing the delicate manipulations. Therefore, we proposed a sensorized suture training pad satisfying the above requirements. Three surgical residents participated in an experiment to measure time sequences of three-dimensional forces applied to the pad when executing a transfixion suture. As a result, the differences in manipulation patterns among the residents were found by dividing the transfixion operation based on the sequential force data and recorded videos.

Integrated Cognitive Architecture for Robot Learning of Action and Language.

The manner in which humans learn, plan, and decide actions is a very compelling subject. Moreover, the mechanism behind high-level cognitive functions, such as action planning, language understanding, and logical thinking, has not yet been fully implemented in robotics. In this paper, we propose a framework for the simultaneously comprehension of concepts, actions, and language as a first step toward this goal. This can be achieved by integrating various cognitive modules and leveraging mainly multimodal categorization by using multilayered multimodal latent Dirichlet allocation (mMLDA). The integration of reinforcement learning and mMLDA enables actions based on understanding. Furthermore, the mMLDA, in conjunction with grammar learning and based on the Bayesian hidden Markov model (BHMM), allows the robot to verbalize its own actions and understand user utterances. We verify the potential of the proposed architecture through experiments using a real robot.

Mexican-Hat-Like Response in a Flexible Tactile Sensor Using a Magnetorheological Elastomer.

A significant challenge in robotics is providing a sense of touch to robots. Even though several types of flexible tactile sensors have been proposed, they still have various technical issues such as a large amount of deformation that fractures the sensing elements, a poor maintainability and a deterioration in the sensitivity caused by the presence of a thick and soft covering. As one solution for these issues, we proposed a flexible tactile sensor composed of a magnet, magnetic transducer and dual-layer elastomer, which consists of a magnetorheological and nonmagnetic elastomer sheet. In this study, we first investigated the sensitivity of the sensor, which was found to be high (approximately 161 mV/N with a signal-to-noise ratio of 42.2 dB); however, the sensor has a speed-dependent hysteresis in its sensor response curve. Then, we investigated the spatial response and observed the following results: (1) the sensor response was a distorted Mexican-hat-like bipolar shape, namely a negative response area was observed around the positive response area; (2) the negative response area disappeared when we used a compressible sponge sheet instead of the incompressible nonmagnetic elastomer. We concluded that the characteristic negative response in the Mexican-hat-like response is derived from the incompressibility of the nonmagnetic elastomer.