Motion-Binding Property Contributes to Accurate Temporal-Order Perception in Audiovisual Synchrony.

Temporal perception in multisensory processing is important for an accurate and efficient understanding of the physical world. In general, it is executed in a dynamic environment in our daily lives. In particular, the motion-binding property is important for correctly identifying moving objects in the external environment. However, how this property affects multisensory temporal perception remains unclear. We investigate whether the motion-binding property influences audiovisual temporal integration. The study subjects performed four types of temporal-order judgment (TOJ) task experiments using three types of perception. In Experiment 1, the subjects conducted audiovisual TOJ tasks in the motion-binding condition, between two flashes, and in the simultaneous condition, in which the two flashes are perceived as simultaneous stimuli without motion. In Experiment 2, subjects conducted audiovisual TOJ tasks in the motion-binding condition and the short and long successive interval condition, in which the two stimuli are perceived as successive with no motion. The results revealed that the point of subjective simultaneity (PSS) and the just-noticeable difference (JND) in the motion-binding condition differed significantly from those in the simultaneous and short and long successive interval conditions. Specifically, the PSS in the motion-binding condition was shifted toward a sound-lead stimulus in which the PSS became closer to zero (i.e., physical simultaneity) and the JND became narrower compared to other conditions. This suggests that the motion-binding property contributes to accurate temporal integration in multisensory processing by precisely encoding the temporal order of the physical stimuli.

Head motion synchrony in unidirectional and bidirectional verbal communication.

Interpersonal communication includes verbal and nonverbal communication. Verbal communication comprises one-way (e.g., a speech or lecture) and interactive verbal communication (e.g., daily conversations or meetings), which we frequently encounter. Nonverbal communication has considerable influence on interpersonal communication, and body motion synchrony is known to be an important factor for successful communication and social interaction. However, most research on body motion synchrony has been elucidated by either the setting of one-way verbal transmission or the verbal interaction setting, and it remains unclear whether verbal directionality and interactivity affect body motion synchrony. One-way and two-way (interactive) verbal communication is implicated in designed or undesigned leader-follower relationships, and also in the complexity and diversity of interpersonal interactions, where two-way verbal communication is more complex and diverse than in the one-way condition. In this study, we tested head motion synchrony between the one-way verbal communication condition (in which the roles of the speaker and listener are fixed) and the two-way verbal communication condition (where the speaker and listener can freely engage in a conversation). Therefore, although no statistically significant difference in synchrony activity (relative frequency) was found, a statistically significant difference was observed in synchrony direction (temporal lead-lag structure as mimicry) and intensity. Specifically, the synchrony direction in two-way verbal communication was close to zero, but this in one-way verbal communication was synchronized with the listener's movement predominantly delayed. Furthermore, synchrony intensity, in terms of the degree of variation in the phase difference distribution, was significantly higher in the one-way verbal communication than in the two-way condition, with bigger time-shifts being observed in the latter. This result suggests that verbal interaction does not affect the overall frequency of head motion synchrony but does affect the temporal lead-lag structure and coherence.

Japanese Sound-Symbolic Words for Representing the Hardness of an Object Are Judged Similarly by Japanese and English Speakers.

Contrary to the assumption of arbitrariness in modern linguistics, sound symbolism, which is the non-arbitrary relationship between sounds and meanings, exists. Sound symbolism, including the "Bouba-Kiki" effect, implies the universality of such relationships; individuals from different cultural and linguistic backgrounds can similarly relate sound-symbolic words to referents, although the extent of these similarities remains to be fully understood. Here, we examined if subjects from different countries could similarly infer the surface texture properties from words that sound-symbolically represent hardness in Japanese. We prepared Japanese sound-symbolic words of which novelty was manipulated by a genetic algorithm (GA). Japanese speakers in Japan and English speakers in both Singapore and the United States rated these words based on surface texture properties (hardness, warmness, and roughness), as well as familiarity. The results show that hardness-related words were rated as harder and rougher than softness-related words, regardless of novelty and countries. Multivariate analyses of the ratings classified the hardness-related words along the hardness-softness dimension at over 80% accuracy, regardless of country. Multiple regression analyses revealed that the number of speech sounds /g/ and /k/ predicted the ratings of the surface texture properties in non-Japanese countries, suggesting a systematic relationship between phonetic features of a word and perceptual quality represented by the word across culturally and linguistically diverse samples.

Computer Vision System for Expressing Texture Using Sound-Symbolic Words.

The major goals of texture research in computer vision are to understand, model, and process texture and ultimately simulate human visual information processing using computer technologies. The field of computer vision has witnessed remarkable advancements in material recognition using deep convolutional neural networks (DCNNs), which have enabled various computer vision applications, such as self-driving cars, facial and gesture recognition, and automatic number plate recognition. However, for computer vision to "express" texture like human beings is still difficult because texture description has no correct or incorrect answer and is ambiguous. In this paper, we develop a computer vision method using DCNN that expresses texture of materials. To achieve this goal, we focus on Japanese "sound-symbolic" words, which can describe differences in texture sensation at a fine resolution and are known to have strong and systematic sensory-sound associations. Because the phonemes of Japanese sound-symbolic words characterize categories of texture sensations, we develop a computer vision method to generate the phonemes and structure comprising sound-symbolic words that probabilistically correspond to the input images. It was confirmed that the sound-symbolic words output by our system had about 80% accuracy rate in our evaluation.

Brain networks underlying the processing of sound symbolism related to softness perception.

Unlike the assumption of modern linguistics, there is non-arbitrary association between sound and meaning in sound symbolic words. Neuroimaging studies have suggested the unique contribution of the superior temporal sulcus to the processing of sound symbolism. However, because these findings are limited to the mapping between sound symbolism and visually presented objects, the processing of sound symbolic information may also involve the sensory-modality dependent mechanisms. Here, we conducted a functional magnetic resonance imaging experiment to test whether the brain regions engaged in the tactile processing of object properties are also involved in mapping sound symbolic information with tactually perceived object properties. Thirty-two healthy subjects conducted a matching task in which they judged the congruency between softness perceived by touch and softness associated with sound symbolic words. Congruency effect was observed in the orbitofrontal cortex, inferior frontal gyrus, insula, medial superior frontal gyrus, cingulate gyrus, and cerebellum. This effect in the insula and medial superior frontal gyri was overlapped with softness-related activity that was separately measured in the same subjects in the tactile experiment. These results indicate that the insula and medial superior frontal gyrus play a role in processing sound symbolic information and relating it to the tactile softness information.

The Effects of Vibratory Frequency and Temporal Interval on Tactile Apparent Motion.

Vibrotactile stimuli can be used to generate the haptic sensation of a static object or the motion of a dynamic object. Here, in this article, we investigated the effects of vibratory frequency and temporal interval on tactile apparent motion. In the experiment, we examined the effect of vibratory frequency with different temporal intervals on tactile apparent motion that results from two successive tactile stimuli on the index fingerpad. Results indicated that tactile apparent motion was perceived not only when both stimuli were either "flutter" or "vibration" stimuli, but also when one of each type was used. Specifically, when the first stimulus was introduced at 40Hz, "continuous motion" was viewed at all combinations of stimulus frequency, and "continuous motion" was clearly noted at the high-frequency combination instead of the low-frequency combination. Also, tactile apparent motion was predominantly viewed in the SOA range of 105 ms to 125 ms. We anticipate that our findings and further research will be essential resources for the design of tactile devices to represent the motion of dynamic objects.

Physical origin of a complicated tactile sensation: 'shittori feel'.

Shittori feel is defined as a texture that is moderately moisturized; however, many people experience 'shittori feel' when they touch a dry solid material containing little liquid. Here, shittori feel was evaluated for 12 materials. We found that the highest score of shittori feel was achieved by powders. Multiple regression analysis showed that shittori feel is a complex sense of moist and smooth feels. We analysed the relationship between the physical properties and the moist/smooth feels to show how subjects felt certain feels simultaneously. The moist and smooth feels are related to the surface roughness and friction characteristics of the materials. The moist and smooth feels can be perceived when the finger starts to move on the material surface and when the finger moves and rubs the material surface, respectively.

Brain networks underlying tactile softness perception: A functional magnetic resonance imaging study.

Humans are adept at perceiving physical properties of an object through touch. Tangible object properties can be categorized into two types: macro-spatial properties, including shape and orientation; and material properties, such as roughness, softness, and temperature. Previous neuroimaging studies have shown that roughness and temperature are extracted at nodes of a network, such as that involving the parietal operculum and insula, which is different from the network engaged in processing macro-spatial properties. However, it is unclear whether other perceptual dimensions pertaining to material properties engage the same regions. Here, we conducted a functional magnetic resonance imaging study to test whether the parietal operculum and insula were involved in extracting tactually-perceived softness magnitude. Fifty-six healthy right-handed participants estimated perceived softness magnitude using their right middle finger. We presented three stimuli that had the same shape but different compliances. The force applied to the finger was manipulated at two levels. Classical mass-univariate analysis showed that activity in the parietal operculum, insula, and medial prefrontal cortex was positively associated with perceived softness magnitude, regardless of the applied force. Softness-related activity was stronger in the ventral striatum in the high-force condition than in the low-force condition. The multivariate voxel pattern analysis showed higher accuracy than chance levels and control regions in the parietal operculum/insula, postcentral gyrus, posterior parietal lobule, and middle occipital gyrus. These results indicate that a distributed set of the brain regions, including the parietal operculum and insula, is involved in representing perceived softness.

Optimal linguistic expression in negotiations depends on visual appearance.

We investigate the influence of the visual appearance of a negotiator on persuasiveness within the context of negotiations. Psychological experiments were conducted to quantitatively analyze the relationship between visual appearance and the use of language. Male and female participants were shown three female and male photographs, respectively. They were asked to report how they felt about each photograph using a seven-point semantic differential (SD) scale for six affective factors (positive impression, extraversion, intelligence, conscientiousness, emotional stability, and agreeableness). Participants then answered how they felt about each negotiation scenario (they were presented with pictures and a situation combined with negotiation sentences) using a seven-point SD scale for seven affective factors (positive impression, extraversion, intelligence, conscientiousness, emotional stability, agreeableness, and degree of persuasion). Two experiments were conducted using different participant groups depending on the negotiation situations. Photographs with good or bad appearances were found to show high or low degrees of persuasion, respectively. A multiple regression equation was obtained, indicating the importance of the three language factors (euphemistic, honorific, and sympathy expressions) to impressions made during negotiation. The result shows that there are optimal negotiation sentences based on various negotiation factors, such as visual appearance and use of language. For example, persons with good appearance might worsen their impression during negotiations by using certain language, although their initial impression was positive, and persons with bad appearance could effectively improve their impressions in negotiations through their use of language, although the final impressions of their negotiation counterpart might still be more negative than those for persons with good appearance. In contrast, the impressions made by persons of normal appearance were not easily affected by their use of language. The results of the present study have significant implications for future studies of effective negotiation strategies considering visual appearance as well as gender.

The simultaneous perception of auditory-tactile stimuli in voluntary movement.

The simultaneous perception of multimodal information in the environment during voluntary movement is very important for effective reactions to the environment. Previous studies have found that voluntary movement affects the simultaneous perception of auditory and tactile stimuli. However, the results of these experiments are not completely consistent, and the differences may be attributable to methodological differences in the previous studies. In this study, we investigated the effect of voluntary movement on the simultaneous perception of auditory and tactile stimuli using a temporal order judgment task with voluntary movement, involuntary movement, and no movement. To eliminate the potential effect of stimulus predictability and the effect of spatial information associated with large-scale movement in the previous studies, we randomized the interval between the start of movement and the first stimulus, and used small-scale movement. As a result, the point of subjective simultaneity (PSS) during voluntary movement shifted from the tactile stimulus being first during involuntary movement or no movement to the auditory stimulus being first. The just noticeable difference (JND), an indicator of temporal resolution, did not differ across the three conditions. These results indicate that voluntary movement itself affects the PSS in auditory-tactile simultaneous perception, but it does not influence the JND. In the discussion of these results, we suggest that simultaneous perception may be affected by the efference copy.

Detection of Nonverbal Synchronization through Phase Difference in Human Communication.

Nonverbal communication is an important factor in human communication, and body movement synchronization in particular is an important part of nonverbal communication. Some researchers have analyzed body movement synchronization by focusing on changes in the amplitude of body movements. However, the definition of "body movement synchronization" is still unclear. From a theoretical viewpoint, phase difference is the most important factor in synchronization analysis. Therefore, there is a need to measure the synchronization of body movements using phase difference. The purpose of this study was to provide a quantitative definition of the phase difference distribution for detecting body movement synchronization in human communication. The phase difference distribution was characterized using four statistical measurements: density, mean phase difference, standard deviation (SD) and kurtosis. To confirm the effectiveness of our definition, we applied it to human communication in which the roles of speaker and listener were defined. Specifically, we examined the difference in the phase difference distribution between two different communication situations: face-to-face communication with visual interaction and remote communication with unidirectional visual perception. Participant pairs performed a task supposing lecture in the face-to-face communication condition and in the remote communication condition via television. Throughout the lecture task, we extracted a set of phase differences from the time-series data of the acceleration norm of head nodding motions between two participants. Statistical analyses of the phase difference distribution revealed the characteristics of head nodding synchronization. Although the mean phase differences in synchronized head nods did not differ significantly between the conditions, there were significant differences in the densities, the SDs and the kurtoses of the phase difference distributions of synchronized head nods. These results show the difference in nonverbal synchronization between different communication types. Our study indicates that the phase difference distribution is useful in detecting nonverbal synchronization in various human communication situations.

The effect of visual apparent motion on audiovisual simultaneity.

Visual motion information from dynamic environments is important in multisensory temporal perception. However, it is unclear how visual motion information influences the integration of multisensory temporal perceptions. We investigated whether visual apparent motion affects audiovisual temporal perception. Visual apparent motion is a phenomenon in which two flashes presented in sequence in different positions are perceived as continuous motion. Across three experiments, participants performed temporal order judgment (TOJ) tasks. Experiment 1 was a TOJ task conducted in order to assess audiovisual simultaneity during perception of apparent motion. The results showed that the point of subjective simultaneity (PSS) was shifted toward a sound-lead stimulus, and the just noticeable difference (JND) was reduced compared with a normal TOJ task with a single flash. This indicates that visual apparent motion affects audiovisual simultaneity and improves temporal discrimination in audiovisual processing. Experiment 2 was a TOJ task conducted in order to remove the influence of the amount of flash stimulation from Experiment 1. The PSS and JND during perception of apparent motion were almost identical to those in Experiment 1, but differed from those for successive perception when long temporal intervals were included between two flashes without motion. This showed that the result obtained under the apparent motion condition was unaffected by the amount of flash stimulation. Because apparent motion was produced by a constant interval between two flashes, the results may be accounted for by specific prediction. In Experiment 3, we eliminated the influence of prediction by randomizing the intervals between the two flashes. However, the PSS and JND did not differ from those in Experiment 1. It became clear that the results obtained for the perception of visual apparent motion were not attributable to prediction. Our findings suggest that visual apparent motion changes temporal simultaneity perception and improves temporal discrimination in audiovisual processing.