The effect of processing partial information in dynamic face perception.

Face perception is a major topic in vision research. Most previous research has concentrated on (holistic) spatial representations of faces, often with static faces as stimuli. However, faces are highly dynamic stimuli containing important temporal information. How sensitive humans are regarding temporal information in dynamic faces is not well understood. Studies investigating temporal information in dynamic faces usually focus on the processing of emotional expressions. However, faces also contain relevant temporal information without any strong emotional expression. To investigate cues that modulate human sensitivity to temporal order, we utilized muted dynamic neutral face videos in two experiments. We varied the orientation of the faces (upright and inverted) and the presence/absence of eye blinks as partial dynamic cues. Participants viewed short, muted, monochromic videos of models vocalizing a widely known text (National Anthem). Videos were played either forward (in the correct temporal order) or backward. Participants were asked to determine the direction of the temporal order for each video, and (at the end of the experiment) whether they had understood the speech. We found that face orientation, and the presence/absence of an eye blink affected sensitivity, criterion (bias) and reaction time: Overall, sensitivity was higher for upright compared to inverted faces, and in the condition where an eye blink was present compared to the condition without an eye blink. Reaction times were mostly faster in the conditions with higher sensitivity. A bias to report inverted faces as 'backward' observed in Experiment I, where upright and inverted faces were presented randomly interleaved within each block, was absent when presenting upright and inverted faces in different blocks in Experiment II. Language comprehension results revealed that there was higher sensitivity when understanding the speech compared to not understanding the speech in both experiments. Taken together, our results showed higher sensitivity with upright compared to inverted faces, suggesting that the perception of dynamic, task-relevant information was superior with the canonical orientation of the faces. Furthermore, partial information coming from eye blinks, in addition to mouth movements, seemed to play a significant role in dynamic face perception, both when faces were presented upright and inverted. We suggest that studying the perception of facial dynamics beyond emotional expressions will help us to better understand the mechanisms underlying the temporal integration of facial information from different -partial and holistic- sources, and that our results show how different strategies, depending on the available information, are employed by human observers when judging the temporal order of faces.

The Sabanci University Dynamic Face Database (SUDFace): Development and validation of an audiovisual stimulus set of recited and free speeches with neutral facial expressions.

Faces convey a wide range of information, including one's identity, and emotional and mental states. Face perception is a major research topic in many research fields, such as cognitive science, social psychology, and neuroscience. Frequently, stimuli are selected from a range of available face databases. However, even though faces are highly dynamic, most databases consist of static face stimuli. Here, we introduce the Sabanci University Dynamic Face (SUDFace) database. The SUDFace database consists of 150 high-resolution audiovisual videos acquired in a controlled lab environment and stored with a resolution of 1920 × 1080 pixels at a frame rate of 60 Hz. The multimodal database consists of three videos of each human model in frontal view in three different conditions: vocalizing two scripted texts (conditions 1 and 2) and one Free Speech (condition 3). The main focus of the SUDFace database is to provide a large set of dynamic faces with neutral facial expressions and natural speech articulation. Variables such as face orientation, illumination, and accessories (piercings, earrings, facial hair, etc.) were kept constant across all stimuli. We provide detailed stimulus information, including facial features (pixel-wise calculations of face length, eye width, etc.) and speeches (e.g., duration of speech and repetitions). In two validation experiments, a total number of 227 participants rated each video on several psychological dimensions (e.g., neutralness and naturalness of expressions, valence, and the perceived mental states of the models) using Likert scales. The database is freely accessible for research purposes.

Neural correlates of integration processes during dynamic face perception.

Integrating the spatiotemporal information acquired from the highly dynamic world around us is essential to navigate, reason, and decide properly. Although this is particularly important in a face-to-face conversation, very little research to date has specifically examined the neural correlates of temporal integration in dynamic face perception. Here we present statistically robust observations regarding the brain activations measured via electroencephalography (EEG) that are specific to the temporal integration. To that end, we generate videos of neutral faces of individuals and non-face objects, modulate the contrast of the even and odd frames at two specific frequencies ([Formula: see text] and [Formula: see text]) in an interlaced manner, and measure the steady-state visual evoked potential as participants view the videos. Then, we analyze the intermodulation components (IMs: ([Formula: see text]), a linear combination of the fundamentals with integer multipliers) that consequently reflect the nonlinear processing and indicate temporal integration by design. We show that electrodes around the medial temporal, inferior, and medial frontal areas respond strongly and selectively when viewing dynamic faces, which manifests the essential processes underlying our ability to perceive and understand our social world. The generation of IMs is only possible if even and odd frames are processed in succession and integrated temporally, therefore, the strong IMs in our frequency spectrum analysis show that the time between frames (1/60 s) is sufficient for temporal integration.

Measuring Integration Processes in Visual Symmetry with Frequency-Tagged EEG.

Symmetry is a highly salient feature of the natural world which requires integration of visual features over space. The aim of the current work is to isolate dynamic neural correlates of symmetry-specific integration processes. We measured steady-state visual evoked potentials (SSVEP) as participants viewed symmetric patterns comprised of distinct spatial regions presented at two different frequencies (f1 and f2). We measured intermodulation components, shown to reflect non-linear processing at the neural level, indicating integration of spatially separated parts of the pattern. We generated a wallpaper pattern containing two reflection symmetry axes by tiling the plane with a two-fold reflection symmetric unit-pattern and split each unit-pattern diagonally into separate parts which could be presented at different frequencies. We compared SSVEPs measured for wallpapers and control patterns for which both images were equal in terms of translation and rotation symmetry but reflection symmetry could only emerge for the wallpaper pattern through integration of the image-pairs. We found that low-frequency intermodulation components differed between the wallpaper and control stimuli, indicating the presence of integration mechanisms specific to reflection symmetry. These results showed that spatial integration specific to symmetry perception can be isolated through a combination of stimulus design and the frequency tagging approach.

EEG frequency tagging reveals higher order intermodulation components as neural markers of learned holistic shape representations.

Shape perception is intrinsically holistic: combinations of features give rise to configurations with emergent properties that are different from the sum of the parts. The current study investigated neural markers of holistic shape representations learned by means of categorization training. We used the EEG frequency tagging technique, where two parts of a shape stimulus were 'tagged' by modifying their contrast at different temporal frequencies. Signals from both parts are integrated and, as a result, emergent frequency components (so-called, intermodulation responses, IMs), caused by nonlinear interaction of two frequency signals, are observed in the EEG spectrum. First, participants were trained in 4 sessions to discriminate highly similar, unfamiliar shapes into two categories, defined based on the combination of features. After training, EEG was recorded while frequency-tagged shapes from either the trained or the untrained shape family were presented. For all IMs combined, no learning effects were detected, but post hoc analyses of higher-order IMs revealed stronger occipital and occipito-temporal IMs for both trained and untrained exemplars of the trained shape family as compared to the untrained shape family. In line with recent findings, we suggest that the higher-order IMs may reflect high-level visual computations, like holistic shape categorization, resulting from a cascade of non-linear operations. Higher order frequency responses are relatively low in power, hence results should be interpreted cautiously and future research is needed to confirm these effects. In general, these findings are, to our knowledge, the first to show IMs as a neural correlate of perceptual learning.

EEG frequency tagging dissociates between neural processing of motion synchrony and human quality of multiple point-light dancers.

Do we perceive a group of dancers moving in synchrony differently from a group of drones flying in-sync? The brain has dedicated networks for perception of coherent motion and interacting human bodies. However, it is unclear to what extent the underlying neural mechanisms overlap. Here we delineate these mechanisms by independently manipulating the degree of motion synchrony and the humanoid quality of multiple point-light displays (PLDs). Four PLDs moving within a group were changing contrast in cycles of fixed frequencies, which permits the identification of the neural processes that are tagged by these frequencies. In the frequency spectrum of the steady-state EEG we found two emergent frequency components, which signified distinct levels of interactions between PLDs. The first component was associated with motion synchrony, the second with the human quality of the moving items. These findings indicate that visual processing of synchronously moving dancers involves two distinct neural mechanisms: one for the perception of a group of items moving in synchrony and one for the perception of a group of moving items with human quality. We propose that these mechanisms underlie high-level perception of social interactions.

Frequency tagging yields an objective neural signature of Gestalt formation.

The human visual system integrates separate visual inputs into coherently organized percepts, going beyond the information given. A striking example is the perception of an illusory square when physically separated inducers are positioned and oriented in a square-like configuration (illusory condition). This illusory square disappears when the specific configuration is broken, for instance, by rotating each inducer (non-illusory condition). Here we used frequency tagging and electroencephalography (EEG) to identify an objective neural signature of the global integration required for illusory surface perception. Two diagonal inducers were contrast-modulated at different frequency rates f1 and f2, leading to EEG responses exactly at these frequencies over the occipital cortex. Most importantly, nonlinear intermodulation (IM) components (e.g., f1+f2) appeared in the frequency spectrum, and were much larger in response to the illusory square figure than the non-illusory control condition. Since the IMs reflect long-range interactions between the signals from the inducers, these data provide an objective (i.e., at a precise and predicted EEG frequency) signature of neural processes involved in the emergence of illusory surface perception. More generally, these findings help to establish EEG frequency-tagging as a highly valuable approach to investigate the underlying neural mechanisms of subjective Gestalt phenomena in an objective and quantitative manner, at the system level in humans.

Depth perception of illusory surfaces.

The perception of an illusory surface, a subjectively perceived surface that is not given in the image, is one of the most intriguing phenomena in vision. It strongly influences the perception of some fundamental properties, namely, depth, lightness and contours. Recently, we suggested (1) that the context-sensitive mechanism of depth computation plays a key role in creating the illusion, (2) that the illusory lightness perception can be explained by an influence of depth perception on the lightness computation, and (3) that the perception of variations of the Kanizsa figure can be well-reproduced by implementing these principles in a model (Kogo, Strecha, et al., 2010). However, depth perception, lightness perception, contour perception, and their interactions can be influenced by various factors. It is essential to measure the differences between the variation figures in these aspects separately to further understand the mechanisms. As a first step, we report here the results of a new experimental paradigm to compare the depth perception of the Kanizsa figure and its variations. One of the illusory figures was presented side-by-side with a non-illusory variation whose stereo disparities were varied. Participants had to decide in which of these two figures the central region appeared closer. The results indicate that the depth perception of the illusory surface was indeed different in the variation figures. Furthermore, there was a non-linear interaction between the occlusion cues and stereo disparity cues. Implications of the results for the neuro-computational mechanisms are discussed.