Spatiotemporal cortical dynamics for visual scene processing as revealed by EEG decoding.

The human visual system rapidly recognizes the categories and global properties of complex natural scenes. The present study investigated the spatiotemporal dynamics of neural signals involved in visual scene processing using electroencephalography (EEG) decoding. We recorded visual evoked potentials from 11 human observers for 232 natural scenes, each of which belonged to one of 13 natural scene categories (e.g., a bedroom or open country) and had three global properties (naturalness, openness, and roughness). We trained a deep convolutional classification model of the natural scene categories and global properties using EEGNet. Having confirmed that the model successfully classified natural scene categories and the three global properties, we applied Grad-CAM to the EEGNet model to visualize the EEG channels and time points that contributed to the classification. The analysis showed that EEG signals in the occipital electrodes at short latencies (approximately 80 ~ ms) contributed to the classifications, whereas those in the frontal electrodes at relatively long latencies (200 ~ ms) contributed to the classification of naturalness and the individual scene category. These results suggest that different global properties are encoded in different cortical areas and with different timings, and that the combination of the EEGNet model and Grad-CAM can be a tool to investigate both temporal and spatial distribution of natural scene processing in the human brain.

Perceiving the representative surface color of real-world materials.

Natural surfaces such as soil, grass, and skin usually involve far more complex and heterogenous structures than the perfectly uniform surfaces assumed in studies on color and material perception. Despite this, we can easily perceive the representative color of these surfaces. Here, we investigated the visual mechanisms underlying the perception of representative surface color using 120 natural images of diverse materials and their statistically synthesized images. Our matching experiments indicated that the perceived representative color revealed was not significantly different from the Portilla-Simoncelli-synthesized images or phase-randomized images except for one sample, even though the perceived shape and material properties were greatly impaired in the synthetic stimuli. The results also showed that the matched representative colors were predictable from the saturation-enhanced color of the brightest point in the image, excluding the high-intensity outliers. The results support the notion that humans judge the representative color and lightness of real-world surfaces depending on simple image measurements.

A two-stage spectral model for sound texture perception: Synthesis and psychophysics.

The natural environment is filled with a variety of auditory events such as wind blowing, water flowing, and fire crackling. It has been suggested that the perception of such textural sounds is based on the statistics of the natural auditory events. Inspired by a recent spectral model for visual texture perception, we propose a model that can describe the perceived sound texture only with the linear spectrum and the energy spectrum. We tested the validity of the model by using synthetic noise sounds that preserve the two-stage amplitude spectra of the original sound. Psychophysical experiment showed that our synthetic noises were perceived as like the original sounds for 120 real-world auditory events. The performance was comparable with the synthetic sounds produced by McDermott-Simoncelli's model which considers various classes of auditory statistics. The results support the notion that the perception of natural sound textures is predictable by the two-stage spectral signals.

Spatial attention in perceptual decision making as revealed by response-locked classification image analysis.

In many situations, humans serially sample information from many locations in an image to make an appropriate decision about a visual target. Spatial attention and eye movements play a crucial role in this serial vision process. To investigate the effect of spatial attention in such dynamic decision making, we applied a classification image (CI) analysis locked to the observer's reaction time (RT). We asked human observers to detect as rapidly as possible a target whose contrast gradually increased on the left or right side of dynamic noise, with the presentation of a spatial cue. The analysis revealed a spatiotemporally biphasic profile of the CI which peaked at ~ 350 ms before the observer's response. We found that a valid cue presented at the target location shortened the RT and increased the overall amplitude of the CI, especially when the cue appeared 500-1250 ms before the observer's response. The results were quantitatively accounted for by a simple perceptual decision mechanism that accumulates the outputs of the spatiotemporal contrast detector, whose gain is increased by sustained attention to the cued location.

Building a decoder of perceptual decisions from microsaccades and pupil size.

Many studies have reported neural correlates of visual awareness across several brain regions, including the sensory, parietal, and frontal areas. In most of these studies, participants were instructed to explicitly report their perceptual experience through a button press or verbal report. It is conceivable, however, that explicit reporting itself may trigger specific neural responses that can confound the direct examination of the neural correlates of visual awareness. This suggests the need to assess visual awareness without explicit reporting. One way to achieve this is to develop a technique to predict the visual awareness of participants based on their peripheral responses. Here, we used eye movements and pupil sizes to decode trial-by-trial changes in the awareness of a stimulus whose visibility was deteriorated due to adaptation-induced blindness (AIB). In the experiment, participants judged whether they perceived a target stimulus and rated the confidence they had in their perceptual judgment, while their eye movements and pupil sizes were recorded. We found that not only perceptual decision but also perceptual confidence can be separately decoded from the eye movement and pupil size. We discuss the potential of this technique with regard to assessing visual awareness in future neuroimaging experiments.

Ensemble perception without phenomenal awareness of elements.

Humans efficiently recognize complex scenes by grouping multiple features and objects into ensembles. It has been suggested that ensemble processing does not require, or even impairs, conscious discrimination of individual element properties. The present study examined whether ensemble perception requires phenomenal awareness of elements. We asked observers to judge the mean orientation of a line-based texture pattern whose central region was made invisible by backward masks. Masks were composed of either a Mondrian pattern (Exp. 1) or of an annular contour (Exp. 2) which, unlike the Mondrian, did not overlap spatially with elements in the central region. In the Mondrian-mask experiment, perceived mean orientation was determined only by visible elements outside the central region. However, in the annular-mask experiment, perceived mean orientation matched the mean orientation of all elements, including invisible elements within the central region. Results suggest that the visual system can compute spatial ensembles even without phenomenal awareness of stimuli.

Photorealistic Reconstruction of Visual Texture From EEG Signals.

Recent advances in brain decoding have made it possible to classify image categories based on neural activity. Increasing numbers of studies have further attempted to reconstruct the image itself. However, because images of objects and scenes inherently involve spatial layout information, the reconstruction usually requires retinotopically organized neural data with high spatial resolution, such as fMRI signals. In contrast, spatial layout does not matter in the perception of "texture," which is known to be represented as spatially global image statistics in the visual cortex. This property of "texture" enables us to reconstruct the perceived image from EEG signals, which have a low spatial resolution. Here, we propose an MVAE-based approach for reconstructing texture images from visual evoked potentials measured from observers viewing natural textures such as the textures of various surfaces and object ensembles. This approach allowed us to reconstruct images that perceptually resemble the original textures with a photographic appearance. The present approach can be used as a method for decoding the highly detailed "impression" of sensory stimuli from brain activity.

Textures vs Non-Textures: A Simple Computational Method for Classifying Perceived 'Texturality' in Natural Images.

The visual system represents textural image regions as simple statistics that are useful for the rapid perception of scenes and surfaces. What images 'textures' are, however, has so far mostly been subjectively defined. The present study investigated the empirical conditions under which natural images are processed as texture. We first show that 'texturality' - i.e., whether or not an image is perceived as a texture - is strongly correlated with the perceived similarity between an original image and its Portilla-Simoncelli (PS) synthesized image. We found that both judgments are highly correlated with specific PS statistics of the image. We also demonstrate that a discriminant model based on a small set of image statistics could discriminate whether a given image was perceived as a texture with over 90% accuracy. The results provide a method to determine whether a given image region is represented statistically by the human visual system.

Response-locked classification image analysis of perceptual decision making in contrast detection.

In many situations, humans make decisions based on serially sampled information through the observation of visual stimuli. To quantify the critical information used by the observer in such dynamic decision making, we here applied a classification image (CI) analysis locked to the observer's reaction time (RT) in a simple detection task for a luminance target that gradually appeared in dynamic noise. We found that the response-locked CI shows a spatiotemporally biphasic weighting profile that peaked about 300 ms before the response, but this profile substantially varied depending on RT; positive weights dominated at short RTs and negative weights at long RTs. We show that these diverse results are explained by a simple perceptual decision mechanism that accumulates the output of the perceptual process as modelled by a spatiotemporal contrast detector. We discuss possible applications and the limitations of the response-locked CI analysis.

Analysis and Synthesis of Natural Texture Perception From Visual Evoked Potentials.

The primate visual system analyzes statistical information in natural images and uses it for the immediate perception of scenes, objects, and surface materials. To investigate the dynamical encoding of image statistics in the human brain, we measured visual evoked potentials (VEPs) for 166 natural textures and their synthetic versions, and performed a reverse-correlation analysis of the VEPs and representative texture statistics of the image. The analysis revealed occipital VEP components strongly correlated with particular texture statistics. VEPs correlated with low-level statistics, such as subband SDs, emerged rapidly from 100 to 250 ms in a spatial frequency dependent manner. VEPs correlated with higher-order statistics, such as subband kurtosis and cross-band correlations, were observed at slightly later times. Moreover, these robust correlations enabled us to inversely estimate texture statistics from VEP signals via linear regression and to reconstruct texture images that appear similar to those synthesized with the original statistics. Additionally, we found significant differences in VEPs at 200-300 ms between some natural textures and their Portilla-Simoncelli (PS) synthesized versions, even though they shared almost identical texture statistics. This differential VEP was related to the perceptual "unnaturalness" of PS-synthesized textures. These results suggest that the visual cortex rapidly encodes image statistics hidden in natural textures specifically enough to predict the visual appearance of a texture, while it also represents high-level information beyond image statistics, and that electroencephalography can be used to decode these cortical signals.

Human Texture Vision as Multi-Order Spectral Analysis.

Texture information plays a critical role in the rapid perception of scenes, objects, and materials. Here, we propose a novel model in which visual texture perception is essentially determined by the 1st-order (2D-luminance) and 2nd-order (4D-energy) spectra. This model is an extension of the dimensionality of the Filter-Rectify-Filter (FRF) model, and it also corresponds to the frequency representation of the Portilla-Simoncelli (PS) statistics. We show that preserving two spectra and randomizing phases of a natural texture image result in a perceptually similar texture, strongly supporting the model. Based on only two single spectral spaces, this model provides a simpler framework to describe and predict texture representations in the primate visual system. The idea of multi-order spectral analysis is consistent with the hierarchical processing principle of the visual cortex, which is approximated by a multi-layer convolutional network.

Spatiotemporal frequency characteristics of the visual unpleasantness of dynamic bandpass noise.

Recent psychophysical evidence shows that visual discomfort and unpleasantness are related to particular image features such as the spatial frequency and orientation spectrum. We also have a strong unpleasant feeling toward moving objects such as swarming worms, but it is poorly understood how motion information relates to a feeling of unpleasantness. The present study investigated spatiotemporal frequency characteristics that cause visual unpleasantness using bandpass noise with variable spatial frequencies, temporal frequencies, temporal frequency bandwidths, and orientation bandwidths. Results show that dynamic noise with relatively low temporal frequencies (0.5-2 Hz) was markedly more unpleasant than static noise, including that judged as the most unpleasant in a previous study. Remarkably, translational motion of the noise did not increase the feeling of unpleasantness. A subsequent experiment using a dynamic texture in which elements moved in a variable range of random directions showed that the variegated motion direction plays a critical role in promoting visual unpleasantness. Natural scenes have regularity in that features inside an object usually move in the same direction and rarely at random, and the present results further support the notion that deviation from the statistical regularity of natural scenes in images and movies induces negative emotions.

Peak-at-end rule: adaptive mechanism predicts time-dependent decision weighting.

Humans make decisions under various natural circumstances, integrating multiple pieces of information that are distributed over space and time. Although psychophysical and physiological studies have investigated temporal dynamics underlying perceptual decision making, weighting profiles for inliers and outliers during temporal integration have yet to be fully investigated in most studies. Here, we examined the temporal weighting profile of a computational model characterized by a leaky integrator of sensory evidence. As a corollary of its leaky nature, the model predicts the recency effect and overweights outlying elements around the end of the stream. Moreover, we found that the model underweights outlying values occurring earlier in the stream (i.e., robust averaging). We also show that human observers exhibit exactly the same weighting profile in an average estimation task. These findings suggest that the adaptive decision process in the brain results in the time-dependent decision weighting, the "peak-at-end" rule, rather than the peak-end rule in behavioral economics.

Differential Effects of Orientation and Spatial-Frequency Spectra on Visual Unpleasantness.

Increasing psychophysical evidence suggests that specific image features - or statistics - can appear unpleasant or induce visual discomfort in humans. Such unpleasantness tends to be particularly profound if the image's amplitude spectrum deviates from the regular 1/f spatial-frequency falloff expected in natural scenes. Here, we show that profound unpleasant impressions also result if the orientation spectrum of the image becomes flatter. Using bandpass noise with variable orientation and spatial-frequency bandwidths, we found that unpleasantness ratings decreased with spatial- frequency bandwidth but increased with orientation bandwidth. Similarly, a subsequent experiment revealed that sinusoidal modulations in the amplitude spectrum of 1/f noise along the spatial frequency increased unpleasantness, but modulations along the orientation decreased it. Given that natural scenes tend to have a linear slope along the spatial frequency but an uneven spectrum along the orientation dimension, our opposing results in the spatial-frequency and orientation domains commonly support the idea that images deviating from the spectral regularity of natural scenes can give rise to unpleasant impressions.

Distinct strategies for estimating the temporal average of numerical and perceptual information.

Humans can estimate global trends in dynamic information presented either as perceptual features or as symbolic codes such as numbers. Previous studies on temporal statistics estimation have shown that observers judge the temporal average of visual attributes according to information from the last few frames of the presentation sequence (in what is referred to as the recency effect). Here, we investigated how humans estimate the temporal average of number vs. orientation using identical stimuli for the two tasks. In Experiment 1, a randomly-selected single-digit number was serially presented at orientations randomly varying over time. In Experiment 2, a texture comprising a random number of Gabor elements was shown at orientations randomly varying over time. In both experiments, observers judged the temporal averages of the numerical values and orientations in separate blocks. Results showed that observers judging the temporal average of orientation relied upon information from later frames as predicted by a typical model of perceptual decision making. By contrast, for the judgement of numerical values, we found that the impacts of each temporal frame were constant or varied little across temporal frames regardless of whether the numerical information was given as digits or by the number of texture elements. The results are interpreted as evidence that distinct computational strategies may be involved in estimating the temporal averages of perceptual features and numerical information.

Adaptive comparison matrix: An efficient method for psychological scaling of large stimulus sets.

Studies on natural and social vision often need to quantify subjective intensity along a particular dimension for a large number of stimuli whose perceptual ordering is unknown. Here, we introduce an easy experimental protocol of comparative judgments that can rank and scale subjective stimulus intensity using a comparatively small number of trials. On each trial in our protocol, the observer initially views M stimuli sampled from a space of N stimuli and selects the stimulus that elicits maximum subjective response along a given dimension (e.g., the most attractive). The selected stimulus is subsequently discarded, the observer then performs a judgment on the remaining stimuli, and the process is iterated until the last stimulus remains and a new trial begins. The method relies on sorting perceived stimulus order in the N x N comparison matrix via logistic regression and sampling the next set of M stimuli such that responses will be collected only for stimulus pairs whose expected response ratio is most informative. Numerical simulations demonstrate that this method can estimate psychological scale with a small number of responses. Psychophysical experiments confirm that the method can quickly estimate the contrast response function for gratings and the perceived glossiness of naturalistic objects. This protocol would be useful for characterizing human judgments along various dimensions, especially those with no physical image correlates such as emotional and social attributes.

Perception and decision mechanisms involved in average estimation of spatiotemporal ensembles.

A number of studies on texture and ensemble perception have shown that humans can immediately estimate the average of spatially distributed visual information. The present study characterized mechanisms involved in estimating averages for information distributed over both space and time. Observers viewed a rapid sequence of texture patterns in which elements' orientation were determined by dynamic Gaussian noise with variable spatial and temporal standard deviations (SDs). We found that discrimination thresholds increased beyond a certain spatial SD if temporal SD was small, but if temporal SD was large, thresholds remained nearly constant regardless of spatial SD. These data are at odds with predictions that threshold is uniquely determined by spatiotemporal SD. Moreover, a reverse correlation analysis revealed that observers judged the spatiotemporal average orientation largely depending on the spatial average orientation over the last few frames of the texture sequence - a recency effect widely observed in studies of perceptual decision making. Results are consistent with the notion that the visual system rapidly computes spatial ensembles and adaptively accumulates information over time to make a decision on spatiotemporal average. A simple computational model based on this notion successfully replicated observed data.

Perception of human skin conditions and image statistics.

The color and texture of human skin provide useful information in evaluating a person's health, emotion, attractiveness, and so on. Whereas the appearance of skin is a product of optical properties originating from its own biophysical substrates, recent psychophysical studies of surface material perception indicate a possibility that humans can perceive skin conditions from relatively simple image features such as texture statistics. The present study investigated the structure of multidimensional perceptual rating data for 289 female skin images and analyzed diagnostic low-level image statistics. We found that various skin appearances can be summarized into two dimensions, pleasantness and glossiness, and either dimension is well correlated with a small set of subband luminance and color statistics. We confirmed that manipulations of these critical image statistics significantly alter the appearance of skin in the expected direction. The results support a notion that humans perceive skin properties essentially based on two cardinal dimensions as predicted by simple image statistics.

Perceiving Natural Speed in Natural Movies.

The visual system uses the physical laws of nature as constraints for perceiving objects and events. Images violating natural laws would therefore tend to be perceived as unnatural. To understand vision's implicit knowledge of natural speed in the real world, we examined visual tolerance to artificial speed deviations in 22 natural movies. For most movies, perception could tolerate deviations from original speed by as much as a factor 2×. However, for movies including human body movements or falling objects, perception only tolerated a significantly narrower range of speed deviations. In general, human observers are poor at judging the naturalness of speed in natural scenes except for events involving gravitational or biological motions.

Attention Periodically Binds Visual Features As Single Events Depending on Neural Oscillations Phase-Locked to Action.

Recent psychophysical studies have demonstrated that periodic attention in the 4-8 Hz range facilitates performance on visual detection. The present study examined the periodicity of feature binding, another major function of attention, in human observers (3 females and 5 males for behavior, with 7 males added for the EEG experiment). In a psychophysical task, observers reported a synchronous pair of brightness (light/dark) and orientation (clockwise/counterclockwise) patterns from two combined brightness-orientation pairs presented in rapid succession. We found that temporal binding performance exhibits periodic oscillations at ∼8 Hz as a function of stimulus onset delay from a self-initiated button press in conditions where brightness-orientation pairs were spatially separated. However, as one would expect from previous studies on pre-attentive binding, significant oscillations were not apparent in conditions where brightness-orientation pairs were spatially superimposed. EEG results, while fully compatible with behavioral oscillations, also revealed a significant dependence of binding performance across trials on prestimulus neural oscillatory phases within the corresponding band. The peak frequency of this dependence was found to be correlated with intertrial phase coherence (ITPC) around the timing of button press in parietal sensors. Moreover, the peak frequency of the ITPC was found to predict behavioral frequency in individual observers. Together, these results suggest that attention operates periodically (at ∼8 Hz) on the perceptual binding of multimodal visual information and is mediated by neural oscillations phase-locked to voluntary action.SIGNIFICANCE STATEMENT Recent studies in neuroscience suggest that the brain's attention network operates rhythmically at 4-8 Hz. The present behavioral task revealed that attentional binding of visual features is performed periodically at ∼8 Hz, and EEG analysis showed a dependence of binding performance on prestimulus neural oscillatory phase. Furthermore, this association between perceptual and neural oscillations is triggered by voluntary action. Periodic processes driven by attention appear to contribute not only to sensory processing but also to the temporal binding of diverse information into a conscious event synchronized with action.