Perceptual task drives later fixations and long latency saccades, while early fixations and short latency saccades are more automatic.

We used a simple stimulus, dissociating perceptually relevant information in space, to differentiate between bottom-up and task-driven fixations. Six participants viewed a dynamic scene showing the reaction of an elastic object fixed to the ceiling being hit. In one condition they had to judge the object's stiffness and in the other condition its lightness. The results show that initial fixations tend to land in the centre of an object, independent of the task. After the initial fixation, participants tended to look at task diagnostic regions. This fixation behaviour correlates with high perceptual performance. Similarly, low-latency saccades lead to fixations that do not depend on the task, whereas higher latency does.

Material category of visual objects computed from specular image structure.

Recognizing materials and their properties visually is vital for successful interactions with our environment, from avoiding slippery floors to handling fragile objects. Yet there is no simple mapping of retinal image intensities to physical properties. Here, we investigated what image information drives material perception by collecting human psychophysical judgements about complex glossy objects. Variations in specular image structure-produced either by manipulating reflectance properties or visual features directly-caused categorical shifts in material appearance, suggesting that specular reflections provide diagnostic information about a wide range of material classes. Perceived material category appeared to mediate cues for surface gloss, providing evidence against a purely feedforward view of neural processing. Our results suggest that the image structure that triggers our perception of surface gloss plays a direct role in visual categorization, and that the perception and neural processing of stimulus properties should be studied in the context of recognition, not in isolation.

Assessing the representational structure of softness activated by words.

Softness is an important material property that can be judged directly, by interacting with an object, but also indirectly, by simply looking at an image of a material. The latter is likely possible by filling in relevant multisensory information from prior experiences with soft materials. Such experiences are thought to lead to associations that make up our representations about perceptual softness. Here, we investigate the structure of this representational space when activated by words, and compare it to haptic and visual perceptual spaces that we obtained in earlier work. To this end, we performed an online study where people rated different sensory aspects of soft materials, presented as written names. We compared the results with the previous studies where identical ratings were made on the basis of visual and haptic information. Correlation and Procrustes analyses show that, overall, the representational spaces of verbally presented materials were similar to those obtained from haptic and visual experiments. However, a classifier analysis showed that verbal representations could better be predicted from those obtained from visual than from haptic experiments. In a second study we rule out that these larger discrepancies in representations between verbal and haptic conditions could be due to difficulties in material identification in haptic experiments. We discuss the results with respect to the recent idea that at perceived softness is a multidimensional construct.

EEG Decoding Reveals Neural Predictions for Naturalistic Material Behaviors.

Material properties, such as softness or stickiness, determine how an object can be used. Based on our real-life experience, we form strong expectations about how objects should behave under force, given their typical material properties. Such expectations have been shown to modulate perceptual processes, but we currently do not know how expectation influences the temporal dynamics of the cortical visual analysis for objects and their materials. Here, we tracked the neural representations of expected and unexpected material behaviors using time-resolved EEG decoding in a violation-of-expectation paradigm, where objects fell to the ground and deformed in expected or unexpected ways. Participants were 25 men and women. Our study yielded three key results: First, both objects and materials were represented rapidly and in a temporally sustained fashion. Second, objects exhibiting unexpected material behaviors were more successfully decoded than objects exhibiting expected behaviors within 190 ms after the impact, which might indicate additional processing demands when expectations are unmet. Third, general signals of expectation fulfillment that generalize across specific objects and materials were found within the first 150 ms after the impact. Together, our results provide new insights into the temporal neural processing cascade that underlies the analysis of real-world material behaviors. They reveal a sequence of predictions, with cortical signals progressing from a general signature of expectation fulfillment toward increased processing of unexpected material behaviors.SIGNIFICANCE STATEMENT In the real world, we can make accurate predictions about how an object's material shapes its behavior: For instance, we know that cups are typically made of porcelain and shatter when we accidentally drop them. Here, we use EEG to experimentally test how expectations about material behaviors impact neural processing. We showed our participants videos of objects that exhibited expected material behaviors (e.g., a glass shattering when falling to the ground) or unexpected material behaviors (e.g., a glass melting on impact). Our results reveal a hierarchy of predictions in cortex: The visual system rapidly generates signals that index whether expectations about material behaviors are met. These signals are followed by increased processing of objects displaying unexpected material behaviors.

Unmet expectations about material properties delay perceptual decisions.

Based on our expectations about material properties, we can implicitly predict an object's future states, e.g., a wine glass falling down will break when it hits the ground. How these expectations affect relatively low-level perceptual decisions, however, has not been systematically studied previously. To seek an answer to this question, we conducted a behavioral experiment using animations of various familiar objects (e.g., key, wine glass, etc.) freely falling and hitting the ground. During a training session, participants first built expectations about the dynamic properties of those objects. Half of the participants (N = 28) built expectations consistent with their daily lives (e.g., a key bounces rigidly), whereas the other half learned an atypical behavior (e.g., a key wobbles). This was followed by experimental sessions, in which expectations were unmet in 20% of the trials. In both training and experimental sessions, the participant's task was to report whether the objects broke or not upon hitting the ground. Critically, a specific object always remained intact or broke - only the manner in which it did so differed. For example, a key could wobble or remain rigid but never break. We found that participants' reaction times were longer when expectations were unmet, not only for typical material behavior but also when those expectations were atypical and learned during the training session. Furthermore, we found an interplay between long-term and newly learned expectations. Overall, our results show that expectations about material properties can impact relatively low-level perceptual decision-making processes.

Hand explorations are determined by the characteristics of the perceptual space of real-world materials from silk to sand.

Perceiving mechanical properties of objects, i.e., how they react to physical forces, is a crucial ability in many aspects of life, from choosing an avocado to picking your clothes. There is, a wide variety of materials that differ substantially in their mechanical properties. For example, both, silk and sand deform and change shape in response to exploration forces, but each does so in very different ways. Studies show that the haptic perceptual space has multiple dimensions corresponding to the physical properties of textures, however in these experiments the range of materials or exploratory movements were restricted. Here we investigate the perceptual dimensionality in a large set of real materials in a free haptic exploration task. Thirty-two participants actively explored deformable and non-deformable materials with their hands and rated them on several attributes. Using the semantic differential technique, video analysis and linear classification, we found four haptic dimensions, each associated with a distinct set of hand and finger movements during active exploration. Taken together our findings suggest that the physical, particularly the mechanical, properties of a material systematically affect how it is explored on a much more fine-grained level than originally thought.

The look and feel of soft are similar across different softness dimensions.

The softness of objects can be perceived through several senses. For instance, to judge the softness of a cat's fur, we do not only look at it, we often also run our fingers through its coat. Recently, we have shown that haptically perceived softness covaries with the compliance, viscosity, granularity, and furriness of materials (Dovencioglu, Üstün, Doerschner, & Drewing, 2020). However, it is unknown whether vision can provide similar information about the various aspects of perceived softness. Here, we investigated this question in an experiment with three conditions: in the haptic condition, blindfolded participants explored materials with their hands, in the static visual condition participants were presented with close-up photographs of the same materials, and in the dynamic visual condition participants watched videos of the hand-material interactions that were recorded in the haptic condition. After haptically or visually exploring the materials, participants rated them on various attributes. Our results show a high overall perceptual correspondence among the three experimental conditions. With a few exceptions, this correspondence tended to be strongest between haptic and dynamic visual conditions. These results are discussed with respect to information potentially available through the senses, or through prior experience, when judging the softness of materials.

Cortical connectivity in the face of congenital structural changes-A case of homozygous LAMC3 mutation.

The homozygous LAMC3 gene mutation is associated with severe bilateral smoothening and thickening of the lateral occipital cortex . Despite this and further significant changes in gray matter structure, a patient harboring this mutation exhibited a range of remarkably intact perceptual abilities . One possible explanation of this perceptual sparing could be that the white matter structural integrity and functional connectivity in relevant pathways remained intact. To test this idea, we used diffusion tensor and functional magnetic resonance imaging to investigate functional connectivity in resting-state networks in major structural pathways involved in object perception and visual attention and corresponding microstructural integrity in a patient with homozygous LAMC3 mutation and sex, age, education, and socioeconomically matched healthy control group. White matter microstructural integrity results indicated widespread disruptions in both intra- and interhemispheric structural connections except inferior longitudinal fasciculus. With a few exceptions, the functional connectivity between the patient's adjacent gray matter regions of major white matter tracts of interest was conserved. In addition, functional localizers for face, object, and place areas showed similar results with a representative control, providing an explanation for the patient's intact face, place, and object recognition abilities. To generalize this finding, we also compared functional connectivity between early visual areas and face, place, and object category-selective areas, and we found that the functional connectivity of the patient was not different from the control group. Overall, our results provided complementary information about the effects of LAMC3 gene mutation on the human brain including intact temporo-occipital structural and functional connectivity that are compatible with preserved perceptual abilities.

The color appearance of curved transparent objects.

Studies on colored transparent objects have elucidated potential mechanisms, but these studies have mainly focused on flat filters overlaying flat backgrounds. While they have provided valuable insight, these studies have not captured all aspects of transparency, like caustics, specular reflections/highlights, and shadows. Here, we investigate color-matching experiments with curved transparent objects for different matching stimuli: a uniform patch and a flat filter. Two instructions were tested: simply match the color of the glass object and the test element (patch and flat filter) or match the color of the dye that was used to tint the transparent object (patch). Observers' matches differed from the mean, the most frequent, and the most saturated color of the transparent stimuli, whereas the brightest regions captured the chromaticity, but not the lightness, of patch matches. We applied four models from flat filter studies: the convergence model, the ratios of either the means (RMC) or standard deviations (RSD) of cone excitations, and a robust ratio model. The original convergence model does not fully generalize but does not perform poorly, and with modifications, we find that curved transparent objects cause a convergence of filtered colors toward a point in color space, similar to flat filters. Considering that, the RMC and robust ratio models generalized more than the RSD, with the RMC performing best across the stimuli we tested. We conclude that the RMC is probably the strongest factor for determining the color. The RSD seems instead to be related to the perceived "clarity" of glass objects.

Task and Material Properties Interactively Affect Softness Explorations Along Different Dimensions.

Haptic research has frequently equated softness with the compliance of elastic objects. However, in a recent study we have suggested that compliance is not the only perceived material dimension underlying what is commonly called softness [1 ]. Here, we investigate, whether the different perceptual dimensions of softness affect how materials are haptically explored. Specifically, we tested whether also the task, i.e., the attribute that a material is being judged on, might affect how a material is explored. To this end we selected 15 adjectives and 19 materials that each associate with different softness dimensions for the study. In the experiment, while participants freely explored and rated the materials, we recorded their hand movements. These movements were subsequently categorized into distinct exploratory procedures (EPs) and analyzed in a multivariate analysis of variance (MANOVA). The results of this analysis suggest that the pattern of EPs depended not only on the material's softness dimension and the task (i.e., what attributes were rated), but also on an interaction between the two factors. Taken together, our findings support the notion of multiple perceptual dimensions of softness and suggest that participants actively adapt their EPs in a nuanced way when judging a particular softness dimensions for a given material.

Dynamic dot displays reveal material motion network in the human brain.

There is growing research interest in the neural mechanisms underlying the recognition of material categories and properties. This research field, however, is relatively more recent and limited compared to investigations of the neural mechanisms underlying object and scene category recognition. Motion is particularly important for the perception of non-rigid materials, but the neural basis of non-rigid material motion remains unexplored. Using fMRI, we investigated which brain regions respond preferentially to material motion versus other types of motion. We introduce a new database of stimuli - dynamic dot materials - that are animations of moving dots that induce vivid percepts of various materials in motion, e.g. flapping cloth, liquid waves, wobbling jelly. Control stimuli were scrambled versions of these same animations and rigid three-dimensional rotating dots. Results showed that isolating material motion properties with dynamic dots (in contrast with other kinds of motion) activates a network of cortical regions in both ventral and dorsal visual pathways, including areas normally associated with the processing of surface properties and shape, and extending to somatosensory and premotor cortices. We suggest that such a widespread preference for material motion is due to strong associations between stimulus properties. For example viewing dots moving in a specific pattern not only elicits percepts of material motion; one perceives a flexible, non-rigid shape, identifies the object as a cloth flapping in the wind, infers the object's weight under gravity, and anticipates how it would feel to reach out and touch the material. These results are a first important step in mapping out the cortical architecture and dynamics in material-related motion processing.

Expectations affect the perception of material properties.

Many objects that we encounter have typical material qualities: spoons are hard, pillows are soft, and Jell-O dessert is wobbly. Over a lifetime of experiences, strong associations between an object and its typical material properties may be formed, and these associations not only include how glossy, rough, or pink an object is, but also how it behaves under force: we expect knocked over vases to shatter, popped bike tires to deflate, and gooey grilled cheese to hang between two slices of bread when pulled apart. Here we ask how such rich visual priors affect the visual perception of material qualities and present a particularly striking example of expectation violation. In a cue conflict design, we pair computer-rendered familiar objects with surprising material behaviors (a linen curtain shattering, a porcelain teacup wrinkling, etc.) and find that material qualities are not solely estimated from the object's kinematics (i.e., its physical [atypical] motion while shattering, wrinkling, wobbling etc.); rather, material appearance is sometimes "pulled" toward the "native" motion, shape, and optical properties that are associated with this object. Our results, in addition to patterns we find in response time data, suggest that visual priors about materials can set up high-level expectations about complex future states of an object and show how these priors modulate material appearance.

Kandinsky or Me? How Free Is the Eye of the Beholder in Abstract Art?

We investigated in "art-naïve" German and Chinese participants the perception of color and spatial balance in abstract art. For color perception, we asked participants (a) to adjust the color of a single element in 24 paintings according to their liking and (b) to indicate whether they preferred their version of the painting or the original. For spatial perception, we asked participants (a) to determine the "balance point" of an artwork and (b) to indicate their preferences for the original or left-right reversed orientation of previously seen and unfamiliar paintings. Results of the color experiments suggest that, even though the interactive task was of a rather open-ended nature, observers' color adjustments were not random but systematically influenced by each painting's color palette. Overall, participants liked their own color choices about as much as the original composition. Results of the spatial experiments reveal a remarkable consistency between participants in their balance point settings. The perceived lateral position of the balance point systematically affected the left-right orientation preference for a given painting. We conclude that "art-naïve" observers are sensitive to the composition of colors and spatial structures in abstract art and are influenced by their cultural backgrounds when experiencing abstract paintings.

Disentangling simultaneous changes of surface and illumination.

Retinally incident light is an ambiguous product of spectral distributions of light in the environment and their interactions with reflecting, absorbing, and transmitting materials. An ideal color constant observer would unravel these confounded sources of information and account for changes in each factor. Scene statistics have been proposed as a way to compensate for changes in the illumination, but few theories consider changes of 3-dimensional surfaces. Here, we investigated the visual system's capacity to deal with simultaneous changes in illumination and surfaces. Spheres were imaged with a hyperspectral camera in a white box and their colors, as well as that of the illumination were varied along "red-green" and "blue-yellow" axes. Both the original hyperspectral images and replica scenes rendered with Mitsuba were used as stimuli, including rendered scenes with Glavens (Acta Psychologica, 2009, 132, 259-266). Observers viewed sequential, random pairs of our images, with either the whole scene, only the object, or only a part of the background being present. They judged how much the illuminant and object color changed on a scale of 0-100%. Observers could extract simultaneous illumination and reflectance changes when provided with a view of the whole scene, but global scene statistics did not fully account for their behavior, while local scene statistics improved the situation. There was no effect of color axis, shape, or simulated vs. original hyperspectral images. Observers appear to be making use of various sources of local information to complete the task.

Gloss and Speed Judgments Yield Different Fine Tuning of Saccadic Sampling in Dynamic Scenes.

Image motion contains potential cues about the material properties of objects. In earlier work, we proposed motion cues that could predict whether a moving object would be perceived as shiny or matte. However, whether the visual system uses these cues is still uncertain. Herein, we use the tracking of eye movements as a tool to understand what visual information observers use when engaged in material perception. Observers judged either the gloss or the speed of moving blobby shapes in an eye tracking experiment. Results indicate that during glossiness judgments, participants tend to look at gloss-diagnostic dynamic features more than during speed judgments. This suggests a fine tuning of the visual system to properties of moving stimuli: Task relevant information is actively singled out and processed in a dynamically changing environment.

Homozygous LAMC3 mutation links to structural and functional changes in visual attention networks.

The occipital lobe contains a substantial part of the neural machinery involved in visual perception. Mutations in the LAMC3 gene have recently been shown to cause complex bilateral occipital cortical gyration abnormalities. However, to what extent these structural changes impact visual behavior is not known. We recorded responses for two screening test batteries targeting visual function (Leuven - Perceptual Organization Screening Test, Cortical Vision Screening Test) and measured eye fixation performance in a visual attention experiment from a patient with homozygous LAMC3 gene mutation. Using voxel-based morphometry (VBM) we quantitatively assessed the extent of structural changes brought on by the genetic mutation by comparing mean cortical curvature, cortical thickness, and gray matter volume in 34 cortical areas between patient and an age-, sex-, and education-matched control group. Anatomical connectivity between these cortical areas was investigated by a structural covariance analysis. Visual screening-, and behavioral results revealed that the patient's impairments were predominantly in visuo-spatial attention. Consistent with this, VBM and structural connectivity results revealed significant structural changes in cortical regions subserving attentional functions. We conclude that the LAMC3 gene mutation affects cortical areas beyond the occipital lobe and primarily those visual functions that involve heavily distributed networks - such as visuo-spatial attention.

Abnormal subcortical activity in congenital mirror movement disorder with RAD51 mutation.

PURPOSE: Congenital mirror movement disorder (CMMD) is characterized by unintended, nonsuppressible, homologous mirroring activity contralateral to the movement on the intended side of the body. In healthy controls, unilateral movements are accompanied with predominantly contralateral cortical activity, whereas in CMMD, in line with the abnormal behavior, bilateral cortical activity is observed for unilateral motor tasks. However, task-related activities in subcortical structures, which are known to play critical roles in motor actions, have not been investigated in CMMD previously. METHODS: We investigated the functional activation patterns of the motor components in CMMD patients. By using linkage analysis and exome sequencing, common mutations were revealed in seven affected individuals from the same family. Next, using functional magnetic resonance imaging (fMRI) we investigated cortical and subcortical activity during manual motor actions in two right-handed affected brothers and sex, age, education, and socioeconomically matched healthy individuals. RESULTS: Genetic analyses revealed heterozygous RAD51 c.401C>T mutation which cosegregated with the phenotype in two affected members of the family. Consistent with previous literature, our fMRI results on these two affected individuals showed that mirror movements were closely related to abnormal cortical activity in M1 and SMA during unimanual movements. Furthermore, we have found previously unknown abnormal task-related activity in subcortical structures. Specifically, we have found increased and bilateral activity during unimanual movements in thalamus, striatum, and globus pallidus in CMMD patients. CONCLUSION: These findings reveal further neural correlates of CMMD, and may guide our understanding of the critical roles of subcortical structures for unimanual movements in healthy individuals.

Seeing through transparent layers.

The human visual system is remarkably good at decomposing local and global deformations in the flow of visual information into different perceptual layers, a critical ability for daily tasks such as driving through rain or fog or catching that evasive trout. In these scenarios, changes in the visual information might be due to a deforming object or deformations due to a transparent medium, such as structured glass or water, or a combination of these. How does the visual system use image deformations to make sense of layering due to transparent materials? We used eidolons to investigate equivalence classes for perceptually similar transparent layers. We created a stimulus space for perceptual equivalents of a fiducial scene by systematically varying the local disarray parameters reach and grain. This disarray in eidolon space leads to distinct impressions of transparency, specifically, high reach and grain values vividly resemble water whereas smaller grain values appear diffuse like structured glass. We asked observers to adjust image deformations so that the objects in the scene looked like they were seen (a) under water, (b) behind haze, or (c) behind structured glass. Observers adjusted image deformation parameters by moving the mouse horizontally (grain) and vertically (reach). For two conditions, water and glass, we observed high intraobserver consistency: responses were not random. Responses yielded a concentrated equivalence class for water and structured glass.

Shatter and splatter: The contribution of mechanical and optical properties to the perception of soft and hard breaking materials.

Research on the visual perception of materials has mostly focused on the surface qualities of rigid objects. The perception of substance like materials is less explored. Here, we investigated the contribution of, and interaction between, surface optics and mechanical properties to the perception of nonrigid, breaking materials. We created novel animations of materials ranging from soft to hard bodies that broke apart differently when dropped. In Experiment 1, animations were rendered as point-light movies varying in dot density, as well as "full-cue" optical versions ranging from translucent glossy to opaque matte under a natural illumination field. Observers used a scale to rate each substance on different attributes. In Experiment 2 we investigated how much shape contributed to ratings of the full-cue stimuli in Experiment 1, by comparing ratings when observers were shown movies versus one frame of the animation. The results showed that optical and mechanical properties had an interactive effect on ratings of several material attributes. We also found that motion and static cues each provided a lot of information about the material qualities; however, when combined, they influenced observers' ratings interactively. For example, in some conditions, motion dominated over optical information; in other conditions, it enhanced the effect of optics. Our results suggest that rating multiple attributes is an effective way to measure underlying perceptual differences between nonrigid breaking materials, and this study is the first to our knowledge to show interactions between optical and mechanical properties in a task involving judgments of perceptual qualities.

Specular motion and 3D shape estimation.

Dynamic visual information facilitates three-dimensional shape recognition. It is still unclear, however, whether the motion information generated by moving specularities across a surface is congruent to that available from optic flow produced by a matte-textured shape. Whereas the latter is directly linked to the first-order properties of the shape and its motion relative to the observer, the specular flow, the image flow generated by a specular object, is less sensitive to the object's motion and is tightly related to second-order properties of the shape. We therefore hypothesize that the perceived bumpiness (a perceptual attribute related to curvature magnitude) is more stable to changes in the type of motion in specular objects compared with their matte-textured counterparts. Results from two two-interval forced-choice experiments in which observers judged the perceived bumpiness of perturbed spherelike objects support this idea and provide an additional layer of evidence for the capacity of the visual system to exploit image information for shape inference.