Visual perception of liquids: Insights from deep neural networks.

Visually inferring material properties is crucial for many tasks, yet poses significant computational challenges for biological vision. Liquids and gels are particularly challenging due to their extreme variability and complex behaviour. We reasoned that measuring and modelling viscosity perception is a useful case study for identifying general principles of complex visual inferences. In recent years, artificial Deep Neural Networks (DNNs) have yielded breakthroughs in challenging real-world vision tasks. However, to model human vision, the emphasis lies not on best possible performance, but on mimicking the specific pattern of successes and errors humans make. We trained a DNN to estimate the viscosity of liquids using 100.000 simulations depicting liquids with sixteen different viscosities interacting in ten different scenes (stirring, pouring, splashing, etc). We find that a shallow feedforward network trained for only 30 epochs predicts mean observer performance better than most individual observers. This is the first successful image-computable model of human viscosity perception. Further training improved accuracy, but predicted human perception less well. We analysed the network's features using representational similarity analysis (RSA) and a range of image descriptors (e.g. optic flow, colour saturation, GIST). This revealed clusters of units sensitive to specific classes of feature. We also find a distinct population of units that are poorly explained by hand-engineered features, but which are particularly important both for physical viscosity estimation, and for the specific pattern of human responses. The final layers represent many distinct stimulus characteristics-not just viscosity, which the network was trained on. Retraining the fully-connected layer with a reduced number of units achieves practically identical performance, but results in representations focused on viscosity, suggesting that network capacity is a crucial parameter determining whether artificial or biological neural networks use distributed vs. localized representations.

Visual Features in the Perception of Liquids.

Perceptual constancy-identifying surfaces and objects across large image changes-remains an important challenge for visual neuroscience [1-8]. Liquids are particularly challenging because they respond to external forces in complex, highly variable ways, presenting an enormous range of images to the visual system. To achieve constancy, the brain must perform a causal inference [9-11] that disentangles the liquid's viscosity from external factors-like gravity and object interactions-that also affect the liquid's behavior. Here, we tested whether the visual system estimates viscosity using "midlevel" features [12-14] that respond more to viscosity than other factors. Observers reported the perceived viscosity of simulated liquids ranging from water to molten glass exhibiting diverse behaviors (e.g., pouring, stirring). A separate group of observers rated the same animations for 20 midlevel 3D shape and motion features. Applying factor analysis to the feature ratings reveals that a weighted combination of four underlying factors (distribution, irregularity, rectilinearity, and dynamics) predicted perceived viscosity very well across this wide range of contexts (R2 = 0.93). Interestingly, observers unknowingly ordered their midlevel judgments according to the one common factor across contexts: variation in viscosity. Principal component analysis reveals that across the features, the first component lines up almost perfectly with the viscosity (R2 = 0.96). Our findings demonstrate that the visual system achieves constancy by representing stimuli in a multidimensional feature space-based on complementary, midlevel features-which successfully cluster very different stimuli together and tease similar stimuli apart, so that viscosity can be read out easily.

Inferring the stiffness of unfamiliar objects from optical, shape, and motion cues.

Visually inferring the stiffness of objects is important for many tasks but is challenging because, unlike optical properties (e.g., gloss), mechanical properties do not directly affect image values. Stiffness must be inferred either (a) by recognizing materials and recalling their properties (associative approach) or (b) from shape and motion cues when the material is deformed (estimation approach). Here, we investigated interactions between these two inference types. Participants viewed renderings of unfamiliar shapes with 28 materials (e.g., nickel, wax, cork). In Experiment 1, they viewed nondeformed, static versions of the objects and rated 11 material attributes (e.g., soft, fragile, heavy). The results confirm that the optical materials elicited a wide range of apparent properties. In Experiment 2, using a blue plastic material with intermediate apparent softness, the objects were subjected to physical simulations of 12 shape-transforming processes (e.g., twisting, crushing, stretching). Participants rated softness and extent of deformation. Both correlated with the physical magnitude of deformation. Experiment 3 combined variations in optical cues with shape cues. We find that optical cues completely dominate. Experiment 4 included the entire motion sequence of the deformation, yielding significant contributions of optical as well as motion cues. Our findings suggest participants integrate shape, motion, and optical cues to infer stiffness, with optical cues playing a major role for our range of stimuli.

Shape, motion, and optical cues to stiffness of elastic objects.

Nonrigid materials, such as jelly, rubber, or sponge move and deform in distinctive ways depending on their stiffness. Which cues do we use to infer stiffness? We simulated cubes of varying stiffness and optical appearance (e.g., wood, metal, wax, jelly) being subjected to two kinds of deformation: (a) a rigid cylinder pushing downwards into the cube to various extents (shape change, but little motion: shape dominant), (b) a rigid cylinder retracting rapidly from the cube (same initial shapes, differences in motion: motion dominant). Observers rated the apparent softness/hardness of the cubes. In the shape-dominant condition, ratings mainly depended on how deeply the rod penetrated the cube and were almost unaffected by the cube's intrinsic physical properties. In contrast, in the motion-dominant condition, ratings varied systematically with the cube's intrinsic stiffness, and were less influenced by the extent of the perturbation. We find that both results are well predicted by the absolute magnitude of deformation, suggesting that when asked to judge stiffness, observers resort to simple heuristics based on the amount of deformation. Softness ratings for static, unperturbed cubes varied substantially and systematically depending on the optical properties. However, when animated, the ratings were again dominated by the extent of the deformation, and the effect of optical appearance was negligible. Together, our results suggest that to estimate stiffness, the visual system strongly relies on measures of the extent to which an object changes shape in response to forces.

Influence of optical material properties on the perception of liquids.

In everyday life we encounter a wide range of liquids (e.g., water, custard, toothpaste) with distinctive optical appearances and viscosities. Optical properties (e.g., color, translucency) are physically independent of viscosity, but, based on experience with real liquids, we may associate specific appearances (e.g., water, caramel) with certain viscosities. Conversely, the visual system may discount optical properties, enabling "viscosity constancy" based primarily on the liquid's shape and motion. We investigated whether optical characteristics affect the perception of viscosity and other properties of liquids. We simulated pouring liquids with viscosities ranging from water to molten glass and rendered them with nine different optical characteristics. In Experiment 1, observers (a) adjusted a match stimulus until it had the same perceived viscosity as a test stimulus with different optical properties, and (b) rated six physical properties of the test stimuli (runniness, shininess, sliminess, stickiness, warmth, wetness). We tested moving and static stimuli. In Experiment 2, observers had to associate names with every liquid in the stimulus set. We find that observers' viscosity matches correlated strongly with the true viscosities and that optical properties had almost no effect. However, some ratings of liquid properties did show substantial interactions between viscosity and optical properties. Observers associate liquid names primarily with optical cues, although some materials are associated with a specific viscosity or combination of viscosity and optics. These results suggest viscosity is inferred primarily from shape and motion cues but that optical characteristics influence recognition of specific liquids and inference of other physical properties.

Highlight shapes and perception of gloss for real and photographed objects.

Gloss perception strongly depends on the three-dimensional shape and the illumination of the object under consideration. In this study we investigated the influence of the spatial structure of the illumination on gloss perception. A diffuse light box in combination with differently shaped masks was used to produce a set of six simple and complex highlight shapes. The geometry of the simple highlight shapes was inspired by conventional artistic practice (e.g., ring flash for photography, window shape for painting and disk or square for cartoons). In the box we placed spherical stimuli that were painted in six degrees of glossiness. This resulted in a stimulus set of six highlight shapes and six gloss levels, a total of 36 stimuli. We performed three experiments of which two took place using digital photographs on a computer monitor and one with the real spheres in the light box. The observers had to perform a comparison task in which they chose which of two stimuli was glossiest and a rating task in which they rated the glossiness. The results show that, perhaps surprisingly, more complex highlight shapes were perceived to produce a less glossy appearance than simple highlight shapes such as a disk or square. These findings were confirmed for both viewing conditions, on a computer display and in a real setting. The results show that variations in the spatial structure of "rather simple" illumination of the "extended source" type highlight influences perceived glossiness.