Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks.

Primates can recognize objects despite 3D geometric variations such as in-depth rotations. The computational mechanisms that give rise to such invariances are yet to be fully understood. A curious case of partial invariance occurs in the macaque face-patch AL and in fully connected layers of deep convolutional networks in which neurons respond similarly to mirror-symmetric views (e.g. left and right profiles). Why does this tuning develop? Here, we propose a simple learning-driven explanation for mirror-symmetric viewpoint tuning. We show that mirror-symmetric viewpoint tuning for faces emerges in the fully connected layers of convolutional deep neural networks trained on object recognition tasks, even when the training dataset does not include faces. First, using 3D objects rendered from multiple views as test stimuli, we demonstrate that mirror-symmetric viewpoint tuning in convolutional neural network models is not unique to faces: it emerges for multiple object categories with bilateral symmetry. Second, we show why this invariance emerges in the models. Learning to discriminate among bilaterally symmetric object categories induces reflection-equivariant intermediate representations. AL-like mirror-symmetric tuning is achieved when such equivariant responses are spatially pooled by downstream units with sufficiently large receptive fields. These results explain how mirror-symmetric viewpoint tuning can emerge in neural networks, providing a theory of how they might emerge in the primate brain. Our theory predicts that mirror-symmetric viewpoint tuning can emerge as a consequence of exposure to bilaterally symmetric objects beyond the category of faces, and that it can generalize beyond previously experienced object categories.

Deep neural networks are not a single hypothesis but a language for expressing computational hypotheses.

An ideal vision model accounts for behavior and neurophysiology in both naturalistic conditions and designed lab experiments. Unlike psychological theories, artificial neural networks (ANNs) actually perform visual tasks and generate testable predictions for arbitrary inputs. These advantages enable ANNs to engage the entire spectrum of the evidence. Failures of particular models drive progress in a vibrant ANN research program of human vision.

Emergence of brain-like mirror-symmetric viewpoint tuning in convolutional neural networks.

Primates can recognize objects despite 3D geometric variations such as in-depth rotations. The computational mechanisms that give rise to such invariances are yet to be fully understood. A curious case of partial invariance occurs in the macaque face-patch AL and in fully connected layers of deep convolutional networks in which neurons respond similarly to mirror-symmetric views (e.g., left and right profiles). Why does this tuning develop? Here, we propose a simple learning-driven explanation for mirror-symmetric viewpoint tuning. We show that mirror-symmetric viewpoint tuning for faces emerges in the fully connected layers of convolutional deep neural networks trained on object recognition tasks, even when the training dataset does not include faces. First, using 3D objects rendered from multiple views as test stimuli, we demonstrate that mirror-symmetric viewpoint tuning in convolutional neural network models is not unique to faces: it emerges for multiple object categories with bilateral symmetry. Second, we show why this invariance emerges in the models. Learning to discriminate among bilaterally symmetric object categories induces reflection-equivariant intermediate representations. AL-like mirror-symmetric tuning is achieved when such equivariant responses are spatially pooled by downstream units with sufficiently large receptive fields. These results explain how mirror-symmetric viewpoint tuning can emerge in neural networks, providing a theory of how they might emerge in the primate brain. Our theory predicts that mirror-symmetric viewpoint tuning can emerge as a consequence of exposure to bilaterally symmetric objects beyond the category of faces, and that it can generalize beyond previously experienced object categories.

Face dissimilarity judgments are predicted by representational distance in morphable and image-computable models.

Human vision is attuned to the subtle differences between individual faces. Yet we lack a quantitative way of predicting how similar two face images look and whether they appear to show the same person. Principal component-based three-dimensional (3D) morphable models are widely used to generate stimuli in face perception research. These models capture the distribution of real human faces in terms of dimensions of physical shape and texture. How well does a "face space" based on these dimensions capture the similarity relationships humans perceive among faces? To answer this, we designed a behavioral task to collect dissimilarity and same/different identity judgments for 232 pairs of realistic faces. Stimuli sampled geometric relationships in a face space derived from principal components of 3D shape and texture (Basel face model [BFM]). We then compared a wide range of models in their ability to predict the data, including the BFM from which faces were generated, an active appearance model derived from face photographs, and image-computable models of visual perception. Euclidean distance in the BFM explained both dissimilarity and identity judgments surprisingly well. In a comparison against 16 diverse models, BFM distance was competitive with representational distances in state-of-the-art deep neural networks (DNNs), including novel DNNs trained on BFM synthetic identities or BFM latents. Models capturing the distribution of face shape and texture across individuals are not only useful tools for stimulus generation. They also capture important information about how faces are perceived, suggesting that human face representations are tuned to the statistical distribution of faces.

Self-supervised Natural Image Reconstruction and Large-scale Semantic Classification from Brain Activity.

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs of images and their corresponding fMRI responses, which span the huge space of natural images, is prohibitive. We present a novel self-supervised approach that goes well beyond the scarce paired data, for achieving both: (i) state-of-the art fMRI-to-image reconstruction, and (ii) first-ever large-scale semantic classification from fMRI responses. By imposing cycle consistency between a pair of deep neural networks (from image-to-fMRI & from fMRI-to-image), we train our image reconstruction network on a large number of "unpaired" natural images (images without fMRI recordings) from many novel semantic categories. This enables to adapt our reconstruction network to a very rich semantic coverage without requiring any explicit semantic supervision. Specifically, we find that combining our self-supervised training with high-level perceptual losses, gives rise to new reconstruction & classification capabilities. In particular, this perceptual training enables to classify well fMRIs of never-before-seen semantic classes, without requiring any class labels during training. This gives rise to: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing), and (ii) Large-scale semantic classification of categories that were never-before-seen during network training. Such large-scale (1000-way) semantic classification from fMRI recordings has never been demonstrated before. Finally, we provide evidence for the biological consistency of our learned model.

Controversial stimuli: Pitting neural networks against each other as models of human cognition.

Distinct scientific theories can make similar predictions. To adjudicate between theories, we must design experiments for which the theories make distinct predictions. Here we consider the problem of comparing deep neural networks as models of human visual recognition. To efficiently compare models' ability to predict human responses, we synthesize controversial stimuli: images for which different models produce distinct responses. We applied this approach to two visual recognition tasks, handwritten digits (MNIST) and objects in small natural images (CIFAR-10). For each task, we synthesized controversial stimuli to maximize the disagreement among models which employed different architectures and recognition algorithms. Human subjects viewed hundreds of these stimuli, as well as natural examples, and judged the probability of presence of each digit/object category in each image. We quantified how accurately each model predicted the human judgments. The best-performing models were a generative analysis-by-synthesis model (based on variational autoencoders) for MNIST and a hybrid discriminative-generative joint energy model for CIFAR-10. These deep neural networks (DNNs), which model the distribution of images, performed better than purely discriminative DNNs, which learn only to map images to labels. None of the candidate models fully explained the human responses. Controversial stimuli generalize the concept of adversarial examples, obviating the need to assume a ground-truth model. Unlike natural images, controversial stimuli are not constrained to the stimulus distribution models are trained on, thus providing severe out-of-distribution tests that reveal the models' inductive biases. Controversial stimuli therefore provide powerful probes of discrepancies between models and human perception.

Neural network models and deep learning.

Originally inspired by neurobiology, deep neural network models have become a powerful tool of machine learning and artificial intelligence. They can approximate functions and dynamics by learning from examples. Here we give a brief introduction to neural network models and deep learning for biologists. We introduce feedforward and recurrent networks and explain the expressive power of this modeling framework and the backpropagation algorithm for setting the parameters. Finally, we consider how deep neural network models might help us understand brain computation.

Resting-State Activity in High-Order Visual Areas as a Window into Natural Human Brain Activations.

A major limitation of conventional human brain research has been its basis in highly artificial laboratory experiments. Due to technical constraints, little is known about the nature of cortical activations during ecological real life. We have previously proposed the "spontaneous trait reactivation (STR)" hypothesis arguing that resting-state patterns, which emerge spontaneously in the absence of external stimulus, reflect the statistics of habitual cortical activations during real life. Therefore, these patterns can serve as a window into daily life cortical activity. A straightforward prediction of this hypothesis is that spontaneous patterns should preferentially correlate to patterns generated by naturalistic stimuli compared with artificial ones. Here we targeted high-level category-selective visual areas and tested this prediction by comparing BOLD functional connectivity patterns formed during rest to patterns formed in response to naturalistic stimuli, as well as to more artificial category-selective, dynamic stimuli. Our results revealed a significant correlation between the resting-state patterns and functional connectivity patterns generated by naturalistic stimuli. Furthermore, the correlations to naturalistic stimuli were significantly higher than those found between resting-state patterns and those generated by artificial control stimuli. These findings provide evidence of a stringent link between spontaneous patterns and the activation patterns during natural vision.

Increasing suppression of saccade-related transients along the human visual hierarchy.

A key hallmark of visual perceptual awareness is robustness to instabilities arising from unnoticeable eye and eyelid movements. In previous human intracranial (iEEG) work (Golan et al., 2016) we found that excitatory broadband high-frequency activity transients, driven by eye blinks, are suppressed in higher-level but not early visual cortex. Here, we utilized the broad anatomical coverage of iEEG recordings in 12 eye-tracked neurosurgical patients to test whether a similar stabilizing mechanism operates following small saccades. We compared saccades (1.3°-3.7°) initiated during inspection of large individual visual objects with similarly-sized external stimulus displacements. Early visual cortex sites responded with positive transients to both conditions. In contrast, in both dorsal and ventral higher-level sites the response to saccades (but not to external displacements) was suppressed. These findings indicate that early visual cortex is highly unstable compared to higher-level visual regions which apparently constitute the main target of stabilizing extra-retinal oculomotor influences.

Cortical representation of persistent visual stimuli.

Research into visual neural activity has focused almost exclusively on onset- or change-driven responses and little is known about how information is encoded in the brain during sustained periods of visual perception. We used intracranial recordings in humans to determine the degree to which the presence of a visual stimulus is persistently encoded by neural activity. The correspondence between stimulus duration and neural response duration was strongest in early visual cortex and gradually diminished along the visual hierarchy, such that is was weakest in inferior-temporal category-selective regions. A similar posterior-anterior gradient was found within inferior temporal face-selective regions, with posterior but not anterior sites showing persistent face-selective activity. The results suggest that regions that appear uniform in terms of their category selectivity are dissociated by how they temporally represent a stimulus in support of ongoing visual perception, and delineate a large-scale organizing principle of the ventral visual stream.

Spontaneously Emerging Patterns in Human Visual Cortex Reflect Responses to Naturalistic Sensory Stimuli.

In the absence of stimulus or task, the cortex spontaneously generates rich and consistent functional connectivity patterns (termed resting state networks) which are evident even within individual cortical areas. We and others have previously hypothesized that habitual cortical network activations during daily life contribute to the shaping of these connectivity patterns. Here we tested this hypothesis by comparing, using blood oxygen level-dependent-functional magnetic resonance imaging, the connectivity patterns that spontaneously emerge during rest in retinotopic visual areas to the patterns generated by naturalistic visual stimuli (repeated movie segments). These were then compared with connectivity patterns produced by more standard retinotopic mapping stimuli (polar and eccentricity mapping). Our results reveal that the movie-driven patterns were significantly more similar to the spontaneously emerging patterns, compared with the connectivity patterns of either eccentricity or polar mapping stimuli. Intentional visual imagery of naturalistic stimuli was unlikely to underlie these results, since they were duplicated when participants were engaged in an auditory task. Our results suggest that the connectivity patterns that appear during rest better reflect naturalistic activations rather than controlled, artificially designed stimuli. The results are compatible with the hypothesis that the spontaneous connectivity patterns in human retinotopic areas reflect the statistics of cortical coactivations during natural vision.

Human intracranial recordings link suppressed transients rather than 'filling-in' to perceptual continuity across blinks.

We hardly notice our eye blinks, yet an externally generated retinal interruption of a similar duration is perceptually salient. We examined the neural correlates of this perceptual distinction using intracranially measured ECoG signals from the human visual cortex in 14 patients. In early visual areas (V1 and V2), the disappearance of the stimulus due to either invisible blinks or salient blank video frames ('gaps') led to a similar drop in activity level, followed by a positive overshoot beyond baseline, triggered by stimulus reappearance. Ascending the visual hierarchy, the reappearance-related overshoot gradually subsided for blinks but not for gaps. By contrast, the disappearance-related drop did not follow the perceptual distinction - it was actually slightly more pronounced for blinks than for gaps. These findings suggest that blinks' limited visibility compared with gaps is correlated with suppression of blink-related visual activity transients, rather than with "filling-in" of the occluded content during blinks.

Intracranial recordings reveal transient response dynamics during information maintenance in human cerebral cortex.

Despite an extensive body of work, it is still not clear how short term maintenance of information is implemented in the human brain. Most prior research has focused on "working memory"-typically involving the storage of a number of items, requiring the use of a phonological loop and focused attention during the delay period between encoding and retrieval. These studies largely support a model of enhanced activity in the delay interval as the central mechanism underlying working memory. However, multi-item working memory constitutes only a subset of storage phenomena that may occur during daily life. A common task in naturalistic situations is short term memory of a single item-for example, blindly reaching to a previously placed cup of coffee. Little is known about such single-item, effortless, storage in the human brain. Here, we examined the dynamics of brain responses during a single-item maintenance task, using intracranial recordings implanted for clinical purpose in patients (ECoG). Our results reveal that active electrodes were dominated by transient short latency visual and motor responses, reflected in broadband high frequency power increases in occipito-temporal, frontal, and parietal cortex. Only a very small set of electrodes showed activity during the early part of the delay period. Interestingly, no cortical site displayed a significant activation lasting to the response time. These results suggest that single item encoding is characterized by transient high frequency ECoG responses, while the maintenance of information during the delay period may be mediated by mechanisms necessitating only low-levels of neuronal activations.

Association and dissociation between detection and discrimination of objects of expertise: Evidence from visual search.

Expertise in face recognition is characterized by high proficiency in distinguishing between individual faces. However, faces also enjoy an advantage at the early stage of basic-level detection, as demonstrated by efficient visual search for faces among nonface objects. In the present study, we asked (1) whether the face advantage in detection is a unique signature of face expertise, or whether it generalizes to other objects of expertise, and (2) whether expertise in face detection is intrinsically linked to expertise in face individuation. We compared how groups with varying degrees of object and face expertise (typical adults, developmental prosopagnosics [DP], and car experts) search for objects within and outside their domains of expertise (faces, cars, airplanes, and butterflies) among a variable set of object distractors. Across all three groups, search efficiency (indexed by reaction time slopes) was higher for faces and airplanes than for cars and butterflies. Notably, the search slope for car targets was considerably shallower in the car experts than in nonexperts. Although the mean face slope was slightly steeper among the DPs than in the other two groups, most of the DPs' search slopes were well within the normative range. This pattern of results suggests that expertise in object detection is indeed associated with expertise at the subordinate level, that it is not specific to faces, and that the two types of expertise are distinct facilities. We discuss the potential role of experience in bridging between low-level discriminative features and high-level naturalistic categories.

Allicin purified from fresh garlic cloves induces apoptosis in colon cancer cells via Nrf2.

Allicin (diallyl thiosulfinate) is the best-known biologically active component in freshly crushed garlic extract. We developed a novel, simple method to isolate active allicin, which yielded a stable compound in aqueous solution amenable for use in in vitro and in vivo studies. We focused on the in vitro effects of allicin on cell proliferation of colon cancer cell lines HCT-116, LS174T, HT-29, and Caco-2 and assessed the underlying mechanisms. This allicin preparation exerted a time- and dose-dependent cytostatic effect on these cells at concentrations ranging from 6.2 to 310 µM. Treatment with allicin resulted in HCT-116 apoptotic cell death as demonstrated by enhanced hypodiploid DNA content, decreased levels of B-cell non-Hodgkin lymphoma-2 (Bcl-2), increased levels of bax and increased capability of releasing cytochrome c from mitochondria to the cytosol. Allicin also induced translocation of NF-E2-related factor-2 (Nrf2) to the nuclei of HCT-116 cells. Luciferase reporter gene assay showed that allicin induces Nrf2-mediated luciferase transactivation activity. SiRNA knock down of Nrf2 significantly affected the capacity of allicin to inhibit HCT-116 proliferation. These results suggest that Nrf2 mediates the allicin-induced apoptotic death of colon cancer cells.

The wide window of face detection.

Faces are detected more rapidly than other objects in visual scenes and search arrays, but the cause for this face advantage has been contested. In the present study, we found that under conditions of spatial uncertainty, faces were easier to detect than control targets (dog faces, clocks and cars) even in the absence of surrounding stimuli, making an explanation based only on low-level differences unlikely. This advantage improved with eccentricity in the visual field, enabling face detection in wider visual windows, and pointing to selective sparing of face detection at greater eccentricities. This face advantage might be due to perceptual factors favoring face detection. In addition, the relative face advantage is greater under flanked than non-flanked conditions, suggesting an additional, possibly attention-related benefit enabling face detection in groups of distracters.

The Kishon affair: science, law, and the politics of causation.

This article describes how science and law were called upon (and failed) to resolve a controversy that created a painful rift between the Israeli State and some of its elite soldiers. The controversy, which came to be known as "the Kishon affair," erupted in 2000, when veterans of an elite and secretive unit in the Israeli navy claimed that pollution in the Kishon River where they had trained and dived during their military service had been the cause of a rash of cancers. The veterans demanded that the Ministry of Defense take responsibility for their illnesses, finance their medical treatment, and support their families if they die. The military denied the causal connection between the polluted river and the veterans' cancers and rejected their demands. The dispute quickly escalated into a bitter public controversy, and a high-rank commission comprised of one of Israel's top jurists and two prominent scientists was called upon to study the disputed causal relation and reveal its true nature. However, after nearly three years of intense inquiry the jurist and the scientists reached opposing conclusions: the jurist found a causal connection while the scientists rejected it.

Human face preference in gamma-frequency EEG activity.

Previous studies demonstrated that induced EEG activity in the gamma band (iGBA) plays an important role in object recognition and is modulated by stimulus familiarity and its compatibility with pre-existent representations. In the present study we investigated the modulation of iGBA by the degree of familiarity and perceptual expertise that observers have with stimuli from different categories. Specifically, we compared iGBA in response to human faces versus stimuli which subjects are not expert with (ape faces, human hands, buildings and watches). iGBA elicited by human faces was higher and peaked earlier than that elicited by all other categories, which did not differ significantly from each other. These findings can be accounted for by two characteristics of perceptual expertise. One is the activation of a richer, stronger and, therefore, more easily accessible mental representation of human faces. The second is the more detailed perceptual processing necessary for within-category distinctions, which is the hallmark of perceptual expertise. In addition, the sensitivity of iGBA to human but not ape faces was contrasted with the face-sensitive N170-effect, which was similar for human and ape faces. In concert with previous studies, this dissociation suggests a multi-level neuronal model of face recognition, manifested by these two electrophysiological measures, discussed in this paper.

The emergence of the silent witness: the legal and medical reception of X-rays in the USA.

The late 19th-century discovery of X-rays befuddled not only the scientific world but also the medical and legal worlds. The possibility of looking into the human body as if through an open window challenged the time-honored medical monopoly over the inner cavities of the human body. Likewise, the possibility of visualizing objects unavailable to the naked eye challenged the established legal theories and practices of illustration and proof. This paper describes the reactions to those challenges by the medical and the legal professions in the USA. The two professions are treated as connected social institutions, producing ongoing negotiations through which legal doctrines affect medicine no less than scientific discoveries and medical applications affect the law. This joint analysis rewards us with a rich story about an early and overlooked chapter in X-ray history on the professionalization of radiology, the origins of defensive medicine, and the evolution of the legal theory and practice of visual evidence.