Gaze and Arrows: does the Gaze Following Patch in the posterior temporal cortex differentiate social and symbolic spatial cues?

The Gaze Following Patch (GFP) is located in the posterior temporal cortex and has been described as a cortical module dedicated to processing other people's gaze direction in a domain-specific manner. Thus, it appears to be the neural correlate of Baron-Cohen's Eye-Direction Detector (EDD) which is one of the core modules in his Mindreading System - a neurocognitive model for the Theory of Mind concept. Inspired by Jerry Fodor's ideas on the modularity of the mind, Baron-Cohen proposed that, among other things, the individual modules are domain-specific. In the case of the EDD this means that it exclusively processes eye-like stimuli to extract gaze direction and that other stimuli, that may carry directional information as well, are processed elsewhere. If the GFP is indeed EDD's neural correlate it must meet this expectation. To test this, we compared the GFP's BOLD activity during gaze-direction following with the activity during arrow-direction following in the present human fMRI study. Contrary to the expectation based on the assumption of domain specificity we did not find a differentiation between gaze and arrow-direction following. In fact, we were not able to reproduce the GFP as presented in previous studies. A possible explanation is that in the present study - unlike previous work -, the gaze stimuli did not contain an obvious change of direction that represented a visual motion. Hence, the critical stimulus component responsible for the identification of the GFP in previous experiments might have been visual motion.Significance Statement This study presents evidence against the notion of domain-specificity of an area in the posterior temporal cortex (the gaze-following-patch; GFP) previously described to specifically serve eye gaze following. This conclusion is suggested by the finding that using arrows to identify a target object among distractors is accompanied by a comparable or even larger BOLD response than when the participants are asked to use the gaze direction of a demonstrator face for target selection. The fact that even the best candidate to date, the posterior temporal GFP, does not stand up to critical scrutiny casts doubt on the assumption that the brain uses a specific module to enable gaze following, as proposed by Simon Baron-Cohen.

Nonhuman primates exploit the prior assumption that the visual world is vertical.

When human subjects tilt their heads in dark surroundings, the noisiness of vestibular information impedes precise reports on objects' orientation with respect to Earth's vertical axis. This difficulty is mitigated if a vertical visual background is available. Tilted visual backgrounds induce feelings of head tilt in subjects who are in fact upright. This is often explained as a result of the brain resorting to the prior assumption that natural visual backgrounds are vertical. Here, we tested whether monkeys show comparable perceptual mechanisms. To this end we trained two monkeys to align a visual arrow to a vertical reference line that had variable luminance across trials, while including a large, clearly visible background square whose orientation changed from trial to trial. On ∼20% of all trials, the vertical reference line was left out to measure the subjective visual vertical (SVV). When the frame was upright, the monkeys' SVV was aligned with the gravitational vertical. In accordance with the perceptual reports of humans, however, when the frame was tilted it induced an illusion of head tilt as indicated by a bias in SVV toward the frame orientation. Thus all primates exploit the prior assumption that the visual world is vertical.NEW & NOTEWORTHY Here we show that the principles that characterize the human perception of the vertical are shared by another old world primate species, the rhesus monkey, suggesting phylogenetic continuity. In both species the integration of visual and vestibular information on the orientation of the head relative to the world is similarly constrained by the prior assumption that the visual world is vertical in the sense of having an orientation that is congruent with the gravity vector.

Resilience of FEF neuronal saccade code to V4 perturbations.

The frontal eye field (FEF) plays a key role in initiating rapid eye movements known as saccades. Accumulation models have been proposed to explain the dynamic of these neurons and how they may enable the initiation of saccades. To update the scope of the viability of this model, we studied single neurons recorded from the FEF of two rhesus monkeys while they performed a memory-guided saccade task. We evaluated the degree to which each type of FEF neuron complied with these models by quantifying how precisely their discharge predicted an imminent saccade based on their immediate presaccadic activity. We found that decoders trained on single neurons with a stronger motor component performed better than decoders trained on neurons with a stronger visual component in predicting the saccade. Importantly, despite a dramatic effect on the reaction times, the perturbations delivered to the FEF neurons via area V4 did not impact their saccade predictability. Our results demonstrate a high degree of resilience of the FEF neuronal presaccadic discharge patterns, fulfilling the predictions of accumulation models.NEW & NOTEWORTHY We studied neurons in the brain's frontal eye field (FEF) to understand how these neurons predict swift eye shifts called saccades. We found that neurons with more movement-related activity were better at predicting saccades than those with sensory-related activity. Interestingly, electrical disruptions of this region strongly impacted saccade onset times but did not affect the individual neuron's saccade predictability, consistent with models suggesting that a specific threshold in neural activity triggers the saccade.

Non-shared coding of observed and executed actions prevails in macaque ventral premotor mirror neurons.

According to the mirror mechanism the discharge of F5 mirror neurons of a monkey observing another individual performing an action is a motor representation of the observed action that may serve to understand or learn from the action. This hypothesis, if strictly interpreted, requires mirror neurons to exhibit an action tuning that is shared between action observation and execution. Due to insufficient data it remains contentious if this requirement is met. To fill in the gaps, we conducted an experiment in which identical objects had to be manipulated in three different ways in order to serve distinct action goals. Using three methods, including cross-task classification, we found that at most time points F5 mirror neurons did not encode observed actions with the same code underlying action execution. However, in about 20% of neurons there were time periods with a shared code. These time periods formed a distinct cluster and cannot be considered a product of chance. Population classification yielded non-shared coding for observed actions in the whole population, which was at times optimal and consistently better than shared coding in differentially selected subpopulations. These results support the hypothesis of a representation of observed actions based on a strictly defined mirror mechanism only for small subsets of neurons and only under the assumption of time-resolved readout. Considering alternative concepts and recent findings, we propose that during observation mirror neurons represent the process of a goal pursuit from the observer's viewpoint. Whether the observer's goal pursuit, in which the other's action goal becomes the observer's action goal, or the other's goal pursuit is represented remains to be clarified. In any case, it may allow the observer to use expectations associated with a goal pursuit to directly intervene in or learn from another's action.

Causal manipulation of gaze-following in the macaque temporal cortex.

Gaze-following, the ability to shift one's own attention to places or objects others are looking at, is essential for social interactions. Single unit recordings from the monkey cortex and neuroimaging work on the human and monkey brain suggest that a distinct region in the temporal cortex, the gaze-following patch (GFP), underpins this ability. Since previous studies of the GFP have relied on correlational techniques, it remains unclear whether gaze-following related activity in the GFP indicates a causal role rather than being just a reverberation of behaviorally relevant information produced elsewhere. To answer this question, we applied focal electrical and pharmacological perturbation to the GFP. Both approaches, when applied to the GFP, disrupted gaze-following if the monkeys had been instructed to follow gaze, along with the ability to suppress it if vetoed by the context. Hence the GFP is necessary for gaze-following as well as its cognitive control.

Multidimensional cerebellar computations for flexible kinematic control of movements.

Both the environment and our body keep changing dynamically. Hence, ensuring movement precision requires adaptation to multiple demands occurring simultaneously. Here we show that the cerebellum performs the necessary multi-dimensional computations for the flexible control of different movement parameters depending on the prevailing context. This conclusion is based on the identification of a manifold-like activity in both mossy fibers (MFs, network input) and Purkinje cells (PCs, output), recorded from monkeys performing a saccade task. Unlike MFs, the PC manifolds developed selective representations of individual movement parameters. Error feedback-driven climbing fiber input modulated the PC manifolds to predict specific, error type-dependent changes in subsequent actions. Furthermore, a feed-forward network model that simulated MF-to-PC transformations revealed that amplification and restructuring of the lesser variability in the MF activity is a pivotal circuit mechanism. Therefore, the flexible control of movements by the cerebellum crucially depends on its capacity for multi-dimensional computations.

A prosocial function of head-gaze aversion and head-cocking in common marmosets.

Gaze aversion is a behavior adopted by several mammalian and non-mammalian species in response to eye contact, and is usually interpreted as a reaction to a perceived threat. Unlike many other primate species, common marmosets (Callithrix jacchus) are thought to have a high tolerance for direct gaze, barely exhibiting gaze avoidance towards conspecifics and humans. Here we show that this does not hold for marmosets interacting with a familiar experimenter who suddenly establishes eye contact in a playful interaction (peekaboo). Video footage synchronously recorded from the perspective of the marmoset and the experimenter showed that the monkeys consistently alternated between eye contact and head-gaze aversion, and that these responses were often preceded by head-cocking. We hypothesize that this behavioral strategy helps marmosets to temporarily disengage from emotionally overwhelming social stimulation due to sight of another individual's face, in order to prepare for a new round of affiliative face-to-face interactions.

Variability of neuronal responses in the posterior superior temporal sulcus predicts choice behavior during social interactions.

Recent studies have shown that neural activity in a well-defined patch in the posterior superior temporal sulcus (the "gaze-following patch," GFP) of the primate brain is strongly modulated when the other's gaze attracts the observer's attention to locations/objects, the other is looking at. Changes of the mean discharge rate of neurons in the monkey GFP indicate that they are involved in two distinct computations: the allocation of spatial attention guided by the other's gaze vector and the suppression of gaze following if inappropriate in a given situation. Here, we asked if and how the discharge variability of neurons in the GFP is related to the task and if it carries information on behavioral performance. To this end, we calculated the Fano factor as a measure of across-trial discharge variability as a function of time. Our results show that all neurons exhibiting a task-related discharge-rate modulation also exhibit a stimulus onset-dependent drop in the Fano factor. Furthermore, the amplitude of the Fano factor reduction is modulated by task condition and the neuron's selectivity in this regard. We found that these effects are directly related to the monkeys' behavioral performance in that the Fano factor is predictive about upcoming correct or wrong decisions. Our results indicate that neuronal discharge variability as gauged by the Fano factor, hitherto primarily studied in the context of visual perception or motor control, is an informative measure also in studies of the neural underpinnings of complex social behavior.NEW & NOTEWORTHY Quenching of neural variability following stimulus onset is a widely accepted phenomenon. However, the relevance of quenching for the shaping of complex social behaviors remains to be explored. Here, we show that task selective neurons in the GFP exhibit a higher degree of variability quenching than their neighboring unselective neurons. Furthermore, we demonstrate that behavioral errors are not only associated with lower firing rates but also less variability quenching, suggesting that both facilitate optimal performance.

Dark-habituation increases the dark-background-contingent upshift of gaze in macaque monkeys.

What is the effect of prior experience on sensorimotor behavior? We studied the following intriguing behavior: monkeys fixating a small target direct their gaze above the target if the background is dark. Fixating a target once on a bright background, then on a dark background, yields 2 gaze directions, typically one above the other; hence the name, 'dark-background-contingent upshift of gaze', which is abbreviated as 'upshift'. Is the upshift only an attempt to avoid using the fovea in the dark? If it is, we might expect to also observe a downshift and a sideshift. We studied gaze direction in a large group of 10 rhesus monkeys from Tübingen, to which we added published data from 4 cynomolgus monkeys from Rehovot. The upshift was ubiquitous, and there was no systematic sideshift. What is the function of the upshift? Is it related to vision in the dark? Here, we concentrate on the effect of the monkeys' previous training. Seven of the 14 monkeys were accustomed to working in the dark ('dark-habituated'), while the other 7 had worked in bright ambient light ('bright-habituated'). The main result of this study is that the dark-habituated monkeys had a much larger upshift: the mean upshift was 2.2° in the dark-habituated monkeys, versus 0.5° in the bright-habituated. Thus, the upshift depends on habituation; the size of the upshift reflects months-long cumulative experience. These findings suggest that the function of the upshift is indeed related to seeing in the dark.

Cerebellar complex spikes multiplex complementary behavioral information.

Purkinje cell (PC) discharge, the only output of cerebellar cortex, involves 2 types of action potentials, high-frequency simple spikes (SSs) and low-frequency complex spikes (CSs). While there is consensus that SSs convey information needed to optimize movement kinematics, the function of CSs, determined by the PC's climbing fiber input, remains controversial. While initially thought to be specialized in reporting information on motor error for the subsequent amendment of behavior, CSs seem to contribute to other aspects of motor behavior as well. When faced with the bewildering diversity of findings and views unraveled by highly specific tasks, one may wonder if there is just one true function with all the other attributions wrong? Or is the diversity of findings a reflection of distinct pools of PCs, each processing specific streams of information conveyed by climbing fibers? With these questions in mind, we recorded CSs from the monkey oculomotor vermis deploying a repetitive saccade task that entailed sizable motor errors as well as small amplitude saccades, correcting them. We demonstrate that, in addition to carrying error-related information, CSs carry information on the metrics of both primary and small corrective saccades in a time-specific manner, with changes in CS firing probability coupled with changes in CS duration. Furthermore, we also found CS activity that seemed to predict the upcoming events. Hence PCs receive a multiplexed climbing fiber input that merges complementary streams of information on the behavior, separable by the recipient PC because they are staggered in time.

Shape-invariant encoding of dynamic primate facial expressions in human perception.

Dynamic facial expressions are crucial for communication in primates. Due to the difficulty to control shape and dynamics of facial expressions across species, it is unknown how species-specific facial expressions are perceptually encoded and interact with the representation of facial shape. While popular neural network models predict a joint encoding of facial shape and dynamics, the neuromuscular control of faces evolved more slowly than facial shape, suggesting a separate encoding. To investigate these alternative hypotheses, we developed photo-realistic human and monkey heads that were animated with motion capture data from monkeys and humans. Exact control of expression dynamics was accomplished by a Bayesian machine-learning technique. Consistent with our hypothesis, we found that human observers learned cross-species expressions very quickly, where face dynamics was represented largely independently of facial shape. This result supports the co-evolution of the visual processing and motor control of facial expressions, while it challenges appearance-based neural network theories of dynamic expression recognition.

Sensitivity of express saccades to the expected value of the target.

Express saccades, a distinct fast mode of visually guided saccades, are probably underpinned by a specific pathway that is at least partially different from the one underlying regular saccades. Whether and how this pathway deals with information on the subjective value of a saccade target is unknown. We studied the influence of varying reward expectancies and compared it with the impact of a temporal gap between the disappearance of the fixation dot and the appearance of the target on the visually guided saccades of two rhesus macaques (Macaca mulatta). We found that increasing reward expectancy increased the probability and decreased the reaction time of express saccades. The latter influence was stronger in the later parts of the reaction time distribution of express saccades, satisfactorily captured by a linear shift model of change in the saccadic reaction time distribution. Although different in strength, increasing reward expectancy and inserting a temporal gap resulted in similar effects on saccadic reaction times, suggesting that these two factors summon the same mechanism to facilitate saccadic reaction times.NEW & NOTEWORTHY Express saccades are the fastest visually driven way of shifting gaze to targets of interest. We examined whether the pathway underlying these saccades has access to information on the value of saccade targets. We found that not only regular saccades but also express saccades occur earlier in case of higher expectations of reward. Yet, the sensitivity of express saccades to reward decreases linearly when approaching the earliest possible reaction time.

Representation of the observer's predicted outcome value in mirror and nonmirror neurons of macaque F5 ventral premotor cortex.

In the search for the function of mirror neurons, a previous study reported that F5 mirror neuron responses are modulated by the value that the observing monkey associates with the grasped object. Yet we do not know whether mirror neurons are modulated by the expected reward value for the observer or also by other variables, which are causally dependent on value (e.g., motivation, attention directed at the observed action, arousal). To clarify this, we trained two rhesus macaques to observe a grasping action on an object kept constant, followed by four fully predictable outcomes of different values (2 outcomes with positive and 2 with negative emotional valence). We found a consistent order in population activity of both mirror and nonmirror neurons that matches the order of the value of this predicted outcome but that does not match the order of the above-mentioned value-dependent variables. These variables were inferred from the probability not to abort a trial, saccade latency, modulation of eye position during action observation, heart rate, and pupil size. Moreover, we found subpopulations of neurons tuned to each of the four predicted outcome values. Multidimensional scaling revealed equal normalized distances of 0.25 between the two positive and between the two negative outcomes suggesting the representation of a relative value, scaled to the task setting. We conclude that F5 mirror neurons and nonmirror neurons represent the observer's predicted outcome value, which in the case of mirror neurons may be transferred to the observed object or action.NEW & NOTEWORTHY Both the populations of F5 mirror neurons and nonmirror neurons represent the predicted value of an outcome resulting from the observation of a grasping action. Value-dependent motivation, arousal, and attention directed at the observed action do not provide a better explanation for this representation. The population activity's metric suggests an optimal scaling of value representation to task setting.

V1 neurons encode the perceptual compensation of false torsion arising from Listing's law.

We try to deploy the retinal fovea to optimally scrutinize an object of interest by directing our eyes to it. The horizontal and vertical components of eye positions acquired by goal-directed saccades are determined by the object's location. However, the eccentric eye positions also involve a torsional component, which according to Donder's law is fully determined by the two-dimensional (2D) eye position acquired. According to von Helmholtz, knowledge of the amount of torsion provided by Listing's law, an extension of Donder's law, alleviates the perceptual interpretation of the image tilt that changes with 2D eye position, a view supported by psychophysical experiments he pioneered. We address the question of where and how Listing's law is implemented in the visual system and we show that neurons in monkey area V1 use knowledge of eye torsion to compensate the image tilt associated with specific eye positions as set by Listing's law.

A Naturalistic Dynamic Monkey Head Avatar Elicits Species-Typical Reactions and Overcomes the Uncanny Valley.

Research on social perception in monkeys may benefit from standardized, controllable, and ethologically valid renditions of conspecifics offered by monkey avatars. However, previous work has cautioned that monkeys, like humans, show an adverse reaction toward realistic synthetic stimuli, known as the "uncanny valley" effect. We developed an improved naturalistic rhesus monkey face avatar capable of producing facial expressions (fear grin, lip smack and threat), animated by motion capture data of real monkeys. For validation, we additionally created decreasingly naturalistic avatar variants. Eight rhesus macaques were tested on the various videos and avoided looking at less naturalistic avatar variants, but not at the most naturalistic or the most unnaturalistic avatar, indicating an uncanny valley effect for the less naturalistic avatar versions. The avoidance was deepened by motion and accompanied by physiological arousal. Only the most naturalistic avatar evoked facial expressions comparable to those toward the real monkey videos. Hence, our findings demonstrate that the uncanny valley reaction in monkeys can be overcome by a highly naturalistic avatar.

Using deep neural networks to detect complex spikes of cerebellar Purkinje cells.

One of the most powerful excitatory synapses in the brain is formed by cerebellar climbing fibers, originating from neurons in the inferior olive, that wrap around the proximal dendrites of cerebellar Purkinje cells. The activation of a single olivary neuron is capable of generating a large electrical event, called "complex spike," at the level of the postsynaptic Purkinje cell, comprising of an initial large-amplitude spike followed by a long polyphasic tail of small-amplitude spikelets. Several ideas discussing the role of the cerebellum in motor control are centered on these complex spike events. However, these events, only occurring one to two times per second, are extremely rare relative to Purkinje cell "simple spikes" (standard sodium-potassium action potentials). As a result, drawing conclusions about their functional role has been very challenging. In fact, because standard spike sorting approaches cannot fully handle the polyphasic shape of complex spike waveforms, the only safe way to avoid omissions and false detections has been to rely on visual inspection by experts, which is both tedious and, because of attentional fluctuations, error prone. Here we present a deep learning algorithm for rapidly and reliably detecting complex spikes. Our algorithm, utilizing both action potential and local field potential signals, not only detects complex spikes much faster than human experts, but it also reliably provides complex spike duration measures similar to those of the experts. A quantitative comparison of our algorithm's performance to both classic and novel published approaches addressing the same problem reveals that it clearly outperforms these approaches.NEW & NOTEWORTHY Purkinje cell "complex spikes", fired at perplexingly low rates, play a crucial role in cerebellum-based motor learning. Careful interpretations of these spikes require manually detecting them, since conventional online or offline spike sorting algorithms are optimized for classifying much simpler waveform morphologies. We present a novel deep learning approach for identifying complex spikes, which also measures additional relevant neurophysiological features, with an accuracy level matching that of human experts yet with very little time expenditure.

Decoding of the other's focus of attention by a temporal cortex module.

Faces attract the observer's attention toward objects and locations of interest for the other, thereby allowing the two agents to establish joint attention. Previous work has delineated a network of cortical "patches" in the macaque cortex, processing faces, eventually also extracting information on the other's gaze direction. Yet, the neural mechanism that links information on gaze direction, guiding the observer's attention to the relevant object, has remained elusive. Here we present electrophysiological evidence for the existence of a distinct "gaze-following patch" (GFP) with neurons that establish this linkage in a highly flexible manner. The other's gaze and the object, singled out by the gaze, are linked only if this linkage is pertinent within the prevailing social context. The properties of these neurons establish the GFP as a key switch in controlling social interactions based on the other's gaze.