From Prior Information to Saccade Selection: Evolution of Frontal Eye Field Activity during Natural Scene Search.

Prior knowledge about our environment influences our actions. How does this knowledge evolve into a final action plan and how does the brain represent this? Here, we investigated this question in the monkey oculomotor system during self-guided search of natural scenes. In the frontal eye field (FEF), we found a subset of neurons, "Early neurons," that contain information about the upcoming saccade long before it is executed, often before the previous saccade had even ended. Crucially, much of this early information did not relate to the actual saccade that would eventually be selected. Rather, it related to prior information about the probabilities of possible upcoming saccades based on the presaccade fixation location. Nearer to the time of saccade onset, a greater proportion of these neurons' activities related to the saccade selection, although prior information continued to influence activity throughout. A separate subset of FEF neurons, "Late neurons," only represented the final action plan near saccade onset and not prior information. Our results demonstrate how, across the population of FEF neurons, prior information evolves into definitive saccade plans.

Sensitivity to biomechanical limitations during postural decision-making depends on the integrity of posterior superior parietal cortex.

Most object-directed limb movements can be carried out with a comfortable grasp posture. However, the orientation of an object relative to our bodies can sometimes lead us to select an uncomfortable or awkward grasp posture due to limitations imposed by the biomechanics of the arm. In a series of experiments, we identified a network of cortical areas that are engaged during the selection of movement strategies. Neurologically intact participants and two brain-damaged patients with overlapping lesions in the right posterior superior parietal lobule (pSPL) performed a grasp posture selection task in which biomechanical constraints were the primary consideration for selecting an action. The task induced states of bistable actions whereby the same stimulus gave rise to categorically different grasp postures. In a behavioral experiment, the two patients displayed a large range of manual bistability with the contralesional hand, resulting in a higher incidence of awkward grasping postures. In neurologically intact participants, a separate functional magnetic resonance imaging (fMRI) experiment revealed activation of a parieto-frontal network, which included the posterior intraparietal sulcus (pIPS) along the banks of the pSPL that was parametrically modulated by the degree of bistability in grasp posture selection. Superimposing this activation over the patients' structural MRIs revealed that the pIPS/pSPL activation in the neurologically intact participants overlapped with lesioned cortical tissue in both patients; all other areas of activation overlapped with intact cortical tissue in the patients. These results provide converging evidence that the posterior parietal cortex plays a critical role in selecting biomechanically appropriate postures during reach-to-grasp behaviors.

A Trial-by-Trial Window into Sensorimotor Transformations in the Human Motor Periphery.

UNLABELLED: The appearance of a novel visual stimulus generates a rapid stimulus-locked response (SLR) in the motor periphery within 100 ms of stimulus onset. Here, we recorded SLRs from an upper limb muscle while humans reached toward (pro-reach) or away (anti-reach) from a visual stimulus. The SLR on anti-reaches encoded the location of the visual stimulus rather than the movement goal. Further, SLR magnitude was attenuated when subjects reached away from rather than toward the visual stimulus. Remarkably, SLR magnitudes also correlated with reaction times on both pro-reaches and anti-reaches, but did so in opposite ways: larger SLRs preceded shorter latency pro-reaches but longer latency anti-reaches. Although converging evidence suggests that the SLR is relayed via a tectoreticulospinal pathway, our results show that task-related signals modulate visual signals feeding into this pathway. The SLR therefore provides a trial-by-trial window into how visual information is integrated with cognitive control in humans. SIGNIFICANCE STATEMENT: The presentation of a visual stimulus elicits a trial-by-trial stimulus-locked response (SLR) on the human limb within 100 ms. Here, we show that the SLR continues to reflect stimulus location even when subjects move in the opposite direction (an anti-reach). Remarkably, the attenuation of SLR magnitude reflected the cognitive control required to generate a correct anti-reach, with greater degrees of attenuation preceding shorter-latency anti-reaches and no attenuation preceding error trials. Our results are strikingly similar to neurophysiological recordings in the superior colliculus of nonhuman primates generating anti-saccades, implicating the tectoreticulospinal pathway. Measuring SLR magnitude therefore provides an unprecedented trial-by-trial opportunity to assess the influence of cognitive control on the initial processing of a visual stimulus in humans.

Feature-based attention and spatial selection in frontal eye fields during natural scene search.

When we search for visual objects, the features of those objects bias our attention across the visual landscape (feature-based attention). The brain uses these top-down cues to select eye movement targets (spatial selection). The frontal eye field (FEF) is a prefrontal brain region implicated in selecting eye movements and is thought to reflect feature-based attention and spatial selection. Here, we study how FEF facilitates attention and selection in complex natural scenes. We ask whether FEF neurons facilitate feature-based attention by representing search-relevant visual features or whether they are primarily involved in selecting eye movement targets in space. We show that search-relevant visual features are weakly predictive of gaze in natural scenes and additionally have no significant influence on FEF activity. Instead, FEF activity appears to primarily correlate with the direction of the upcoming eye movement. Our result demonstrates a concrete need for better models of natural scene search and suggests that FEF activity during natural scene search is explained primarily by spatial selection.

Role of expected reward in frontal eye field during natural scene search.

When a saccade is expected to result in a reward, both neural activity in oculomotor areas and the saccade itself (e.g., its vigor and latency) are altered (compared with when no reward is expected). As such, it is unclear whether the correlations of neural activity with reward indicate a representation of reward beyond a movement representation; the modulated neural activity may simply represent the differences in motor output due to expected reward. Here, to distinguish between these possibilities, we trained monkeys to perform a natural scene search task while we recorded from the frontal eye field (FEF). Indeed, when reward was expected (i.e., saccades to the target), FEF neurons showed enhanced responses. Moreover, when monkeys accidentally made eye movements to the target, firing rates were lower than when they purposively moved to the target. Thus, neurons were modulated by expected reward rather than simply the presence of the target. We then fit a model that simultaneously included components related to expected reward and saccade parameters. While expected reward led to shorter latency and higher velocity saccades, these behavioral changes could not fully explain the increased FEF firing rates. Thus, FEF neurons appear to encode motivational factors such as reward expectation, above and beyond the kinematic and behavioral consequences of imminent reward.

Transient visual responses reset the phase of low-frequency oscillations in the skeletomotor periphery.

We recorded muscle activity from an upper limb muscle while human subjects reached towards peripheral targets. We tested the hypothesis that the transient visual response sweeps not only through the central nervous system, but also through the peripheral nervous system. Like the transient visual response in the central nervous system, stimulus-locked muscle responses (< 100 ms) were sensitive to stimulus contrast, and were temporally and spatially dissociable from voluntary orienting activity. Also, the arrival of visual responses reduced the variability of muscle activity by resetting the phase of ongoing low-frequency oscillations. This latter finding critically extends the emerging evidence that the feedforward visual sweep reduces neural variability via phase resetting. We conclude that, when sensory information is relevant to a particular effector, detailed information about the sensorimotor transformation, even from the earliest stages, is found in the peripheral nervous system.

Cross-species comparison of anticipatory and stimulus-driven neck muscle activity well before saccadic gaze shifts in humans and nonhuman primates.

Recent studies have described a phenomenon wherein the onset of a peripheral visual stimulus elicits short-latency (<100 ms) stimulus-locked recruitment (SLR) of neck muscles in nonhuman primates (NHPs), well before any saccadic gaze shift. The SLR is thought to arise from visual responses within the intermediate layers of the superior colliculus (SCi), hence neck muscle recordings may reflect presaccadic activity within the SCi, even in humans. We obtained bilateral intramuscular recordings from splenius capitis (SPL, an ipsilateral head-turning muscle) from 28 human subjects performing leftward or rightward visually guided eye-head gaze shifts. Evidence of an SLR was obtained in 16/55 (29%) of samples; we also observed examples where the SLR was present only unilaterally. We compared these human results with those recorded from a sample of eight NHPs from which recordings of both SPL and deeper suboccipital muscles were available. Using the same criteria, evidence of an SLR was obtained in 8/14 (57%) of SPL recordings, but in 26/29 (90%) of recordings from suboccipital muscles. Thus, both species-specific and muscle-specific factors contribute to the low SLR prevalence in human SPL. Regardless of the presence of the SLR, neck muscle activity in both human SPL and in NHPs became predictive of the reaction time of the ensuing saccade gaze shift ∼70 ms after target appearance; such pregaze recruitment likely reflects developing SCi activity, even if the tectoreticulospinal pathway does not reliably relay visually related activity to SPL in humans.

Counting on the motor system: rapid action planning reveals the format- and magnitude-dependent extraction of numerical quantity.

Symbolic numbers (e.g., "2") acquire their meaning by becoming linked to the core nonsymbolic quantities they represent (e.g., two items). However, the extent to which symbolic and nonsymbolic information converges onto the same internal core representations of quantity remains a point of considerable debate. As nearly all previous work on this topic has employed perceptual tasks requiring the conscious reporting of numerical magnitudes, here we question the extent to which numerical processing via the visual-motor system might shed further light on the fundamental basis of how different number formats are encoded. We show, using a rapid reaching task and a detailed analysis of initial arm trajectories, that there are key differences in how the quantity information extracted from symbolic Arabic numerals and nonsymbolic collections of discrete items are used to guide action planning. In particular, we found that the magnitude derived from discrete dots resulted in movements being biased by an amount directly proportional to the actual quantities presented whereas the magnitude derived from numerals resulted in movements being biased only by the relative (e.g., larger than) quantities presented. In addition, we found that initial motor plans were more sensitive to changes in numerical quantity within small (1-3) than large (5-15) number ranges, irrespective of their format (dots or numerals). In light of previous work, our visual-motor results clearly show that the processing of numerical quantity information is both format and magnitude dependent.

Connecting the dots: object connectedness deceives perception but not movement planning.

The perceptual system parses complex scenes into discrete objects. Parsing is also required for planning visually guided movements when more than one potential target is present. To examine whether visual perception and motor planning use the same or different parsing strategies, we used the connectedness illusion, in which observers typically report seeing fewer targets if pairs of targets are connected by short lines. We found that despite this illusion, when observers are asked to make speeded reaches toward targets in such displays, their reaches are unaffected by the presence of the connecting lines. Instead, their movement plans, as revealed by their movement trajectories, are influenced by the number of potential targets irrespective of whether connecting lines are present or not. This suggests that scene parsing for perception depends on mechanisms that are distinct from those that allow observers to plan rapid and efficient target-directed movements in situations with multiple potential targets.

Visual salience dominates early visuomotor competition in reaching behavior.

In this study, we investigated whether visual salience influences the competition between potential targets during reach planning. Participants initiated rapid pointing movements toward multiple potential targets, with the final target being cued only after the reach was initiated. We manipulated visual salience by varying the luminance of potential targets. Across two separate experiments, we demonstrate that initial reach trajectories are directed toward more salient targets, even when there are twice as many targets (and therefore twice the likelihood of the final target appearing) on the opposite side of space. We also show that this salience bias is time-dependent, as evidenced by the return of spatially averaged reach trajectories when participants were given an additional 500-ms preview of the target display prior to the cue to move. This study shows both when and to what extent task-irrelevant luminance differences affect the planning of reaches to multiple potential targets.

One to four, and nothing more: nonconscious parallel individuation of objects during action planning.

Much of the current understanding about the capacity limits on the number of objects that can be simultaneously processed comes from studies of visual short-term memory, attention, and numerical cognition. Consistent reports suggest that, despite large variability in the perceptual tasks administered (e.g., object tracking, counting), a limit of three to four visual items can be independently processed in parallel. In the research reported here, we asked whether this limit also extends to the domain of action planning. Using a unique rapid visuomotor task and a novel analysis of reach trajectories, we demonstrated an upper limit to the number of targets that can be simultaneously encoded for action, a capacity limit that also turns out to be no more than three to four. Our findings suggest that conscious perceptual processing and nonconscious movement planning are constrained by a common underlying mechanism limited by the number of items that can be simultaneously represented.

Selection of wrist posture in conditions of motor ambiguity.

In our everyday motor interactions with objects, we often encounter situations where the features of an object are determinate (i.e., not perceptually ambiguous), but the mapping between those features and appropriate movement patterns is indeterminate, resulting in a lack of any clear preference for one posture over another. We call this indeterminacy in stimulus-response mapping 'motor ambiguity'. Here, we use a grasping task to investigate the decision mechanisms that mediate the basic behavior of selecting one wrist posture over another in conditions of motor ambiguity. Using one of two possible wrist postures, participants grasped a dowel that was presented at various orientations. At most orientations, there was a clear preference for one wrist posture over the other. Within a small range of orientations, however, participants were variable in their posture selection due to the fact that the dowel was ambiguous with respect to the hand posture it afforded. We observed longer reaction times (RT) during 'ambiguous' trials than during the 'unambiguous' trials. In two subsequent experiments, we explored the effects of foreknowledge and trial history on the selection of wrist posture. We found that foreknowledge led to shorter RT unless the previous trial involved selecting a posture in the ambiguous region, in which case foreknowledge gave no RT advantage. These results are discussed within the context of existing models of sensorimotor decision making.

Short-term motor plasticity revealed in a visuomotor decision-making task.

Selecting and executing an action toward only one object in our complex environments presents the visuomotor system with a significant challenge. To overcome this problem, the motor system is thought to simultaneously encode multiple motor plans, which then compete for selection. The decision between motor plans is influenced both by incoming sensory information and previous experience-which itself is comprised of long-term (e.g. weeks, months) and recent (seconds, minutes, hours) information. In this study, we were interested in how the recent trial-to-trial visuomotor experience would be factored into upcoming movement decisions made between competing potential targets. To this aim, we used a unique rapid reaching task to investigate how reach trajectories would be spatially influenced by previous decisions. Our task required subjects to initiate speeded reaches toward multiple potential targets before one was cued in-flight. A novel statistical analysis of the reach trajectories revealed that in cases of target uncertainty, subjects initiated a spatially averaged trajectory toward the midpoint of potential target locations before correcting to the selected target location. Interestingly, when the same target location was consecutively cued, reaches were biased toward that location on the next trial and this effect accumulated across trials. Beyond providing supporting evidence that potential reach locations are encoded and compete in parallel, our results strongly suggest that this motor competition is biased by recent trial history.

Reaching for the unknown: multiple target encoding and real-time decision-making in a rapid reach task.

Decision-making is central to human cognition. Fundamental to every decision is the ability to internally represent the available choices and their relative costs and benefits. The most basic and frequent decisions we make occur as our motor system chooses and executes only those actions that achieve our current goals. Although these interactions with the environment may appear effortless, this belies what must be incredibly sophisticated visuomotor decision-making processes. In order to measure how visuomotor decisions unfold in real-time, we used a unique reaching paradigm that forced participants to initiate rapid hand movements toward multiple potential targets, with only one being cued after reach onset. We show across three experiments that, in cases of target uncertainty, trajectories are spatially sensitive to the probabilistic distribution of targets within the display. Specifically, when presented with two or three target displays, subjects initiate their reaches toward an intermediary or 'averaged' location before correcting their trajectory in-flight to the cued target location. A control experiment suggests that our effect depends on the targets acting as potential reach locations and not as distractors. This study is the first to show that the 'averaging' of target-directed reaching movements depends not only on the spatial position of the targets in the display but also the probability of acting at each target location.