Behavior in a visual search task with moving dot stimuli.

Understanding the neuronal mechanisms underlying the processing of visual attention requires a well-designed behavioral task that allows investigators to clearly describe the behavioral effects of attention. Here, we introduce a behavioral paradigm in which one, two or four moving dot stimuli are used in a visual search paradigm that includes two additional attentional conditions. Two animals were trained to make a saccade to a target (a dot patch with net rightward motion) and hold central fixation if no target was present. To implement covert visual attention, we included trials in which a 100% valid spatial cue appeared and trials in which the color of the fixation point indicated, with 100% validity, which of four colored dot patches the animals should attend to. We analyzed the behavior in terms of reaction times and signal detection theory metrics d-prime (representing sensitivity) and criteria. In both animals, we found that reaction times were greater for larger set-sizes and that the introduction of an attentional cue reduced the reaction times substantially. We also found that both animals showed increases in criteria, but no change in sensitivity, as set-size increased and the attentional cues led to an increase in sensitivity, with only some change in criteria. Our results illustrate how the animals perform this task and imply that both animals chose similar strategies. Importantly, this will allow future neurophysiological studies to probe not only the effects of attention, but will give the possibility of seeing whether different neural mechanisms drive changes in criteria and d-prime.

The roles of the lateral intraparietal area and frontal eye field in guiding eye movements in free viewing search behavior.

The lateral intraparietal area (LIP) and frontal eye field (FEF) have been shown to play significant roles in oculomotor control, yet most studies have found that the two areas behave similarly. To identify the unique roles each area plays in guiding eye movements, we recorded 200 LIP neurons and 231 FEF neurons from four animals performing a free viewing visual foraging task. We analyzed how neuronal responses were modulated by stimulus identity and the animals' choice of where to make a saccade. We additionally analyzed the comodulation of the sensory signals and the choice signal to identify how the sensory signals drove the choice. We found a clearly defined division of labor: LIP provided a stable map integrating task rules and stimulus identity, whereas FEF responses were dynamic, representing more complex information and, just before the saccade, were integrated with task rules and stimulus identity to decide where to move the eye.NEW & NOTEWORTHY The lateral intrapareital area (LIP) and frontal eye field (FEF) are known to contribute to guiding eye movements, but little is known about the unique roles that each area plays. Using a free viewing visual search task, we found that LIP provides a stable map of the visual world, integrating task rules and stimulus identity. FEF activity is consistently modulated by more complex information but, just before the saccade, integrates all the information to make the final decision about where to move.

Bio-Inspired Haptic Feedback for Artificial Palpation in Robotic Surgery.

Adding haptic feedback has been reported to improve the outcome of minimally invasive robotic surgery. In this study, we seek to determine whether an algorithm based on simulating responses of a cutaneous afferent population can be implemented to improve the performance of presenting haptic feedback for robot-assisted surgery. We propose a bio-inspired controlling model to present vibration and force feedback to help surgeons localize underlying structures in phantom tissue. A single pair of actuators was controlled by outputs of a model of a population of cutaneous afferents based on the pressure signal from a single sensor embedded in surgical forceps. We recruited 25 subjects including 10 expert surgeons to evaluate the performance of the bio-inspired controlling model in an artificial palpation task using the da Vinci surgical robot. Among the control methods tested, the bio-inspired system was unique in allowing both novices and experts to easily identify the locations of all classes of tumors and did so with reduced contact force and tumor contact time. This work demonstrates the utility of our bio-inspired multi-modal feedback system, which resulted in superior performance for both novice and professional users, in comparison to a traditional linear and the existing piecewise discrete algorithms of haptic feedback.

A Simplified Model for Simulating Population Responses of Tactile Afferents and Receptors in the Skin.

Tactile information about an object can only be extracted from population responses of tactile receptors and their afferents. Thus, to best control tactile information in robots, neuroprostheses or haptic devices, inputs should represent responses from full populations of afferents. Here, we describe a simplified model that recreates afferent population responses of thousands of tactile afferents in a personal computer. The whole model includes a resistance network model to simplify the skin mechanics and an improved version of a single unit model that we have previously described. The whole model was implemented by short and efficient python code. The parameters of the model were fit based on a simple vibrating stimulus, but the simulated outputs generalize to match receptive field sizes, edge enhancement, and neurophysiological responses to dot textures, embossed letters and curved surfaces. We discuss how to use this work to model haptic perception and provide guidance in designing and controlling highly realistic tactile interfaces in robots, neural prostheses and haptic devices.

The functional roles of neural remapping in cortex.

Remapping is a property of some cortical and subcortical neurons that update their responses around the time of an eye movement to account for the shift of stimuli on the retina due to the saccade. Physiologically, remapping is traditionally tested by briefly presenting a single stimulus around the time of the saccade and looking at the onset of the response and the locations in space to which the neuron is responsive. Here we suggest that a better way to understand the functional role of remapping is to look at the time at which the neural signal emerges when saccades are made across a stable scene. Based on data obtained using this approach, we suggest that remapping in the lateral intraparietal area is sufficient to play a role in maintaining visual stability across saccades, whereas in the frontal eye field, remapped activity carries information that affects future saccadic choices and, in a separate subset of neurons, is used to maintain a map of locations in the scene that have been previously fixated.

Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic Surgery.

Minimally invasive robotic surgery allows for many advantages over traditional surgical procedures, but the loss of force feedback combined with a potential for strong grasping forces can result in excessive tissue damage. Single modality haptic feedback systems have been designed and tested in an attempt to diminish grasping forces, but the results still fall short of natural performance. A multi-modal pneumatic feedback system was designed to allow for tactile, kinesthetic, and vibrotactile feedback, with the aims of more closely imitating natural touch and further improving the effectiveness of HFS in robotic surgical applications and tasks such as tissue grasping and manipulation. Testing of the multi-modal system yielded very promising results with an average force reduction of nearly 50% between the no feedback and hybrid (tactile and kinesthetic) trials (p < 1.0E-16). The multi-modal system demonstrated an increased reduction over single modality feedback solutions and indicated that the system can help users achieve average grip forces closer to those normally possible with the human hand.

The neural instantiation of a priority map.

The term priority map is commonly used to describe a map of the visual scene, in which objects and locations are represented by their attentional priority, which itself is a combination of low-level salience and top-down control. The aim of this review is to examine how such a map may be represented at the neuronal level. We propose that there is not a single, common map in the brain, but that a number of cortical areas work together to generate the resultant behavior. Specifically, we suggest that the lateral intraparietal area (LIP) of posterior parietal cortex provides a simple representation of attentional priority, which remaps across saccades, so that there is an apparent allocentric map in a region with retinocentric encoding scheme. We propose that the frontal eye field (FEF) of prefrontal cortex receives the responses from LIP, but can suppress them to control the flow of eye movement behavior, and that the intermediate layers of the superior colliculus (SCi) reflect the final saccade goal. Together, these areas function to guide eye movements and may play a similar role in allocating covert visual attention.

Neurons in FEF Keep Track of Items That Have Been Previously Fixated in Free Viewing Visual Search.

When searching a visual scene for a target, we tend not to look at items or locations we have already searched. It is thought that this behavior is driven by an inhibitory tagging mechanism that inhibits responses on priority maps to the relevant items. We hypothesized that this inhibitory tagging signal should be represented as an elevated response in neurons that keep track of stimuli that have been fixated. We recorded from 231 neurons in the frontal eye field (FEF) of 2 male animals performing a visual foraging task, in which they had to find a reward linked to one of five identical targets (Ts) among five distractors. We identified 38 neurons with activity that was significantly greater when the stimulus in the receptive field had been fixated previously in the trial than when it had not been fixated. The response to a fixated object began before the saccade ended, suggesting that this information is remapped. Unlike most FEF neurons, the activity in these cells was not suppressed during active fixation, had minimal motor responses, and did not change through the trial. Yet using traditional classifications from a memory-guided saccade, they were indistinguishable from the rest of the FEF population. We propose that these neurons keep track of any items that have been fixated within the trial and this signal is propagated by remapping. These neurons could be the source of the inhibitory tagging signal to parietal cortex, where a neuronal instantiation of inhibitory tagging is seen.SIGNIFICANCE STATEMENT When we search a scene for an item, we rarely examine the same location twice. It is thought that this is due to a neural mechanism that keeps track of the items at which we have looked. Here we identified a subset of neurons in the frontal eye field that preferentially responded to items that had been fixated earlier in the trial. These responses were remapped, appearing before the saccade even ended, and were not suppressed during maintained fixation. We propose that these neurons keep track of which items have been examined in search and could be the source of feedback that creates the inhibitory tagging seen in parietal cortex.

Artificial palpation in robotic surgery using haptic feedback.

BACKGROUND: The loss of tactile feedback in minimally invasive robotic surgery remains a major challenge to the expanding field. With visual cue compensation alone, tissue characterization via palpation proves to be immensely difficult. This work evaluates a bimodal vibrotactile system as a means of conveying applied forces to simulate haptic feedback in two sets of studies simulating an artificial palpation task using the da Vinci surgical robot. METHODS: Subjects in the first study were tasked with localizing an embedded vessel in a soft tissue phantom using a single-sensor unit. In the second study, subjects localized tumor-like structures using a three-sensor array. In both sets of studies, subjects completed the task under three trial conditions: no feedback, normal force tactile feedback, and hybrid vibrotactile feedback. Recordings of correct localization, incorrect localization, and time-to-completion were used to evaluate performance outcomes. RESULTS: With the addition of vibrotactile and pneumatic feedback, significant improvements in the percentage of correct localization attempts were detected (p = 0.0001 and p = 0.0459, respectively) during the first experiment with phantom vessels. Similarly, significant improvements in correct localization were found with the addition of vibrotactile (p = 2.57E-5) and pneumatic significance (p = 8.54E-5) were observed in the second experiment involving tumor phantoms. CONCLUSIONS: This work demonstrates not only the superior benefits of a multi-modal feedback over traditional single-modality feedback, but also the effectiveness of vibration in providing haptic feedback to artificial palpation systems.

Suppression of frontal eye field neuronal responses with maintained fixation.

The decision of where to make an eye movement is thought to be driven primarily by responses to stimuli in neurons' receptive fields (RFs) in oculomotor areas, including the frontal eye field (FEF) of prefrontal cortex. It is also thought that a saccade may be generated when the accumulation of this activity in favor of one location or another reaches a threshold. However, in the reading and scene perception fields, it is well known that the properties of the stimulus at the fovea often affect when the eyes leave that stimulus. We propose that if FEF plays a role in generating eye movements, then the identity of the stimulus at fixation should affect the FEF responses so as to reduce the probability of making a saccade when fixating an item of interest. Using a visual foraging task in which animals could make multiple eye movements within a single trial, we found that responses were strongly modulated by the identity of the stimulus at the fovea. Specifically, responses to the stimulus in the RF were suppressed when the animal maintained fixation for longer durations on a stimulus that could be associated with a reward. We suggest that this suppression, which was predicted by models of eye movement behavior, could be a mechanism by which FEF can modulate the temporal flow of saccades based on the importance of the stimulus at the fovea.

Activity in LIP, But not V4, Matches Performance When Attention is Spread.

The enhancement of neuronal responses in many visual areas while animals perform spatial attention tasks has widely been thought to be the neural correlate of visual attention, but it is unclear whether the presence or absence of this modulation contributes to our striking inability to notice changes in change blindness examples. We asked whether neuronal responses in visual area V4 and the lateral intraparietal area (LIP) in posterior parietal cortex could explain the limited ability of subjects to attend multiple items in a display. We trained animals to perform a change detection task in which they had to compare 2 arrays of stimuli separated briefly in time and found that each animal's performance decreased as function of set-size. Neuronal discriminability in V4 was consistent across set-sizes, but decreased for higher set-sizes in LIP. The introduction of a reward bias produced attentional enhancement in V4, but this could not explain the vast improvement in performance, whereas the enhancement in LIP responses could. We suggest that behavioral set-size effects and the marked improvement in performance with focused attention may not be related to response enhancement in V4 but, instead, may occur in or on the way to LIP.

Object comparison in the lateral intraparietal area.

We can search for and locate specific objects in our environment by looking for objects with similar features. Object recognition involves stimulus similarity responses in ventral visual areas and task-related responses in prefrontal cortex. We tested whether neurons in the lateral intraparietal area (LIP) of posterior parietal cortex could form an intermediary representation, collating information from object-specific similarity map representations to allow general decisions about whether a stimulus matches the object being looked for. We hypothesized that responses to stimuli would correlate with how similar they are to a sample stimulus. When animals compared two peripheral stimuli to a sample at their fovea, the response to the matching stimulus was similar, independent of the sample identity, but the response to the nonmatch depended on how similar it was to the sample: the more similar, the greater the response to the nonmatch stimulus. These results could not be explained by task difficulty or confidence. We propose that LIP uses its known mechanistic properties to integrate incoming visual information, including that from the ventral stream about object identity, to create a dynamic representation that is concise, low dimensional, and task relevant and that signifies the choice priorities in mental matching behavior.NEW & NOTEWORTHY Studies in object recognition have focused on the ventral stream, in which neurons respond as a function of how similar a stimulus is to their preferred stimulus, and on prefrontal cortex, where neurons indicate which stimulus is being looked for. We found that parietal area LIP uses its known mechanistic properties to form an intermediary representation in this process. This creates a perceptual similarity map that can be used to guide decisions in prefrontal areas.

Visual-perceptual mismatch in robotic surgery.

BACKGROUND: The principal objective of the experiment was to analyze the effects of the clutch operation of robotic surgical systems on the performance of the operator. The relative coordinate system introduced by the clutch operation can introduce a visual-perceptual mismatch which can potentially have negative impact on a surgeon's performance. We also assess the impact of the introduction of additional tactile sensory information on reducing the impact of visual-perceptual mismatch on the performance of the operator. METHODS: We asked 45 novice subjects to complete peg transfers using the da Vinci IS 1200 system with grasper-mounted, normal force sensors. The task involves picking up a peg with one of the robotic arms, passing it to the other arm, and then placing it on the opposite side of the view. Subjects were divided into three groups: aligned group (no mismatch), the misaligned group (10 cm z axis mismatch), and the haptics-misaligned group (haptic feedback and z axis mismatch). Each subject performed the task five times, during which the grip force, time of completion, and number of faults were recorded. RESULTS: Compared to the subjects that performed the tasks using a properly aligned controller/arm configuration, subjects with a single-axis misalignment showed significantly more peg drops (p = 0.011) and longer time to completion (p < 0.001). Additionally, it was observed that addition of tactile feedback helps reduce the negative effects of visual-perceptual mismatch in some cases. Grip force data recorded from grasper-mounted sensors showed no difference between the different groups. CONCLUSIONS: The visual-perceptual mismatch created by the misalignment of the robotic controls relative to the robotic arms has a negative impact on the operator of a robotic surgical system. Introduction of other sensory information and haptic feedback systems can help in potentially reducing this effect.

Evaluating tactile feedback in robotic surgery for potential clinical application using an animal model.

INTRODUCTION: The aims of this study were to evaluate (1) grasping forces with the application of a tactile feedback system in vivo and (2) the incidence of tissue damage incurred during robotic tissue manipulation. Robotic-assisted minimally invasive surgery has been shown to be beneficial in a variety of surgical specialties, particularly radical prostatectomy. This innovative surgical tool offers advantages over traditional laparoscopic techniques, such as improved wrist-like maneuverability, stereoscopic video displays, and scaling of surgical gestures to increase precision. A widely cited disadvantage associated with robotic systems is the absence of tactile feedback. METHODS AND PROCEDURE: Nineteen subjects were categorized into two groups: 5 experts (six or more robotic cases) and 14 novices (five cases or less). The subjects used the da Vinci with integrated tactile feedback to run porcine bowel in the following conditions: (T1: deactivated tactile feedback; T2: activated tactile feedback; and T3: deactivated tactile feedback). The grasping force, incidence of tissue damage, and the correlation of grasping force and tissue damage were analyzed. Tissue damage was evaluated both grossly and histologically by a pathologist blinded to the sample. RESULTS: Tactile feedback resulted in significantly decreased grasping forces for both experts and novices (P < 0.001 in both conditions). The overall incidence of tissue damage was significantly decreased in all subjects (P < 0.001). A statistically significant correlation was found between grasping forces and incidence of tissue damage (P = 0.008). The decreased forces and tissue damage were retained through the third trial when the system was deactivated (P > 0.05 in all subjects). CONCLUSION: The in vivo application of integrated tactile feedback in the robotic system demonstrates significantly reduced grasping forces, resulting in significantly less tissue damage. This tactile feedback system may improve surgical outcomes and broaden the use of robotic-assisted minimally invasive surgery.

Remapping, Spatial Stability, and Temporal Continuity: From the Pre-Saccadic to Postsaccadic Representation of Visual Space in LIP.

As our eyes move, we have a strong percept that the world is stable in space and time; however, the signals in cortex coming from the retina change with each eye movement. It is not known how this changing input produces the visual percept we experience, although the predictive remapping of receptive fields has been described as a likely candidate. To explain how remapping accounts for perceptual stability, we examined responses of neurons in the lateral intraparietal area while animals performed a visual foraging task. When a stimulus was brought into the response field of a neuron that exhibited remapping, the onset of the postsaccadic representation occurred shortly after the saccade ends. Whenever a stimulus was taken out of the response field, the presaccadic representation abruptly ended shortly after the eyes stopped moving. In the 38% (20/52) of neurons that exhibited remapping, there was no more than 30 ms between the end of the presaccadic representation and the start of the postsaccadic representation and, in some neurons, and the population as a whole, it was continuous. We conclude by describing how this seamless shift from a presaccadic to postsaccadic representation could contribute to spatial stability and temporal continuity.

LIP activity in the interstimulus interval of a change detection task biases the behavioral response.

When looking around at the world, we can only attend to a limited number of locations. The lateral intraparietal area (LIP) is thought to play a role in guiding both covert attention and eye movements. In this study, we tested the involvement of LIP in both mechanisms with a change detection task. In the task, animals had to indicate whether an element changed during a blank in the trial by making a saccade to it. If no element changed, they had to maintain fixation. We examine how the animal's behavior is biased based on LIP activity prior to the presentation of the stimulus the animal must respond to. When the activity was high, the animal was more likely to make an eye movement toward the stimulus, even if there was no change; when the activity was low, the animal either had a slower reaction time or maintained fixation, even if a change occurred. We conclude that LIP activity is involved in both covert and overt attention, but when decisions about eye movements are to be made, this role takes precedence over guiding covert attention.

The what, where, and why of priority maps and their interactions with visual working memory.

Priority maps are winner-take-all neural mechanisms thought to guide the allocation of covert and overt attention. Here, we go beyond this standard definition and argue that priority maps play a much broader role in controlling goal-directed behavior. We start by defining what priority maps are and where they might be found in the brain; we then ask why they exist-the function that they serve. We propose that this function is to communicate a goal state to the different effector systems, thereby guiding behavior. Within this framework, we speculate on how priority maps interact with visual working memory and introduce our common source hypothesis, the suggestion that this goal state is maintained in visual working memory and used to construct all of the priority maps controlling the various motor systems. Finally, we look ahead and suggest questions about priority maps that should be asked next.

Extrafoveal preview benefit during free-viewing visual search in the monkey.

Previous studies have shown that subjects require less time to process a stimulus at the fovea after a saccade if they have viewed the same stimulus in the periphery immediately prior to the saccade. This extrafoveal preview benefit indicates that information about the visual form of an extrafoveally viewed stimulus can be transferred across a saccade. Here, we extend these findings by demonstrating and characterizing a similar extrafoveal preview benefit in monkeys during a free-viewing visual search task. We trained two monkeys to report the orientation of a target among distractors by releasing one of two bars with their hand; monkeys were free to move their eyes during the task. Both monkeys took less time to indicate the orientation of the target after foveating it, when the target lay closer to the fovea during the previous fixation. An extrafoveal preview benefit emerged even if there was more than one intervening saccade between the preview and the target fixation, indicating that information about target identity could be transferred across more than one saccade and could be obtained even if the search target was not the goal of the next saccade. An extrafoveal preview benefit was also found for distractor stimuli. These results aid future physiological investigations of the extrafoveal preview benefit.

Evidence for differential top-down and bottom-up suppression in posterior parietal cortex.

When searching for an object, we usually avoid items that are visually different from the target and objects or places that have been searched already. Previous studies have shown that neural activity in the lateral intraparietal area (LIP) can be used to guide this behaviour; responses to task irrelevant stimuli or to stimuli that have been fixated previously in the trial are reduced compared with responses to potential targets. Here, we test the hypothesis that these reduced responses have a different genesis. Two animals were trained on a visual foraging task, in which they had to find a target among a number of physically identical potential targets (T) and task irrelevant distractors. We recorded neural activity and local field potentials (LFPs) in LIP while the animals performed the task. We found that LFP power was similar for potential targets and distractors but was greater in the alpha and low beta bands when a previously fixated T was in the response field. We interpret these data to suggest that the reduced single-unit response to distractors is a bottom-up feed-forward result of processing in earlier areas and the reduced response to previously fixated Ts is a result of active top-down suppression.