Ethical implications of visual neuroprostheses-a systematic review.

Objective. The aim of this review was to systematically identify the ethical implications of visual neuroprostheses.Approach. A systematic search was performed in both PubMed and Embase using a search string that combined synonyms for visual neuroprostheses, brain-computer interfaces (BCIs), cochlear implants (CIs), and ethics. We chose to include literature on BCIs and CIs, because of their ethically relavant similarities and functional parallels with visual neuroprostheses.Main results. We included 84 articles in total. Six focused specifically on visual prostheses. The other articles focused more broadly on neurotechnologies, on BCIs or CIs. We identified 169 ethical implications that have been categorized under seven main themes: (a) benefits for health and well-being; (b) harm and risk; (c) autonomy; (d) societal effects; (e) clinical research; (f) regulation and governance; and (g) involvement of experts, patients and the public.Significance. The development and clinical use of visual neuroprostheses is accompanied by ethical issues that should be considered early in the technological development process. Though there is ample literature on the ethical implications of other types of neuroprostheses, such as motor neuroprostheses and CIs, there is a significant gap in the literature regarding the ethical implications of visual neuroprostheses. Our findings can serve as a starting point for further research and normative analysis.

Slow brain oscillations of sleep, resting state, and vigilance.

The most important quest of cognitive neuroscience may be to unravel the mechanisms by which the brain selects, links, consolidates, and integrates new information into its neuronal network, while preventing saturation to occur. During the past decade, neuroscientists working within several disciplines have observed an important involvement of the specific types of brain oscillations that occur during sleep--the cortical slow oscillations; during the resting state--the fMRI resting state networks including the default-mode network (DMN); and during task performance--the performance modulations that link as well to modulations in electroencephalography or magnetoencephalography frequency content. Understanding the role of these slow oscillations thus appears to be essential for our fundamental understanding of brain function. Brain activity is characterized by oscillations occurring in spike frequency, field potentials or blood oxygen level-dependent functional magnetic resonance imaging signals. Environmental stimuli, reaching the brain through our senses, activate or inactivate neuronal populations and modulate ongoing activity. The effect they sort is to a large extent determined by the momentary state of the slow endogenous oscillations of the brain. In the absence of sensory input, as is the case during rest or sleep, brain activity does not cease. Rather, its oscillations continue and change with respect to their dominant frequencies and coupling topography. This chapter briefly introduces the topics that will be addressed in this dedicated volume of Progress in Brain Research on slow oscillations and sets the stage for excellent papers discussing their molecular, cellular, network physiological and cognitive performance aspects. Getting to know about slow oscillations is essential for our understanding of plasticity, memory, brain structure from synapse to DMN, cognition, consciousness, and ultimately for our understanding of the mechanisms and functions of sleep and vigilance.

Time course of attentional modulation in the frontal eye field during curve tracing.

Neurons in the frontal eye fields (FEFs) register incoming visual information and select visual stimuli that are relevant for behavior. Here we investigated the timing of the visual response and the timing of selection by recording from single FEF neurons in a curve-tracing task that requires shifts of attention followed by an oculomotor response. We found that the behavioral selection signal in area FEF had a latency of 147 ms and that it was delayed substantially relative to the visual response, which occurred 50 ms after stimulus presentation. We compared the FEF responses to activity previously recorded in the primary visual cortex (area V1) during the same task. Visual responses in area V1 preceded the FEF responses, but the latencies of selection signals in areas V1 and FEF were similar. The similarity of timing of selection signals in structures at opposite ends of the visual cortical processing hierarchy supports the view that stimulus selection occurs in an interaction between widely separated cortical regions.

The effect of items in working memory on the deployment of attention and the eyes during visual search.

Paying attention to an object facilitates its storage in working memory. The authors investigate whether the opposite is also true: whether items in working memory influence the deployment of attention. Participants performed a search for a prespecified target while they held another item in working memory. In some trials this memory item was present in the search display as a distractor. Such a distractor has no effect on search time if the search target is in the display. In that case, the item in working memory is unlikely to be selected as a target for an eye movement, and if the eyes do land on it, fixation duration is short. In the absence of the target, however, there is a small but significant effect of the memory item on search time. The authors conclude that the target for visual search has a special status in working memory that allows it to guide attention. Guidance of attention by other items in working memory is much weaker and can be observed only if the search target is not present in the display.

A gradual spread of attention during mental curve tracing.

The visual system has to segregate objects that are relevant to behavior from other objects and the background, if they are embedded in a visual scene. This segregation process can be time consuming, especially if the relevant object is spatially extended and overlaps with other image components, but the cause of the delays is presently not well understood. In the present study, we used a curve-tracing task to investigate processing delays during the grouping of contour segments into elongated curves. Our results indicate that contour segments that need to be grouped together are labeled with visual attention. Attention gradually spreads from contour segments that were labeled previously to other contours that are colinear and connected to them. The contour-grouping task is completed as soon as attention is directed to the entire curve. We conclude that processing delays during contour grouping are caused by a time-consuming spread of visual attention.

The representation of erroneously perceived stimuli in the primary visual cortex.

In order to attain a correct interpretation of an ambiguous visual stimulus, the brain may have to elaborate on the sensory evidence. Are the neurons that carry the sensory evidence also involved in generating an interpretation? To address this question, we studied the activity of neurons in the primary visual cortex of macaque monkeys involved in a task in which they have to trace a curve mentally, without moving their eyes. On a percentage of trials, the monkeys made errors and traced the wrong curve. Here, we show that these errors are predicted by activity in area V1. Thus, neurons in the primary visual cortex do not only represent sensory events, but also the way in which they are interpreted by the monkey.

The spatial profile of visual attention in mental curve tracing.

In a curve-tracing task, subjects have to judge whether items are located on a single, continuous curve. Spatially separate segments of such a curve are related to each other through grouping criteria, like collinearity and connectedness. These grouping cues need to be exploited during curve tracing, but it is still an open issue how grouping of contour segments is achieved by the visual system. Many contemporary theories of visual perception assume that grouping operations are carried out pre-attentively, with unlimited capacity. The present study examines this assumption by investigating the involvement of attention in curve tracing. The results show that attention is directed to contour segments that need to be grouped together. The distribution of attention is guided by grouping criteria, such as connectedness. Apparently, attention is required to group spatially separate contour segments into a coherent representation of a curve.

The distinct modes of vision offered by feedforward and recurrent processing.

An analysis of response latencies shows that when an image is presented to the visual system, neuronal activity is rapidly routed to a large number of visual areas. However, the activity of cortical neurons is not determined by this feedforward sweep alone. Horizontal connections within areas, and higher areas providing feedback, result in dynamic changes in tuning. The differences between feedforward and recurrent processing could prove pivotal in understanding the distinctions between attentive and pre-attentive vision as well as between conscious and unconscious vision. The feedforward sweep rapidly groups feature constellations that are hardwired in the visual brain, yet is probably incapable of yielding visual awareness; in many cases, recurrent processing is necessary before the features of an object are attentively grouped and the stimulus can enter consciousness.

The implementation of visual routines.

Many visual tasks can be decomposed into a sequence of simpler subtasks. Ullman suggested that such subtasks are carried out by elemental operations that are implemented by specialized processes in the visual brain [Ullman, S. (1984). Visual routines. Cognition (18), 97-159]. According to this hypothesis, there are a limited number of elemental operations that, since they can be applied sequentially, may nevertheless give rise to a large number of visual routines. Examples of such elemental operations are visual search, texture segregation and contour grouping. Here we attempt to delineate how such elemental operations are implemented in the visual brain. When an image appears, feedforward processing rapidly leads to an activity pattern that is distributed across many visual areas. Thereafter, elemental operations come into play, and these are implemented by the modulation of firing rates. Firing rate modulations effectuate grouping of neural responses into coherent object representations. Moreover, they permit transfer of information from one operator to the next, which allows flexibility in the sequencing of operations. We discuss how the elemental operations provide a tool to relate cortical physiology to psychophysics, and suggest a reclassification of pre-attentive and attentive processes.

The role of primary visual cortex (V1) in visual awareness.

In the search for the neural correlate of visual awareness, much controversy exists about the role of primary visual cortex. Here, the neurophysiological data from V1 recordings in awake monkeys are examined in light of two general classes of models of visual awareness. In the first model type, visual awareness is seen as being mediated either by a particular set of areas or pathways, or alternatively by a specific set of neurons. In these models, the role of V1 seems rather limited, as the mere activity of V1 cells seems insufficient to mediate awareness. In the second model type, awareness is hypothesized to be mediated by a global mechanism, i.e. a specific kind of activity not linked to a particular area or cell type. Two separate versions of global models are discussed, synchronous oscillations and spike rate modulations. It is shown that V1 synchrony does not reflect perception but rather the horizontal connections between neurons, indicating that V1 synchrony cannot be a direct neural correlate of conscious percepts. However, the rate of spike discharges of V1 neurons is strongly modulated by perceptual context, and these modulations correlate very well with aspects of perceptual organization, visual awareness, and attention. If these modulations serve as a neural correlate of visual awareness, then V1 contributes to that neural correlate. Whether V1 plays a role in the neural correlate of visual awareness thus strongly depends on the way visual awareness is hypothesized to be implemented in the brain.

The effects of pair-wise and higher order correlations on the firing rate of a post-synaptic neuron.

Coincident firing of neurons projecting to a common target cell is likely to raise the probability of firing of this postsynaptic cell. Therefore, synchronized firing constitutes a significant event for postsynaptic neurons and is likely to play a role in neuronal information processing. Physiological data on synchronized firing in cortical networks are based primarily on paired recordings and cross-correlation analysis. However, pair-wise correlations among all inputs onto a postsynaptic neuron do not uniquely determine the distribution of simultaneous postsynaptic events. We develop a framework in order to calculate the amount of synchronous firing that, based on maximum entropy, should exist in a homogeneous neural network in which the neurons have known pair-wise correlations and higher-order structure is absent. According to the distribution of maximal entropy, synchronous events in which a large proportion of the neurons participates should exist even in the case of weak pair-wise correlations. Network simulations also exhibit these highly synchronous events in the case of weak pair-wise correlations. If such a group of neurons provides input to a common postsynaptic target, these network bursts may enhance the impact of this input, especially in the case of a high postsynaptic threshold. The proportion of neurons participating in synchronous bursts can be approximated by our method under restricted conditions. When these conditions are not fulfilled, the spike trains have less than maximal entropy, which is indicative of the presence of higher-order structure. In this situation, the degree of synchronicity cannot be derived from the pair-wise correlations.

Temporal constraints on the grouping of contour segments into spatially extended objects.

The speed of contour integration was investigated in a task that can be solved by grouping contour segments into elongated curves. Subjects had to detect a continuous curve, which could be intersected by one or two other curves. At locations where these curves came in close proximity, the assignment of contour segments to the different curves could be based on collinearity. Reaction times exhibited a strong dependence on (1) the presence of intersections among curves; and (2) the context provided by the stimulus set from which individual stimuli were selected. Reaction times were shortest when grouping of contour segments depended on information at a single location in the visual field. In this condition, responses to stimuli containing an intersection were faster than responses to stimuli that did not. When responses were determined by information at spatially separate locations, responses were delayed, and every intersection increased the reaction time considerably. This result contrasts with earlier investigations which have suggested that contour integration on the basis of collinearity is performed pre-attentively but is in accordance with studies on curve tracing. We propose that the assignment of contour segments to equally coherent curves, a process which may be called figure-figure segregation, is a function of object-based attention. Moreover, the protracted reaction times for some of the stimuli indicate that spread of attention within an object costs time. This implies that object recognition is not always as fast as is sometimes assumed.

Object-based attention in the primary visual cortex of the macaque monkey.

Typical natural visual scenes contain many objects, which need to be segregated from each other and from the background. Present theories subdivide the processes responsible for this segregation into a pre-attentive and attentive system. The pre-attentive system segregates image regions that 'pop out' rapidly and in parallel across the visual field. In the primary visual cortex, responses to pre-attentively selected image regions are enhanced. When objects do not segregate automatically from the rest of the image, the time-consuming attentive system is recruited. Here we investigate whether attentive selection is also associated with a modulation of firing rates in area V1 of the brain in monkeys trained to perform a curve-tracing task. Neuronal responses to the various segments of a target curve were simultaneously enhanced relative to responses evoked by a distractor curve, even if the two curves crossed each other. This indicates that object-based attention is associated with a response enhancement at the earliest level of the visual cortical processing hierarchy.

Solutions for the binding problem.

Visual cortical neurons are broadly tuned to one or a few feature dimensions, like color and motion. This is advantageous because broadly tuned neurons can contribute to the representation of many visual scenes. However, if there are multiple objects in a visual scene, the cortex is at risk to combine features of different objects as if they belong to a single object. The term "binding problem" was introduced to refer to the difficulties that may occur in sorting out those responses that are evoked by a single perceptual object. The present article reviews proposals suggesting that the binding problem is solved by labelling an assembly of neurons that is responsive to a single perceptual object. Evidence is reviewed in favor of two possible assembly-labels: rate enhancement due to visual attention and neuronal synchrony. Assembly-labels should be spread through the cortical network to all neurons that have to participate in an assembly. The present article tries to shed light on the mechanisms that subserve such a selective spread of assembly labels. Moreover, it is suggested that assembly labels may fulfill an equivalent role in the motor system, since binding problems can also occur during the generation of useful patterns of motor activity.

Detecting connectedness.

Natural visual images are typically composed of multiple objects, which need to be segregated from each other and from the background. The visual system has evolved to capture a great variety of cues that allow a meaningful segmentation of the visual input. One of these cues is connectedness. Connected image regions are likely to belong to a single visual object, whereas disconnected image regions typically belong to different objects. The visual system should therefore be rather proficient in recovering connected image regions. In the present article we will review evidence in favour of an important role of connectedness detection for figure-ground segmentation, and speculate on the physiological mechanisms that allow the visual system to perform this non-trivial task. We argue that biologically plausible feedforward networks are maladapted for the detection of connectedness. It is proposed that neurons that respond to connected image regions are linked by a network of recurrent connections that we call the interaction skeleton. Neurons spread a tag through the interaction skeleton, which labels cells that respond to the same perceptual object. Tag-spreading costs time and is therefore inconsistent with extremely rapid object recognition. We will discuss the pros and cons of two such tags: synchrony and rate modulation.

Synchronization of oscillatory responses in visual cortex correlates with perception in interocular rivalry.

In subjects suffering from early onset strabismus, signals conveyed by the two eyes are not perceived simultaneously but in alternation. We exploited this phenomenon of interocular suppression to investigate the neuronal correlate of binocular rivalry in primary visual cortex of awake strabismic cats. Monocularly presented stimuli that were readily perceived by the animal evoked synchronized discharges with an oscillatory patterning in the gamma-frequency range. Upon dichoptic stimulation, neurons responding to the stimulus that continued to be perceived increased the synchronicity and the regularity of their oscillatory patterning while the reverse was true for neurons responding to the stimulus that was no longer perceived. These differential changes were not associated with modifications of discharge rate, suggesting that at early stages of visual processing the degree of synchronicity rather than the amplitude of responses determines which signals are perceived and control behavioral responses.

Role of the temporal domain for response selection and perceptual binding.

Most cognitive functions are based on highly parallel and distributed information processing by the brain. A paradigmatic example is provided by the vertebrate visual system where numerous cortical areas have been described which analyse different types of visual information. At present, it is unclear how information can be integrated and how coherent representational states can be established in such distributed systems. We suggest that this so-called 'binding problem' may be solved in the temporal domain. The hypothesis is that synchronization of neuronal discharges can serve for the integration of distributed neurons into cell assemblies and that this process may underlie the selection of perceptually and behaviourally relevant information. We review experimental results, mainly obtained in the visual system, which support this temporal binding hypothesis.

Visuomotor integration is associated with zero time-lag synchronization among cortical areas.

Information processing in the cerebral cortex invariably involves the activation of millions of neurons that are widely distributed over its various areas. These distributed activity patterns need to be integrated into coherent representational states. A candidate mechanism for the integration and coordination of neuronal activity between different brain regions is synchronization on a fine temporal scale. In the visual cortex, synchronization occurs selectively between the responses of neurons that represent related features and that need to be integrated for the generation of coherent percepts; neurons in other areas of the cerebral cortex also synchronize their discharges. However, little is known about the patterns and the behavioural correlates of synchrony among widely separated cortical regions. Here we report that synchronization occurs between areas of the visual and parietal cortex, and between areas of the parietal and motor cortex, in the awake cat. When cats responded to a sudden change of a visual pattern, neuronal activity in cortical areas exhibited synchrony without time lags; this synchrony was particularly strong between areas subserving related functions. During reward and inter-trial episodes, zero-time-lag synchrony was lost and replaced by interactions exhibiting large and unsystematic time lags.

Neuronal assemblies: necessity, signature and detectability.

The ease with which highly developed brains can generate representations of a virtually unlimited diversity of perceptual objects indicates that they have developed very efficient mechanisms to analyse and represent relations among incoming signals. Here, we propose that two complementary strategies are applied to cope with these combinatorial problems. First, elementary relations are represented by the tuned responses of individual neurons that acquire their specific response properties (feature selectivity) through appropriate convergence of input connections in hierarchically structured feed-forward architectures. Second, complex relations that cannot be represented economically by the responses of individual neurons are represented by assemblies of cells that are generated by dynamic association of individual, featureselective cells. The signature identifying the responses of an assembly as components of a coherent code is thought to be the synchronicity of the respective discharges. The compatibility of this hypothesis is examined in the context of recent data on the dynamics of synchronization phenomena, the dependence of synchronization on central states and the relations between the synchronization behaviour of neurons and perception.