Theory of optimal balance predicts and explains the amplitude and decay time of synaptic inhibition.

Synaptic inhibition counterbalances excitation, but it is not known what constitutes optimal inhibition. We previously proposed that perfect balance is achieved when the peak of an excitatory postsynaptic potential (EPSP) is exactly at spike threshold, so that the slightest variation in excitation determines whether a spike is generated. Using simulations, we show that the optimal inhibitory postsynaptic conductance (IPSG) increases in amplitude and decay rate as synaptic excitation increases from 1 to 800 Hz. As further proposed by theory, we show that optimal IPSG parameters can be learned through anti-Hebbian rules. Finally, we compare our theoretical optima to published experimental data from 21 types of neurons, in which rates of synaptic excitation and IPSG decay times vary by factors of about 100 (5-600 Hz) and 50 (1-50 ms), respectively. From an infinite range of possible decay times, theory predicted experimental decay times within less than a factor of 2. Across a distinct set of 15 types of neuron recorded in vivo, theory predicted the amplitude of synaptic inhibition within a factor of 1.7. Thus, the theory can explain biophysical quantities from first principles.

T-type calcium channels cause bursts of spikes in motor but not sensory thalamic neurons during mimicry of natural patterns of synaptic input.

Although neurons within intact nervous systems can be classified as 'sensory' or 'motor,' it is not known whether there is any general distinction between sensory and motor neurons at the cellular or molecular levels. Here, we extend and test a theory according to which activation of certain subtypes of voltage-gated ion channel (VGC) generate patterns of spikes in neurons of motor systems, whereas VGC are proposed to counteract patterns in sensory neurons. We previously reported experimental evidence for the theory from visual thalamus, where we found that T-type calcium channels (TtCCs) did not cause bursts of spikes but instead served the function of 'predictive homeostasis' to maximize the causal and informational link between retinogeniculate excitation and spike output. Here, we have recorded neurons in brain slices from eight sensory and motor regions of rat thalamus while mimicking key features of natural excitatory and inhibitory post-synaptic potentials. As predicted by theory, TtCC did cause bursts of spikes in motor thalamus. TtCC-mediated responses in motor thalamus were activated at more hyperpolarized potentials and caused larger depolarizations with more spikes than in visual and auditory thalamus. Somatosensory thalamus is known to be more closely connected to motor regions relative to auditory and visual thalamus, and likewise the strength of its TtCC responses was intermediate between these regions and motor thalamus. We also observed lower input resistance, as well as limited evidence of stronger hyperpolarization-induced ('H-type') depolarization, in nuclei closer to motor output. These findings support our theory of a specific difference between sensory and motor neurons at the cellular level.

T-type calcium channels promote predictive homeostasis of input-output relations in thalamocortical neurons of lateral geniculate nucleus.

A general theory views the function of all neurons as prediction, and one component of this theory is that of "predictive homeostasis" or "prediction error." It is well established that sensory systems adapt so that neuronal output maintains sensitivity to sensory input, in accord with information theory. Predictive homeostasis applies the same principle at the cellular level, where the challenge is to maintain membrane excitability at the optimal homeostatic level so that spike generation is maximally sensitive to small gradations in synaptic drive. Negative feedback is a hallmark of homeostatic mechanisms, as exemplified by depolarization-activated potassium channels. In contrast, T-type calcium channels exhibit positive feedback that appears at odds with the theory. In thalamocortical neurons of lateral geniculate nucleus (LGN), T-type channels are capable of causing bursts of spikes with an all-or-none character in response to excitation from a hyperpolarized potential. This "burst mode" would partially uncouple visual input from spike output and reduce the information spikes convey about gradations in visual input. However, past observations of T-type-driven bursts may have resulted from unnaturally high membrane excitability. Here we have mimicked within rat brain slices the patterns of synaptic conductance that occur naturally during vision. In support of the theory of predictive homeostasis, we found that T-type channels restored excitability toward its homeostatic level during periods of hyperpolarization. Thus, activation of T-type channels allowed two retinal input spikes to cause one output spike on average, and we observed almost no instances in which output count exceeded input count (a "burst"). T-type calcium channels therefore help to maintain a single optimal mode of transmission rather than creating a second mode. More fundamentally our results support the general theory, which seeks to predict the properties of a neuron's ion channels and synapses given knowledge of natural patterns of synaptic input.

The meaning of spikes from the neuron's point of view: predictive homeostasis generates the appearance of randomness.

The conventional interpretation of spikes is from the perspective of an external observer with knowledge of a neuron's inputs and outputs who is ignorant of the contents of the "black box" that is the neuron. Here we consider a neuron to be an observer and we interpret spikes from the neuron's perspective. We propose both a descriptive hypothesis based on physics and logic, and a prescriptive hypothesis based on biological optimality. Our descriptive hypothesis is that a neuron's membrane excitability is "known" and the amplitude of a future excitatory postsynaptic conductance (EPSG) is "unknown". Therefore excitability is an expectation of EPSG amplitude and a spike is generated only when EPSG amplitude exceeds its expectation ("prediction error"). Our prescriptive hypothesis is that a diversity of synaptic inputs and voltage-regulated ion channels implement "predictive homeostasis", working to insure that the expectation is accurate. The homeostatic ideal and optimal expectation would be achieved when an EPSP reaches precisely to spike threshold, so that spike output is exquisitely sensitive to small variations in EPSG input. To an external observer who knows neither EPSG amplitude nor membrane excitability, spikes would appear random if the neuron is making accurate predictions. We review experimental evidence that spike probabilities are indeed maintained near an average of 0.5 under natural conditions, and we suggest that the same principles may also explain why synaptic vesicle release appears to be "stochastic". Whereas the present hypothesis accords with principles of efficient coding dating back to Barlow (1961), it contradicts decades of assertions that neural activity is substantially "random" or "noisy". The apparent randomness is by design, and like many other examples of apparent randomness, it corresponds to the ignorance of external macroscopic observers about the detailed inner workings of a microscopic system.

Two dimensions of value: dopamine neurons represent reward but not aversiveness.

Whereas reward (appetitiveness) and aversiveness (punishment) have been distinguished as two discrete dimensions within psychology and behavior, physiological and computational models of their neural representation have treated them as opposite sides of a single continuous dimension of "value." Here, I show that although dopamine neurons of the primate ventral midbrain are activated by evidence for reward and suppressed by evidence against reward, they are insensitive to aversiveness. This indicates that reward and aversiveness are represented independently as two dimensions, even by neurons that are closely related to motor function. Because theory and experiment support the existence of opponent neural representations for value, the present results imply four types of value-sensitive neurons corresponding to reward-ON (dopamine), reward-OFF, aversive-ON, and aversive-OFF.

Diversity and homogeneity in responses of midbrain dopamine neurons.

Dopamine neurons of the ventral midbrain have been found to signal a reward prediction error that can mediate positive reinforcement. Despite the demonstration of modest diversity at the cellular and molecular levels, there has been little analysis of response diversity in behaving animals. Here we examine response diversity in rhesus macaques to appetitive, aversive, and neutral stimuli having relative motivational values that were measured and controlled through a choice task. First, consistent with previous studies, we observed a continuum of response variability and an apparent absence of distinct clusters in scatter plots, suggesting a lack of statistically discrete subpopulations of neurons. Second, we found that a group of "sensitive" neurons tend to be more strongly suppressed by a variety of stimuli and to be more strongly activated by juice. Third, neurons in the "ventral tier" of substantia nigra were found to have greater suppression, and a subset of these had higher baseline firing rates and late "rebound" activation after suppression. These neurons could belong to a previously identified subgroup of dopamine neurons that express high levels of H-type cation channels but lack calbindin. Fourth, neurons further rostral exhibited greater suppression. Fifth, although we observed weak activation of some neurons by aversive stimuli, this was not associated with their aversiveness. In conclusion, we find a diversity of response properties, distributed along a continuum, within what may be a single functional population of neurons signaling reward prediction error.

Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli.

The transient response of dopamine neurons has been described as reward prediction error (RPE), with activation or suppression by events that are better or worse than expected, respectively. However, at least a minority of neurons are activated by aversive or high-intensity stimuli, casting doubt on the generality of RPE in describing the dopamine signal. To overcome limitations of previous studies, we studied neuronal responses to a wider variety of high-intensity and aversive stimuli, and we quantified and controlled aversiveness through a choice task in which macaques sacrificed juice to avoid aversive stimuli. Whereas most previous work has portrayed the RPE as a single impulse or "phase," here we demonstrate its multiphasic temporal dynamics. Aversive or high-intensity stimuli evoked a triphasic sequence of activation-suppression-activation extending over a period of 40-700 ms. The initial activation at short latencies (40-120 ms) reflected sensory intensity. The influence of motivational value became dominant between 150 and 250 ms, with activation in the case of appetitive stimuli, and suppression in the case of aversive and neutral stimuli. The previously unreported late activation appeared to be a modest "rebound" after strong suppression. Similarly, strong activation by reward was often followed by suppression. We suggest that these "rebounds" may result from overcompensation by homeostatic mechanisms in some cells. Our results are consistent with a realistic RPE, which evolves over time through a dynamic balance of excitation and inhibition.

Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement.

Activation of dopamine receptors in forebrain regions, for minutes or longer, is known to be sufficient for positive reinforcement of stimuli and actions. However, the firing rate of dopamine neurons is increased for only about 200 milliseconds following natural reward events that are better than expected, a response which has been described as a "reward prediction error" (RPE). Although RPE drives reinforcement learning (RL) in computational models, it has not been possible to directly test whether the transient dopamine signal actually drives RL. Here we have performed optical stimulation of genetically targeted ventral tegmental area (VTA) dopamine neurons expressing Channelrhodopsin-2 (ChR2) in mice. We mimicked the transient activation of dopamine neurons that occurs in response to natural reward by applying a light pulse of 200 ms in VTA. When a single light pulse followed each self-initiated nose poke, it was sufficient in itself to cause operant reinforcement. Furthermore, when optical stimulation was delivered in separate sessions according to a predetermined pattern, it increased locomotion and contralateral rotations, behaviors that are known to result from activation of dopamine neurons. All three of the optically induced operant and locomotor behaviors were tightly correlated with the number of VTA dopamine neurons that expressed ChR2, providing additional evidence that the behavioral responses were caused by activation of dopamine neurons. These results provide strong evidence that the transient activation of dopamine neurons provides a functional reward signal that drives learning, in support of RL theories of dopamine function.

Towards a general theory of neural computation based on prediction by single neurons.

Although there has been tremendous progress in understanding the mechanics of the nervous system, there has not been a general theory of its computational function. Here I present a theory that relates the established biophysical properties of single generic neurons to principles of Bayesian probability theory, reinforcement learning and efficient coding. I suggest that this theory addresses the general computational problem facing the nervous system. Each neuron is proposed to mirror the function of the whole system in learning to predict aspects of the world related to future reward. According to the model, a typical neuron receives current information about the state of the world from a subset of its excitatory synaptic inputs, and prior information from its other inputs. Prior information would be contributed by synaptic inputs representing distinct regions of space, and by different types of non-synaptic, voltage-regulated channels representing distinct periods of the past. The neuron's membrane voltage is proposed to signal the difference between current and prior information ("prediction error" or "surprise"). A neuron would apply a Hebbian plasticity rule to select those excitatory inputs that are the most closely correlated with reward but are the least predictable, since unpredictable inputs provide the neuron with the most "new" information about future reward. To minimize the error in its predictions and to respond only when excitation is "new and surprising," the neuron selects amongst its prior information sources through an anti-Hebbian rule. The unique inputs of a mature neuron would therefore result from learning about spatial and temporal patterns in its local environment, and by extension, the external world. Thus the theory describes how the structure of the mature nervous system could reflect the structure of the external world, and how the complexity and intelligence of the system might develop from a population of undifferentiated neurons, each implementing similar learning algorithms.

Explicit neural signals reflecting reward uncertainty.

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.

The temporal precision of reward prediction in dopamine neurons.

Midbrain dopamine neurons are activated when reward is greater than predicted, and this error signal could teach target neurons both the value of reward and when it will occur. We used the dopamine error signal to measure how the expectation of reward was distributed over time. Animals were trained with fixed-duration intervals of 1-16 s between conditioned stimulus onset and reward. In contrast to the weak responses that have been observed after short intervals (1-2 s), activations to reward increased steeply and linearly with the logarithm of the interval. Results with varied stimulus-reward intervals suggest that the neural expectation was substantial after just half an interval had elapsed. Thus, the neural expectation of reward in these experiments was not highly precise and the precision declined sharply with interval duration. The neural precision of expectation appeared to be at least qualitatively similar to the precision of anticipatory licking behavior.

Adaptive coding of reward value by dopamine neurons.

It is important for animals to estimate the value of rewards as accurately as possible. Because the number of potential reward values is very large, it is necessary that the brain's limited resources be allocated so as to discriminate better among more likely reward outcomes at the expense of less likely outcomes. We found that midbrain dopamine neurons rapidly adapted to the information provided by reward-predicting stimuli. Responses shifted relative to the expected reward value, and the gain adjusted to the variance of reward value. In this way, dopamine neurons maintained their reward sensitivity over a large range of reward values.

Discrete coding of reward probability and uncertainty by dopamine neurons.

Uncertainty is critical in the measure of information and in assessing the accuracy of predictions. It is determined by probability P, being maximal at P = 0.5 and decreasing at higher and lower probabilities. Using distinct stimuli to indicate the probability of reward, we found that the phasic activation of dopamine neurons varied monotonically across the full range of probabilities, supporting past claims that this response codes the discrepancy between predicted and actual reward. In contrast, a previously unobserved response covaried with uncertainty and consisted of a gradual increase in activity until the potential time of reward. The coding of uncertainty suggests a possible role for dopamine signals in attention-based learning and risk-taking behavior.