Is N-Hacking Ever OK? The consequences of collecting more data in pursuit of statistical significance.

Upon completion of an experiment, if a trend is observed that is "not quite significant," it can be tempting to collect more data in an effort to achieve statistical significance. Such sample augmentation or "N-hacking" is condemned because it can lead to an excess of false positives, which can reduce the reproducibility of results. However, the scenarios used to prove this rule tend to be unrealistic, assuming the addition of unlimited extra samples to achieve statistical significance, or doing so when results are not even close to significant; an unlikely situation for most experiments involving patient samples, cultured cells, or live animals. If we were to examine some more realistic scenarios, could there be any situations where N-hacking might be an acceptable practice? This Essay aims to address this question, using simulations to demonstrate how N-hacking causes false positives and to investigate whether this increase is still relevant when using parameters based on real-life experimental settings.

Different Forms of Variability Could Explain a Difference Between Human and Rat Decision Making.

When observers make rapid, difficult perceptual decisions, their response time is highly variable from trial to trial. In a visual motion discrimination task, it has been reported that human accuracy declines with increasing response time, whereas rat accuracy increases with response time. This is of interest because different mathematical theories of decision-making differ in their predictions regarding the correlation of accuracy with response time. On the premise that perceptual decision-making mechanisms are likely to be conserved among mammals, we seek to unify the rodent and primate results in a common theoretical framework. We show that a bounded drift diffusion model (DDM) can explain both effects with variable parameters: trial-to-trial variability in the starting point of the diffusion process produces the pattern typically observed in rats, whereas variability in the drift rate produces the pattern typically observed in humans. We further show that the same effects can be produced by deterministic biases, even in the absence of parameter stochasticity or parameter change within a trial.

Rational regulation of water-seeking effort in rodents.

In the laboratory, animals' motivation to work tends to be positively correlated with reward magnitude. But in nature, rewards earned by work are essential to survival (e.g., working to find water), and the payoff of that work can vary on long timescales (e.g., seasonally). Under these constraints, the strategy of working less when rewards are small could be fatal. We found that instead, rats in a closed economy did more work for water rewards when the rewards were stably smaller, a phenomenon also observed in human labor supply curves. Like human consumers, rats showed elasticity of demand, consuming far more water per day when its price in effort was lower. The neural mechanisms underlying such "rational" market behaviors remain largely unexplored. We propose a dynamic utility maximization model that can account for the dependence of rat labor supply (trials/day) on the wage rate (milliliter/trial) and also predict the temporal dynamics of when rats work. Based on data from mice, we hypothesize that glutamatergic neurons in the subfornical organ in lamina terminalis continuously compute the instantaneous marginal utility of voluntary work for water reward and causally determine the amount and timing of work.

The Interaction Between Elapsed Time and Decision Accuracy Differs Between Humans and Rats.

A stochastic visual motion discrimination task is widely used to study rapid decision-making in humans and animals. Among trials of the same sensory difficulty within a block of fixed decision strategy, humans and monkeys are widely reported to make more errors in the individual trials with longer reaction times. This finding has posed a challenge for the drift-diffusion model of sensory decision-making, which in its basic form predicts that errors and correct responses should have the same reaction time distributions. We previously reported that rats also violate this model prediction, but in the opposite direction: for rats, motion discrimination accuracy was highest in the trials with the longest reaction times. To rule out task differences as the cause of our divergent finding in rats, the present study tested humans and rats using the same task and analyzed their data identically. We confirmed that rats' accuracy increased with reaction time, whereas humans' accuracy decreased with reaction time in the same task. These results were further verified using a new temporally local analysis method, ruling out that the observed trend was an artifact of non-stationarity in the data of either species. The main effect was found whether the signal strength (motion coherence) was varied in randomly interleaved trials or held constant within a block. The magnitude of the effects increased with motion coherence. These results provide new constraints useful for refining and discriminating among the many alternative mathematical theories of decision-making.

Training Rats Using Water Rewards Without Water Restriction.

High-throughput behavioral training of rodents has been a transformative development for systems neuroscience. Water or food restriction is typically required to motivate task engagement. We hypothesized a gap between physiological water need and hedonic water satiety that could be leveraged to train rats for water rewards without water restriction. We show that when Citric Acid (CA) is added to water, female rats drink less, yet consume enough to maintain long term health. With 24 h/day access to a visual task with water rewards, rats with ad lib CA water performed 84% ± 18% as many trials as in the same task under water restriction. In 2-h daily sessions, rats with ad lib CA water performed 68% ± 13% as many trials as under water restriction. Using reward sizes <25 µl, rats with ad lib CA performed 804 ± 285 trials/day in live-in sessions or 364 ± 82 trials/day in limited duration daily sessions. The safety of CA water amendment was previously shown for male rats, and the gap between water need and satiety was similar to what we observed in females. Therefore, it is likely that this method will generalize to male rats, though this remains to be shown. We conclude that at least in some contexts rats can be trained using water rewards without water restriction, benefitting both animal welfare and scientific productivity.

Temporal and spatial tuning of dorsal lateral geniculate nucleus neurons in unanesthetized rats.

Visual response properties of neurons in the dorsolateral geniculate nucleus (dLGN) have been well described in several species, but not in rats. Analysis of responses from the unanesthetized rat dLGN will be needed to develop quantitative models that account for visual behavior of rats. We recorded visual responses from 130 single units in the dLGN of 7 unanesthetized rats. We report the response amplitudes, temporal frequency, and spatial frequency sensitivities in this population of cells. In response to 2-Hz visual stimulation, dLGN cells fired 15.9 ± 11.4 spikes/s (mean ± SD) modulated by 10.7 ± 8.4 spikes/s about the mean. The optimal temporal frequency for full-field stimulation ranged from 5.8 to 19.6 Hz across cells. The temporal high-frequency cutoff ranged from 11.7 to 33.6 Hz. Some cells responded best to low temporal frequency stimulation (low pass), and others were strictly bandpass; most cells fell between these extremes. At 2- to 4-Hz temporal modulation, the spatial frequency of drifting grating that drove cells best ranged from 0.008 to 0.18 cycles per degree (cpd) across cells. The high-frequency cutoff ranged from 0.01 to 1.07 cpd across cells. The majority of cells were driven best by the lowest spatial frequency tested, but many were partially or strictly bandpass. We conclude that single units in the rat dLGN can respond vigorously to temporal modulation up to at least 30 Hz and spatial detail up to 1 cpd. Tuning properties were heterogeneous, but each fell along a continuum; we found no obvious clustering into discrete cell types along these dimensions.

Speed and accuracy of visual motion discrimination by rats.

Animals must continuously evaluate sensory information to select the preferable among possible actions in a given context, including the option to wait for more information before committing to another course of action. In experimental sensory decision tasks that replicate these features, reaction time distributions can be informative about the implicit rules by which animals determine when to commit and what to do. We measured reaction times of Long-Evans rats discriminating the direction of motion in a coherent random dot motion stimulus, using a self-paced two-alternative forced-choice (2-AFC) reaction time task. Our main findings are: (1) When motion strength was constant across trials, the error trials had shorter reaction times than correct trials; in other words, accuracy increased with response latency. (2) When motion strength was varied in randomly interleaved trials, accuracy increased with motion strength, whereas reaction time decreased. (3) Accuracy increased with reaction time for each motion strength considered separately, and in the interleaved motion strength experiment overall. (4) When stimulus duration was limited, accuracy improved with stimulus duration, whereas reaction time decreased. (5) Accuracy decreased with response latency after stimulus offset. This was the case for each stimulus duration considered separately, and in the interleaved duration experiment overall. We conclude that rats integrate visual evidence over time, but in this task the time of their response is governed more by elapsed time than by a criterion for sufficient evidence.

Evidence that primary visual cortex is required for image, orientation, and motion discrimination by rats.

The pigmented Long-Evans rat has proven to be an excellent subject for studying visually guided behavior including quantitative visual psychophysics. This observation, together with its experimental accessibility and its close homology to the mouse, has made it an attractive model system in which to dissect the thalamic and cortical circuits underlying visual perception. Given that visually guided behavior in the absence of primary visual cortex has been described in the literature, however, it is an empirical question whether specific visual behaviors will depend on primary visual cortex in the rat. Here we tested the effects of cortical lesions on performance of two-alternative forced-choice visual discriminations by Long-Evans rats. We present data from one highly informative subject that learned several visual tasks and then received a bilateral lesion ablating >90% of primary visual cortex. After the lesion, this subject had a profound and persistent deficit in complex image discrimination, orientation discrimination, and full-field optic flow motion discrimination, compared with both pre-lesion performance and sham-lesion controls. Performance was intact, however, on another visual two-alternative forced-choice task that required approaching a salient visual target. A second highly informative subject learned several visual tasks prior to receiving a lesion ablating >90% of medial extrastriate cortex. This subject showed no impairment on any of the four task categories. Taken together, our data provide evidence that these image, orientation, and motion discrimination tasks require primary visual cortex in the Long-Evans rat, whereas approaching a salient visual target does not.

Speed and accuracy of visual image discrimination by rats.

The trade-off between speed and accuracy of sensory discrimination has most often been studied using sensory stimuli that evolve over time, such as random dot motion discrimination tasks. We previously reported that when rats perform motion discrimination, correct trials have longer reaction times than errors, accuracy increases with reaction time, and reaction time increases with stimulus ambiguity. In such experiments, new sensory information is continually presented, which could partly explain interactions between reaction time and accuracy. The present study shows that a changing physical stimulus is not essential to those findings. Freely behaving rats were trained to discriminate between two static visual images in a self-paced, two-alternative forced-choice reaction time task. Each trial was initiated by the rat, and the two images were presented simultaneously and persisted until the rat responded, with no time limit. Reaction times were longer in correct trials than in error trials, and accuracy increased with reaction time, comparable to results previously reported for rats performing motion discrimination. In the motion task, coherence has been used to vary discrimination difficulty. Here morphs between the previously learned images were used to parametrically vary the image similarity. In randomly interleaved trials, rats took more time on average to respond in trials in which they had to discriminate more similar stimuli. For both the motion and image tasks, the dependence of reaction time on ambiguity is weak, as if rats prioritized speed over accuracy. Therefore we asked whether rats can change the priority of speed and accuracy adaptively in response to a change in reward contingencies. For two rats, the penalty delay was increased from 2 to 6 s. When the penalty was longer, reaction times increased, and accuracy improved. This demonstrates that rats can flexibly adjust their behavioral strategy in response to the cost of errors.

Rats and humans differ in processing collinear visual features.

Behavioral studies in humans and rats demonstrate that visual detection of a target stimulus is sensitive to surrounding spatial patterns. In both species, the detection of an oriented visual target is affected when the surrounding region contains flanking stimuli that are collinear to the target. In many studies, collinear flankers have been shown to improve performance in humans, both absolutely (compared to performance with no flankers) and relative to non-collinear flankers. More recently, collinear flankers have been shown to impair performance in rats both absolutely and relative to non-collinear flankers. However, these observations spanned different experimental paradigms. Past studies in humans have shown that the magnitude and even sign of flanker effects can depend critically on the details of stimulus and task design. Therefore either task differences or species could explain the opposite findings. Here we provide a direct comparison of behavioral data between species and show that these differences persist--collinear flankers improve performance in humans, and impair performance in rats--in spite of controls that match stimuli, experimental paradigm, and learning procedure. There is evidence that the contrasts of the target and the flankers could affect whether surround processing is suppressive or facilitatory. In a second experiment, we explored a range of contrast conditions in the rat, to determine if contrast could explain the lack of collinear facilitation. Using different pairs of target and flanker contrast, the rat's collinear impairment was confirmed to be robust across a range of contrast conditions. We conclude that processing of collinear features is indeed different between rats and humans. We speculate that the observed difference between rat and human is caused by the combined impact of differences in the statistics in natural retinal images, the representational capacity of neurons in visual cortex, and attention.

Dorsal lateral geniculate substructure in the long-evans rat: a cholera toxin B subunit study.

The pigmented rat is an increasingly important model in visual neuroscience research, yet the lamination of retinal projections in the dLGN has not been examined in sufficient detail. From previous studies it was known that most of the rat dLGN receives monocular input from the contralateral eye, with a small island receiving predominantly ipsilateral projections. Here we revisit the question using cholera toxin B subunit, a tracer that efficiently fills retinal terminals after intra-ocular injection. We imaged retinal termini throughout the dLGN at 0.5 µm resolution and traced areas of ipsilateral and contralateral terminals to obtain a high resolution 3D reconstruction of the projection pattern. Retinal termini in the dLGN are well segregated by eye of origin, as expected. We find, however, that the ipsilateral projections form multiple discrete projection zones in three dimensions, not the single island previously described. It remains to be determined whether these subdomains represent distinct functional sublaminae, as is the case in other mammals.

Rat performance on visual detection task modeled with divisive normalization and adaptive decision thresholds.

Performance on any perceptual task depends on both the perceptual capacity and the decision strategy of the subject. We provide a model to fit both aspects and apply it to data from rats performing a detection task. When rats must detect a faint visual target, the presence of other nearby stimuli ("flankers") increases the difficulty of the task. In this study, we consider two specific factors. First, flankers could diminish the sensory response to the target via spatial contrast normalization in early visual processing. Second, rats may treat the sensory signal caused by the flankers as if it belonged to the target. We call this source confusion, which may be sensory, cognitive, or both. We account for contrast normalization and source confusion by fitting model parameters to the likelihood of the observed behavioral data. We test multiple combinations of target and flanker contrasts using a yes/no detection task. Contrast normalization was crucial to explain the rats' flanker-induced detection impairment. By adding a decision variable to the contrast normalization framework, our model provides a new tool to assess differences in visual or cognitive brain function between normal and abnormal rodents.

Collinear features impair visual detection by rats.

We measure rats' ability to detect an oriented visual target grating located between two flanking stimuli ("flankers"). Flankers varied in contrast, orientation, angular position, and sign. Rats are impaired at detecting visual targets with collinear flankers, compared to configurations where flankers differ from the target in orientation or angular position. In particular, rats are more likely to miss the target when flankers are collinear. The same impairment is found even when the flanker luminance was sign-reversed relative to the target. These findings suggest that contour alignment alters visual processing in rats, despite their lack of orientation columns in the visual cortex. This is the first report that the arrangement of visual features relative to each other affects visual behavior in rats. To provide a conceptual framework for our findings, we relate our stimuli to a contrast normalization model of early visual processing. We suggest a pattern-sensitive generalization of the model that could account for a collinear deficit. These experiments were performed using a novel method for automated high-throughput training and testing of visual behavior in rodents.

Intact performance on feature-ambiguous discriminations in rats with lesions of the perirhinal cortex.

We developed a behavioral paradigm for the rat that made it possible to separate the evaluation of memory functions from the evaluation of perceptual functions. Animals were given extensive training on an automated two-choice discrimination task and then maintained their memory performance at a high level while interpolated probe trials tested visual perceptual ability. The probe trials systematically varied the degree of feature ambiguity between the stimuli, such that perceptual functions could be tested across 14 different levels of difficulty. As feature ambiguity increased, performance declined in an orderly, monotonic manner (from 87% correct to chance, 50% correct). Bilateral lesions of the perirhinal cortex fully spared the capacity to make feature-ambiguous discriminations and the performance of lesioned and intact animals was indistinguishable at every difficulty level. In contrast, the perirhinal lesions did impair recognition memory. The findings suggest that the perirhinal cortex is important for memory and not for perceptual functions.

Information measure for analyzing specific spiking patterns and applications to LGN bursts.

Neural spiking responses can include a variety of spiking patterns. However, neither the mere presence of the patterns nor the pattern's frequency indicates that the pattern conveys distinct stimulus information. Here, we present an in-depth analysis of a Pattern Information measure, which quantifies how informative it is to distinguish a particular pattern of spikes from either a single spike or an another pattern. (1) We show how a shuffle-controlled estimation method minimizes the impact of sampling bias. (2) We describe how the Pattern Information could arise from time-varying firing rates, and we demonstrate an analysis to determine whether Pattern Information associated with a particular pattern captures structure not contained in the time-varying firing rate. (3) Because patterns may contain several spikes or inter-spike intervals, we extend the Pattern Information measure to determine whether the complete pattern carries information distinct from sub-patterns containing only a fraction of these spikes or intervals. (4) The Pattern Information is applied to determine whether a plurality of patterns carry distinct stimulus information from one another. In particular, we demonstrate these concepts using data from cells of the lateral geniculate nucleus (LGN), thereby extending previous analysis demonstrating that distinguishes between bursts of spikes and single spikes providing visual information.

The inner life of bursts.

In the thalamus, bursts and single spikes are elicited by distinct visual stimuli, suggesting distinct visual functions. In this issue of Neuron, Wang et al. make use of intracellular recordings of thalamic neurons in vivo to provide a clear, detailed explanation of how natural stimuli are converted into a neural code that uses both bursts and single spikes.

Benefits of contrast normalization demonstrated in neurons and model cells.

The large dynamic range of natural stimuli poses a challenge for neural coding: how is a neuron to encode large differences at high contrast while remaining sensitive to small differences at low contrast? Many sensory neurons exhibit contrast normalization: gain depends on the range of stimuli presented, such that firing-rate modulation is not proportional to contrast. However, coding depends strongly on the precision of spike timing and the reliability of spike number, neither of which can be predicted from neural gain. The presumption that contrast normalization is associated with maintained coding efficiency remained untested. We report that, as contrast decreases, responses are more variable and encode less information, as expected. Nevertheless, these changes can be small, and information transmission is even better preserved across contrasts than rate modulation. The extent of contrast normalization is correlated with the extent to which information transmission is preserved across contrasts. Specifically, normalization is associated with maintaining the bits of information per spike rather than bits per second. Finally, we show that a nonadapting model can exhibit both contrast normalization and the associated information preservation.

Contrast adaptation in a nonadapting LGN model.

Sensory neurons appear to adapt their gain to match the variance of signals along the dimension they encode, a property we shall call "contrast normalization." Contrast normalization has been the subject of extensive physiological and theoretical study. We previously found that neurons in the lateral geniculate nucleus (LGN) exhibit contrast normalization in their responses to full-field flickering white-noise stimuli, and that neurons with the strongest contrast normalization best preserved information transmission across a range of contrasts. We have also shown that both of these properties could be reproduced by nonadapting model cells. Here we present a detailed comparison of this nonadapting model to physiological data from the LGN. First, the model cells recapitulated other contrast dependencies of LGN responses: decreasing stimulus contrast resulted in an increase in spike-timing jitter and spike-number variability. Second, we find that the extent of contrast normalization in this model depends on model parameters related to refractoriness and to noise. Third, we show that the model cells exhibit rapid, transient changes in firing rate just after changes in contrast, and that this is sufficient to produce the transient changes in information transmission that have been reported in other neurons. It is known that intrinsic properties of neurons change during contrast adaptation. Nevertheless the model demonstrates that the spiking nonlinearity of neurons can produce many of the temporal aspects of contrast gain control, including normalization to input variance and transient effects of contrast change.

Visual control of burst priming in the anesthetized lateral geniculate nucleus.

Thalamic relay cells fire bursts of action potentials. Once a long hyperpolarization "primes" (deinactivates) the T-type calcium channel, a depolarizing input will "trigger" a calcium spike with a burst of action potentials. During sleep, bursts are frequent, rhythmic, and nonvisual. Bursts have been observed in alert animals, and burst timing is known to carry visual information under light anesthesia. We extend this finding by showing that bursts without visual triggers are rare. Nevertheless, if the channel were primed at random with respect to the stimulus, then bursts would have the same visual significance as single spikes. We find, however, that visual signals influence when the channel is primed. First, natural time-varying stimuli evoke more bursts than white noise. Second, specific visual stimuli reproducibly elicit bursts, whereas others reliably elicit single spikes. Therefore, visual information is encoded by the selective tagging of some responses as bursts. The visual information attributable to visual priming (as distinct from the information attributable to visual triggering of the bursts) was two bits per burst on average. Although bursts are reportedly rare in alert animals, this must be investigated as a function of visual stimulus. Moreover, we propose methods to measure the extent of both visual triggering and visual priming of bursts. Whether or not bursts are rare, our methods could help determine whether bursts in alert animals carry a distinct visual signal.