Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics.

In this critical review, we examine the application of predictive models, for example, classifiers, trained using machine learning (ML) to assist in interpretation of functional neuroimaging data. Our primary goal is to summarize how ML is being applied and critically assess common practices. Our review covers 250 studies published using ML and resting-state functional MRI (fMRI) to infer various dimensions of the human functional connectome. Results for holdout ("lockbox") performance was, on average, ∼13% less accurate than performance measured through cross-validation alone, highlighting the importance of lockbox data, which was included in only 16% of the studies. There was also a concerning lack of transparency across the key steps in training and evaluating predictive models. The summary of this literature underscores the importance of the use of a lockbox and highlights several methodological pitfalls that can be addressed by the imaging community. We argue that, ideally, studies are motivated both by the reproducibility and generalizability of findings as well as the potential clinical significance of the insights. We offer recommendations for principled integration of machine learning into the clinical neurosciences with the goal of advancing imaging biomarkers of brain disorders, understanding causative determinants for health risks, and parsing heterogeneous patient outcomes.

Using attribute amnesia to test the limits of hyper-binding and associative deficits in working memory.

Previous work has shown mixed evidence regarding age-related deficits for binding in working memory. The current study used the newly developed attribute amnesia effect (H. Chen & Wyble, 2015a) to test the associative-deficit hypothesis during working memory and to probe whether hyper-binding extends to include binding of de-selected information. In studies of attribute amnesia, participants use target attributes (e.g., identity, color) to demonstrate near ceiling levels of reporting of a second target attribute (e.g., location) across a series of trials (H. Chen & Wyble, 2015a, 2016). Yet, despite having just processed the target-defining attribute, they have difficulty reporting it on a surprise trial. This effect provides several predictions for associative binding in aging. The associative-deficit hypothesis predicts age-related decline on the surprise trial, whereas an extension of hyper-binding predicts age-related increase in performance in older adults. In Experiment 1, when working memory load was low, older adults demonstrated attribute amnesia equal to that found in younger adults. When load increased in Experiment 2, older adults again demonstrated attribute amnesia as well as an age deficit for reporting target attributes. In lieu of spontaneous binding, results suggest that expectancy plays a critical role in older adults' propensity to encode and bind target attributes in working memory. Results further suggest that expectancy alone is not enough for older adults to form bound representations when task demands are high. Taken together results revealed a boundary condition of hyper-binding and further provided conditional support for the associative-deficit hypothesis in working memory. (PsycINFO Database Record

Adults blink more deeply: a comparative study of the attentional blink across different age groups.

The attentional blink (AB) is thought to help the visual system parse and categorize rapidly changing information by segmenting it into temporal chunks, and is elicited using Rapid Serial Visual Presentation. It is reflected in a decrease in accuracy at detecting the second of two targets presented within 200-500 ms of the first, and its development appears to be protracted on tasks that require set-shifting. Here, younger (M = 8.5 years) and older (M = 12.8 years) children and adults (M = 19.13 years) completed a simple AB task with no set-shift requirement in which participants detected two letters in a stream of numbers presented at a rate of 135 ms/item. In addition to assessing the developmental course of the AB on this simple task, we also assessed temporal order errors, or swaps. The AB and its associated characteristics are present in both groups but developmental differences were noted in the depth of the AB, and the presence or absence of lag-1 sparing. These developmental changes were explained by changes in a single parameter, inhibition, using the eTST model, which suggests that the AB is an adaptive function of the visual system.

Children with Autism Detect Targets at Very Rapid Presentation Rates with Similar Accuracy as Adults.

Enhanced perception may allow for visual search superiority by individuals with Autism Spectrum Disorder (ASD), but does it occur over time? We tested high-functioning children with ASD, typically developing (TD) children, and TD adults in two tasks at three presentation rates (50, 83.3, and 116.7 ms/item) using rapid serial visual presentation. In the Color task, participants detected a purple target letter amongst black letter distractors. In the Category task, participants detected a letter amongst number distractors. Slower rates resulted in higher accuracy. Children with ASD were more accurate than TD children and similar to adults at the fastest rate when detecting color-marked targets, indicating atypical neurodevelopment in ASD may cause generalized perceptual enhancement relative to typically developing peers.

Analysis of theta power in hippocampal EEG during bar pressing and running behavior in rats during distinct behavioral contexts.

These experiments examine changes in theta power as measured by wavelet analysis in five rats performing a conditional visual discrimination task and a simple running task. In the conditional task, rats were trained to press one lever to initiate a trial and then to press one of two choice levers, each corresponding to one of two cue lights. Analysis of theta power in this operant task found a large decrease in theta power during the choice bar presses, in contrast to the increase in theta power during trial initiation bar presses. This result seems to stand counter to results that propose consistent relationships between motor actions and theta power (Vanderwolf, EEG Clin Neurophys 26:407-418, 1969), as well as studies suggesting that the lack of bar-press theta is the result of habituation. However, these data can be seen as being in broad agreement with the theoretical framework of sensorimotor integration (Bland and Oddie, Behav Brain Res 127:119-136, 2001). To investigate further the power of theta observed at the termination of type 1 motor activity, a runway task was devised in which rats ran back and forth between two ends of a linear track, one of which was always rewarded and the other never rewarded. Theta power decreased sharply 240 ms before movement ended at the rewarded end, but not at the unrewarded end of the track. These data extend the current scope of theory in demonstrating that hippocampal theta activity can end abruptly 200-400 ms prior to the end of type 1 motor movement when approaching the end of a motor sequence.

Stimulation in hippocampal region CA1 in behaving rats yields long-term potentiation when delivered to the peak of theta and long-term depression when delivered to the trough.

Experimental evidence suggests that the hippocampal theta rhythm plays a critical role in learning. Previous studies have shown long-term potentiation (LTP) to be preferentially induced with stimulation on the peak of local theta rhythm in region CA1 in anesthetized rats and with stimulation of the perforant path at the peak of theta in both anesthetized and behaving animals. We set out to determine the effects of tetanic burst stimulation in stratum radiatum of region CA1 in awake behaving animals, delivered during either the peak or the trough of the theta rhythm in the EEG. Bursts delivered to the peak resulted in an increase of 17.9 +/- 0.94% in potential slope. When identical stimulation bursts were delivered to the trough of local theta waves, the potential slope decreased 12.9 +/- 1.03%. This is the first report of LTP being preferentially induced at the peak of local theta rhythm in behaving animals in region CA1 and that LTD was found in response to tetanic stimulation at the trough of the local theta wave. The results are discussed within the framework of a recent theory that proposes that the theta rhythm sets the dynamics for alternating phases of encoding and retrieval (Hasselmo et al., 20021).

A proposed function for hippocampal theta rhythm: separate phases of encoding and retrieval enhance reversal of prior learning.

The theta rhythm appears in the rat hippocampal electroencephalogram during exploration and shows phase locking to stimulus acquisition. Lesions that block theta rhythm impair performance in tasks requiring reversal of prior learning, including reversal in a T-maze, where associations between one arm location and food reward need to be extinguished in favor of associations between the opposite arm location and food reward. Here, a hippocampal model shows how theta rhythm could be important for reversal in this task by providing separate functional phases during each 100-300 msec cycle, consistent with physiological data. In the model, effective encoding of new associations occurs in the phase when synaptic input from entorhinal cortex is strong and long-term potentiation (LTP) of excitatory connections arising from hippocampal region CA3 is strong, but synaptic currents arising from region CA3 input are weak (to prevent interference from prior learned associations). Retrieval of old associations occurs in the phase when entorhinal input is weak and synaptic input from region CA3 is strong, but when depotentiation occurs at synapses from CA3 (to allow extinction of prior learned associations that do not match current input). These phasic changes require that LTP at synapses arising from region CA3 should be strongest at the phase when synaptic transmission at these synapses is weakest. Consistent with these requirements, our recent data show that synaptic transmission in stratum radiatum is weakest at the positive peak of local theta, which is when previous data show that induction of LTP is strongest in this layer.