Data leakage in deep learning studies of translational EEG.

A growing number of studies apply deep neural networks (DNNs) to recordings of human electroencephalography (EEG) to identify a range of disorders. In many studies, EEG recordings are split into segments, and each segment is randomly assigned to the training or test set. As a consequence, data from individual subjects appears in both the training and the test set. Could high test-set accuracy reflect data leakage from subject-specific patterns in the data, rather than patterns that identify a disease? We address this question by testing the performance of DNN classifiers using segment-based holdout (in which segments from one subject can appear in both the training and test set), and comparing this to their performance using subject-based holdout (where all segments from one subject appear exclusively in either the training set or the test set). In two datasets (one classifying Alzheimer's disease, and the other classifying epileptic seizures), we find that performance on previously-unseen subjects is strongly overestimated when models are trained using segment-based holdout. Finally, we survey the literature and find that the majority of translational DNN-EEG studies use segment-based holdout. Most published DNN-EEG studies may dramatically overestimate their classification performance on new subjects.

Estimating the statistical power to detect set-size effects in contralateral delay activity.

The contralateral delay activity (CDA) is an event-related potential component commonly used to examine the online processes of visual working memory. Here, we provide a robust analysis of the statistical power that is needed to achieve reliable and reproducible results with the CDA. Using two very large EEG datasets that examined the contrast between CDA amplitude with set sizes 2 and 6 items and set sizes 2 and 4 items, we present a subsampling analysis that estimates the statistical power achieved with varying numbers of subjects and trials based on the proportion of significant tests in 10,000 iterations. We also generated simulated data using Bayesian multilevel modeling to estimate power beyond the bounds of the original datasets. The number of trials and subjects required depends critically on the effect size. Detecting the presence of the CDA-a reliable difference between contralateral and ipsilateral electrodes during the memory period-required only 30-50 clean trials with a sample of 25 subjects to achieve approximately 80% statistical power. However, for detecting a difference in CDA amplitude between two set sizes, a substantially larger number of trials and subjects were required; approximately 400 clean trials with 25 subjects to achieve 80% power. Thus, to achieve robust tests of how CDA activity differs across conditions, it is essential to be mindful of the estimated effect size. We recommend researchers designing experiments to detect set-size differences in the CDA collect substantially more trials per subject.

No Evidence for an Object Working Memory Capacity Benefit with Extended Viewing Time.

Visual working memory is the ability to hold visual information temporarily in mind. A key feature of working memory is its starkly limited capacity, such that only a few simple items can be remembered at once. Prior work has shown that this capacity limit cannot be circumvented by providing additional encoding time, whether providing just 200 ms or up to 1300 ms, capacity is still limited to only three to four items. In contrast, Brady et al. (2016) hypothesized that real-world objects, but not simple items used in prior research, benefit from additional encoding time and are not subject to traditional capacity limits. They supported this hypothesis with results from both behavior and the contralateral delay activity (CDA), an EEG marker of working memory storage, and concluded that familiar, complex stimuli are necessary to observe encoding time effects. Here, we conducted three replications of Brady et al.'s key manipulation with a larger number of human participants and more trials per condition. We failed to replicate their primary behavioral result (objects benefit more than colors from additional encoding time) and failed to observe an object-specific increase in the CDA. Instead, we found that encoding time benefitted both simple color items and real-world objects, in contrast to both the findings by Brady et al., and some prior work on this topic. Overall, we observed no support for the hypothesis that real-world objects have a different capacity than colored squares. We discuss the implications of our findings for theories of visual working memory (VWM).

Visual search and the N2pc in children.

While there is growing understanding of visual selective attention in children, some aspects such as selection in the presence of distractors are not well understood. Adult studies suggest that when presented with a visual search task, an enhanced negativity is seen beginning around 200 ms (the N2pc) that reflects selection of a target item among distractors. However, it is not known if similar selective attention-related activity is seen in children during visual search. This study was designed to investigate the presence of the N2pc in children. Nineteen children (ages 9-12 years) and 21 adults (ages 18-22 years) completed a visual search task in which they were asked to attend to a fixation surrounded by both a target and a distractor stimulus. Three types of displays were analyzed at parietal electrodes P7 and P8; lateral target/lateral distractor, lateral target/midline distractor, and midline target/lateral distractor. Both adults and children showed a significant increased negativity contralateral compared to ipsilateral to the target (reflected in the N2pc) in both displays with a lateral target while no such effect was seen in displays with a midline target. This suggests that children also utilized additional resources to select a target item when distractors are present. These findings demonstrate that the N2pc can be used as a marker of attentional object selection in children.