Consistent spectro-spatial features of human ECoG successfully decode naturalistic behavioral states.

Objective: Understanding the neural correlates of naturalistic behavior is critical for extending and confirming the results obtained from trial-based experiments and designing generalizable brain-computer interfaces that can operate outside laboratory environments. In this study, we aimed to pinpoint consistent spectro-spatial features of neural activity in humans that can discriminate between naturalistic behavioral states. Approach: We analyzed data from five participants using electrocorticography (ECoG) with broad spatial coverage. Spontaneous and naturalistic behaviors such as "Talking" and "Watching TV" were labeled from manually annotated videos. Linear discriminant analysis (LDA) was used to classify the two behavioral states. The parameters learned from the LDA were then used to determine whether the neural signatures driving classification performance are consistent across the participants. Main results: Spectro-spatial feature values were consistently discriminative between the two labeled behavioral states across participants. Mainly, θ, α, and low and high γ in the postcentral gyrus, precentral gyrus, and temporal lobe showed significant classification performance and feature consistency across participants. Subject-specific performance exceeded 70%. Combining neural activity from multiple cortical regions generally does not improve decoding performance, suggesting that information regarding the behavioral state is non-additive as a function of the cortical region. Significance: To the best of our knowledge, this is the first attempt to identify specific spectro-spatial neural correlates that consistently decode naturalistic and active behavioral states. The aim of this work is to serve as an initial starting point for developing brain-computer interfaces that can be generalized in a realistic setting and to further our understanding of the neural correlates of naturalistic behavior in humans.

Spatiotemporal dynamics of human high gamma discriminate naturalistic behavioral states.

In analyzing the neural correlates of naturalistic and unstructured behaviors, features of neural activity that are ignored in a trial-based experimental paradigm can be more fully studied and investigated. Here, we analyze neural activity from two patients using electrocorticography (ECoG) and stereo-electroencephalography (sEEG) recordings, and reveal that multiple neural signal characteristics exist that discriminate between unstructured and naturalistic behavioral states such as "engaging in dialogue" and "using electronics". Using the high gamma amplitude as an estimate of neuronal firing rate, we demonstrate that behavioral states in a naturalistic setting are discriminable based on long-term mean shifts, variance shifts, and differences in the specific neural activity's covariance structure. Both the rapid and slow changes in high gamma band activity separate unstructured behavioral states. We also use Gaussian process factor analysis (GPFA) to show the existence of salient spatiotemporal features with variable smoothness in time. Further, we demonstrate that both temporally smooth and stochastic spatiotemporal activity can be used to differentiate unstructured behavioral states. This is the first attempt to elucidate how different neural signal features contain information about behavioral states collected outside the conventional experimental paradigm.

High γ Activity in Cortex and Hippocampus Is Correlated with Autonomic Tone during Sleep.

Studies in animals have demonstrated a strong relationship between cortical and hippocampal activity, and autonomic tone. However, the extent, distribution, and nature of this relationship have not been investigated with intracranial recordings in humans during sleep. Cortical and hippocampal population neuronal firing was estimated from high γ band activity (HG) from 70 to 110 Hz in local field potentials (LFPs) recorded from 15 subjects (nine females) during nonrapid eye movement (NREM) sleep. Autonomic tone was estimated from heart rate variability (HRV). HG and HRV were significantly correlated in the hippocampus and multiple cortical sites in NREM stages N1-N3. The average correlation between HG and HRV could be positive or negative across patients given anatomic location and sleep stage and was most profound in lateral temporal lobe in N3, suggestive of greater cortical activity associated with sympathetic tone. Patient-wide correlation was related to δ band activity (1-4 Hz), which is known to be correlated with high γ activity during sleep. The percentage of statistically correlated channels was weaker in N1 and N2 as compared with N3, and was strongest in regions that have previously been associated with autonomic processes, such as anterior hippocampus and insula. The anatomic distribution of HRV-HG correlations during sleep did not reproduce those usually observed with positron emission tomography (PET) or functional magnetic resonance imaging (fMRI) during waking. This study aims to characterize the relationship between autonomic tone and neuronal firing rate during sleep and further studies are needed to investigate finer temporal resolutions, denser coverages, and different frequency bands in both waking and sleep.

Coarse behavioral context decoding.

OBJECTIVE: Current brain-computer interface (BCI) studies demonstrate the potential to decode neural signals obtained from structured and trial-based tasks to drive actuators with high performance within the context of these tasks. Ideally, to maximize utility, such systems will be applied to a wide range of behavioral settings or contexts. Thus, we explore the potential to augment such systems with the ability to decode abstract behavioral contextual states from neural activity. APPROACH: To demonstrate the feasibility of such context decoding, we used electrocorticography (ECoG) and stereo-electroencephalography (sEEG) data recorded from the cortical surface and deeper brain structures, respectively, continuously across multiple days from three subjects. During this time, the subjects were engaged in a range of naturalistic behaviors in a hospital environment. Behavioral contexts were labeled manually from video and audio recordings; four states were considered: engaging in dialogue, rest, using electronics, and watching television. We decode these behaviors using a factor analysis and support vector machine (SVM) approach. MAIN RESULTS: We demonstrate that these general behaviors can be decoded with high accuracies of 73% for a four-class classifier for one subject and 71% and 62% for a three-class classifier for two subjects. SIGNIFICANCE: To our knowledge, this is the first demonstration of the potential to disambiguate abstract naturalistic behavioral contexts from neural activity recorded throughout the day from implanted electrodes. This work motivates further study of context decoding for BCI applications using continuously recorded naturalistic activity in the clinical setting.

Patient-Specific Pose Estimation in Clinical Environments.

Reliable posture labels in hospital environments can augment research studies on neural correlates to natural behaviors and clinical applications that monitor patient activity. However, many existing pose estimation frameworks are not calibrated for these unpredictable settings. In this paper, we propose a semi-automated approach for improving upper-body pose estimation in noisy clinical environments, whereby we adapt and build around an existing joint tracking framework to improve its robustness to environmental uncertainties. The proposed framework uses subject-specific convolutional neural network models trained on a subset of a patient's RGB video recording chosen to maximize the feature variance of each joint. Furthermore, by compensating for scene lighting changes and by refining the predicted joint trajectories through a Kalman filter with fitted noise parameters, the extended system yields more consistent and accurate posture annotations when compared with the two state-of-the-art generalized pose tracking algorithms for three hospital patients recorded in two research clinics.