Temporal RIT-Eyes: From Real Infrared Eye-Images to Synthetic Sequences of Gaze Behavior.

Current methods for segmenting eye imagery into skin, sclera, pupil, and iris cannot leverage information about eye motion. This is because the datasets on which models are trained are limited to temporally non-contiguous frames. We present Temporal RIT-Eyes, a Blender pipeline that draws data from real eye videos for the rendering of synthetic imagery depicting natural gaze dynamics. These sequences are accompanied by ground-truth segmentation maps that may be used for training image-segmentation networks. Temporal RIT-Eyes relies on a novel method for the extraction of 3D eyelid pose (top and bottom apex of eyelids/eyeball boundary) from raw eye images for the rendering of gaze-dependent eyelid pose and blink behavior. The pipeline is parameterized to vary in appearance, eye/head/camera/illuminant geometry, and environment settings (indoor/outdoor). We present two open-source datasets of synthetic eye imagery: sGiW is a set of synthetic-image sequences whose dynamics are modeled on those of the Gaze in Wild dataset, and sOpenEDS2 is a series of temporally non-contiguous eye images that approximate the OpenEDS-2019 dataset. We also analyze and demonstrate the quality of the rendered dataset qualitatively and show significant overlap between latent-space representations of the source and the rendered datasets.

Ultrasound for Gaze Estimation-A Modeling and Empirical Study.

Most eye tracking methods are light-based. As such, they can suffer from ambient light changes when used outdoors, especially for use cases where eye trackers are embedded in Augmented Reality glasses. It has been recently suggested that ultrasound could provide a low power, fast, light-insensitive alternative to camera-based sensors for eye tracking. Here, we report on our work on modeling ultrasound sensor integration into a glasses form factor AR device to evaluate the feasibility of estimating eye-gaze in various configurations. Next, we designed a benchtop experimental setup to collect empirical data on time of flight and amplitude signals for reflected ultrasound waves for a range of gaze angles of a model eye. We used this data as input for a low-complexity gradient-boosted tree machine learning regression model and demonstrate that we can effectively estimate gaze (gaze RMSE error of 0.965 ± 0.178 degrees with an adjusted R2 score of 90.2 ± 4.6).

OpenEDS2020 Challenge on Gaze Tracking for VR: Dataset and Results.

This paper summarizes the OpenEDS 2020 Challenge dataset, the proposed baselines, and results obtained by the top three winners of each competition: (1) Gaze prediction Challenge, with the goal of predicting the gaze vector 1 to 5 frames into the future based on a sequence of previous eye images, and (2) Sparse Temporal Semantic Segmentation Challenge, with the goal of using temporal information to propagate semantic eye labels to contiguous eye image frames. Both competitions were based on the OpenEDS2020 dataset, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display with two synchronized eye-facing cameras. The dataset, which we make publicly available for the research community, consists of 87 subjects performing several gaze-elicited tasks, and is divided into 2 subsets, one for each competition task. The proposed baselines, based on deep learning approaches, obtained an average angular error of 5.37 degrees for gaze prediction, and a mean intersection over union score (mIoU) of 84.1% for semantic segmentation. The winning solutions were able to outperform the baselines, obtaining up to 3.17 degrees for the former task and 95.2% mIoU for the latter.

Automated detection and sorting of microencapsulation via machine learning.

Microfluidic-based microencapsulation requires significant oversight to prevent material and quality loss due to sporadic disruptions in fluid flow that routinely arise. State-of-the-art microcapsule production is laborious and relies on experts to monitor the process, e.g. through a microscope. Unnoticed defects diminish the quality of collected material and/or may cause irreversible clogging. To address these issues, we developed an automated monitoring and sorting system that operates on consumer-grade hardware in real-time. Using human-labeled microscope images acquired during typical operation, we train a convolutional neural network that assesses microencapsulation. Based on output from the machine learning algorithm, an integrated valving system collects desirable microcapsules or diverts waste material accordingly. Although the system notifies operators to make necessary adjustments to restore microencapsulation, we can extend the system to automate corrections. Since microfluidic-based production platforms customarily collect image and sensor data, machine learning can help to scale up and improve microfluidic techniques beyond microencapsulation.

Emergent gamma synchrony in all-to-all interneuronal networks.

We investigate the emergence of in-phase synchronization in a heterogeneous network of coupled inhibitory interneurons in the presence of spike timing dependent plasticity (STDP). Using a simple network of two mutually coupled interneurons (2-MCI), we first study the effects of STDP on in-phase synchronization. We demonstrate that, with STDP, the 2-MCI network can evolve to either a state of stable 1:1 in-phase synchronization or exhibit multiple regimes of higher order synchronization states. We show that the emergence of synchronization induces a structural asymmetry in the 2-MCI network such that the synapses onto the high frequency firing neurons are potentiated, while those onto the low frequency firing neurons are de-potentiated, resulting in the directed flow of information from low frequency firing neurons to high frequency firing neurons. Finally, we demonstrate that the principal findings from our analysis of the 2-MCI network contribute to the emergence of robust synchronization in the Wang-Buzsaki network (Wang and Buzsáki, 1996) of all-to-all coupled inhibitory interneurons (100-MCI) for a significantly larger range of heterogeneity in the intrinsic firing rate of the neurons in the network. We conclude that STDP of inhibitory synapses provide a viable mechanism for robust neural synchronization.

Genesis of interictal spikes in the CA1: a computational investigation.

Interictal spikes (IISs) are spontaneous high amplitude, short time duration <400 ms events often observed in electroencephalographs (EEG) of epileptic patients. In vitro analysis of resected mesial temporal lobe tissue from patients with refractory temporal lobe epilepsy has revealed the presence of IIS in the CA1 subfield. In this paper, we develop a biophysically relevant network model of the CA1 subfield and investigate how changes in the network properties influence the susceptibility of CA1 to exhibit an IIS. We present a novel template based approach to identify conditions under which synchronization of paroxysmal depolarization shift (PDS) events evoked in CA1 pyramidal (Py) cells can trigger an IIS. The results from this analysis are used to identify the synaptic parameters of a minimal network model that is capable of generating PDS in response to afferent synaptic input. The minimal network model parameters are then incorporated into a detailed network model of the CA1 subfield in order to address the following questions: (1) How does the formation of an IIS in the CA1 depend on the degree of sprouting (recurrent connections) between the CA1 Py cells and the fraction of CA3 Shaffer collateral (SC) connections onto the CA1 Py cells? and (2) Is synchronous afferent input from the SC essential for the CA1 to exhibit IIS? Our results suggest that the CA1 subfield with low recurrent connectivity (absence of sprouting), mimicking the topology of a normal brain, has a very low probability of producing an IIS except when a large fraction of CA1 neurons (>80%) receives a barrage of quasi-synchronous afferent input (input occurring within a temporal window of ≤24 ms) via the SC. However, as we increase the recurrent connectivity of the CA1 (P sprout > 40); mimicking sprouting in a pathological CA1 network, the CA1 can exhibit IIS even in the absence of a barrage of quasi-synchronous afferents from the SC (input occurring within temporal window >80 ms) and a low fraction of CA1 Py cells (≈30%) receiving SC input. Furthermore, we find that in the presence of Poisson distributed random input via SC, the CA1 network is able to generate spontaneous periodic IISs (≈3 Hz) for high degrees of recurrent Py connectivity (P sprout > 70). We investigate the conditions necessary for this phenomenon and find that spontaneous IISs closely depend on the degree of the network's intrinsic excitability.

Computational modeling of channelrhodopsin-2 photocurrent characteristics in relation to neural signaling.

Channelrhodopsins-2 (ChR2) are a class of light sensitive proteins that offer the ability to use light stimulation to regulate neural activity with millisecond precision. In order to address the limitations in the efficacy of the wild-type ChR2 (ChRwt) to achieve this objective, new variants of ChR2 that exhibit fast mon-exponential photocurrent decay characteristics have been recently developed and validated. In this paper, we investigate whether the framework of transition rate model with 4 states, primarily developed to mimic the biexponential photocurrent decay kinetics of ChRwt, as opposed to the low complexity 3 state model, is warranted to mimic the mono-exponential photocurrent decay kinetics of the newly developed fast ChR2 variants: ChETA (Gunaydin et al., Nature Neurosci. 13:387-392, 2010) and ChRET/TC (Berndt et al., Proc. Natl. Acad. Sci. 108:7595-7600, 2011). We begin by estimating the parameters of the 3-state and 4-state models from experimental data on the photocurrent kinetics of ChRwt, ChETA, and ChRET/TC. We then incorporate these models into a fast-spiking interneuron model (Wang and Buzsaki, J. Neurosci. 16:6402-6413, 1996) and a hippocampal pyramidal cell model (Golomb et al., J. Neurophysiol. 96:1912-1926, 2006) and investigate the extent to which the experimentally observed neural response to various optostimulation protocols can be captured by these models. We demonstrate that for all ChR2 variants investigated, the 4 state model implementation is better able to capture neural response consistent with experiments across wide range of optostimulation protocol. We conclude by analytically investigating the conditions under which the characteristic specific to the 3-state model, namely the monoexponential photocurrent decay of the newly developed variants of ChR2, can occur in the framework of the 4-state model.

Phase shift in the 24-hour rhythm of hippocampal EEG spiking activity in a rat model of temporal lobe epilepsy.

For over a century epileptic seizures have been known to cluster at specific times of the day. Recent studies have suggested that the circadian regulatory system may become permanently altered in epilepsy, but little is known about how this affects neural activity and the daily pattern of seizures. To investigate, we tracked long-term changes in the rate of spontaneous hippocampal EEG spikes (SPKs) in a rat model of temporal lobe epilepsy. In healthy animals, SPKs oscillated with near 24-h period; however, after injury by status epilepticus, a persistent phase shift of ∼12 h emerged in animals that later went on to develop chronic spontaneous seizures. Additional measurements showed that global 24-h rhythms, including core body temperature and theta state transitions, did not phase shift. Instead, we hypothesized that locally impaired circadian input to the hippocampus might be responsible for the SPK phase shift. This was investigated with a biophysical computer model in which we showed that subtle changes in the relative strengths of circadian input could produce a phase shift in hippocampal neural activity. MRI provided evidence that the medial septum, a putative circadian relay center for the hippocampus, exhibits signs of damage and therefore could contribute to local circadian impairment. Our results suggest that balanced circadian input is critical to maintaining natural circadian phase in the hippocampus and that damage to circadian relay centers, such as the medial septum, may disrupt this balance. We conclude by discussing how abnormal circadian regulation may contribute to the daily rhythms of epileptic seizures and related cognitive dysfunction.

Computational models of epilepsy.

PURPOSE: Approximately 30% of epilepsy patients suffer from medically refractory epilepsy, in which seizures can not controlled by the use of anti-epileptic drugs (AEDs). Understanding the mechanisms underlying these forms of drug-resistant epileptic seizures and the development of alternative effective treatment strategies are fundamental challenges for modern epilepsy research. In this context, computational modeling has gained prominence as an important tool for tackling the complexity of the epileptic phenomenon. In this review article, we present a survey of computational models of epilepsy from the point of view that epilepsy is a dynamical brain disease that is primarily characterized by unprovoked spontaneous epileptic seizures. METHOD: We introduce key concepts from the mathematical theory of dynamical systems, such as multi-stability and bifurcations, and explain how these concepts aid in our understanding of the brain mechanisms involved in the emergence of epileptic seizures. RESULTS: We present a literature survey of the different computational modeling approaches that are used in the study of epilepsy. Special emphasis is placed on highlighting the fine balance between the degree of model simplification and the extent of biological realism that modelers seek in order to address relevant questions. In this context, we discuss three specific examples from published literature, which exemplify different approaches used for developing computational models of epilepsy. We further explore the potential of recently developed optogenetics tools to provide novel avenue for seizure control. CONCLUSION: We conclude with a discussion on the utility of computational models for the development of new epilepsy treatment protocols.

Detecting effective connectivity in networks of coupled neuronal oscillators.

The application of data-driven time series analysis techniques such as Granger causality, partial directed coherence and phase dynamics modeling to estimate effective connectivity in brain networks has recently gained significant prominence in the neuroscience community. While these techniques have been useful in determining causal interactions among different regions of brain networks, a thorough analysis of the comparative accuracy and robustness of these methods in identifying patterns of effective connectivity among brain networks is still lacking. In this paper, we systematically address this issue within the context of simple networks of coupled spiking neurons. Specifically, we develop a method to assess the ability of various effective connectivity measures to accurately determine the true effective connectivity of a given neuronal network. Our method is based on decision tree classifiers which are trained using several time series features that can be observed solely from experimentally recorded data. We show that the classifiers constructed in this work provide a general framework for determining whether a particular effective connectivity measure is likely to produce incorrect results when applied to a dataset.

Challenges for emerging neurostimulation-based therapies for real-time seizure control.

In step with the worthwhile aim of this special issue, two junior investigators impart their insights on the therapeutic challenges imposed by pharmacoresistant epilepsies and offer viable approaches to improvement of treatment outcomes. Sunderam's comprehensive perspective addresses issues of critical importance for the design of efficacious therapies. Talathi delves into the thorny roles of so-called "interictal" spikes in ictio- and epileptogenesis, roles that are central to understanding the dynamics of these phenomena and implicitly of how to prevent them or abort them. First, however, Osorio and co-workers illustrate the complex behavior of the epileptogenic network and point to the importance of real-time intraindividual adaptation and optimization of therapies for seizures originating from the same epileptogenic network.

Control of neural synchrony using channelrhodopsin-2: a computational study.

In this paper, we present an optical stimulation based approach to induce 1:1 in-phase synchrony in a network of coupled interneurons wherein each interneuron expresses the light sensitive protein channelrhodopsin-2 (ChR2). We begin with a transition rate model for the channel kinetics of ChR2 in response to light stimulation. We then define "functional optical time response curve (fOTRC)" as a measure of the response of a periodically firing interneuron (transfected with ChR2 ion channel) to a periodic light pulse stimulation. We specifically consider the case of unidirectionally coupled (UCI) network and propose an open loop control architecture that uses light as an actuation signal to induce 1:1 in-phase synchrony in the UCI network. Using general properties of the spike time response curves (STRCs) for Type-1 neuron model (Ermentrout, Neural Comput 8:979-1001, 1996) and fOTRC, we estimate the (open loop) optimal actuation signal parameters required to induce 1:1 in-phase synchrony. We then propose a closed loop controller architecture and a controller algorithm to robustly sustain stable 1:1 in-phase synchrony in the presence of unknown deviations in the network parameters. Finally, we test the performance of this closed-loop controller in a network of mutually coupled (MCI) interneurons.

Support vector machines for seizure detection in an animal model of chronic epilepsy.

We compare the performance of three support vector machine (SVM) types: weighted SVM, one-class SVM and support vector data description (SVDD) for the application of seizure detection in an animal model of chronic epilepsy. Large EEG datasets (273 h and 91 h respectively, with a sampling rate of 1 kHz) from two groups of rats with chronic epilepsy were used in this study. For each of these EEG datasets, we extracted three energy-based seizure detection features: mean energy, mean curve length and wavelet energy. Using these features we performed twofold cross-validation to obtain the performance statistics: sensitivity (S), specificity (K) and detection latency (tau) as a function of control parameters for the given SVM. Optimal control parameters for each SVM type that produced the best seizure detection statistics were then identified using two independent strategies. Performance of each SVM type is ranked based on the overall seizure detection performance through an optimality index metric (O). We found that SVDD not only performed better than the other SVM types in terms of highest value of the mean optimality index metric (O⁻) but also gave a more reliable performance across the two EEG datasets.

Granger causality relationships between local field potentials in an animal model of temporal lobe epilepsy.

An understanding of the in vivo spatial emergence of abnormal brain activity during spontaneous seizure onset is critical to future early seizure detection and closed-loop seizure prevention therapies. In this study, we use Granger causality (GC) to determine the strength and direction of relationships between local field potentials (LFPs) recorded from bilateral microelectrode arrays in an intermittent spontaneous seizure model of chronic temporal lobe epilepsy before, during, and after Racine grade partial onset generalized seizures. Our results indicate distinct patterns of directional GC relationships within the hippocampus, specifically from the CA1 subfield to the dentate gyrus, prior to and during seizure onset. Our results suggest sequential and hierarchical temporal relationships between the CA1 and dentate gyrus within and across hippocampal hemispheres during seizure. Additionally, our analysis suggests a reversal in the direction of GC relationships during seizure, from an abnormal pattern to more anatomically expected pattern. This reversal correlates well with the observed behavioral transition from tonic to clonic seizure in time-locked video. These findings highlight the utility of GC to reveal dynamic directional temporal relationships between multichannel LFP recordings from multiple brain regions during unprovoked spontaneous seizures.

Effects of phase on homeostatic spike rates.

Recent experimental results by Talathi et al. (Neurosci Lett 455:145-149, 2009) showed a divergence in the spike rates of two types of population spike events, representing the putative activity of the excitatory and inhibitory neurons in the CA1 area of an animal model for temporal lobe epilepsy. The divergence in the spike rate was accompanied by a shift in the phase of oscillations between these spike rates leading to a spontaneous epileptic seizure. In this study, we propose a model of homeostatic synaptic plasticity which assumes that the target spike rate of populations of excitatory and inhibitory neurons in the brain is a function of the phase difference between the excitatory and inhibitory spike rates. With this model of homeostatic synaptic plasticity, we are able to simulate the spike rate dynamics seen experimentally by Talathi et al. in a large network of interacting excitatory and inhibitory neurons using two different spiking neuron models. A drift analysis of the spike rates resulting from the homeostatic synaptic plasticity update rule allowed us to determine the type of synapse that may be primarily involved in the spike rate imbalance in the experimental observation by Talathi et al. We find excitatory neurons, particularly those in which the excitatory neuron is presynaptic, have the most influence in producing the diverging spike rates and causing the spike rates to be anti-phase. Our analysis suggests that the excitatory neuronal population, more specifically the excitatory to excitatory synaptic connections, could be implicated in a methodology designed to control epileptic seizures.

Synchrony with shunting inhibition in a feedforward inhibitory network.

Recent experiments have shown that GABA(A) receptor mediated inhibition in adult hippocampus is shunting rather than hyperpolarizing. Simulation studies of realistic interneuron networks with strong shunting inhibition have been demonstrated to exhibit robust gamma band (20-80 Hz) synchrony in the presence of heterogeneity in the intrinsic firing rates of individual neurons in the network. In order to begin to understand how shunting can contribute to network synchrony in the presence of heterogeneity, we develop a general theoretical framework using spike time response curves (STRC's) to study patterns of synchrony in a simple network of two unidirectionally coupled interneurons (UCI network) interacting through a shunting synapse in the presence of heterogeneity. We derive an approximate discrete map to analyze the dynamics of synchronous states in the UCI network by taking into account the nonlinear contributions of the higher order STRC terms. We show how the approximate discrete map can be used to successfully predict the domain of synchronous 1:1 phase locked state in the UCI network. The discrete map also allows us to determine the conditions under which the two interneurons can exhibit in-phase synchrony. We conclude by demonstrating how the information from the study of the discrete map for the dynamics of the UCI network can give us valuable insight into the degree of synchrony in a larger feed-forward network of heterogeneous interneurons.

Predicting synchrony in heterogeneous pulse coupled oscillators.

Pulse coupled oscillators (PCOs) represent an ubiquitous model for a number of physical and biological systems. Phase response curves (PRCs) provide a general mathematical framework to analyze patterns of synchrony generated within these models. A general theoretical approach to account for the nonlinear contributions from higher-order PRCs in the generation of synchronous patterns by the PCOs is still lacking. Here, by considering a prototypical example of a PCO network, i.e., two synaptically coupled neurons, we present a general theory that extends beyond the weak-coupling approximation, to account for higher-order PRC corrections in the derivation of an approximate discrete map, the stable fixed point of which can predict the domain of 1:1 phase locked synchronous states generated by the PCO network.

Circadian control of neural excitability in an animal model of temporal lobe epilepsy.

We provide experimental evidence for the emerging imbalance in the firing activity of two distinct classes (type 1 and type 2) of population spikes recorded from the hippocampal area CA1 in an animal model of temporal lobe epilepsy. We show that during the latent period of epileptogenesis following status epilepticus inducing brain injury, there is a sustained increase in the firing rate of type 1 population spikes (PS1) with a concurrent decrease in the firing rate of type 2 population spikes (PS2). Both PS1 and PS2 firing rates are observed to follow a circadian rhythm and are in-phase in control rats. Following brain injury there is an abrupt phase shift in the circadian activity of the PS firing rates. We hypothesize that this abrupt phase shift is the underlying cause for the emergence of imbalance in the firing activity of the two PS. We test our hypothesis in the framework of a simple two-dimensional Wilson-Cowan model that describes the interaction between firing activities of populations of excitatory and inhibitory neurons.

Temporal lobe epilepsy: anatomical and effective connectivity.

While temporal lobe epilepsy (TLE) has been treatable with anti-seizure medications over the past century, there still remain a large percentage of patients whose seizures remain untreatable pharmacologically. To better understand and treat TLE, our laboratory uses several in vivo analytical techniques to estimate connectivity in epilepsy. This paper reviews two different connectivity-based approaches with an emphasis on application to the study of epilepsy. First, we present effective connectivity techniques, such as Granger causality, that has been used to assess the dynamic directional relationships among brain regions. These measures are used to better understand how seizure activity initiates, propagates, and terminates. Second, structural techniques, such as magnetic resonance imaging, can be used to assess changes in the underlying neural structures that result in seizure. This paper also includes in vivo epilepsy-centered examples of both effective and anatomical connectivity analysis. These analyses are performed on data collected in vivo from a spontaneously seizing animal model of TLE. Future work in vivo on epilepsy will no doubt benefit from a fusion of these different techniques. We conclude by discussing the interesting possibilities, implications, and challenges that a unified analysis would present.

Temporal spike pattern learning.

Sensory systems pass information about an animal's environment to higher nervous system units through sequences of action potentials. When these action potentials have essentially equivalent wave forms, all information is contained in the interspike intervals (ISIs) of the spike sequence. How do neural circuits recognize and read these ISI sequences? We address this issue of temporal sequence learning by a neuronal system utilizing spike timing dependent plasticity (STDP). We present a general architecture of neural circuitry that can perform the task of ISI recognition. The essential ingredients of this neural circuit, which we refer to as "interspike interval recognition unit" (IRU) are (i) a spike selection unit, the function of which is to selectively distribute input spikes to downstream IRU circuitry; (ii) a time-delay unit that can be tuned by STDP; and (iii) a detection unit, which is the output of the IRU and a spike from which indicates successful ISI recognition by the IRU. We present two distinct configurations for the time-delay circuit within the IRU using excitatory and inhibitory synapses, respectively, to produce a delayed output spike at time t_{0}+tau(R) in response to the input spike received at time t_{0} . R is the tunable parameter of the time-delay circuit that controls the timing of the delayed output spike. We discuss the forms of STDP rules for excitatory and inhibitory synapses, respectively, that allow for modulation of R for the IRU to perform its task of ISI recognition. We then present two specific implementations for the IRU circuitry, derived from the general architecture that can both learn the ISIs of a training sequence and then recognize the same ISI sequence when it is presented on subsequent occasions.