Fatigue Assessment from Facial Videos using Deep Neural Networks and Engineered Features Informed by Domain Knowledge.

Fatigue impairs cognitive and motor function, potentially leading to mishaps in high-pressure occupations such as aviation and emergency medical services. The current approach is primarily based on self-assessment, which is subjective and error-prone. An objective method is needed to detect severe and likely dangerous levels of fatigue quickly and accurately. Here, we present a quantitative evaluation tool that uses less than two minutes of facial video, captured using an iPad, to assess fatigue vs. alertness. The tool is fast, easy to use, and scalable since it uses cameras readily available on consumer-electronic devices. We compared the classification performance between a Long Short-Term Memory (LSTM) deep neural network and a Random Forest (RF) classifier applied to engineered features informed by domain knowledge. The preliminary results on an 11-subject dataset show that RF outperforms LSTM, with added interpretability on the features used. For the RF classifiers, the average areas under the receiver operating characteristic curve, based on the 11-fold and individualized 11-fold cross validations, are 0.72 ± 0.16 and 0.8 ± 0.12, respectively. Equal error rates are 0.34 and 0.26, respectively. This study presents a promising approach for rapid fatigue detection. Additional data will be collected to assess the generalizability across populations.

Changes in Eye Tracking Features Across Periods of Overpressure Exposure.

INTRODUCTION: Repetitive exposure to blast overpressure waves can be a part of routine military and law enforcement training. However, our understanding of the effects of that repetitive exposure on human neurophysiology remains limited. To link an individual's cumulative exposure with their neurophysiological effects, overpressure dosimetry needs to be concurrently collected with relevant physiological signals. Eye tracking has shown promise for providing insight into neurophysiological change because of neural injury, but video-based technology limits usage to a laboratory or clinic. In the present work, we show capability for using electrooculography-based eye tracking to enable physiological assessment in the field during activities involved repetitive blast exposures. MATERIALS AND METHODS: Overpressure dosimetry was accomplished by using a body-worn measurement system that captures continuous sound pressure levels as well as pressure waveforms of blast event in the range of 135-185 dB peak (0.1-36 kPa). Electrooculography eye tracking was performed using a commercial Shimmer Sensing system, which captured horizontal eye movements of both the left and right eyes, as well as vertical eye movements of the right eye, from which blinks can also be extracted. Data were collected during breaching activities that included repetitive use of explosives. Participants in the study were U.S. Army Special Operators and Federal Bureau of Investigations special agents. Approval for research was received by the Massachucetts Institute of Technology Committee on the Use of Humans as Experimental Subjects, the Air Force Human Research Protections Office, and the Federal Bureau of Investigations Institutional Review Board. RESULTS: The energy from overpressure events was accumulated and summarized into an 8-hour equivalent of sound pressure level (i.e., LZeq8hr). The total exposure in a single day, i.e., the LZeq8hr, ranged from 110 to 160 dB. Oculomotor features, such as blink and saccade rate, as well as variance in blink waveforms, show changes across the period of overpressure exposure. However, the features that showed significant change across the population were not necessarily the ones that showed significant correlation with the levels of overpressure exposure. A regression model built to predict overpressure levels from oculomotor features alone showed a significant association (R = 0.51, P < .01). Investigation of the model indicates that changes in the saccade rate and blink waveforms are driving the relationship. CONCLUSIONS: This study successfully demonstrated that eye tracking can be performed during training activities, such as explosive breaching, and that the modality may provide insight into neurophysiological change across periods of overpressure exposure. The results presented herein show that electrooculography-based eye tracking may be a useful method of assessing individualized physiological effects of overpressure exposure in the field. Future work is focused on time-dependent modeling to assess continuous changes in eye movements as this will enable building dose-response curves.

Using Body-worn Accelerometers to Detect Physiological Changes During Periods of Blast Overpressure Exposure.

Repetitive exposure to non-concussive blast expo-sure may result in sub-clinical neurological symptoms. These changes may be reflected in the neural control gait and balance. In this study, we collected body-worn accelerometry data on individuals who were exposed to repetitive blast overpressures as part of their occupation. Accelerometry features were gener-ated within periods of low-movement and gait. These features were the eigenvalues of high-dimensional correlation matrices, which were constructed with time-delay embedding at multiple delay scales. When focusing on the gait windows, there were significant correlations of the changes in features with the cumulative dose of blast exposure. When focusing on the low-movement frames, the correlation with exposure were lower than that of the gait frames and statistically insignificant. In a cross-validated model, the overpressure exposure was predicted from gait features alone. The model was statistically significant and yielded an RMSE of 1.27 dB. With continued development, the model may be used to assess the physiological effects of repetitive blast exposure and guide training procedures to minimize impact on the individual.

EEG alpha and pupil diameter reflect endogenous auditory attention switching and listening effort.

Everyday environments often contain distracting competing talkers and background noise, requiring listeners to focus their attention on one acoustic source and reject others. During this auditory attention task, listeners may naturally interrupt their sustained attention and switch attended sources. The effort required to perform this attention switch has not been well studied in the context of competing continuous speech. In this work, we developed two variants of endogenous attention switching and a sustained attention control. We characterized these three experimental conditions under the context of decoding auditory attention, while simultaneously evaluating listening effort and neural markers of spatial-audio cues. A least-squares, electroencephalography (EEG)-based, attention decoding algorithm was implemented across all conditions. It achieved an accuracy of 69.4% and 64.0% when computed over nonoverlapping 10 and 5-s correlation windows, respectively. Both decoders illustrated smooth transitions in the attended talker prediction through switches at approximately half of the analysis window size (e.g., the mean lag taken across the two switch conditions was 2.2 s when the 5-s correlation window was used). Expended listening effort, as measured by simultaneous EEG and pupillometry, was also a strong indicator of whether the listeners sustained attention or performed an endogenous attention switch (peak pupil diameter measure [ p=0.034 ] and minimum parietal alpha power measure [ p=0.016 ]). We additionally found evidence of talker spatial cues in the form of centrotemporal alpha power lateralization ( p=0.0428 ). These results suggest that listener effort and spatial cues may be promising features to pursue in a decoding context, in addition to speech-based features.

Using Natural Head Movements to Continually Calibrate EOG Signals.

Electrooculography (EOG) is the measurement of eye movements using surface electrodes adhered around the eye. EOG systems can be designed to have an unobtrusive form-factor that is ideal for eye tracking in free-living over long durations, but the relationship between voltage and gaze direction requires frequent re-calibration as the skin-electrode impedance and retinal adaptation vary over time. Here we propose a method for automatically calibrating the EOG-gaze relationship by fusing EOG signals with gyroscopic measurements of head movement whenever the vestibulo-ocular reflex (VOR) is active. The fusion is executed as recursive inference on a hidden Markov model that accounts for all rotational degrees-of-freedom and uncertainties simultaneously. This enables continual calibration using natural eye and head movements while minimizing the impact of sensor noise. No external devices like monitors or cameras are needed. On average, our method's gaze estimates deviate by 3.54° from those of an industry-standard desktop video-based eye tracker. Such discrepancy is on par with the latest mobile video eye trackers. Future work is focused on automatically detecting moments of VOR in free-living.

Psychophysiological Markers of Performance and Learning during Simulated Marksmanship in Immersive Virtual Reality.

The fusion of immersive virtual reality, kinematic movement tracking, and EEG offers a powerful test bed for naturalistic neuroscience research. Here, we combined these elements to investigate the neuro-behavioral mechanisms underlying precision visual-motor control as 20 participants completed a three-visit, visual-motor, coincidence-anticipation task, modeled after Olympic Trap Shooting and performed in immersive and interactive virtual reality. Analyses of the kinematic metrics demonstrated learning of more efficient movements with significantly faster hand RTs, earlier trigger response times, and higher spatial precision, leading to an average of 13% improvement in shot scores across the visits. As revealed through spectral and time-locked analyses of the EEG beta band (13-30 Hz), power measured prior to target launch and visual-evoked potential amplitudes measured immediately after the target launch correlated with subsequent reactive kinematic performance in the shooting task. Moreover, both launch-locked and shot/feedback-locked visual-evoked potentials became earlier and more negative with practice, pointing to neural mechanisms that may contribute to the development of visual-motor proficiency. Collectively, these findings illustrate EEG and kinematic biomarkers of precision motor control and changes in the neurophysiological substrates that may underlie motor learning.

Using Dynamics of Eye Movements, Speech Articulation and Brain Activity to Predict and Track mTBI Screening Outcomes.

Repeated subconcussive blows to the head during sports or other contact activities may have a cumulative and long lasting effect on cognitive functioning. Unobtrusive measurement and tracking of cognitive functioning is needed to enable preventative interventions for people at elevated risk of concussive injury. The focus of the present study is to investigate the potential for using passive measurements of fine motor movements (smooth pursuit eye tracking and read speech) and resting state brain activity (measured using fMRI) to complement existing diagnostic tools, such as the Immediate Post-concussion Assessment and Cognitive Testing (ImPACT), that are used for this purpose. Thirty-one high school American football and soccer athletes were tracked through the course of a sports season. Hypotheses were that (1) measures of complexity of fine motor coordination and of resting state brain activity are predictive of cognitive functioning measured by the ImPACT test, and (2) within-subject changes in these measures over the course of a sports season are predictive of changes in ImPACT scores. The first principal component of the six ImPACT composite scores was used as a latent factor that represents cognitive functioning. This latent factor was positively correlated with four of the ImPACT composites: verbal memory, visual memory, visual motor speed and reaction speed. Strong correlations, ranging between r = 0.26 and r = 0.49, were found between this latent factor and complexity features derived from each sensor modality. Based on a regression model, the complexity features were combined across sensor modalities and used to predict the latent factor on out-of-sample subjects. The predictions correlated with the true latent factor with r = 0.71. Within-subject changes over time were predicted with r = 0.34. These results indicate the potential to predict cognitive performance from passive monitoring of fine motor movements and brain activity, offering initial support for future application in detection of performance deficits associated with subconcussive events.

Using Oculomotor Features to Predict Changes in Optic Nerve Sheath Diameter and ImPACT Scores From Contact-Sport Athletes.

There is mounting evidence linking the cumulative effects of repetitive head impacts to neuro-degenerative conditions. Robust clinical assessment tools to identify mild traumatic brain injuries are needed to assist with timely diagnosis for return-to-field decisions and appropriately guide rehabilitation. The focus of the present study is to investigate the potential for oculomotor features to complement existing diagnostic tools, such as measurements of Optic Nerve Sheath Diameter (ONSD) and Immediate Post-concussion Assessment and Cognitive Testing (ImPACT). Thirty-one high school American football and soccer athletes were tracked through the course of a sports season. Given the high risk of repetitive head impacts associated with both soccer and football, our hypotheses were that (1) ONSD and ImPACT scores would worsen through the season and (2) oculomotor features would effectively capture both neurophysiological changes reflected by ONSD and neuro-functional status assessed via ImPACT. Oculomotor features were used as input to Linear Mixed-Effects Regression models to predict ONSD and ImPACT scores as outcomes. Prediction accuracy was evaluated to identify explicit relationships between eye movements, ONSD, and ImPACT scores. Significant Pearson correlations were observed between predicted and actual outcomes for ONSD (Raw = 0.70; Normalized = 0.45) and for ImPACT (Raw = 0.86; Normalized = 0.71), demonstrating the capability of oculomotor features to capture neurological changes detected by both ONSD and ImPACT. The most predictive features were found to relate to motor control and visual-motor processing. In future work, oculomotor models, linking neural structures to oculomotor function, can be built to gain extended mechanistic insights into neurophysiological changes observed through seasons of participation in contact sports.

Sensorimotor conflict tests in an immersive virtual environment reveal subclinical impairments in mild traumatic brain injury.

Current clinical tests lack the sensitivity needed for detecting subtle balance impairments associated with mild traumatic brain injury (mTBI). Patient-reported symptoms can be significant and have a huge impact on daily life, but impairments may remain undetected or poorly quantified using clinical measures. Our central hypothesis was that provocative sensorimotor perturbations, delivered in a highly instrumented, immersive virtual environment, would challenge sensory subsystems recruited for balance through conflicting multi-sensory evidence, and therefore reveal that not all subsystems are performing optimally. The results show that, as compared to standard clinical tests, the provocative perturbations illuminate balance impairments in subjects who have had mild traumatic brain injuries. Perturbations delivered while subjects were walking provided greater discriminability (average accuracy ≈ 0.90) than those delivered during standing (average accuracy ≈ 0.65) between mTBI subjects and healthy controls. Of the categories of features extracted to characterize balance, the lower limb accelerometry-based metrics proved to be most informative. Further, in response to perturbations, subjects with an mTBI utilized hip strategies more than ankle strategies to prevent loss of balance and also showed less variability in gait patterns. We have shown that sensorimotor conflicts illuminate otherwise-hidden balance impairments, which can be used to increase the sensitivity of current clinical procedures. This augmentation is vital in order to robustly detect the presence of balance impairments after mTBI and potentially define a phenotype of balance dysfunction that enhances risk of injury.

Predicting Cognitive Load and Operational Performance in a Simulated Marksmanship Task.

Modern operational environments can place significant demands on a service member's cognitive resources, increasing the risk of errors or mishaps due to overburden. The ability to monitor cognitive burden and associated performance within operational environments is critical to improving mission readiness. As a key step toward a field-ready system, we developed a simulated marksmanship scenario with an embedded working memory task in an immersive virtual reality environment. As participants performed the marksmanship task, they were instructed to remember numbered targets and recall the sequence of those targets at the end of the trial. Low and high cognitive load conditions were defined as the recall of three- and six-digit strings, respectively. Physiological and behavioral signals recorded included speech, heart rate, breathing rate, and body movement. These features were input into a random forest classifier that significantly discriminated between the low- and high-cognitive load conditions (AUC = 0.94). Behavioral features of gait were the most informative, followed by features of speech. We also showed the capability to predict performance on the digit recall (AUC = 0.71) and marksmanship (AUC = 0.58) tasks. The experimental framework can be leveraged in future studies to quantify the interaction of other types of stressors and their impact on operational cognitive and physical performance.

Assessment of speech and fine motor coordination in children with autism spectrum disorder.

Autism Spectrum Disorder (ASD) is a developmental disorder characterized by difficulty in communication, which includes a high incidence of speech production errors. We hypothesize that these errors are partly due to underlying deficits in motor coordination and control, which are also manifested in degraded fine motor control of facial expressions and purposeful hand movements. In this pilot study, we computed correlations of acoustic, video, and handwriting time-series derived from five children with ASD and five children with neurotypical development during speech and handwriting tasks. These correlations and eigenvalues derived from the correlations act as a proxy for motor coordination across articulatory, laryngeal, and respiratory speech production systems and for fine motor skills. We utilized features derived from these correlations to discriminate between children with and without ASD. Eigenvalues derived from these correlations highlighted differences in complexity of coordination across speech subsystems and during handwriting, and helped discriminate between the two subject groups. These results suggest differences in coupling within speech production and fine motor skill systems in children with ASD. Our long-term goal is to create a platform assessing motor coordination in children with ASD in order to track progress from speech and motor interventions administered by clinicians.

Probabilistic inferential decision-making under time pressure in rhesus macaques (Macaca mulatta).

Decisions often involve the consideration of multiple cues, each of which may inform selection on the basis of learned probabilities. Our ability to use probabilistic inference for decisions is bounded by uncertainty and constraints such as time pressure. Previous work showed that when humans choose between visual objects in a multiple-cue, probabilistic task, they cope with time pressure by discounting the least informative cues, an example of satisficing or "good enough" decision-making. We tested two rhesus macaques (Macaca mulatta) on a similar task to assess their capacity for probabilistic inference and satisficing in comparison with humans. In each trial, a monkey viewed two compound stimuli consisting of four cue dimensions. Each dimension (e.g., color) had two possible states (e.g., red or blue) with different probabilistic weights. Selecting the stimulus with highest total weight yielded higher odds of receiving reward. Both monkeys learned the assigned weights at high accuracy. Under time pressure, both monkeys were less accurate as a result of decreased use of cue information. One monkey adopted the same satisficing strategy used by humans, ignoring the least informative cue dimension. Both monkeys, however, exhibited a strategy not reported for humans, a "group-the-best" strategy in which the top two cues were used similarly despite their different assigned weights. The results validate macaques as an animal model of probabilistic decision-making, establishing their capacity to discriminate between objects using at least four visual dimensions simultaneously. The time pressure data suggest caution, however, in using macaques as models of human satisficing. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Sensorimotor Learning during a Marksmanship Task in Immersive Virtual Reality.

Sensorimotor learning refers to improvements that occur through practice in the performance of sensory-guided motor behaviors. Leveraging novel technical capabilities of an immersive virtual environment, we probed the component kinematic processes that mediate sensorimotor learning. Twenty naïve subjects performed a simulated marksmanship task modeled after Olympic Trap Shooting standards. We measured movement kinematics and shooting performance as participants practiced 350 trials while receiving trial-by-trial feedback about shooting success. Spatiotemporal analysis of motion tracking elucidated the ballistic and refinement phases of hand movements. We found systematic changes in movement kinematics that accompanied improvements in shot accuracy during training, though reaction and response times did not change over blocks. In particular, we observed longer, slower, and more precise ballistic movements that replaced effort spent on corrections and refinement. Collectively, these results leverage developments in immersive virtual reality technology to quantify and compare the kinematics of movement during early learning of full-body sensorimotor orienting.

Circuits for presaccadic visual remapping.

Saccadic eye movements rapidly displace the image of the world that is projected onto the retinas. In anticipation of each saccade, many neurons in the visual system shift their receptive fields. This presaccadic change in visual sensitivity, known as remapping, was first documented in the parietal cortex and has been studied in many other brain regions. Remapping requires information about upcoming saccades via corollary discharge. Analyses of neurons in a corollary discharge pathway that targets the frontal eye field (FEF) suggest that remapping may be assembled in the FEF's local microcircuitry. Complementary data from reversible inactivation, neural recording, and modeling studies provide evidence that remapping contributes to transsaccadic continuity of action and perception. Multiple forms of remapping have been reported in the FEF and other brain areas, however, and questions remain about the reasons for these differences. In this review of recent progress, we identify three hypotheses that may help to guide further investigations into the structure and function of circuits for remapping.

Neural Network Evidence for the Coupling of Presaccadic Visual Remapping to Predictive Eye Position Updating.

As we look around a scene, we perceive it as continuous and stable even though each saccadic eye movement changes the visual input to the retinas. How the brain achieves this perceptual stabilization is unknown, but a major hypothesis is that it relies on presaccadic remapping, a process in which neurons shift their visual sensitivity to a new location in the scene just before each saccade. This hypothesis is difficult to test in vivo because complete, selective inactivation of remapping is currently intractable. We tested it in silico with a hierarchical, sheet-based neural network model of the visual and oculomotor system. The model generated saccadic commands to move a video camera abruptly. Visual input from the camera and internal copies of the saccadic movement commands, or corollary discharge, converged at a map-level simulation of the frontal eye field (FEF), a primate brain area known to receive such inputs. FEF output was combined with eye position signals to yield a suitable coordinate frame for guiding arm movements of a robot. Our operational definition of perceptual stability was "useful stability," quantified as continuously accurate pointing to a visual object despite camera saccades. During training, the emergence of useful stability was correlated tightly with the emergence of presaccadic remapping in the FEF. Remapping depended on corollary discharge but its timing was synchronized to the updating of eye position. When coupled to predictive eye position signals, remapping served to stabilize the target representation for continuously accurate pointing. Graded inactivations of pathways in the model replicated, and helped to interpret, previous in vivo experiments. The results support the hypothesis that visual stability requires presaccadic remapping, provide explanations for the function and timing of remapping, and offer testable hypotheses for in vivo studies. We conclude that remapping allows for seamless coordinate frame transformations and quick actions despite visual afferent lags. With visual remapping in place for behavior, it may be exploited for perceptual continuity.

Visual continuity across saccades is influenced by expectations.

As we make saccades, the image on each retina is displaced, yet our visual perception is uninterrupted. This is commonly referred to as transsaccadic perceptual stability, but such a description is inadequate. Some visual objects are stable (e.g., rocks) and should be perceived as such across saccades, but other objects may move at any time (e.g., birds). Stability is probabilistic in natural scenes. Here we extend the common notion of transsaccadic visual stability to a more general, ecologically based hypothesis of transsaccadic visual continuity in which postsaccadic percepts of objects depend on expectations about their probability of movement. Subjects made a saccade to a target and reported whether it seemed displaced after the saccade. Targets had varying probabilities of movement (ranging from 0.1-0.9) that corresponded to their color (spectrum from blue to red). Performance was compared before and after subjects were told about the color-probability pairings ("uninformed" vs. "informed" conditions). Analyses focused on signal detection and psychometric threshold measures. We found that in the uninformed condition, performance was similar across color-probability pairings, but in the informed condition, response biases varied with probability of movement, and movement-detection sensitivities were higher for rarely moving targets. We conclude that subjects incorporate priors about object movement into their judgments of visual continuity across saccades.

Speech intelligibility estimation using multi-resolution spectral features for speakers undergoing cancer treatment.

Head and neck cancer can significantly hamper speech production which often reduces speech intelligibility. A method of extracting spectral features is presented. The method uses a multi-resolution sinusoidal transform scheme, which enables better representation of spectral and harmonic characteristics. Regression methods were used to predict interval-scaled intelligibility scores of utterances in the NKI-CCRT speech corpus. The inclusion of these features lowered the mean squared estimation error from 0.43 to 0.39 on a scale from 1 to 7, with a p-value less than 0.001. For binary intelligibility classification, their inclusion resulted in an improvement by 5.0 percentage points when tested on a disjoint set.

Simultaneous transcranial magnetic stimulation and single-neuron recording in alert non-human primates.

Transcranial magnetic stimulation (TMS) is a widely used, noninvasive method for stimulating nervous tissue, yet its mechanisms of effect are poorly understood. Here we report new methods for studying the influence of TMS on single neurons in the brain of alert non-human primates. We designed a TMS coil that focuses its effect near the tip of a recording electrode and recording electronics that enable direct acquisition of neuronal signals at the site of peak stimulus strength minimally perturbed by stimulation artifact in awake monkeys (Macaca mulatta). We recorded action potentials within ∼1 ms after 0.4-ms TMS pulses and observed changes in activity that differed significantly for active stimulation as compared with sham stimulation. This methodology is compatible with standard equipment in primate laboratories, allowing easy implementation. Application of these tools will facilitate the refinement of next generation TMS devices, experiments and treatment protocols.

Toward prediction of the local onset of alternans in the heart.

A beat-to-beat variation in the cardiac action potential duration is a phenomenon known as alternans. Alternans has been linked to ventricular fibrillation, and thus the ability to predict the onset of alternans could be clinically beneficial. Theoretically, it has been proposed that the slope of a restitution curve, which relates the duration of the action potential to the preceding diastolic interval, can predict the onset of alternans. Experimentally, however, this hypothesis has not been consistently proven, mainly because of the intrinsic complexity of the dynamics of cardiac tissue. It was recently shown that the restitution portrait, which combines several restitution curves simultaneously, is associated with the onset of alternans in isolated myocytes. Our main purpose in this study was to determine whether the restitution portrait is correlated with the onset of alternans in the heart, where the dynamics include a spatial complexity. We performed optical mapping experiments in isolated Langendorff-perfused rabbit hearts in which alternans was induced by periodic pacing at different frequencies, and identified the local onset of alternans, B(onset). We identified two regions of the heart: the area that exhibited alternans at B(onset) (1:1(alt)) and the area that did not (1:1). We constructed two-dimensional restitution portraits for the epicardial surface of the heart and measured the spatial distribution of three different slopes (the dynamic restitution slope, S(dyn)(RP), and two local S1-S2 slopes, S(12) and S(12)(max)) separately for these two regions. We found that the S(12) and S(12)(max) slopes differed significantly between the 1:1(alt) and 1:1 regions just before the onset of alternans, and S(dyn)(RP) slopes were statistically similar. In addition, we found that the slopes of the dynamic restitution curve S(dyn) were also statistically similar between these two regions. On the other hand, the quantitative values of all slopes were significantly different from the theoretically predicted value of one. These results demonstrate that the slopes measured in the restitution portrait correlate with the onset of alternans in the heart.