Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review.

Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real-world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless-sensor networks, in addressing single-sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG), electrocardiography (ECG), and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real-time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.

Electroencephalographic Workload Indicators During Teleoperation of an Unmanned Aerial Vehicle Shepherding a Swarm of Unmanned Ground Vehicles in Contested Environments.

Background: Although many electroencephalographic (EEG) indicators have been proposed in the literature, it is unclear which of the power bands and various indices are best as indicators of mental workload. Spectral powers (Theta, Alpha, and Beta) and ratios (Beta/(Alpha + Theta), Theta/Alpha, Theta/Beta) were identified in the literature as prominent indicators of cognitive workload. Objective: The aim of the present study is to identify a set of EEG indicators that can be used for the objective assessment of cognitive workload in a multitasking setting and as a foundational step toward a human-autonomy augmented cognition system. Methods: The participants' perceived workload was modulated during a teleoperation task involving an unmanned aerial vehicle (UAV) shepherding a swarm of unmanned ground vehicles (UGVs). Three sources of data were recorded from sixteen participants (n = 16): heart rate (HR), EEG, and subjective indicators of the perceived workload using the Air Traffic Workload Input Technique (ATWIT). Results: The HR data predicted the scores from ATWIT. Nineteen common EEG features offered a discriminatory power of the four workload setups with high classification accuracy (82.23%), exhibiting a higher sensitivity than ATWIT and HR. Conclusion: The identified set of features represents EEG indicators for the objective assessment of cognitive workload across subjects. These common indicators could be used for augmented intelligence in human-autonomy teaming scenarios, and form the basis for our work on designing a closed-loop augmented cognition system for human-swarm teaming.

Effects of strategic early-morning caffeine gum administration on association between salivary alpha-amylase and neurobehavioural performance during 50 h of sleep deprivation.

Self-assessment is the most common method for monitoring performance and safety in the workplace. However, discrepancies between subjective and objective measures have increased interest in physiological assessment of performance. In a double-blind placebo-controlled study, 23 healthy adults were randomly assigned to either a placebo (n = 11; 5 F, 6 M) or caffeine condition (n = 12; 4 F, 8 M) while undergoing 50 h (i.e. two days) of total sleep deprivation. In previous work, higher salivary alpha-amylase (sAA) levels were associated with improved psychomotor vigilance and simulated driving performance in the placebo condition. In this follow-up article, the effects of strategic caffeine administration on the previously reported diurnal profiles of sAA and performance, and the association between sAA and neurobehavioural performance were investigated. Participants were given a 10 h baseline sleep opportunity (monitored via standard polysomnography techniques) prior to undergoing sleep deprivation (total sleep time: placebo = 8.83 ±â€¯0.48 h; caffeine = 9.01 ±â€¯0.48 h). During sleep deprivation, caffeine gum (200 mg) was administered at 01:00 h, 03:00 h, 05:00 h, and 07:00 h to participants in the caffeine condition (n = 12). This strategic administration of caffeine gum (200 mg) has been shown to be effective at maintaining cognitive performance during extended wakefulness. Saliva samples were collected, and psychomotor vigilance and simulated driving performance assessed at three-hour intervals throughout wakefulness. Caffeine effects on diurnal variability were compared with previously reported findings in the placebo condition (n = 11). The impact of caffeine on the circadian profile of sAA coincided with changes in neurobehavioural performance. Higher sAA levels were associated with improved performance on the psychomotor vigilance test during the first 24 h of wakefulness in the caffeine condition. However, only the association between sAA and response speed (i.e. reciprocal-transform of mean reaction time) was consistent across both days of sleep deprivation. The association between sAA and driving performance was not consistent across both days of sleep deprivation. Results show that the relationship between sAA and reciprocal-transform of mean reaction time on the psychomotor vigilance test persisted in the presence of caffeine, however the association was relatively weaker as compared with the placebo condition.

The impact of caffeine consumption during 50 hr of extended wakefulness on glucose metabolism, self-reported hunger and mood state.

Caffeine is known for its capacity to mitigate performance decrements. The metabolic side-effects are less well understood. This study examined the impact of cumulative caffeine doses on glucose metabolism, self-reported hunger and mood state during 50 hr of wakefulness. In a double-blind laboratory study, participants were assigned to caffeine (n = 9, 6M, age 21.3 ± 2.1 years; body mass index 21.9 ± 1.6 kg/m2 ) or placebo conditions (n = 8, 4M, age 23.0 ± 2.8 years; body mass index 21.8 ± 1.6 kg/m2 ). Following a baseline sleep (22:00 hours-08:00 hours), participants commenced 50 hr of sleep deprivation. Meal timing and composition were controlled throughout the study. Caffeine (200 mg) or placebo gum was chewed for 5 min at 01:00 hours, 03:00 hours, 05:00 hours and 07:00 hours during each night of sleep deprivation. Continual glucose monitors captured interstitial glucose 2 hr post-breakfast, at 5-min intervals. Hunger and mood state were assessed at 10:00 hours, 16:30 hours, 22:30 hours and 04:30 hours. Caffeine did not affect glucose area under the curve (p = 0.680); however, glucose response to breakfast significantly increased after 2 nights of extended wakefulness compared with baseline (p = 0.001). There was a significant main effect of day, with increased tiredness (p < 0.001), mental exhaustion (p < 0.001), irritability (p = 0.002) and stress (p < 0.001) on the second day of extended wake compared with day 1. Caffeine attenuated the rise in tiredness (p < 0.001), mental exhaustion (p = 0.044) and irritability (p = 0.018) on day 1 but not day 2. Self-reported hunger was not affected by sleep deprivation or caffeine. These data confirm the effectiveness of caffeine in improving performance under conditions of sleep deprivation by reducing feelings of tiredness, mental exhaustion and irritability without exacerbating glucose metabolism and feelings of hunger.

Decreased salivary alpha-amylase levels are associated with performance deficits during sleep loss.

During sleep deprivation, neurobehavioral functions requiring sustained levels of attention and alertness are significantly impaired. Discrepancies between subjective measures of sleepiness and objective performance during sustained operations have led to interest in physiological monitoring of operator performance. Alertness, vigilance, and arousal are modulated by the wake-promoting actions of the central noradrenergic system. Salivary alpha-amylase (sAA) has been proposed as a sensitive peripheral measure of noradrenergic activity, but limited research has investigated the relationship between sAA and performance. In a laboratory-controlled environment, we investigated the relationship between sAA levels, subjective sleepiness, and performance during two days (50h) of total sleep deprivation. Beginning at 09:00, twelve healthy participants (5 females) aged 22.5±2.5years (mean±SD) provided saliva samples, recorded ratings of subjective sleepiness, completed a brief 3-min psychomotor vigilance task (PVT-B) and performed a 40-min simulated driving task, at regular 3h intervals during wakefulness. Ratings of subjective sleepiness exhibited a constant linear increase (p<0.001) during sleep deprivation. In contrast, sAA levels showed a marked diurnal profile, with levels increasing during the day (p<0.001) and steadily declining in the evening and early-morning (p<0.001). PVT-B (mean reaction time and mean slowest 10% reaction time) and simulated driving performance (speed deviation and lane deviation) also exhibited diurnal profiles across the two days of sleep deprivation. Performance peaked in the afternoon (p<0.001) and then steadily worsened as wakefulness continued into the evening and early-morning (p<0.001). Further analysis revealed that higher sAA levels in the hour preceding each performance assessment were associated with better PVT-B and driving performance (p<0.001). These findings suggest that sAA measures may be suitable indicators of performance deficits during sustained wakefulness and highlight the potential for sAA to be considered for physiological monitoring of performance. In operational environments sAA levels, as part of a panel of physiological measures, may be useful for assessing fitness-for-duty prior to safety being compromised or when performance deficits are unknown.