Facial Micro-Expression Recognition Enhanced by Score Fusion and a Hybrid Model from Convolutional LSTM and Vision Transformer.

In the billions of faces that are shaped by thousands of different cultures and ethnicities, one thing remains universal: the way emotions are expressed. To take the next step in human-machine interactions, a machine (e.g., a humanoid robot) must be able to clarify facial emotions. Allowing systems to recognize micro-expressions affords the machine a deeper dive into a person's true feelings, which will take human emotion into account while making optimal decisions. For instance, these machines will be able to detect dangerous situations, alert caregivers to challenges, and provide appropriate responses. Micro-expressions are involuntary and transient facial expressions capable of revealing genuine emotions. We propose a new hybrid neural network (NN) model capable of micro-expression recognition in real-time applications. Several NN models are first compared in this study. Then, a hybrid NN model is created by combining a convolutional neural network (CNN), a recurrent neural network (RNN, e.g., long short-term memory (LSTM)), and a vision transformer. The CNN can extract spatial features (within a neighborhood of an image), whereas the LSTM can summarize temporal features. In addition, a transformer with an attention mechanism can capture sparse spatial relations residing in an image or between frames in a video clip. The inputs of the model are short facial videos, while the outputs are the micro-expressions recognized from the videos. The NN models are trained and tested with publicly available facial micro-expression datasets to recognize different micro-expressions (e.g., happiness, fear, anger, surprise, disgust, sadness). Score fusion and improvement metrics are also presented in our experiments. The results of our proposed models are compared with that of literature-reported methods tested on the same datasets. The proposed hybrid model performs the best, where score fusion can dramatically increase recognition performance.

Finding Explanations in AI Fusion of Electro-Optical/Passive Radio-Frequency Data.

In the Information Age, the widespread usage of blackbox algorithms makes it difficult to understand how data is used. The practice of sensor fusion to achieve results is widespread, as there are many tools to further improve the robustness and performance of a model. In this study, we demonstrate the utilization of a Long Short-Term Memory (LSTM-CCA) model for the fusion of Passive RF (P-RF) and Electro-Optical (EO) data in order to gain insights into how P-RF data are utilized. The P-RF data are constructed from the in-phase and quadrature component (I/Q) data processed via histograms, and are combined with enhanced EO data via dense optical flow (DOF). The preprocessed data are then used as training data with an LSTM-CCA model in order to achieve object detection and tracking. In order to determine the impact of the different data inputs, a greedy algorithm (explainX.ai) is implemented to determine the weight and impact of the canonical variates provided to the fusion model on a scenario-by-scenario basis. This research introduces an explainable LSTM-CCA framework for P-RF and EO sensor fusion, providing novel insights into the sensor fusion process that can assist in the detection and differentiation of targets and help decision-makers to determine the weights for each input.

An Efficient and Uncertainty-Aware Decision Support System for Disaster Response Using Aerial Imagery.

Efficient and robust search and rescue actions are always required when natural or technical disasters occur. Empowered by remote sensing techniques, building damage assessment can be achieved by fusing aerial images of pre- and post-disaster environments through computational models. Existing methods pay over-attention to assessment accuracy without considering model efficiency and uncertainty quantification in such a life-critical application. Thus, this article proposes an efficient and uncertain-aware decision support system (EUDSS) that evolves the recent computational models into an efficient decision support system, realizing the uncertainty during building damage assessment (BDA). Specifically, a new efficient and uncertain-aware BDA integrates the recent advances in computational models such as Fourier attention and Monte Carlo Dropout for uncertainty quantification efficiently. Meanwhile, a robust operation (RO) procedure is designed to invite experts for manual reviews if the uncertainty is high due to external factors such as cloud clutter and poor illumination. This procedure can prevent rescue teams from missing damaged houses during operations. The effectiveness of the proposed system is demonstrated on a public dataset from both quantitative and qualitative perspectives. The solution won the first place award in International Overhead Imagery Hackathon.

Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition.

Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these shortcomings for a long-term monitoring solution, an interpretable, passive, multi-modal, sensor fusion system PRF-PIR is proposed in this work. PRF-PIR is composed of one software-defined radio (SDR) device and one novel passive infrared (PIR) sensor system. A recurrent neural network (RNN) is built as the HIAR model for this proposed solution to handle the temporal dependence of passive information captured by both modalities. We validate our proposed PRF-PIR system for a potential human monitoring system through the data collection of eleven activities from twelve human subjects in an academic office environment. From our data collection, the efficacy of the sensor fusion system is proven via an accuracy of 0.9866 for human identification and an accuracy of 0.9623 for activity recognition. The results of the system are supported with explainable artificial intelligence (XAI) methodologies to serve as a validation for sensor fusion over the deployment of single sensor solutions. PRF-PIR provides a passive, non-intrusive, and highly accurate system that allows for robustness in uncertain, highly similar, and complex at-home activities performed by a variety of human subjects.

The Study of Aviation Safe Incapacitating Device Based on LED Technology with a Smart-Illumination Sensor Unit.

This paper deals with a design and implementation of optical defensive device for protection of aviation personnel. The design is built on the basic characteristics of human eyesight, illumination sensing of the environment, and microcontroller implementation for adaptation over sensed power, flash duration, and person distance. The aviation safe LED-based optical dazzler equipment (ASLODE) utilizes light emitting diode (LED) technology implemented with constant current regulators to control several modes of effects based on situational sensing. The temporarily incapacitating device can be extended by means of real-time illumination sensing to improve power efficiency and reach the highest level of safety. The smart pulse sets the flashing frequency from 8Hz for high-level light intensities and up to 20 Hz in low-level lighting conditions. Experimental results demonstrate the effectiveness of the ASLODE device over numerous experiments with controlled onboard aircraft scenarios that adapt the energy, flash rate, and processing to the sensed environmental illumination to meet aviation hygienic standards for people without eyesight defects.

Spatial Disorientation Impact on the Precise Approach in Simulated Flight.

BACKGROUND: The risks posed by flight illusions impacting pilot spatial orientation have been determined as a safety concern from numerous past aviation accident investigations. Early demonstration of the adverse effects of flight illusions on spatial orientation would be desirable for all pilots, especially at the early training stages to deeply embed good practices for onset detection, flight correction, and response mitigation.METHOD: Simulated flights on a disorientation demonstrator were performed by 19 pilots for 3 conditions: no illusion, somatogyral illusion, and Coriolis illusion. An objective approach for assessing pilot performance degradation due to flight illusions can be done by using a defined flight profile: instrument landing system (ILS) flight trajectory during final instrument approach. Deviations to the standard ILS profile were recorded to measure and evaluate the influence of the demonstrated flight illusion on pilot performance.RESULTS: The results show the expectation that the smallest deviations from the ideal trajectory are caused by pilot tracking error (no illusion), and the greatest deviations are caused by the Coriolis illusion. Results demonstrated a statistically significant effect of illusions on performance. According to statements from pilots, training for flight illusion response is essential to complement training in aircraft regulations and aerodynamics.DISCUSSION: Measuring the influence of vestibular illusions on flight profile with a simulator allows assessment of individual differences and improvement of pilot performance under the conditions of no illusion, the somatogyral illusion, and the Coriolis illusion.Boril J, Smrz V, Blasch E, Lone M. Spatial disorientation impact on the precise approach in simulated flight. Aerosp Med Hum Perform. 2020; 91(10):767775.

Human Occupancy Detection via Passive Cognitive Radio.

Human occupancy detection (HOD) in an enclosed space, such as indoors or inside of a vehicle, via passive cognitive radio (CR) is a new and challenging research area. Part of the difficulty arises from the fact that a human subject cannot easily be detected due to spectrum variation. In this paper, we present an advanced HOD system that dynamically reconfigures a CR to collect passive radio frequency (RF) signals at different places of interest. Principal component analysis (PCA) and recursive feature elimination with logistic regression (RFE-LR) algorithms are applied to find the frequency bands sensitive to human occupancy when the baseline spectrum changes with locations. With the dynamically collected passive RF signals, four machine learning (ML) classifiers are applied to detect human occupancy, including support vector machine (SVM), k-nearest neighbors (KNN), decision tree (DT), and linear SVM with stochastic gradient descent (SGD) training. The experimental results show that the proposed system can accurately detect human subjects-not only in residential rooms-but also in commercial vehicles, demonstrating that passive CR is a viable technique for HOD. More specifically, the RFE-LR with SGD achieves the best results with a limited number of frequency bands. The proposed adaptive spectrum sensing method has not only enabled robust detection performance in various environments, but also improved the efficiency of the CR system in terms of speed and power consumption.

Detecting Malicious False Frame Injection Attacks on Surveillance Systems at the Edge Using Electrical Network Frequency Signals.

Over the past few years, the importance of video surveillance in securing national critical infrastructure has significantly increased, with applications including the detection of failures and anomalies. Accompanied by the proliferation of video is the increasing number of attacks against surveillance systems. Among the attacks, False Frame Injection (FFI) attacks that replay video frames from a previous recording to mask the live feed has the highest impact. While many attempts have been made to detect FFI frames using features from the video feeds, video analysis is computationally too intensive to be deployed on-site for real-time false frame detection. In this paper, we investigated the feasibility of FFI attacks on compromised surveillance systems at the edge and propose an effective technique to detect the injected false video and audio frames by monitoring the surveillance feed using the embedded Electrical Network Frequency (ENF) signals. An ENF operates at a nominal frequency of 60 Hz/50 Hz based on its geographical location and maintains a stable value across the entire power grid interconnection with minor fluctuations. For surveillance system video/audio recordings connected to the power grid, the ENF signals are embedded. The time-varying nature of the ENF component was used as a forensic application for authenticating the surveillance feed. The paper highlights the ENF signal collection from a power grid creating a reference database and ENF extraction from the recordings using conventional short-time Fourier Transform and spectrum detection for robust ENF signal analysis in the presence of noise and interference caused in different harmonics. The experimental results demonstrated the effectiveness of ENF signal detection and/or abnormalities for FFI attacks.

A Real-Time High Performance Computation Architecture for Multiple Moving Target Tracking Based on Wide-Area Motion Imagery via Cloud and Graphic Processing Units.

This paper presents the first attempt at combining Cloud with Graphic Processing Units (GPUs) in a complementary manner within the framework of a real-time high performance computation architecture for the application of detecting and tracking multiple moving targets based on Wide Area Motion Imagery (WAMI). More specifically, the GPU and Cloud Moving Target Tracking (GC-MTT) system applied a front-end web based server to perform the interaction with Hadoop and highly parallelized computation functions based on the Compute Unified Device Architecture (CUDA©). The introduced multiple moving target detection and tracking method can be extended to other applications such as pedestrian tracking, group tracking, and Patterns of Life (PoL) analysis. The cloud and GPUs based computing provides an efficient real-time target recognition and tracking approach as compared to methods when the work flow is applied using only central processing units (CPUs). The simultaneous tracking and recognition results demonstrate that a GC-MTT based approach provides drastically improved tracking with low frame rates over realistic conditions.

Encoding color information for visual tracking: Algorithms and benchmark.

While color information is known to provide rich discriminative clues for visual inference, most modern visual trackers limit themselves to the grayscale realm. Despite recent efforts to integrate color in tracking, there is a lack of comprehensive understanding of the role color information can play. In this paper, we attack this problem by conducting a systematic study from both the algorithm and benchmark perspectives. On the algorithm side, we comprehensively encode 10 chromatic models into 16 carefully selected state-of-the-art visual trackers. On the benchmark side, we compile a large set of 128 color sequences with ground truth and challenge factor annotations (e.g., occlusion). A thorough evaluation is conducted by running all the color-encoded trackers, together with two recently proposed color trackers. A further validation is conducted on an RGBD tracking benchmark. The results clearly show the benefit of encoding color information for tracking. We also perform detailed analysis on several issues, including the behavior of various combinations between color model and visual tracker, the degree of difficulty of each sequence for tracking, and how different challenge factors affect the tracking performance. We expect the study to provide the guidance, motivation, and benchmark for future work on encoding color in visual tracking.

Automatic association of chats and video tracks for activity learning and recognition in aerial video surveillance.

We describe two advanced video analysis techniques, including video-indexed by voice annotations (VIVA) and multi-media indexing and explorer (MINER). VIVA utilizes analyst call-outs (ACOs) in the form of chat messages (voice-to-text) to associate labels with video target tracks, to designate spatial-temporal activity boundaries and to augment video tracking in challenging scenarios. Challenging scenarios include low-resolution sensors, moving targets and target trajectories obscured by natural and man-made clutter. MINER includes: (1) a fusion of graphical track and text data using probabilistic methods; (2) an activity pattern learning framework to support querying an index of activities of interest (AOIs) and targets of interest (TOIs) by movement type and geolocation; and (3) a user interface to support streaming multi-intelligence data processing. We also present an activity pattern learning framework that uses the multi-source associated data as training to index a large archive of full-motion videos (FMV). VIVA and MINER examples are demonstrated for wide aerial/overhead imagery over common data sets affording an improvement in tracking from video data alone, leading to 84% detection with modest misdetection/false alarm results due to the complexity of the scenario. The novel use of ACOs and chat Sensors 2014, 14 19844 messages in video tracking paves the way for user interaction, correction and preparation of situation awareness reports.

Efficient minimum error bounded particle resampling L1 tracker with occlusion detection.

Recently, sparse representation has been applied to visual tracking to find the target with the minimum reconstruction error from a target template subspace. Though effective, these L1 trackers require high computational costs due to numerous calculations for l1 minimization. In addition, the inherent occlusion insensitivity of the l1 minimization has not been fully characterized. In this paper, we propose an efficient L1 tracker, named bounded particle resampling (BPR)-L1 tracker, with a minimum error bound and occlusion detection. First, the minimum error bound is calculated from a linear least squares equation and serves as a guide for particle resampling in a particle filter (PF) framework. Most of the insignificant samples are removed before solving the computationally expensive l1 minimization in a two-step testing. The first step, named τ testing, compares the sample observation likelihood to an ordered set of thresholds to remove insignificant samples without loss of resampling precision. The second step, named max testing, identifies the largest sample probability relative to the target to further remove insignificant samples without altering the tracking result of the current frame. Though sacrificing minimal precision during resampling, max testing achieves significant speed up on top of τ testing. The BPR-L1 technique can also be beneficial to other trackers that have minimum error bounds in a PF framework, especially for trackers based on sparse representations. After the error-bound calculation, BPR-L1 performs occlusion detection by investigating the trivial coefficients in the l1 minimization. These coefficients, by design, contain rich information about image corruptions, including occlusion. Detected occlusions are then used to enhance the template updating. For evaluation, we conduct experiments on three video applications: biometrics (head movement, hand holding object, singers on stage), pedestrians (urban travel, hallway monitoring), and cars in traffic (wide area motion imagery, ground-mounted perspectives). The proposed BPR-L1 method demonstrates an excellent performance as compared with nine state-of-the-art trackers on eleven challenging benchmark sequences.

Real-time probabilistic covariance tracking with efficient model update.

The recently proposed covariance region descriptor has been proven robust and versatile for a modest computational cost. The covariance matrix enables efficient fusion of different types of features, where the spatial and statistical properties, as well as their correlation, are characterized. The similarity between two covariance descriptors is measured on Riemannian manifolds. Based on the same metric but with a probabilistic framework, we propose a novel tracking approach on Riemannian manifolds with a novel incremental covariance tensor learning (ICTL). To address the appearance variations, ICTL incrementally learns a low-dimensional covariance tensor representation and efficiently adapts online to appearance changes of the target with only O(1) computational complexity, resulting in a real-time performance. The covariance-based representation and the ICTL are then combined with the particle filter framework to allow better handling of background clutter, as well as the temporary occlusions. We test the proposed probabilistic ICTL tracker on numerous benchmark sequences involving different types of challenges including occlusions and variations in illumination, scale, and pose. The proposed approach demonstrates excellent real-time performance, both qualitatively and quantitatively, in comparison with several previously proposed trackers.

Multiscale analysis of biological data by scale-dependent lyapunov exponent.

Physiological signals often are highly non-stationary (i.e., mean and variance change with time) and multiscaled (i.e., dependent on the spatial or temporal interval lengths). They may exhibit different behaviors, such as non-linearity, sensitive dependence on small disturbances, long memory, and extreme variations. Such data have been accumulating in all areas of health sciences and rapid analysis can serve quality testing, physician assessment, and patient diagnosis. To support patient care, it is very desirable to characterize the different signal behaviors on a wide range of scales simultaneously. The Scale-Dependent Lyapunov Exponent (SDLE) is capable of such a fundamental task. In particular, SDLE can readily characterize all known types of signal data, including deterministic chaos, noisy chaos, random 1/f(α) processes, stochastic limit cycles, among others. SDLE also has some unique capabilities that are not shared by other methods, such as detecting fractal structures from non-stationary data and detecting intermittent chaos. In this article, we describe SDLE in such a way that it can be readily understood and implemented by non-mathematically oriented researchers, develop a SDLE-based consistent, unifying theory for the multiscale analysis, and demonstrate the power of SDLE on analysis of heart-rate variability (HRV) data to detect congestive heart failure and analysis of electroencephalography (EEG) data to detect seizures.