Finding Explanations in AI Fusion of Electro-Optical/Passive Radio-Frequency Data.

In the Information Age, the widespread usage of blackbox algorithms makes it difficult to understand how data is used. The practice of sensor fusion to achieve results is widespread, as there are many tools to further improve the robustness and performance of a model. In this study, we demonstrate the utilization of a Long Short-Term Memory (LSTM-CCA) model for the fusion of Passive RF (P-RF) and Electro-Optical (EO) data in order to gain insights into how P-RF data are utilized. The P-RF data are constructed from the in-phase and quadrature component (I/Q) data processed via histograms, and are combined with enhanced EO data via dense optical flow (DOF). The preprocessed data are then used as training data with an LSTM-CCA model in order to achieve object detection and tracking. In order to determine the impact of the different data inputs, a greedy algorithm (explainX.ai) is implemented to determine the weight and impact of the canonical variates provided to the fusion model on a scenario-by-scenario basis. This research introduces an explainable LSTM-CCA framework for P-RF and EO sensor fusion, providing novel insights into the sensor fusion process that can assist in the detection and differentiation of targets and help decision-makers to determine the weights for each input.

Interpretable Passive Multi-Modal Sensor Fusion for Human Identification and Activity Recognition.

Human monitoring applications in indoor environments depend on accurate human identification and activity recognition (HIAR). Single modality sensor systems have shown to be accurate for HIAR, but there are some shortcomings to these systems, such as privacy, intrusion, and costs. To combat these shortcomings for a long-term monitoring solution, an interpretable, passive, multi-modal, sensor fusion system PRF-PIR is proposed in this work. PRF-PIR is composed of one software-defined radio (SDR) device and one novel passive infrared (PIR) sensor system. A recurrent neural network (RNN) is built as the HIAR model for this proposed solution to handle the temporal dependence of passive information captured by both modalities. We validate our proposed PRF-PIR system for a potential human monitoring system through the data collection of eleven activities from twelve human subjects in an academic office environment. From our data collection, the efficacy of the sensor fusion system is proven via an accuracy of 0.9866 for human identification and an accuracy of 0.9623 for activity recognition. The results of the system are supported with explainable artificial intelligence (XAI) methodologies to serve as a validation for sensor fusion over the deployment of single sensor solutions. PRF-PIR provides a passive, non-intrusive, and highly accurate system that allows for robustness in uncertain, highly similar, and complex at-home activities performed by a variety of human subjects.

Human Occupancy Detection via Passive Cognitive Radio.

Human occupancy detection (HOD) in an enclosed space, such as indoors or inside of a vehicle, via passive cognitive radio (CR) is a new and challenging research area. Part of the difficulty arises from the fact that a human subject cannot easily be detected due to spectrum variation. In this paper, we present an advanced HOD system that dynamically reconfigures a CR to collect passive radio frequency (RF) signals at different places of interest. Principal component analysis (PCA) and recursive feature elimination with logistic regression (RFE-LR) algorithms are applied to find the frequency bands sensitive to human occupancy when the baseline spectrum changes with locations. With the dynamically collected passive RF signals, four machine learning (ML) classifiers are applied to detect human occupancy, including support vector machine (SVM), k-nearest neighbors (KNN), decision tree (DT), and linear SVM with stochastic gradient descent (SGD) training. The experimental results show that the proposed system can accurately detect human subjects-not only in residential rooms-but also in commercial vehicles, demonstrating that passive CR is a viable technique for HOD. More specifically, the RFE-LR with SGD achieves the best results with a limited number of frequency bands. The proposed adaptive spectrum sensing method has not only enabled robust detection performance in various environments, but also improved the efficiency of the CR system in terms of speed and power consumption.