On the value of CTIS imagery for neural-network-based classification: a simulation perspective.

The computed tomography imaging spectrometer (CTIS) is a snapshot hyperspectral imaging system. Its output is a 2D image of multiplexed spatiospectral projections of the hyperspectral cube of the scene. Traditionally, the 3D cube is reconstructed from this image before further analysis. In this paper, we show that it is possible to learn information directly from the CTIS raw output, by training a neural network to perform binary classification on such images. The use case we study is an agricultural one, as snapshot imagery is used substantially in this field: the detection of apple scab lesions on leaves. To train the network appropriately and to study several degrees of scab infection, we simulated CTIS images of scabbed leaves. This was made possible with a novel CTIS simulator, where special care was taken to preserve realistic pixel intensities compared to true images. To the best of our knowledge, this is the first application of compressed learning on a simulated CTIS system.

Semantic Event Fusion of Different Visual Modality Concepts for Activity Recognition.

Combining multimodal concept streams from heterogeneous sensors is a problem superficially explored for activity recognition. Most studies explore simple sensors in nearly perfect conditions, where temporal synchronization is guaranteed. Sophisticated fusion schemes adopt problem-specific graphical representations of events that are generally deeply linked with their training data and focused on a single sensor. This paper proposes a hybrid framework between knowledge-driven and probabilistic-driven methods for event representation and recognition. It separates semantic modeling from raw sensor data by using an intermediate semantic representation, namely concepts. It introduces an algorithm for sensor alignment that uses concept similarity as a surrogate for the inaccurate temporal information of real life scenarios. Finally, it proposes the combined use of an ontology language, to overcome the rigidity of previous approaches at model definition, and a probabilistic interpretation for ontological models, which equips the framework with a mechanism to handle noisy and ambiguous concept observations, an ability that most knowledge-driven methods lack. We evaluate our contributions in multimodal recordings of elderly people carrying out IADLs. Results demonstrated that the proposed framework outperforms baseline methods both in event recognition performance and in delimiting the temporal boundaries of event instances.