Lip Reading by Alternating between Spatiotemporal and Spatial Convolutions.

Lip reading (LR) is the task of predicting the speech utilizing only the visual information of the speaker. In this work, for the first time, the benefits of alternating between spatiotemporal and spatial convolutions for learning effective features from the LR sequences are studied. In this context, a new learnable module named ALSOS (Alternating Spatiotemporal and Spatial Convolutions) is introduced in the proposed LR system. The ALSOS module consists of spatiotemporal (3D) and spatial (2D) convolutions along with two conversion components (3D-to-2D and 2D-to-3D) providing a sequence-to-sequence-mapping. The designed LR system utilizes the ALSOS module in-between ResNet blocks, as well as Temporal Convolutional Networks (TCNs) in the backend for classification. The whole framework is composed by feedforward convolutional along with residual layers and can be trained end-to-end directly from the image sequences in the word-level LR problem. The ALSOS module can capture spatiotemporal dynamics and can be advantageous in the task of LR when combined with the ResNet topology. Experiments with different combinations of ALSOS with ResNet are performed on a dataset in Greek language simulating a medical support application scenario and on the popular large-scale LRW-500 dataset of English words. Results indicate that the proposed ALSOS module can improve the performance of a LR system. Overall, the insertion of ALSOS module into the ResNet architecture obtained higher classification accuracy since it incorporates the contribution of the temporal information captured at different spatial scales of the framework.

Muscle Type and Gender Recognition Utilising High-Level Textural Representation in Musculoskeletal Ultrasonography.

Human assistive technology and computer-aided diagnosis is an emerging field in the area of medical imaging. Following the recent advances in this domain, a study for integrating machine learning techniques in musculoskeletal ultrasonography images was conducted. The goal of this attempt was to investigate how feature extraction techniques, that capture higher-level information, perform in identifying human characteristics. The potential success of these techniques could lead to significant improvement of the current assessment methods-as the gray-scale image analysis-for distinguishing healthy and pathologic conditions, that are heavily dependent on the image-acquisition system. The contribution of this work is threefold. First, a new privately held data set of 74 healthy patients was presented. This data set included musculoskeletal ultrasound images from four muscles of the human body, namely the biceps brachii, tibialis anterior, gastrocnemius medialis and rectus femoris, recorded in the transverse and longitudinal plane. Second, two classification tasks were performed, namely, gender and muscle-type recognition, to assess the performance of the proposed method for successfully identifying differences in the texture of the examined muscle sections. Third, a novel method used with great success in the computer vision domain was presented, allowing the extraction of a high-level feature representation, by encoding the distribution of locally invariant texture descriptors. On the muscle-type recognition our method achieved an 87.07% classification rate, and on the task of gender recognition it surpassed state-of-the-art textural representations, reported in the literature in almost all the examined muscle sections.

A novel symbolization scheme for multichannel recordings with emphasis on phase information and its application to differentiate EEG activity from different mental tasks.

UNLABELLED: Symbolic dynamics is a powerful tool for studying complex dynamical systems. So far many techniques of this kind have been proposed as a means to analyze brain dynamics, but most of them are restricted to single-sensor measurements. Analyzing the dynamics in a channel-wise fashion is an invalid approach for multisite encephalographic recordings, since it ignores any pattern of coordinated activity that might emerge from the coherent activation of distinct brain areas. We suggest, here, the use of neural-gas algorithm (Martinez et al. in IEEE Trans Neural Netw 4:558-569, 1993) for encoding brain activity spatiotemporal dynamics in the form of a symbolic timeseries. A codebook of k prototypes, best representing the instantaneous multichannel data, is first designed. Each pattern of activity is then assigned to the most similar code vector. The symbolic timeseries derived in this way is mapped to a network, the topology of which encapsulates the most important phase transitions of the underlying dynamical system. Finally, global efficiency is used to characterize the obtained topology. We demonstrate the approach by applying it to EEG-data recorded from subjects while performing mental calculations. By working in a contrastive-fashion, and focusing in the phase aspects of the signals, we show that the underlying dynamics differ significantly in their symbolic representations. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1007/s11571-011-9186-5) contains supplementary material, which is available to authorized users.

Tracking brain dynamics via time-dependent network analysis.

Complex network analysis is currently employed in neuroscience research to describe the neuron pathways in the brain with a small number of computable measures that have neurobiological meaning. Connections in biological neural networks might fluctuate over time; therefore, surveillance can provide a more useful picture of brain dynamics than the standard approach that relies on a static graph to represent functional connectivity. Using the application of well-known measures of neural synchrony over short segments of brain activity in a time series, we attempted a time-dependent characterization of brain connectivity by investigating functional segregation and integration. In our implementation, a frequency-dependent time window was employed and regularly spaced (defined as overlapping segments), and a novel, parameter-free method was introduced to derive the required adjacency matrices. The resulting characterization was compared against conventional approaches that rely on static and time-evolving graphs, which are constructed from non-overlapping segments of arbitrarily defined durations. Our approach is demonstrated using EEG recordings during mental calculations. The derived consecutive values of network metrics were then compared with values from randomized networks. The results revealed the dynamic small-world character of the brain's functional connectivity, which otherwise can be hidden from estimators that rely on either long or stringent time-windows. Moreover, by involving a network-metric time series (NMTS) in a summarizing procedure that was based on replicator dynamics, consistent hubs that facilitated communication in the underlying networks were identified. Finally, the scale-free character of brain networks was also demonstrated based on the significant edges selected with the introduced approach.

A region dissimilarity relation that combines feature-space and spatial information for color image segmentation.

This paper proposes a methodology that incorporates principles from cluster analysis and graph representation to achieve efficient image segmentation results. More specifically, a feature-based, inter-region dissimilarity relation is considered here in order to determine the dissimilarity matrix in a graph-based segmentation scheme. The calculation of the dissimilarity function between adjacent elementary image regions is based on the proximity of each region's feature vector to the main clusters that are formed by the image samples in the feature space. In contrast to typical segmentation approaches of the literature, the global feature space information is included in the spatial graph representation that was derived from the initial Watershed partitioning. A region grouping process is applied next to form the final segmentation results. The proposed approach was also compared to approaches that use feature-based, or spatial information exclusively, to indicate its effectiveness.