A comprehensive multimodal dataset for contactless lip reading and acoustic analysis.

Small-scale motion detection using non-invasive remote sensing techniques has recently garnered significant interest in the field of speech recognition. Our dataset paper aims to facilitate the enhancement and restoration of speech information from diverse data sources for speakers. In this paper, we introduce a novel multimodal dataset based on Radio Frequency, visual, text, audio, laser and lip landmark information, also called RVTALL. Specifically, the dataset consists of 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77 GHz frequency modulated continuous wave (FMCW) data from millimeter wave (mmWave) radar, visual and audio information, lip landmarks and laser data, offering a unique multimodal approach to speech recognition research. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words, and 16 sentences. The dataset has been validated and has potential for the investigation of lip reading and multimodal speech recognition.

OPERAnet, a multimodal activity recognition dataset acquired from radio frequency and vision-based sensors.

This paper presents a comprehensive dataset intended to evaluate passive Human Activity Recognition (HAR) and localization techniques with measurements obtained from synchronized Radio-Frequency (RF) devices and vision-based sensors. The dataset consists of RF data including Channel State Information (CSI) extracted from a WiFi Network Interface Card (NIC), Passive WiFi Radar (PWR) built upon a Software Defined Radio (SDR) platform, and Ultra-Wideband (UWB) signals acquired via commercial off-the-shelf hardware. It also consists of vision/Infra-red based data acquired from Kinect sensors. Approximately 8 hours of annotated measurements are provided, which are collected across two rooms from 6 participants performing 6 daily activities. This dataset can be exploited to advance WiFi and vision-based HAR, for example, using pattern recognition, skeletal representation, deep learning algorithms or other novel approaches to accurately recognize human activities. Furthermore, it can potentially be used to passively track a human in an indoor environment. Such datasets are key tools required for the development of new algorithms and methods in the context of smart homes, elderly care, and surveillance applications.

Through Wall Radar Classification of Human Micro-Doppler Using Singular Value Decomposition Analysis.

The ability to detect the presence as well as classify the activities of individuals behind visually obscuring structures is of significant benefit to police, security and emergency services in many situations. This paper presents the analysis from a series of experimental results generated using a through-the-wall (TTW) Frequency Modulated Continuous Wave (FMCW) C-Band radar system named Soprano. The objective of this analysis was to classify whether an individual was carrying an item in both hands or not using micro-Doppler information from a FMCW sensor. The radar was deployed at a standoff distance, of approximately 0.5 m, outside a residential building and used to detect multiple people walking within a room. Through the application of digital filtering, it was shown that significant suppression of the primary wall reflection is possible, significantly enhancing the target signal to clutter ratio. Singular Value Decomposition (SVD) signal processing techniques were then applied to the micro-Doppler signatures from different individuals. Features from the SVD information have been used to classify whether the person was carrying an item or walking free handed. Excellent performance of the classifier was achieved in this challenging scenario with accuracies up to 94%, suggesting that future through wall radar sensors may have the ability to reliably recognize many different types of activities in TTW scenarios using these techniques.

High-speed optical observations and simulation results of SonoVue microbubbles at low-pressure insonation.

Abstract-Modified Rayleigh-Plesset models are commonly used to characterize the acoustic response of microbubbles under ultrasound exposure. In most instances these models have been parameterized through acoustic measurements taken from bulk suspensions of microbubbles. The aim of this study was to parameterize the Hoff model for the commercial contrast agent SonoVue using optically observed oscillations from individual microbubbles recorded with a high-speed camera. The shell elasticity model term was tuned to fit simulation data to the measured oscillations while the shell viscosity parameter was held constant at 1 Pa??s. The results demonstrate a limited ability of the model to predict the microbubble behavior. The shell elasticity parameter was found to vary proportionally between 10 and 80 MPa with the initial microbubble diameter, implying the viscoelastic shell terms are not a constant property of the shell material. Further analysis using a moving window optimization to probe the microbubble responses suggests that the elasticity of the shell can increase by up to 50% over the course of insonation, particularly for microbubbles oscillating nearer to their resonant frequency. Microbubble oscillations were modeled more successfully by incorporating a varying elasticity term into the model.

Microbubble contrast agent detection using binary coded pulses.

Real-time visualization of microbubbles in the microvasculature of deep tissues remains a challenge for existing nonlinear microbubble imaging techniques. A technique with high sensitivity to nonlinear signals is required to compensate for the effects of limited power used to avoid bubble destruction and the high attenuation of the overlying tissues for deeper targets. The use of coded pulses in ultrasound imaging is well established as a means of improving the signal-to-noise ratio (SNR) within B-mode ultrasound imaging, but the feasibility of this approach for detecting microbubbles has not been well studied. In this work we investigate the use of binary phase encoding together with phase and amplitude modulation (PIAM) for the detection of nonlinear signals from microbubbles. A series of simulation experiments were conducted using a modified Rayleigh-Plesset model together with Golay and Barker coding techniques to investigate (i) the ability of binary encoded PIAM to detect nonlinear signals, (ii) the effect of the SNR and insonating pressure on the detection process, (iii) the sensitivity of different pulse encoding approaches and (iv) the effects of bubble resonance behavior on the detection process. The results show that the binary encoding approach combined with PIAM is able to detect nonlinear signals from microbubbles. It was found that nonlinear scattering from the microbubbles degrades the sensitivity of the binary encoded approach such that at high SNR there is no advantage in using these pulses over existing short-pulse PIAM. However, at lower SNR (<20 dB) the increased pulse length provides improved sensitivity without significant loss of spatial resolution, even under conditions in which the detection failed completely for existing approaches. The results also show that both the insonating acoustic pressure and resonance behavior of bubbles have an effect on the detection sensitivity and spatial resolution for the binary encoded approach.

Investigating the nonlinear microbubble response to chirp encoded, multipulse sequences.

A modified Rayleigh-Plesset model was used to investigate the nonlinear acoustic response of ultrasound contrast microbubbles to multipulse phase and amplitude modulated, chirp encoded sequences. Trade-offs between the signal-to-noise ratio (SNR) and axial resolution were quantified for differing chirp time-bandwidth products and methods for minimising the artifacts formed in the postprocessing stages were developed. It was found that the chirp length can be increased and bandwidth reduced to improve SNR, though resolution is sacrificed. Results from the simulated chirp, pulse inverted, amplitude modulated (chirp PIAM) sequences were also compared with equivalent short pulse PIAM sequences and it was found that the chirp sequences preserve their extra energy after scattering, which translates to an improved SNR after processing. Compression artifacts were reduced by using chirps with a centre frequency and bandwidth tuned to the frequency response of the microbubble and reversing the frequency sweep of one chirp in the sequence.