Correction: Impacts of multicollinearity on CAPT modalities: An heterogeneous machine learning framework for computer-assisted French phoneme pronunciation training.

[This corrects the article DOI: 10.1371/journal.pone.0257901.].

Performance analysis of remote photoplethysmography deep filtering using long short-term memory neural network.

BACKGROUND: Remote photoplethysmography (rPPG) is a technique developed to estimate heart rate using standard video cameras and ambient light. Due to the multiple sources of noise that deteriorate the quality of the signal, conventional filters such as the bandpass and wavelet-based filters are commonly used. However, after using conventional filters, some alterations remain, but interestingly an experienced eye can easily identify them. RESULTS: We studied a long short-term memory (LSTM) network in the rPPG filtering task to identify these alterations using many-to-one and many-to-many approaches. We used three public databases in intra-dataset and cross-dataset scenarios, along with different protocols to analyze the performance of the method. We demonstrate how the network can be easily trained with a set of 90 signals totaling around 45 min. On the other hand, we show the stability of the LSTM performance with six state-of-the-art rPPG methods. CONCLUSIONS: This study demonstrates the superiority of the LSTM-based filter experimentally compared with conventional filters in an intra-dataset scenario. For example, we obtain on the VIPL database an MAE of 3.9 bpm, whereas conventional filtering improves performance on the same dataset from 10.3 bpm to 7.7 bpm. The cross-dataset approach presents a dependence in the network related to the average signal-to-noise ratio on the rPPG signals, where the closest signal-to-noise ratio values in the training and testing set the better. Moreover, it was demonstrated that a relatively small amount of data are sufficient to successfully train the network and outperform the results obtained by classical filters. More precisely, we have shown that about 45 min of rPPG signal could be sufficient to train an effective LSTM deep-filter.

Automatic Assessment of Emotion Dysregulation in American, French, and Tunisian Adults and New Developments in Deep Multimodal Fusion: Cross-sectional Study.

BACKGROUND: Emotion dysregulation is a key dimension of adult psychological functioning. There is an interest in developing a computer-based, multimodal, and automatic measure. OBJECTIVE: We wanted to train a deep multimodal fusion model to estimate emotion dysregulation in adults based on their responses to the Multimodal Developmental Profile, a computer-based psychometric test, using only a small training sample and without transfer learning. METHODS: Two hundred and forty-eight participants from 3 different countries took the Multimodal Developmental Profile test, which exposed them to 14 picture and music stimuli and asked them to express their feelings about them, while the software extracted the following features from the video and audio signals: facial expressions, linguistic and paralinguistic characteristics of speech, head movements, gaze direction, and heart rate variability derivatives. Participants also responded to the brief version of the Difficulties in Emotional Regulation Scale. We separated and averaged the feature signals that corresponded to the responses to each stimulus, building a structured data set. We transformed each person's per-stimulus structured data into a multimodal codex, a grayscale image created by projecting each feature's normalized intensity value onto a cartesian space, deriving each pixel's position by applying the Uniform Manifold Approximation and Projection method. The codex sequence was then fed to 2 network types. First, 13 convolutional neural networks dealt with the spatial aspect of the problem, estimating emotion dysregulation by analyzing each of the codified responses. These convolutional estimations were then fed to a transformer network that decoded the temporal aspect of the problem, estimating emotional dysregulation based on the succession of responses. We introduce a Feature Map Average Pooling layer, which computes the mean of the convolved feature maps produced by our convolution layers, dramatically reducing the number of learnable weights and increasing regularization through an ensembling effect. We implemented 8-fold cross-validation to provide a good enough estimation of the generalization ability to unseen samples. Most of the experiments mentioned in this paper are easily replicable using the associated Google Colab system. RESULTS: We found an average Pearson correlation (r) of 0.55 (with an average P value of <.001) between ground truth emotion dysregulation and our system's estimation of emotion dysregulation. An average mean absolute error of 0.16 and a mean concordance correlation coefficient of 0.54 were also found. CONCLUSIONS: In psychometry, our results represent excellent evidence of convergence validity, suggesting that the Multimodal Developmental Profile could be used in conjunction with this methodology to provide a valid measure of emotion dysregulation in adults. Future studies should replicate our findings using a hold-out test sample. Our methodology could be implemented more generally to train deep neural networks where only small training samples are available.

Impacts of multicollinearity on CAPT modalities: An heterogeneous machine learning framework for computer-assisted French phoneme pronunciation training.

Phoneme pronunciations are usually considered as basic skills for learning a foreign language. Practicing the pronunciations in a computer-assisted way is helpful in a self-directed or long-distance learning environment. Recent researches indicate that machine learning is a promising method to build high-performance computer-assisted pronunciation training modalities. Many data-driven classifying models, such as support vector machines, back-propagation networks, deep neural networks and convolutional neural networks, are increasingly widely used for it. Yet, the acoustic waveforms of phoneme are essentially modulated from the base vibrations of vocal cords, and this fact somehow makes the predictors collinear, distorting the classifying models. A commonly-used solution to address this issue is to suppressing the collinearity of predictors via partial least square regressing algorithm. It allows to obtain high-quality predictor weighting results via predictor relationship analysis. However, as a linear regressor, the classifiers of this type possess very simple topology structures, constraining the universality of the regressors. For this issue, this paper presents an heterogeneous phoneme recognition framework which can further benefit the phoneme pronunciation diagnostic tasks by combining the partial least square with support vector machines. A French phoneme data set containing 4830 samples is established for the evaluation experiments. The experiments of this paper demonstrates that the new method improves the accuracy performance of the phoneme classifiers by 0.21 - 8.47% comparing to state-of-the-arts with different data training data density.

Multimodal imaging as optical biopsy system for gastritis diagnosis in humans, and input of the mouse model.

BACKGROUND: Gastric inflammation is a major risk factor for gastric cancer. Current endoscopic methods are not able to efficiently detect and characterize gastric inflammation, leading to a sub-optimal patients' care. New non-invasive methods are needed. Reflectance mucosal light analysis is of particular interest in this context. The aim of our study was to analyze reflectance light and specific autofluorescence signals, both in humans and in a mouse model of gastritis. METHODS: We recruited patients undergoing gastroendoscopic procedure during which reflectance was analysed with a multispectral camera. In parallel, the gastritis mouse model of Helicobacter pylori infection was used to investigate reflectance from ex vivo gastric samples using a spectrometer. In both cases, autofluorescence signals were measured using a confocal microscope. FINDINGS: In gastritis patients, reflectance modifications were significant in near-infrared spectrum, with a decrease between 610 and 725 nm and an increase between 750 and 840 nm. Autofluorescence was also modified, showing variations around 550 nm of emission. In H. pylori infected mice developing gastric inflammatory lesions, we observed significant reflectance modifications 18 months after infection, with increased intensity between 617 and 672 nm. Autofluorescence was significantly modified after 1, 3 and 6 months around 550 and 630 nm. Both in human and in mouse, these reflectance data can be considered as biomarkers and accurately predicted inflammatory state. INTERPRETATION: In this pilot study, using a practical measuring device, we identified in humans, modification of reflectance spectra in the visible spectrum and for the first time in near-infrared, associated with inflammatory gastric states. Furthermore, both in the mouse model and humans, we also observed modifications of autofluorescence associated with gastric inflammation. These innovative data pave the way to deeper validation studies on larger cohorts, for further development of an optical biopsy system to detect gastritis and finally to better surveil this important gastric cancer risk factor. FUNDING: The project was funded by the ANR EMMIE (ANR-15-CE17-0015) and the French Gastroenterology Society (SNFGE).

Multispectral imaging detects gastritis consistently in mouse model and in humans.

Gastritis constitutes the initial step of the gastric carcinogenesis process. Gastritis diagnosis is based on histological examination of biopsies. Non-invasive real-time methods to detect mucosal inflammation are needed. Tissue optical properties modify reemitted light, i.e. the proportion of light that is emitted by a tissue after stimulation by a light flux. Analysis of light reemitted by gastric tissue could predict the inflammatory state. The aim of our study was to investigate a potential association between reemitted light and gastric tissue inflammation. We used two models and three multispectral analysis methods available on the marketplace. We used a mouse model of Helicobacter pylori infection and included patients undergoing gastric endoscopy. In mice, the reemitted light was measured using a spectrometer and a multispectral camera. We also exposed patient's gastric mucosa to specific wavelengths and analyzed reemitted light. In both mouse model and humans, modifications of reemitted light were observed around 560 nm, 600 nm and 640 nm, associated with the presence of gastritis lesions. These results pave the way for the development of improved endoscopes in order to detect real-time gastritis without the need of biopsies. This would allow a better prevention of gastric cancer alongside with cost efficient endoscopies.

Video-Based Pulse Rate Variability Measurement Using Periodic Variance Maximization and Adaptive Two-Window Peak Detection.

Many previous studies have shown that the remote photoplethysmography (rPPG) can measure the Heart Rate (HR) signal with very high accuracy. The remote measurement of the Pulse Rate Variability (PRV) signal is also possible, but this is much more complicated because it is then necessary to detect the peaks on the temporal rPPG signal, which is usually quite noisy and has a lower temporal resolution than PPG signals obtained by contact equipment. Since the PRV signal is vital for various applications such as remote recognition of stress and emotion, the improvement of PRV measurement by rPPG is a critical task. Contact based PRV measurement has already been investigated, but the research on remotely measured PRV is very limited. In this paper, we propose to use the Periodic Variance Maximization (PVM) method to extract the rPPG signal and event-related Two-Window algorithm to improve the peak detection for PRV measurement. We have made several contributions. Firstly, we show that the newly proposed PVM method and Two-Window algorithm can be used for PRV measurement in the non-contact scenario. Secondly, we propose a method to adaptively determine the parameters of the Two-Window method. Thirdly, we compare the algorithm with other attempts for improving the non-contact PRV measurement such as the Slope Sum Function (SSF) method and the Local Maximum method. We calculated several features and compared the accuracy based on the ground truth provided by contact equipment. Our experiments showed that this algorithm performed the best of all the algorithms.

Intrinsic RGB and multispectral images recovery by independent quadratic programming.

This work introduces a method to estimate reflectance, shading, and specularity from a single image. Reflectance, shading, and specularity are intrinsic images derived from the dichromatic model. Estimation of these intrinsic images has many applications in computer vision such as shape recovery, specularity removal, segmentation, or classification. The proposed method allows for recovering the dichromatic model parameters thanks to two independent quadratic programming steps. Compared to the state of the art in this domain, our approach has the advantage to address a complex inverse problem into two parallelizable optimization steps that are easy to solve and do not require learning. The proposed method is an extension of a previous algorithm that is rewritten to be numerically more stable, has better quantitative and qualitative results, and applies to multispectral images. The proposed method is assessed qualitatively and quantitatively on standard RGB and multispectral datasets.

Remote photoplethysmography with constrained ICA using periodicity and chrominance constraints.

BACKGROUND: Remote photoplethysmography (rPPG) has been in the forefront recently for measuring cardiac pulse rates from live or recorded videos. It finds advantages in scenarios requiring remote monitoring, such as medicine and fitness, where contact based monitoring is limiting and cumbersome. The blood volume pulse, defined as the pulsative flow of arterial blood, gives rise to periodic changes in the skin color which are then quantified to estimate a temporal signal. This temporal signal can be analysed using various methods to extract the representative cardiac signal. METHODS: We present a novel method for measuring rPPG signals using constrained independent component analysis (cICA). We incorporate a priori information into the cICA algorithm to aid in the extraction of the most prominent rPPG signal. This a priori information is implemented using two constraints: first, based on periodicity using autocorrelation, and second, a chrominance-based constraint exploiting the physical characteristics of the skin. RESULTS AND CONCLUSION: Our method showed improved performances over traditional blind source separation methods like ICA and chrominance based methods with mean absolute errors of 0.62 beats per minute (BPM) and 3.14 BPM for the two datasets in our inhouse video database UBFC-RPPG, and 4.69 BPM for the public MMSE-HR dataset. Its performance was also better in comparison to other state of the art methods in terms of accuracy and robustness. Our UBFC-RPPG database is also made publicly available and is specifically aimed towards testing rPPG measurements.

Skin parameter map retrieval from a dedicated multispectral imaging system applied to dermatology/cosmetology.

In vivo quantitative assessment of skin lesions is an important step in the evaluation of skin condition. An objective measurement device can help as a valuable tool for skin analysis. We propose an explorative new multispectral camera specifically developed for dermatology/cosmetology applications. The multispectral imaging system provides images of skin reflectance at different wavebands covering visible and near-infrared domain. It is coupled with a neural network-based algorithm for the reconstruction of reflectance cube of cutaneous data. This cube contains only skin optical reflectance spectrum in each pixel of the bidimensional spatial information. The reflectance cube is analyzed by an algorithm based on a Kubelka-Munk model combined with evolutionary algorithm. The technique allows quantitative measure of cutaneous tissue and retrieves five skin parameter maps: melanin concentration, epidermis/dermis thickness, haemoglobin concentration, and the oxygenated hemoglobin. The results retrieved on healthy participants by the algorithm are in good accordance with the data from the literature. The usefulness of the developed technique was proved during two experiments: a clinical study based on vitiligo and melasma skin lesions and a skin oxygenation experiment (induced ischemia) with healthy participant where normal tissues are recorded at normal state and when temporary ischemia is induced.

Posture recognition based on fuzzy logic for home monitoring of the elderly.

We propose in this paper a computer vision-based posture recognition method for home monitoring of the elderly. The proposed system performs human detection prior to the posture analysis; posture recognition is performed only on a human silhouette. The human detection approach has been designed to be robust to different environmental stimuli. Thus, posture is analyzed with simple and efficient features that are not designed to manage constraints related to the environment but only designed to describe human silhouettes. The posture recognition method, based on fuzzy logic, identifies four static postures and is robust to variation in the distance between the camera and the person, and to the person's morphology. With an accuracy of 74.29% of satisfactory posture recognition, this approach can detect emergency situations such as a fall within a health smart home.