Emotion Classification Based on Pulsatile Images Extracted from Short Facial Videos via Deep Learning.

Most human emotion recognition methods largely depend on classifying stereotypical facial expressions that represent emotions. However, such facial expressions do not necessarily correspond to actual emotional states and may correspond to communicative intentions. In other cases, emotions are hidden, cannot be expressed, or may have lower arousal manifested by less pronounced facial expressions, as may occur during passive video viewing. This study improves an emotion classification approach developed in a previous study, which classifies emotions remotely without relying on stereotypical facial expressions or contact-based methods, using short facial video data. In this approach, we desire to remotely sense transdermal cardiovascular spatiotemporal facial patterns associated with different emotional states and analyze this data via machine learning. In this paper, we propose several improvements, which include a better remote heart rate estimation via a preliminary skin segmentation, improvement of the heartbeat peaks and troughs detection process, and obtaining a better emotion classification accuracy by employing an appropriate deep learning classifier using an RGB camera input only with data. We used the dataset obtained in the previous study, which contains facial videos of 110 participants who passively viewed 150 short videos that elicited the following five emotion types: amusement, disgust, fear, sexual arousal, and no emotion, while three cameras with different wavelength sensitivities (visible spectrum, near-infrared, and longwave infrared) recorded them simultaneously. From the short facial videos, we extracted unique high-resolution spatiotemporal, physiologically affected features and examined them as input features with different deep-learning approaches. An EfficientNet-B0 model type was able to classify participants' emotional states with an overall average accuracy of 47.36% using a single input spatiotemporal feature map obtained from a regular RGB camera.

Atmospheric Turbulence Degraded Video Restoration with Recurrent GAN (ATVR-GAN).

Atmospheric turbulence (AT) can change the path and direction of light during video capturing of a target in space due to the random motion of the turbulent medium, a phenomenon that is most noticeable when shooting videos at long ranges, resulting in severe video dynamic distortion and blur. To mitigate geometric distortion and reduce spatially and temporally varying blur, we propose a novel Atmospheric Turbulence Video Restoration Generative Adversarial Network (ATVR-GAN) with a specialized Recurrent Neural Network (RNN) generator, which is trained to predict the scene's turbulent optical flow (OF) field and utilizes a recurrent structure to catch both spatial and temporal dependencies. The new architecture is trained using a newly combined loss function that counts for the spatiotemporal distortions, specifically tailored to the AT problem. Our network was tested on synthetic and real imaging data and compared against leading algorithms in the field of AT mitigation and image restoration. The proposed method outperformed these methods for both synthetic and real data examined.

Turbulence video simulation using the spatiotemporal cross correlation of distortion fields.

Previous simulations of atmospheric turbulence in videos are computationally complex. The purpose of this study is to develop an efficient algorithm for simulating spatiotemporal videos affected by atmospheric turbulence, given a static image. We extend an existing method for the simulation of atmospheric turbulence in a single image by incorporating turbulence properties in the time domain and the blurring effect. We accomplish this through analysis of the correlation between turbulence image distortions in time and in space. The significance of this method is the ease with which it will be possible to produce a simulation, given properties of the turbulence (including turbulence strength, object distance, and height). We apply the simulation to low and high frame rate videos, and we show that the spatiotemporal cross correlation of the distortion fields in the simulated video matches the physical spatiotemporal cross correlation function. Such a simulation can be useful when developing algorithms that apply to videos degraded by atmospheric turbulence and require a large amount of imaging data for training.

3D Object Detection via 2D Segmentation-Based Computational Integral Imaging Applied to a Real Video.

This study aims to achieve accurate three-dimensional (3D) localization of multiple objects in a complicated scene using passive imaging. It is challenging, as it requires accurate localization of the objects in all three dimensions given recorded 2D images. An integral imaging system captures the scene from multiple angles and is able to computationally produce blur-based depth information about the objects in the scene. We propose a method to detect and segment objects in a 3D space using integral-imaging data obtained by a video camera array. Using objects' two-dimensional regions detected via deep learning, we employ local computational integral imaging in detected objects' depth tubes to estimate the depth positions of the objects along the viewing axis. This method analyzes object-based blurring characteristics in the 3D environment efficiently. Our camera array produces an array of multiple-view videos of the scene, called elemental videos. Thus, the proposed 3D object detection applied to the video frames allows for 3D tracking of the objects with knowledge of their depth positions along the video. Results show successful 3D object detection with depth localization in a real-life scene based on passive integral imaging. Such outcomes have not been obtained in previous studies using integral imaging; mainly, the proposed method outperforms them in its ability to detect the depth locations of objects that are in close proximity to each other, regardless of the object size. This study may contribute when robust 3D object localization is desired with passive imaging, but it requires a camera or lens array imaging apparatus.

Fast and Enhanced MMW Imaging System Using a Simple Row Detector Circuit with GDDs as Sensor Elements and an FFT-Based Signal Acquisition System.

The relatively high atmospheric propagation of millimeter-waves (MMW) was found to be one of the most critical reasons for the development of reliable sensors for MMW detection. According to previous research works, it has been already shown that incident MMW radiation on a glow discharge detector (GDD) can increase the discharge current. Hence, the electrical mode of detection can be employed to detect the presence of MMW radiation. In this article, a new design of a row detector using GDDs as pixel elements, and the influence of MMW incidence on GDD's discharge current, were acquired using an elementary data acquisition (DAQ) platform. The DAQ system computes the averaged Fast Fourier Transform (FFT) spectrum of the time signal and returns the FFT results as magnitude based on the level of detection. An FFT-based signal acquisition proved to be a better alternative to the lock-in detection that was commonly used in MMW detection systems. This improved detection circuit provides enhanced noise filtering, thereby resulting in better MMW images within a short time. The overhead expense of the entire system is very low, as it can avoid lock-in amplifier stages that were previously used for signal enhancement. A scanning mechanism using a motorized translation stage (step motor) is involved to place and align the row detector in the image plane. The scanning can be carried out vertically to perform the imaging, by configuring the step motor after selecting the desired step size and position. A simplified version of the MMW detection circuit with a dedicated over-voltage protection facility is presented here. This made the detection system more stable and reliable during its operation. The MMW detection circuit demonstrated in this work was found to be a milestone to develop larger focal plane arrays (FPA) with very inexpensive sensor elements.

A Machine Learning Approach to Design of Aperiodic, Clustered-Dot Halftone Screens via Direct Binary Search.

Aperiodic, clustered-dot, halftone patterns have recently become popular for commercial printing of continuous-tone images with laser, electrophotographic presses, because of their inherent stability and resistance to moiré artifacts. Halftone screens designed using the multistage, multipass, clustered direct binary search (MS-MP-CLU-DBS) algorithm can yield halftone patterns with very high visual quality. But the characteristics of these halftone patterns depend on three input parameters for which there are no known formulas to choose their values to yield halftone patterns of a certain quality level and scale. Using machine learning methods, two predictors are developed that take as input these three parameters. One predicts the quality level of the halftone pattern. The other one predicts the scale of the halftone pattern. To provide ground truth information for training these predictors, human subjects viewed a large number of halftone patches generated from MS-MP-CLU-DBS-designed screens and assigned each patch to one of four quality levels. For each patch, the location of the peak in the radially averaged power spectrum (RAPS) is calculated as a measure of the scale or effective line frequency of the pattern. Experimental results demonstrate the accuracy of the two predictors and the effectiveness of screen design procedures based on these predictors to generate both monochrome and color high quality halftone images.

Classification of emotional states via transdermal cardiovascular spatiotemporal facial patterns using multispectral face videos.

We describe a new method for remote emotional state assessment using multispectral face videos, and present our findings: unique transdermal, cardiovascular and spatiotemporal facial patterns associated with different emotional states. The method does not rely on stereotypical facial expressions but utilizes different wavelength sensitivities (visible spectrum, near-infrared, and long-wave infrared) to gauge correlates of autonomic nervous system activity spatially and temporally distributed across the human face (e.g., blood flow, hemoglobin concentration, and temperature). We conducted an experiment where 110 participants viewed 150 short emotion-eliciting videos and reported their emotional experience, while three cameras recorded facial videos with multiple wavelengths. Spatiotemporal multispectral features from the multispectral videos were used as inputs to a machine learning model that was able to classify participants' emotional state (i.e., amusement, disgust, fear, sexual arousal, or no emotion) with satisfactory results (average ROC AUC score of 0.75), while providing feature importance analysis that allows the examination of facial occurrences per emotional state. We discuss findings concerning the different spatiotemporal patterns associated with different emotional states as well as the different advantages of the current method over existing approaches to emotion detection.

Simulating the perceptual effects of electrode-retina distance in prosthetic vision.

Objective. Retinal prostheses aim to restore some vision in retinitis pigmentosa and age-related macular degeneration blind patients. Many spatial and temporal aspects have been found to affect prosthetic vision. Our objective is to study the impact of the space-variant distance between the stimulating electrodes and the surface of the retina on prosthetic vision and how to mitigate this impact.Approach. A prosthetic vision simulation was built to demonstrate the perceptual effects of the electrode-retina distance (ERD) with different random spatial variations, such as size, brightness, shape, dropout, and spatial shifts. Three approaches for reducing the ERD effects are demonstrated: electrode grouping (quads), ERD-based input-image enhancement, and object scanning with and without phosphene persistence. A quantitative assessment for the first two approaches was done based on experiments with 20 subjects and three vision-based computational image similarity metrics.Main results.The effects of various ERDs on phosphenes' size, brightness, and shape were simulated. Quads, chosen according to the ERDs, effectively elicit phosphenes without exceeding the safe charge density limit, whereas single electrodes with large ERD cannot do so. Input-image enhancement reduced the ERD effects effectively. These two approaches significantly improved ERD-affected prosthetic vision according to the experiment and image similarity metrics. A further reduction of the ERD effects was achieved by scanning an object while moving the head.Significance.ERD has multiple effects on perception with retinal prostheses. One of them is vision loss caused by the incapability of electrodes with large ERD to evoke phosphenes. The three approaches presented in this study can be used separately or together to mitigate the impact of ERD. A consideration of our approaches in reducing the perceptual effects of the ERD may help improve the perception with current prosthetic technology and influence the design of future prostheses.

Retinal prosthetic vision simulation: temporal aspects.

Objective. The perception of individuals fitted with retinal prostheses is not fully understood, although several retinal implants have been tested and commercialized. Realistic simulations of perception with retinal implants would be useful for future development and evaluation of such systems.Approach.We implemented a retinal prosthetic vision simulation, including temporal features, which have not been previously simulated. In particular, the simulation included temporal aspects such as persistence and perceptual fading of phosphenes and the electrode activation rate.Main results.The simulated phosphene persistence showed an effective reduction in flickering at low electrode activation rates. Although persistence has a positive effect on static scenes, it smears dynamic scenes. Perceptual fading following continuous stimulation affects prosthetic vision of both static and dynamic scenes by making them disappear completely or partially. However, we showed that perceptual fading of a static stimulus might be countered by head-scanning motions, which together with the persistence revealed the contours of the faded object. We also showed that changing the image polarity may improve simulated prosthetic vision in the presence of persistence and perceptual fading.Significance.Temporal aspects have important roles in prosthetic vision, as illustrated by the simulations. Considering these aspects may improve the future design, the training with, and evaluation of retinal prostheses.

New Design for Compact Color Screen Sets for High-End Digital Color Press.

Digital halftoning is an essential part of the process for printing color, continuous-tone content. Traditionally, the highest quality has been achieved with analog, offset lithographic presses, using color screen sets that yield periodic, clustereddot halftone patterns. Increasingly, these systems are being supplanted by digital presses that are based on either electrophotographic or inkjet marking processes. Due to the inherent instability of the electrophotographic marking process, periodic, clustered-dot halftone patterns are also widely used with such presses. However, digital presses have much lower resolution than their analog counterparts. Simply mimicking the traditional screen designs used with commercial, offset presses will result in halftone patterns that are more susceptible to moire due to the interaction between the periodic patterns used to render the different color channels. This causes instability in the printed colors. The moire can be reduced by increasing the frequency of the halftone patterns. But this may make the print appear grainier than its analog counterpart. In this paper, we introduce a principled design procedure that allows one to design color screen sets that generate periodic, clustered-dot halftone patterns that improve color stability without increasing graininess. We present experimental results to support the benefits of our new color screen set design framework.

Ultra-wideband and inexpensive glow discharge detector for millimeter-wave wireless communication based on upconversion to visual light.

Data traffic is increasing rapidly, especially on wireless channels, pushing the carrier frequency to the X, K, and millimeter-wave (MMW) bands. This requires development of new technologies and communication components operating at those bands. The detectors and receivers for those bands are very expensive, have high sensitivity to electrostatic discharge, and can be damaged by high incident power. An ultra-wideband and inexpensive glow discharge detector (GDD) is presented here. The GDD was found to be an excellent microwave and MMW radiation detector. The detection mechanism presented here is based on upconversion of microwave and MMW radiation to visual light. The experimental results demonstrate ultra-wideband detection at X and at MMW bands. These results present a proof of concept for the ability of our system to be used as a detector in wireless communication for the 5th generation.

Pulse Oximetry with Two Infrared Wavelengths without Calibration in Extracted Arterial Blood.

Oxygen saturation in arterial blood (SaO2) provides information about the performance of the respiratory system. Non-invasive measurement of SaO2 by commercial pulse oximeters (SpO2) make use of photoplethysmographic pulses in the red and infrared regions and utilizes the different spectra of light absorption by oxygenated and de-oxygenated hemoglobin. Because light scattering and optical path-lengths differ between the two wavelengths, commercial pulse oximeters require empirical calibration which is based on SaO2 measurement in extracted arterial blood. They are still prone to error, because the path-lengths difference between the two wavelengths varies among different subjects. We have developed modified pulse oximetry, which makes use of two nearby infrared wavelengths that have relatively similar scattering constants and path-lengths and does not require an invasive calibration step. In measurements performed on adults during breath holding, the two-infrared pulse oximeter and a commercial pulse oximeter showed similar changes in SpO2. The two pulse oximeters showed similar accuracy when compared to SaO2 measurement in extracted arterial blood (the gold standard) performed in intensive care units on newborns and children with an arterial line. Errors in SpO2 because of variability in path-lengths difference between the two wavelengths are expected to be smaller in the two-infrared pulse oximeter.

Effects of elemental images' quantity on three-dimensional segmentation using computational integral imaging: publisher's note.

This publisher's note corrects the Funding section in Appl. Opt.56, 2132 (2017)APOPAI0003-693510.1364/AO.56.002132.

Optimized depth of field methodology using annular liquid crystal modulator assisted by image processing.

An optical-digital tunable depth of field (DOF) methodology is presented. The suggested methodology forms a fused image based on the sharpest similar depth regions from a set of source images taken with different phase masks. Each phase mask contains a different degree of DOF extension and is implemented by using an annular liquid crystal spatial light modulator, which consists of 16-ring electrodes positioned in the pupil plane. A detailed description of the optical setup and characterization of selected pupil phase masks as well as optimization of the binary phase mask for maximal DOF extension is presented. Experimental results are investigated both qualitatively and quantitatively. In addition, the algorithm's results are compared with those of some well-known fusion algorithms and proved its supremacy.

Effects of elemental images' quantity on three-dimensional segmentation using computational integral imaging.

3D object detection and isolation can be achieved algorithmically using computational integral-imaging data. The 3D scene is acquired by a multi-channel system, where each channel (elemental image) captures the scene from a shifted perspective angle. The number of these channels affects the weight, the cost, and the computational load of the segmentation process, while a lower number of channels may reduce the performance of the objects' separation in the 3D scene. This research examines the effect of the elemental images' quantity on the 3D object detection and segmentation, under both regular and noisy conditions. Moreover, based on our previous works, we perform an improvement of the 3D object segmentation quality using an adapted active-contour method.

Automatic 3D object localization and isolation using computational integral imaging.

Detecting objects in three-dimensional (3D) space may be useful for various applications. We present a method to detect the 3D locations of objects using computationally reconstructed images obtained by integral imaging. The new algorithm exploits the space-variant blurring properties of the reconstructed images to detect and isolate objects at their depth locations, while removing traces of objects from other depths. With regard to previous work, the proposed method is more efficient and more resistant to noise; it gives more information about the detected object's depth, and improves object isolation and presentation.

Active confocal imaging for visual prostheses.

There are encouraging advances in prosthetic vision for the blind, including retinal and cortical implants, and other "sensory substitution devices" that use tactile or electrical stimulation. However, they all have low resolution, limited visual field, and can display only few gray levels (limited dynamic range), severely restricting their utility. To overcome these limitations, image processing or the imaging system could emphasize objects of interest and suppress the background clutter. We propose an active confocal imaging system based on light-field technology that will enable a blind user of any visual prosthesis to efficiently scan, focus on, and "see" only an object of interest while suppressing interference from background clutter. The system captures three-dimensional scene information using a light-field sensor and displays only an in-focused plane with objects in it. After capturing a confocal image, a de-cluttering process removes the clutter based on blur difference. In preliminary experiments we verified the positive impact of confocal-based background clutter removal on recognition of objects in low resolution and limited dynamic range simulated phosphene images. Using a custom-made multiple-camera system based on light-field imaging, we confirmed that the concept of a confocal de-cluttered image can be realized effectively.

Spatial-phase-shift imaging interferometry using a spectrally modulated white light source.

An extension of the white light spatial-phase-shift (WLSPS) for object surface measurements is described. Using WLSPS, surface measurements can be obtained from any real object image without the need of a reference beam, thus achieving inherent vibration cancellation. The surface topography is obtained by acquiring multiple images of an object illuminated by a spectrally modulated white light source and using an appropriate algorithm. The modulation of the light source obviates the need for the continuous phase delay to obtain the interferograms.