3D object tracking using integral imaging with mutual information and Bayesian optimization.

Integral imaging has proven useful for three-dimensional (3D) object visualization in adverse environmental conditions such as partial occlusion and low light. This paper considers the problem of 3D object tracking. Two-dimensional (2D) object tracking within a scene is an active research area. Several recent algorithms use object detection methods to obtain 2D bounding boxes around objects of interest in each frame. Then, one bounding box can be selected out of many for each object of interest using motion prediction algorithms. Many of these algorithms rely on images obtained using traditional 2D imaging systems. A growing literature demonstrates the advantage of using 3D integral imaging instead of traditional 2D imaging for object detection and visualization in adverse environmental conditions. Integral imaging's depth sectioning ability has also proven beneficial for object detection and visualization. Integral imaging captures an object's depth in addition to its 2D spatial position in each frame. A recent study uses integral imaging for the 3D reconstruction of the scene for object classification and utilizes the mutual information between the object's bounding box in this 3D reconstructed scene and the 2D central perspective to achieve passive depth estimation. We build over this method by using Bayesian optimization to track the object's depth in as few 3D reconstructions as possible. We study the performance of our approach on laboratory scenes with occluded objects moving in 3D and show that the proposed approach outperforms 2D object tracking. In our experimental setup, mutual information-based depth estimation with Bayesian optimization achieves depth tracking with as few as two 3D reconstructions per frame which corresponds to the theoretical minimum number of 3D reconstructions required for depth estimation. To the best of our knowledge, this is the first report on 3D object tracking using the proposed approach.

Underwater object detection and temporal signal detection in turbid water using 3D-integral imaging and deep learning.

Underwater scattering caused by suspended particles in the water severely degrades signal detection performance and poses significant challenges to the problem of object detection. This paper introduces an integrated dual-function deep learning-based underwater object detection and classification and temporal signal detection algorithm using three-dimensional (3D) integral imaging (InIm) under degraded conditions. The proposed system is an efficient object classification and temporal signal detection system for degraded environments such as turbidity and partial occlusion and also provides the object range in the scene. A camera array captures the underwater objects in the scene and the temporally encoded binary signals transmitted for the purpose of communication. The network is trained using a clear underwater scene without occlusion, whereas test data is collected in turbid water with partial occlusion. Reconstructed 3D data is the input to a You Look Only Once (YOLOv4) neural network for object detection and a convolutional neural network-based bidirectional long short-term memory network (CNN-BiLSTM) is used for temporal optical signal detection. Finally, the transmitted signal is decoded. In our experiments, 3D InIm provides better image reconstruction in a degraded environment over 2D sensing-based methods. Also, reconstructed 3D images segment out the object of interest from occlusions and background which improves the detection accuracy of the network with 3D InIm. To the best of our knowledge, this is the first report that combines deep learning with 3D InIm for simultaneous and integrated underwater object detection and optical signal detection in degraded environments.

Generalization of the two-point-source resolution criterion in the presence of noise.

The two-point-source resolution criterion is widely used to quantify the performance of imaging systems. The two main approaches for the computation of the two-point-source resolution are the detection theoretic and visual analyses. The first assumes a shift-invariant system and lacks the ability to incorporate two different point spread functions (PSFs), which may be required in certain situations like computing axial resolution. The latter approach, which includes the Rayleigh criterion, relies on the peak-to-valley ratio and does not properly account for the presence of noise. We present a heuristic generalization of the visual two-point-source resolution criterion using Gaussian processes (GP). This heuristic criterion is applicable to both shift-invariant and shift-variant imaging modalities. This criterion can also incorporate different definitions of resolution expressed in terms of varying peak-to-valley ratios. Our approach implicitly incorporates information about noise statistics such as the variance or signal-to-noise ratio by making assumptions about the spatial correlation of PSFs in the form of kernel functions. Also, it does not rely on an analytic form of the PSF.

3D object detection through fog and occlusion: passive integral imaging vs active (LiDAR) sensing.

In this paper, we address the problem of object recognition in degraded environments including fog and partial occlusion. Both long wave infrared (LWIR) imaging systems and LiDAR (time of flight) imaging systems using Azure Kinect, which combine conventional visible and lidar sensing information, have been previously demonstrated for object recognition in ideal conditions. However, the object detection performance of Azure Kinect depth imaging systems may decrease significantly in adverse weather conditions such as fog, rain, and snow. The concentration of fog degrades the depth images of Azure Kinect camera, and the overall visibility of RGBD images (fused RGB and depth image), which can make object recognition tasks challenging. LWIR imaging may avoid these issues of lidar-based imaging systems. However, due to poor spatial resolution of LWIR cameras, thermal imaging provides limited textural information within a scene and hence may fail to provide adequate discriminatory information to identify between objects of similar texture, shape and size. To improve the object detection task in fog and occlusion, we use three-dimensional (3D) integral imaging (InIm) system with a visible range camera. 3D InIm provides depth information, mitigates the occlusion and fog in front of the object, and improves the object recognition capabilities. For object recognition, the YOLOv3 neural network is used for each of the tested imaging systems. Since the concentration of fog affects the images from different sensors (visible, LWIR, and Azure Kinect depth cameras) in different ways, we compared the performance of the network on these images in terms of average precision and average miss rate. For the experiments we conducted, the results indicate that in degraded environment 3D InIm using visible range cameras can provide better image reconstruction as compared to the LWIR camera and Azure Kinect RGBD camera, and therefore it may improve the detection accuracy of the network. To the best of our knowledge, this is the first report comparing the performance of object detection between passive integral imaging system vs active (LiDAR) sensing in degraded environments such as fog and partial occlusion.

Lowlight object recognition by deep learning with passive three-dimensional integral imaging in visible and long wave infrared wavelengths.

Traditionally, long wave infrared imaging has been used in photon starved conditions for object detection and classification. We investigate passive three-dimensional (3D) integral imaging (InIm) in visible spectrum for object classification using deep neural networks in photon-starved conditions and under partial occlusion. We compare the proposed passive 3D InIm operating in the visible domain with that of the long wave infrared sensing in both 2D and 3D imaging cases for object classification in degraded conditions. This comparison is based on average precision, recall, and miss rates. Our experimental results demonstrate that cold and hot object classification using 3D InIm in the visible spectrum may outperform both 2D and 3D imaging implemented in long wave infrared spectrum for photon-starved and partially occluded scenes. While these experiments are not comprehensive, they demonstrate the potential of 3D InIm in the visible spectrum for low light applications. Imaging in the visible spectrum provides higher spatial resolution, more compact optics, and lower cost hardware compared with long wave infrared imaging. In addition, higher spatial resolution obtained in the visible spectrum can improve object classification accuracy. Our experimental results provide a proof of concept for implementing visible spectrum imaging in place of the traditional LWIR spectrum imaging for certain object recognition tasks.

Three-dimensional polarimetric image restoration in low light with deep residual learning and integral imaging.

Polarimetric imaging can become challenging in degraded environments such as low light illumination conditions or in partial occlusions. In this paper, we propose the denoising convolutional neural network (DnCNN) model with three-dimensional (3D) integral imaging to enhance the reconstructed image quality of polarimetric imaging in degraded environments such as low light and partial occlusions. The DnCNN is trained based on the physical model of the image capture in degraded environments to enhance the visualization of polarimetric imaging where simulated low light polarimetric images are used in the training process. The DnCNN model is experimentally tested on real polarimetric images captured in real low light environments and in partial occlusion. The performance of DnCNN model is compared with that of total variation denoising. Experimental results demonstrate that DnCNN performs better than total variation denoising for polarimetric integral imaging in terms of signal-to-noise ratio and structural similarity index measure in low light environments as well as low light environments under partial occlusions. To the best of our knowledge, this is the first report of polarimetric 3D object visualization and restoration in low light environments and occlusions using DnCNN with integral imaging. The proposed approach is also useful for 3D image restoration in conventional (non-polarimetric) integral imaging in a degraded environment.

Deep learning polarimetric three-dimensional integral imaging object recognition in adverse environmental conditions.

Polarimetric imaging is useful for object recognition and material classification because of its ability to discriminate objects based on polarimetric signatures of materials. Polarimetric imaging of an object captures important physical properties such as shape and surface properties and can be effective even in low light environments. Integral imaging is a passive three-dimensional (3D) imaging approach that takes advantage of multiple 2D imaging perspectives to perform 3D reconstruction. In this paper, we propose a unified polarimetric detection and classification of objects in degraded environments such as low light and the presence of occlusion. This task is accomplished using a deep learning model for 3D polarimetric integral imaging data captured in the visible spectral domain. The neural network system is designed and trained for 3D object detection and classification using polarimetric integral images. We compare the detection and classification results between polarimetric and non-polarimetric 2D and 3D imaging. The system performance in degraded environmental conditions is evaluated using average miss rate, average precision, and F-1 score. The results indicate that for the experiments we have performed, polarimetric 3D integral imaging outperforms 2D polarimetric imaging as well as non-polarimetric 2D and 3D imaging for object recognition in adverse conditions such as low light and occlusions. To the best of our knowledge, this is the first report for polarimetric 3D object recognition in low light environments and occlusions using a deep learning-based integral imaging. The proposed approach is attractive because low light polarimetric object recognition in the visible spectral band benefits from much higher spatial resolution, more compact optics, and lower system cost compared with long wave infrared imaging which is the conventional imaging approach for low light environments.

Three-dimensional polarimetric integral imaging in photon-starved conditions: performance comparison between visible and long wave infrared imaging.

Three-dimensional (3D) polarimetric integral imaging (InIm) to extract the 3D polarimetric information of objects in photon-starved conditions is investigated using a low noise visible range camera and a long wave infrared (LWIR) range camera, and the performance between the two sensors is compared. Stokes polarization parameters and degree of polarization (DoP) are calculated to extract the polarimetric information of the 3D scene while integral imaging reconstruction provides depth information and improves the performance of low-light imaging tasks. An LWIR wire grid polarizer and a linear polarizer film are used as polarimetric objects for the LWIR range and visible range cameras, respectively. To account for a limited number of photons per pixel using the visible range camera in low light conditions, we apply a mathematical restoration model at each elemental image of visible camera to enhance the signal. We show that the low noise visible range camera may outperform the LWIR camera in detection of polarimetric objects under low illumination conditions. Our experiments indicate that for 3D polarimetric measurements under photon-starved conditions, visible range sensing may produce a signal-to-noise ratio (SNR) that is not lower than the LWIR range sensing. We derive the probability density function (PDF) of the 2D and 3D degree of polarization (DoP) images and show that the theoretical model demonstrates agreement to that of the experimentally obtained results. To the best of our knowledge, this is the first report comparing the polarimetric imaging performance between visible range and infrared (IR) range sensors under photon-starved conditions and the relevant statistical models of 3D polarimetric integral imaging.

Relationship between the source size at the diffuser plane and the longitudinal spatial coherence function of the optical coherence microscopy system.

Coherence properties of light sources are indispensable for optical coherence microscopy/tomography as they greatly influence the signal-to-noise ratio, axial resolution, and penetration depth of the system. In the present paper, we report the investigation of longitudinal spatial coherence properties of a pseudothermal light source (PTS) as a function of the laser spot size at the rotating diffuser plate. The laser spot size is varied by translating a microscope objective lens toward or away from the diffuser plate. The longitudinal spatial coherence length, which governs the axial resolution of the coherence microscope, is found to be minimum for the beam spot size of 3.5 mm at the diffuser plate. The axial resolution of the system is found to be equal to an $\sim{13}\,\,{\rm \unicode{x00B5}{\rm m}}$∼13µm at 3.5 mm beam spot size. The change in the axial resolution of the system is confirmed by performing the experiments on standard gauge blocks of a height difference of 15 µm by varying the spot size at the diffuser plate. Thus, by appropriately choosing the beam spot size at the diffuser plane, any monochromatic laser light source can be utilized to obtain high axial resolution irrespective of the source's temporal coherence length. It can provide speckle-free tomographic images of multilayered biological specimens with large penetration depth. In addition, a PTS avoids the use of any chromatic-aberration-corrected optics and dispersion-compensation mechanism unlike conventional setups.