Multi-channel residual network model for accurate estimation of spatially-varying and depth-dependent defocus kernels.

Digital projectors have been increasingly utilized in various commercial and scientific applications. However, they are prone to the out-of-focus blurring problem since their depth-of-fields are typically limited. In this paper, we explore the feasibility of utilizing a deep learning-based approach to analyze the spatially-varying and depth-dependent defocus properties of digital projectors. A multimodal displaying/imaging system is built for capturing images projected at various depths. Based on the constructed dataset containing well-aligned in-focus, out-of-focus, and depth images, we propose a novel multi-channel residual deep network model to learn the end-to-end mapping function between the in-focus and out-of-focus image patches captured at different spatial locations and depths. To the best of our knowledge, it is the first research work revealing that the complex spatially-varying and depth-dependent blurring effects can be accurately learned from a number of real-captured image pairs instead of being hand-crafted as before. Experimental results demonstrate that our proposed deep learning-based method significantly outperforms the state-of-the-art defocus kernel estimation techniques and thus leads to better out-of-focus compensation for extending the dynamic ranges of digital projectors.

On-the-fly extrinsic calibration of multimodal sensing system for fast 3D thermographic scanning.

The fusion of three-dimensional (3D) geometrical and two-dimensional (2D) thermal information provides a promising method for characterizing temperature distribution of 3D objects, extending infrared imaging from 2D to 3D to support various thermal inspection applications. In this paper, we present an effective on-the-fly calibration approach for accurate alignment of depth and thermal data to facilitate dynamic and fast-speed 3D thermal scanning tasks. For each pair of depth and thermal frames, we estimate their relative pose by minimizing the objective function that measures the temperature consistency between a 2D infrared image and the reference 3D thermographic model. Our proposed frame-to-model mapping scheme can be seamlessly integrated into a generic 3D thermographic reconstruction framework. Through graphics-processing-unit-based acceleration, our method requires less than 10 ms to generate a pair of well-aligned depth and thermal images without hardware synchronization and improves the robustness of the system against significant camera motion.

Exploiting fusion architectures for multispectral pedestrian detection and segmentation.

Recent research has demonstrated that the fusion of complementary information captured by multi-modal sensors (visible and infrared cameras) enables robust pedestrian detection under various surveillance situations (e.g., daytime and nighttime). In this paper, we investigate a number of fusion architectures in an attempt to identify the optimal way of incorporating multispectral information for joint semantic segmentation and pedestrian detection. We made two important findings: (1) the sum fusion strategy, which computes the sum of two feature maps at the same spatial locations, delivers the best performance of multispectral detection, while the most commonly used concatenation fusion surprisingly performs the worst; and (2) two-stream semantic segmentation without multispectral fusion is the most effective scheme to infuse semantic information as supervision for learning human-related features. Based on these studies, we present a unified multispectral fusion framework for joint training of semantic segmentation and target detection that outperforms state-of-the-art multispectral pedestrian detectors by a large margin on the KAIST benchmark dataset.

Single-image-based nonuniformity correction of uncooled long-wave infrared detectors: a deep-learning approach.

Fixed-pattern noise (FPN), which is caused by the nonuniform opto-electronic responses of microbolometer focal-plane-array (FPA) optoelectronics, imposes a challenging problem in infrared imaging systems. In this paper, we successfully demonstrate that a better single-image-based non-uniformity correction (NUC) operator can be directly learned from a large number of simulated training images instead of being handcrafted as before. Our proposed training scheme, which is based on convolutional neural networks (CNNs) and a column FPN simulation module, gives rise to a powerful technique to reconstruct the noise-free infrared image from its corresponding noisy observation. Specifically, a comprehensive column FPN model is utilized to depict the nonlinear characteristics of column amplifiers in the readout circuit of FPA. A large number of high-fidelity training images are simulated based on this model and the end-to-end residual deep network is capable of learning the intrinsic difference between undesirable FPN and original image details. Therefore, column FPN can be accurately estimated and further subtracted from the raw infrared images to obtain NUC results. Comparative results with state-of-the-art single-image-based NUC methods, using real-captured noisy infrared images, demonstrate that our proposed deep-learning-based approach delivers better performances of FPN removal, detail preservation, and artifact suppression.

Depth and thermal sensor fusion to enhance 3D thermographic reconstruction.

Three-dimensional geometrical models with incorporated surface temperature data provide important information for various applications such as medical imaging, energy auditing, and intelligent robots. In this paper we present a robust method for mobile and real-time 3D thermographic reconstruction through depth and thermal sensor fusion. A multimodal imaging device consisting of a thermal camera and a RGB-D sensor is calibrated geometrically and used for data capturing. Based on the underlying principle that temperature information remains robust against illumination and viewpoint changes, we present a Thermal-guided Iterative Closest Point (T-ICP) methodology to facilitate reliable 3D thermal scanning applications. The pose of sensing device is initially estimated using correspondences found through maximizing the thermal consistency between consecutive infrared images. The coarse pose estimate is further refined by finding the motion parameters that minimize a combined geometric and thermographic loss function. Experimental results demonstrate that complimentary information captured by multimodal sensors can be utilized to improve performance of 3D thermographic reconstruction. Through effective fusion of thermal and depth data, the proposed approach generates more accurate 3D thermal models using significantly less scanning data.

Single-image-based solution for optics temperature-dependent nonuniformity correction in an uncooled long-wave infrared camera.

In this Letter, we propose an efficient and accurate solution to remove temperature-dependent nonuniformity effects introduced by the imaging optics. This single-image-based approach computes optics-related fixed pattern noise (FPN) by fitting the derivatives of correction model to the gradient components, locally computed on an infrared image. A modified bilateral filtering algorithm is applied to local pixel output variations, so that the refined gradients are most likely caused by the nonuniformity associated with optics. The estimated bias field is subtracted from the raw infrared imagery to compensate the intensity variations caused by optics. The proposed method is fundamentally different from the existing nonuniformity correction (NUC) techniques developed for focal plane arrays (FPAs) and provides an essential image processing functionality to achieve completely shutterless NUC for uncooled long-wave infrared (LWIR) imaging systems.

Shutterless solution for simultaneous focal plane array temperature estimation and nonuniformity correction in uncooled long-wave infrared camera.

In uncooled long-wave infrared (LWIR) microbolometer imaging systems, temperature fluctuations of the focal plane array (FPA) result in thermal drift and spatial nonuniformity. In this paper, we present a novel approach based on single-image processing to simultaneously estimate temperature variances of FPAs and compensate the resulting temperature-dependent nonuniformity. Through well-controlled thermal calibrations, empirical behavioral models are derived to characterize the relationship between the responses of microbolometer and FPA temperature variations. Then, under the assumption that strong dependency exists between spatially adjacent pixels, we estimate the optimal FPA temperature so as to minimize the global intensity variance across the entire thermal infrared image. We make use of the estimated FPA temperature to infer an appropriate nonuniformity correction (NUC) profile. The performance and robustness of the proposed temperature-adaptive NUC method are evaluated on realistic IR images obtained by a 640 × 512 pixels uncooled LWIR microbolometer imaging system operating in a significantly changed temperature environment.

Low-cost miniature wide-angle imaging for self-motion estimation.

This paper examines the performance of a low-cost, miniature, wide field-of-view (FOV) visual sensor that includes advanced pinhole optics and most recent CMOS imager technology. The pinhole camera may often be disregarded because of its apparent simplicity, low aperture and image finesse. However, its angular field can be dramatically improved using only a few off-the-shelf micro-optical elements. With modern high-sensitivity silicon-based digital retina, we show that it could be a practical device for developing self-motion estimation sensor in mobile applications, such as stabilization of a robotic micro flyer.