Deep Dynamic Scene Deblurring for Unconstrained Dual-Lens Cameras.

Dual-lens (DL) cameras capture depth information, and hence enable several important vision applications. Most present-day DL cameras employ unconstrained settings in the two views in order to support extended functionalities. But a natural hindrance to their working is the ubiquitous motion blur encountered due to camera motion, object motion, or both. However, there exists not a single work for the prospective unconstrained DL cameras that addresses this problem (so called dynamic scene deblurring). Due to the unconstrained settings, degradations in the two views need not be the same, and consequently, naive deblurring approaches produce inconsistent left-right views and disrupt scene-consistent disparities. In this paper, we address this problem using Deep Learning and make three important contributions. First, we address the root cause of view-inconsistency in standard deblurring architectures using a Coherent Fusion Module. Second, we address an inherent problem in unconstrained DL deblurring that disrupts scene-consistent disparities by introducing a memory-efficient Adaptive Scale-space Approach. This signal processing formulation allows accommodation of different image-scales in the same network without increasing the number of parameters. Finally, we propose a module to address the Space-variant and Image-dependent nature of dynamic scene blur. We experimentally show that our proposed techniques have substantial practical merit.

Deep end-to-end rolling shutter rectification.

CMOS sensors employ a row-wise acquisition mechanism while imaging a scene, which can result in undesired motion artifacts known as rolling shutter (RS) distortions in the captured image. Existing single image RS rectification methods attempt to account for these distortions by using either algorithms tailored for a specific class of scenes that warrants information of intrinsic camera parameters or a learning-based framework with known ground truth motion parameters. In this paper, we propose an end-to-end deep neural network for the challenging task of single image RS rectification. Our network consists of a motion block, a trajectory module, a row block, an RS rectification module, and an RS regeneration module (which is used only during training). The motion block predicts the camera pose for every row of the input RS distorted image, while the trajectory module fits estimated motion parameters to a third-order polynomial. The row block predicts the camera motion that must be associated with every pixel in the target, i.e., RS rectified image. Finally, the RS rectification module uses motion trajectory and the output of a row block to warp the input RS image to arrive at a distortion-free image. For faster convergence during training, we additionally use an RS regeneration module that compares the input RS image with the ground truth image distorted by estimated motion parameters. The end-to-end formulation in our model does not constrain the estimated motion to ground truth motion parameters, thereby successfully rectifying the RS images with complex real-life camera motion. Experiments on synthetic and real datasets reveal that our network outperforms prior art both qualitatively and quantitatively.

Local Proximity for Enhanced Visibility in Haze.

Atmospheric medium often constrains the visibility of outdoor scenes due to scattering of light rays. This causes attenuation in the irradiance reaching the imaging device along with an additive component to render a hazy effect in the image. The visibility is further reduced for poorly illuminated scenes. The attenuation becomes wavelength dependent in underwater scenario, causing undesired color cast along with hazy effect. In order to suppress the effect of different atmospheric/underwater conditions such as haze and to enhance the contrast of such images, we reformulate local haziness in a generalized manner. The parameters are estimated by harnessing the similarity of patches within a local neighborhood. Unlike existing methods, our approach is developed based on the assumption that for outdoor scenes the depth of patches changes gradually in a local neighborhood surrounding the patch. This change in depth can be approximated by patch similarity in that neighborhood. As the attenuation in irradiance of an image in presence of atmospheric medium relies on the depth of the scene, the coefficients related to the attenuation are estimated from the weights of patch similarity. The additive haze effect is deduced using non-local mean of the patch. Our experimental results demonstrate the effectiveness of our approach in reducing the haze component as well as in enhancing the image under different conditions of haze (daytime, nighttime, and underwater).

Multilevel weighted enhancement for underwater image dehazing.

Attenuation and scattering of light are responsible for haziness in images of underwater scenes. To reduce this effect, we propose an approach for single-image dehazing by multilevel weighted enhancement of the image. The underlying principle is that enhancement at different levels of detail can undo the degradation caused by underwater haze. The depth information is captured implicitly while going through different levels of details due to the depth-variant nature of haze. Hence, we judiciously assign weights to different levels of image details and reveal that their linear combination along with the coarsest information can successfully restore the image. Results demonstrate the efficacy of our approach as compared to state-of-the-art underwater dehazing methods.

Change detection in underwater imagery.

In this work, we deal with the problem of change detection in an underwater scenario given an unblurred-blurred image pair of a planar scene taken at different times. The blur is primarily due to the dynamic nature of the water surface and its nature is space-invariant in the presence of cyclic water flows. Exploiting the sparsity of the induced blur as well as the occlusions, we propose a distort-difference pipeline that employs an alternating minimization framework to perform change detection in the presence of geometric distortions (skew) as well as photometric degradations (blur and global illumination variations). The method can effectively yield both sharp and blurred occluder maps. Using synthetic as well as real data, we demonstrate how the proposed technique advances the state of the art.

Unified multiframe super-resolution of matte, foreground, and background.

Reconstruction of a super-resolved image from multiple frames and extraction of matte are two popular topics that have been solved independently. In this paper, we advocate a unified framework that assimilates matting within the super-resolution model. We show that joint estimation is advantageous, as super-resolved edge information helps in obtaining a sharp matte, while the matte in turn aids in resolving fine details. We propose a multiframe approach to increase the spatial resolution of the matte, foreground, and background. This is validated extensively on examples from standard matting datasets.

Shape-from-focus by tensor voting.

In this correspondence, we address the task of recovering shape-from-focus (SFF) as a perceptual organization problem in 3-D. Using tensor voting, depth hypotheses from different focus operators are validated based on their likelihood to be part of a coherent 3-D surface, thereby exploiting scene geometry and focus information to generate reliable depth estimates. The proposed method is fast and yields significantly better results compared with existing SFF methods.

Depth from motion and optical blur with an unscented Kalman filter.

Space-variantly blurred images of a scene contain valuable depth information. In this paper, our objective is to recover the 3-D structure of a scene from motion blur/optical defocus. In the proposed approach, the difference of blur between two observations is used as a cue for recovering depth, within a recursive state estimation framework. For motion blur, we use an unblurred-blurred image pair. Since the relationship between the observation and the scale factor of the point spread function associated with the depth at a point is nonlinear, we propose and develop a formulation of unscented Kalman filter for depth estimation. There are no restrictions on the shape of the blur kernel. Furthermore, within the same formulation, we address a special and challenging scenario of depth from defocus with translational jitter. The effectiveness of our approach is evaluated on synthetic as well as real data, and its performance is also compared with contemporary techniques.

Natural matting for degraded pictures.

A new approach for image matting is proposed based on the Kalman filter, to extract the matte and original foreground, despite the presence of noise in the observed image. Different filter formulations with a discontinuity-adaptive Markov random field prior are proposed for handling additive white Gaussian noise and film-grain noise.

Dealing with parallax in shape-from-focus.

We propose a new method that extends the capability of shape-from-focus (SFF) to estimate the depth profile of 3-D objects in the presence of structure-dependent pixel motion. Existing SFF techniques work under the constraint that there is no parallax in the captured stack of frames. However, in off-the-shelf cameras, there can be appreciable pixel motion among the observations when there is relative motion between the object and the camera. In such a scenario, the depth estimates will be erroneous if the parallax effect is not factored in. Our degradation model accounts for pixel migration effects in the observations due to parallax resulting in a generalization of the SFF technique. We show that pixel motion and defocus blur therein are tightly coupled to the underlying shape of the 3-D object. Simultaneous reconstruction of the underlying 3-D structure and the all-in-focus image is carried out within an optimization framework using local image operations. The proposed method when tested on many examples, both synthetic and real, is very effective and delivers state-of-the-art performance.

Recursive framework for joint inpainting and de-noising of photographic films.

We address the problem of inpainting noisy photographs. We present a recursive image recovery scheme based on the unscented Kalman filter (UKF) to simultaneously inpaint identified damaged portions in an image and suppress film-grain noise. Inpainting of the missing observations is guided by a mask-dependent reconstruction of the image edges. Prediction within the UKF is based on a discontinuity-adaptive Markov random field prior that attempts to preserve edges while achieving noise reduction in uniform regions. We demonstrate the capability of the proposed method with many examples.

Joint image and depth completion in shape-from-focus: taking a cue from parallax.

Shape-from-focus (SFF) uses a sequence of space-variantly defocused observations captured with relative motion between camera and scene. It assumes that there is no motion parallax in the frames. This is a restriction and constrains the working environment. Moreover, SFF cannot recover the structure information when there are missing data in the frames due to CCD sensor damage or unavoidable occlusions. The capability of filling-in plausible information in regions devoid of data is of critical importance in many applications. Images of 3D scenes captured by off-the-shelf cameras with relative motion commonly exhibit parallax-induced pixel motion. We demonstrate the interesting possibility of exploiting motion parallax cue in the images captured in SFF with a practical camera to jointly inpaint the focused image and depth map.

Resolution enhancement in multi-image stereo.

Under stereo settings, the twin problems of image superresolution (SR) and high-resolution (HR) depth estimation are intertwined. The subpixel registration information required for image superresolution is tightly coupled to the 3D structure. The effects of parallax and pixel averaging (inherent in the downsampling process) preclude a priori estimation of pixel motion for superresolution. These factors also compound the correspondence problem at low resolution (LR), which in turn affects the quality of the LR depth estimates. In this paper, we propose an integrated approach to estimate the HR depth and the SR image from multiple LR stereo observations. Our results demonstrate the efficacy of the proposed method in not only being able to bring out image details but also in enhancing the HR depth over its LR counterpart.

Image matching with higher-order geometric features.

We propose a geometric matching technique in which line segments and elliptical arcs are used as edge features. The use of these higher-order features renders feature representation efficient. We derive distance measures to evaluate the similarity between the features of the model and those of the image. The model transformation parameters are found by searching a 3-D transformation space using cell-decomposition. The performance of the proposed method is quite good when tested on a variety of images.

A recursive filter for despeckling SAR images.

This correspondence proposes a recursive algorithm for noise reduction in synthetic aperture radar imagery. Excellent despeckling in conjunction with feature preservation is achieved by incorporating a discontinuity-adaptive Markov random field prior within the unscented Kalman filter framework through importance sampling. The performance of this method is demonstrated on both synthetic and real examples.

Extension of the shape from focus method for reconstruction of high-resolution images.

Shape from focus (SFF) estimates the depth profile of a 3D object using a sequence of observations. Due to the finite depth of field of real aperture cameras and the 3D nature of the object, none of the observations is completely in focus. However, in many applications, it is important to examine finer image details of the underlying object in conjunction with the depth map. We propose an extension to the traditional SFF method to optimally estimate a high-resolution image of the 3D object, given the low-resolution observations and the depth map derived from traditional SFF. Using the observation stack, we show that it is possible to achieve significant improvement in resolution. We also analyze the special case of region of interest superresolution and show analytically that an optional interframe separation exists for which the quality of the estimated high-resolution image is the best.

Improving shape from focus using defocus cue.

The shape-from-focus (SFF) method uses a sequence of frames to estimate the structure of a 3-D object. Its accuracy depends on the step size by which the translational table is moved while capturing the images. Existing SFF algorithms use an ad hoc interpolation strategy to account for the error due to the finite step size. We propose an improved SFF method that uses relative defocus blur derived from actual image data to arrive at the final estimates of the structure of the object. A space-variant image restoration scheme is also proposed to obtain a focused image of the 3-D object. The reconstructed 3-D structure as well as the quality of the restored image are superior for the proposed method in comparison to traditional SFF.

Background learning for robust face recognition with PCA in the presence of clutter.

We propose a new method within the framework of principal component analysis (PCA) to robustly recognize faces in the presence of clutter. The traditional eigenface recognition (EFR) method, which is based on PCA, works quite well when the input test patterns are faces. However, when confronted with the more general task of recognizing faces appearing against a background, the performance of the EFR method can be quite poor. It may miss faces completely or may wrongly associate many of the background image patterns to faces in the training set. In order to improve performance in the presence of background, we argue in favor of learning the distribution of background patterns and show how this can be done for a given test image. An eigenbackground space is constructed corresponding to the given test image and this space in conjunction with the eigenface space is used to impart robustness. A suitable classifier is derived to distinguish nonface patterns from faces. When tested on images depicting face recognition in real situations against cluttered background, the performance of the proposed method is quite good with fewer false alarms.

Image recovery under nonlinear and non-Gaussian degradations.

A new two-dimensional recursive filter for recovering degraded images is proposed that is based on particle-filter theory. The main contribution of this work lies in evolving a framework that has the potential to recover images suffering from a general class of degradations such as system nonlinearity and non-Gaussian observation noise. Samples of the prior probability distribution of the original image are obtained by propagating the samples through an appropriate state model. Given the measurement model and the degraded image, the weights of the samples are computed. The samples and their corresponding weights are used to calculate the conditional mean that yields an estimate of the original image. The proposed method is validated by demonstrating its effectiveness in recovering images degraded by film-grain noise. Synthetic as well as real examples are considered for this purpose. Performance is also compared with that of an existing scheme.

Depth estimation and image restoration using defocused stereo pairs.

We propose a method for estimating depth from images captured with a real aperture camera by fusing defocus and stereo cues. The idea is to use stereo-based constraints in conjunction with defocusing to obtain improved estimates of depth over those of stereo or defocus alone. The depth map as well as the original image of the scene are modeled as Markov random fields with a smoothness prior, and their estimates are obtained by minimizing a suitable energy function using simulated annealing. The main advantage of the proposed method, despite being computationally less efficient than the standard stereo or DFD method, is simultaneous recovery of depth as well as space-variant restoration of the original focused image of the scene.