Camera Shutter-Independent Registration and Rectification.

Inevitable camera motion during exposure does not augur well for free-hand photography. Distortions introduced in images can be of different types and mainly depend on the structure of the scene, the nature of camera motion, and the shutter mechanism of the camera. In this paper, we address the problem of registering images taken from global shutter and rolling shutter cameras and reveal the constraints on camera motion that admit registration, change detection, and rectification. Our analysis encompasses degradations arising from camera motion during exposure and differences in shutter mechanisms. We also investigate conditions under which camera motions causing distortions in reference and target image can be decoupled to yield the underlying latent image through RS rectification. We validate our approach using several synthetic and real examples.

Multi-Image Blind Super-Resolution of 3D Scenes.

We address the problem of estimating the latent high-resolution (HR) image of a 3D scene from a set of non-uniformly motion blurred low-resolution (LR) images captured in the burst mode using a hand-held camera. Existing blind super-resolution (SR) techniques that account for motion blur are restricted to fronto-parallel planar scenes. We initially develop an SR motion blur model to explain the image formation process in 3D scenes. We then use this model to solve for the three unknowns-the camera trajectories, the depth map of the scene, and the latent HR image. We first compute the global HR camera motion corresponding to each LR observation from patches lying on a reference depth layer in the input images. Using the estimated trajectories, we compute the latent HR image and the underlying depth map iteratively using an alternating minimization framework. Experiments on synthetic and real data reveal that our proposed method outperforms the state-of-the-art techniques by a significant margin.

Image Registration and Change Detection under Rolling Shutter Motion Blur.

In this paper, we address the problem of registering a distorted image and a reference image of the same scene by estimating the camera motion that had caused the distortion. We simultaneously detect the regions of changes between the two images. We attend to the coalesced effect of rolling shutter and motion blur that occurs frequently in moving CMOS cameras. We first model a general image formation framework for a 3D scene following a layered approach in the presence of rolling shutter and motion blur. We then develop an algorithm which performs layered registration to detect changes. This algorithm includes an optimisation problem that leverages the sparsity of the camera trajectory in the pose space and the sparsity of changes in the spatial domain. We create a synthetic dataset for change detection in the presence of motion blur and rolling shutter effect covering different types of camera motion for both planar and 3D scenes. We compare our method with existing registration methods and also show several real examples captured with CMOS cameras.

Recognizing blurred, nonfrontal, illumination, and expression variant partially occluded faces.

The focus of this paper is on the problem of recognizing faces across space-varying motion blur, changes in pose, illumination, and expression, as well as partial occlusion, when only a single image per subject is available in the gallery. We show how the blur, incurred due to relative motion between the camera and the subject during exposure, can be estimated from the alpha matte of pixels that straddle the boundary between the face and the background. We also devise a strategy to automatically generate the trimap required for matte estimation. Having computed the motion via the matte of the probe, we account for pose variations by synthesizing from the intensity image of the frontal gallery a face image that matches the pose of the probe. To handle illumination, expression variations, and partial occlusion, we model the probe as a linear combination of nine blurred illumination basis images in the synthesized nonfrontal pose, plus a sparse occlusion. We also advocate a recognition metric that capitalizes on the sparsity of the occluded pixels. The performance of our method is extensively validated on synthetic as well as real face data.

Face recognition across non-uniform motion blur, illumination, and pose.

Existing methods for performing face recognition in the presence of blur are based on the convolution model and cannot handle non-uniform blurring situations that frequently arise from tilts and rotations in hand-held cameras. In this paper, we propose a methodology for face recognition in the presence of space-varying motion blur comprising of arbitrarily-shaped kernels. We model the blurred face as a convex combination of geometrically transformed instances of the focused gallery face, and show that the set of all images obtained by non-uniformly blurring a given image forms a convex set. We first propose a non-uniform blur-robust algorithm by making use of the assumption of a sparse camera trajectory in the camera motion space to build an energy function with l1 -norm constraint on the camera motion. The framework is then extended to handle illumination variations by exploiting the fact that the set of all images obtained from a face image by non-uniform blurring and changing the illumination forms a bi-convex set. Finally, we propose an elegant extension to also account for variations in pose.

Deskewing of underwater images.

We address the problem of restoring a static planar scene degraded by skewing effect when imaged through adynamic water surface. In particular, we investigate geometric distortions due to unidirectional cyclic waves and circular ripples,phenomena that are most prevalent in fluid flow. Although the camera and scene are stationary, light rays emanating from a scene undergo refraction at the fluid­air interface. This refraction effect is time varying for dynamic fluids and results in nonrigid distortions (skew) in the captured image. These distortions can be associated with motion blur depending on the exposure time of the camera. In the first part of this paper, we establish the condition under which the blur induced due to unidirectional cyclic waves can be treated as space invariant. We proceed to derive a mathematical model for blur formation and propose a restoration scheme using a single degraded observation. In the second part, we reveal how the blur induced by circular ripples(though space variant) can be modeled as uniform in the polar domain and develop a method for deskewing. The proposed methods are tested on synthetic as well as real examples.

Non-uniform deblurring in HDR image reconstruction.

Hand-held cameras inevitably result in blurred images caused by camera-shake, and even more so in high dynamic range imaging applications where multiple images are captured over a wide range of exposure settings. The degree of blurring depends on many factors such as exposure time, stability of the platform, and user experience. Camera shake involves not only translations but also rotations resulting in nonuniform blurring. In this paper, we develop a method that takes input non-uniformly blurred and differently exposed images to extract the deblurred, latent irradiance image. We use transformation spread function (TSF) to effectively model the blur caused by camera motion. We first estimate the TSFs of the blurred images from locally derived point spread functions by exploiting their linear relationship. The scene irradiance is then estimated by minimizing a suitably derived cost functional. Two important cases are investigated wherein 1) only the higher exposures are blurred and 2) all the captured frames are blurred.