Convolutional neural network for improved event-based Shack-Hartmann wavefront reconstruction.

Shack-Hartmann wavefront sensing is a technique for measuring wavefront aberrations, whose use in adaptive optics relies on fast position tracking of an array of spots. These sensors conventionally use frame-based cameras operating at a fixed sampling rate to report pixel intensities, even though only a fraction of the pixels have signal. Prior in-lab experiments have shown feasibility of event-based cameras for Shack-Hartmann wavefront sensing (SHWFS), asynchronously reporting the spot locations as log intensity changes at a microsecond time scale. In our work, we propose a convolutional neural network (CNN) called event-based wavefront network (EBWFNet) that achieves highly accurate estimation of the spot centroid position in real time. We developed a custom Shack-Hartmann wavefront sensing hardware with a common aperture for the synchronized frame- and event-based cameras so that spot centroid locations computed from the frame-based camera may be used to train/test the event-CNN-based centroid position estimation method in an unsupervised manner. Field testing with this hardware allows us to conclude that the proposed EBWFNet achieves sub-pixel accuracy in real-world scenarios with substantial improvement over the state-of-the-art event-based SHWFS. An ablation study reveals the impact of data processing, CNN components, and training cost function; and an unoptimized MATLAB implementation is shown to run faster than 800 Hz on a single GPU.

Continuously variable Fourier filters fabricated using varying angle glancing angle deposition for chip-scale spectroscopy.

Low-cost spectroscopy has received a great deal of attention in recent years in applications such as food inspection, disease detection, and manufacturing. Current spectroscopic systems rely on multiple optical components, making them mechanically fragile systems. In our previous work, we demonstrated the use of Fourier filtering using thin dielectric films. The sampling effect from the cavity resonances can be used to decompose a signal into its Fourier components. Although the thin films were deposited directly on the face of the detectors, filters of varying thicknesses were needed, which required multiple lithographic processes. To overcome this challenge, in this work, we use a continuously variable filters deposited by a single-step electron-beam evaporation technique. We demonstrate a novel, to our knowledge, method that utilizes the glancing angle deposition technique with a continuously varying angle in order to produce tens of variable Fourier filters in a single deposition run. To prove this technique, we deposit this variable filter on a 38-channel linear detector and show the results from this device.

Time-Ordered Recent Event (TORE) Volumes for Event Cameras.

Event cameras are an exciting, new sensor modality enabling high-speed imaging with extremely low-latency and wide dynamic range. Unfortunately, most machine learning architectures are not designed to directly handle sparse data, like that generated from event cameras. Many state-of-the-art algorithms for event cameras rely on interpolated event representations-obscuring crucial timing information, increasing the data volume, and limiting overall network performance. This paper details an event representation called Time-Ordered Recent Event (TORE) volumes. TORE volumes are designed to compactly store raw spike timing information with minimal information loss. This bio-inspired design is memory efficient, computationally fast, avoids time-blocking (i.e., fixed and predefined frame rates), and contains "local memory" from past data. The design is evaluated on a wide range of challenging tasks (e.g., event denoising, image reconstruction, classification, and human pose estimation) and is shown to dramatically improve state-of-the-art performance. TORE volumes are an easy-to-implement replacement for any algorithm currently utilizing event representations.

Framework for improving DoLP and AoLP reconstruction quality in microgrid polarimeters.

In polarimetric imaging, degree and angle of linear polarization (DoLP and AoLP, respectively) are computed from ratios of Stokes parameters. In snapshot imagers, DoLP and AoLP are degraded by inherent mismatches between the spatial bandwidth of the S0, S1, and S2 parameters reconstructed by demosaicking from microgrid polarizer array (MPA)-sampled data. To overcome this, we rigorously show that log-MPA-sampled data approximately decouples DoLP and AoLP from the intensity component (S0) in the spatial Fourier domain. Based on this analysis, we propose an alternative demosaicking strategy aimed at estimating DoLP and AoLP directly from MPA-sampled data. Our method bypasses Stokes parameter estimation, alleviating the spatial bandwidth mismatch problems altogether and reducing computational complexity. We experimentally verify the superior DoLP and AoLP reconstructions of the proposed log-MPA demosaicking compared to the conventional Stokes parameter demosaicking approach in simulation. We simulated the conventional 2 × 2 MPA patterns as well as the more recently introduced 2 × 4 MPA patterns, and report quantitative results (mean squared error, structural similarity index, and polarization angular error) using five demosaicking approaches drawn from the literature. We also provide a closed-form error analysis on the log-MPA-sampled data to demonstrate that the approximation error is negligible for real practical applications.

Lossless White Balance for Improved Lossless CFA Image and Video Compression.

Color filter array is a spatial multiplexing of pixel-sized filters fabricated over pixel sensors in most color image sensors. The state-of-the-art lossless coding techniques of raw sensor data captured by such sensors leverage spatial or cross-color correlation using lifting schemes. In this paper, we propose a lifting-based lossless white balance algorithm. When applied to the raw sensor data, the spatial bandwidth of the implied chrominance signals decreases. We propose to use this white balance as a pre-processing step to lossless CFA subsampled image/video compression, improving the overall coding efficiency of the raw sensor data.

Tunable optical filter using phase change materials for smart IR night vision applications.

In this paper we present a tunable filter using Ge2Sb2Se4Te1 (GSST) phase change material. The design principle of the filter is based on a metal-insulator-metal (MIM) cavity operating in the reflection mode. This is intended for night vision applications that utilize 850nm as the illumination source. The filter allows us to selectively reject the 850nm band in one state. This is illustrated through several daytime and nighttime imaging applications.

Multi-Resolution Aitchison Geometry Image Denoising for Low-Light Photography.

In the low-photon imaging regime, noise in the image sensors is dominated by shot noise, best modeled statistically as Poisson distribution. In this work, we show that the Poisson likelihood function is very well matched with the Bayesian estimation of the "difference of log of contrast of pixel intensities." More specifically, our work is rooted in statistical compositional data analysis, whereby we reinterpret the Aitchison geometry as a multi-resolution analysis in the log-pixel domain. We demonstrate that the difference-log-contrast has wavelet-like properties that correspond well with the human visual system, while being robust to illumination variations. We derive a denoising technique based on an approximate conjugate prior for the latent Aitchison variable that gives rise to an explicit minimum mean squared error estimation. The resulting denoising technique preserves image contrast details that are arguably more meaningful to human vision than the pixel intensity values themselves.

L's approach: illumination pattern estimation technique for optical microscopy.

We propose a novel solution to the correction of illumination nonuniformity without removing the imaging sample. Calibration of the spatial illumination pattern in reflectance microscopy is challenging due to the fact that the illumination source is colocated with the objective lens and therefore cannot be observed directly. Our proposed methodology overcomes this by collecting three spatially translated images in a strategic way. We prove that 'log-illumination pattern' recovery can be reformulated as a deconvolution of log-ratio of captured images, and develop an efficient, noise-robust implementation. Experiments with simulated and reflectance microscopy data verify the effectiveness of our proposed approach. LAY DESCRIPTION: The proposed technique overcomes a common problem of nonuniform illumination in microscopy. For the case of reflectance microscopy, we show that a few images with large amounts of overlap can be used to recover the illumination pattern. In conjunction with an image formation model, the recovered illumination pattern may be used to correct the images captured under similar illumination conditions.

Distance Surface for Event-Based Optical Flow.

We propose DistSurf-OF, a novel optical flow method for neuromorphic cameras. Neuromorphic cameras (or event detection cameras) are an emerging sensor modality that makes use of dynamic vision sensors (DVS) to report asynchronously the log-intensity changes (called "events") exceeding a predefined threshold at each pixel. In absence of the intensity value at each pixel location, we introduce a notion of "distance surface"-the distance transform computed from the detected events-as a proxy for object texture. The distance surface is then used as an input to the intensity-based optical flow methods to recover the two dimensional pixel motion. Real sensor experiments verify that the proposed DistSurf-OF accurately estimates the angle and speed of each events.

Corrupted Reference Image Quality Assessment of Denoised Images.

We propose corrupted reference image quality assessment (CRIQA), a novel foundation for reasoning about image quality and image denoising problems jointly. In order to assess the visual quality of a processed image relative to an ideal reference image (not provided), we predict the full-reference image quality assessment (FRIQA) scores of denoised images without having the direct access to the ideal reference image, but with the help of the observed corrupted image, instead. Our simulation studies verify that the CRIQA scores of denoised images indeed agree with the corresponding FRIQA scores, and human subject studies confirm that CRIQA scores are more consistent with the perceived image denoising quality than the NRIQA scores. We demonstrated the usefulness of CRIQA with an application in denoising parameter tuning.

Monte-Carlo Acceleration of Bilateral Filter and Non-Local Means.

We propose stochastic bilateral filter (SBF) and stochastic non-local means (SNLM), efficient randomized processes that agree with conventional bilateral filter (BF) and non-local means (NLM) on average, respectively. By Monte-Carlo, we repeat this process a few times with different random instantiations so that they can be averaged to attain the correct BF/NLM output. The computational bottleneck of the SBF and SNLM are constant with respect to the window size and the color dimension of the edge image, meaning the execution times for color and hyperspectral images are nearly the same as for the grayscale images. In addition, for SNLM, the complexity is constant with respect to the block size. The proposed stochastic filter implementations are considerably faster than the conventional and existing "fast" implementations for high dimensional image data.

Towards Optimal Denoising of Image Contrast.

Most conventional imaging modalities detect light indirectly by observing high-energy photons. The random nature of photon emission and detection is often the dominant sources of noise in imaging. Such case is referred to as photon-limited imaging, and the noise distribution is well modeled as Poisson. Multiplicative multiscale innovation (MMI) presents a natural model for Poisson count measurement, where the inter-scale relation is represented as random partitioning (binomial distribution) or local image contrast. In this paper, we propose a nonparametric empirical Bayes estimator that minimizes the mean square error of MMI coefficients. The proposed method achieves better performance compared with state-of-the-art methods in both synthetic and real sensor image experiments under low illumination.

Camera-Aware Multi-Resolution Analysis for Raw Image Sensor Data Compression.

We propose novel lossless and lossy compression schemes for color filter array (CFA) sampled images based on the Camera-A ware Multi-Resolution Analysis, or CAMRA. Specifically, by CAMRA we refer to modifications that we make to wavelet transform of CFA sampled images in order to achieve a very high degree of decorrelation at the finest scale wavelet coefficients; and a series of color processing steps applied to the coarse scale wavelet coefficients, aimed at limiting the propagation of lossy compression errors through the subsequent camera processing pipeline. We validated our theoretical analysis and the performance of the proposed compression schemes using the images of natural scenes captured in a raw format. The experimental results verify that our proposed methods improve coding efficiency relative to the standard and the state-of-the-art compression schemes for CFA sampled images.

Pixel Binning for High Dynamic Range Color Image Sensor Using Square Sampling Lattice.

We propose a new pixel binning scheme for color image sensors. We minimized distortion caused by binning by requiring that the superpixels lie on a square sampling lattice. The proposed binning schemes achieve the equivalent of 4.42 times signal strength improvement with the image resolution loss of 5 times, higher in noise performance and in resolution than the existing binning schemes. As a result, the proposed binning has considerably less artifacts and better noise performance compared with the existing binning schemes. In addition, we provide an extension to the proposed binning scheme for performing single-shot high dynamic range image acquisition.

Improved Denoising via Poisson Mixture Modeling of Image Sensor Noise.

This paper describes a study aimed at comparing the real image sensor noise distribution to the models of noise often assumed in image denoising designs. A quantile analysis in pixel, wavelet transform, and variance stabilization domains reveal that the tails of Poisson, signal-dependent Gaussian, and Poisson-Gaussian models are too short to capture real sensor noise behavior. A new Poisson mixture noise model is proposed to correct the mismatch of tail behavior. Based on the fact that noise model mismatch results in image denoising that undersmoothes real sensor data, we propose a mixture of Poisson denoising method to remove the denoising artifacts without affecting image details, such as edge and textures. Experiments with real sensor data verify that denoising for real image sensor data is indeed improved by this new technique.

Joint Defogging and Demosaicking.

Image defogging is a technique used extensively for enhancing visual quality of images in bad weather conditions. Even though defogging algorithms have been well studied, defogging performance is degraded by demosaicking artifacts and sensor noise amplification in distant scenes. In order to improve the visual quality of restored images, we propose a novel approach to perform defogging and demosaicking simultaneously. We conclude that better defogging performance with fewer artifacts can be achieved when a defogging algorithm is combined with a demosaicking algorithm simultaneously. We also demonstrate that the proposed joint algorithm has the benefit of suppressing noise amplification in distant scenes. In addition, we validate our theoretical analysis and observations for both synthesized data sets with ground truth fog-free images and natural scene data sets captured in a raw format.

Blind Deblurring and Denoising of Images Corrupted by Unidirectional Object Motion Blur and Sensor Noise.

Low light photography suffers from blur and noise. In this paper, we propose a novel method to recover a dense estimate of spatially varying blur kernel as well as a denoised and deblurred image from a single noisy and object motion blurred image. A proposed method takes the advantage of the sparse representation of double discrete wavelet transform-a generative model of image blur that simplifies the wavelet analysis of a blurred image-and the Bayesian perspective of modeling the prior distribution of the latent sharp wavelet coefficient and the likelihood function that makes the noise handling explicit. We demonstrate the effectiveness of the proposed method on moderate noise and severely blurred images using simulated and real camera data.

Defocus Blur-Invariant Scale-Space Feature Extractions.

We propose modifications to scale-space feature extraction techniques scale-invariant feature transform (SIFT) and speeded up robust features (SURFs) that make the feature detection and description invariant to defocus blur. Specifically, the scale-space blob detection relies on the second derivative responses of images. Our analysis of circular defocus blur (which sufficiently approximates a real camera blur kernel) and its effect on scale-space blob detection suggests that fourth derivative-and not the usual second derivative-is optimal for detecting the blurred blobs, while multi-scale descriptors of blurred blobs are effective at establishing correspondences between the blurred images. The proposed defocus blur-invariant (DBI) scale-space feature extraction techniques-which we refer to as DBI-SIFT and DBI-SURF-do not require image deblurring nor blur kernel estimation, meaning that their accuracy does not depend on the quality of image deblurring. We offer empirical evidence of blur invariance by establishing interest point correspondences between sharp or blurred reference images and blurred target images.

Fourier Spectral Filter Array for Optimal Multispectral Imaging.

Limitations to existing multispectral imaging modalities include speed, cost, range, spatial resolution, and application-specific system designs that lack versatility of the hyperspectral imaging modalities. In this paper, we propose a novel general-purpose single-shot passive multispectral imaging modality. Central to this design is a new type of spectral filter array (SFA) based not on the notion of spatially multiplexing narrowband filters, but instead aimed at enabling single-shot Fourier transform spectroscopy. We refer to this new SFA pattern as Fourier SFA, and we prove that this design solves the problem of optimally sampling the hyperspectral image data.

Fourier multispectral imaging.

Current multispectral imaging systems use narrowband filters to capture the spectral content of a scene, which necessitates different filters to be designed for each application. In this paper, we demonstrate the concept of Fourier multispectral imaging which uses filters with sinusoidally varying transmittance. We designed and built these filters employing a single-cavity resonance, and made spectral measurements with a multispectral LED array. The measurements show that spectral features such as transmission and absorption peaks are preserved with this technique, which makes it a versatile technique than narrowband filters for a wide range of multispectral imaging applications.