Neuromorphic Imaging With Joint Image Deblurring and Event Denoising.

Neuromorphic imaging reacts to per-pixel brightness changes of a dynamic scene with high temporal precision and responds with asynchronous streaming events as a result. It also often supports a simultaneous output of an intensity image. Nevertheless, the raw events typically involve a large amount of noise due to the high sensitivity of the sensor, while capturing fast-moving objects at low frame rates results in blurry images. These deficiencies significantly degrade human observation and machine processing. Fortunately, the two information sources are inherently complementary - events with microsecond-level temporal resolution, which are triggered by the edges of objects recorded in a latent sharp image, can supply rich motion details missing from the blurry one. In this work, we bring the two types of data together and introduce a simple yet effective unifying algorithm to jointly reconstruct blur-free images and noise-robust events in an iterative coarse-to-fine fashion. Specifically, an event-regularized prior offers precise high-frequency structures and dynamic features for blind deblurring, while image gradients serve as a kind of faithful supervision in regulating neuromorphic noise removal. Comprehensively evaluated on real and synthetic samples, such a synergy delivers superior reconstruction quality for both images with severe motion blur and raw event streams with a storm of noise, and also exhibits greater robustness to challenging realistic scenarios such as varying levels of illumination, contrast and motion magnitude. Meanwhile, it can be driven by much fewer events and holds a competitive edge at computational time overhead, rendering itself preferable as available computing resources are limited. Our solution gives impetus to the improvement of both sensing data and paves the way for highly accurate neuromorphic reasoning and analysis.

Group Sparsity Mixture Model and Its Application on Image Denoising.

Prior learning is a fundamental problem in the field of image processing. In this paper, we conduct a detailed study on (1) how to model and learn the prior of the image patch group, which consists of a group of non-local similar image patches, and (2) how to apply the learned prior to the whole image denoising task. To tackle the first problem, we propose a new prior model named Group Sparsity Mixture Model (GSMM). With the bilateral matrix multiplication, the GSMM can model both the local feature of a single patch and the relation among non-local similar patches, and thus it is very suitable for patch group based prior learning. This is supported by the parameter analysis which demonstrates that the learned GSMM successfully captures the inherent strong sparsity embodied in the image patch group. Besides, as a mixture model, GSMM can be used for patch group classification. This makes the image denoising method based on GSMM capable of processing patch groups flexibly. To tackle the second problem, we propose an efficient and effective patch group based image denoising framework, which is plug-and-play and compatible with any patch group prior model. Using this framework, we construct two versions of GSMM based image denoising methods, both of which outperform the competing methods based on other prior models, e.g., Field of Experts (FoE) and Gaussian Mixture Model (GMM). Also, the better version is competitive with the state-of-the-art model based method WNNM with about ×8 faster average running speed.

Image Regularizations Based on the Sparsity of Corner Points.

Many analysis-based regularizations proposed so far employ a common prior information, i.e., edges in an image are sparse. However, in local edge regions and texture regions, this prior may not hold. As a result, the performance of regularizations based on the edge sparsity may be unsatisfactory in such regions for image-related inverse problems. These regularizations tend to smooth out the edges while eliminating the noise. In other words, these regularizations' abilities of preserving edges are limited. In this paper, a new prior that the corner points in a natural image are sparse was proposed to construct regularizations. Intuitively, even in local edge regions and texture regions, the sparsity of corner points may still exist, and hence, the regularizations based on it can achieve better performance than those based on the edge sparsity. As an example, by utilizing the sparsity of corner points, we proposed a new regularization based on Noble's corner measure function. Our experiments demonstrated the excellent performance of the proposed regularization for both image denoising and deblurring problems, especially in local edge regions and texture regions.

Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy.

To increase the temporal resolution and maximal imaging time of super-resolution (SR) microscopy, we have developed a deconvolution algorithm for structured illumination microscopy based on Hessian matrixes (Hessian-SIM). It uses the continuity of biological structures in multiple dimensions as a priori knowledge to guide image reconstruction and attains artifact-minimized SR images with less than 10% of the photon dose used by conventional SIM while substantially outperforming current algorithms at low signal intensities. Hessian-SIM enables rapid imaging of moving vesicles or loops in the endoplasmic reticulum without motion artifacts and with a spatiotemporal resolution of 88 nm and 188 Hz. Its high sensitivity allows the use of sub-millisecond excitation pulses followed by dark recovery times to reduce photobleaching of fluorescent proteins, enabling hour-long time-lapse SR imaging of actin filaments in live cells. Finally, we observed the structural dynamics of mitochondrial cristae and structures that, to our knowledge, have not been observed previously, such as enlarged fusion pores during vesicle exocytosis.