Formulating Event-Based Image Reconstruction as a Linear Inverse Problem With Deep Regularization Using Optical Flow.

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event_based_image_rec_inverse_problem.

Closed-form solution for thin lens image irradiance under arbitrary solid angle.

Optical imaging systems are found everywhere in modern society. They are integral to computer vision, where the goal is often to infer geometric and radiometric information about a 3D environment given limited sensing resources. It is helpful to develop relationships between these real-world properties and the actual measurements that are taken, such as 2D images. To this end, we propose a new relationship between object radiance and image irradiance based on power conservation and a thin lens imaging model. The relationship has a closed-form solution for in-focus points and can be solved via numerical integration for points that are not focused. It can be thought of as a generalization of Horn's commonly accepted irradiance equation. Through both ray tracing simulations and comparison to the intensity values of actual images, we believe our equation provides better accuracy than Horn's equation. An improvement is most notable for large lenses and near-focused images where the pinhole imaging model implicit in Horn's derivation breaks down. Outside of this regime, our model validates the use of Horn's approximation through a more thorough theoretical foundation.

Adjoint active surfaces for localization and imaging.

This paper addresses the problem of localizing and segmenting regions embedded within a surrounding medium by characterizing their boundaries, as opposed to imaging the entirety of the volume. Active surfaces are used to directly reconstruct the shape of the region of interest. We describe the procedure for finding the optimal surface, which is computed iteratively via gradient descent that exploits the sensitivity of an error minimization functional to changes of the active surface. In doing so, we introduce the adjoint model to compute the sensitivity, and in this respect, the method shares common ground with several other disciplines, such as optimal control. Finally, we illustrate the proposed active surface technique in the framework of wave propagation governed by the scalar Helmholtz equation. Potential applications include electromagnetics, acoustics, geophysics, nondestructive testing, and medical imaging.

3D Topology Preserving Flows for Viewpoint-Based Cortical Unfolding.

We present a variational method for unfolding of the cortex based on a user-chosen point of view as an alternative to more traditional global flattening methods, which incur more distortion around the region of interest. Our approach involves three novel contributions. The first is an energy function and its corresponding gradient flow to measure the average visibility of a region of interest of a surface with respect to a given viewpoint. The second is an additional energy function and flow designed to preserve the 3D topology of the evolving surface. The third is a method that dramatically improves the computational speed of the 3D topology preservation approach by creating a tree structure of the 3D surface and using a recursion technique. Experiments results show that the proposed approach can successfully unfold highly convoluted surfaces such as the cortex while preserving their topology.

A hybrid Eulerian-Lagrangian approach for thickness, correspondence, and gridding of annular tissues.

We present a novel approach to efficiently compute thickness, correspondence, and gridding of tissues between two simply connected boundaries. The solution of Laplace's equation within the tissue region provides a harmonic function whose gradient flow determines the correspondence trajectories going from one boundary to the other. The proposed method uses and expands upon two recently introduced techniques in order to compute thickness and correspondences based on these trajectories. Pairs of partial differential equations are efficiently computed within an Eulerian framework and combined with a Lagrangian approach so that correspondences trajectories are partially constructed when necessary. Examples are presented in order to compare the performance of this method with those of the pure Lagrangian and pure Eulerian approaches. Results show that the proposed technique takes advantage of both the speed of the Eulerian approach and the accuracy of the Lagrangian approach.

Integral invariants for shape matching.

For shapes represented as closed planar contours, we introduce a class of functionals which are invariant with respect to the Euclidean group and which are obtained by performing integral operations. While such integral invariants enjoy some of the desirable properties of their differential counterparts, such as locality of computation (which allows matching under occlusions) and uniqueness of representation (asymptotically), they do not exhibit the noise sensitivity associated with differential quantities and, therefore, do not require presmoothing of the input shape. Our formulation allows the analysis of shapes at multiple scales. Based on integral invariants, we define a notion of distance between shapes. The proposed distance measure can be computed efficiently and allows warping the shape boundaries onto each other; its computation results in optimal point correspondence as an intermediate step. Numerical results on shape matching demonstrate that this framework can match shapes despite the deformation of subparts, missing parts and noise. As a quantitative analysis, we report matching scores for shape retrieval from a database.

An Eulerian PDE approach for computing tissue thickness.

We outline an Eulerian framework for computing the thickness of tissues between two simply connected boundaries that does not require landmark points or parameterizations of either boundary. Thickness is defined as the length of correspondence trajectories, which run from one tissue boundary to the other, and which follow a smooth vector field constructed in the region between the boundaries. A pair of partial differential equations (PDEs) that are guided by this vector field are then solved over this region, and the sum of their solutions yields the thickness of the tissue region. Unlike other approaches, this approach does not require explicit construction of any correspondence trajectories. An efficient, stable, and computationally fast solution to these PDEs is found by careful selection of finite differences according to an upwinding condition. The behavior and performance of our method is demonstrated on two simulations and two magnetic resonance imaging data sets in two and three dimensions. These experiments reveal very good performance and show strong potential for application in tissue thickness visualization and quantification.