Temporal Super-Resolution Using a Multi-Channel Illumination Source.

While sensing in high temporal resolution is necessary for a wide range of applications, it is still limited nowadays due to the camera sampling rate. In this work, we try to increase the temporal resolution beyond the Nyquist frequency, which is limited by the sensor's sampling rate. This work establishes a novel approach to temporal super-resolution that uses the object-reflecting properties from an active illumination source to go beyond this limit. Following theoretical derivation and the development of signal-processing-based algorithms, we demonstrate how to increase the detected temporal spectral range by a factor of six and possibly even more. Our method is supported by simulations and experiments, and we demonstrate (via application) how we use our method to dramatically improve the accuracy of object motion estimation. We share our simulation code on GitHub.

Cost Function Unrolling in Unsupervised Optical Flow.

Steepest descent algorithms, which are commonly used in deep learning, use the gradient as the descent direction, either as-is or after a direction shift using preconditioning. In many scenarios calculating the gradient is numerically hard due to complex or non-differentiable cost functions, specifically next to singular points. This has been commonly overcome by increased DNN model sizes and complexity. In this work we propose a novel mechanism we refer to as Cost Unrolling, for improving the ability of a given DNN model to solve a complex cost function, without modifying its architecture or increasing computational complexity. We focus on the derivation of the Total Variation (TV) smoothness constraint commonly used in unsupervised cost functions. We introduce an iterative differentiable alternative to the TV smoothness constraint, which is demonstrated to produce more stable gradients during training, enable faster convergence and improve the predictions of a given DNN model. We test our method in several tasks, including image denoising and unsupervised optical flow. Replacing the TV smoothness constraint with our loss during DNN training, we report improved results in all tested scenarios. Specifically, our method improves flows predicted at occluded regions, a crucial task by itself, resulting in sharper motion boundaries.

Illumination-Based Color Reconstruction for the Dynamic Vision Sensor.

This work demonstrates a novel, state-of-the-art method to reconstruct colored images via the dynamic vision sensor (DVS). The DVS is an image sensor that indicates only a binary change in brightness, with no information about the captured wavelength (color) or intensity level. However, the reconstruction of the scene's color could be essential for many tasks in computer vision and DVS. We present a novel method for reconstructing a full spatial resolution, colored image utilizing the DVS and an active colored light source. We analyze the DVS response and present two reconstruction algorithms: linear-based and convolutional-neural-network-based. Our two presented methods reconstruct the colored image with high quality, and they do not suffer from any spatial resolution degradation as other methods. In addition, we demonstrate the robustness of our algorithm to changes in environmental conditions, such as illumination and distance. Finally, compared with previous works, we show how we reach the state-of-the-art results. We share our code on GitHub.

Learn2Reg: Comprehensive Multi-Task Medical Image Registration Challenge, Dataset and Evaluation in the Era of Deep Learning.

Image registration is a fundamental medical image analysis task, and a wide variety of approaches have been proposed. However, only a few studies have comprehensively compared medical image registration approaches on a wide range of clinically relevant tasks. This limits the development of registration methods, the adoption of research advances into practice, and a fair benchmark across competing approaches. The Learn2Reg challenge addresses these limitations by providing a multi-task medical image registration data set for comprehensive characterisation of deformable registration algorithms. A continuous evaluation will be possible at https://learn2reg.grand-challenge.org. Learn2Reg covers a wide range of anatomies (brain, abdomen, and thorax), modalities (ultrasound, CT, MR), availability of annotations, as well as intra- and inter-patient registration evaluation. We established an easily accessible framework for training and validation of 3D registration methods, which enabled the compilation of results of over 65 individual method submissions from more than 20 unique teams. We used a complementary set of metrics, including robustness, accuracy, plausibility, and runtime, enabling unique insight into the current state-of-the-art of medical image registration. This paper describes datasets, tasks, evaluation methods and results of the challenge, as well as results of further analysis of transferability to new datasets, the importance of label supervision, and resulting bias. While no single approach worked best across all tasks, many methodological aspects could be identified that push the performance of medical image registration to new state-of-the-art performance. Furthermore, we demystified the common belief that conventional registration methods have to be much slower than deep-learning-based methods.

Hardware analysis for motion estimation task.

This work introduces hardware metrics to evaluate imaging sensor (camera) ability to cope with temporal change (motion). Shifting from images towards moving elements demands better tools for evaluation than just refresh rate, and this work is here to close that gap. We focus on the sampling frequency, signal to noise ratio, rolling shutter, and modulation transfer function as a set of parameters to define four fundamental conditions to evaluate and compare the quality of motion sensing. We further examine our theory on existing hardware used in modern equipment and report our findings.

Author Correction: Active Printed Materials for Complex Self-Evolving Deformations.

A correction has been published and is appended to both the HTML and PDF versions of this paper. The error has not been fixed in the paper.

All Photons Imaging Through Volumetric Scattering.

Imaging through thick highly scattering media (sample thickness ≫ mean free path) can realize broad applications in biomedical and industrial imaging as well as remote sensing. Here we propose a computational "All Photons Imaging" (API) framework that utilizes time-resolved measurement for imaging through thick volumetric scattering by using both early arrived (non-scattered) and diffused photons. As opposed to other methods which aim to lock on specific photons (coherent, ballistic, acoustically modulated, etc.), this framework aims to use all of the optical signal. Compared to conventional early photon measurements for imaging through a 15 mm tissue phantom, our method shows a two fold improvement in spatial resolution (4db increase in Peak SNR). This all optical, calibration-free framework enables widefield imaging through thick turbid media, and opens new avenues in non-invasive testing, analysis, and diagnosis.

Locating and classifying fluorescent tags behind turbid layers using time-resolved inversion.

The use of fluorescent probes and the recovery of their lifetimes allow for significant advances in many imaging systems, in particular, medical imaging systems. Here we propose and experimentally demonstrate reconstructing the locations and lifetimes of fluorescent markers hidden behind a turbid layer. This opens the door to various applications for non-invasive diagnosis, analysis, flowmetry and inspection. The method is based on a time-resolved measurement that captures information about both fluorescence lifetime and spatial position of the probes. To reconstruct the scene, the method relies on a sparse optimization framework to invert time-resolved measurements. This wide-angle technique does not rely on coherence, and does not require the probes to be directly in line of sight of the camera, making it potentially suitable for long-range imaging.

Active printed materials for complex self-evolving deformations.

We propose a new design of complex self-evolving structures that vary over time due to environmental interaction. In conventional 3D printing systems, materials are meant to be stable rather than active and fabricated models are designed and printed as static objects. Here, we introduce a novel approach for simulating and fabricating self-evolving structures that transform into a predetermined shape, changing property and function after fabrication. The new locally coordinated bending primitives combine into a single system, allowing for a global deformation which can stretch, fold and bend given environmental stimulus.

Pose estimation using time-resolved inversion of diffuse light.

We present a novel approach for evaluation of position and orientation of geometric shapes from scattered time-resolved data. Traditionally, imaging systems treat scattering as unwanted and are designed to mitigate the effects. Instead, we show here that scattering can be exploited by implementing a system based on a femtosecond laser and a streak camera. The result is accurate estimation of object pose, which is a fundamental tool in analysis of complex scenarios and plays an important role in our understanding of physical phenomena. Here, we experimentally show that for a given geometry, a single incident illumination point yields enough information for pose estimation and tracking after multiple scattering events. Our technique can be used for single-shot imaging behind walls or through turbid media.

Design of aerosol face masks for children using computerized 3D face analysis.

BACKGROUND: Aerosol masks were originally developed for adults and downsized for children. Overall fit to minimize dead space and a tight seal are problematic, because children's faces undergo rapid and marked topographic and internal anthropometric changes in their first few months/years of life. Facial three-dimensional (3D) anthropometric data were used to design an optimized pediatric mask. METHODS: Children's faces (n=271, aged 1 month to 4 years) were scanned with 3D technology. Data for the distance from the bridge of the nose to the tip of the chin (H) and the width of the mouth opening (W) were used to categorize the scans into "small," "medium," and "large" "clusters." RESULTS: "Average" masks were developed from each cluster to provide an optimal seal with minimal dead space. The resulting computerized contour, W and H, were used to develop the SootherMask® that enables children, "suckling" on their own pacifier, to keep the mask on their face, mainly by means of subatmospheric pressure. The relatively wide and flexible rim of the mask accommodates variations in facial size within and between clusters. CONCLUSIONS: Unique pediatric face masks were developed based on anthropometric data obtained through computerized 3D face analysis. These masks follow facial contours and gently seal to the child's face, and thus may minimize aerosol leakage and dead space.

Graph isomorphisms and automorphisms via spectral signatures.

An isomorphism between two graphs is a connectivity preserving bijective mapping between their sets of vertices. Finding isomorphisms between graphs, or between a graph and itself (automorphisms), is of great importance in applied sciences. The inherent computational complexity of this problem is as yet unknown. Here, we introduce an efficient method to compute such mappings using heat kernels associated with the graph Laplacian. While the problem is combinatorial in nature, in practice we experience polynomial runtime in the number of vertices. As we demonstrate, the proposed method can handle a variety of graphs and is competitive with state-of-the-art packages on various important examples.