Unsupervised learning with a physics-based autoencoder for estimating the thickness and mixing ratio of pigments.

Layered surface objects represented by decorated tomb murals and watercolors are in danger of deterioration and damage. To address these dangers, it is necessary to analyze the pigments' thickness and mixing ratio and record the current status. This paper proposes an unsupervised autoencoder model for thickness and mixing ratio estimation. The input of our autoencoder is spectral data of layered surface objects. Our autoencoder is unique, to our knowledge, in that the decoder part uses a physical model, the Kubelka-Munk model. Since we use the Kubelka-Munk model for the decoder, latent variables in the middle layer can be interpretable as the pigment thickness and mixing ratio. We conducted a quantitative evaluation using synthetic data and confirmed that our autoencoder provides a highly accurate estimation. We measured an object with layered surface pigments for qualitative evaluation and confirmed that our method is valid in an actual environment. We also present the superiority of our unsupervised autoencoder over supervised learning.

Descattering for transmissive inspection in production line using slanted linear image sensors.

We propose a descattering method that can be easily applied to food production lines. The system consists of several sets of linear image sensors and linear light sources slanted at different angles. The images captured by these sensors are partially clear along the direction perpendicular to the sensors. We computationally integrate these images on the frequency domain into a single clear image. The effectiveness of the proposed method is assessed by simulation and real-world experiments. The results show that our method recovers clear images. We demonstrate the applicability of the proposed method to a real production line by a prototype system.

Super-resolution for a dispersive spectrometer using a tilted area sensor and spectrally varying blur kernel interpolation.

The grating, lens, and linear sensor determine a spectrometer's wavelength resolution and measurement range. While conventional methods have tried to improve the optical design to obtain a better resolution, they have a limitation caused by the physical property. To improve the resolution, we introduce a super-resolution method from the computer vision field. We propose tilting an area sensor to realize accurate subpixel shifting and recover a high-resolution spectrum using interpolated spectrally varying kernels. We experimentally validate that the proposed method achieved a high spectral resolution of 0.141nm in 400-800nm by just tilting the sensor in the spectrometer.

Time-of-flight imaging in fog using multiple time-gated exposures.

We propose a time-of-flight measurement algorithm for depth and intensity that is robust to fog. The key idea of the algorithm is to compensate for the scattering effects of fog by using multiple time-gating and assigning one time-gated exposure for scattering property estimation. Once the property is estimated, the depth and intensity can be reconstructed from the rest of the exposures via a physics-based model. Several experiments with artificial fog show that our method can measure depth and intensity irrespective of the traits of the fog. We also confirm the effectiveness of our method in real fog through an outdoor experiment.

Reconstruction of Geometric and Optical Parameters of Non-Planar Objects with Thin Film.

Here, we propose a novel method to estimate the parameters of non-planar objects with thin film surfaces. Being able to estimate the optical parameters of objects with thin film surfaces has a wide range of applications from industrial inspections to biological and archaeology research. However, there are many challenging issues that need to be overcome to model such parameters. The appearance of thin film objects is highly dependent on the surface orientation and optical parameters such as the refractive index and film thickness. First, we therefore analyzed the optical parameters of non-planar objects with thin film surfaces. Next, we proposed and implemented an analysis procedure and demonstrated its effectiveness for studying planar objects with thin film surfaces. Finally, we developed a device to acquire the shapes and optical parameters of objects with thin film surfaces using a camera and demonstrated the effectiveness of our method experimentally. Then, we surveyed the errors caused by the light source. We discussed the difference between the theoretically obtained parameters and experimental data obtained using a hyper spectral camera.

Programmable Non-Epipolar Indirect Light Transport: Capture and Analysis.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

Time-Resolved Far Infrared Light Transport Decomposition for Thermal Photometric Stereo.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, the transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated, and therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

Depth from phasor distortions in fog.

This paper presents a time-of-flight (ToF) measurement method for use in foggy weather. The depth measured by a ToF camera is greatly distorted in fog because the light scattered in the fog reaches the camera much faster than the target reflection. We reveal that the multi-frequency measurements contain a cue whether two arbitrary pixels have the same depth. After clustering the same depth pixels using this cue, the original depth can be recovered for each cluster by line fitting in the Cartesian coordinate frame. The effectiveness of this method is evaluated numerically via real-world and road-scale experiments.

Material Classification from Time-of-Flight Distortions.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. The proposed method is built upon a key observation that the depth measurement by a ToF camera is distorted for objects with certain materials, especially with translucent materials. We show that this distortion is due to the variation of time domain impulse responses across materials and also due to the measurement mechanism of the ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the object material, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as a feature for classification and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluations and real-world experiments, showing its capability of material classification, even for visually indistinguishable objects.

Recovering Inner Slices of Layered Translucent Objects by Multi-Frequency Illumination.

This paper describes a method for recovering appearance of inner slices of translucent objects. The appearance of a layered translucent object is the summed appearance of all layers, where each layer is blurred by a depth-dependent point spread function (PSF). By exploiting the difference of low-pass characteristics of depth-dependent PSFs, we develop a multi-frequency illumination method for obtaining the appearance of individual inner slices. Specifically, by observing the target object with varying the spatial frequency of checker-pattern illumination, our method recovers the appearance of inner slices via computation. We study the effect of non-uniform transmission due to inhomogeneity of translucent objects and develop a method for recovering clear inner slices based on the pixel-wise PSF estimates under the assumption of spatial smoothness of inner slice appearances. We quantitatively evaluate the accuracy of the proposed method by simulations and qualitatively show faithful recovery using real-world scenes.

Adherent Raindrop Modeling, Detectionand Removal in Video.

Raindrops adhered to a windscreen or window glass can significantly degrade the visibility of a scene. Modeling, detecting and removing raindrops will, therefore, benefit many computer vision applications, particularly outdoor surveillance systems and intelligent vehicle systems. In this paper, a method that automatically detects and removes adherent raindrops is introduced. The core idea is to exploit the local spatio-temporal derivatives of raindrops. To accomplish the idea, we first model adherent raindrops using law of physics, and detect raindrops based on these models in combination with motion and intensity temporal derivatives of the input video. Having detected the raindrops, we remove them and restore the images based on an analysis that some areas of raindrops completely occludes the scene, and some other areas occlude only partially. For partially occluding areas, we restore them by retrieving as much as possible information of the scene, namely, by solving a blending function on the detected partially occluding areas using the temporal intensity derivative. For completely occluding areas, we recover them by using a video completion technique. Experimental results using various real videos show the effectiveness of our method.

Optical tomography with discretized path integral.

We present a framework for optical tomography based on a path integral. Instead of directly solving the radiative transport equations, which have been widely used in optical tomography, we use a path integral that has been developed for rendering participating media based on the volume rendering equation in computer graphics. For a discretized two-dimensional layered grid, we develop an algorithm to estimate the extinction coefficients of each voxel with an interior point method. Numerical simulation results are shown to demonstrate that the proposed method works well.

Analysis of photometric factors based on photometric linearization.

We propose a method for analyzing photometric factors, such as diffuse reflection, specular reflection, attached shadow, and cast shadow. For analyzing real images, we utilize the photometric linearization method, which was originally proposed for image synthesis. First, we show that each pixel can be photometrically classified by a simple comparison of the pixel intensity. Our classification algorithm requires neither 3D shape information nor color information of the scene. Then, we show that the accuracy of the photometric linearization can be improved by introducing a new classification-based criterion to the linearization process. Experimental results show that photometric factors can be correctly classified without any special devices. A further experiment shows that the proposed method is effective for photometric stereo.