Deep Learning Methods for Calibrated Photometric Stereo and Beyond.

Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues, i.e., modeling the relationship between surface orientation and intensity at each pixel. Photometric stereo prevails in superior per-pixel resolution and fine reconstruction details. However, it is a complicated problem because of the non-linear relationship caused by non-Lambertian surface reflectance. Recently, various deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces. This paper provides a comprehensive review of existing deep learning-based calibrated photometric stereo methods utilizing orthographic cameras and directional light sources. We first analyze these methods from different perspectives, including input processing, supervision, and network architecture. We summarize the performance of deep learning photometric stereo models on the most widely-used benchmark data set. This demonstrates the advanced performance of deep learning-based photometric stereo methods. Finally, we give suggestions and propose future research trends based on the limitations of existing models.

GR-PSN: Learning to Estimate Surface Normal and Reconstruct Photometric Stereo Images.

In this paper, we propose a novel method, namely GR-PSN, which learns surface normals from photometric stereo images and generates the photometric images under distant illumination from different lighting directions and surface materials. The framework is composed of two subnetworks, named GeometryNet and ReconstructNet, which are cascaded to perform shape reconstruction and image rendering in an end-to-end manner. ReconstructNet introduces additional supervision for surface-normal recovery, forming a closed-loop structure with GeometryNet. We also encode lighting and surface reflectance in ReconstructNet, to achieve arbitrary rendering. In training, we set up a parallel framework to simultaneously learn two arbitrary materials for an object, providing an additional transform loss. Therefore, our method is trained based on the supervision by three different loss functions, namely the surface-normal loss, reconstruction loss, and transform loss. We alternately input the predicted surface-normal map and the ground-truth into ReconstructNet, to achieve stable training for ReconstructNet. Experiments show that our method can accurately recover the surface normals of an object with an arbitrary number of inputs, and can re-render images of the object with arbitrary surface materials. Extensive experimental results show that our proposed method outperforms those methods based on a single surface recovery network and shows realistic rendering results on 100 different materials. Our code can be found in https://github.com/Kelvin-Ju/GR-PSN.

Three-Dimensional Reconstruction with a Laser Line Based on Image In-Painting and Multi-Spectral Photometric Stereo.

This paper presents a multi-spectral photometric stereo (MPS) method based on image in-painting, which can reconstruct the shape using a multi-spectral image with a laser line. One of the difficulties in multi-spectral photometric stereo is to extract the laser line because the required illumination for MPS, e.g., red, green, and blue light, may pollute the laser color. Unlike previous methods, through the improvement of the network proposed by Isola, a Generative Adversarial Network based on image in-painting was proposed, to separate a multi-spectral image with a laser line into a clean laser image and an uncorrupted multi-spectral image without the laser line. Then these results were substituted into the method proposed by Fan to obtain high-precision 3D reconstruction results. To make the proposed method applicable to real-world objects, a rendered image dataset obtained using the rendering models in ShapeNet has been used for training the network. Evaluation using the rendered images and real-world images shows the superiority of the proposed approach over several previous methods.

Recovering Surface Normal and Arbitrary Images: A Dual Regression Network for Photometric Stereo.

Photometric stereo recovers three-dimensional (3D) object surface normal from multiple images under different illumination directions. Traditional photometric stereo methods suffer from the problem of non-Lambertian surfaces with general reflectance. By leveraging deep neural networks, learning-based methods are capable of improving the surface normal estimation under general non-Lambertian surfaces. These state-of-the-art learning-based methods however do not associate surface normal with reconstructed images and, therefore, they cannot explore the beneficial effect of such association on the estimation of the surface normal. In this paper, we specifically exploit the positive impact of this association and propose a novel dual regression network for both fine surface normals and arbitrary reconstructed images in calibrated photometric stereo. Our work unifies the 3D reconstruction and rendering tasks in a deep learning framework, with the explorations including: 1. generating specified reconstructed images under arbitrary illumination directions, which provides more intuitive perception of the reflectance and is extremely useful for visual applications, such as virtual reality, and 2. our dual regression scheme introduces an additional constraint on observed images and reconstructed images, which forms a closed-loop to provide additional supervision. Experiments show that our proposed method achieves accurate reconstructed images under arbitrarily specified illumination directions and it significantly outperforms the state-of-the-art learning-based single regression methods in calibrated photometric stereo.

Immune and anti-oxidant functions of ethanol extracts of Scutellaria baicalensis Georgi in mice bearing U14 cervical cancers.

BACKGROUND: The objective was to study the effect of Scutellaria baicalensis Georgi ethanol extracts (SBGE) on immune and anti-oxidant function in U14 tumor-bearing mice. MATERIALS AND METHODS: U14 tumor-bearing mice were randomly divided into eight groups: a control group, a cyclophosphamide (CTX) group, three dose groups of SBGEI (high, medium, low), and three dose groups of SBGEII (high, medium, low). After two weeks, the thymus and spleen weight indices of mice bearing U14 cervical cancer were calculated. Enzyme linked immunosorbent assays (ELISA) was used to determine the levels of serum IL-2, TNF-α, IL-8, and PCNA. MDA activity and SOD activity in plasma were measured with detection kits. RESULTS: In the SBGE groups, thymus weight and spleen weight indices of U14 tumor-bearing mice were significantly higher than in the control group or CTX group (p<0.05). Compared to control group, the levels of serum IL-2 and TNF-α in U14 tumor-bearing mice increased significantly, whereas the contents of serum IL-8 and PCNA decreased (p<0.05). The activity of SOD increased with the growing dose of SBGE, while the activity of MDA decreased significantly in the higher- dose groups of SBGE. CONCLUSIONS: These findings suggested that SBGE, especially at high dose, 1000 mg/kg, showed significant immune and anti-oxidant effects in U14 tumor-bearing mice, which might be the mechanisms of SBGE inhibition of tumor growth.