Towards practical single-shot phase retrieval with physics-driven deep neural network.

Phase retrieval (PR), a long-established challenge for recovering a complex-valued signal from its Fourier intensity-only measurements, has attracted considerable attention due to its widespread applications in optical imaging. Recently, deep learning-based approaches were developed and allowed single-shot PR. However, due to the substantial disparity between the input and output domains of the PR problems, the performance of these approaches using vanilla deep neural networks (DNN) still has much room to improve. To increase the reconstruction accuracy, physics-informed approaches were suggested to incorporate the Fourier intensity measurements into an iterative estimation procedure. Since the approach is iterative, they require a lengthy computation process, and the accuracy is still not satisfactory for images with complex structures. Besides, many of these approaches work on simulation data that ignore some common problems such as saturation and quantization errors in practical optical PR systems. In this paper, a novel physics-driven multi-scale DNN structure dubbed PPRNet is proposed. Similar to other deep learning-based PR methods, PPRNet requires only a single Fourier intensity measurement. It is physics-driven that the network is guided to follow the Fourier intensity measurement at different scales to enhance the reconstruction accuracy. PPRNet has a feedforward structure and can be end-to-end trained. Thus, it is much faster and more accurate than the traditional physics-driven PR approaches. Extensive simulations and experiments on an optical platform were conducted. The results demonstrate the superiority and practicality of the proposed PPRNet over the traditional learning-based PR methods.

SiSPRNet: end-to-end learning for single-shot phase retrieval.

With the success of deep learning methods in many image processing tasks, deep learning approaches have also been introduced to the phase retrieval problem recently. These approaches are different from the traditional iterative optimization methods in that they usually require only one intensity measurement and can reconstruct phase images in real-time. However, because of tremendous domain discrepancy, the quality of the reconstructed images given by these approaches still has much room to improve to meet the general application requirements. In this paper, we design a novel deep neural network structure named SiSPRNet for phase retrieval based on a single Fourier intensity measurement. To effectively utilize the spectral information of the measurements, we propose a new feature extraction unit using the Multi-Layer Perceptron (MLP) as the front end. It allows all pixels of the input intensity image to be considered together for exploring their global representation. The size of the MLP is carefully designed to facilitate the extraction of the representative features while reducing noises and outliers. A dropout layer is also equipped to mitigate the possible overfitting problem in training the MLP. To promote the global correlation in the reconstructed images, a self-attention mechanism is introduced to the Up-sampling and Reconstruction (UR) blocks of the proposed SiSPRNet. These UR blocks are inserted into a residual learning structure to prevent the weak information flow and vanishing gradient problems due to their complex layer structure. Extensive evaluations of the proposed model are performed using different testing datasets of phase-only images and images with linearly related magnitude and phase. Experiments were conducted on an optical experimentation platform (with defocusing to reduce the saturation problem) to understand the performance of different deep learning methods when working in a practical environment. The results demonstrate that the proposed approach consistently outperforms other deep learning methods in single-shot maskless phase retrieval. The source codes of the proposed method have been released in Github [see references].

Joint Spine Segmentation and Noise Removal From Ultrasound Volume Projection Images With Selective Feature Sharing.

Volume Projection Imaging from ultrasound data is a promising technique to visualize spine features and diagnose Adolescent Idiopathic Scoliosis. In this paper, we present a novel multi-task framework to reduce the scan noise in volume projection images and to segment different spine features simultaneously, which provides an appealing alternative for intelligent scoliosis assessment in clinical applications. Our proposed framework consists of two streams: i) A noise removal stream based on generative adversarial networks, which aims to achieve effective scan noise removal in a weakly-supervised manner, i.e., without paired noisy-clean samples for learning; ii) A spine segmentation stream, which aims to predict accurate bone masks. To establish the interaction between these two tasks, we propose a selective feature-sharing strategy to transfer only the beneficial features, while filtering out the useless or harmful information. We evaluate our proposed framework on both scan noise removal and spine segmentation tasks. The experimental results demonstrate that our proposed method achieves promising performance on both tasks, which provides an appealing approach to facilitating clinical diagnosis.

Invertible Image Decolorization.

Invertible image decolorization is a useful color compression technique to reduce the cost in multimedia systems. Invertible decolorization aims to synthesize faithful grayscales from color images, which can be fully restored to the original color version. In this paper, we propose a novel color compression method to produce invertible grayscale images using invertible neural networks (INNs). Our key idea is to separate the color information from color images, and encode the color information into a set of Gaussian distributed latent variables via INNs. By this means, we force the color information lost in grayscale generation to be independent of the input color image. Therefore, the original color version can be efficiently recovered by randomly re-sampling a new set of Gaussian distributed variables, together with the synthetic grayscale, through the reverse mapping of INNs. To effectively learn the invertible grayscale, we introduce the wavelet transformation into a UNet-like INN architecture, and further present a quantization embedding to prevent the information omission in format conversion, which improves the generalizability of the framework in real-world scenarios. Extensive experiments on three widely used benchmarks demonstrate that the proposed method achieves a state-of-the-art performance in terms of both qualitative and quantitative results, which shows its superiority in multimedia communication and storage systems.

Improved Multiple-Image-Based Reflection Removal Algorithm Using Deep Neural Networks.

When imaging through a semi-reflective medium such as glass, the reflection of another scene can often be found in the captured images. It degrades the quality of the images and affects their subsequent analyses. In this paper, a novel deep neural network approach for solving the reflection problem in imaging is presented. Traditional reflection removal methods not only require long computation time for solving different optimization functions, their performance is also not guaranteed. As array cameras are readily available in nowadays imaging devices, we first suggest in this paper a multiple-image based depth estimation method using a convolutional neural network (CNN). The proposed network avoids the depth ambiguity problem due to the reflection in the image, and directly estimates the depths along the image edges. They are then used to classify the edges as belonging to the background or reflection. Since edges having similar depth values are error prone in the classification, they are removed from the reflection removal process. We suggest a generative adversarial network (GAN) to regenerate the removed background edges. Finally, the estimated background edge map is fed to another auto-encoder network to assist the extraction of the background from the original image. Experimental results show that the proposed reflection removal algorithm achieves superior performance both quantitatively and qualitatively as compared to the state-of-the-art methods. The proposed algorithm also shows much faster speed compared to the existing approaches using the traditional optimization methods.

Robust Reflection Removal Based on Light Field Imaging.

In daily photography, it is common to capture images in the reflection of an unwanted scene. This circumstance arises frequently when imaging through a semi-reflecting material such as glass. The unwanted reflection will affect the visibility of the background image and introduce ambiguity that perturbs the subsequent analysis on the image. It is a very challenging task to remove the reflection of an image since the problem is severely ill-posed. In this paper, we propose a novel algorithm to solve the reflection removal problem based on light field (LF) imaging. For the proposed algorithm, we first show that the strong gradient points of an LF epipolar plane image (EPI) are preserved after adding to the EPI of another LF image. We can then make use of these strong gradient points to give a rough estimation of the background and reflection. Rather than assuming that the background and reflection have absolutely different disparity ranges, we propose a sandwich layer model to allow them to have common disparities, which is more realistic in practical situations. Then, the background image is refined by recovering the components in the shared disparity range using an iterative enhancement process. Our experimental results show that the proposed algorithm achieves superior performance over traditional approaches both qualitatively and quantitatively. These results verify the robustness of the proposed algorithm when working with images captured from real-life scenes.

Robust Single-shot Fringe Projection Profilometry Based on Morphological Component Analysis.

In a fringe projection profilometry (FPP) process, the captured fringe images can be modeled as the superimposition of the projected fringe patterns on the texture of the objects. Extracting the fringe patterns from the captured fringe images is an essential procedure in FPP; but traditional single-shot FPP methods often fail to perform if the objects have a highly textured surface. In this paper, a new single-shot FPP algorithm which allows the object texture and fringe pattern to be estimated simultaneously is proposed. The heart of the proposed algorithm is an enhanced morphological component analysis (MCA) tailored for FPP problems. Conventional MCA methods which use a uniform threshold in an iterative optimization process are inefficient to separate fringe-like patterns from image texture. We extend the conventional MCA by taking advantage of the low-rank structure of the fringe's sparse representation to enable an adaptive thresholding process. It ends up with a robust single-shot FPP algorithm that can extract the fringe pattern even if the object has a highly textured surface. The proposed approach has a side benefit that the object texture can be simultaneously obtained in the fringe pattern estimation process, which is useful in many FPP applications. Experimental results have demonstrated the improved performance of the proposed algorithm over the conventional single-shot FPP approaches.

Robust Fringe Projection Profilometry via Sparse Representation.

In this paper, a robust fringe projection profilometry (FPP) algorithm using the sparse dictionary learning and sparse coding techniques is proposed. When reconstructing the 3D model of objects, traditional FPP systems often fail to perform if the captured fringe images have a complex scene, such as having multiple and occluded objects. It introduces great difficulty to the phase unwrapping process of an FPP system that can result in serious distortion in the final reconstructed 3D model. For the proposed algorithm, it encodes the period order information, which is essential to phase unwrapping, into some texture patterns and embeds them to the projected fringe patterns. When the encoded fringe image is captured, a modified morphological component analysis and a sparse classification procedure are performed to decode and identify the embedded period order information. It is then used to assist the phase unwrapping process to deal with the different artifacts in the fringe images. Experimental results show that the proposed algorithm can significantly improve the robustness of an FPP system. It performs equally well no matter the fringe images have a simple or complex scene, or are affected due to the ambient lighting of the working environment.

Inpainting for Fringe Projection Profilometry Based on Geometrically Guided Iterative Regularization.

Conventional fringe projection profilometry methods often have difficulty in reconstructing the 3D model of objects when the fringe images have the so-called highlight regions due to strong illumination from nearby light sources. Within a highlight region, the fringe pattern is often overwhelmed by the strong reflected light. Thus, the 3D information of the object, which is originally embedded in the fringe pattern, can no longer be retrieved. In this paper, a novel inpainting algorithm is proposed to restore the fringe images in the presence of highlights. The proposed method first detects the highlight regions based on a Gaussian mixture model. Then, a geometric sketch of the missing fringes is made and used as the initial guess of an iterative regularization procedure for regenerating the missing fringes. The simulation and experimental results show that the proposed algorithm can accurately reconstruct the 3D model of objects even when their fringe images have large highlight regions. It significantly outperforms the traditional approaches in both quantitative and qualitative evaluations.

Efficient blind image restoration using discrete periodic radon transform.

Restoring an image from its convolution with an unknown blur function is a well-known ill-posed problem in image processing. Many approaches have been proposed to solve the problem and they have shown to have good performance in identifying the blur function and restoring the original image. However, in actual implementation, various problems incurred due to the large data size and long computational time of these approaches are undesirable even with the current computing machines. In this paper, an efficient algorithm is proposed for blind image restoration based on the discrete periodic Radon transform (DPRT). With DPRT, the original two-dimensional blind image restoration problem is converted into one-dimensional ones, which greatly reduces the memory size and computational time required. Experimental results show that the resulting approach is faster in almost an order of magnitude as compared with the traditional approach, while the quality of the restored image is similar.