Passive imaging through inhomogeneous scattering media.

According to the atmospheric scattering model (ASM), the object signal's attenuation diminishes exponentially as the imaging distance increases. This imposes limitations on ASM-based methods in situations where the scattering medium one wish to look through is inhomogeneous. Here, we extend ASM by taking into account the spatial variation of the medium density, and propose a two-step method for imaging through inhomogeneous scattering media. In the first step, the proposed method eliminates the direct current component of the scattered pattern by subscribing to the estimated global distribution (background). In the second step, it eliminates the randomized components of the scattered light by using threshold truncation, followed by the histogram equalization to further enhance the contrast. Outdoor experiments were carried out to demonstrate the proposed method.

Adaptive imaging through dense dynamic scattering media using transfer learning.

Imaging through scattering media is a long-standing challenge in optical imaging, holding substantial importance in fields like biology, transportation, and remote sensing. Recent advancements in learning-based methods allow accurate and rapid imaging through optically thick scattering media. However, the practical application of data-driven deep learning faces substantial hurdles due to its inherent limitations in generalization, especially in scenarios such as imaging through highly non-static scattering media. Here we utilize the concept of transfer learning toward adaptive imaging through dense dynamic scattering media. Our approach specifically involves using a known segment of the imaging target to fine-tune the pre-trained de-scattering model. Since the training data of downstream tasks used for transfer learning can be acquired simultaneously with the current test data, our method can achieve clear imaging under varying scattering conditions. Experiment results show that the proposed approach (with transfer learning) is capable of providing more than 5dB improvements when optical thickness varies from 11.6 to 13.1 compared with the conventional deep learning approach (without transfer learning). Our method holds promise for applications in video surveillance and beacon guidance under dense dynamic scattering conditions.

Learning-based adaptive under-sampling for Fourier single-pixel imaging.

In this Letter, we present a learning-based method for efficient Fourier single-pixel imaging (FSI). Based on the auto-encoder, the proposed adaptive under-sampling technique (AuSamNet) manages to optimize a sampling mask and a deep neural network at the same time to achieve both under-sampling of the object image's Fourier spectrum and high-quality reconstruction from the under-sampled measurements. It is thus helpful in determining the best encoding and decoding scheme for FSI. Simulation and experiments demonstrate that AuSamNet can reconstruct high-quality natural color images even when the sampling ratio is as low as 7.5%. The proposed adaptive under-sampling strategy can be used for other computational imaging modalities, such as tomography and ptychography. We have released our source code.

DeepSCI: scalable speckle correlation imaging using physics-enhanced deep learning.

In this Letter we present a physics-enhanced deep learning approach for speckle correlation imaging (SCI), i.e., DeepSCI. DeepSCI incorporates the theoretical model of SCI into both the training and test stages of a neural network to achieve interpretable data preprocessing and model-driven fine-tuning, allowing the full use of data and physics priors. It can accurately reconstruct the image from the speckle pattern and is highly scalable to both medium perturbations and domain shifts. Our experimental results demonstrate the suitability and effectiveness of DeepSCI for solving the problem of limited generalization generally encountered in data-driven approaches.

Dammann gratings-based truly parallel optical matrix multiplication accelerator.

Matrix multiplication (MM) is a fundamental operation in various scientific and engineering computations, as well as in artificial intelligence algorithms. Efficient implementation of MM is crucial for speeding up numerous applications. Photonics presents an opportunity for efficient acceleration of dense matrix computation, owing to its intrinsic advantages, such as huge parallelism, low latency, and low power consumption. However, most optical matrix computing architectures have been limited to realizing single-channel vector-matrix multiplication or using complex configurations to expand the number of channels, which does not fully exploit the parallelism of optics. In this study, we propose a novel, to the best of our knowledge, scheme for the implementation of large-scale two-dimensional optical MM with truly massive parallelism based on a specially designed Dammann grating. We demonstrate a sequence of MMs of 50 pairs of randomly generated 4 × 8 and 8 × 4 matrices in our proof-of-principle experiment. The results indicate that the mean relative error is approximately 0.048, thereby demonstrating optical robustness and high accuracy.

Multifunctional Metasurfaces Enabled by Multifold Geometric Phase Interference.

Geometric phase is frequently used in artificially designed metasurfaces; it is typically used only once in reported works, leading to conjugate responses of two spins. Supercells containing multiple nanoantennas can break this limitation by introducing more degrees of freedom to generate new modulation capabilities. Here, we provide a method for constructing supercells for geometric phases using triple rotations, each of which achieves a specific modulation function. The physical meaning of each rotation is revealed by stepwise superposition. Based on this idea, spin-selective holography, nanoprinting, and their hybrid displays are demonstrated. As a typical application, we have designed a metalens that enables spin-selective transmission, allowing for high-quality imaging with only one spin state, which can serve as a plug-and-play chiral detection device. Finally, we analyzed how the size of supercells and the phase distribution inside it can affect the higher order diffraction, which may help in designing supercells for different scenarios.

Approximating the uncertainty of deep learning reconstruction predictions in single-pixel imaging.

Single-pixel imaging (SPI) has the advantages of high-speed acquisition over a broad wavelength range and system compactness. Deep learning (DL) is a powerful tool that can achieve higher image quality than conventional reconstruction approaches. Here, we propose a Bayesian convolutional neural network (BCNN) to approximate the uncertainty of the DL predictions in SPI. Each pixel in the predicted image represents a probability distribution rather than an image intensity value, indicating the uncertainty of the prediction. We show that the BCNN uncertainty predictions are correlated to the reconstruction errors. When the BCNN is trained and used in practical applications where the ground truths are unknown, the level of the predicted uncertainty can help to determine whether system, data, or network adjustments are needed. Overall, the proposed BCNN can provide a reliable tool to indicate the confidence levels of DL predictions as well as the quality of the model and dataset for many applications of SPI.

Arbitrary Jones matrix on-demand design in metasurfaces using multiple meta-atoms.

Super cells or multi-layer metasurfaces are used to realize various multi-functional and exotic functional devices. In such methods, the design space expands exponentially as more variable parameters are introduced; however, this will necessitate huge computational effort without special treatment. The function of a metasurface can be described mathematically by using a Jones matrix. When the gap between adjacent atoms is sufficiently large, the overall Jones matrix of a 3D lattice which is composed of multiple meta-atoms can be obtained by adding or multiplying each meta-atom's Jones matrix for a parallel or cascaded arrangement, respectively. Reversely, an arbitrary Jones matrix can be decomposed to achieve a combination of diagonal and rotation matrices. This means that the devices with various functions can be constructed by combining, cascading, and rotating a kind of atom, and thus the computation requirements will be reduced significantly. In this work, the feasibility of this approach is demonstrated with two cases, circular polarization selective transmission and resemble optical activity. Both the simulation and experiment are consistent with the hypothesis. This method can manipulate all degrees of freedom in a Jones matrix and reduce design complexity and may find applications to extend the scope of meta-optics.

Optical random micro-phase-shift DropConnect in a diffractive deep neural network.

The formulation and training of unitary neural networks is the basis of an active modulation diffractive deep neural network. In this Letter, an optical random phase DropConnect is implemented on an optical weight to manipulate a jillion of optical connections in the form of massively parallel sub-networks, in which a micro-phase assumed as an essential ingredient is drilled into Bernoulli holes to enable training convergence, and malposed deflections of the geometrical phase ray are reformulated constantly in epochs, allowing for enhancement of statistical inference. Optically, the random micro-phase-shift acts like a random phase sparse griddle with respect to values and positions, and is operated in the optical path of a projective imaging system. We investigate the performance of the full-drilling and part-drilling phenomena. In general, random micro-phase-shift part-drilling outperforms its full-drilling counterpart both in the training and inference since there are more possible recombinations of geometrical ray deflections induced by random phase DropConnect.

Single-shot noninvasive imaging through scattering medium under white-light illumination.

We experimentally investigate image reconstruction through a scattering medium under white-light illumination. To solve the inverse problem of noninvasive scattering imaging, a modified iterative algorithm is employed with an interpretable constraint on the optical transfer function (OTF). As a result, a sparse and real object can be retrieved whether it is illuminated with a narrowband or broadband light. Compared with the well-known speckle correlation technique (SCT), the proposed method requires no restrictions on the speckle autocorrelation and shows a potential advantage in scattering imaging.

Far-field super-resolution ghost imaging with a deep neural network constraint.

Ghost imaging (GI) facilitates image acquisition under low-light conditions by single-pixel measurements and thus has great potential in applications in various fields ranging from biomedical imaging to remote sensing. However, GI usually requires a large amount of single-pixel samplings in order to reconstruct a high-resolution image, imposing a practical limit for its applications. Here we propose a far-field super-resolution GI technique that incorporates the physical model for GI image formation into a deep neural network. The resulting hybrid neural network does not need to pre-train on any dataset, and allows the reconstruction of a far-field image with the resolution beyond the diffraction limit. Furthermore, the physical model imposes a constraint to the network output, making it effectively interpretable. We experimentally demonstrate the proposed GI technique by imaging a flying drone, and show that it outperforms some other widespread GI techniques in terms of both spatial resolution and sampling ratio. We believe that this study provides a new framework for GI, and paves a way for its practical applications.

Non-line-of-sight imaging under white-light illumination: a two-step deep learning approach.

Non-line-of-sight (NLOS) imaging has received considerable attentions for its ability to recover occluded objects from an indirect view. Various NLOS imaging techniques have been demonstrated recently. Here, we propose a white-light NLOS imaging method that is equipped only with an ordinary camera, and not necessary to operate under active coherent illumination as in other existing NLOS systems. The central idea is to incorporate speckle correlation-based model into a deep neural network (DNN), and form a two-step DNN strategy that endeavors to learn the optimization of the scattered pattern autocorrelation and object image reconstruction, respectively. Optical experiments are carried out to demonstrate the proposed method.

Optical random phase dropout in a diffractive deep neural network.

Unitary learning is a backpropagation (BP) method that serves to update unitary weights in fully connected deep complex-valued neural networks, meeting a prior unitary in an active modulation diffractive deep neural network. However, the square matrix characteristic of unitary weights in each layer results in its learning belonging to a small-sample training, which produces an almost useless network that has a fairly poor generalization capability. To alleviate such a serious over-fitting problem, in this Letter, optical random phase dropout is formulated and designed. The equivalence between unitary forward and diffractive networks deduces a synthetic mask that is seamlessly compounded with a computational modulation and a random sampling comb called dropout. The dropout is filled with random phases in its zero positions that satisfy the Bernoulli distribution, which could slightly deflect parts of transmitted optical rays in each output end to generate statistical inference networks. The enhancement of generalization benefits from the fact that massively parallel full connection with different optical links is involved in the training. The random phase comb introduced into unitary BP is in the form of conjugation, which indicates the significance of optical BP.

Two-step training deep learning framework for computational imaging without physics priors.

Deep learning (DL) is a powerful tool in computational imaging for many applications. A common strategy is to use a preprocessor to reconstruct a preliminary image as the input to a neural network to achieve an optimized image. Usually, the preprocessor incorporates knowledge of the physics priors in the imaging model. One outstanding challenge, however, is errors that arise from imperfections in the assumed model. Model mismatches degrade the quality of the preliminary image and therefore affect the DL predictions. Another main challenge is that many imaging inverse problems are ill-posed and the networks are over-parameterized; DL networks have flexibility to extract features from the data that are not directly related to the imaging model. This can lead to suboptimal training and poorer image reconstruction results. To solve these challenges, a two-step training DL (TST-DL) framework is proposed for computational imaging without physics priors. First, a single fully-connected layer (FCL) is trained to directly learn the inverse model with the raw measurement data as the inputs and the images as the outputs. Then, this pre-trained FCL is fixed and concatenated with an un-trained deep convolutional network with a U-Net architecture for a second-step training to optimize the output image. This approach has the advantage that does not rely on an accurate representation of the imaging physics since the first-step training directly learns the inverse model. Furthermore, the TST-DL approach mitigates network over-parameterization by separately training the FCL and U-Net. We demonstrate this framework using a linear single-pixel camera imaging model. The results are quantitatively compared with those from other frameworks. The TST-DL approach is shown to perform comparable to approaches which incorporate perfect knowledge of the imaging model, to be robust to noise and model ill-posedness, and to be more robust to model mismatch than approaches which incorporate imperfect knowledge of the imaging model. Furthermore, TST-DL yields better results than end-to-end training while suffering from less overfitting. Overall, this TST-DL framework is a flexible approach for image reconstruction without physics priors, applicable to diverse computational imaging systems.

Single-shot three-dimensional imaging with a scattering layer [Invited].

In this paper, we propose a single-shot three-dimensional imaging technique. This is achieved by simply placing a normal thin scattering layer in front of a two-dimensional image sensor, making it a light-field-like camera. The working principle of the proposed technique is based on the statistical independence and spatial ergodicity of the speckle produced by the scattering layer. Thus, the local point responses of the scattering layer should be measured in advance and are used for image reconstruction. We demonstrate the proposed method with proof-of-concept experiments and analyze the factors that affect its performance.

Recovery of the topological charge of a vortex beam propagated through a scattering layer.

Coherent vortex beams have shown great potential in many applications including information transmission under non-ideal conditions, as information can be encoded in the orbital angular momentum. However, inhomogeneity of atmosphere tends to scramble the vortex structure and give rise to speckle. It is therefore of great interest to reconstruct the topological charge of a vortex beam after it propagates through a scattering medium. Here, we propose a feasible solution for this. The proposed method measures holographically the scattered field and reconstructs the spiral phase from it by taking advantage of both the deterministic nature and the ergodicity of the scattering process. Our preliminary experiments show promising results and suggest that the proposed method can have great potential in information transmission under non-ideal conditions.

Sparse regularization-based reconstruction for 3D flame chemiluminescence tomography.

Flame chemiluminescence tomography (FCT) is a non-intrusive method that is based on using cameras to measure projections, and it plays a crucial role in combustion diagnostics and measurement. Mathematically, the inversion problem is ill-posed, and in the case of limited optical accessibility in practical applications, it is rank deficient. Therefore, the solution process should ideally be supported by prior information, which can be based on the known physics. In this work, the total variation (TV) regularization has been combined with the well-known algebraic reconstruction technique (ART) for practical FCT applications. The TV method endorses smoothness while also preserving typical flame features such as the flame front. Split Bregman iteration has been adopted for TV minimization. Five different noise conditions and the chosen regularization parameter have been tested in numerical studies. Additionally, for the 12 perspectives, an experimental FCT system is demonstrated, which is utilized to recover the three-dimensional (3D) chemiluminescence distribution of candle flames. Both the numerical and experimental studies show that the typical line artifacts that appear with the conventional ART algorithm when recovering the continuous chemiluminescence field of the flames are significantly reduced with the proposed algorithm.

Phase imaging with an untrained neural network.

Most of the neural networks proposed so far for computational imaging (CI) in optics employ a supervised training strategy, and thus need a large training set to optimize their weights and biases. Setting aside the requirements of environmental and system stability during many hours of data acquisition, in many practical applications, it is unlikely to be possible to obtain sufficient numbers of ground-truth images for training. Here, we propose to overcome this limitation by incorporating into a conventional deep neural network a complete physical model that represents the process of image formation. The most significant advantage of the resulting physics-enhanced deep neural network (PhysenNet) is that it can be used without training beforehand, thus eliminating the need for tens of thousands of labeled data. We take single-beam phase imaging as an example for demonstration. We experimentally show that one needs only to feed PhysenNet a single diffraction pattern of a phase object, and it can automatically optimize the network and eventually produce the object phase through the interplay between the neural network and the physical model. This opens up a new paradigm of neural network design, in which the concept of incorporating a physical model into a neural network can be generalized to solve many other CI problems.

Chosen-plaintext attack on the double random polarization encryption.

Images can be optically encrypted by random encoding in the phase, the polarization, or even the coherence of a light field. It is important for these optical encryption methods to undergo rigorous cryptanalysis. However, only phase-encoding-based encryption has been rigorously analyzed to date. In this manuscript, we demonstrate that the double random polarization encryption (DRPolE) is vulnerable to chosen-plaintext attack (CPA). We show that the keys can be retrieved if one can choose the polarization states of two plaintext images and collect the corresponding cyphertext images. Our study reveals a serious concern regarding the DRPolE that should be addressed in the design of polarization-based optical encryption methods.

Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging.

Artificial intelligence (AI) techniques such as deep learning (DL) for computational imaging usually require to experimentally collect a large set of labeled data to train a neural network. Here we demonstrate that a practically usable neural network for computational imaging can be trained by using simulation data. We take computational ghost imaging (CGI) as an example to demonstrate this method. We develop a one-step end-to-end neural network, trained with simulation data, to reconstruct two-dimensional images directly from experimentally acquired one-dimensional bucket signals, without the need of the sequence of illumination patterns. This is in particular useful for image transmission through quasi-static scattering media as little care is needed to take to simulate the scattering process when generating the training data. We believe that the concept of training using simulation data can be used in various DL-based solvers for general computational imaging.