Enhancing scanning electron microscopy imaging quality of weakly conductive samples through unsupervised learning.

Scanning electron microscopy (SEM) is a crucial tool for analyzing submicron-scale structures. However, the attainment of high-quality SEM images is contingent upon the high conductivity of the material due to constraints imposed by its imaging principles. For weakly conductive materials or structures induced by intrinsic properties or organic doping, the SEM imaging quality is significantly compromised, thereby impeding the accuracy of subsequent structure-related analyses. Moreover, the unavailability of paired high-low quality images in this context renders the supervised-based image processing methods ineffective in addressing this challenge. Here, an unsupervised method based on Cycle-consistent Generative Adversarial Network (CycleGAN) was proposed to enhance the quality of SEM images for weakly conductive samples. The unsupervised model can perform end-to-end learning using unpaired blurred and clear SEM images from weakly and well-conductive samples, respectively. To address the requirements of material structure analysis, an edge loss function was further introduced to recover finer details in the network-generated images. Various quantitative evaluations substantiate the efficacy of the proposed method in SEM image quality improvement with better performance than the traditional methods. Our framework broadens the application of artificial intelligence in materials analysis, holding significant implications in fields such as materials science and image restoration.

Dual-Domain Reconstruction Network Incorporating Multi-Level Wavelet Transform and Recurrent Convolution for Sparse View Computed Tomography Imaging.

Sparse view computed tomography (SVCT) aims to reduce the number of X-ray projection views required for reconstructing the cross-sectional image of an object. While SVCT significantly reduces X-ray radiation dose and speeds up scanning, insufficient projection data give rise to issues such as severe streak artifacts and blurring in reconstructed images, thereby impacting the diagnostic accuracy of CT detection. To address this challenge, a dual-domain reconstruction network incorporating multi-level wavelet transform and recurrent convolution is proposed in this paper. The dual-domain network is composed of a sinogram domain network (SDN) and an image domain network (IDN). Multi-level wavelet transform is employed in both IDN and SDN to decompose sinograms and CT images into distinct frequency components, which are then processed through separate network branches to recover detailed information within their respective frequency bands. To capture global textures, artifacts, and shallow features in sinograms and CT images, a recurrent convolution unit (RCU) based on convolutional long and short-term memory (Conv-LSTM) is designed, which can model their long-range dependencies through recurrent calculation. Additionally, a self-attention-based multi-level frequency feature normalization fusion (MFNF) block is proposed to assist in recovering high-frequency components by aggregating low-frequency components. Finally, an edge loss function based on the Laplacian of Gaussian (LoG) is designed as the regularization term for enhancing the recovery of high-frequency edge structures. The experimental results demonstrate the effectiveness of our approach in reducing artifacts and enhancing the reconstruction of intricate structural details across various sparse views and noise levels. Our method excels in both performance and robustness, as evidenced by its superior outcomes in numerous qualitative and quantitative assessments, surpassing contemporary state-of-the-art CNNs or Transformer-based reconstruction methods.

On the use of deep learning for phase recovery.

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

Single-shot image restoration via a model-enhanced network with unpaired supervision in an optical sparse aperture system.

We propose a model-enhanced network with unpaired single-shot data for solving the imaging blur problem of an optical sparse aperture (OSA) system. With only one degraded image captured from the system and one "arbitrarily" selected unpaired clear image, the cascaded neural network is iteratively trained for denoising and restoration. With the computational image degradation model enhancement, our method is able to improve contrast, restore blur, and suppress noise of degraded images in simulation and experiment. It can achieve better restoration performance with fewer priors than other algorithms. The easy selectivity of unpaired clear images and the non-strict requirement of a custom kernel make it suitable and applicable for single-shot image restoration of any OSA system.

Real-time phase imaging with physics-enhanced network and equivariance.

Learning-based phase imaging balances high fidelity and speed. However, supervised training requires unmistakable and large-scale datasets, which are often hard or impossible to obtain. Here, we propose an architecture for real-time phase imaging based on physics-enhanced network and equivariance (PEPI). The measurement consistency and equivariant consistency of physical diffraction images are used to optimize the network parameters and invert the process from a single diffraction pattern. In addition, we propose a regularization method based total variation kernel (TV-K) function constraint to output more texture details and high-frequency information. The results show that PEPI can produce the object phase quickly and accurately, and the proposed learning strategy performs closely to the fully supervised method in the evaluation function. Moreover, the PEPI solution can handle high-frequency details better than the fully supervised method. The reconstruction results validate the robustness and generalization ability of the proposed method. Specially, our results show that PEPI leads to considerable performance improvement on the imaging inverse problem, thereby paving the way for high-precision unsupervised phase imaging.

Single-shot Fresnel incoherent correlation holography via deep learning based phase-shifting technology.

Fresnel incoherent correlation holography (FINCH) realizes non-scanning three-dimension (3D) images using spatial incoherent illumination, but it requires phase-shifting technology to remove the disturbance of the DC term and twin term that appears in the reconstruction field, thus increasing the complexity of the experiment and limits the real-time performance of FINCH. Here, we propose a single-shot Fresnel incoherent correlation holography via deep learning based phase-shifting (FINCH/DLPS) method to realize rapid and high-precision image reconstruction using only a collected interferogram. A phase-shifting network is designed to implement the phase-shifting operation of FINCH. The trained network can conveniently predict two interferograms with the phase shift of 2/3 π and 4/3 π from one input interferogram. Using the conventional three-step phase-shifting algorithm, we can conveniently remove the DC term and twin term of the FINCH reconstruction and obtain high-precision reconstruction through the back propagation algorithm. The Mixed National Institute of Standards and Technology (MNIST) dataset is used to verify the feasibility of the proposed method through experiments. In the test with the MNIST dataset, the reconstruction results demonstrate that in addition to high-precision reconstruction, the proposed FINCH/DLPS method also can effectively retain the 3D information by calibrating the back propagation distance in the case of reducing the complexity of the experiment, further indicating the feasibility and superiority of the proposed FINCH/DLPS method.

Deep learning-based method for non-uniform motion-induced error reduction in dynamic microscopic 3D shape measurement.

The non-uniform motion-induced error reduction in dynamic fringe projection profilometry is complex and challenging. Recently, deep learning (DL) has been successfully applied to many complex optical problems with strong nonlinearity and exhibits excellent performance. Inspired by this, a deep learning-based method is developed for non-uniform motion-induced error reduction by taking advantage of the powerful ability of nonlinear fitting. First, a specially designed dataset of motion-induced error reduction is generated for network training by incorporating complex nonlinearity. Then, the corresponding DL-based architecture is proposed and it contains two parts: in the first part, a fringe compensation module is developed as network pre-processing to reduce the phase error caused by fringe discontinuity; in the second part, a deep neural network is employed to extract the high-level features of error distribution and establish a pixel-wise hidden nonlinear mapping between the phase with motion-induced error and the ideal one. Both simulations and real experiments demonstrate the feasibility of the proposed method in dynamic macroscopic measurement.

PredictionNet: a long short-term memory-based attention network for atmospheric turbulence prediction in adaptive optics.

Adaptive optics (AO) has great applications in many fields and has attracted wide attention from researchers. However, both traditional and deep learning-based AO methods have inherent time delay caused by wavefront sensors and controllers, leading to the inability to truly achieve real-time atmospheric turbulence correction. Hence, future turbulent wavefront prediction plays a particularly important role in AO. Facing the challenge of accurately predicting stochastic turbulence, we combine the convolutional neural network with a turbulence correction time series model and propose a long short-term memory attention-based network, named PredictionNet, to achieve real-time AO correction. Especially, PredictionNet takes the spatiotemporal coupling characteristics of turbulence wavefront into consideration and can improve the accuracy of prediction effectively. The combination of the numerical simulation by a professional software package and the real turbulence experiment by digital holography demonstrates in detail that PredictionNet is more accurate and more stable than traditional methods. Furthermore, the result compared with AO without prediction confirms that predictive AO with PredictionNet is useful.

Registration of multi-modal images under a complex background combining multiscale features extraction and semantic segmentation.

Multi-modal imaging technology has a very broad application value in target recognition and other fields, and image registration is one of its key technologies. In this paper, a multi-modal image registration algorithm that combines multiscale features extraction and semantic segmentation is proposed to achieve accurate registration of polarized images and near-infrared images under complex backgrounds. A classical convolutional neural network ResNet is employed to capture the robust feature descriptors, and a convolutional neural network with an attention mechanism is trained to filter out the irrelevant feature points. Further, the two multi-modal images can be further registered. The experimental results show the feasibility and effectiveness of the proposed method.

Simultaneous measurement of near-water-film air temperature and humidity fields based on dual-wavelength digital holographic interferometry.

We present a method to reconstruct the near-water-film air temperature and humidity distributions synchronously by measuring the phase delays based on dual-wavelength digital holographic interferometry. A falling water film device was used to create a water film evaporation environment and generate axially uniform temperature and humidity fields. The relationship between air temperature, humidity and phase delay is derived from the Edlen equations. With such relationship, the temperature and humidity distributions can be solved directly according to phase delays of two different wavelengths. An edge phase enhancement method and an error elimination method with PSO are presented to improve the measurement accuracy. The temperature and humidity fields in the falling water film model were experimentally reconstructed with temperature deviation of 0.06% and relative humidity deviation of 2.61%.

Light-field focusing and modulation through scattering media based on dual-polarization-encoded digital optical phase conjugation.

Digital optical phase conjugation (DOPC) can be applied for light-field focusing and imaging through or within scattering media. Traditional DOPC only recovers the phase but loses the polarization information of the original incident beam. In this Letter, we propose a dual-polarization-encoded DOPC to recover the full information (both phase and polarization) of the incident beam. The phase distributions of two orthogonal polarization components of the speckle field coming from a multimode fiber are first measured by using digital holography. Then, the phase distributions are separately modulated on two beams and their conjugations are superposed to recover the incident beam through the fiber. By changing the phase difference or amplitude ratio between the two conjugate beams, light fields with complex polarization distribution can also be generated. This method will broaden the application scope of DOPC in imaging through scattering media.

Dual-wavelength surface plasmon resonance holographic microscopy for simultaneous measurements of cell-substrate distance and cytoplasm refractive index.

Studying the basic characteristics of living cells is of great significance in biological research. Bio-physical parameters, including cell-substrate distance and cytoplasm refractive index (RI), can be used to reveal cellular properties. In this Letter, we propose a dual-wavelength surface plasmon resonance holographic microscopy (SPRHM) to simultaneously measure the cell-substrate distance and cytoplasm RI of live cells in a wide-field and non-intrusive manner. Phase-contrast surface plasmon resonance (SPR) images of individual cells at wavelengths of 632.8 nm and 690 nm are obtained using an optical system. The two-dimensional distributions of cell-substrate distance and cytoplasm RI are then demodulated from the phase-contrast SPR images of the cells. MDA-MB-231 cells and IDG-SW3 cells are experimentally measured to verify the feasibility of this approach. Our method provides a useful tool in biological fields for dual-parameter detection and characterization of live cells.

Optical tweezers integrated surface plasmon resonance holographic microscopy for characterizing cell-substrate interactions under noninvasive optical force stimuli.

The rapid development of bio-mechanical research increases the significance of studying cell behaviors near the substrate under the force stimuli in a real-time manner. Here, we present an optical tweezers (OT) integrated surface plasmon resonance holographic microscopy (SPRHM) to realize the dynamical and in-situ characterizations of cell-substrate interactions with noninvasive optical force stimulations. Using the OT integrated SPRHM (OT-SPRHM), we dynamically manipulate the living cells by OT, and simultaneously, the phase-contrast surface plasmon resonance images of the living cells are obtained and the cell-substrate distance is determined via SPRHM. We show that OT-SPRHM has the advanced capabilities of measuring the optical force and its tiny variations applied to the K562 cells near the substrate. Also, we for the first time reveal the manipulation of the MC3T3-E1 cells by OT. Demonstrating its robustness, this technique provides a powerful tool to explore the responses of various biological specimens to the force stimuli along the cell-substrate interface in the bio-sensing area.

Analysis of the effects of magnetic levitation to simulate microgravity environment on the Arp2/3 complex pathway in macrophage.

With dwindling natural resources on earth, current and future generations will need to explore space to new planets that will require travel under no or varying gravity conditions. Hence, long-term space missions and anticipated impacts on human biology such as changes in immune function are of growing research interest. Here, we reported new findings on mechanisms of immune response to microgravity with a focus on macrophage as a cellular model. We employed a superconducting magnet to generate a simulated microgravity environment and evaluated the effects of simulated microgravity on RAW 264.7 mouse macrophage cell line in three time frames: 8, 24, and 48 h. As study endpoints, we measured cell viability, phagocytosis, and used next-generation sequencing to explore its changing mechanism. Macrophage cell viability and phagocytosis both showed a marked decrease under microgravity. The differentially expressed genes (DEG) were identified in two ways: (1) gravity-dependent DEG, compared µg samples and 1 g samples at each time point; (2) time-dependent DEG, compared time-point samples within the same gravitational field. Through transcriptome analysis and confirmed by molecular biological verification, our findings firstly suggest that microgravity might affect macrophage phagocytosis by targeting Arp2/3 complex involved cytoskeleton synthesis and causing macrophage immune dysfunction. Our findings contribute to an emerging body of scholarship on biological effects of space travel.

Dual-channel illumination surface plasmon resonance holographic microscopy for resolution improvement.

Surface plasmon resonance holographic microscopy (SPRHM), combining digital holographic microscopy with surface plasmon resonance (SPR), can simultaneously obtain the amplitude and phase distributions of the reflected beam carrying specimen information in SPR. Due to the decaying length of the surface plasmon wave as large as tens of micrometers, the spatial resolution of SPRHM is lower than that of ordinary optical microscopes. In this work, we propose a scheme to improve the spatial resolution of SPRHM by applying dual-channel SPR excitations. Through the polarization multiplexing technique, two holograms carrying the information of SPR excited in orthogonal directions are simultaneously acquired. Via a numerical reconstruction and filtering algorithm for holograms, the lateral spatial resolution of SPRHM can be effectively enhanced to reach nearly 1 µm at a wavelength of 632.8 nm. This is comparable to the resolution of traditional optical microscopes, while possessing the advantages of wide-field imaging and high measurement sensitivity of SPR.

Sparse-view imaging of a fiber internal structure in holographic diffraction tomography via a convolutional neural network.

Deep learning has recently shown great potential in computational imaging. Here, we propose a deep-learning-based reconstruction method to realize the sparse-view imaging of a fiber internal structure in holographic diffraction tomography. By taking the sparse-view sinogram as the input and the cross-section image obtained by the dense-view sinogram as the ground truth, the neural network can reconstruct the cross-section image from the sparse-view sinogram. It performs better than the corresponding filtered back-projection algorithm with a sparse-view sinogram, both in the case of simulated data and real experimental data.

Real-time and wide-field mapping of cell-substrate adhesion gap and its evolution via surface plasmon resonance holographic microscopy.

As one of the most common biological phenomena, cell adhesion plays a vital role in the cellular activities such as the growth and apoptosis, attracting tremendous research interests over the past decades. Taking the cell evolution under drug injection as an example, the dynamics of cell-substrate adhesion gap can provide valuable information in the fundamental research of cell contacts. A robust technique of monitoring the cell adhesion gap and its evolution in real time is highly desired. Herein, we develop a surface plasmon resonance holographic microscopy to achieve the novel functionality of real-time and wide-field mapping of the cell-substrate adhesion gap and its evolution in situ. The cell adhesion gap images of mouse osteoblast cells and human breast cancer cells have been effectively extracted in a dynamic and label-free manner. The proposed technique opens up a new avenue of revealing the cell-substrate interaction mechanism and renders the wide applications in the biosensing area.

Y4-Net: a deep learning solution to one-shot dual-wavelength digital holographic reconstruction.

In this Letter, a deep learning solution (Y4-Net, four output channels network) to one-shot dual-wavelength digital holography is proposed to simultaneously reconstruct the complex amplitude information of both wavelengths from a single digital hologram with high efficiency. In the meantime, by using single-wavelength results as network ground truth to train the Y4-Net, the challenging spectral overlapping problem in common-path situations is solved with high accuracy.

Classification of cell morphology with quantitative phase microscopy and machine learning.

We describe and compare two machine learning approaches for cell classification based on label-free quantitative phase imaging with transport of intensity equation methods. In one approach, we design a multilevel integrated machine learning classifier including various individual models such as artificial neural network, extreme learning machine and generalized logistic regression. In another approach, we apply a pretrained convolutional neural network using transfer learning for the classification. As a validation, we show the performances of both approaches on classification between macrophages cultured in normal gravity and microgravity with quantitative phase imaging. The multilevel integrated classifier achieves average accuracy 93.1%, which is comparable to the average accuracy 93.5% obtained by convolutional neural network. The presented quantitative phase imaging system with two classification approaches could be helpful to biomedical scientists for easy and accurate cell analysis.

Structured light beams created through a multimode fiber via virtual Fourier filtering based on digital optical phase conjugation.

Digital optical phase conjugation (DOPC) is a newly developed technique in wavefront shaping to control light propagation through complex media. Currently, DOPC has been demonstrated for the reconstruction of two- and three-dimensional targets and enabled important applications in many areas. Nevertheless, the reconstruction results are only phase conjugated to the original input targets. Herein, we demonstrate that DOPC could be further developed for creating structured light beams through a multimode fiber (MMF). By applying annular filtering in the virtual Fourier domain of the acquired speckle field, we realize the creation of the quasi-Bessel and donut beams through the MMF. In principle, arbitrary amplitude and/or phase circular symmetry filtering could be performed in the Fourier domain, thus generating the corresponding point spread functions. We expect that the reported technique can be useful for super-resolution endoscopic imaging and optical manipulation through MMFs.