Robust autofocus method based on patterned active illumination and image cross-correlation analysis.

For the effectiveness of a computer-aided diagnosis system, the quality of whole-slide image (WSI) is the foundation, and a useful autofocus method is an important part of ensuring the quality of WSI. The existing autofocus methods need to balance focusing speed and focusing accuracy, and need to be optimized separately for different samples or scenes. In this paper, a robust autofocus method based on fiber bundle illumination and image normalization analysis is proposed. For various application scenes, it meets the requirements of autofocusing through active illumination, such as bright field imaging and fluorescence imaging. For different structures on samples, it ensures the autofocusing accuracy through image analysis. The experimental results imply that the autofocusing method in this paper can effectively track the change of the distance from the sample to the focal plane and significantly improve the WSI quality.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy.

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging.

Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 µm3, allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

Real-time image resolution measurement for single molecule localization microscopy.

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

ResNet-based image inpainting method for enhancing the imaging speed of single molecule localization microscopy.

Single molecule localization microscopy (SMLM) is a mainstream method in the field of super-resolution fluorescence microscopy that can achieve a spatial resolution of 20∼30 nm through a simple optical system. SMLM usually requires thousands of raw images to reconstruct a super-resolution image, and thus suffers from a slow imaging speed. Recently, several methods based on image inpainting have been developed to enhance the imaging speed of SMLM. However, these image inpainting methods may also produce erroneous local features (or called image artifacts), for example, incorrectly joined or split filaments. In this study, we use the ResNet generator, a network with strong local feature extraction capability, to replace the popularly-used U-Net generator to minimize the image artifact problem in current image inpainting methods, and develop an image inpainting method called DI-STORM. We validate our method using both simulated and experimental data, and demonstrate that DI-STORM has the best acceleration capability and produces the least artifacts in the repaired images, as compared with VDSR (the simplest CNN-based image inpainting method in SMLM) and ANNA-PALM (the best GAN-based image inpainting method in SMLM). We believe that DI-STORM could facilitate the application of deep learning-based image inpainting methods for SMLM.

Pushing the colorimetry camera-based fluorescence microscopy to low light imaging by denoising and dye combination.

Colorimetry camera-based fluorescence microscopy (CCFM) is a single-frame imaging method for observing multiple biological events simultaneously. Compared with the traditional multi-color fluorescence microscopy methods based on sequential excitation or spectral splitting, the CCFM method simplifies multi-color fluorescence imaging experiments, while keeping a high spatial resolution. However, when the level of the detected fluorescence signal decreases, the image quality, the demosaicking algorithm precision, and the discrimination of fluorescence channels on the colorimetry camera will also decrease. Thus, CCFM has a poor color resolution under a low signal level. For example, the crosstalk will be higher than 10% when the signal is less than 100 photons/pixel. To solve this problem, we developed a new algorithm that combines sCMOS noise correction with demosaicking, and a dye selection method based on the spectral response characteristics of the colorimetry camera. By combining the above two strategies, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and low light single-frame four-color fluorescence imaging was successfully performed on fixed cos-7 cells. This study expands the power of the CCFM method, and provides a simple and efficient way for various bioimaging applications in low-light conditions.

Influence of drift correction precision on super-resolution localization microscopy.

Super-resolution localization microscopy (SRLM) breaks the diffraction limit successfully and improves the resolution of optical imaging systems by nearly an order of magnitude. However, SRLM typically takes several minutes or longer to collect a sufficient number of image frames that are required for reconstructing a final super-resolution image. During this long image acquisition period, system drift should be tightly controlled to ensure the imaging quality; thus, several drift correction methods have been developed. However, it is still unclear whether the performance of these methods is able to ensure sufficient image quality in SRLM. Without a clear answer to this question, it is hard to choose a suitable drift correction method for a specific SRLM experiment. In this paper, we use both theoretical analysis and simulation to investigate the relationship among drift correction precision, localization precision, and position estimation precision. We propose a concept of relative localization precision for evaluating the effect of drift correction on imaging resolution, which would help to select an appropriate drift correction method for a specific experiment.

LuckyProfiler: an ImageJ plug-in capable of quantifying FWHM resolution easily and effectively for super-resolution images.

Quantifying the resolution of a super-resolution image is vital for biologists trying to apply super-resolution microscopy in various research fields. Among the reported image resolution estimation methods, the one that calculates the full width at half maximum (FWHM) of line profile, called FWHM resolution, continues the traditional resolution criteria and has been popularly used by many researchers. However, quantifying the FWHM resolution of a super-resolution image is a time-consuming, labor-intensive, and error-prone process because this method typically involves a manual and careful selection of one or several of the smallest structures. In this paper, we investigate the influencing factors in FWHM resolution quantification systematically and present an ImageJ plug-in called LuckyProfiler for biologists so that they can have an easy and effective way of quantifying the FWHM resolution of super-resolution images.

PCIe-based FPGA-GPU heterogeneous computation for real-time multi-emitter fitting in super-resolution localization microscopy.

Real-time multi-emitter fitting is a key technology for advancing super-resolution localization microscopy (SRLM), especially when it is necessary to achieve dynamic imaging quality control and/or optimization of experimental conditions. However, with the increase of activation densities, the requirements in the computing resources would increase rapidly due to the complexity of the fitting algorithms, making it difficult to realize real-time multi-emitter fitting for emitter density more than 0.6 mol/µm2 in large field of view (FOV), even after acceleration with the popular Graphics Processing Unit (GPU) computation. Here we adopt the task parallelism strategy in computer science to construct a Peripheral Component Interconnect Express (PCIe) based all-in-one heterogeneous computing platform (AIO-HCP), where the data between two major parallel computing hardware, Field Programmable Gate Array (FPGA) and GPU, are interacted directly and executed simultaneously. Using simulated and experimental data, we verify that AIO-HCP could achieve a data throughput of up to ∼ 1.561 GB/s between FPGA and GPU. With this new platform, we develop a multi-emitter fitting method, called AIO-STORM, under big data stream parallel scheduling. We show that AIO-STORM is capable of providing real-time image processing on raw images with 100 µm × 100 µm FOV, 10 ms exposure time and 5.5 mol/µm2 structure density, without scarifying image quality. This study overcomes the data throughput limitation of heterogeneous devices, demonstrates the power of the PCIe-based heterogeneous computation platform, and offers opportunities for multi-scale stitching of super-resolution images.

Single-shot multi-color fluorescence microscopy via a colorimetry camera.

Multi-color fluorescence microscopy presents highly detailed biological samples interactively. However, current multi-color methods suffer from an intricate optical setup, complicated image analysis, or a long acquisition time. To address these issues, here we develop a simple multi-color method based on a customized colorimetry camera to enable the detection of multiple structures from single-shot acquisition. The unfiltered channel (W pixels) and color channels (R, G, B, and NIR pixels) in this customized camera simultaneously provide a broad detection wavelength range and high detection sensitivity. We built a simple optical setup by replacing the monochrome camera in a basic fluorescence microscopy system with a colorimetry camera, and developed effective image analysis procedures to reconstruct a multi-color image from a single frame of a raw image. We demonstrated single-shot four-color wide-field fluorescence imaging on fixed cos-7 cells with < 5% cross talk, which is comparable to the best reported values. Our method greatly simplifies both the optical system and image analysis in the widely used method of multi-color fluorescence microscopy, thus offering an effective and easy way to study multiple objects at the same time.

A labeling strategy with effective preservation of fluorophores for expansion single-molecule localization microscopy (Ex-SMLM).

Expansion microscopy (ExM) significantly improves the resolution of conventional diffraction-limited optical microscopy by using physically expanding biological samples. Combining ExM with single-molecule localization microscopy (SMLM) could further enhance the resolving power of SMLM, which is typically in the order of 20-30 nm. However, to make this combination successful, we need to solve three key issues related to sample preparation, including mainly hydrogel shrinking in an ionic photoswitching buffer, fluorescence photobleaching due to a free-radical reaction and reduced labelling efficiency from protease digestion. Re-embedding polyacrylamide gel or using an improved photoswitching buffer with a low ionic strength is able to minimize or even solve the hydrogel shrinking problem, while the development of post-expansion labelling approaches avoids fluorescence bleaching. However, the preservation of protein epitopes (which determines the labelling efficiency) remains to be challenging. In this paper, we propose to tackle this challenge by introducing the highly selective and stable biotin-streptavidin interaction into the post-expansion labelling strategy. After upgrading the popular immunolabelling linkage scheme from Epitope-Primary antibody-Secondary antibody-Fluorophores to Epitope-Primary antibody-Secondary antibody-Biotin-Streptavidin-Fluorophores, we were able to label protein epitopes with biotin, which was stable during the expansion process, and thus avoid the troublesome problem in preserving protein epitopes or antibodies. We demonstrate that combining Ex-SMLM with the new post-expansion linkage scheme enables new possibilities in resolving the detailed arrangement of Nup133 proteins in the nuclear pore complex, which helps researchers to observe a clearer structure. This study provides new opportunities for studying the ultrastructural details of subcellular organelles or even biomacromolecules, using the conventional SMLM system.

Accelerating multi-emitter localization in super-resolution localization microscopy with FPGA-GPU cooperative computation.

The real-time multi-emitter localization method is essential for advancing high-throughput super-resolution localization microscopy (HT-SRLM). In the past decade, the graphics processing unit (GPU) computation has been dominantly used to accelerate the execution speed of the multi-emitter localization method. However, if HT-SRLM is combined with a scientific complementary metal-oxide-semiconductor (sCMOS) camera working at full frame rate, real-time image processing is still difficult to achieve using this acceleration approach, thus resulting in a massive data storage challenge and even system crash. Here we take advantage of the cooperative acceleration power of field programming gate array (FPGA) computation and GPU computation, and propose a method called HCP-STORM to enable real-time multi-emitter localization. Using simulated images, we verified that HCP-STORM is capable of providing real-time image processing for raw images from a representative Hamamatsu Flash 4 V3 sCMOS camera working at full frame rate (that is, 2048×2048 pixels @ 10 ms exposure time). Using experimental images, we prove that HCP-STORM is 25 times faster than QC-STORM and 295 times faster than ThunderSTORM, with a small but acceptable degradation in image quality. This study shows the potential of FPGA-GPU cooperative computation in accelerating multi-emitter localization, and pushes a significant step toward the maturity of HT-SRLM technology.

High-precision 3D drift correction with differential phase contrast images.

Single molecule localization microscopy (SMLM) usually requires long image acquisition time at the order of minutes and thus suffers from sample drift, which deteriorates image quality. A drift estimation method with high precision is typically used in SMLM, which can be further combined with a drift compensation device to enable active microscope stabilization. Among all the reported methods, the drift estimation method based on bright-field image correlation requires no extra sample preparation or complicated modification to the imaging setup. However, the performance of this method is limited by the contrast of bright-field images, especially for the structures without sufficient features. In this paper, we proposed to use differential phase contrast (DPC) microscopy to enhance the image contrast and presented a 3D drift correction method with higher precision and robustness. This DPC-based drift correction method is suitable even for biological samples without clear morphological features. We demonstrated that this method can achieve a correction precision of < 6 nm in both the lateral direction and axial direction. Using SMLM imaging of microtubules, we verified that this method provides a comparable drift estimation performance as redundant cross-correlation.

Two-color super-resolution localization microscopy via joint encoding of emitter location and color.

Multi-color super-resolution localization microscopy (SRLM) provides great opportunities for studying the structural and functional details of biological samples. However, current multi-color SRLM methods either suffer from medium to high crosstalk, or require a dedicated optical system and a complicated image analysis procedure. To address these problems, here we propose a completely different method to realize multi-color SRLM. This method is built upon a customized RGBW camera with a repeated pattern of filtered (Red, Green, Blue and Near-infrared) and unfiltered (White) pixels. With a new insight that RGBW camera is advantageous for color recognition instead of color reproduction, we developed a joint encoding scheme of emitter location and color. By combing this RGBW camera with the joint encoding scheme and a simple optical set-up, we demonstrated two-color SRLM with ∼20 nm resolution and < 2% crosstalk (which is comparable to the best-reported values). This study significantly reduces the complexity of two-color SRLM (and potentially multi-color SRLM), and thus offers good opportunities for general biomedical research laboratories to use multi-color SRLM, which is currently mastered only by well-trained researchers.

Computational framework for generating large panoramic super-resolution images from localization microscopy.

Combining super-resolution localization microscopy with pathology creates new opportunities for biomedical researches. This combination requires a suitable image mosaic method for generating a panoramic image from many overlapping super-resolution images. However, current image mosaic methods are not suitable for this purpose. Here we proposed a computational framework and developed an image mosaic method called NanoStitcher. We generated ground truth datasets and defined criteria to evaluate this computational framework. We used both simulated and experimental datasets to prove that NanoStitcher exhibits better performance than two representative image mosaic methods. This study is helpful for the mature of super-resolution digital pathology.

Characterizing and correcting camera noise in back-illuminated sCMOS cameras.

With promising properties of fast imaging speed, large field-of-view, relative low cost and many others, back-illuminated sCMOS cameras have been receiving intensive attention for low light level imaging in the past several years. However, due to the pixel-to-pixel difference of camera noise (called noise non-uniformity) in sCMOS cameras, researchers may hesitate to use them in some application fields, and sometimes wonder whether they should optimize the noise non-uniformity of their sCMOS cameras before using them in a specific application scenario. In this paper, we systematically characterize the impact of different types of sCMOS noise on image quality and perform corrections to these types of sCMOS noise using three representative algorithms (PURE, NCS and MLEsCMOS). We verify that it is possible to use appropriate correction methods to push the non-uniformity of major types of camera noise, including readout noise, offset, and photon response, to a satisfactory level for conventional microscopy and single molecule localization microscopy. We further find out that, after these corrections, global read noise becomes a major concern that limits the imaging performance of back-illuminated sCMOS cameras. We believe this study provides new insights into the understanding of camera noise in back-illuminated sCMOS cameras, and also provides useful information for future development of this promising camera technology.

Localization of the WD repeat-containing protein 5 to the Virion Assembly Compartment Facilitates Human Cytomegalovirus Assembly.

We previously reported that human cytomegalovirus (HCMV) utilizes the cellular protein WD repeat-containing protein 5 (WDR5) to facilitate capsid nuclear egress. Here, we further show that HCMV infection results in WDR5 localization in a juxtanuclear region, and that its localization to this cellular site is associated with viral replication and late viral gene expression. Furthermore, WDR5 accumulated in the virion assembly compartment (vAC) and co-localized with vAC markers of gamma-tubulin (γ-tubulin), early endosomes, and viral vAC marker proteins pp65, pp28, and glycoprotein B (gB). WDR5 co-immunoprecipitated with multiple virion proteins, including MCP, pp150, pp65, pIRS1, and pTRS1, which may explain WDR5 accumulation in the vAC during infection. WDR5 fractionated with virions either in the presence or absence of Triton X-100 and was present in purified viral particles, suggesting that WDR5 was incorporated into HCMV virions. Thus, WDR5 localized to the vAC and was incorporated into virions, raising the possibility that in addition to capsid nuclear egress, WDR5 could also participate in cytoplasmic HCMV virion morphogenesis.Importance Human cytomegalovirus (HCMV) has a large (∼235-kb) genome that contains over 170 ORFs and exploits numerous cellular factors to facilitate its replication. In the late phase of HCMV infection cytoplasmic membranes are reorganized to establish the virion assembly compartment (vAC), which has been shown to necessary for efficient assembly of progeny virions. We previously reported that WDR5 facilitates HCMV nuclear egress. Here, we show that WDR5 is localized to the vAC and incorporated into virions, perhaps contributing to efficient virion maturation. Thus, findings in this study identified a potential role for WDR5 in HCMV assembly in the cytoplasmic phase of virion morphogenesis.

Rethinking resolution estimation in fluorescence microscopy: from theoretical resolution criteria to super-resolution microscopy.

Resolution is undoubtedly the most important parameter in optical microscopy by providing an estimation on the maximum resolving power of a certain optical microscope. For centuries, the resolution of an optical microscope is generally considered to be limited only by the numerical aperture of the optical system and the wavelength of light. However, since the invention and popularity of various advanced fluorescence microscopy techniques, especially super-resolution fluorescence microscopy, many new methods have been proposed for estimating the resolution, leading to confusions for researchers who need to quantify the resolution of their fluorescence microscopes. In this paper, we firstly summarize the early concepts and criteria for predicting the resolution limit of an ideal optical system. Then, we discuss some important influence factors that deteriorate the resolution of a certain fluorescence microscope. Finally, we provide methods and examples on how to measure the resolution of a fluorescence microscope from captured fluorescence images. This paper aims to answer as best as possible the theoretical and practical issues regarding the resolution estimation in fluorescence microscopy.

[Research progress of non-coding RNA in regulating the function of T cells in asthma].

Bronchial asthma (i.e. asthma) is a chronic inflammatory disease characterized by airway inflammatory response, hyperresponsiveness and airway remodeling, in which T cells play a vital role, especially T helper cells (Th cells). Non-coding RNAs (ncRNAs) are the RNAs that do not encode proteins, mainly including microRNAs (miRNAs), long non-coding RNAs (lncRNAs), and circular RNAs (circRNAs), which are widely found in eukaryotic genomes and participate in the regulation of various biological processes. Previous studies have shown that ncRNAs play an important role in the activation and transformation of T cells and other biological processes in asthma. The specific molecular mechanism and clinical application are worth in-depth discussion. This article reviewed the research progress in regulation of miRNAs, lncRNAs and circRNAs on T cells in asthma in recent years.

N-terminal pro-brain natriuretic peptide used for screening hemodynamically significant patent ductus arteriosus in very low birth weight infants: How and when?

BACKGROUND: N-terminal pro-B-type natriuretic peptide (NTproBNP) appears to be a useful tool for diagnosing hemodynamically significant patent ductus arteriosus (hsPDA) in preterm infants. However, a consensus for its application has not been reached. OBJECTIVE: The present study aims to evaluate the role of NTproBNP in predicting hsPDA in preterm infants, and explore the optimal cutoff value and testing-time. METHODS: A prospective blind study of 120 preterm infants with birth weights of < 1,500 g was conducted at the NICU of Peking University Shenzhen Hospital. Blood samples were successively collected on the first three days after birth for NTproBNP analysis. Echocardiographies were performed on day three of life to confirm the status of the ductus arteriosus. A receiver operating characteristic curve (ROC) analysis was performed to determine the ability of NTproBNP to recognize hsPDA. RESULTS: NTproBNP was significantly higher in infants with hsPDA, than in infants in the control group, on both day two (P < 0.001) and day three (P < 0.001). On day two, a NTproBNP cutoff value of 3,689.0 pmol/L offered an optimal predictive value for hsPDA, while on day three, the optimal cut-off value for hsPDA was 2,331.5 pmol/L. The investigators proposes day three of life (48-72 hours) as the optimal testing time. CONCLUSION: The NTproBNP biomarker during the early neonatal period can be a useful tool for screening and assessing hsPDA in premature infants, especially on day three of life.