Boosting microscopic object detection via feature activation map guided poisson blending.

Microscopic examination of visible components based on micrographs is the gold standard for testing in biomedical research and clinical diagnosis. The application of object detection technology in bioimages not only improves the efficiency of the analyst but also provides decision support to ensure the objectivity and consistency of diagnosis. However, the lack of large annotated datasets is a significant impediment in rapidly deploying object detection models for microscopic formed elements detection. Standard augmentation methods used in object detection are not appropriate because they are prone to destroy the original micro-morphological information to produce counterintuitive micrographs, which is not conducive to build the trust of analysts in the intelligent system. Here, we propose a feature activation map-guided boosting mechanism dedicated to microscopic object detection to improve data efficiency. Our results show that the boosting mechanism provides solid gains in the object detection model deployed for microscopic formed elements detection. After image augmentation, the mean Average Precision (mAP) of baseline and strong baseline of the Chinese herbal medicine micrograph dataset are increased by 16.3% and 5.8% respectively. Similarly, on the urine sediment dataset, the boosting mechanism resulted in an improvement of 8.0% and 2.6% in mAP of the baseline and strong baseline maps respectively. Moreover, the method shows strong generalizability and can be easily integrated into any main-stream object detection model. The performance enhancement is interpretable, making it more suitable for microscopic biomedical applications.

Augmented Super-Resolution Radial Fluctuations (aSRRF) Pushing the Limits of Structured Illumination Microscopy.

Structured illumination microscopy (SIM) is a versatile super-resolution technique known for its compatibility with a wide range of probes and fast implementation. While 3D SIM is capable of achieving a spatial resolution of ∼120 nm laterally and ∼300 nm axially, attempting to further enhance the resolution through methods such as nonlinear SIM or 4-beam SIM introduces complexities in optical configurations, increased phototoxicity, and reduced temporal resolution. Here, we have developed a novel method that combines SIM with augmented super-resolution radial fluctuations (aSRRF) utilizing a single image through image augmentation. By applying aSRRF reconstruction to SIM images, we can enhance the SIM resolution to ∼50 nm isotopically, without requiring any modifications to the optical system or sample acquisition process. Additionaly, we have incorporated the aSRRF approach into an ImageJ plugin and demonstrated its versatility across various fluorescence microscopy images, showcasing a remarkable two-fold resolution increase.

A deep learning based method for automatic analysis of high-throughput droplet digital PCR images.

Droplet digital PCR (ddPCR) is a technique for absolute quantification of nucleic acid molecules and is widely used in biomedical research and clinical diagnosis. ddPCR partitions the reaction solution containing target molecules into a large number of independent microdroplets for amplification and performs quantitative analysis of target molecules by calculating the proportion of positive droplets by the principle of Poisson distribution. Accurate recognition of positive droplets in ddPCR images is of great importance to guarantee the accuracy of target nucleic acid quantitative analysis. However, hand-designed operators are sensitive to interference and have disadvantages such as low contrast, uneven illumination, low sample copy number, and noise, and their accuracy and robustness still need to be improved. Herein, we developed a deep learning-based high-throughput ddPCR droplet detection framework for robust and accurate ddPCR image analysis, and the experimental results show that our method achieves excellent performance in the recognition of positive droplets (99.71%) within a limited time. By combining the Hough transform and a convolutional neural network (CNN), our novel method can automatically filter out invalid droplets that are difficult to be identified by local or global encoding methods and realize high-precision localization and classification of droplets in ddPCR images under variable exposure, contrast, and uneven illumination conditions without the need for image pre-processing and normalization processes.

Deep learning-enabled fast DNA-PAINT imaging in cells.

DNA-based point accumulation in nanoscale topography (DNA-PAINT) is a well-established technique for single-molecule localization microscopy (SMLM), enabling resolution of up to a few nanometers. Traditionally, DNA-PAINT involves the utilization of tens of thousands of single-molecule fluorescent images to generate a single super-resolution image. This process can be time-consuming, which makes it unfeasible for many researchers. Here, we propose a simplified DNA-PAINT labeling method and a deep learning-enabled fast DNA-PAINT imaging strategy for subcellular structures, such as microtubules. By employing our method, super-resolution reconstruction can be achieved with only one-tenth of the raw data previously needed, along with the option of acquiring the widefield image. As a result, DNA-PAINT imaging is significantly accelerated, making it more accessible to a wider range of biological researchers.

Imaging Single-Vesicle Exocytosis with Total Internal Reflection Fluorescence Microscopy (TIRFM).

Total internal reflection fluorescence microscopy (TIRFM) provides extremely thin optical sectioning with excellent signal-to-noise ratios, which allows for visualization of membrane dynamics at the cell surface with superb spatiotemporal resolution. In this chapter, TIRFM is used to record and analyze exocytosis of single glucose transporter-4 (GLUT4) containing vesicles in 3T3-L1 adipocytes.

DNA-PAINT Imaging Accelerated by Machine Learning.

DNA point accumulation in nanoscale topography (DNA-PAINT) is an easy-to-implement approach for localization-based super-resolution imaging. Conventional DNA-PAINT imaging typically requires tens of thousands of frames of raw data to reconstruct one super-resolution image, which prevents its potential application for live imaging. Here, we introduce a new DNA-PAINT labeling method that allows for imaging of microtubules with both DNA-PAINT and widefield illumination. We develop a U-Net-based neural network, namely, U-PAINT to accelerate DNA-PAINT imaging from a widefield fluorescent image and a sparse single-molecule localization image. Compared with the conventional method, U-PAINT only requires one-tenth of the original raw data, which permits fast imaging and reconstruction of super-resolution microtubules and can be adopted to analyze other SMLM datasets. We anticipate that this machine learning method enables faster and even live-cell DNA-PAINT imaging in the future.

A survey on applications of deep learning in microscopy image analysis.

Advanced microscopy enables us to acquire quantities of time-lapse images to visualize the dynamic characteristics of tissues, cells or molecules. Microscopy images typically vary in signal-to-noise ratios and include a wealth of information which require multiple parameters and time-consuming iterative algorithms for processing. Precise analysis and statistical quantification are often needed for the understanding of the biological mechanisms underlying these dynamic image sequences, which has become a big challenge in the field. As deep learning technologies develop quickly, they have been applied in bioimage processing more and more frequently. Novel deep learning models based on convolution neural networks have been developed and illustrated to achieve inspiring outcomes. This review article introduces the applications of deep learning algorithms in microscopy image analysis, which include image classification, region segmentation, object tracking and super-resolution reconstruction. We also discuss the drawbacks of existing deep learning-based methods, especially on the challenges of training datasets acquisition and evaluation, and propose the potential solutions. Furthermore, the latest development of augmented intelligent microscopy that based on deep learning technology may lead to revolution in biomedical research.

Development of fan-shaped tracker for single particle tracking.

With the development of super-resolution fluorescence microscopy, complex dynamic processes in living cells can be observed and recorded with unprecedented temporal and spatial resolution. Single particle tracking (SPT) is the most important step to explore the relationship between the spatio-temporal dynamics of subcellular molecules and their functions. Although previous studies have developed SPT algorithms to quantitatively analyze particle dynamics in cell, traditional tracking methods have poor performance when dealing with intersecting trajectories. This can be attributed to two main reasons: (a) they do not have point compensation process for overlapping objects; (b) they use inefficient motion prediction models. In this paper, we present a novel fan-shaped tracker (FsT) algorithm to reconstruct the trajectories of subcellular vesicles in living cells. We proposed a customized point compensation method for overlapping objects based on the fan-shaped motion trend of the particles. Furthermore, we validated the performance of the FsT in both simulated time-lapse movies with variable imaging quality and in real vesicle moving images. Meanwhile, we compared the performance of FsT with other five state-of-the-art tracking algorithms by using commonly defined measures. The results showed that our FsT achieves better performance in high signal-to-noise ratio conditions and in tracking of overlapping objects. We anticipate that our FsT method will have vast applications in tracking of moving objects in cell.

Deep learning enables structured illumination microscopy with low light levels and enhanced speed.

Structured illumination microscopy (SIM) surpasses the optical diffraction limit and offers a two-fold enhancement in resolution over diffraction limited microscopy. However, it requires both intense illumination and multiple acquisitions to produce a single high-resolution image. Using deep learning to augment SIM, we obtain a five-fold reduction in the number of raw images required for super-resolution SIM, and generate images under extreme low light conditions (at least 100× fewer photons). We validate the performance of deep neural networks on different cellular structures and achieve multi-color, live-cell super-resolution imaging with greatly reduced photobleaching.

Multi-color live-cell super-resolution volume imaging with multi-angle interference microscopy.

Imaging and tracking of near-surface three-dimensional volumetric nanoscale dynamic processes of live cells remains a challenging problem. In this paper, we propose a multi-color live-cell near-surface-volume super-resolution microscopy method that combines total internal reflection fluorescence structured illumination microscopy with multi-angle evanescent light illumination. We demonstrate that our approach of multi-angle interference microscopy is perfectly adapted to studying subcellular dynamics of mitochondria and microtubule architectures during cell migration.

Optogenetic control of epithelial-mesenchymal transition in cancer cells.

Epithelial-mesenchymal transition (EMT) is one of the most important mechanisms in the initiation and promotion of cancer cell metastasis. The phosphoinositide 3-kinase (PI3K) signaling pathway has been demonstrated to be involved in TGF-ß induced EMT, but the complicated TGF-ß signaling network makes it challenging to dissect the important role of PI3K on regulation of EMT process. Here, we applied optogenetic controlled PI3K module (named 'Opto-PI3K'), which based on CRY2 and the N-terminal of CIB1 (CIBN), to rapidly and reversibly control the endogenous PI3K activity in cancer cells with light. By precisely modulating the kinetics of PI3K activation, we found that E-cadherin is an important downstream target of PI3K signaling. Compared with TGF-ß treatment, Opto-PI3K had more potent effect in down-regulation of E-cadherin expression, which was demonstrated to be regulated in a light dose-dependent manner. Surprisingly, sustained PI3K activation induced partial EMT state in A549 cells that is highly reversible. Furthermore, we demonstrated that Opto-PI3K only partially mimicked TGF-ß effects on promotion of cell migration in vitro. These results reveal the importance of PI3K signaling in TGF-ß induced EMT, suggesting other TGF-ß regulated signaling pathways are necessary for the full and irreversible promotion of EMT in cancer cells. In addition, our study implicates the great promise of optogenetics in cancer research for mapping input-output relationships in oncogenic pathways.

Three-dimensional super-resolved live cell imaging through polarized multi-angle TIRF.

Measuring three-dimensional nanoscale cellular structures is challenging, especially when the structure is dynamic. Owing to the informative total internal reflection fluorescence (TIRF) imaging under varied illumination angles, multi-angle (MA) TIRF has been examined to offer a nanoscale axial and a subsecond temporal resolution. However, conventional MA-TIRF still performs badly in lateral resolution and fails to characterize the depth image in densely distributed regions. Here, we emphasize the lateral super-resolution in the MA-TIRF, exampled by simply introducing polarization modulation into the illumination procedure. Equipped with a sparsity and accelerated proximal algorithm, we examine a more precise 3D sample structure compared with previous methods, enabling live cell imaging with a temporal resolution of 2 s and recovering high-resolution mitochondria fission and fusion processes. We also shared the recovery program, which is the first open-source recovery code for MA-TIRF, to the best of our knowledge.