Lens-free motion analysis via neuromorphic laser speckle imaging.

Laser speckle imaging (LSI) is a powerful tool for motion analysis owing to the high sensitivity of laser speckles. Traditional LSI techniques rely on identifying changes from the sequential intensity speckle patterns, where each pixel performs synchronous measurements. However, a lot of redundant data of the static speckles without motion information in the scene will also be recorded, resulting in considerable resources consumption for data processing and storage. Moreover, the motion cues are inevitably lost during the "blind" time interval between successive frames. To tackle such challenges, we propose neuromorphic laser speckle imaging (NLSI) as an efficient alternative approach for motion analysis. Our method preserves the motion information while excluding the redundant data by exploring the use of the neuromorphic event sensor, which acquires only the relevant information of the moving parts and responds asynchronously with a much higher sampling rate. This neuromorphic data acquisition mechanism captures fast-moving objects on the order of microseconds. In the proposed NLSI method, the moving object is illuminated using a coherent light source, and the reflected high frequency laser speckle patterns are captured with a bare neuromorphic event sensor. We present the data processing strategy to analyze motion from event-based laser speckles, and the experimental results demonstrate the feasibility of our method at different motion speeds.

Event-based laser speckle correlation for micro motion estimation: erratum.

We present an erratum to our Letter [Opt. Lett.46, 3885 (2021)OPLEDP0146-959210.1364/OL.430419]. This erratum corrects an inadvertent error in Eq. (4). The corrections have no influence on the results and conclusions of the original Letter.

High-speed laser-scanning biological microscopy using FACED.

Laser scanning is used in advanced biological microscopy to deliver superior imaging contrast, resolution and sensitivity. However, it is challenging to scale up the scanning speed required for interrogating a large and heterogeneous population of biological specimens or capturing highly dynamic biological processes at high spatiotemporal resolution. Bypassing the speed limitation of traditional mechanical methods, free-space angular-chirp-enhanced delay (FACED) is an all-optical, passive and reconfigurable laser-scanning approach that has been successfully applied in different microscopy modalities at an ultrafast line-scan rate of 1-80 MHz. Optimal FACED imaging performance requires optimized experimental design and implementation to enable specific high-speed applications. In this protocol, we aim to disseminate information allowing FACED to be applied to a broader range of imaging modalities. We provide (i) a comprehensive guide and design specifications for the FACED hardware; (ii) step-by-step optical implementations of the FACED module including the key custom components; and (iii) the overall image acquisition and reconstruction pipeline. We illustrate two practical imaging configurations: multimodal FACED imaging flow cytometry (bright-field, fluorescence and second-harmonic generation) and kHz 2D two-photon fluorescence microscopy. Users with basic experience in optical microscope operation and software engineering should be able to complete the setup of the FACED imaging hardware and software in ~2-3 months.

Event-based laser speckle correlation for micro motion estimation.

Micro motion estimation has important applications in various fields such as microfluidic particle detection and biomedical cell imaging. Conventional methods analyze the motion from intensity images captured using frame-based imaging sensors such as the complementary metal-oxide semiconductor (CMOS) and the charge-coupled device (CCD). Recently, event-based sensors have evolved with the special capability to record asynchronous light changes with high dynamic range, high temporal resolution, low latency, and no motion blur. In this Letter, we explore the potential of using the event sensor to estimate the micro motion based on the laser speckle correlation technique.

High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction.

We consider the problem of high-dimensional light field reconstruction and develop a learning-based framework for spatial and angular super-resolution. Many current approaches either require disparity clues or restore the spatial and angular details separately. Such methods have difficulties with non-Lambertian surfaces or occlusions. In contrast, we formulate light field super-resolution (LFSR) as tensor restoration and develop a learning framework based on a two-stage restoration with 4-dimensional (4D) convolution. This allows our model to learn the features capturing the geometry information encoded in multiple adjacent views. Such geometric features vary near the occlusion regions and indicate the foreground object border. To train a feasible network, we propose a novel normalization operation based on a group of views in the feature maps, design a stage-wise loss function, and develop the multi-range training strategy to further improve the performance. Evaluations are conducted on a number of light field datasets including real-world scenes, synthetic data, and microscope light fields. The proposed method achieves superior performance and less execution time comparing with other state-of-the-art schemes.

Deep-learning-assisted biophysical imaging cytometry at massive throughput delineates cell population heterogeneity.

The association of the intrinsic optical and biophysical properties of cells to homeostasis and pathogenesis has long been acknowledged. Defining these label-free cellular features obviates the need for costly and time-consuming labelling protocols that perturb the living cells. However, wide-ranging applicability of such label-free cell-based assays requires sufficient throughput, statistical power and sensitivity that are unattainable with current technologies. To close this gap, we present a large-scale, integrative imaging flow cytometry platform and strategy that allows hierarchical analysis of intrinsic morphological descriptors of single-cell optical and mass density within a population of millions of cells. The optofluidic cytometry system also enables the synchronous single-cell acquisition of and correlation with fluorescently labeled biochemical markers. Combined with deep neural network and transfer learning, this massive single-cell profiling strategy demonstrates the label-free power to delineate the biophysical signatures of the cancer subtypes, to detect rare populations of cells in the heterogeneous samples (10-5), and to assess the efficacy of targeted therapeutics. This technique could spearhead the development of optofluidic imaging cell-based assays that stratify the underlying physiological and pathological processes based on the information-rich biophysical cellular phenotypes.

RedCap: residual encoder-decoder capsule network for holographic image reconstruction.

A capsule network, as an advanced technique in deep learning, is designed to overcome information loss in the pooling operation and internal data representation of a convolutional neural network (CNN). It has shown promising results in several applications, such as digit recognition and image segmentation. In this work, we investigate for the first time the use of capsule network in digital holographic reconstruction. The proposed residual encoder-decoder capsule network, which we call RedCap, uses a novel windowed spatial dynamic routing algorithm and residual capsule block, which extends the idea of a residual block. Compared with the CNN-based neural network, RedCap exhibits much better experimental results in digital holographic reconstruction, while having a dramatic 75% reduction in the number of parameters. It indicates that RedCap is more efficient in the way it processes data and requires a much less memory storage for the learned model, which therefore makes it possible to be applied to some challenging situations with limited computational resources, such as portable devices.

PARC: ultrafast and accurate clustering of phenotypic data of millions of single cells.

MOTIVATION: New single-cell technologies continue to fuel the explosive growth in the scale of heterogeneous single-cell data. However, existing computational methods are inadequately scalable to large datasets and therefore cannot uncover the complex cellular heterogeneity. RESULTS: We introduce a highly scalable graph-based clustering algorithm PARC-Phenotyping by Accelerated Refined Community-partitioning-for large-scale, high-dimensional single-cell data (>1 million cells). Using large single-cell flow and mass cytometry, RNA-seq and imaging-based biophysical data, we demonstrate that PARC consistently outperforms state-of-the-art clustering algorithms without subsampling of cells, including Phenograph, FlowSOM and Flock, in terms of both speed and ability to robustly detect rare cell populations. For example, PARC can cluster a single-cell dataset of 1.1 million cells within 13 min, compared with >2 h for the next fastest graph-clustering algorithm. Our work presents a scalable algorithm to cope with increasingly large-scale single-cell analysis. AVAILABILITY AND IMPLEMENTATION: https://github.com/ShobiStassen/PARC. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

A Real-Time Coprime Line Scan Super-Resolution System for Ultra-Fast Microscopy.

A fundamental technical challenge for ultra-fast cell microscopy is the tradeoff between imaging throughput and resolution. In addition to throughput, real-time applications such as image-based cell sorting further requires ultra-low imaging latency to facilitate rapid decision making on a single-cell level. Using a novel coprime line scan sampling scheme, a real-time low-latency hardware super-resolution system for ultra-fast time-stretch microscopy is presented. The proposed scheme utilizes analog-to-digital converter with a carefully tuned sampling pattern (shifted sampling grid) to enable super-resolution image reconstruction using line scan input from an optical front-end. A fully pipelined FPGA-based system is built to efficiently handle the real-time high-resolution image reconstruction process with the input subpixel samples while achieving minimal output latency. The proposed super-resolution sampling and reconstruction scheme is parametrizable and is readily applicable to different line scan imaging systems. In our experiments, an imaging latency of 0.29 µs has been achieved based on a pixel-stream throughput of 4.123 giga pixels per second, which translates into imaging throughput of approximately 120000 cells per second.

Quantitative Phase Imaging Flow Cytometry for Ultra-Large-Scale Single-Cell Biophysical Phenotyping.

Cellular biophysical properties are the effective label-free phenotypes indicative of differences in cell types, states, and functions. However, current biophysical phenotyping methods largely lack the throughput and specificity required in the majority of cell-based assays that involve large-scale single-cell characterization for inquiring the inherently complex heterogeneity in many biological systems. Further confounded by the lack of reported robust reproducibility and quality control, widespread adoption of single-cell biophysical phenotyping in mainstream cytometry remains elusive. To address this challenge, here we present a label-free imaging flow cytometer built upon a recently developed ultrafast quantitative phase imaging (QPI) technique, coined multi-ATOM, that enables label-free single-cell QPI, from which a multitude of subcellularly resolvable biophysical phenotypes can be parametrized, at an experimentally recorded throughput of >10,000 cells/s-a capability that is otherwise inaccessible in current QPI. With the aim to translate multi-ATOM into mainstream cytometry, we report robust system calibration and validation (from image acquisition to phenotyping reproducibility) and thus demonstrate its ability to establish high-dimensional single-cell biophysical phenotypic profiles at ultra-large-scale (>1,000,000 cells). Such a combination of throughput and content offers sufficiently high label-free statistical power to classify multiple human leukemic cell types at high accuracy (~92-97%). This system could substantiate the significance of high-throughput QPI flow cytometry in enabling next frontier in large-scale image-derived single-cell analysis applied in biological discovery and cost-effective clinical diagnostics. © 2019 International Society for Advancement of Cytometry.

Computational image speckle suppression using block matching and machine learning.

We develop an image despeckling method that combines nonlocal self-similarity filters with machine learning, which makes use of convolutional neural network (CNN) denoisers. It consists of three major steps: block matching, CNN despeckling, and group shrinkage. Through the use of block matching, we can take advantage of the similarity across image patches as a regularizer to augment the performance of data-driven denoising using a pre-trained network. The outputs from the CNN denoiser and the group coordinates from block matching are further used to form 3D groups of similar patches, which are then filtered through a wavelet-domain shrinkage. The experimental results show that the proposed method achieves noticeable improvement compared with state-of-the-art speckle suppression techniques in both visual inspection and objective assessments.

Multi-ATOM: Ultrahigh-throughput single-cell quantitative phase imaging with subcellular resolution.

A growing body of evidence has substantiated the significance of quantitative phase imaging (QPI) in enabling cost-effective and label-free cellular assays, which provides useful insights into understanding the biophysical properties of cells and their roles in cellular functions. However, available QPI modalities are limited by the loss of imaging resolution at high throughput and thus run short of sufficient statistical power at the single-cell precision to define cell identities in a large and heterogeneous population of cells-hindering their utility in mainstream biomedicine and biology. Here we present a new QPI modality, coined multiplexed asymmetric-detection time-stretch optical microscopy (multi-ATOM) that captures and processes quantitative label-free single-cell images at ultrahigh throughput without compromising subcellular resolution. We show that multi-ATOM, based upon ultrafast phase-gradient encoding, outperforms state-of-the-art QPI in permitting robust phase retrieval at a QPI throughput of >10 000 cell/sec, bypassing the need for interferometry which inevitably compromises QPI quality under ultrafast operation. We employ multi-ATOM for large-scale, label-free, multivariate, cell-type classification (e.g. breast cancer subtypes, and leukemic cells vs peripheral blood mononuclear cells) at high accuracy (>94%). Our results suggest that multi-ATOM could empower new strategies in large-scale biophysical single-cell analysis with applications in biology and enriching disease diagnostics.

High-Throughput Line Buffer Microarchitecture for Arbitrary Sized Streaming Image Processing.

Parallel hardware designed for image processing promotes vision-guided intelligent applications. With the advantages of high-throughput and low-latency, streaming architecture on FPGA is especially attractive to real-time image processing. Notably, many real-world applications, such as region of interest (ROI) detection, demand the ability to process images continuously at different sizes and resolutions in hardware without interruptions. FPGA is especially suitable for implementation of such flexible streaming architecture, but most existing solutions require run-time reconfiguration, and hence cannot achieve seamless image size-switching. In this paper, we propose a dynamically-programmable buffer architecture (D-SWIM) based on the Stream-Windowing Interleaved Memory (SWIM) architecture to realize image processing on FPGA for image streams at arbitrary sizes defined at run time. D-SWIM redefines the way that on-chip memory is organized and controlled, and the hardware adapts to arbitrary image size with sub-100 ns delay that ensures minimum interruptions to the image processing at a high frame rate. Compared to the prior SWIM buffer for high-throughput scenarios, D-SWIM achieved dynamic programmability with only a slight overhead on logic resource usage, but saved up to 56 % of the BRAM resource. The D-SWIM buffer achieves a max operating frequency of 329.5 MHz and reduction in power consumption by 45.7 % comparing with the SWIM scheme. Real-world image processing applications, such as 2D-Convolution and the Harris Corner Detector, have also been used to evaluate D-SWIM's performance, where a pixel throughput of 4.5 Giga Pixel/s and 4.2 Giga Pixel/s were achieved respectively in each case. Compared to the implementation with prior streaming frameworks, the D-SWIM-based design not only realizes seamless image size-switching, but also improves hardware efficiency up to 30 × .

All-passive pixel super-resolution of time-stretch imaging.

Based on image encoding in a serial-temporal format, optical time-stretch imaging entails a stringent requirement of state-of-the-art fast data acquisition unit in order to preserve high image resolution at an ultrahigh frame rate - hampering the widespread utilities of such technology. Here, we propose a pixel super-resolution (pixel-SR) technique tailored for time-stretch imaging that preserves pixel resolution at a relaxed sampling rate. It harnesses the subpixel shifts between image frames inherently introduced by asynchronous digital sampling of the continuous time-stretch imaging process. Precise pixel registration is thus accomplished without any active opto-mechanical subpixel-shift control or other additional hardware. Here, we present the experimental pixel-SR image reconstruction pipeline that restores high-resolution time-stretch images of microparticles and biological cells (phytoplankton) at a relaxed sampling rate (≈2-5 GSa/s)-more than four times lower than the originally required readout rate (20 GSa/s) - is thus effective for high-throughput label-free, morphology-based cellular classification down to single-cell precision. Upon integration with the high-throughput image processing technology, this pixel-SR time-stretch imaging technique represents a cost-effective and practical solution for large scale cell-based phenotypic screening in biomedical diagnosis and machine vision for quality control in manufacturing.

High-throughput time-stretch imaging flow cytometry for multi-class classification of phytoplankton.

Time-stretch imaging has been regarded as an attractive technique for high-throughput imaging flow cytometry primarily owing to its real-time, continuous ultrafast operation. Nevertheless, two key challenges remain: (1) sufficiently high time-stretch image resolution and contrast is needed for visualizing sub-cellular complexity of single cells, and (2) the ability to unravel the heterogeneity and complexity of the highly diverse population of cells - a central problem of single-cell analysis in life sciences - is required. We here demonstrate an optofluidic time-stretch imaging flow cytometer that enables these two features, in the context of high-throughput multi-class (up to 14 classes) phytoplantkton screening and classification. Based on the comprehensive feature extraction and selection procedures, we show that the intracellular texture/morphology, which is revealed by high-resolution time-stretch imaging, plays a critical role of improving the accuracy of phytoplankton classification, as high as 94.7%, based on multi-class support vector machine (SVM). We also demonstrate that high-resolution time-stretch images, which allows exploitation of various feature domains, e.g. Fourier space, enables further sub-population identification - paving the way toward deeper learning and classification based on large-scale single-cell images. Not only applicable to biomedical diagnostic, this work is anticipated to find immediate applications in marine and biofuel research.