Universal inverse modeling of point spread functions for SMLM localization and microscope characterization.

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Depth-enhanced high-throughput microscopy by compact PSF engineering.

High-throughput microscopy is vital for screening applications, where three-dimensional (3D) cellular models play a key role. However, due to defocus susceptibility, current 3D high-throughput microscopes require axial scanning, which lowers throughput and increases photobleaching and photodamage. Point spread function (PSF) engineering is an optical method that enables various 3D imaging capabilities, yet it has not been implemented in high-throughput microscopy due to the cumbersome optical extension it typically requires. Here we demonstrate compact PSF engineering in the objective lens, which allows us to enhance the imaging depth of field and, combined with deep learning, recover 3D information using single snapshots. Beyond the applications shown here, this work showcases the usefulness of high-throughput microscopy in obtaining training data for deep learning-based algorithms, applicable to a variety of microscopy modalities.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization.

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

Near index matching enables solid diffractive optical element fabrication via additive manufacturing.

Diffractive optical elements (DOEs) have a wide range of applications in optics and photonics, thanks to their capability to perform complex wavefront shaping in a compact form. However, widespread applicability of DOEs is still limited, because existing fabrication methods are cumbersome and expensive. Here, we present a simple and cost-effective fabrication approach for solid, high-performance DOEs. The method is based on conjugating two nearly refractive index-matched solidifiable transparent materials. The index matching allows for extreme scaling up of the elements in the axial dimension, which enables simple fabrication of a template using commercially available 3D printing at tens-of-micrometer resolution. We demonstrated the approach by fabricating and using DOEs serving as microlens arrays, vortex plates, including for highly sensitive applications such as vector beam generation and super-resolution microscopy using MINSTED, and phase-masks for three-dimensional single-molecule localization microscopy. Beyond the advantage of making DOEs widely accessible by drastically simplifying their production, the method also overcomes difficulties faced by existing methods in fabricating highly complex elements, such as high-order vortex plates, and spectrum-encoding phase masks for microscopy.

Fluorescence Microscopy: a statistics-optics perspective.

Fundamental properties of light unavoidably impose features on images collected using fluorescence microscopes. Modeling these features is ever more important in quantitatively interpreting microscopy images collected at scales on par or smaller than light's wavelength. Here we review the optics responsible for generating fluorescent images, fluorophore properties, microscopy modalities leveraging properties of both light and fluorophores, in addition to the necessarily probabilistic modeling tools imposed by the stochastic nature of light and measurement.

Diffractive optical system design by cascaded propagation.

Modern design of complex optical systems relies heavily on computational tools. These frequently use geometrical optics as well as Fourier optics. Fourier optics is typically used for designing thin diffractive elements, placed in the system's aperture, generating a shift-invariant Point Spread Function (PSF). A major bottleneck in applying Fourier Optics in many cases of interest, e.g. when dealing with multiple, or out-of-aperture elements, comes from numerical complexity. In this work, we propose and implement an efficient and differentiable propagation model based on the Collins integral, which enables the optimization of diffractive optical systems with unprecedented design freedom using backpropagation. We demonstrate the applicability of our method, numerically and experimentally, by engineering shift-variant PSFs via thin plate elements placed in arbitrary planes inside complex imaging systems, performing cascaded optimization of multiple planes, and designing optimal machine-vision systems by deep learning.

Quantifying cell-cycle-dependent chromatin dynamics during interphase by live 3D tracking.

The study of cell cycle progression and regulation is important to our understanding of fundamental biophysics, aging, and disease mechanisms. Local chromatin movements are generally considered to be constrained and relatively consistent during all interphase stages, although recent advances in our understanding of genome organization challenge this claim. Here, we use high spatiotemporal resolution, 4D (x, y, z and time) localization microscopy by point-spread-function (PSF) engineering and deep learning-based image analysis, for live imaging of mouse embryonic fibroblast (MEF 3T3) and MEF 3T3 double Lamin A Knockout (LmnaKO) cell lines, to characterize telomere diffusion during the interphase. We detected varying constraint levels imposed on chromatin, which are prominently decreased during G0/G1. Our 4D measurements of telomere diffusion offer an effective method to investigate chromatin dynamics and reveal cell-cycle-dependent motion constraints, which may be caused by various cellular processes.

Deep-ROCS: from speckle patterns to superior-resolved images by deep learning in rotating coherent scattering microscopy.

Rotating coherent scattering (ROCS) microscopy is a label-free imaging technique that overcomes the optical diffraction limit by adding up the scattered laser light from a sample obliquely illuminated from different angles. Although ROCS imaging achieves 150 nm spatial and 10 ms temporal resolution, simply summing different speckle patterns may cause loss of sample information. In this paper we present Deep-ROCS, a neural network-based technique that generates a superior-resolved image by efficient numerical combination of a set of differently illuminated images. We show that Deep-ROCS can reconstruct super-resolved images more accurately than conventional ROCS microscopy, retrieving high-frequency information from a small number (6) of speckle images. We demonstrate the performance of Deep-ROCS experimentally on 200 nm beads and by computer simulations, where we show its potential for even more complex structures such as a filament network.

Multiplexed PSF Engineering for Three-Dimensional Multicolor Particle Tracking.

Three-dimensional spatiotemporal tracking of microscopic particles in multiple colors is a challenging optical imaging task. Existing approaches require a trade-off between photon efficiency, field of view, mechanical complexity, spectral specificity, and speed. Here, we introduce multiplexed point-spread-function engineering that achieves photon-efficient, 3D multicolor particle tracking over a large field of view. This is accomplished by first chromatically splitting the emission path of a microscope to different channels, engineering the point-spread function of each, and then recombining them onto the same region of the camera. We demonstrate our technique for simultaneously tracking five types of emitters in vitro as well as colocalization of DNA loci in live yeast cells.

3D printable diffractive optical elements by liquid immersion.

Diffractive optical elements (DOEs) are used to shape the wavefront of incident light. This can be used to generate practically any pattern of interest, albeit with varying efficiency. A fundamental challenge associated with DOEs comes from the nanoscale-precision requirements for their fabrication. Here we demonstrate a method to controllably scale up the relevant feature dimensions of a device from tens-of-nanometers to tens-of-microns by immersing the DOEs in a near-index-matched solution. This makes it possible to utilize modern 3D-printing technologies for fabrication, thereby significantly simplifying the production of DOEs and decreasing costs by orders of magnitude, without hindering performance. We demonstrate the tunability of our design for varying experimental conditions, and the suitability of this approach to ultrasensitive applications by localizing the 3D positions of single molecules in cells using our microscale fabricated optical element to modify the point-spread-function (PSF) of a microscope.

Learning Optimal Wavefront Shaping for Multi-Channel Imaging.

Fast acquisition of depth information is crucial for accurate 3D tracking of moving objects. Snapshot depth sensing can be achieved by wavefront coding, in which the point-spread function (PSF) is engineered to vary distinctively with scene depth by altering the detection optics. In low-light applications, such as 3D localization microscopy, the prevailing approach is to condense signal photons into a single imaging channel with phase-only wavefront modulation to achieve a high pixel-wise signal to noise ratio. Here we show that this paradigm is generally suboptimal and can be significantly improved upon by employing multi-channel wavefront coding, even in low-light applications. We demonstrate our multi-channel optimization scheme on 3D localization microscopy in densely labelled live cells where detectability is limited by overlap of modulated PSFs. At extreme densities, we show that a split-signal system, with end-to-end learned phase masks, doubles the detection rate and reaches improved precision compared to the current state-of-the-art, single-channel design. We implement our method using a bifurcated optical system, experimentally validating our approach by snapshot volumetric imaging and 3D tracking of fluorescently labelled subcellular elements in dense environments.

Microscopic scan-free surface profiling over extended axial ranges by point-spread-function engineering.

The shape of a surface, i.e., its topography, influences many functional properties of a material; hence, characterization is critical in a wide variety of applications. Two notable challenges are profiling temporally changing structures, which requires high-speed acquisition, and capturing geometries with large axial steps. Here, we leverage point-spread-function engineering for scan-free, dynamic, microsurface profiling. The presented method is robust to axial steps and acquires full fields of view at camera-limited framerates. We present two approaches for implementation: fluorescence-based and label-free surface profiling, demonstrating the applicability to a variety of sample geometries and surface types.

Publisher Correction: DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning.

An outstanding challenge in single-molecule localization microscopy is the accurate and precise localization of individual point emitters in three dimensions in densely labeled samples. One established approach for three-dimensional single-molecule localization is point-spread-function (PSF) engineering, in which the PSF is engineered to vary distinctively with emitter depth using additional optical elements. However, images of dense emitters, which are desirable for improving temporal resolution, pose a challenge for algorithmic localization of engineered PSFs, due to lateral overlap of the emitter PSFs. Here we train a neural network to localize multiple emitters with densely overlapping Tetrapod PSFs over a large axial range. We then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach experimentally with super-resolution reconstructions of mitochondria and volumetric imaging of fluorescently labeled telomeres in cells. Our approach, DeepSTORM3D, enables the study of biological processes in whole cells at timescales that are rarely explored in localization microscopy.

VIPR: vectorial implementation of phase retrieval for fast and accurate microscopic pixel-wise pupil estimation.

In microscopy, proper modeling of the image formation has a substantial effect on the precision and accuracy in localization experiments and facilitates the correction of aberrations in adaptive optics experiments. The observed images are subject to polarization effects, refractive index variations, and system specific constraints. Previously reported techniques have addressed these challenges by using complicated calibration samples, computationally heavy numerical algorithms, and various mathematical simplifications. In this work, we present a phase retrieval approach based on an analytical derivation of the vectorial diffraction model. Our method produces an accurate estimate of the system's phase information, without any prior knowledge about the aberrations, in under a minute.

Three-dimensional localization microscopy in live flowing cells.

Capturing the dynamics of live cell populations with nanoscale resolution poses a significant challenge, primarily owing to the speed-resolution trade-off of existing microscopy techniques. Flow cytometry would offer sufficient throughput, but lacks subsample detail. Here we show that imaging flow cytometry, in which the point detectors of flow cytometry are replaced with a camera to record 2D images, is compatible with 3D localization microscopy through point-spread-function engineering, which encodes the depth of the emitter into the emission pattern captured by the camera. The extraction of 3D positions from sub-cellular objects of interest is achieved by calibrating the depth-dependent response of the imaging system using fluorescent beads mixed with the sample buffer. This approach enables 4D imaging of up to tens of thousands of objects per minute and can be applied to characterize chromatin dynamics and the uptake and spatial distribution of nanoparticles in live cancer cells.

Ultrasensitive Refractometry via Supercritical Angle Fluorescence.

Refractometry, namely, the measurement of refractive index (RI), provides information about various sample properties such as concentrations and molecular structure. One physical phenomenon which enables precise determination of a sample's RI in a microscope is the supercritical-angle fluorescence. This effect is observed when the fluorescence from an emitter near a glass-medium interface is captured by an objective lens with a high numerical aperture. The materials' index mismatch creates a distinguishable transition in the intensity pattern at the back focal plane of the objective that changes proportionally to the RI of the media. Here, we present a refractometry approach in which the fluorophores are preattached to the bottom surface of a microfluidic channel, enabling highly sensitive determination of the RI using tiny amounts of liquid (picoliters). With this method, we attained a standard deviation of 3.1 × 10-5 and a repeatability of 2.7 × 10-5 RI units. We first determine the capabilities of the device for glycerol-water solutions and then demonstrate the relevance of our system for monitoring changes in biological systems. As a model system, we show that we can detect single bacteria ( Escherichia coli) and measure population growth.