Image restoration in frequency space using complex-valued CNNs.

Real-valued convolutional neural networks (RV-CNNs) in the spatial domain have outperformed classical approaches in many image restoration tasks such as image denoising and super-resolution. Fourier analysis of the results produced by these spatial domain models reveals the limitations of these models in properly processing the full frequency spectrum. This lack of complete spectral information can result in missing textural and structural elements. To address this limitation, we explore the potential of complex-valued convolutional neural networks (CV-CNNs) for image restoration tasks. CV-CNNs have shown remarkable performance in tasks such as image classification and segmentation. However, CV-CNNs for image restoration problems in the frequency domain have not been fully investigated to address the aforementioned issues. Here, we propose several novel CV-CNN-based models equipped with complex-valued attention gates for image denoising and super-resolution in the frequency domains. We also show that our CV-CNN-based models outperform their real-valued counterparts for denoising super-resolution structured illumination microscopy (SR-SIM) and conventional image datasets. Furthermore, the experimental results show that our proposed CV-CNN-based models preserve the frequency spectrum better than their real-valued counterparts in the denoising task. Based on these findings, we conclude that CV-CNN-based methods provide a plausible and beneficial deep learning approach for image restoration in the frequency domain.

Graphene Microelectrode Arrays, 4D Structured Illumination Microscopy, and a Machine Learning Spike Sorting Algorithm Permit the Analysis of Ultrastructural Neuronal Changes During Neuronal Signaling in a Model of Niemann-Pick Disease Type C.

Simultaneously recording network activity and ultrastructural changes of the synapse is essential for advancing understanding of the basis of neuronal functions. However, the rapid millisecond-scale fluctuations in neuronal activity and the subtle sub-diffraction resolution changes of synaptic morphology pose significant challenges to this endeavor. Here, specially designed graphene microelectrode arrays (G-MEAs) are used, which are compatible with high spatial resolution imaging across various scales as well as permit high temporal resolution electrophysiological recordings to address these challenges. Furthermore, alongside G-MEAs, an easy-to-implement machine learning algorithm is developed to efficiently process the large datasets collected from MEA recordings. It is demonstrated that the combined use of G-MEAs, machine learning (ML) spike analysis, and 4D structured illumination microscopy (SIM) enables monitoring the impact of disease progression on hippocampal neurons which are treated with an intracellular cholesterol transport inhibitor mimicking Niemann-Pick disease type C (NPC), and show that synaptic boutons, compared to untreated controls, significantly increase in size, leading to a loss in neuronal signaling capacity.

Visualizing highly bright and uniform cellular ultrastructure by expansion-microscopy with tetrahedral DNA nanostructures.

Expansion Microscopy (ExM) is a widely used super-resolution technique that enables imaging of structures beyond the diffraction limit of light. However, ExM suffers from weak labeling signals and expansion distortions, limiting its applicability. Here, we present an innovative approach called Tetrahedral DNA nanostructure Expansion Microscopy (TDN-ExM), addressing these limitations by using tetrahedral DNA nanostructures (TDNs) for fluorescence labeling. Our approach demonstrates a 3- to 10-fold signal amplification due to the multivertex nature of TDNs, allowing the modification of multiple dyes. Previous studies have confirmed minimal distortion on a large scale, and our strategy can reduce the distortion at the ultrastructural level in samples because it does not rely on anchoring agents and is not affected by digestion. This results in a brighter fluorescence, better uniformity, and compatibility with different labeling strategies and optical super-resolution technologies. We validated the utility of TDN-ExM by imaging various biological structures with improved resolutions and signal-to-noise ratios.

Localized Plasmonic Structured Illumination Microscopy Using Hybrid Inverse Design.

Super-resolution fluorescence imaging has offered unprecedented insights and revolutionized our understanding of biology. In particular, localized plasmonic structured illumination microscopy (LPSIM) achieves video-rate super-resolution imaging with ∼50 nm spatial resolution by leveraging subdiffraction-limited nearfield patterns generated by plasmonic nanoantenna arrays. However, the conventional trial-and-error design process for LPSIM arrays is time-consuming and computationally intensive, limiting the exploration of optimal designs. Here, we propose a hybrid inverse design framework combining deep learning and genetic algorithms to refine LPSIM arrays. A population of designs is evaluated using a trained convolutional neural network, and a multiobjective optimization method optimizes them through iteration and evolution. Simulations demonstrate that the optimized LPSIM substrate surpasses traditional substrates, exhibiting higher reconstruction accuracy, robustness against noise, and increased tolerance for fewer measurements. This framework not only proves the efficacy of inverse design for tailoring LPSIM substrates but also opens avenues for exploring new plasmonic nanostructures in imaging applications.

Posterior approach to correct for focal plane offsets in lattice light-sheet structured illumination microscopy.

Significance: Lattice light-sheet structured illumination microscopy (latticeSIM) has proven highly effective in producing three-dimensional images with super resolution rapidly and with minimal photobleaching. However, due to the use of two separate objectives, sample-induced aberrations can result in an offset between the planes of excitation and detection, causing artifacts in the reconstructed images. Aim: We introduce a posterior approach to detect and correct the axial offset between the excitation and detection focal planes in latticeSIM and provide a method to minimize artifacts in the reconstructed images. Approach: We utilized the residual phase information within the overlap regions of the laterally shifted structured illumination microscopy information components in frequency space to retrieve the axial offset between the excitation and the detection focal planes in latticeSIM. Results: We validated our technique through simulations and experiments, encompassing a range of samples from fluorescent beads to subcellular structures of adherent cells. We also show that using transfer functions with the same axial offset as the one present during data acquisition results in reconstructed images with minimal artifacts and salvages otherwise unusable data. Conclusion: We envision that our method will be a valuable addition to restore image quality in latticeSIM datasets even for those acquired under non-ideal experimental conditions.

Metamaterial-Assisted Illumination Nanoscopy with Exceptional Axial Resolution.

Recent advancements in optical metamaterials have opened new possibilities in the exciting field of super-resolution microscopies. The far-field metamaterial-assisted illumination nanoscopies (MAINs) have, very recently, enhanced the lateral resolution to one-fifteenth of the optical wavelength. However, the axial localization accuracy of fluorophores in the MAINs remains rarely explored. Here, a MAIN with a nanometer-scale axial localization accuracy is demonstrated by monitoring the distance-dependent photobleaching dynamics of the fluorophores on top of an organic hyperbolic metamaterial (OHM) substrate under a wide-field single-objective microscope. With such a regular experimental configuration, 3D imaging of various biological samples with the resolution of ≈40 nm in the lateral dimensions and ≈5 nm in the axial dimension is realized. The demonstrated imaging modality enables the resolution of the 3D morphology of nanoscopic cellular structures with a significantly simplified experimental setup.

Surpassing light inhomogeneities in structured-illumination microscopy with FlexSIM.

Super-resolution structured-illumination microscopy (SIM) is a powerful technique that allows one to surpass the diffraction limit by up to a factor two. Yet, its practical use is hampered by its sensitivity to imaging conditions which makes it prone to reconstruction artefacts. In this work, we present FlexSIM, a flexible SIM reconstruction method capable to handle highly challenging data. Specifically, we demonstrate the ability of FlexSIM to deal with the distortion of patterns, the high level of noise encountered in live imaging, as well as out-of-focus fluorescence. Moreover, we show that FlexSIM achieves state-of-the-art performance over a variety of open SIM datasets.

Identification of z-axis filopodia in growth cones using super-resolution microscopy.

A growth cone is a highly motile tip of an extending axon that is crucial for neural network formation. Three-dimensional-structured illumination microscopy, a type of super-resolution light microscopy with a resolution that overcomes the optical diffraction limitation (ca. 200 nm) of conventional light microscopy, is well suited for studying the molecular dynamics of intracellular events. Using this technique, we discovered a novel type of filopodia distributed along the z-axis ("z-filopodia") within the growth cone. Z-filopodia were typically oriented in the direction of axon growth, not attached to the substratum, protruded spontaneously without microtubule invasion, and had a lifetime that was considerably shorter than that of conventional filopodia. Z-filopodia formation and dynamics were regulated by actin-regulatory proteins, such as vasodilator-stimulated phosphoprotein, fascin, and cofilin. Chromophore-assisted laser inactivation of cofilin induced the rapid turnover of z-filopodia. An axon guidance receptor, neuropilin-1, was concentrated in z-filopodia and was transported together with them, whereas its ligand, semaphorin-3A, was selectively bound to them. Membrane domains associated with z-filopodia were also specialized and resembled those of lipid rafts, and their behaviors were closely related to those of neuropilin-1. The results suggest that z-filopodia have unique turnover properties, and unlike xy-filopodia, do not function as force-generating structures for axon extension.

Profiling Chromosome Topological Features by Super-Resolution 3D Structured Illumination Microscopy.

Three-dimensional structured illumination microscopy (3D-SIM) and fluorescence in situ hybridization on three-dimensional preserved cells (3D-FISH) have proven to be robust and efficient methodologies for analyzing nuclear architecture and profiling the genome's topological features. These methods have allowed the simultaneous visualization and evaluation of several target structures at super-resolution. In this chapter, we focus on the application of 3D-SIM for the visualization of 3D-FISH preparations of chromosomes in interphase, known as Chromosome Territories (CTs). We provide a workflow and detailed guidelines for sample preparation, image acquisition, and image analysis to obtain quantitative measurements for profiling chromosome topological features. In parallel, we address a practical example of these protocols in the profiling of CTs 9 and 22 involved in the translocation t(9;22) in Chronic Myeloid Leukemia (CML). The profiling of chromosome topological features described in this chapter allowed us to characterize a large-scale topological disruption of CTs 9 and 22 that correlates directly with patients' response to treatment and as a possible potential change in the inheritance systems. These findings open new insights into how the genome structure is associated with the response to cancer treatments, highlighting the importance of microscopy in analyzing the topological features of the genome.

Array tomography of in vivo labeled synaptic receptors.

Array tomography (AT) allows one to localize sub-cellular components within the structural context of cells in 3D through the imaging of serial sections. Using this technique, the z-resolution can be improved physically by cutting ultra-thin sections. Nevertheless, conventional immunofluorescence staining of those sections is time consuming and requires relatively large amounts of costly antibody solutions. Moreover, epitopes are only readily accessible at the section's surface, leaving the volume of the serial sections unlabeled. Localization of receptors at neuronal synapses in 3D in their native cellular ultrastructural context is important for understanding signaling processes. Here, we present in vivo labeling of receptors via fluorophore-coupled tags in combination with super-resolution AT. We present two workflows where we label receptors at the plasma membrane: first, in vivo labeling via microinjection with a setup consisting of readily available components and self-manufactured microscope table equipment and second, live receptor labeling by using a cell-permeable tag. To take advantage of a near-to-native preservation of tissues for subsequent scanning electron microscopy (SEM), we also apply high-pressure freezing and freeze substitution. The advantages and disadvantages of our workflows are discussed.

Modulated illumination microscopy: Application perspectives in nuclear nanostructure analysis.

The structure of the cell nucleus of higher organisms has become a major topic of advanced light microscopy. So far, a variety of methods have been applied, including confocal laser scanning fluorescence microscopy, 4Pi, STED and localisation microscopy approaches, as well as different types of patterned illumination microscopy, modulated either laterally (in the object plane) or axially (along the optical axis). Based on our experience, we discuss here some application perspectives of Modulated Illumination Microscopy (MIM) and its combination with single-molecule localisation microscopy (SMLM). For example, spatially modulated illumination microscopy/SMI (illumination modulation along the optical axis) has been used to determine the axial extension (size) of small, optically isolated fluorescent objects between ≤ 200 nm and ≥ 40 nm diameter with a precision down to the few nm range; it also allows the axial positioning of such structures down to the 1 nm scale; combined with laterally structured illumination/SIM, a 3D localisation precision of ≤1 nm is expected using fluorescence yields typical for SMLM applications. Together with the nanosizing capability of SMI, this can be used to analyse macromolecular nuclear complexes with a resolution approaching that of cryoelectron microscopy.

Enhancing resolution and contrast in fibre bundle-based fluorescence microscopy using generative adversarial network.

Fibre bundle (FB)-based endoscopes are indispensable in biology and medical science due to their minimally invasive nature. However, resolution and contrast for fluorescence imaging are limited due to characteristic features of the FBs, such as low numerical aperture (NA) and individual fibre core sizes. In this study, we improved the resolution and contrast of sample fluorescence images acquired using in-house fabricated high-NA FBs by utilising generative adversarial networks (GANs). In order to train our deep learning model, we built an FB-based multifocal structured illumination microscope (MSIM) based on a digital micromirror device (DMD) which improves the resolution and the contrast substantially compared to basic FB-based fluorescence microscopes. After network training, the GAN model, employing image-to-image translation techniques, effectively transformed wide-field images into high-resolution MSIM images without the need for any additional optical hardware. The results demonstrated that GAN-generated outputs significantly enhanced both contrast and resolution compared to the original wide-field images. These findings highlight the potential of GAN-based models trained using MSIM data to enhance resolution and contrast in wide-field imaging for fibre bundle-based fluorescence microscopy. Lay Description: Fibre bundle (FB) endoscopes are essential in biology and medicine but suffer from limited resolution and contrast for fluorescence imaging. Here we improved these limitations using high-NA FBs and generative adversarial networks (GANs). We trained a GAN model with data from an FB-based multifocal structured illumination microscope (MSIM) to enhance resolution and contrast without additional optical hardware. Results showed significant enhancement in contrast and resolution, showcasing the potential of GAN-based models for fibre bundle-based fluorescence microscopy.

Correlative light and X-ray tomography jointly unveil the critical role of connexin43 channels on inflammation-induced cellular ultrastructural alterations.

Non-junctional connexin43 (Cx43) plasma membrane hemichannels have been implicated in several inflammatory diseases, particularly playing a role in ATP release that triggers activation of the inflammasome. Therapies targeting the blocking of the hemichannels to prevent the pathological release or uptake of ions and signalling molecules through its pores are of therapeutic interest. To date, there is no close-to-native, high-definition documentation of the impact of Cx43 hemichannel-mediated inflammation on cellular ultrastructure, neither is there a robust account of the ultrastructural changes that occur following treatment with selective Cx43 hemichannel blockers such as Xentry-Gap19 (XG19). A combination of same-sample correlative high-resolution three-dimensional fluorescence microscopy and soft X-ray tomography at cryogenic temperatures, enabled in the identification of novel 3D molecular interactions within the cellular milieu when comparing behaviour in healthy states and during the early onset or late stages under inflammatory conditions. Notably, our findings suggest that XG19 blockage of connexin hemichannels under pro-inflammatory conditions may be crucial in preventing the direct degradation of connexosomes by lysosomes, without affecting connexin protein translation and trafficking. We also delineated fine and gross cellular phenotypes, characteristic of inflammatory insult or road-to-recovery from inflammation, where XG19 could indirectly prevent and reverse inflammatory cytokine-induced mitochondrial swelling and cellular hypertrophy through its action on Cx43 hemichannels. Our findings suggest that XG19 might have prophylactic and therapeutic effects on the inflammatory response, in line with functional studies.

Impact of manufacturing processes on glycerolipid and polar lipid composition and ultrastructure in infant formula.

The introduction of exogenous lipids in the production of infant formula induces significant alterations in milk lipid composition, content, and membrane structure, thus affecting the lipid digestion, absorption, and utilization. This study meticulously tracks these changes throughout the manufacturing process. Pasteurization has a significant effect on phosphatidylcholine and sphingomyelin in the outer membrane, decreasing their relative contents to total polar lipids from 12.52% and 17.34% to 7.72% and 12.59%, respectively. Subsequent processes, including bactericidal-concentration and spray-drying, demonstrate the thermal stability of sphingomyelin and ceramides, while glycerolipids with arachidonic acid/docosahexaenoic acid and glycerophospholipids, particularly phosphatidylethanolamine, diminish significantly. Polar lipids addition and freeze-drying technology significantly enhance the polar lipid content and improve microscopic morphology of infant formula. These findings reveal the diverse effects of technological processes on glycerolipid and polar lipid compositions, concentration, and ultrastructure in infant formulas, thus offering crucial insights for optimizing lipid content and structure within infant formula.

Evaluation of Swin Transformer and knowledge transfer for denoising of super-resolution structured illumination microscopy data.

BACKGROUND: Convolutional neural network (CNN)-based methods have shown excellent performance in denoising and reconstruction of super-resolved structured illumination microscopy (SR-SIM) data. Therefore, CNN-based architectures have been the focus of existing studies. However, Swin Transformer, an alternative and recently proposed deep learning-based image restoration architecture, has not been fully investigated for denoising SR-SIM images. Furthermore, it has not been fully explored how well transfer learning strategies work for denoising SR-SIM images with different noise characteristics and recorded cell structures for these different types of deep learning-based methods. Currently, the scarcity of publicly available SR-SIM datasets limits the exploration of the performance and generalization capabilities of deep learning methods. RESULTS: In this work, we present SwinT-fairSIM, a novel method based on the Swin Transformer for restoring SR-SIM images with a low signal-to-noise ratio. The experimental results show that SwinT-fairSIM outperforms previous CNN-based denoising methods. Furthermore, as a second contribution, two types of transfer learning-namely, direct transfer and fine-tuning-were benchmarked in combination with SwinT-fairSIM and CNN-based methods for denoising SR-SIM data. Direct transfer did not prove to be a viable strategy, but fine-tuning produced results comparable to conventional training from scratch while saving computational time and potentially reducing the amount of training data required. As a third contribution, we publish four datasets of raw SIM images and already reconstructed SR-SIM images. These datasets cover two different types of cell structures, tubulin filaments and vesicle structures. Different noise levels are available for the tubulin filaments. CONCLUSION: The SwinT-fairSIM method is well suited for denoising SR-SIM images. By fine-tuning, already trained models can be easily adapted to different noise characteristics and cell structures. Furthermore, the provided datasets are structured in a way that the research community can readily use them for research on denoising, super-resolution, and transfer learning strategies.

Visualizing the intracellular aggregation behavior of gold nanoclusters via structured illumination microscopy and scanning transmission electron microscopy.

Given the growing concerns about nanotoxicity, numerous studies have focused on providing mechanistic insights into nanotoxicity by imaging the intracellular fate of nanoparticles. A suitable imaging strategy is necessary to uncover the intracellular behavior of nanoparticles. Although each conventional technique has its own limitations, scanning transmission electron microscopy (STEM) and three-dimensional structured illumination microscopy (3D-SIM) combine the advantages of chemical element mapping, ultrastructural analysis, and cell dynamic tracking. Gold nanoclusters (AuNCs), synthesized using 6-aza-2 thiothymine (ATT) and L-arginine (Arg) as reducing and protecting ligands, referred to as Arg@ATT-AuNCs, have been widely used in biological sensing and imaging, medicine, and catalyst yield. Based on their intrinsic fluorescence and high electron density, Arg@ATT-AuNCs were selected as a model. STEM imaging showed that both the single-particle and aggregated states of Arg@ATT-AuNCs were compartmentally distributed within a single cell. Real-time 3D-SIM imaging showed that the fluorescent Arg@ATT-AuNCs gradually aggregated after being located in the lysosomes of living cells, causing lysosomal damage. The aggregate formation of Arg@ATT-AuNCs was triggered by the low-pH medium, particularly in the lysosomal acidic environment. The proposed dual imaging strategy was verified using other types of AuNCs, which is valuable for studying nano-cell interactions and any associated cytotoxicity, and has the potential to be a useful approach for exploring the interaction of cells with various nanoparticles.

Localization of the Remnant of a Cell Wall Sorting Signal and Its Interaction with a Sensor Kinase.

Sortases are highly conserved enzymes with endopeptidase and transpeptidase activities in Gram-positive bacteria. Sortase A cleaves within an LPXTG-motif and covalently crosslinks cell wall proteins to become anchored to the peptidoglycan of the cell wall. We showed that a peptide cleaved by sortase A from the C-terminus (C-pep) of the LPXTG-adhesin SspA intercalates in the cell membrane. Nested in the membrane, this C-pep docks with the intramembrane sensor histidine kinase, SraS, to activate the response regulator, SraR. SraR signals that the C-pep has been cleaved as an indicator of the fidelity of sortase A processing. SraSR also signals that key LPXTG-proteins in concert with lipoteichoic acid engage the mucin, MUC5B, which elicits a different transcriptional response than the binding of other salivary constituents. To visualize the C-pep intercalating in the cell membrane in vivo, we used Structured Illumination Microscopy (SIM). And to show that the C-pep complexes with SraS, we used bimolecular fluorescence experiments. The C-pep and SraS were each expressed with one or the other half of yellow fluorescence protein (YFP). Reconstitution of the complete YFP signal indicated that the C-pep and SraS interacted at molecular distances within the cell membrane in vivo. Using these imaging protocols, we learned that the C-pep functions as a signaling molecule within the cell membrane of the streptococcal cell.

Dynamics of mitochondrial membranes under photo-oxidative stress with high spatiotemporal resolution.

In our study, we harnessed an original Enhanced Speed Structured Illumination Microscopy (Fast-SIM) imaging setup to explore the dynamics of mitochondrial and inner membrane ultrastructure under specific photo-oxidation stress induced by Chlorin-e6 and light irradiation. Notably, our Fast-SIM system allowed us to observe and quantify a distinct remodeling and shortening of the mitochondrial structure after 60-80 s of irradiation. These changes were accompanied by fusion events of adjacent inner membrane cristae and global swelling of the organelle. Preceding these alterations, a larger sequence was characterized by heightened dynamics within the mitochondrial network, featuring events such as mitochondrial fission, rapid formation of tubular prolongations, and fluctuations in cristae structure. Our findings provide compelling evidence that, among enhanced-resolution microscopy techniques, Fast-SIM emerges as the most suitable approach for non-invasive dynamic studies of mitochondrial structure in living cells. For the first time, this approach allows quantitative and qualitative characterization of successive steps in the photo-induced oxidation process with sufficient spatial and temporal resolution.

Correlative Cryo-imaging Using Soft X-Ray Tomography for the Study of Virus Biology in Cells and Tissues.

Viruses are obligate intracellular pathogens that depend on their host cell machinery and metabolism for their replicative life cycle. Virus entry, replication, and assembly are dynamic processes that lead to the reorganisation of host cell components. Therefore, a complete understanding of the viral processes requires their study in the cellular context where advanced imaging has been proven valuable in providing the necessary information. Among the available imaging techniques, soft X-ray tomography (SXT) at cryogenic temperatures can provide three-dimensional mapping to 25 nm resolution and is ideally suited to visualise the internal organisation of virus-infected cells. In this chapter, the principles and practices of synchrotron-based cryo-soft X-ray tomography (cryo-SXT) in virus research are presented. The potential of the cryo-SXT in correlative microscopy platforms is also demonstrated through working examples of reovirus and hepatitis research at Beamline B24 (Diamond Light Source Synchrotron, UK) and BL09-Mistral beamline (ALBA Synchrotron, Spain), respectively.

Enhanced synaptic protein visualization by multicolor super-resolution expansion microscopy.

Significance: Understanding the organization of biomolecules into complexes and their dynamics is crucial for comprehending cellular functions and dysfunctions, particularly in neuronal networks connected by synapses. In the last two decades, various powerful super-resolution (SR) microscopy techniques have been developed that produced stunning images of synapses and their molecular organization. However, current SR microscopy methods do not permit multicolor fluorescence imaging with 20 to 30 nm spatial resolution. Aim: We developed a method that enables 4-color fluorescence imaging of synaptic proteins in neurons with 20 to 30 nm lateral resolution. Approach: We used post-expansion immunolabeling of eightfold expanded hippocampal neurons in combination with Airyscan and structured illumination microscopy (SIM). Results: We demonstrate that post-expansion immunolabeling of approximately eightfold expanded hippocampal neurons enables efficient labeling of synaptic proteins in crowded compartments with minimal linkage error and enables in combination with Airyscan and SIM four-color three-dimensional fluorescence imaging with 20 to 30 nm lateral resolution. Using immunolabeling of Synaptobrevin 2 as an efficient marker of the vesicle pool allowed us to identify individual synaptic vesicles colocalized with Rab3-interacting molecule 1 and 2 (RIM1/2), a marker of pre-synaptic fusion sites. Conclusions: Our optimized expansion microscopy approach improves the visualization and location of pre- and post-synaptic proteins and can thus provide invaluable insights into the spatial organization of proteins at synapses.