Merlin: A Vision Language Foundation Model for 3D Computed Tomography.

Over 85 million computed tomography (CT) scans are performed annually in the US, of which approximately one quarter focus on the abdomen. Given the current shortage of both general and specialized radiologists, there is a large impetus to use artificial intelligence to alleviate the burden of interpreting these complex imaging studies while simultaneously using the images to extract novel physiological insights. Prior state-of-the-art approaches for automated medical image interpretation leverage vision language models (VLMs) that utilize both the image and the corresponding textual radiology reports. However, current medical VLMs are generally limited to 2D images and short reports. To overcome these shortcomings for abdominal CT interpretation, we introduce Merlin - a 3D VLM that leverages both structured electronic health records (EHR) and unstructured radiology reports for pretraining without requiring additional manual annotations. We train Merlin using a high-quality clinical dataset of paired CT scans (6+ million images from 15,331 CTs), EHR diagnosis codes (1.8+ million codes), and radiology reports (6+ million tokens) for training. We comprehensively evaluate Merlin on 6 task types and 752 individual tasks. The non-adapted (off-the-shelf) tasks include zero-shot findings classification (31 findings), phenotype classification (692 phenotypes), and zero-shot cross-modal retrieval (image to findings and image to impressions), while model adapted tasks include 5-year chronic disease prediction (6 diseases), radiology report generation, and 3D semantic segmentation (20 organs). We perform internal validation on a test set of 5,137 CTs, and external validation on 7,000 clinical CTs and on two public CT datasets (VerSe, TotalSegmentator). Beyond these clinically-relevant evaluations, we assess the efficacy of various network architectures and training strategies to depict that Merlin has favorable performance to existing task-specific baselines. We derive data scaling laws to empirically assess training data needs for requisite downstream task performance. Furthermore, unlike conventional VLMs that require hundreds of GPUs for training, we perform all training on a single GPU. This computationally efficient design can help democratize foundation model training, especially for health systems with compute constraints. We plan to release our trained models, code, and dataset, pending manual removal of all protected health information.

Carbonized Polymer Dot Probe for Two-Photon Fluorescence Imaging of Lipid Droplets in Living Cells and Tissues.

As a dynamic and multifunctional organelle, lipid droplets (LDs) are essential in maintaining lipid balance and transducing biological signals. LD accumulation and catabolism are closely associated with energy metabolism and cell signaling. In order to easily trace LDs in living cells, a novel carbonized polymer dot (CPD)-based fluorescent nanoprobe is reported to serve the needs of LD-targeting imaging. This probe exhibits the advantages of excellent biocompatibility, simple preparation, good lipophilicity, and high compatibility with commercial dyes. Transient absorption spectroscopy was employed to discuss the luminescence mechanism of CPDs, and the results indicate that the excellent fluorescence property and the environment-responsive feature of our CPDs are derived from the intramolecular charge transfer (ICT) characteristics and the D-π-A structure that possibly formed in CPD. This nanoprobe is available for one-photon fluorescence (OPF) and two-photon fluorescence (TPF) imaging and is also practicable for staining LDs in living/fixed cells and lipids in tissue sections. The staining process is completed within several seconds, with no washing step. The intracellular LDs involving the intranuclear LDs (nLDs) can be selectively lit up. This probe is feasible for visualizing dynamic interactions among LDs, which suggests its great potential in revealing the secret of LD metabolism. The in situ TPF spectra were analyzed to determine surrounding microenvironment according to the polarity-responsive feature of our CPDs. This work expands the applications of CPDs in biological imaging, helps design new LD-selective fluorescent probes, and has implications for studying LD-related metabolism and diseases.

Single-Cell VEGF Analysis by Fluorescence Imaging-Microfluidic Droplet Platform: An Immunosandwich Strategy on the Cell Surface.

Despite recent advances in single-cell analysis techniques, the ability of single-cell analysis platforms to track specific cells that secreted cytokines remains limited. Here, we report a microfluidic droplet-based fluorescence imaging platform that can analyze single cell-secreted vascular endothelial growth factor (VEGF), an important regulator of physiological and pathological angiogenesis, to explore cellular physiological clues at the single-cell level. Two kinds of silica nanoparticle (NP)-based immunoprobes were developed, and they were bioconjugated to the membrane proteins of the probed cell surface via the bridging of secreted VEGF. Thus, an immunosandwich assay was built above the probed cell via fluorescence imaging analysis of each cell in isolated droplets. This analytical platform was used to compare the single-cell VEGF secretion ability of three cell lines (MCF-7, HeLa, and H8), which experimentally demonstrates the cellular heterogeneity of cells in secreting cytokines. The uniqueness of this method is that the single-cell assay is carried out above the cell of interest, and no additional carriers (beads or reporter cells) for capturing analytes are needed, which dramatically improves the availability of microdroplets. This single-cell analytical platform can be applied for determining other secreted cytokines at the single-cell level by changing other immune pairs, which will be an available tool for exploring single-cell metabonomics.

Spatially confined photoexcitation with triplet-triplet annihilation upconversion.

Two-photon absorption (TPA) has been widely used in confocal microscopy, photo-initiated three-dimensional (3D) polymerization, and 3D-micro/nanofabrication. These applications are based on the spatial confinement of the TPA excitation, due to the quadratic excitation power dependency of the excitation. However, an expensive and high-power femtosecond (fs) pulsed laser has to be used. Herein, we show a new technique as a promising alternative to the TPA to achieve spatially confined excitation, but no fs laser and TPA dyes are required. This new spatially confined excitation with a continuous wave laser is based on triplet-triplet-annihilation upconversion. The potential of the new technique was demonstrated by spatially confined photopolymerization.

Tuning Organic Microcrystal Morphologies through Crystal Engineering Strategies toward Anisotropic Optical Waveguide.

The construction of organic optoelectronic materials with desirable size and morphology remains a challenge now. Crystal engineering strategies (polymorphs and cocrystals) provide convenience for tailoring molecular packing and further controlling the growth morphology and photofunctionality of materials. Herein, we prepare polymorphic 2D plate crystals and 3D microhelixes by assembly of a cyanostilbene derivative (2-(3',5'-bis(trifluoromethyl)-biphenyl-4-yl)-3-(4-(pyridin-4-yl)phenyl)acrylonitrile, CF3-CN-Py). The former emits blue emission, while the latter emits green emission. Different crystallization environments contribute to the adjustable morphologies. Then, novel cocrystals are fabricated with the introduction of 1,4-diiodotetrafluorobenzene (FDIB) to CF3-CN-Py. Both molecular conformation and packing are totally changed in the cocrystal system. Such cocrystal displays a 1D sky-blue emissive rod shape on account of a long-range ordered π-stacking of molecules. In addition, the 2D plate crystal and 1D rod cocrystal are further applied to optical waveguides. In the plate crystal, a packing of transition dipole moment (µ) inclined to the upper surface leads to an anisotropic optical waveguide. In the cocrystal, owing to the nearly horizontal µ orientation, the cocrystal exhibits light propagation along the primary growth direction and a low optical loss coefficient. The present study supplies an effective way to construct materials with controlled morphology and optical waveguide.

Plasmon-Enhanced Four-Wave Mixing Imaging for Microdroplet-Based Single-Cell Analysis.

A high-throughput single-cell analytical technique based on the microdroplet array integrated with the plasmon-enhanced-four-wave mixing (PE-FWM) imaging was developed, which is applicable for the highly sensitive and automatic assessment of the surface receptors of cells. The metal nanoprobes were prepared by simply decorating metal nanoparticles with capturing molecules (antibody or molecules with surface identification function). Owing to the multifrequency selection of lasers via resonating their plasmonic bands, these metal nanoprobes are highly recognizable under the FWM imaging and display high photostability above fluorescent dyes. This PE-FWM imaging technique shows superior to dark-field imaging due to almost no interference from off-resonant species and exhibits the antifade feature that is suitable for long-period cell monitoring. The automated processing of images is available for the analysis of cell heterogeneity according to the cell surface receptors. Emerging applications such as single-cell analysis, bioimaging, metabolite, and drug tracing offer many biological and medical possibilities with broad application prospects.

A two-photon fluorescence, carbonized polymer dot (CPD)-based, wide range pH nanosensor: a view from the surface state.

A green-emitting, low-toxicity carbonized polymer dot (CPD) with a high fluorescence quantum yield was synthesised by a simple hydrothermal method, and has been applied as a three-mode pH indicator and the pH readouts involve the intensity ratio of the absorption bands, the single-photon fluorescence, and the two-photon fluorescence (TPF) signals. The pH sensing mechanism of this CPD is dependent on the hydrogen ion regulation on its surface states, which is evidenced for the first time by transient spectroscopy. The rich surface states of this CPD allow a wider pH-responsive range relative to other carbon nanodot-based pH nanosensors. Its ultra-small size, low cell toxicity, high brightness and stability are conducive to intracellular pH sensing under the TPF imaging. Our study is helpful for the development of novel carbon-based sensing materials based on the design of the surface states. It also provides a new candidate for up-conversion photoluminescence-responsive imaging agents and it has potential applications in the diagnosis and dynamic monitoring of cells relying on the pH evolution.

Deep Red Emissive Carbonized Polymer Dots with Unprecedented Narrow Full Width at Half Maximum.

Development of high-performance carbon dots (CDs) with emission wavelength longer than 660 nm (deep red emission) is critical in deep-tissue bioimaging, yet it is still a major challenge to obtain CDs with both narrow full width at half maximum (FWHM) and high deep red/near-infrared emission yield. Here, deep red emissive carbonized polymer dots (CPDs) with unprecedented FWHM of 20 nm are synthesized. The purified CPDs in dimethyl sulfoxide (DMSO) solution possess quantum yield (QY) as high as 59% under 413 nm excitation, as well as recorded QY of 31% under 660 nm excitation in the deep red fluorescent window. Detailed characterizations identify that CPDs have unique polymer characteristics, consisting of carbon cores and the shells of polymer chains, and π conjugated system formed with N heterocycles and aromatic rings governs the single photoluminescence (PL) center, which is responsible for high QY in deep red emissive CPDs with narrow FWHM. The CPDs exhibit strong absorption and emission in the deep red light region, low toxicity, and good biocompatibility, making them an efficient probe for both one-photon and two-photon bioimaging. CPDs are rapidly excreted via the kidney system and hepatobiliary system.

Sparse Embedding for Interpretable Hospital Admission Prediction.

This paper introduces a sparse embedding for electronic health record (EHR) data in order to predict hospital admission. We use a k-sparse autoencoder to embed the original registry data into a much lower dimension, with sparsity as a goal. Then, t-SNE is used to show the embedding of each patient's data in a 2D plot. We then demonstrate the predictive accuracy in different existing machine learning algorithms. Our sparse embedding performs competitively against the original data and traditional embedding vectors with an AUROC of 0.878. In addition, we demonstrate the expressive power of our sparse embedding, i.e. interpretability. Sparse embedding can discover more phenotypes in t-SNE visualization than original data or traditional embedding. The discovered phenotypes can be regarded as different risk groups, through which we can study the driving risk factors for each patient phenotype.