Convolutional neural network advances in demosaicing for fluorescent cancer imaging with color-near-infrared sensors.

Significance: Single-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets. Aim: We introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor. Approach: A residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation. Results: Our optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities. Conclusions: We showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments.

Influence of signal-to-noise ratio on DoLP and AoP measurements during reflectance-mode division-of-focal plane Stokes polarimetry of biological tissues.

Stokes polarimeter based endoscopes are emerging as an area of technology where polarization imaging can greatly impact clinical care by improving diagnostic tools without the use of exogenous contrast. Image acquisition in minimally invasive surgical settings is often beset by inherently limited illumination. A comprehensive analysis of how signal-to-noise (SNR) propagates through Stokes polarimetric outcomes such as degree of linear polarization (DoLP) and angle of polarization (AoP) in low light is important for future interpretation of data acquired in low-light conditions. A previously developed theoretical model of quantitative polarized light imaging (QPLI) analysis described SNR as a function of both incident light intensity and DoLP. When polarized light interacts with biological tissues, the resultant DoLP of exiting light is dependent on the underlying tissue microstructure. Therefore, in this study we explore how low light impacts SNR of QPLI outcomes of DoLP and AoP differently in tissue phantoms of varying microstructures. Data are compared to theoretical solutions of SNR of DoLP and AoP. Tissues were additionally loaded to varying magnitudes of strain to investigate how variable SNR affects the ability to discern dynamic realignment in biological tissues. We observed a high degree of congruency between experimental and theoretical data, with SNR depending on both light intensity and DoLP. Additionally, we found that AoP may have a greater resilience to noise overall than DoLP and, as such, may be particularly useful in conditions where light is inherently limited.

Learning a global underwater geolocalization model with sectoral transformer.

Polarization-based underwater geolocalization presents an innovative method for positioning unmanned autonomous devices beneath the water surface, in environments where GPS signals are ineffective. While the state-of-the-art deep neural network (DNN) method achieves high-precision geolocalization based on sun polarization patterns in same-site tasks, its learning-based nature limits its generalizability to unseen sites and subsequently impairs its performance on cross-site tasks, where an unavoidable domain gap between training and test data exists. In this paper, we present an advanced Deep Neural Network (DNN) methodology, which includes a neural network built on a Transformer architecture, similar to the core of large language models such as ChatGPT, and integrates an unscented Kalman filter (UKF) for estimating underwater geolocation using polarization-based images. This combination effectively simulates the sun's daily trajectory, yielding enhanced performance across different locations and quicker inference speeds compared to current benchmarks. Following thorough analysis of over 10 million polarization images from four global locations, we conclude that our proposed technique significantly boosts cross-site geolocalization accuracy by around 28% when contrasted with traditional DNN methods.

Biomimetic-Membrane-Protected Plasmonic Nanostructures as Dual-Modality Contrast Agents for Correlated Surface-Enhanced Raman Scattering and Photoacoustic Detection of Hidden Tumor Lesions.

Optical imaging and spectroscopic modalities are of considerable current interest for in vivo cancer detection and image-guided surgery, but the turbid or scattering nature of biomedical tissues has severely limited their abilities to detect buried or occluded tumor lesions. Here we report the development of a dual-modality plasmonic nanostructure based on colloidal gold nanostars (AuNSs) for simultaneous surface-enhanced Raman scattering (SERS) and photoacoustic (PA) detection of tumor phantoms embedded (hidden) in ex vivo animal tissues. By using red blood cell membranes as a naturally derived biomimetic coating, we show that this class of dual-modality contrast agents can provide both Raman spectroscopic and PA signals for the detection and differentiation of hidden solid tumors with greatly improved depths of tissue penetration. Compared to previous polymer-coated AuNSs, the biomimetic coatings are also able to minimize protein adsorption and cellular uptake when exposed to human plasma without compromising their SERS or PA signals. We further show that tumor-targeting peptides (such as cyclic RGD) can be noncovalently inserted for targeting the ανß3-integrin receptors expressed on metastatic cancer cells and tracked via both SERS and PA imaging (PAI). Finally, we demonstrate image-guided resections of tumor-mimicking phantoms comprising metastatic tumor cells buried under layers of skin and fat tissues (6 mm in thickness). Specifically, PAI was used to determine the precise tumor location, while SERS spectroscopic signals were used for tumor identification and differentiation. This work opens the possibility of using these biomimetic dual-modality nanoparticles with superior signal and biological stability for intraoperative cancer detection and resection.

Bioinspired, vertically stacked, and perovskite nanocrystal-enhanced CMOS imaging sensors for resolving UV spectral signatures.

Imaging and identifying target signatures and biomedical markers in the ultraviolet (UV) spectrum is broadly important to medical imaging, military target tracking, remote sensing, and industrial automation. However, current silicon-based imaging sensors are fundamentally limited because of the rapid absorption and attenuation of UV light, hindering their ability to resolve UV spectral signatures. Here, we present a bioinspired imaging sensor capable of wavelength-resolved imaging in the UV range. Inspired by the UV-sensitive visual system of the Papilio xuthus butterfly, the sensor monolithically combines vertically stacked photodiodes and perovskite nanocrystals. This imaging design combines two complementary UV detection mechanisms: The nanocrystal layer converts a portion of UV signals into visible fluorescence, detected by the photodiode array, while the remaining UV light is detected by the top photodiode. Our label-free UV fluorescence imaging data from aromatic amino acids and cancer/normal cells enables real-time differentiation of these biomedical materials with 99% confidence.

Fluorescence-guided surgical system using holographic display: from phantom studies to canine patients.

Significance: Holographic display technology is a promising area of research that can lead to significant advancements in cancer surgery. We present the benefits of combining bioinspired multispectral imaging technology with holographic goggles for fluorescence-guided cancer surgery. Through a series of experiments with 43D-printed phantoms, small animal models of cancer, and surgeries on canine patients with head and neck cancer, we showcase the advantages of this holistic approach. Aim: The aim of our study is to demonstrate the feasibility and potential benefits of utilizing holographic display for fluorescence-guided surgery through a series of experiments involving 3D-printed phantoms and canine patients with head and neck cancer. Approach: We explore the integration of a bioinspired camera with a mixed reality headset to project fluorescent images as holograms onto a see-through display, and we demonstrate the potential benefits of this technology through benchtop and in vivo animal studies. Results: Our complete imaging and holographic display system showcased improved delineation of fluorescent targets in phantoms compared with the 2D monitor display approach and easy integration into the veterinarian surgical workflow. Conclusions: Based on our findings, it is evident that our comprehensive approach, which combines a bioinspired multispectral imaging sensor with holographic goggles, holds promise in enhancing the presentation of fluorescent information to surgeons during intraoperative scenarios while minimizing disruptions.

Effect of matrix properties on transmission and reflectance mode division-of-focal-plane Stokes polarimetry.

Significance: Division-of-focal-plane Stokes polarimetry is emerging as a powerful tool for the microstructural characterization of soft tissues. How individual extracellular matrix (ECM) properties influence polarimetric signals in reflectance or transmission modes of quantitative polarized light imaging (QPLI) is not well understood. Aim: We aimed to investigate how ECM properties affect outcomes obtained from division-of-focal-plane polarimetric imaging in reflectance or transmission modes. Approach: Tunable collagen gel phantoms were used to modulate ECM properties of anisotropy, collagen density, crosslinking, and absorber density; the effects of degree of linear polarization (DoLP) and angle of polarization (AoP) on polarimetry outcomes were assessed. A model biological tissue (i.e., bovine tendon) was similarly imaged and evaluated using both reflectance and transmission modes. Results: Reflectance QPLI resulted in decreased DoLP compared with transmission mode. A 90 deg shift in AoP was observed between modes but yielded similar spatial patterns. Collagen density had the largest effect on outcomes besides anisotropy in both imaging modes. Conclusions: Both imaging modes were sufficiently sensitive to detect structural anisotropy differences in gels of varying fiber alignment. Conclusions drawn from phantom experiments should carry over when interpreting data from more complex tissues and can help provide context for interpretation of other Stokes polarimetry data.

Cell-Membrane Coated Nanoparticles for Tumor Delineation and Qualitative Estimation of Cancer Biomarkers at Single Wavelength Excitation in Murine and Phantom Models.

Real-time guidance through fluorescence imaging improves the surgical outcomes of tumor resections, reducing the chances of leaving positive margins behind. As tumors are heterogeneous, it is imperative to interrogate multiple overexpressed cancer biomarkers with high sensitivity and specificity to improve surgical outcomes. However, for accurate tumor delineation and ratiometric detection of tumor biomarkers, current methods require multiple excitation wavelengths to image multiple biomarkers, which is impractical in a clinical setting. Here, we have developed a biomimetic platform comprising near-infrared fluorescent semiconducting polymer nanoparticles (SPNs) with red blood cell membrane (RBC) coating, capable of targeting two representative cell-surface biomarkers (folate, αυß3 integrins) using a single excitation wavelength for tumor delineation during surgical interventions. We evaluate our single excitation ratiometric nanoparticles in in vitro tumor cells, ex vivo tumor-mimicking phantoms, and in vivo mouse xenograft tumor models. Favorable biological properties (improved biocompatibility, prolonged blood circulation, reduced liver uptake) are complemented by superior spectral features: (i) specific fluorescence enhancement in tumor regions with high tumor-to-normal tissue (T/NT) ratios in ex vivo samples and (ii) estimation of cell-surface tumor biomarkers with single wavelength excitation providing insights about cancer progression (metastases). Our single excitation, dual output approach has the potential to differentiate between the tumor and healthy regions and simultaneously provide a qualitative indicator of cancer progression, thereby guiding surgeons in the operating room with the resection process.

Bioinspired color-near infrared endoscopic imaging system for molecular guided cancer surgery.

Significance: Fluorescently guided minimally invasive surgery is improving patient outcomes and disease-free survival, but biomarker variability hinders complete tumor resection with single molecular probes. To overcome this, we developed a bioinspired endoscopic system that images multiple tumor-targeted probes, quantifies volumetric ratios in cancer models, and detects tumors in ex vivo samples. Aim: We present a new rigid endoscopic imaging system (EIS) that can capture color images while simultaneously resolving two near-infrared (NIR) probes. Approach: Our optimized EIS integrates a hexa-chromatic image sensor, a rigid endoscope optimized for NIR-color imaging, and a custom illumination fiber bundle. Results: Our optimized EIS achieves a 60% improvement in NIR spatial resolution when compared to a leading FDA-approved endoscope. Ratio-metric imaging of two tumor-targeted probes is demonstrated in vials and animal models of breast cancer. Clinical data gathered from fluorescently tagged lung cancer samples on the operating room's back table demonstrate a high tumor-to-background ratio and consistency with the vial experiments. Conclusions: We investigate key engineering breakthroughs for the single-chip endoscopic system, which can capture and distinguish numerous tumor-targeting fluorophores. As the molecular imaging field shifts toward a multi-tumor targeted probe methodology, our imaging instrument can aid in assessing these concepts during surgical procedures.

Angle of polarization calibration for omnidirectional polarization cameras.

Polarization cameras quantify one of the fundamental properties of light and capture intrinsic properties of the imaged environment that are otherwise omitted by color sensors. Many polarization applications, such as underwater geolocalization and sky-based polarization compass, require simultaneous imaging of the entire radial optical field with omnidirectional lenses. However, the reconstructed angle of polarization captured with omnidirectional lenses has a radial offset due to redirection of the light rays within these lenses. In this paper, we describe a calibration method for correcting angle of polarization images captured with omnidirectional lenses. Our calibration method reduces the variance of reconstructed angle of polarization from 76.2 ∘ to 4.1 ∘. Example images collected both on an optical bench and in nature, demonstrate the improved accuracy of the reconstructed angle of polarization with our calibration method. The improved accuracy in the angle of polarization images will aid the development of polarization-based applications with omnidirectional lenses.

Interpolant-based demosaicing routines for dual-mode visible/near-infrared imaging systems.

Dual-mode visible/near-infrared imaging systems, including a bioinspired six-channel design and more conventional four-channel implementations, have transitioned from a niche in surveillance to general use in machine vision. However, the demosaicing routines that transform the raw images from these sensors into processed images that can be consumed by humans or computers rely on assumptions that may not be appropriate when the two portions of the spectrum contribute different information about a scene. A solution can be found in a family of demosaicing routines that utilize interpolating polynomials and splines of different dimensionalities and orders to process images with minimal assumptions.

Protease-activated indocyanine green nanoprobes for intraoperative NIR fluorescence imaging of primary tumors.

Tumor-targeted fluorescent probes in the near-infrared spectrum can provide invaluable information about the location and extent of primary and metastatic tumors during intraoperative procedures to ensure no residual tumors are left in the patient's body. Even though the first fluorescence-guided surgery was performed more than 50 years ago, it is still not accepted as a standard of care in part due to the lack of efficient and non-toxic targeted probes approved by regulatory agencies around the world. Herein, we report protease-activated cationic gelatin nanoparticles encapsulating indocyanine green (ICG) for the detection of primary breast tumors in murine models with high tumor-to-background ratios. Upon intravenous administration, these nanoprobes remain optically silent due to the energy resonance transfer among the bound ICG molecules. As the nanoprobes extravasate and are exposed to the acidic tumor microenvironment, their positive surface charges increase, facilitating cellular uptake. The internalized nanoprobes are activated upon proteolytic degradation of gelatin to allow high contrast between the tumor and normal tissue. Since both gelatin and ICG are FDA-approved for intravenous administration, this activatable nanoprobe can lead to quick clinical adoption and improve the treatment of patients undergoing image-guided cancer surgery.

Decoupling channel count from field of view and spatial resolution in single-sensor imaging systems for fluorescence image-guided surgery.

Significance: Near-infrared fluorescence image-guided surgery is often thought of as a spectral imaging problem where the channel count is the critical parameter, but it should also be thought of as a multiscale imaging problem where the field of view and spatial resolution are similarly important. Aim: Conventional imaging systems based on division-of-focal-plane architectures suffer from a strict relationship between the channel count on one hand and the field of view and spatial resolution on the other, but bioinspired imaging systems that combine stacked photodiode image sensors and long-pass/short-pass filter arrays offer a weaker tradeoff. Approach: In this paper, we explore how the relevant changes to the image sensor and associated image processing routines affect image fidelity during image-guided surgeries for tumor removal in an animal model of breast cancer and nodal mapping in women with breast cancer. Results: We demonstrate that a transition from a conventional imaging system to a bioinspired one, along with optimization of the image processing routines, yields improvements in multiple measures of spectral and textural rendition relevant to surgical decision-making. Conclusions: These results call for a critical examination of the devices and algorithms that underpin image-guided surgery to ensure that surgeons receive high-quality guidance and patients receive high-quality outcomes as these technologies enter clinical practice.

Biomimetic Surface-Enhanced Raman Scattering Nanoparticles with Improved Dispersibility, Signal Brightness, and Tumor Targeting Functions.

The development of biocompatible and nontoxic surface-enhanced Raman scattering (SERS) nanoparticles is of considerable current interest because of their attractive biomedical applications such as ultrasensitive in vitro diagnostics, in vivo tumor imaging, and spectroscopy-guided cancer surgery. However, current SERS nanoparticles are prepared and stored in aqueous solution, have limited stability and dispersibility, and are not suitable for lyophilization and storage by freeze-drying or other means. Here, we report a simple but robust method to coat colloidal SERS nanoparticles by naturally derived biomimetic red blood cell membranes (RBCM), leading to a dramatic improvement in stability and dispersibility under freeze-thawing, lyophilization, heating, and physiological conditions. The results demonstrate that the lyophilized SERS nanoparticles in the solid form can be readily dissolved and dispersed in physiological buffer solutions. A surprising finding is that the RBCM-coated SERS particles are considerably brighter (by as much as 5-fold) than PEGylated SERS particles under similar experimental conditions. This additional enhancement is believed to arise from the hydrophobic nature of RBCM's hydrocarbon chains, which is known to reduce electronic dampening and boost electromagnetic field enhancement. A further advantage in using biomimetic membrane coatings is that the bilayer membrane structure allows nonvalent insertion of molecular ligands for tumor targeting. In particular, we show that cyclic-RGD, a tumor-targeting peptide, can be efficiently inserted into the membrane coatings of SERS nanoparticles for targeting the ανß3 integrin receptors expressed on cancer cells. Thus, biomimetic RBCMs provide major advantages over traditional polyethylene glycols for preparing SERS nanoparticles with improved dispersibility, higher signal intensity, and more efficient biofunctionalization.

Hexachromatic bioinspired camera for image-guided cancer surgery.

Cancer affects one in three people worldwide. Surgery remains the primary curative option for localized cancers, but good prognoses require complete removal of primary tumors and timely recognition of metastases. To expand surgical capabilities and enhance patient outcomes, we developed a six-channel color/near-infrared image sensor inspired by the mantis shrimp visual system that enabled near-infrared fluorescence image guidance during surgery. The mantis shrimp's unique eye, which maximizes the number of photons contributing to and the amount of information contained in each glimpse of its surroundings, is recapitulated in our single-chip imaging system that integrates arrays of vertically stacked silicon photodetectors and pixelated spectral filters. To provide information about tumor location unavailable from a single instrument, we tuned three color channels to permit an intuitive perspective of the surgical procedure and three near-infrared channels to permit multifunctional imaging of optical probes highlighting cancerous tissue. In nude athymic mice bearing human prostate tumors, our image sensor enabled simultaneous detection of two tumor-targeted fluorophores, distinguishing diseased from healthy tissue in an estimated 92% of cases. It also permitted extraction of near-infrared structured illumination enabling the mapping of the three-dimensional topography of tumors and surgical sites to within 1.2-mm error. In the operating room, during surgical resection in 18 patients with breast cancer, our image sensor further enabled sentinel lymph node mapping using clinically approved near-infrared fluorophores. The flexibility and performance afforded by this simple and compact architecture highlights the benefits of biologically inspired sensors in image-guided surgery.

Comparison of the polarization contrast of gelatinous zooplankton and a transparent single-use plastic bag-Implications for marine animals.

Plastic pollution in the ocean is an increasingly detrimental issue for marine organisms. As a form of polarized light pollution, transparent plastic debris may be more visible and pose additional threats to organisms that can detect and interpret polarized light. Plastic can mimic the visual features of common marine prey items, such as transparent gelatinous zooplankton, which may lead to more significant plastic ingestion. We measured, in situ, the polarization and radiance contrast between a transparent plastic bag and gelatinous zooplankton with an underwater video polarimeter. The plastic bag had significantly higher polarization contrast than the gelatinous zooplankton, yet both shared similar radiance contrasts. This higher polarization contrast may contribute to the observed high ingestion rates of transparent plastic by marine organisms. Further study into the connection between polarization-sensitive organisms and plastic ingestion is recommended.

Thresholds of polarization vision in octopuses.

Polarization vision is widespread in nature, mainly among invertebrates, and is used for a range of tasks including navigation, habitat localization and communication. In marine environments, some species such as those from the Crustacea and Cephalopoda that are principally monochromatic, have evolved to use this adaptation to discriminate objects across the whole visual field, an ability similar to our own use of colour vision. The performance of these polarization vision systems varies, and the few cephalopod species tested so far have notably acute thresholds of discrimination. However, most studies to date have used artificial sources of polarized light that produce levels of polarization much higher than found in nature. In this study, the ability of octopuses to detect polarization contrasts varying in angle of polarization (AoP) was investigated over a range of different degrees of linear polarization (DoLP) to better judge their visual ability in more ecologically relevant conditions. The 'just-noticeable-differences' (JND) of AoP contrasts varied consistently with DoLP. These JND thresholds could be largely explained by their 'polarization distance', a neurophysical model that effectively calculates the level of activity in opposing horizontally and vertically oriented polarization channels in the cephalopod visual system. Imaging polarimetry from the animals' natural environment was then used to illustrate the functional advantage that these polarization thresholds may confer in behaviourally relevant contexts.

Implementation of a logarithmic division-of-focal-plane polarimeter to quantify changes in collagen alignment at varying levels of illumination.

We examine the impact of illumination, aperture, and sample thickness on two division-of-focal-plane (DoFP) polarimeters, one created using a standard 3 T pixel and the other with a forward-biased, logarithmic pixel. Across all measured metrics the logarithmic DoFP polarimeter was better able to track real-time changes in collagen alignment than the standard DoFP polarimeter.

Polarisation signals: a new currency for communication.

Most polarisation vision studies reveal elegant examples of how animals, mainly the invertebrates, use polarised light cues for navigation, course-control or habitat selection. Within the past two decades it has been recognised that polarised light, reflected, blocked or transmitted by some animal and plant tissues, may also provide signals that are received or sent between or within species. Much as animals use colour and colour signalling in behaviour and survival, other species additionally make use of polarisation signalling, or indeed may rely on polarisation-based signals instead. It is possible that the degree (or percentage) of polarisation provides a more reliable currency of information than the angle or orientation of the polarised light electric vector (e-vector). Alternatively, signals with specific e-vector angles may be important for some behaviours. Mixed messages, making use of polarisation and colour signals, also exist. While our knowledge of the physics of polarised reflections and sensory systems has increased, the observational and behavioural biology side of the story needs more (and more careful) attention. This Review aims to critically examine recent ideas and findings, and suggests ways forward to reveal the use of light that we cannot see.

Bio-inspired imager improves sensitivity in near-infrared fluorescence image-guided surgery.

Image-guided surgery can enhance cancer treatment by decreasing, and ideally eliminating, positive tumor margins and iatrogenic damage to healthy tissue. Current state-of-the-art near-infrared fluorescence imaging systems are bulky and costly, lack sensitivity under surgical illumination, and lack co-registration accuracy between multimodal images. As a result, an overwhelming majority of physicians still rely on their unaided eyes and palpation as the primary sensing modalities for distinguishing cancerous from healthy tissue. Here we introduce an innovative design, comprising an artificial multispectral sensor inspired by the Morpho butterfly's compound eye, which can significantly improve image-guided surgery. By monolithically integrating spectral tapetal filters with photodetectors, we have realized a single-chip multispectral imager with 1000 × higher sensitivity and 7 × better spatial co-registration accuracy compared to clinical imaging systems in current use. Preclinical and clinical data demonstrate that this technology seamlessly integrates into the surgical workflow while providing surgeons with real-time information on the location of cancerous tissue and sentinel lymph nodes. Due to its low manufacturing cost, our bio-inspired sensor will provide resource-limited hospitals with much-needed technology to enable more accurate value-based health care.