METRIC-GUIDED IMAGE RECONSTRUCTION BOUNDS VIA CONFORMAL PREDICTION.

Recent advancements in machine learning have led to the development of novel medical imaging systems and algorithms that address ill-posed problems. Assessing their trustworthiness and understanding how to deploy them safely at test time remains an important and open problem. In this work, we propose using conformal prediction to compute valid and distribution-free bounds on downstream metrics given reconstructions generated by one algorithm, and retrieve upper/lower bounds and inlier/outlier reconstructions according to the adjusted bounds. Our work offers 1) test time image reconstruction evaluation without ground truth, 2) downstream performance guarantees, 3) meaningful upper/lower bound reconstructions, and 4) meaningful statistical inliers/outlier reconstructions. We demonstrate our method on post-mastectomy radiotherapy planning using 3D breast CT reconstructions, and show 1) that metric-guided bounds have valid coverage for downstream metrics while conventional pixel-wise bounds do not and 2) anatomical differences of upper/lower bounds between metric-guided and pixel-wise methods. Our work paves way for more meaningful and trustworthy test-time evaluation of medical image reconstructions. Code available at https://github.com/matthewyccheung/conformal-metric.

DeepDOF-SE: affordable deep-learning microscopy platform for slide-free histology.

Histopathology plays a critical role in the diagnosis and surgical management of cancer. However, access to histopathology services, especially frozen section pathology during surgery, is limited in resource-constrained settings because preparing slides from resected tissue is time-consuming, labor-intensive, and requires expensive infrastructure. Here, we report a deep-learning-enabled microscope, named DeepDOF-SE, to rapidly scan intact tissue at cellular resolution without the need for physical sectioning. Three key features jointly make DeepDOF-SE practical. First, tissue specimens are stained directly with inexpensive vital fluorescent dyes and optically sectioned with ultra-violet excitation that localizes fluorescent emission to a thin surface layer. Second, a deep-learning algorithm extends the depth-of-field, allowing rapid acquisition of in-focus images from large areas of tissue even when the tissue surface is highly irregular. Finally, a semi-supervised generative adversarial network virtually stains DeepDOF-SE fluorescence images with hematoxylin-and-eosin appearance, facilitating image interpretation by pathologists without significant additional training. We developed the DeepDOF-SE platform using a data-driven approach and validated its performance by imaging surgical resections of suspected oral tumors. Our results show that DeepDOF-SE provides histological information of diagnostic importance, offering a rapid and affordable slide-free histology platform for intraoperative tumor margin assessment and in low-resource settings.

Roadmap for Optical Metasurfaces.

Metasurfaces have recently risen to prominence in optical research, providing unique functionalities that can be used for imaging, beam forming, holography, polarimetry, and many more, while keeping device dimensions small. Despite the fact that a vast range of basic metasurface designs has already been thoroughly studied in the literature, the number of metasurface-related papers is still growing at a rapid pace, as metasurface research is now spreading to adjacent fields, including computational imaging, augmented and virtual reality, automotive, display, biosensing, nonlinear, quantum and topological optics, optical computing, and more. At the same time, the ability of metasurfaces to perform optical functions in much more compact optical systems has triggered strong and constantly growing interest from various industries that greatly benefit from the availability of miniaturized, highly functional, and efficient optical components that can be integrated in optoelectronic systems at low cost. This creates a truly unique opportunity for the field of metasurfaces to make both a scientific and an industrial impact. The goal of this Roadmap is to mark this "golden age" of metasurface research and define future directions to encourage scientists and engineers to drive research and development in the field of metasurfaces toward both scientific excellence and broad industrial adoption.

Mesoscopic calcium imaging in a head-unrestrained male non-human primate using a lensless microscope.

Mesoscopic calcium imaging enables studies of cell-type specific neural activity over large areas. A growing body of literature suggests that neural activity can be different when animals are free to move compared to when they are restrained. Unfortunately, existing systems for imaging calcium dynamics over large areas in non-human primates (NHPs) are table-top devices that require restraint of the animal's head. Here, we demonstrate an imaging device capable of imaging mesoscale calcium activity in a head-unrestrained male non-human primate. We successfully miniaturize our system by replacing lenses with an optical mask and computational algorithms. The resulting lensless microscope can fit comfortably on an NHP, allowing its head to move freely while imaging. We are able to measure orientation columns maps over a 20 mm2 field-of-view in a head-unrestrained macaque. Our work establishes mesoscopic imaging using a lensless microscope as a powerful approach for studying neural activity under more naturalistic conditions.

Broadband thermal imaging using meta-optics.

Subwavelength diffractive optics known as meta-optics have demonstrated the potential to significantly miniaturize imaging systems. However, despite impressive demonstrations, most meta-optical imaging systems suffer from strong chromatic aberrations, limiting their utilities. Here, we employ inverse-design to create broadband meta-optics operating in the long-wave infrared (LWIR) regime (8-12 µm). Via a deep-learning assisted multi-scale differentiable framework that links meta-atoms to the phase, we maximize the wavelength-averaged volume under the modulation transfer function (MTF) surface of the meta-optics. Our design framework merges local phase-engineering via meta-atoms and global engineering of the scatterer within a single pipeline. We corroborate our design by fabricating and experimentally characterizing all-silicon LWIR meta-optics. Our engineered meta-optic is complemented by a simple computational backend that dramatically improves the quality of the captured image. We experimentally demonstrate a six-fold improvement of the wavelength-averaged Strehl ratio over the traditional hyperboloid metalens for broadband imaging.

First-Arrival Differential Counting for SPAD Array Design.

We present a novel architecture for the design of single-photon detecting arrays that captures relative intensity or timing information from a scene, rather than absolute. The proposed method for capturing relative information between pixels or groups of pixels requires very little circuitry, and thus allows for a significantly higher pixel packing factor than is possible with per-pixel TDC approaches. The inherently compressive nature of the differential measurements also reduces data throughput and lends itself to physical implementations of compressed sensing, such as Haar wavelets. We demonstrate this technique for HDR imaging and LiDAR, and describe possible future applications.

Real-time, deep-learning aided lensless microscope.

Traditional miniaturized fluorescence microscopes are critical tools for modern biology. Invariably, they struggle to simultaneously image with a high spatial resolution and a large field of view (FOV). Lensless microscopes offer a solution to this limitation. However, real-time visualization of samples is not possible with lensless imaging, as image reconstruction can take minutes to complete. This poses a challenge for usability, as real-time visualization is a crucial feature that assists users in identifying and locating the imaging target. The issue is particularly pronounced in lensless microscopes that operate at close imaging distances. Imaging at close distances requires shift-varying deconvolution to account for the variation of the point spread function (PSF) across the FOV. Here, we present a lensless microscope that achieves real-time image reconstruction by eliminating the use of an iterative reconstruction algorithm. The neural network-based reconstruction method we show here, achieves more than 10000 times increase in reconstruction speed compared to iterative reconstruction. The increased reconstruction speed allows us to visualize the results of our lensless microscope at more than 25 frames per second (fps), while achieving better than 7 µm resolution over a FOV of 10 mm2. This ability to reconstruct and visualize samples in real-time empowers a more user-friendly interaction with lensless microscopes. The users are able to use these microscopes much like they currently do with conventional microscopes.

SpeckleCam: high-resolution computational speckle contrast tomography for deep blood flow imaging.

Laser speckle contrast imaging is widely used in clinical studies to monitor blood flow distribution. Speckle contrast tomography, similar to diffuse optical tomography, extends speckle contrast imaging to provide deep tissue blood flow information. However, the current speckle contrast tomography techniques suffer from poor spatial resolution and involve both computation and memory intensive reconstruction algorithms. In this work, we present SpeckleCam, a camera-based system to reconstruct high resolution 3D blood flow distribution deep inside the skin. Our approach replaces the traditional forward model using diffuse approximations with Monte-Carlo simulations-based convolutional forward model, which enables us to develop an improved deep tissue blood flow reconstruction algorithm. We show that our proposed approach can recover complex structures up to 6 mm deep inside a tissue-like scattering medium in the reflection geometry. We also conduct human experiments to demonstrate that our approach can detect reduced flow in major blood vessels during vascular occlusion.

CoIR: Compressive Implicit Radar.

Using millimeter wave (mmWave) signals for imaging has an important advantage in that they can penetrate through poor environmental conditions such as fog, dust, and smoke that severely degrade optical-based imaging systems. However, mmWave radars, contrary to cameras and LiDARs, suffer from low angular resolution because of small physical apertures and conventional signal processing techniques. Sparse radar imaging, on the other hand, can increase the aperture size while minimizing the power consumption and read out bandwidth. This paper presents CoIR, an analysis by synthesis method that leverages the implicit neural network bias in convolutional decoders and compressed sensing to perform high accuracy sparse radar imaging. The proposed system is data set-agnostic and does not require any auxiliary sensors for training or testing. We introduce a sparse array design that allows for a 5.5× reduction in the number of antenna elements needed compared to conventional MIMO array designs. We demonstrate our system's improved imaging performance over standard mmWave radars and other competitive untrained methods on both simulated and experimental mmWave radar data.

High-Speed Time-Domain Diffuse Optical Tomography with a Sensitivity Equation-based Neural Network.

Steady progress in time-domain diffuse optical tomography (TD-DOT) technology is allowing for the first time the design of low-cost, compact, and high-performance systems, thus promising more widespread clinical TD-DOT use, such as for recording brain tissue hemodynamics. TD-DOT is known to provide more accurate values of optical properties and physiological parameters compared to its frequency-domain or steady-state counterparts. However, achieving high temporal resolution is still difficult, as solving the inverse problem is computationally demanding, leading to relatively long reconstruction times. The runtime is further compromised by processes that involve 'nontrivial' empirical tuning of reconstruction parameters, which increases complexity and inefficiency. To address these challenges, we present a new reconstruction algorithm that combines a deep-learning approach with our previously introduced sensitivity-equation-based, non-iterative sparse optical reconstruction (SENSOR) code. The new algorithm (called SENSOR-NET) unfolds the iterations of SENSOR into a deep neural network. In this way, we achieve high-resolution sparse reconstruction using only learned parameters, thus eliminating the need to tune parameters prior to reconstruction empirically. Furthermore, once trained, the reconstruction time is not dependent on the number of sources or wavelengths used. We validate our method with numerical and experimental data and show that accurate reconstructions with 1 mm spatial resolution can be obtained in under 20 milliseconds regardless of the number of sources used in the setup. This opens the door for real-time brain monitoring and other high-speed DOT applications.

NeuWS: Neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media.

Diffraction-limited optical imaging through scattering media has the potential to transform many applications such as airborne and space-based imaging (through the atmosphere), bioimaging (through skin and human tissue), and fiber-based imaging (through fiber bundles). Existing wavefront shaping methods can image through scattering media and other obscurants by optically correcting wavefront aberrations using high-resolution spatial light modulators-but these methods generally require (i) guidestars, (ii) controlled illumination, (iii) point scanning, and/or (iv) statics scenes and aberrations. We propose neural wavefront shaping (NeuWS), a scanning-free wavefront shaping technique that integrates maximum likelihood estimation, measurement modulation, and neural signal representations to reconstruct diffraction-limited images through strong static and dynamic scattering media without guidestars, sparse targets, controlled illumination, nor specialized image sensors. We experimentally demonstrate guidestar-free, wide field-of-view, high-resolution, diffraction-limited imaging of extended, nonsparse, and static/dynamic scenes captured through static/dynamic aberrations.

Unrolled-DOT: an interpretable deep network for diffuse optical tomography.

Significance: Imaging through scattering media is critical in many biomedical imaging applications, such as breast tumor detection and functional neuroimaging. Time-of-flight diffuse optical tomography (ToF-DOT) is one of the most promising methods for high-resolution imaging through scattering media. ToF-DOT and many traditional DOT methods require an image reconstruction algorithm. Unfortunately, this algorithm often requires long computational runtimes and may produce lower quality reconstructions in the presence of model mismatch or improper hyperparameter tuning. Aim: We used a data-driven unrolled network as our ToF-DOT inverse solver. The unrolled network is faster than traditional inverse solvers and achieves higher reconstruction quality by accounting for model mismatch. Approach: Our model "Unrolled-DOT" uses the learned iterative shrinkage thresholding algorithm. In addition, we incorporate a refinement U-Net and Visual Geometry Group (VGG) perceptual loss to further increase the reconstruction quality. We trained and tested our model on simulated and real-world data and benchmarked against physics-based and learning-based inverse solvers. Results: In experiments on real-world data, Unrolled-DOT outperformed learning-based algorithms and achieved over 10× reduction in runtime and mean-squared error, compared to traditional physics-based solvers. Conclusion: We demonstrated a learning-based ToF-DOT inverse solver that achieves state-of-the-art performance in speed and reconstruction quality, which can aid in future applications for noninvasive biomedical imaging.

Subdural CMOS optical probe (SCOPe) for bidirectional neural interfacing.

Optical neurotechnologies use light to interface with neurons and can monitor and manipulate neural activity with high spatial-temporal precision over large cortical extents. While there has been significant progress in miniaturizing microscope for head-mounted configurations, these existing devices are still very bulky and could never be fully implanted. Any viable translation of these technologies to human use will require a much more noninvasive, fully implantable form factor. Here, we leverage advances in microelectronics and heterogeneous optoelectronic packaging to develop a transformative, ultrathin, miniaturized device for bidirectional optical stimulation and recording: the subdural CMOS Optical Probe (SCOPe). By being thin enough to lie entirely within the subdural space of the primate brain, SCOPe defines a path for the eventual human translation of a new generation of brain-machine interfaces based on light.

Recent Advances in Lensless Imaging.

Lensless imaging provides opportunities to design imaging systems free from the constraints imposed by traditional camera architectures. Thanks to advances in imaging hardware, fabrication techniques, and new algorithms, researchers have recently developed lensless imaging systems that are extremely compact, lightweight or able to image higher-dimensional quantities. Here we review these recent advances and describe the design principles and their effects that one should consider when developing and using lensless imaging systems.

Snapshot polarimetric diffuse-specular separation.

We present a polarization-based approach to perform diffuse-specular separation from a single polarimetric image, acquired using a flexible, practical capture setup. Our key technical insight is that, unlike previous polarization-based separation methods that assume completely unpolarized diffuse reflectance, we use a more general polarimetric model that accounts for partially polarized diffuse reflections. We capture the scene with a polarimetric sensor and produce an initial analytical diffuse-specular separation that we further pass into a deep network trained to refine the separation. We demonstrate that our combination of analytical separation and deep network refinement produces state-of-the-art diffuse-specular separation, which enables image-based appearance editing of dynamic scenes and enhanced appearance estimation.

Evolution of Mood Symptomatology Through the COVID-19 Pandemic: Findings From the CovidSense Longitudinal Study.

Background The severe acute respiratory syndrome coronavirus 2 global pandemic, with its associated coronavirus disease 2019 (COVID-19) illness, has led to significant mental, physical, social, and economic hardships. Physical distancing, isolation, and fear of illness have significantly affected the mental health of people worldwide. Several studies have documented the cross-sectional elevated prevalence of mental anguish, but due to the sudden nature of the pandemic, very few longitudinal studies have been reported, especially covering the first phase of the pandemic. CovidSense, a longitudinal adaptive study, was initiated to answer some key questions: how did the pandemic and related social and economic conditions affect depression, which groups showed more vulnerability, and what protective factors emerged as the pandemic unfolded? Methodology CovidSense was deployed from April to December 2020. The adaptive design enabled adaption to fluctuating demographics/health status. Participants were regularly queried via SMS messages about their mental health, physical health, and life circumstances. The study included 1,190 participants who answered a total of 18,783 survey panels. This was a prospective longitudinal cohort study following adult participants in the general population through the COVID-19 pandemic. The participant cohort reported self-assessed measures ranging from subjective mood ratings and substance use to validated questionnaires, such as the Quick Inventory of Depressive Symptoms (QIDS) and Cognitive and Affective Mindfulness Scale-Revised (CAMS-R). Results Participants with pre-existing physical (especially pulmonary) or mental conditions had overall higher levels of depression, as measured by the QIDS and self-reported mood. Participants with pre-existing conditions also showed increased vulnerability to the stress caused by watching the news and the increase in COVID-19 cases. Younger participants (aged 18-25 years) were more affected than older groups. People with severe levels of depression had the most variation in QIDS scores, whereas individuals with none to low depressive scores had the most variability in self-reported mood fluctuations. Conclusions The effects of pandemic-related chronic stress were predominant in young adults and individuals with pre-existing mental and medical conditions regardless of whether they had acquired COVID-19 or not. These results point to the possibility of allocating preventive as well as treatment resources based on vulnerability.

The Family Level Assessment of Screen Use-Mobile Approach: Development of an Approach to Measure Children's Mobile Device Use.

BACKGROUND: There is a strong association between increased mobile device use and worse dietary habits, worse sleep outcomes, and poor academic performance in children. Self-report or parent-proxy report of children's screen time has been the most common method of measuring screen time, which may be imprecise or biased. OBJECTIVE: The objective of this study was to assess the feasibility of measuring the screen time of children on mobile devices using the Family Level Assessment of Screen Use (FLASH)-mobile approach, an innovative method that leverages the existing features of the Android platform. METHODS: This pilot study consisted of 2 laboratory-based observational feasibility studies and 2 home-based feasibility studies in the United States. A total of 48 parent-child dyads consisting of a parent and child aged 6 to 11 years participated in the pilot study. The children had to have their own or shared Android device. The laboratory-based studies included a standardized series of tasks while using the mobile device or watching television, which were video recorded. Video recordings were coded by staff for a gold standard comparison. The home-based studies instructed the parent-child dyads to use their mobile device as they typically use it over 3 days. Parents received a copy of the use logs at the end of the study and completed an exit interview in which they were asked to review their logs and share their perceptions and suggestions for the improvement of the FLASH-mobile approach. RESULTS: The final version of the FLASH-mobile approach resulted in user identification compliance rates of >90% for smartphones and >80% for tablets. For laboratory-based studies, a mean agreement of 73.6% (SD 16.15%) was achieved compared with the gold standard (human coding of video recordings) in capturing the target child's mobile use. Qualitative feedback from parents and children revealed that parents found the FLASH-mobile approach useful for tracking how much time their child spends using the mobile device as well as tracking the apps they used. Some parents revealed concerns over privacy and provided suggestions for improving the FLASH-mobile approach. CONCLUSIONS: The FLASH-mobile approach offers an important new research approach to measure children's use of mobile devices more accurately across several days, even when the child shares the device with other family members. With additional enhancement and validation studies, this approach can significantly advance the measurement of mobile device use among young children.

FlatNet3D: intensity and absolute depth from single-shot lensless capture.

Lensless cameras are ultra-thin imaging systems that replace the lens with a thin passive optical mask and computation. Passive mask-based lensless cameras encode depth information in their measurements for a certain depth range. Early works have shown that this encoded depth can be used to perform 3D reconstruction of close-range scenes. However, these approaches for 3D reconstructions are typically optimization based and require strong hand-crafted priors and hundreds of iterations to reconstruct. Moreover, the reconstructions suffer from low resolution, noise, and artifacts. In this work, we propose FlatNet3D-a feed-forward deep network that can estimate both depth and intensity from a single lensless capture. FlatNet3D is an end-to-end trainable deep network that directly reconstructs depth and intensity from a lensless measurement using an efficient physics-based 3D mapping stage and a fully convolutional network. Our algorithm is fast and produces high-quality results, which we validate using both simulated and real scenes captured using PhlatCam.

PS 2 F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing.

We propose a compact snapshot monocular depth estimation technique that relies on an engineered point spread function (PSF). Traditional approaches used in microscopic super-resolution imaging such as the Double-Helix PSF (DHPSF) are ill-suited for scenes that are more complex than a sparse set of point light sources. We show, using the Cramér-Rao lower bound, that separating the two lobes of the DHPSF and thereby capturing two separate images leads to a dramatic increase in depth accuracy. A special property of the phase mask used for generating the DHPSF is that a separation of the phase mask into two halves leads to a spatial separation of the two lobes. We leverage this property to build a compact polarization-based optical setup, where we place two orthogonal linear polarizers on each half of the DHPSF phase mask and then capture the resulting image with a polarization-sensitive camera. Results from simulations and a lab prototype demonstrate that our technique achieves up to 50% lower depth error compared to state-of-the-art designs including the DHPSF and the Tetrapod PSF, with little to no loss in spatial resolution.

In vivo lensless microscopy via a phase mask generating diffraction patterns with high-contrast contours.

The simple and compact optics of lensless microscopes and the associated computational algorithms allow for large fields of view and the refocusing of the captured images. However, existing lensless techniques cannot accurately reconstruct the typical low-contrast images of optically dense biological tissue. Here we show that lensless imaging of tissue in vivo can be achieved via an optical phase mask designed to create a point spread function consisting of high-contrast contours with a broad spectrum of spatial frequencies. We built a prototype lensless microscope incorporating the 'contour' phase mask and used it to image calcium dynamics in the cortex of live mice (over a field of view of about 16 mm2) and in freely moving Hydra vulgaris, as well as microvasculature in the oral mucosa of volunteers. The low cost, small form factor and computational refocusing capability of in vivo lensless microscopy may open it up to clinical uses, especially for imaging difficult-to-reach areas of the body.