Correcting projector lens distortion in real time with a scale-offset model for structured light illumination.

In fringe projection profilometry, inevitable distortion of optical lenses decreases phase accuracy and decreases the quality of 3D point clouds. For camera lens distortion, existing compensation methods include real time look-up tables derived from the related parameters of camera calibration. However, for projector lens distortion, so far, post-undistortion methods iteratively correcting lens distortion are relatively time-consuming while, despite avoiding iteration, pre-distortion methods are not suitable for binary fringe patterns. In this paper, we aim to achieve real-time phase correction for the projector by means of a scale-offset model that characterizes projector distortion by four correction parameters within a small-enough area, and thus we can speed up the post-undistortion by looking up tables. Experiments show that the proposed method can suppress the distortion error by a factor of 20 ×, i.e., the error of root mean square is less than 45 µm/0.7‰, while also proposed improving the computation speed by a factor of 50× over traditional iterative post-undistortion.

BimodalPS: Causes and Corrections for Bimodal Multi-path in Phase-Shifting Structured Light Scanners.

Structured light illumination is an active 3D scanning technique based on projecting and capturing a set of striped patterns and measuring the warping of the patterns as they reflect off a target object's surface. As designed, each pixel in the camera sees exactly one pixel from the projector; however, there are multi-path situations where a camera pixel sees light from multiple projector positions. In the case of bimodal multi-path, the camera pixel receives light from exactly two positions, which occurs along a step edge where the edge slices through a pixel which, therefore, sees both a foreground and background surface. In this paper, we present a general mathematical model to address this bimodal multi-path issue in a phase-shifting or so-called phase-measuring-profilometry scanner to measure the constructive and destructive interference between the two light paths, and by taking advantage of this interference, separate the paths and make two decoupled depth measurements. We validate our algorithm with both simulations and a number of challenging real-world scenarios, significantly outperforming the state-of-the-art methods.

Block-based spectral image reconstruction for compressive spectral imaging using smoothness on graphs.

A novel reconstruction method for compressive spectral imaging is designed by assuming that the spectral image of interest is sufficiently smooth on a collection of graphs. Since the graphs are not known in advance, we propose to infer them from a panchromatic image using a state-of-the-art graph learning method. Our approach leads to solutions with closed-form that can be found efficiently by solving multiple sparse systems of linear equations in parallel. Extensive simulations and an experimental demonstration show the merits of our method in comparison with traditional methods based on sparsity and total variation and more recent methods based on low-rank minimization and deep-based plug-and-play priors. Our approach may be instrumental in designing efficient methods based on deep neural networks and covariance estimation.

Extending epipolar geometry for real-time structured light illumination II: lossless accuracy.

Structured light illumination is a process of 3D scanning using a digital projector to project a series of striped patterns that sweep a target surface, and based on the warping of the stripes viewed by a camera, the shape of the target can be reconstructed. In the case of scanning stripes separately in both horizontal and vertical directions, algorithms have been proposed that achieve real-time reconstruction through look-up tables; however, these look-up tables implement the inversion of one traditional lens projection matrix. In this Letter, we propose look-up tables constructed using (1) both views of a camera and a projector and (2) basic arithmetic operations rather than complex matrix operations to significantly reduce the total number of computations used to reconstruct a point cloud. Experiments show that, with the same accuracy, the proposed tables improve the computation speed by a factor of 6.66×, from 31.00 to 206.61 fps.

Hybrid metasurfaces for simultaneous focusing and filtering.

This work presents the design and fabrication of polymeric, structural optical filters that simultaneously focus light. These filters represent a novel, to the best of our knowledge, design at the boundary between diffractive optics and metasurfaces that may provide significant advantages for both digital and hyperspectral imaging. Filters for visible and near-infrared wavelengths were designed using finite-difference time-domain (FDTD) simulations. Prototype filters were fabricated using two-photon lithography, a form of nanoscale 3D printing, and have geometries suitable to replication by molding. The experimentally measured spectral transmission and focused spot size of each filter show excellent agreement with simulation.

Extending epipolar geometry for real-time structured light illumination.

Structured light illumination, scanning along both horizontal and vertical directions, achieves more robust accuracy. By introducing the constraint of epipolar geometry, we previously proposed real-time 3D reconstruction using lookup tables; however, we only knew these offline derived tables were the combinations of the elements in calibration matrices of a camera and a projector, and suffered from long-time computation. In this Letter, by parameterizing the line perspectively mapping a 3D world coordinate into the camera and projector spaces, we propose to extend the epipolar analysis by defining phase and optical poles. Thus, we can geometrically address these parameters via analytic closed-form equations, with which we can (1) directly derive lookup tables in real time from the calibration matrices and (2) optimally reduce the number of tables from 11 to 5 to save much more memory space while further accelerating the processing rate. Experiments show that with the same level of accuracy, we significantly reduce the time to compute the lookup tables from more than 20 min to 20 ms, and increase the speed of computing point clouds from approximately 320 to 492 fps.

Dual-projector structured light 3D shape measurement.

Structured light illumination is an active three-dimensional scanning technique that uses a projector and camera pair to project and capture a series of stripe patterns; however, with a single camera and single projector, structured light scanning has issues associated with scan occlusions, multi-path, and weak signal reflections. To address these issues, this paper proposes dual-projector scanning using a range of projector/camera arrangements. Unlike previous attempts at dual-projector scanning, the proposed scanner drives both light engines simultaneously, using temporal-frequency multiplexing to computationally decouple the projected patterns. Besides presenting the details of how such a system is built, we also present experimental results demonstrating how multiple projectors can be used to (1) minimize occlusions; (2) achieve higher signal-to-noise ratios having twice a single projector's brightness; (3) reduce the number of component video frames required for a scan; and (4) detect multi-path interference.

Divide and conquer: high-accuracy and real-time 3D reconstruction of static objects using multiple-phase-shifted structured light illumination.

Multiple-phase-shifted structured light illumination achieves high-accuracy 3D reconstructions of static objects, while typically it can't achieve real-time phase computation. In this paper, we propose to compute modulations and phases of multiple scans in real time by using divide-and-conquer solutions. First, we categorize total N = KM images into M groups and each group contains K phase equally shifted images; second, we compute the phase of each group; and finally, we obtain the final phase by averaging all the separately computed phases. When K = 3, 4 or 6, we can use integer-valued intensities of images as inputs and build one or M look-up tables storing real-valued phases computed by using arctangent function. Thus, with addition and/or subtraction operations computing indices of the tables, we can directly access the pre-computed phases and avoid time-consuming arctangent computation. Compared with K-step phase measuring profilometry repeated for M times, the proposed is robust to nonlinear distortion of structured light systems. Experiments show that, first, the proposed is of the same accuracy level as the traditional algorithm, and secondly, with employing one core of a central processing unit, compared with the classical 12-step phase measuring profilometry algorithm, for K = 4 and M = 3, the proposed improves phase computation by a factor of 6 ×.

Snapshot Compressive ToF+Spectral Imaging via Optimized Color-Coded Apertures.

Compressive multispectral imaging systems comprise a new generation of spectral imagers that capture coded projections of a scene where spectral data cubes are reconstructed computationally. Separately, time-of-flight (ToF) cameras obtain 2D range images where each pixel records the distance from the camera sensor to the target surface. The demand for these imaging modalities is rapidly increasing, and thus, there is strong interest in developing new image sensors that can simultaneously acquire multispectral-color-and-depth imagery (MS+D) using a single aperture. Work in this path has been mainly developed via RGB+D imaging. However, in RGB+D, the multispectral image is limited to three spectral channels, and the imaging system often relies on two image sensors. We recently proposed a compressive MS+D imaging device that used a digital-micromirror-device, requiring a bulky double imaging-and-relay path. To overcome the bulkiness and other difficulties of our previous imaging system, this work presents a more-compact MS+D imaging device with snapshot capabilities. It provides better spectral sensing, relying on a static color-coded-aperture (CCA) and a ToF sensor. To guarantee good quality in the recovery, we develop an optimization method for CCA based-on blue-noise-multitoning, solved via the direct-binary-search algorithm. A testbed-setup is reported along with simulated and real experiments that demonstrate the MS+D capabilities of the proposed system over static and dynamic scenes.

Automatic look-up table based real-time phase unwrapping for phase measuring profilometry and optimal reference frequency selection.

For temporal phase unwrapping in phase measuring profilometry, it has recently been reported that two phases with co-prime frequencies can be absolutely unwrapped using a look-up table; however, frequency selection and table construction has been performed manually without optimization. In this paper, a universal phase unwrapping method is proposed to unwrap phase flexibly and automatically by using geometric analysis, and thus we can programmatically build a one-dimensional or two-dimensional look-up table for arbitrary two co-prime frequencies to correctly unwrap phases in real time. Moreover, a phase error model related to the defocus effect is derived to figure out an optimal reference frequency co-prime to the principal frequency. Experimental results verify the correctness and computational efficiency of the proposed method.

Reconstructing 3D point clouds in real time with look-up tables for structured light scanning along both horizontal and vertical directions.

By scanning static, not moving, objects along both the horizontal and vertical axes instead of one, structured light illumination achieves more accurate and robust 3D surface reconstructions but with greater latency on computing 3D point clouds. If scanning is performed along only one axis, it has been reported that look-up tables, manually derived from the calibration matrices of a camera and a projector, can significantly help to speed up computation; however, it has been nearly impossible to manually derive similar look-up tables for phases scanned along two axes. In this Letter, we bridge this divide by introducing the constraint of epipolar geometry to automatically compute look-up tables and thus, significantly speed up computing 3D point clouds with only basic arithmetic operations rather than time-consuming matrix computations. Experimental results show that the proposed method, using only single-thread CPU computing, reduces process latency by an order of magnitude.

Coded aperture design in compressive spectral imaging based on side information.

Coded aperture compressive spectral imagers (CSI) sense a three-dimensional data cube by using two-dimensional projections of the coded and spectrally dispersed input image. Recently, it has been shown that by combining spectral images acquired from a CSI sensor and a complementary sensor leads to substantial improvement in the quality of the fused image. To maximally exploit the benefits of the complementary information, the spatial structure of the coded apertures must be optimized inasmuch as these structures determine the sensing matrix properties and, accordingly, the quality of the reconstructed images. This paper proposes a method to use side information from a red-green-blue sensor to design the coded aperture patterns of a CSI imager, such that more detailed spatial images and wavelength profiles can be reconstructed. The side information is used as the input of an edge detection algorithm to approximate a version of the edges of the spectral images. The coded apertures are designed to follow the spatial structure determined by the estimated spectral edges, such that the high frequencies are promoted, leading to more detailed reconstructed spectral images. Simulations and experimental results indicate that when compared with random coded aperture structures, the designed coded apertures based on side information obtain up to 3 dB improvement in the quality of the reconstructed images.

Multi-spectral compressive snapshot imaging using RGB image sensors.

Compressive sensing is a powerful sensing and reconstruction framework for recovering high dimensional signals with only a handful of observations and for spectral imaging, compressive sensing offers a novel method of multispectral imaging. Specifically, the coded aperture snapshot spectral imager (CASSI) system has been demonstrated to produce multi-spectral data cubes color images from a single snapshot taken by a monochrome image sensor. In this paper, we expand the theoretical framework of CASSI to include the spectral sensitivity of the image sensor pixels to account for color and then investigate the impact on image quality using either a traditional color image sensor that spatially multiplexes red, green, and blue light filters or a novel Foveon image sensor which stacks red, green, and blue pixels on top of one another.

QR images: optimized image embedding in QR codes.

This paper introduces the concept of QR images, an automatic method to embed QR codes into color images with bounded probability of detection error. These embeddings are compatible with standard decoding applications and can be applied to any color image with full area coverage. The QR information bits are encoded into the luminance values of the image, taking advantage of the immunity of QR readers against local luminance disturbances. To mitigate the visual distortion of the QR image, the algorithm utilizes halftoning masks for the selection of modified pixels and nonlinear programming techniques to locally optimize luminance levels. A tractable model for the probability of error is developed and models of the human visual system are considered in the quality metric used to optimize the luminance levels of the QR image. To minimize the processing time, the optimization techniques proposed to consider the mechanics of a common binarization method and are designed to be amenable for parallel implementations. Experimental results show the graceful degradation of the decoding rate and the perceptual quality as a function the embedding parameters. A visual comparison between the proposed and existing methods is presented.

In Vivo measurement of pediatric vocal fold motion using structured light laser projection.

OBJECTIVE: The aim of the study was to present the development of a miniature structured light laser projection endoscope and to quantify vocal fold length and vibratory features related to impact stress of the pediatric glottis using high-speed imaging. STUDY DESIGN: The custom-developed laser projection system consists of a green laser with a 4-mm diameter optics module at the tip of the endoscope, projecting 20 vertical laser lines on the glottis. Measurements of absolute phonatory vocal fold length, membranous vocal fold length, peak amplitude, amplitude-to-length ratio, average closing velocity, and impact velocity were obtained in five children (6-9 years), two adult male and three adult female participants without voice disorders, and one child (10 years) with bilateral vocal fold nodules during modal phonation. RESULTS: Independent measurements made on the glottal length of a vocal fold phantom demonstrated a 0.13mm bias error with a standard deviation of 0.23mm, indicating adequate precision and accuracy for measuring vocal fold structures and displacement. First, in vivo measurements of amplitude-to-length ratio, peak closing velocity, and impact velocity during phonation in pediatric population and a child with vocal fold nodules are reported. CONCLUSION: The proposed laser projection system can be used to obtain in vivo measurements of absolute length and vibratory features in children and adults. Children have large amplitude-to-length ratio compared with typically developing adults, whereas nodules result in larger peak amplitude, amplitude-to-length ratio, average closing velocity, and impact velocity compared with typically developing children.

A Kalman-filtering approach to high dynamic range imaging for measurement applications.

High dynamic range imaging (HDRI) methods in computational photography address situations where the dynamic range of a scene exceeds what can be captured by an image sensor in a single exposure. HDRI techniques have also been used to construct radiance maps in measurement applications; unfortunately, the design and evaluation of HDRI algorithms for use in these applications have received little attention. In this paper, we develop a novel HDRI technique based on pixel-by-pixel Kalman filtering and evaluate its performance using objective metrics that this paper also introduces. In the presented experiments, this new technique achieves as much as 9.4-dB improvement in signal-to-noise ratio and can achieve as much as a 29% improvement in radiometric accuracy over a classic method.

Robust active stereo vision using Kullback-Leibler divergence.

Active stereo vision is a method of 3D surface scanning involving the projecting and capturing of a series of light patterns where depth is derived from correspondences between the observed and projected patterns. In contrast, passive stereo vision reveals depth through correspondences between textured images from two or more cameras. By employing a projector, active stereo vision systems find correspondences between two or more cameras, without ambiguity, independent of object texture. In this paper, we present a hybrid 3D reconstruction framework that supplements projected pattern correspondence matching with texture information. The proposed scheme consists of using projected pattern data to derive initial correspondences across cameras and then using texture data to eliminate ambiguities. Pattern modulation data are then used to estimate error models from which Kullback-Leibler divergence refinement is applied to reduce misregistration errors. Using only a small number of patterns, the presented approach reduces measurement errors versus traditional structured light and phase matching methodologies while being insensitive to gamma distortion, projector flickering, and secondary reflections. Experimental results demonstrate these advantages in terms of enhanced 3D reconstruction performance in the presence of noise, deterministic distortions, and conditions of texture and depth contrast.

Corneal antifibrotic switch identified in genetic and pharmacological deficiency of vimentin.

The type III intermediate filaments (IFs) are essential cytoskeletal elements of mechanosignal transduction and serve critical roles in tissue repair. Mice genetically deficient for the IF protein vimentin (Vim(-/-)) have impaired wound healing from deficits in myofibroblast development. We report a surprising finding made in Vim(-/-) mice that corneas are protected from fibrosis and instead promote regenerative healing after traumatic alkali injury. This reparative phenotype in Vim(-/-) corneas is strikingly recapitulated by the pharmacological agent withaferin A (WFA), a small molecule that binds to vimentin and down-regulates its injury-induced expression. Attenuation of corneal fibrosis by WFA is mediated by down-regulation of ubiquitin-conjugating E3 ligase Skp2 and up-regulation of cyclin-dependent kinase inhibitors p27(Kip1) and p21(Cip1). In cell culture models, WFA exerts G(2)/M cell cycle arrest in a p27(Kip1)- and Skp2-dependent manner. Finally, by developing a highly sensitive imaging method to measure corneal opacity, we identify a novel role for desmin overexpression in corneal haze. We demonstrate that desmin down-regulation by WFA via targeting the conserved WFA-ligand binding site shared among type III IFs promotes further improvement of corneal transparency without affecting cyclin-dependent kinase inhibitor levels in Vim(-/-) mice. This dissociates a direct role for desmin in corneal cell proliferation. Taken together, our findings illuminate a previously unappreciated pathogenic role for type III IF overexpression in corneal fibrotic conditions and also validate WFA as a powerful drug lead toward anti-fibrosis therapeutic development.

Molecular and functional analysis of Drosophila single-minded larval central brain expression.

Developmental regulatory proteins are commonly utilized in multiple cell types throughout development. The Drosophila single-minded (sim) gene acts as master regulator of embryonic CNS midline cell development and transcription. However, it is also expressed in the brain during larval development. In this paper, we demonstrate that sim is expressed in three clusters of anterior central brain neurons: DAMv1/2, BAmas1/2, and TRdm and in three clusters of posterior central brain neurons: a subset of DPM neurons, and two previously unidentified clusters, which we term PLSC and PSC. In addition, sim is expressed in the lamina and medulla of the optic lobes. MARCM studies confirm that sim is expressed at high levels in neurons but is low or absent in neuroblasts (NBs) and ganglion mother cell (GMC) precursors. In the anterior brain, sim(+) neurons are detected in 1st and 2nd instar larvae but rapidly increase in number during the 3rd instar stage. To understand the regulation of sim brain transcription, 12 fragments encompassing 5'-flanking, intronic, and 3'-flanking regions were tested for the presence of enhancers that drive brain expression of a reporter gene. Three of these fragments drove expression in sim(+) brain cells, including all sim(+) neuronal clusters in the central brain and optic lobes. One fragment upstream of sim is autoregulatory and is expressed in all sim(+) brain cells. One intronic fragment drives expression in only the PSC and laminar neurons. Another downstream intronic fragment drives expression in all sim(+) brain neurons, except the PSC and lamina. Thus, together these two enhancers drive expression in all sim(+) brain neurons. Sequence analysis of existing sim mutant alleles identified three likely null alleles to utilize in MARCM experiments to examine sim brain function. Mutant clones of DAMv1/2 neurons revealed a consistent axonal fasciculation defect. Thus, unlike the embryonic roles of sim that control CNS midline neuron and glial formation and differentiation, postembryonic sim, instead, controls aspects of axon guidance in the brain. This resembles the roles of vertebrate sim that have an early role in neuronal migration and a later role in axonogenesis.

A robust model for simultaneously inducing corneal neovascularization and retinal gliosis in the mouse eye.

PURPOSE: To develop an animal model for simultaneously eliciting corneal angiogenesis and retinal gliosis that will enable the assessment of inhibitor efficacy on these two pathological processes in separate anatomic sites of the ocular globe. METHODS: Four to six week-old mice in a C57BL/6J background were anesthetized and 0.15 N NaOH was applied to the cornea, followed by mechanical scraping of the epithelium from limbus and central cornea. After this injury, mice were treated with vehicle or with an inhibitor (withaferin A [WFA]), which were delivered by intraperitoneal injection, to assess the pharmacological effects on angiogenesis and/or gliosis. Mice were sacrificed after 14 days and tissues (corneas and retinas) were prepared for analysis of corneal neovascularization and retinal gliosis by immunohistochemistry and western blotting, respectively. This protocol was also suited for studying earlier disease end points, for assessment of drug dose efficacy or genetic influences and the entire procedure and this analysis was completed in 16-17 days. RESULTS: Both corneal angiogenesis and retinal gliosis were maximally sustained at fourteen days following chemical and mechanical injury of the cornea. 1) Injured corneas showed abundant CD31+ staining, with new blood vessels branching out from the limbus to the central cornea. WFA treatment potently inhibited corneal neovascularization. 2) Retinal gliosis in injured mice was associated with upregulated expression of glial fibrillary acidic protein (GFAP) that appeared as polymeric filaments and soluble forms expressed in reactive Müller glial cells. WFA treatment potently downregulated the expression of soluble and filamentous GFAP; the latter protein was fragmented. CONCLUSIONS: We have developed a mouse model for investigating retinal gliosis and corneal neovascularization. We used this model to demonstrate the simultaneous inhibitory effects of WFA on both of these disease processes. Retinal gliosis occurs in several major degenerative conditions of the eye, including age-related macular degeneration, where angiogenesis is also a prevailing pathological feature. Thus, inhibitors of both gliosis and angiogensis used as combination therapy are currently being explored for treatment of such complex diseases. The model presented here affords a very simple preclinical assay for screening combination of drugs or polypharmacological agents and reduces the numbers of animals because of the different anatomic sites of these pathologies. Finally, given that endogenous mediators elicit angiogenesis and gliosis in this model, the combination of genetics and pharmacology can be exploited to study drug mechanisms and for target validation in vivo.