T-HyperGNNs: Hypergraph Neural Networks via Tensor Representations.

Hypergraph neural networks (HyperGNNs) are a family of deep neural networks designed to perform inference on hypergraphs. HyperGNNs follow either a spectral or a spatial approach, in which a convolution or message-passing operation is conducted based on a hypergraph algebraic descriptor. While many HyperGNNs have been proposed and achieved state-of-the-art performance on broad applications, there have been limited attempts at exploring high-dimensional hypergraph descriptors (tensors) and joint node interactions carried by hyperedges. In this article, we depart from hypergraph matrix representations and present a new tensor-HyperGNN (T-HyperGNN) framework with cross-node interactions (CNIs). The T-HyperGNN framework consists of T-spectral convolution, T-spatial convolution, and T-message-passing HyperGNNs (T-MPHN). The T-spectral convolution HyperGNN is defined under the t-product algebra that closely connects to the spectral space. To improve computational efficiency for large hypergraphs, we localize the T-spectral convolution approach to formulate the T-spatial convolution and further devise a novel tensor-message-passing algorithm for practical implementation by studying a compressed adjacency tensor representation. Compared to the state-of-the-art approaches, our T-HyperGNNs preserve intrinsic high-order network structures without any hypergraph reduction and model the joint effects of nodes through a CNI layer. These advantages of our T-HyperGNNs are demonstrated in a wide range of real-world hypergraph datasets. The implementation code is available at https://github.com/wangfuli/T-HyperGNNs.git.

Hyperspectral image reconstruction via patch attention driven network.

Coded aperture snapshot spectral imaging (CASSI) captures 3D hyperspectral images (HSIs) with 2D compressive measurements. The recovery of HSIs from these measurements is an ill-posed problem. This paper proposes a novel, to our knowledge, network architecture for this inverse problem, which consists of a multilevel residual network driven by patch-wise attention and a data pre-processing method. Specifically, we propose the patch attention module to adaptively generate heuristic clues by capturing uneven feature distribution and global correlations of different regions. By revisiting the data pre-processing stage, we present a complementary input method that effectively integrates the measurements and coded aperture. Extensive simulation experiments illustrate that the proposed network architecture outperforms state-of-the-art methods.

A Multigraph-Defined Distribution Function in a Simulation Model of a Communication Network.

We presented a method based on multigraphs to mathematically define a distribution function in time for the generation of data exchange in a special-purpose communication network. This is needed for the modeling and design of communication networks (CNs) consisting of integrated telecommunications and computer networks (ITCN). Simulation models require a precise definition of network traffic communication. An additional problem for describing the network traffic in simulation models is the mathematical model of data distribution, according to which the generation and exchange of certain types and quantities of data are realized. The application of multigraphs enabled the time and quantity of the data distribution to be displayed as operational procedures for a special-purpose communication unit. A multigraph was formed for each data-exchange time and allowed its associated adjacency matrix to be defined. Using the matrix estimation method allowed the mathematical definition of the distribution function values. The application of the described method for the use of multigraphs enabled a more accurate mathematical description of real traffic in communication networks.

Block-based spectral image reconstruction for compressive spectral imaging using smoothness on graphs.

A novel reconstruction method for compressive spectral imaging is designed by assuming that the spectral image of interest is sufficiently smooth on a collection of graphs. Since the graphs are not known in advance, we propose to infer them from a panchromatic image using a state-of-the-art graph learning method. Our approach leads to solutions with closed-form that can be found efficiently by solving multiple sparse systems of linear equations in parallel. Extensive simulations and an experimental demonstration show the merits of our method in comparison with traditional methods based on sparsity and total variation and more recent methods based on low-rank minimization and deep-based plug-and-play priors. Our approach may be instrumental in designing efficient methods based on deep neural networks and covariance estimation.

Static coded aperture in robotic X-ray tomography systems.

Coded aperture X-ray computed tomography is a computational imaging technique capable of reconstructing inner structures of an object from a reduced set of X-ray projection measurements. Coded apertures are placed in front of the X-ray sources from different views and thus significantly reduce the radiation dose. This paper introduces coded aperture X-ray computed tomography for robotic X-ray systems which offer positioning flexibility. While single coded-aperture 3D tomography was recently introduced for standard trajectory CT scanning, it is shown that significant gains in imaging performance can be attained by simple modifications in the CT scanning trajectories enabled by emerging dual robotic CT systems. In particular, the subject is fixed on a plane and the CT system uniformly rotates around the r -axis which is misaligned with the coordinate axes. A single stationary coded aperture is placed on front of the robotic X-ray source above the plane and the corresponding X-ray projections are measured by a two-dimensional detector on the second arm of the robotic system. The compressive measurements with misalignment enable the reconstruction of high-resolution three-dimensional volumetric images from the low-resolution coded projections on the detector at a sub-sampling rate. An efficient algorithm is proposed to generate the rotation matrix with two basic sub-matrices and thus the forward model is formulated. The stationary coded aperture is designed based on the Pearson product-moment correlation coefficient analysis and the direct binary search algorithm is used to obtain the optimized coded aperture. Simulations using simulated datasets show significant gains in reconstruction performance compared to conventional coded aperture CT systems.

Static coded illumination strategies for low-dose x-ray material decomposition.

Static coded aperture x-ray tomography was introduced recently where a static illumination pattern is used to interrogate an object with a low radiation dose, from which an accurate 3D reconstruction of the object can be attained computationally. Rather than continuously switching the pattern of illumination with each view angle, as traditionally done, static code computed tomography (CT) places a single pattern for all views. The advantages are many, including the feasibility of practical implementation. This paper generalizes this powerful framework to develop single-scan dual-energy coded aperture spectral tomography that enables material characterization at a significantly reduced exposure level. Two sensing strategies are explored: rapid kV switching with a single-static block/unblock coded aperture, and coded apertures with non-uniform thickness. Both systems rely on coded illumination with a plurality of x-ray spectra created by kV switching or 3D coded apertures. The structured x-ray illumination is projected through the objects of interest and measured with standard x-ray energy integrating detectors. Then, based on the tensor representation of projection data, we develop an algorithm to estimate a full set of synthesized measurements that can be used with standard reconstruction algorithms to accurately recover the object in each energy channel. Simulation and experimental results demonstrate the effectiveness of the proposed cost-effective solution to attain material characterization in low-dose dual-energy CT.

Experimental demonstration and optimization of X-ray StaticCodeCT.

As the use of X-ray computed tomography (CT) grows in medical diagnosis, so does the concern for the harm a radiation dose can cause and the biological risks it represents. StaticCodeCT is a new low-dose imaging architecture that uses a single-static coded aperture (CA) in a CT gantry. It exploits the highly correlated data in the projection domain to estimate the unobserved measurements on the detector. We previously analyzed the StaticCodeCT system by emulating the effect of the coded mask on experimental CT data. In contrast, this manuscript presents test-bed reconstructions using an experimental cone-beam X-ray CT system with a CA holder. We analyzed the reconstruction quality using three different techniques to manufacture the CAs: metal additive manufacturing, cold casting, and ceramic additive manufacturing. Furthermore, we propose an optimization method to design the CA pattern based on the algorithm developed for the measurement estimation. The obtained results point to the possibility of the real deployment of StaticCodeCT systems in practice.

Compressive hyperspectral image classification using a 3D coded convolutional neural network.

Hyperspectral image classification (HIC) is an active research topic in remote sensing. Hyperspectral images typically generate large data cubes posing big challenges in data acquisition, storage, transmission and processing. To overcome these limitations, this paper develops a novel deep learning HIC approach based on compressive measurements of coded-aperture snapshot spectral imagers (CASSI), without reconstructing the complete hyperspectral data cube. A new kind of deep learning strategy, namely 3D coded convolutional neural network (3D-CCNN), is proposed to efficiently solve for the classification problem, where the hardware-based coded aperture is regarded as a pixel-wise connected network layer. An end-to-end training method is developed to jointly optimize the network parameters and the coded apertures with periodic structures. The accuracy of classification is effectively improved by exploiting the synergy between the deep learning network and coded apertures. The superiority of the proposed method is assessed over the state-of-the-art HIC methods on several hyperspectral datasets.

Compressive X-ray tomosynthesis using model-driven deep learning.

Compressive X-ray tomosynthesis uses a few two-dimensional projection measurements modulated by coding masks to reconstruct the three-dimensional object that can be sparsely represented on a predefined basis. However, the coding mask optimization and object reconstruction require significant computing resources. In addition, existing methods fall short to exploits the synergy between the encoding and reconstruction stages to approach the global optimum. This paper proposes a model-driven deep learning (MDL) approach to significantly improve the computational efficiency and accuracy of tomosynthesis reconstruction. A unified framework is developed to jointly optimize the coding masks and the neural network parameters, which effectively increase the degrees of optimization freedom. It shows that the computational efficiency of coding mask optimization and image reconstruction can be improved by more than one order of magnitude. Furthermore, the performance of reconstruction results is significantly improved.

Efficient informatics-based source and mask optimization for optical lithography.

Source and mask optimization (SMO) is a widely used computational lithography technology that greatly improves the image fidelity of lithography systems. This paper develops an efficient informatics-based SMO (EISMO) method to improve the image fidelity of lithography systems. First, a communication channel model is established to depict the mechanism of information transmission in the SMO framework, where the source is obtained from the gradient-based SMO algorithm. The manufacturing-aware mask distribution is then optimized to achieve the best mutual information, and the theoretical lower bound of lithography patterning error is obtained. Subsequently, an efficient informatics-based method is proposed to refine the mask optimization result in SMO, further reducing the lithography patterning error. It is shown that the proposed EISMO method is computationally efficient and can achieve superior imaging performance over the conventional SMO method.

Multi-objective optimization for structured illumination in dynamic x-ray tomosynthesis.

Dynamic coded x-ray tomosynthesis (CXT) uses a set of encoded x-ray sources to interrogate objects lying on a moving conveyor mechanism. The object is reconstructed from the encoded measurements received by the uniform linear array detectors. We propose a multi-objective optimization (MO) method for structured illuminations to balance the reconstruction quality and radiation dose in a dynamic CXT system. The MO framework is established based on a dynamic sensing geometry with binary coding masks. The Strength Pareto Evolutionary Algorithm 2 is used to solve the MO problem by jointly optimizing the coding masks, locations of x-ray sources, and exposure moments. Computational experiments are implemented to assess the proposed MO method. They show that the proposed strategy can obtain a set of Pareto optimal solutions with different levels of radiation dose and better reconstruction quality than the initial setting.

Nonlinear reconstruction of coded spectral X-ray CT based on material decomposition.

Coded spectral X-ray computed tomography (CT) based on K-edge filtered illumination is a cost-effective approach to acquire both 3-dimensional structure of objects and their material composition. This approach allows sets of incomplete rays from sparse views or sparse rays with both spatial and spectral encoding to effectively reduce the inspection duration or radiation dose, which is of significance in biological imaging and medical diagnostics. However, reconstruction of spectral CT images from compressed measurements is a nonlinear and ill-posed problem. This paper proposes a material-decomposition-based approach to directly solve the reconstruction problem, without estimating the energy-binned sinograms. This approach assumes that the linear attenuation coefficient map of objects can be decomposed into a few basis materials that are separable in the spectral and space domains. The nonlinear problem is then converted to the reconstruction of the mass density maps of the basis materials. The dimensionality of the optimization variables is thus effectively reduced to overcome the ill-posedness. An alternating minimization scheme is used to solve the reconstruction with regularizations of weighted nuclear norm and total variation. Compared to the state-of-the-art reconstruction method for coded spectral CT, the proposed method can significantly improve the reconstruction quality. It is also capable of reconstructing the spectral CT images at two additional energy bins from the same set of measurements, thus providing more spectral information of the object.

StaticCodeCT: single coded aperture tensorial X-ray CT.

Coded aperture X-ray CT (CAXCT) is a new low-dose imaging technology that promises far-reaching benefits in industrial and clinical applications. It places various coded apertures (CA) at a time in front of the X-ray source to partially block the radiation. The ill-posed inverse reconstruction problem is then solved using l1-norm-based iterative reconstruction methods. Unfortunately, to attain high-quality reconstructions, the CA patterns must change in concert with the view-angles making the implementation impractical. This paper proposes a simple yet radically different approach to CAXCT, which is coined StaticCodeCT, that uses a single-static CA in the CT gantry, thus making the imaging system amenable for practical implementations. Rather than using conventional compressed sensing algorithms for recovery, we introduce a new reconstruction framework for StaticCodeCT. Namely, we synthesize the missing measurements using low-rank tensor completion principles that exploit the multi-dimensional data correlation and low-rank nature of a 3-way tensor formed by stacking the 2D coded CT projections. Then, we use the FDK algorithm to recover the 3D object. Computational experiments using experimental projection measurements exhibit up to 10% gains in the normalized root mean square distance of the reconstruction using the proposed method compared with those attained by alternative low-dose systems.

Blue noise coding for a coherent x-ray diffraction imaging system.

Coded x-ray diffraction imaging (CXRDI) is an emerging computational imaging approach that aims to solve the phase retrieval problem in x-ray crystallography based on the intensity measurements of encoded diffraction patterns. Boolean coding masks (BCMs) with complementary structures have been used to modulate the diffraction pattern in CXRDI. However, the optimal spatial distribution of BCMs still remains an open problem to be studied in depth. Based on the spectral initialization criterion, we provide a theoretical proof for the premise that the optimal complementary BCMs should obey the blue noise distribution in the sense of mathematical expectation. In addition, the benefits of the blue noise coding strategy are assessed by a set of simulations, where better reconstruction quality is observed compared to the random BCMs and other complementary BCMs.

LED-based compressive spectral-temporal imaging.

A compressive spectral-temporal imaging system is reported. A multi-spectral light-emitting diode array is used for target illumination and spectral modulation, while a digital micro-mirror device (DMD) encodes the spatial and temporal frames. Several encoded video frames are captured in a snapshot of an integrating focal plane array (FPA). A high-frame-rate spectral video is reconstructed from the sequence of compressed measurements captured by the grayscale low-frame-rate camera. The imaging system is optimized through the design of the DMD patterns based on the forward model. Laboratory implementation is conducted to validate the performance of the proposed imaging system. We experimentally demonstrate the video acquisition with eight spectral bands and six temporal frames per FPA snapshot, and thus a 256 × 256 × 8 × 6 4D cube is reconstructed from a single 2D measurement.

Model-informed deep learning for computational lithography with partially coherent illumination.

Computational lithography is a key technique to optimize the imaging performance of optical lithography systems. However, the large amount of calculation involved in computational lithography significantly increases the computational complexity. This paper proposes a model-informed deep learning (MIDL) approach to improve its computational efficiency and to enhance the image fidelity of lithography system with partially coherent illumination (PCI). Different from conventional deep learning approaches, the network structure of MIDL is derived from an approximate compact imaging model of PCI lithography system. MIDL has a dual-channel structure, which overcomes the vanishing gradient problem and improves its prediction capacity. In addition, an unsupervised training method is developed based on an accurate lithography imaging model to avoid the computational cost of labelling process. It is shown that the MIDL provides significant gains in terms of computational efficiency and imaging performance of PCI lithography system.

Single-snapshot X-ray imaging for nonlinear compressive tomosynthesis.

Traditional compressive X-ray tomosynthesis uses sequential illumination to interrogate the object, leading to long scanning time and image distortion due to the object variation. This paper proposes a single-snapshot compressive tomosynthesis imaging approach, where the object is simultaneously illuminated by multiple X-ray emitters equipped with coded apertures. Based on rank, intensity and sparsity prior models, a nonlinear image reconstruction framework is established. The coded aperture patterns are optimized based on uniform sensing criteria. Then, a modified split Bregman algorithm is developed to reconstruct the object from the set of nonlinear compressive measurements. It is shown that the proposed method can be used to reduce the inspection time and achieve robust reconstruction with respect to shape variation or motion of objects.

Fast inverse lithography based on dual-channel model-driven deep learning.

Inverse lithography technology (ILT) is extensively used to compensate image distortion in optical lithography systems by pre-warping the photomask at the pixel scale. However, computational complexity is always a central challenge of ILT due to the big throughput of data volume. This paper proposes a dual-channel model-driven deep learning (DMDL) method to overcome the computational burden, while break through the limit of image fidelity over traditional ILT algorithms. The architecture of DMDL network is not inherited from conventional deep learning, but derived from the inverse optimization model under a gradient-based ILT framework. A dual-channel structure is introduced to extend the capacity of the DMDL network, which allows to simultaneously modify the mask contour and insert sub-resolution assist features to further improve the lithography image fidelity. An unsupervised training strategy based on auto-decoder is developed to avoid the time-consuming labelling process. The superiority of DMDL over the state-of-the-art ILT method is verified in both of the computational efficiency and image fidelity obtained on the semiconductor wafer.

Compressive spectral imaging approach using adaptive coded apertures.

The coded aperture snapshot spectral imager (CASSI) acquires three-dimensional spectral images with two-dimensional coded projection measurements. This paper proposes an adaptive design method of the coded apertures, according to a priori knowledge of the target scene, to improve sensing efficiency and imaging performance of the super-resolution CASSI system. The adaptive coded apertures are constructed from the nonlinear thresholding of the grayscale map of the scene. Theoretical proof is provided to demonstrate the superiority of the adaptive coded apertures over traditional random coded apertures. Improvement in reconstruction performance is also verified by a set of simulations based on different spectral data.

Snapshot Compressive ToF+Spectral Imaging via Optimized Color-Coded Apertures.

Compressive multispectral imaging systems comprise a new generation of spectral imagers that capture coded projections of a scene where spectral data cubes are reconstructed computationally. Separately, time-of-flight (ToF) cameras obtain 2D range images where each pixel records the distance from the camera sensor to the target surface. The demand for these imaging modalities is rapidly increasing, and thus, there is strong interest in developing new image sensors that can simultaneously acquire multispectral-color-and-depth imagery (MS+D) using a single aperture. Work in this path has been mainly developed via RGB+D imaging. However, in RGB+D, the multispectral image is limited to three spectral channels, and the imaging system often relies on two image sensors. We recently proposed a compressive MS+D imaging device that used a digital-micromirror-device, requiring a bulky double imaging-and-relay path. To overcome the bulkiness and other difficulties of our previous imaging system, this work presents a more-compact MS+D imaging device with snapshot capabilities. It provides better spectral sensing, relying on a static color-coded-aperture (CCA) and a ToF sensor. To guarantee good quality in the recovery, we develop an optimization method for CCA based-on blue-noise-multitoning, solved via the direct-binary-search algorithm. A testbed-setup is reported along with simulated and real experiments that demonstrate the MS+D capabilities of the proposed system over static and dynamic scenes.