Multispectral Demosaicing Based on Iterative-Linear-Regression Model for Estimating Pseudo-Panchromatic Image.

This paper proposes a method for demosaicing raw images captured by multispectral cameras. The proposed method estimates a pseudo-panchromatic image (PPI) via an iterative-linear-regression model and utilizes the estimated PPI for multispectral demosaicing. The PPI is estimated through horizontal and vertical guided filtering, with the subsampled multispectral-filter-array-(MSFA) image and low-pass-filtered MSFA as the guide image and filtering input, respectively. The number of iterations is automatically determined according to a predetermined criterion. Spectral differences between the estimated PPI and MSFA are calculated for each channel, and each spectral difference is interpolated using directional interpolation. The weights are calculated from the estimated PPI, and each interpolated spectral difference is combined using the weighted sum. The experimental results indicate that the proposed method outperforms the State-of-the-Art methods with regard to spatial and spectral fidelity for both synthetic and real-world images.

Kernel Estimation Using Total Variation Guided GAN for Image Super-Resolution.

Various super-resolution (SR) kernels in the degradation model deteriorate the performance of the SR algorithms, showing unpleasant artifacts in the output images. Hence, SR kernel estimation has been studied to improve the SR performance in several ways for more than a decade. In particular, a conventional research named KernelGAN has recently been proposed. To estimate the SR kernel from a single image, KernelGAN introduces generative adversarial networks(GANs) that utilize the recurrence of similar structures across scales. Subsequently, an enhanced version of KernelGAN, named E-KernelGAN, was proposed to consider image sharpness and edge thickness. Although it is stable compared to the earlier method, it still encounters challenges in estimating sizable and anisotropic kernels because the structural information of an input image is not sufficiently considered. In this paper, we propose a kernel estimation algorithm called Total Variation Guided KernelGAN (TVG-KernelGAN), which efficiently enables networks to focus on the structural information of an input image. The experimental results show that the proposed algorithm accurately and stably estimates kernels, particularly sizable and anisotropic kernels, both qualitatively and quantitatively. In addition, we compared the results of the non-blind SR methods, using SR kernel estimation techniques. The results indicate that the performance of the SR algorithms was improved using our proposed method.

Infrared Image Deconvolution Considering Fixed Pattern Noise.

As the demand for thermal information increases in industrial fields, numerous studies have focused on enhancing the quality of infrared images. Previous studies have attempted to independently overcome one of the two main degradations of infrared images, fixed pattern noise (FPN) and blurring artifacts, neglecting the other problems, to reduce the complexity of the problems. However, this is infeasible for real-world infrared images, where two degradations coexist and influence each other. Herein, we propose an infrared image deconvolution algorithm that jointly considers FPN and blurring artifacts in a single framework. First, an infrared linear degradation model that incorporates a series of degradations of the thermal information acquisition system is derived. Subsequently, based on the investigation of the visual characteristics of the column FPN, a strategy to precisely estimate FPN components is developed, even in the presence of random noise. Finally, a non-blind image deconvolution scheme is proposed by analyzing the distinctive gradient statistics of infrared images compared with those of visible-band images. The superiority of the proposed algorithm is experimentally verified by removing both artifacts. Based on the results, the derived infrared image deconvolution framework successfully reflects a real infrared imaging system.

Crosstalk Correction for Color Filter Array Image Sensors Based on Lp-Regularized Multi-Channel Deconvolution.

In this paper, we propose a crosstalk correction method for color filter array (CFA) image sensors based on Lp-regularized multi-channel deconvolution. Most imaging systems with CFA exhibit a crosstalk phenomenon caused by the physical limitations of the image sensor. In general, this phenomenon produces both color degradation and spatial degradation, which are respectively called desaturation and blurring. To improve the color fidelity and the spatial resolution in crosstalk correction, the feasible solution of the ill-posed problem is regularized by image priors. First, the crosstalk problem with complex spatial and spectral degradation is formulated as a multi-channel degradation model. An objective function with a hyper-Laplacian prior is then designed for crosstalk correction. This approach enables the simultaneous improvement of the color fidelity and the sharpness restoration of the details without noise amplification. Furthermore, an efficient solver minimizes the objective function for crosstalk correction consisting of Lp regularization terms. The proposed method was verified on synthetic datasets according to various crosstalk and noise levels. Experimental results demonstrated that the proposed method outperforms the conventional methods in terms of the color peak signal-to-noise ratio and structural similarity index measure.

Color Demosaicing of RGBW Color Filter Array Based on Laplacian Pyramid.

In recent years, red, green, blue, and white (RGBW) color filter arrays (CFAs) have been developed to solve the problem of low-light conditions. In this paper, we propose a new color demosaicing algorithm for RGBW CFAs using a Laplacian pyramid. Because the white channel has a high correlation to the red, green, and blue channels, the white channel is interpolated first using each color difference channel. After we estimate the white channel, the red, green, and blue channels are interpolated using the Laplacian pyramid decomposition of the estimated white channel. Our proposed method using Laplacian pyramid restoration works with Canon-RGBW CFAs and any other periodic CFAs. The experimental results demonstrated that the proposed method shows superior performance compared with other conventional methods in terms of the color peak signal-to-noise ratio, structural similarity index measure, and average execution time.

Thermal Image Restoration Based on LWIR Sensor Statistics.

An imaging system has natural statistics that reflect its intrinsic characteristics. For example, the gradient histogram of a visible light image generally obeys a heavy-tailed distribution, and its restoration considers natural statistics. Thermal imaging cameras detect infrared radiation, and their signal processors are specialized according to the optical and sensor systems. Thermal images, also known as long wavelength infrared (LWIR) images, suffer from distinct degradations of LWIR sensors and residual nonuniformity (RNU). However, despite the existence of various studies on the statistics of thermal images, thermal image processing has seldom attempted to incorporate natural statistics. In this study, natural statistics of thermal imaging sensors are derived, and an optimization method for restoring thermal images is proposed. To verify our hypothesis about the thermal images, high-frequency components of thermal images from various datasets are analyzed with various measures (correlation coefficient, histogram intersection, chi-squared test, Bhattacharyya distance, and Kullback-Leibler divergence), and generalized properties are derived. Furthermore, cost functions accommodating the validated natural statistics are designed and minimized by a pixel-wise optimization method. The proposed algorithm has a specialized structure for thermal images and outperforms the conventional methods. Several image quality assessments are employed for quantitatively demonstrating the performance of the proposed method. Experiments with synthesized images and real-world images are conducted, and the results are quantified by reference image assessments (peak signal-to-noise ratio and structural similarity index measure) and no-reference image assessments (Roughness (Ro) and Effective Roughness (ERo) indices).

Demosaicing of RGBW Color Filter Array Based on Rank Minimization with Colorization Constraint.

Recently, the white (w) channel has been incorporated in various forms into color filter arrays (CFAs). The advantage of using the W channel is that W pixels have less noise than RGB pixels; therefore, under low-light conditions, pixels with high fidelity can be obtained. However, RGBW CFAs normally suffer from spatial resolution degradation due to a smaller number of color pixels than in RGB CFAs. Therefore, even though the reconstructed colors have higher sensitivity, which results in larger CPSNR values, there are some color aliasing artifacts due to a low resolution. In this paper, we propose a rank minimization-based color interpolation method with a colorization constraint for the RGBW format with a large number of W pixels. The rank minimization can achieve a broad interpolation and preserve the structure in the image, and it thereby eliminates the color artifacts. However, the colors fade from this global process. Therefore, we further incorporate a colorization constraint into the rank minimization process for better reproduction of the colors. Experimental results show that the images can be reconstructed well even from noisy pattern images obtained under low-light conditions.

Sensitivity Improvement of Extremely Low Light Scenes with RGB-NIR Multispectral Filter Array Sensor.

Recently, several red-green-blue near-infrared (RGB-NIR) multispectral filter arrays (MFAs), which include near infrared (NIR) pixels, have been proposed. For extremely low light scenes, the RGB-NIR MFA sensor has been extended to receive NIR light, by adding NIR pixels to supplement for the insufficient visible band light energy. However, the resolution reconstruction of the RGB-NIR MFA, using demosaicing and color restoration methods, is based on the correlation between the NIR pixels and the pixels of other colors; this does not improve the RGB channel sensitivity with respect to the NIR channel sensitivity. In this paper, we propose a color restored image post-processing method to improve the sensitivity and resolution of an RGB-NIR MFA. Although several linear regression based color channel reconstruction methods have taken advantage of the high sensitivity NIR channel, it is difficult to accurately estimate the linear coefficients because of the high level of noise in the color channels under extremely low light conditions. The proposed method solves this problem in three steps: guided filtering, based on the linear similarity between the NIR and color channels, edge preserving smoothing to improve the accuracy of linear coefficient estimation, and residual compensation for lost spatial resolution information. The results show that the proposed method is effective, while maintaining the NIR pixel resolution characteristics, and improving the sensitivity in terms of the signal-to-noise ratio by approximately 13 dB.

Permuted Coordinate-wise Optimizations Applied to Lp-regularized Image Deconvolution.

Image deconvolution is an ill-posed problem that usually requires prior knowledge for regularizing the feasible solutions. In literature, iterative methods estimate an intrinsic image, minimizing a cost function regularized by specific prior information. However, it is difficult to directly minimize the constrained cost function, if a nondifferentiable regularization (e.g., the sparsity constraint) is employed. In this paper, we propose a nonderivative image deconvolution algorithm that solves the under-constrained problem (i.e., a non-blind image deconvolution) by successively solving the permuted subproblems. The subproblems, arranged in permuted sequences, directly minimize the nondifferentiable cost functions. Various Lp-regularized (0 < p ≤ 1, p = 2) objective functions are utilized to demonstrate the pixel-wise optimization, in which the projection operator generates simplified, low-dimensional subproblems for estimating each pixel. The subproblems, after projection, are dealt with in the corresponding hyperplanes containing the adjacent pixels of each image coordinate. Furthermore, successively solving the subproblems can accelerate the deconvolution process with a linear speed-up, by parallelizing the subproblem sequences. The image deconvolution results with various regularization functionals are presented and the linear speed-up is also demonstrated with a parallelized version of the proposed algorithm. Experimental results demonstrate that the proposed method outperforms the conventional methods in terms of the improved-signal-to-noise ratio and structural similarity index measure.

Sensitivity and Resolution Improvement in RGBW Color Filter Array Sensor.

Recently, several red-green-blue-white (RGBW) color filter arrays (CFAs), which include highly sensitive W pixels, have been proposed. However, RGBW CFA patterns suffer from spatial resolution degradation owing to the sensor composition having more color components than the Bayer CFA pattern. RGBW CFA demosaicing methods reconstruct resolution using the correlation between white (W) pixels and pixels of other colors, which does not improve the red-green-blue (RGB) channel sensitivity to the W channel level. In this paper, we thus propose a demosaiced image post-processing method to improve the RGBW CFA sensitivity and resolution. The proposed method decomposes texture components containing image noise and resolution information. The RGB channel sensitivity and resolution are improved through updating the W channel texture component with those of RGB channels. For this process, a cross multilateral filter (CMF) is proposed. It decomposes the smoothness component from the texture component using color difference information and distinguishes color components through that information. Moreover, it decomposes texture components, luminance noise, color noise, and color aliasing artifacts from the demosaiced images. Finally, by updating the texture of the RGB channels with the W channel texture components, the proposed algorithm improves the sensitivity and resolution. Results show that the proposed method is effective, while maintaining W pixel resolution characteristics and improving sensitivity from the signal-to-noise ratio value by approximately 4.5 dB.

Poisson-Gaussian Noise Analysis and Estimation for Low-Dose X-ray Images in the NSCT Domain.

The noise distribution of images obtained by X-ray sensors in low-dosage situations can be analyzed using the Poisson and Gaussian mixture model. Multiscale conversion is one of the most popular noise reduction methods used in recent years. Estimation of the noise distribution of each subband in the multiscale domain is the most important factor in performing noise reduction, with non-subsampled contourlet transform (NSCT) representing an effective method for scale and direction decomposition. In this study, we use artificially generated noise to analyze and estimate the Poisson-Gaussian noise of low-dose X-ray images in the NSCT domain. The noise distribution of the subband coefficients is analyzed using the noiseless low-band coefficients and the variance of the noisy subband coefficients. The noise-after-transform also follows a Poisson-Gaussian distribution, and the relationship between the noise parameters of the subband and the full-band image is identified. We then analyze noise of actual images to validate the theoretical analysis. Comparison of the proposed noise estimation method with an existing noise reduction method confirms that the proposed method outperforms traditional methods.

Edge enhancement algorithm for low-dose X-ray fluoroscopic imaging.

BACKGROUND AND OBJECTIVE: Low-dose X-ray fluoroscopy has continually evolved to reduce radiation risk to patients during clinical diagnosis and surgery. However, the reduction in dose exposure causes quality degradation of the acquired images. In general, an X-ray device has a time-average pre-processor to remove the generated quantum noise. However, this pre-processor causes blurring and artifacts within the moving edge regions, and noise remains in the image. During high-pass filtering (HPF) to enhance edge detail, this noise in the image is amplified. METHODS: In this study, a 2D edge enhancement algorithm comprising region adaptive HPF with the transient improvement (TI) method, as well as artifacts and noise reduction (ANR), was developed for degraded X-ray fluoroscopic images. The proposed method was applied in a static scene pre-processed by a low-dose X-ray fluoroscopy device. First, the sharpness of the X-ray image was improved using region adaptive HPF with the TI method, which facilitates sharpening of edge details without overshoot problems. Then, an ANR filter that uses an edge directional kernel was developed to remove the artifacts and noise that can occur during sharpening, while preserving edge details. RESULTS: The quantitative and qualitative results obtained by applying the developed method to low-dose X-ray fluoroscopic images and visually and numerically comparing the final images with images improved using conventional edge enhancement techniques indicate that the proposed method outperforms existing edge enhancement methods in terms of objective criteria and subjective visual perception of the actual X-ray fluoroscopic image. CONCLUSIONS: The developed edge enhancement algorithm performed well when applied to actual low-dose X-ray fluoroscopic images, not only by improving the sharpness, but also by removing artifacts and noise, including overshoot.

Colorization-Based RGB-White Color Interpolation using Color Filter Array with Randomly Sampled Pattern.

Recently, several RGB-White (RGBW) color filter arrays (CFAs) have been proposed, which have extra white (W) pixels in the filter array that are highly sensitive. Due to the high sensitivity, the W pixels have better SNR (Signal to Noise Ratio) characteristics than other color pixels in the filter array, especially, in low light conditions. However, most of the RGBW CFAs are designed so that the acquired RGBW pattern image can be converted into the conventional Bayer pattern image, which is then again converted into the final color image by using conventional demosaicing methods, i.e., color interpolation techniques. In this paper, we propose a new RGBW color filter array based on a totally different color interpolation technique, the colorization algorithm. The colorization algorithm was initially proposed for colorizing a gray image into a color image using a small number of color seeds. Here, we adopt this algorithm as a color interpolation technique, so that the RGBW color filter array can be designed with a very large number of W pixels to make the most of the highly sensitive characteristics of the W channel. The resulting RGBW color filter array has a pattern with a large proportion of W pixels, while the small-numbered RGB pixels are randomly distributed over the array. The colorization algorithm makes it possible to reconstruct the colors from such a small number of RGB values. Due to the large proportion of W pixels, the reconstructed color image has a high SNR value, especially higher than those of conventional CFAs in low light condition. Experimental results show that many important information which are not perceived in color images reconstructed with conventional CFAs are perceived in the images reconstructed with the proposed method.

Denoising Algorithm for CFA Image Sensors Considering Inter-Channel Correlation.

In this paper, a spatio-spectral-temporal filter considering an inter-channel correlation is proposed for the denoising of a color filter array (CFA) sequence acquired by CCD/CMOS image sensors. Owing to the alternating under-sampled grid of the CFA pattern, the inter-channel correlation must be considered in the direct denoising process. The proposed filter is applied in the spatial, spectral, and temporal domain, considering the spatio-tempo-spectral correlation. First, nonlocal means (NLM) spatial filtering with patch-based difference (PBD) refinement is performed by considering both the intra-channel correlation and inter-channel correlation to overcome the spatial resolution degradation occurring with the alternating under-sampled pattern. Second, a motion-compensated temporal filter that employs inter-channel correlated motion estimation and compensation is proposed to remove the noise in the temporal domain. Then, a motion adaptive detection value controls the ratio of the spatial filter and the temporal filter. The denoised CFA sequence can thus be obtained without motion artifacts. Experimental results for both simulated and real CFA sequences are presented with visual and numerical comparisons to several state-of-the-art denoising methods combined with a demosaicing method. Experimental results confirmed that the proposed frameworks outperformed the other techniques in terms of the objective criteria and subjective visual perception in CFA sequences.

G-Channel Restoration for RWB CFA with Double-Exposed W Channel.

In this paper, we propose a green (G)-channel restoration for a red-white-blue (RWB) color filter array (CFA) image sensor using the dual sampling technique. By using white (W) pixels instead of G pixels, the RWB CFA provides high-sensitivity imaging and an improved signal-to-noise ratio compared to the Bayer CFA. However, owing to this high sensitivity, the W pixel values become rapidly over-saturated before the red-blue (RB) pixel values reach the appropriate levels. Because the missing G color information included in the W channel cannot be restored with a saturated W, multiple captures with dual sampling are necessary to solve this early W-pixel saturation problem. Each W pixel has a different exposure time when compared to those of the R and B pixels, because the W pixels are double-exposed. Therefore, a RWB-to-RGB color conversion method is required in order to restore the G color information, using a double-exposed W channel. The proposed G-channel restoration algorithm restores G color information from the W channel by considering the energy difference caused by the different exposure times. Using the proposed method, the RGB full-color image can be obtained while maintaining the high-sensitivity characteristic of the W pixels.

Hue-preserving and saturation-improved color histogram equalization algorithm.

In this paper, an algorithm is proposed to improve contrast and saturation without color degradation. The local histogram equalization (HE) method offers better performance than the global HE method, whereas the local HE method sometimes produces undesirable results due to the block-based processing. The proposed contrast-enhancement (CE) algorithm reflects the characteristics of the global HE method in the local HE method to avoid the artifacts, while global and local contrasts are enhanced. There are two ways to apply the proposed CE algorithm to color images. One is luminance processing methods, and the other one is each channel processing methods. However, these ways incur excessive or reduced saturation and color degradation problems. The proposed algorithm solves these problems by using channel adaptive equalization and similarity of ratios between the channels. Experimental results show that the proposed algorithm enhances contrast and saturation while preserving the hue and producing better performance than existing methods in terms of objective evaluation metrics.

Color Restoration of RGBN Multispectral Filter Array Sensor Images Based on Spectral Decomposition.

A multispectral filter array (MSFA) image sensor with red, green, blue and near-infrared (NIR) filters is useful for various imaging applications with the advantages that it obtains color information and NIR information simultaneously. Because the MSFA image sensor needs to acquire invisible band information, it is necessary to remove the IR cut-offfilter (IRCF). However, without the IRCF, the color of the image is desaturated by the interference of the additional NIR component of each RGB color channel. To overcome color degradation, a signal processing approach is required to restore natural color by removing the unwanted NIR contribution to the RGB color channels while the additional NIR information remains in the N channel. Thus, in this paper, we propose a color restoration method for an imaging system based on the MSFA image sensor with RGBN filters. To remove the unnecessary NIR component in each RGB color channel, spectral estimation and spectral decomposition are performed based on the spectral characteristics of the MSFA sensor. The proposed color restoration method estimates the spectral intensity in NIR band and recovers hue and color saturation by decomposing the visible band component and the NIR band component in each RGB color channel. The experimental results show that the proposed method effectively restores natural color and minimizes angular errors.

Colorization-based compression using optimization.

In this paper, we formulate the colorization-based coding problem into an optimization problem, i.e., an L1 minimization problem. In colorization-based coding, the encoder chooses a few representative pixels (RP) for which the chrominance values and the positions are sent to the decoder, whereas in the decoder, the chrominance values for all the pixels are reconstructed by colorization methods. The main issue in colorization-based coding is how to extract the RP well therefore the compression rate and the quality of the reconstructed color image becomes good. By formulating the colorization-based coding into an L1 minimization problem, it is guaranteed that, given the colorization matrix, the chosen set of RP becomes the optimal set in the sense that it minimizes the error between the original and the reconstructed color image. In other words, for a fixed error value and a given colorization matrix, the chosen set of RP is the smallest set possible. We also propose a method to construct the colorization matrix that colorizes the image in a multiscale manner. This, combined with the proposed RP extraction method, allows us to choose a very small set of RP. It is shown experimentally that the proposed method outperforms conventional colorization-based coding methods as well as the JPEG standard and is comparable with the JPEG2000 compression standard, both in terms of the compression rate and the quality of the reconstructed color image.

Correction of axial and lateral chromatic aberration with false color filtering.

In this paper, we propose a chromatic aberration (CA) correction algorithm based on a false color filtering technique. In general, CA produces color distortions called color fringes near the contrasting edges of captured images, and these distortions cause false color artifacts. In the proposed method, a false color filtering technique is used to filter out the false color components from the chroma-signals of the input image. The filtering process is performed with the adaptive weights obtained from both the gradient and color differences, and the weights are designed to reduce the various types of color fringes regardless of the colors of the artifacts. Moreover, as preprocessors of the filtering process, a transient improvement (TI) technique is applied to enhance the slow transitions of the red and blue channels that are blurred by the CA. The TI process improves the filtering performance by narrowing the false color regions before the filtering process when severe color fringes (typically purple fringes) occur widely. Last, the CA-corrected chroma-signal is combined with the TI chroma-signal to avoid incorrect color adjustment. The experimental results show that the proposed method substantially reduces the CA artifacts and provides natural-looking replacement colors, while it avoids incorrect color adjustment.

Spatially adaptive high-resolution image reconstruction of DCT-based compressed images.

The problem of recovering a high-resolution image from a sequence of low-resolution DCT-based compressed observations is considered in this paper. The introduction of compression complicates the recovery problem. We analyze the DCT quantization noise and propose to model it in the spatial domain as a colored Gaussian process. This allows us to estimate the quantization noise at low bit-rates without explicit knowledge of the original image frame, and we propose a method that simultaneously estimates the quantization noise along with the high-resolution data. We also incorporate a nonstationary image prior model to address blocking and ringing artifacts while still preserving edges. To facilitate the simultaneous estimate, we employ a regularization functional to determine the regularization parameter without any prior knowledge of the reconstruction procedure. The smoothing functional to be minimized is then formulated to have a global minimizer in spite of its nonlinearity by enforcing convergence and convexity requirements. Experiments illustrate the benefit of the proposed method when compared to traditional high-resolution image reconstruction methods. Quantitative and qualitative comparisons are provided.