Gradient-based residual variance modeling and its applications to motion-compensated video coding.

This paper analyzes the relationship between the residual frame and the previous frame in motion-compensated video coding. It is found that the variance of the residual signal depends on the gradient magnitude. On average, the variance of the residual signal is larger for pixels with larger gradient magnitude. Two applications of this analysis are presented. In the first one, the relationship between the residual signal variance and the gradient magnitude is used to model the second-order statistics of the residual field in a nonstationary way. This modeling enables more efficient residual signal coding. The other application is for pixel decimation-based fast block matching. It is proposed that pixels with the largest gradient magnitude in a block be chosen to participate in the block matching process. It is demonstrated that such a gradient-adaptive subsampling achieves great advantage over two other known subsampling methods.

A parametric solution for optimal overlapped block motion compensation.

We find the optimal window for overlapped block motion compensation (OBMC) by statistically modeling the motion field, the field of block motion estimates and their relationship. This enables us to show how the optimal OBMC window is affected by random field parameters, such as the variance of the motion field and the correlation coefficients of both the intensity field and the motion field. The OBMC window obtained in this fashion is shown to have good performance in reducing the prediction error. Furthermore, this parametric solution provides insight into motion uncertainty and the overlapped motion compensation process.

Multiple description coding using pairwise correlating transforms.

The objective of multiple description coding (MDC) is to encode a source into multiple bitstreams supporting multiple quality levels of decoding. In this paper, we only consider the two-description case, where the requirement is that a high-quality reconstruction should be decodable from the two bitstreams together, while lower, but still acceptable, quality reconstructions should be decodable from either of the two individual bitstreams. This paper describes techniques for meeting MDC objectives in the framework of standard transform-based image coding through the design of pairwise correlating transforms. The correlation introduced by the transform helps to reduce the distortion when only a single description is received, but it also increases the bit rate beyond that prescribed by the rate-distortion function of the source. We analyze the relation between the redundancy (i.e., the extra bit rate) and the single description distortion using this transform-based framework. We also describe an image coder that incorporates the pairwise transform and show its redundancy-rate-distortion performance for real images.

New edge-directed interpolation.

This paper proposes an edge-directed interpolation algorithm for natural images. The basic idea is to first estimate local covariance coefficients from a low-resolution image and then use these covariance estimates to adapt the interpolation at a higher resolution based on the geometric duality between the low-resolution covariance and the high-resolution covariance. The edge-directed property of covariance-based adaptation attributes to its capability of tuning the interpolation coefficients to match an arbitrarily oriented step edge. A hybrid approach of switching between bilinear interpolation and covariance-based adaptive interpolation is proposed to reduce the overall computational complexity. Two important applications of the new interpolation algorithm are studied: resolution enhancement of grayscale images and reconstruction of color images from CCD samples. Simulation results demonstrate that our new interpolation algorithm substantially improves the subjective quality of the interpolated images over conventional linear interpolation.

Inverse halftoning using wavelets.

This work introduces a new approach to inverse halftoning using nonorthogonal wavelets. The distinct features of this wavelet-based approach are: 1) edge information in the highpass wavelet images of a halftone image is extracted and used to assist inverse halftoning, 2) cross-scale correlations in the multiscale wavelet decomposition are used for removing background halftoning noise while preserving important edges in the wavelet lowpass image, and 3) experiments show that our simple wavelet-based approach outperforms the best results obtained from inverse halftoning methods published in the literature, which are iterative in nature.

Image coding based on a morphological representation of wavelet data.

In this paper, an experimental study of the statistical properties of wavelet coefficients of image data is presented, as well as the design of two different morphology-based image coding algorithms that make use of these statistics. A salient feature of the proposed methods is that, by a simple change of quantizers, the same basic algorithm yields high performance embedded or fixed rate coders. Another important feature is that the shape information of morphological sets used in this coder is encoded implicitly by the values of wavelet coefficients, thus avoiding the use of explicit and rate expensive shape descriptors. These proposed algorithms, while achieving nearly the same objective performance of state-of-the-art zerotree based methods, are able to produce reconstructions of a somewhat superior perceptual quality, due to a property of joint compression and noise reduction they exhibit.

Wavelet packet image coding using space-frequency quantization.

We extend our previous work on space-frequency quantization (SFQ) for image coding from wavelet transforms to the more general wavelet packet transforms. The resulting wavelet packet coder offers a universal transform coding framework within the constraints of filterbank structures by allowing joint transform and quantizer design without assuming a priori statistics of the input image. In other words, the new coder adaptively chooses the representation to suit the image and the quantization to suit the representation. Experimental results show that, for some image classes, our new coder gives excellent coding performance.

Motion field modeling for video sequences.

We propose a model for the interframe correspondences existing between pixels of an image sequence. These correspondences form the elements of a field called the motion field. In our model, spatial neighborhoods of motion elements are related based on a generalization of autoregressive (AR) modeling of the time-series. We also propose a joint spatio-temporal model by including spatial neighborhoods of pixel intensities in the motion model. A fundamental difference of our approach with most previous approaches to modeling motion is in basing our model on concepts from statistical signal processing. The developments in this paper give rise to the promise of extending well-understood tools of signal processing (e.g., filtering) to the analysis and processing of motion fields. Simulation results presented show the performance of our models in interframe prediction; specifically, on average the motion model performs 29% better in terms of the mean squared error energy over a commonly used pel-recursive approach. The spatio-temporal model improves the prediction efficiencies by 8% over the motion model. Our model can also be used to obtain estimates of the optical flow field as the simulations demonstrate.

Optimized nonorthogonal transforms for image compression.

The transform coding of images is analyzed from a common standpoint in order to generate a framework for the design of optimal transforms. It is argued that all transform coders are alike in the way they manipulate the data structure formed by transform coefficients. A general energy compaction measure is proposed to generate optimized transforms with desirable characteristics particularly suited to the simple transform coding operation of scalar quantization and entropy coding. It is shown that the optimal linear decoder (inverse transform) must be an optimal linear estimator, independent of the structure of the transform generating the coefficients. A formulation that sequentially optimizes the transforms is presented, and design equations and algorithms for its computation provided. The properties of the resulting transform systems are investigated. In particular, it is shown that the resulting basis are nonorthogonal and complete, producing energy compaction optimized, decorrelated transform coefficients. Quantization issues related to nonorthogonal expansion coefficients are addressed with a simple, efficient algorithm. Two implementations are discussed, and image coding examples are given. It is shown that the proposed design framework results in systems with superior energy compaction properties and excellent coding results.

Space-frequency quantization for wavelet image coding.

A new class of image coding algorithms coupling standard scalar quantization of frequency coefficients with tree-structured quantization (related to spatial structures) has attracted wide attention because its good performance appears to confirm the promised efficiencies of hierarchical representation. This paper addresses the problem of how spatial quantization modes and standard scalar quantization can be applied in a jointly optimal fashion in an image coder. We consider zerotree quantization (zeroing out tree-structured sets of wavelet coefficients) and the simplest form of scalar quantization (a single common uniform scalar quantizer applied to all nonzeroed coefficients), and we formalize the problem of optimizing their joint application. We develop an image coding algorithm for solving the resulting optimization problem. Despite the basic form of the two quantizers considered, the resulting algorithm demonstrates coding performance that is competitive, often outperforming the very best coding algorithms in the literature.

Joint space-frequency segmentation using balanced wavelet packet trees for least-cost image representation.

We examine the question of how to choose a space varying filterbank tree representation that minimizes some additive cost function for an image. The idea is that for a particular cost function, e.g., energy compaction or quantization distortion, some tree structures perform better than others. While the wavelet tree represents a good choice for many signals, it is generally outperformed by the best tree from the library of wavelet packet frequency-selective trees. The double-tree library of bases performs better still, by allowing different wavelet packet trees over all binary spatial segments of the image. We build on this foundation and present efficient new pruning algorithms for both one- and two-dimensional (1-D and 2-D) trees that will find the best basis from a library that is many times larger than the library of the single-tree or double-tree algorithms. The augmentation of the library of bases overcomes the constrained nature of the spatial variation in the double-tree bases, and is a significant enhancement in practice. Use of these algorithms to select the least-cost expansion for images with a rate-distortion cost function gives a very effective signal adaptive compression scheme. This scheme is universal in the sense that, without assuming a model for the signal or making use of training data, it performs very well over a large class of signal types. In experiments it achieves compression rates that are competitive with the best training-based schemes.

Interframe coding of magnetic resonance images.

Presents a new interframe coding method for medical images, in particular magnetic resonance (MR) images. Until now, attempts in using interframe redundancies for coding MR images have been unsuccessful. The authors believe that the main reason for this is twofold: unsuitable interframe estimation models and the thermal noise inherent in magnetic resonance imaging (MRI). The interframe model used here is a continuous affine mapping based on (and optimized by) deforming triangles. The inherent noise of MRI is dealt with by using a median filter within the estimation loop. The residue frames are quantized with a zero-tree wavelet coder, which includes arithmetic entropy coding. This particular method of quantization allows for progressive transmission, which aside from avoiding buffer control problems is very attractive in medical imaging applications.

Overlapped block motion compensation: an estimation-theoretic approach.

We present an estimation-theoretic analysis of motion compensation that, when used with fields of block-based motion vectors, leads to the development of overlapped block algorithms with improved compensation accuracy. Overlapped block motion compensation (OBMC) is formulated as a probabilistic linear estimator of pixel intensities given the limited block motion information available to the decoder. Although overlapped techniques have been observed to reduce blocking artifacts in video coding, this analysis establishes for the first time how (and why) OBMC can offer substantial reductions in prediction error as well, even with no change in the encoder's search and no extra side information. Performance can be further enhanced with the use of state variable conditioning in the compensation process. We describe the design of optimized windows for OBMC. We also demonstrate how, with additional encoder complexity, a motion estimation algorithm optimized for OBMC offers further significant gains in compensation accuracy. Overall mean-square prediction improvements in the range of 16 to 40% (0.8 to 2.2 dB) are demonstrated.