Backpropagation algorithms and Reservoir Computing in Recurrent Neural Networks for the forecasting of complex spatiotemporal dynamics.

We examine the efficiency of Recurrent Neural Networks in forecasting the spatiotemporal dynamics of high dimensional and reduced order complex systems using Reservoir Computing (RC) and Backpropagation through time (BPTT) for gated network architectures. We highlight advantages and limitations of each method and discuss their implementation for parallel computing architectures. We quantify the relative prediction accuracy of these algorithms for the long-term forecasting of chaotic systems using as benchmarks the Lorenz-96 and the Kuramoto-Sivashinsky (KS) equations. We find that, when the full state dynamics are available for training, RC outperforms BPTT approaches in terms of predictive performance and in capturing of the long-term statistics, while at the same time requiring much less training time. However, in the case of reduced order data, large scale RC models can be unstable and more likely than the BPTT algorithms to diverge. In contrast, RNNs trained via BPTT show superior forecasting abilities and capture well the dynamics of reduced order systems. Furthermore, the present study quantifies for the first time the Lyapunov Spectrum of the KS equation with BPTT, achieving similar accuracy as RC. This study establishes that RNNs are a potent computational framework for the learning and forecasting of complex spatiotemporal systems.

Fractal properties of robust strange nonchaotic attractors.

For a simple class of quasiperiodically forced dynamical systems, we present a rigorous result supporting the idea that the attractors for this class of systems, although nonchaotic, are strange in the sense that their box-counting dimension is two while their information dimension is one. Furthermore, this result is stable to changes of the system, suggesting that the basic features leading to it may be present in typical quasiperiodically forced systems.

Local low dimensionality of atmospheric dynamics.

A statistic, the BV (bred vector) dimension, is introduced to measure the effective local finite-time dimensionality of a spatiotemporally chaotic system. It is shown that the Earth's atmosphere often has low BV dimension, and the implications for improving weather forecasting are discussed.

Blur identification from vector quantizer encoder distortion.

Blur identification is a crucial first step in many image restoration techniques. An approach for identifying image blur using vector quantizer encoder distortion is proposed. The blur in an image is identified by choosing from a finite set of candidate blur functions. The method requires a set of training images produced by each of the blur candidates. Each of these sets is used to train a vector quantizer codebook. Given an image degraded by unknown blur, it is first encoded with each of these codebooks. The blur in the image is then estimated by choosing from among the candidates, the one corresponding to the codebook that provides the lowest encoder distortion. Simulations are performed at various bit rates and with different levels of noise. Results show that the method performs well even at a signal-to-noise ratio (SNR) as low as 10 dB.

Image restoration with the Viterbi algorithm.

The Viterbi algorithm (VA) is known to given an optimal solution to the problem of estimating one-dimensional sequences of discrete-valued pixels corrupted by finite-support blur and memoryless noise. A row-by-row estimation along with decision feedback and vector quantization is used to reduce the computational complexity of the VA and allow the estimation of two-dimensional images. This reduced-complexity VA (RCVA) is shown to produce near-optimal estimation of random binary images. In addition, simulated restorations of gray-scale images show the RCVA estimates to be an improvement over the estimates obtained by the conventional Wiener filter (WF). Unlike the WF, the RCVA is capable of superresolution and is adaptable for use in restoring data from signal-dependent Poisson noise corruption. Experimental restorations of random binary data gathered from an optical imaging system support the simulations and show that the RCVA estimate has fewer than one third of the errors of the WF estimate.

Lapped nonlinear interpolative vector quantization and image super-resolution.

This correspondence presents an improved version of an algorithm designed to perform image restoration via nonlinear interpolative vector quantization (NLIVQ). The improvement results from using lapped blocks during the decoding process. The algorithm is trained on original and diffraction-limited image pairs. The discrete cosine transform is again used in the codebook design process to control complexity. Simulation results are presented which demonstrate improvements over the nonlapped algorithm in both observed image quality and peak signal-to-noise ratio. In addition, the nonlinearity of the algorithm is shown to produce super-resolution in the restored images.

Superresolution of binary images with a nonlinear interpolative neural network.

Superresolution is the process by which the bandwidth of a diffraction-limited spectrum is extended beyond the optical passband. Many algorithms exist that are capable of superresolution; however, most are iterative methods, which are ill suited for real-time operation. One approach that has been virtually ignored is the neural-network approach. We consider the feedforward architecture known as a multilayer perceptron and present results on simulated binary images blurred by a diffraction-limited, circular-aperture optical transfer function and sampled at the Nyquist rate. To avoid aliasing, the network performs as a nonlinear spatial interpolator while simultaneously extrapolating in the frequency domain.

Training of a neural network for image superresolution based on a nonlinear interpolative vector quantizer.

Superresolution is the process of extending the spectrum of a diffraction-limited image beyond the optical passband. We consider the neural-network approach to accomplish superresolution and present results on simulated gray-scale images degraded by diffraction blur and additive noise. Images are assumed to be sampled at the Nyquist rate, which requires spatial interpolation for avoiding aliasing, in addition to frequency-domain extrapolation. A novel, to our knowledge, use of vector quantization for the generation of training data sets is also presented. This is accomplished by training of a nonlinear vector quantizer, whose codebooks are subsequently used in the supervised training of the neural network with backpropagation.

Box-counting dimension without boxes: computing D0 from average expansion rates.

We propose an efficient iterative scheme for calculating the box-counting (capacity) dimension of a chaotic attractor in terms of its average expansion rates. Similar to the Kaplan-Yorke conjecture for the information dimension, this scheme provides a connection between a geometric property of a strange set and its underlying dynamical properties. Our conjecture is demonstrated analytically with an exactly solvable two-dimensional hyperbolic map, and numerically with a more complicated higher-dimensional nonhyperbolic map.

Universal trellis coded quantization.

A new form of trellis coded quantization based on uniform quantization thresholds and "on-the-fly" quantizer training is presented. The universal trellis coded quantization (UTCQ) technique requires neither stored codebooks nor a computationally intense codebook design algorithm. Its performance is comparable with that of fully optimized entropy-constrained trellis coded quantization (ECTCQ) for most encoding rates. The codebook and trellis geometry of UTCQ are symmetric with respect to the trellis superset. This allows sources with a symmetric probability density to be encoded with a single variable-rate code. Rate allocation and quantizer modeling procedures are given for UTCQ which allow access to continuous quantization rates. An image coding application based on adaptive wavelet coefficient subblock classification, arithmetic coding, and UTCQ is presented. The excellent performance of this coder demonstrates the efficacy of UTCQ. We also present a simple scheme to improve the perceptual performance of UTCQ for certain imagery at low bit rates. This scheme has the added advantage of being applied during image decoding, without the need to reencode the original image.

Image reconstruction from pairs of Fourier-transform magnitude.

The retrieval of phase information from only the magnitude of the Fourier transform of a signal remains an important problem for many applications. We present an algorithm for phase retrieval when there exist two related sets of Fourier-transform magnitude data. The data are assumed to come from a single object observed in two different polarizations through a distorting medium, so the phase component of the Fourier transform of the object is corrupted. Phase retrieval is accomplished by minimization of a suitable criterion function, which can take three different forms.

A vector quantizer for image restoration.

This paper presents a novel technique for image restoration based on nonlinear interpolative vector quantization (NLIVQ). The algorithm performs nonlinear restoration of diffraction-limited images concurrently with quantization. It is trained on image pairs consisting of an original image and its diffraction-limited counterpart. The discrete cosine transform is used in the codebook design process to control complexity. Simulation results are presented that demonstrate improvements in visual quality and peak signal-to-noise ratio of the restored images.

Hyperspectral image compression using entropy-constrained predictive trellis coded quantization.

A training-sequence-based entropy-constrained predictive trellis coded quantization (ECPTCQ) scheme is presented for encoding autoregressive sources. For encoding a first-order Gauss-Markov source, the mean squared error (MSE) performance of an eight-state ECPTCQ system exceeds that of entropy-constrained differential pulse code modulation (ECDPCM) by up to 1.0 dB. In addition, a hyperspectral image compression system is developed, which utilizes ECPTCQ. A hyperspectral image sequence compressed at 0.125 b/pixel/band retains an average peak signal-to-noise ratio (PSNR) of greater than 43 dB over the spectral bands.

A multiresolution approach to computer verification of handwritten signatures.

We took a multi-resolution approach to the signature verification problem. The top-level representation of signatures was the global geometric features. A multi-resolution representation of signatures was obtained using the wavelet transformation. We built VQ and network classifiers to demonstrate the advantages of the multi-resolution approach. High verification rates were achieved based on a limited database.

Image coding using adaptive recursive interpolative DPCM.

A predictive image coder having minimal decoder complexity is presented. The image coder utilizes recursive interpolative DPCM in conjunction with adaptive classification, entropy-constrained trellis coded quantization, and optimal rate allocation to obtain signal-to-noise ratios (SNRs) in the range of those provided by the most advanced transform coders.

Signal-detection tradeoff-analysis of optical vs digital Fourier transform computers.

Signal detection theory is used to develop analytic models which yield comparisons between optical and digital Fourier transform computers in terms of their ability to detect transformed signals within small spectral regions of their Fourier domains. Stochastic noise models are first given describing the quantization noise introduced by the finite register length involved in a digital transformation. The signal detection models are then developed which describe the detectability of a transformed signal among this kind of noise, with models given for fixed-point and floating-point machines and for the signal-known-exactly and the signalunknown detection cases. These models provide the optimum detection statistic to be used in each case, a means for choosing the cutoff points used in the detection process, the over-all performance curve of the detector, and detection indices which summarize this performance. The optical and digital computers are compared by equating their detectabilities as obtained from these models, thus allowing a digital processor with given specifications to be paired with an optical processor with a specific SNR in its output plane. Analytical results are presented demonstrating these comparisons in which computer number-representation, register length, transform-array size, detection-array size, and type of detection are the independent variables under consideration.

Optical computing for image bandwidth compression: analysis and simulation.

Image bandwidth compression is dominated by digital methods for carrying out the required computations. This paper discusses the general problem of using optics to realize the computations in bandwidth compression. A common method of digital bandwidth compression, feedback differential pulse code modulation (DPCM), is reviewed, and the obstacles to making a direct optical analogy to feedback DPCM are discussed. Instead of a direct optical analogy to DPCM, an optical system which captures the essential features of DPCM without optical feedback is introduced. The essential features of this incoherent optical system are encoding of low-frequency information and generation of difference samples which can be coded with a small number of bits. A simulation of this optical system by means of digital image processing is presented, and performance data are also included.

Comparison of image restoration methods.

Three image restoration methods are compared in a variety of blur and noise conditions. Both numerical and subjective data are evaluated. It is demonstrated that, in certain conditions, one restoration method is preferable to others.