Parallel cascade identification and its application to protein family prediction.

Parallel cascade identification is a method for modeling dynamic systems with possibly high order nonlinearities and lengthy memory, given only input/output data for the system gathered in an experiment. While the method was originally proposed for nonlinear system identification, two recent papers have illustrated its utility for protein family prediction. One strength of this approach is the capability of training effective parallel cascade classifiers from very little training data. Indeed, when the amount of training exemplars is limited, and when distinctions between a small number of categories suffice, parallel cascade identification can outperform some state-of-the-art techniques. Moreover, the unusual approach taken by this method enables it to be effectively combined with other techniques to significantly improve accuracy. In this paper, parallel cascade identification is first reviewed, and its use in a variety of different fields is surveyed. Then protein family prediction via this method is considered in detail, and some particularly useful applications are pointed out.

On the use of separable Volterra networks to model discrete-time Volterra systems.

A paper by Marmarelis and Zhao (1997) describes the use of what the authors call a "separable Volterra network" for modeling high-order Volterra systems. This model is identical to a parallel cascade of dynamic linear/polynomial static nonlinear elements, which has been extensively studied since 1982 for the same purpose.

Automatic classification of protein sequences into structure/function groups via parallel cascade identification: a feasibility study.

A recent paper introduced the approach of using nonlinear system identification as a means for automatically classifying protein sequences into their structure/function families. The particular technique utilized, known as parallel cascade identification (PCI), could train classifiers on a very limited set of exemplars from the protein families to be distinguished and still achieve impressively good two-way classifications. For the nonlinear system classifiers to have numerical inputs, each amino acid in the protein was mapped into a corresponding hydrophobicity value, and the resulting hydrophobicity profile was used in place of the primary amino acid sequence. While the ensuing classification accuracy was gratifying, the use of (Rose scale) hydrophobicity values had some disadvantages. These included representing multiple amino acids by the same value, weighting some amino acids more heavily than others, and covering a narrow numerical range, resulting in a poor input for system identification. This paper introduces binary and multilevel sequence codes to represent amino acids, for use in protein classification. The new binary and multilevel sequences, which are still able to encode information such as hydrophobicity, polarity, and charge, avoid the above disadvantages and increase classification accuracy. Indeed, over a much larger test set than in the original study, parallel cascade models using numerical profiles constructed with the new codes achieved slightly higher two-way classification rates than did hidden Markov models (HMMs) using the primary amino acid sequences, and combining PCI and HMM approaches increased accuracy.

Iterative fast orthogonal search algorithm for MDL-based training of generalized single-layer networks.

The generalized single-layer network (GSLN) architecture, which implements a sum of arbitrary basis functions defined on its inputs, is potentially a flexible and efficient structure for approximating arbitrary nonlinear functions. A drawback of GSLNs is that a large number of weights and basis functions may be required to provide satisfactory approximations. In this paper, we present a new approach in which an algorithm known as iterative fast orthogonal search (IFOS) is coupled with the minimum description length (MDL) criterion to provide automatic structure selection and parameter estimation for GSLNs. The resulting algorithm, dubbed IFOS-MDL, performs both network growth and pruning to construct sparse GSLNs from potentially large spaces of candidate basis functions.

3D-1D threading methods for protein fold recognition.

The threading approach to protein fold recognition attempts to evaluate how well a query sequence fits into an already-solved fold. 3D-1D threaders rely on matching 1-dimensional strings of 3-dimensional information predicted from the query sequence with corresponding features of the target structure. In many cases this is combined with a sequence comparison. The combination of sequence and structure information has been shown to improve the accuracy of fold recognition, relative to the exclusive use of sequence or structure. In this paper, we review progress made since the introduction of threading methods a decade ago, highlighting recent advances. We focus on two emerging methods that are unconventional 3D-1D threaders: proximity correlation matrices and parallel cascade identification.

Two methods for identifying Wiener cascades having noninvertible static nonlinearities.

Two methods are proposed for identifying the component elements of a Wiener cascade that is comprised of a dynamic linear element (L) followed by a static nonlinearity (N). Both methods avoid potential problems of instability in a procedure presented by Paulin [M. G. Paulin, Biol. Cybern. 69: 67-76, 1993], which itself is a modification of a method described earlier by Hunter and Korenberg [I. W. Hunter and M. J. Korenberg, Biol. Cybern. 55: 135-144, 1996]. The latter method is a rapidly convergent iterative procedure that produces accurate estimates of the L and N elements from short data records, provided that the static nonlinearity N is invertible. Subsequently, Paulin introduced a modification that removed this limitation and enabled identification of Wiener cascades with nonmonotonic static nonlinearities. However, Paulin presented his modification employing an autoregressive moving average (ARMA) model representation for the dynamic linear element. To remove the possibility that the estimated ARMA model could be unstable, we recast the procedure by utilizing instead a rapid method for finding an impulse response representation for the dynamic linear element. However, in this form the procedure did not have good convergence properties, so we introduced two key ideas, both of which provide effective alternatives for identifying Wiener cascades whether or not the static nonlinearities therein are invertible. The new procedures are illustrated on challenging examples involving high-degree polynomial static nonlinearities, of odd or even symmetry, a high-pass linear element, and output noise corruption of 50%.

Iterative fast orthogonal search for modeling by a sum of exponentials or sinusoids.

Accurate sinusoidal series models of biological time-series data may be obtained using a modeling algorithm known as fast orthogonal search (FOS). FOS does not require equally spaced data, and can resolve sinusoidal frequencies much more closely spaced than can a discrete Fourier transform. FOS has been less successful at obtaining accurate exponential series models. We here consider a modification of FOS in which iteration of the original procedure is used to further reduce the mean-squared error (m.s.e.) between model and data, approaching a minimum in the m.s.e. Iteration of the FOS procedure greatly improves the accuracy of estimated exponential series models. The application of iterative FOS (IFOS) to exponential and sinusoidal series models is described. Finally, the use of FOS and IFOS procedures for finding a single model from the results of multiple experiments is described.

Two methods for calculating the responses of photoreceptors to moving objects.

The responses of photoreceptor cells to moving stimuli are crucial to understanding motion detection in visual systems. However, these responses are not well characterized quantitatively because they result from a combination of spatial optical behavior in the lens systems with temporal behavior in the phototransduction mechanism. While both these processes can now be modeled quite well by relatively simple equations, their combination cannot be easily obtained in a closed form. Here, we present two approaches to this problem, based on well-established models for the lens and photoreceptors systems of the fly compound eye. The first approach leads to a recursive formula for predicting the photoreceptor response to a moving point object. The second method is approximate, but almost equally accurate and more rapid.

Processing of color- and noncolor-coded signals in the gourami retina. III. Ganglion cells.

The dynamics of intracellular responses from ganglion cells, as well as that of spike discharges, were studied with the stimulus regimens and analytic procedures identical to those used to study the dynamics of the responses from horizontal and amacrine cells (,). The stimuli used were large fields of red and green light given as a pulsatile input or modulation about a mean luminance by a white-noise signal. Spike discharges evoked by a white-noise stimulus were analyzed in exactly the same manner as that used for analysis of analog responses. The canonical nature of kernels allowed us to correlate the first- and second-order components in a spike train with those of the intracellular responses from horizontal, amacrine, and ganglion cells. Both red and green stimuli given alone in darkness produced noncolor-coded responses from all ganglion cells. In the case of some cells, steady red illumination changed the polarity or waveform of the response to green light. Color-coded ganglions responded only to simultaneous color contrast. Nonlinearities recovered from intracellular responses, and spike discharges were similar to those found in responses from amacrine cells and were of two types, one characteristic of the C amacrine cells and the other characteristic of the N amacrine cells. The first-order kernels of most ganglion cells could be divided into two basic types, biphasic and triphasic. The combination of kernels of these two basic types with different polarities can produce a wide range of responses. Addition of two types of second-order nonlinearity could render color coding in this relatively simple retina as an extremely complex process. Color information appeared to be represented by the polarity, as well as the waveform, of the first-order kernel. The response dynamics is a means of transmission of color-coded information. Second-order components carry information about changes around a mean luminance regardless of the color of an input. Some spike discharges produced a well-defined cross-kernel between red and green inputs to show that a particular time sequence of red and green stimuli was detected by the retinal neuron network. The similarity between signatures of second-order kernels for both amacrine and ganglion cells indicates that signals undergo a minimal transformation in the temporal domain when they are transmitted from amacrine to ganglion cells and then transformed into a spike train. Under our experimental conditions, a single spike train carried simultaneously information about red and green inputs, as well as about linear and nonlinear components. In addition, the spike train also carries a cross-talk component. A spike train is a carrier of multiple signals. Conversely, many types of information in a stimulus are independently encoded into a spike train.

Application of fast orthogonal search to linear and nonlinear stochastic systems.

Standard deterministic autoregressive moving average (ARMA) models consider prediction errors to be unexplainable noise sources. The accuracy of the estimated ARMA model parameters depends on producing minimum prediction errors. In this study, an accurate algorithm is developed for estimating linear and nonlinear stochastic ARMA model parameters by using a method known as fast orthogonal search, with an extended model containing prediction errors as part of the model estimation process. The extended algorithm uses fast orthogonal search in a two-step procedure in which deterministic terms in the nonlinear difference equation model are first identified and then reestimated, this time in a model containing the prediction errors. Since the extended algorithm uses an orthogonal procedure, together with automatic model order selection criteria, the significant model terms are estimated efficiently and accurately. The model order selection criteria developed for the extended algorithm are also crucial in obtaining accurate parameter estimates. Several simulated examples are presented to demonstrate the efficacy of the algorithm.

Automatic discrimination of myoelectric signals via parallel cascade identification.

It has recently been shown that it is possible to discriminate accurately among myoelectric signals underlying different muscle contraction types, specifically elbow flexion and extension and forearm pronation and supination. It was reported that once a number of distinctive features had been extracted from the myoelectric signals, a neural network could be trained to distinguish the contraction types with an impressively high accuracy. In the present paper, we show that a technique known as parallel cascade identification can be used to construct classifiers that can also accurately differentiate the contraction types. The use of parallel cascades has the benefit of dispensing with the need for feature extraction, so that raw myoelectric signal data can be used directly. In addition, very little data are required to train the parallel cascades to distinguish accurately novel incoming myoelectric signals. Results of using parallel cascades to distinguish forearm pronation, supination, and elbow flexion are presented.

Complexity and frequency hierarchies in the catfish retina.

The intricate connectivity and interactions between neurons in the vertebrate retina have made their individual roles in signal processing very difficult to elucidate. We have used a recently developed mathematical tool, fast orthogonal search (FOS), to probe the catfish outer (distal) and inner (proximal) retina, and study the signal processing within. Through FOS, a given waveform can be decomposed into a parsimonious sinusoidal series containing the most significant constituent frequencies. In particular, we examined the light-evoked first-order Wiener kernels of horizontal cells and on-bipolar cells, and on-off, off- and on-amacrine and ganglion cells. Here we report a hierarchy (correlation coefficient up to 0.86) in preferred frequency and complexity of response corresponding to the retina's structural hierarchy. In addition, clear differences between on-, on-off and off-cell functional characteristics were detected. For example, the kernel waveform for the on-amacrine cell was found to be more complex and to have a higher preferred frequency than that for the off-amacrine cell. Indeed FOS analysis revealed that both off- (sustained) amacrine and off-ganglion cells exhibit significantly less complexity in their waveforms for signal processing of light input than do the corresponding on- and on-off cells. This shows a clear breakdown in symmetry between on- and off-pathways, and suggests that connections to off-cells may provide fewer or a smaller variety of inputs than those to on- and on-off cells. Many of our new findings can be appreciated by assuming an underlying cascade structure for the retinal information processing. The FOS findings in particular support the following previously advanced hypothesis: the transition in nonlinear processing from on-off amacrine to on- off-amacrine cells is due to high-pass linear filtering. Furthermore, our results indicate that the high-pass filtering is more sharply differentiating for the on-amacrine than for the off-amacrine cell.

The identification of nonlinear biological systems: Volterra kernel approaches.

Representation, identification, and modeling are investigated for nonlinear biomedical systems. We begin by considering the conditions under which a nonlinear system can be represented or accurately approximated by a Volterra series (or functional expansion). Next, we examine system identification through estimating the kernels in a Volterra functional expansion approximation for the system. A recent kernel estimation technique that has proved to be effective in a number of biomedical applications is investigated as to running time and demonstrated on both clean and noisy data records, then it is used to illustrate identification of cascades of alternating dynamic linear and static nonlinear systems, both single-input single-output and multivariable cascades. During the presentation, we critically examine some interesting biological applications of kernel estimation techniques.

The identification of nonlinear biological systems: Volterra kernel approaches.

Representation, identification, and modeling are investigated for nonlinear biomedical systems. We begin by considering the conditions under which a nonlinear system can be represented or accurately approximated by a Volterra series (or functional expansion). Next, we examine system identification through estimating the kernels in a Volterra functional expansion approximation for the system. A recent kernel estimation technique that has proved to be effective in a number of biomedical applications is investigated as to running time and demonstrated on both clean and noisy data records, then it is used to illustrate identification of cascades of alternating dynamic linear and static nonlinear systems, both single-input single-output and multivariable cascades. During the presentation, we critically examine some interesting biological applications of kernel estimation techniques.

Nonlinear models of the first synapse in the light-adapted fly retina.

1. Randomly modulated light stimuli were used to characterize the nonlinear dynamic properties of the synapse between photoreceptors and large monopolar neurons (LMC) in the fly retina. Membrane potential fluctuations produced by constant variance contrast stimuli were recorded at eight different levels of background light intensity. 2. Representation of the photoreceptor-LMC input-output data in the form of traditional characteristic curves indicated that synaptic gain was reduced by light adaptation. However, this representation did not include the time-dependent properties of the synaptic function, which are known to be nonlinear. Therefore nonlinear systems analysis was used to characterize the synapse. 3. The responses of photoreceptors and LMCs to random light fluctuations were characterized by second-order Volterra series, with kernel estimation by the parallel cascade method. Photoreceptor responses were approximately linear, but LMC responses were clearly nonlinear. 4. Synaptic input-output relationships were measured by passing the light stimuli to LMCs through the measured photoreceptor characteristics to obtain an estimate of the synaptic input. The resulting nonlinear synaptic functions were well characterized by second-order Volterra series. They could not be modeled by a linear-nonlinear-linear cascade but were better approximated by a nonlinear-linear-nonlinear cascade. 5. These results support two possible structural models of the synapse, the first having two parallel paths for signal flow between the photoreceptor and LMC, and the second having two distinct nonlinear operations, occurring before and after chemical transmission. 6. The two models were cach used to calculate the synaptic gain to a brief change in photoreceptor membrane potential. Both models predicted that synaptic gain is reduced by light adaptation.

The dynamic nonlinear behavior of fly photoreceptors evoked by a wide range of light intensities.

Fly photoreceptor cells were stimulated with steps of light over a wide intensity range. First- and second-order Volterra kernels were then computed from sequences of combined step responses. Diagonal values of the second-order Volterra kernels were much greater than the off-diagonal values, and the diagonal values were roughly proportional to the corresponding first-order kernels, suggesting that the response could be approximated by a static nonlinearity followed by a dynamic linear component (Hammerstein model). The amplitudes of the second-order kernels were much smaller in light-adapted than in dark-adapted photoreceptors. Hammerstein models constructed from the step input/output measurements gave reasonable approximations to the actual photoreceptor responses, with light-adapted responses being relatively better fitted. However, Hammerstein models could not account for several features of the photoreceptor behavior, including the dependence of the step response shape on step amplitude. A model containing an additional static nonlinearity after the dynamic linear component gave significantly better fits to the data. These results indicate that blowfly photoreceptors have a strong early gain control nonlinearity acting before the processes that create the characteristic time course of the response, in addition to the nonlinearities caused by membrane conductances.

Identification of complex-cell intensive nonlinearities in a cascade model of cat visual cortex.

Complex cells in the cat's visual cortex show nonlinearities in processing of image luminance and movement. To study mechanisms, initially we have represented the chain of neurons from retina to cortex as a black-box model. Independent information about the visual system has helped us cast this "Wiener-kernel" model into a dynamic-linear/static-nonlinear/dynamic-linear (LNL) cascade. We then use system identification techniques to define the nature of these transformations directly from responses of the neuron to a single presentation of a stimulus composed of a sequence of white-noise-modulated luminance values. The two dynamic linear filters are mainly low-pass, and the static nonlinearity is mainly of even polynomial degree. This approximate squaring function may be effected in the animal by soft-thresholding each of the linear ON- and OFF-channel signals and then summing them, which account for "ON-OFF" responses and for the squaring operation needed for computation of "motion energy", both observed in these neurons.

Parallel cascade identification and kernel estimation for nonlinear systems.

We consider the representation and identification of nonlinear systems through the use of parallel cascades of alternating dynamic linear and static nonlinear elements. Building on the work of Palm and others, we show that any discrete-time finite-memory nonlinear system having a finite-order Volterra series representation can be exactly represented by a finite number of parallel LN cascade paths. Each LN path consists of a dynamic linear system followed by a static nonlinearity (which can be a polynomial). In particular, we provide an upper bound for the number of parallel LN paths required to represent exactly a discrete-time finite-memory Volterra functional of a given order. Next, we show how to obtain a parallel cascade representation of a nonlinear system from a single input-output record. The input is not required to be Gaussian or white, nor to have special autocorrelation properties. Next, our parallel cascade identification is applied to measure accurately the kernels of nonlinear systems (even those with lengthy memory), and to discover the significant terms to include in a nonlinear difference equation model for a system. In addition, the kernel estimation is used as a means of studying individual signals to distinguish deterministic from random behaviour, in an alternative to the use of chaotic dynamics. Finally, an alternate kernel estimation scheme is presented.

Dissection of a nonlinear cascade model for sensory encoding.

Action potential encoding in the cockroach tactile spine neuron may be treated as a single-input, single-output dynamic nonlinear process, where the input is the electric current flowing across the neuronal membrane and the output is the resultant train of action potentials. The nonlinear behavior of the system may be characterized by a functional expansion method which efficiently and accurately yields similar kernels to the Wiener method. A simple nonlinear cascade consisting of sequential dynamic linear, static nonlinear, and dynamic linear components was identified and gives a good approximation to the response of the neuron to random stimulation. Next, we attempted to study the components of the cascade by the use of a drug, phentolamine, which selectively modifies the dynamic behavior of the encoder. Application of phentolamine to the neuron caused a significant change in the first dynamic linear component of the cascade without affecting the other components. The change was much larger than the variability between results obtained from individual animals. This finding has implications for the biophysical processes which are involved in the components of the cascade.

Nonlinear mechanisms for gain adaptation in locust photoreceptors.

Intracellular membrane potential responses were recorded from locust photoreceptors under two stimulus conditions: pairs of flashes to dark-adapted receptors, and white-noise modulated light at a range of background intensities from 500 to 15,000 effective photons per second. Nonlinear analysis of the input-output relationships were performed by estimating the Volterra and Wiener kernels of the system. The Volterra kernels obtained from the double-flash experiments were similar to the Wiener kernels obtained from the white-noise experiments, except for a change of time scale. The structure of the second-order kernels obtained with either method gave evidence for a gain control mechanism acting at an early stage of the cascade. Both feedforward and feedback nonlinearities could account for the observed system behavior at any one background level. The differences in amplitude between the kernels obtained at different background levels could be accounted for by an adaptation process which further decreased the gain of the system, acting on a slower time scale, also at some early stage of the cascade.