Ensemble Conformalized Quantile Regression for Probabilistic Time Series Forecasting.

This article presents a novel probabilistic forecasting method called ensemble conformalized quantile regression (EnCQR). EnCQR constructs distribution-free and approximately marginally valid prediction intervals (PIs), which are suitable for nonstationary and heteroscedastic time series data. EnCQR can be applied on top of a generic forecasting model, including deep learning architectures. EnCQR exploits a bootstrap ensemble estimator, which enables the use of conformal predictors for time series by removing the requirement of data exchangeability. The ensemble learners are implemented as generic machine learning algorithms performing quantile regression (QR), which allow the length of the PIs to adapt to local variability in the data. In the experiments, we predict time series characterized by a different amount of heteroscedasticity. The results demonstrate that EnCQR outperforms models based only on QR or conformal prediction (CP), and it provides sharper, more informative, and valid PIs.

Age prediction by deep learning applied to Greenland halibut (Reinhardtius hippoglossoides) otolith images.

Otoliths (ear-stones) in the inner ears of vertebrates containing visible year zones are used extensively to determine fish age. Analysis of otoliths is a time-consuming and difficult task that requires the education of human experts. Human age estimates are inconsistent, as several readings by the same human expert might result in different ages assigned to the same otolith, in addition to an inherent bias between readers. To improve efficiency and resolve inconsistent results in the age reading from otolith images by human experts, an automated procedure based on convolutional neural networks (CNNs), a class of deep learning models suitable for image processing, is investigated. We applied CNNs that perform image regression to estimate the age of Greenland halibut (Reinhardtius hippoglossoides) with good results for individual ages as well as the overall age distribution, with an average CV of about 10% relative to the read ages by experts. In addition, the density distribution of predicted ages resembles the density distribution of the ground truth. By using k*l-fold cross-validation, we test all available samples, and we show that the results are rather sensitive to the choice of test set. Finally, we apply explanation techniques to analyze the decision process of deep learning models. In particular, we produce heatmaps indicating which input features that are the most important in the computation of predicted age.

Understanding Pooling in Graph Neural Networks.

Many recent works in the field of graph machine learning have introduced pooling operators to reduce the size of graphs. In this article, we present an operational framework to unify this vast and diverse literature by describing pooling operators as the combination of three functions: selection, reduction, and connection (SRC). We then introduce a taxonomy of pooling operators, based on some of their key characteristics and implementation differences under the SRC framework. Finally, we propose three criteria to evaluate the performance of pooling operators and use them to investigate the behavior of different operators on a variety of tasks.

Code-Aligned Autoencoders for Unsupervised Change Detection in Multimodal Remote Sensing Images.

Image translation with convolutional autoencoders has recently been used as an approach to multimodal change detection (CD) in bitemporal satellite images. A main challenge is the alignment of the code spaces by reducing the contribution of change pixels to the learning of the translation function. Many existing approaches train the networks by exploiting supervised information of the change areas, which, however, is not always available. We propose to extract relational pixel information captured by domain-specific affinity matrices at the input and use this to enforce alignment of the code spaces and reduce the impact of change pixels on the learning objective. A change prior is derived in an unsupervised fashion from pixel pair affinities that are comparable across domains. To achieve code space alignment, we enforce pixels with similar affinity relations in the input domains to be correlated also in code space. We demonstrate the utility of this procedure in combination with cycle consistency. The proposed approach is compared with the state-of-the-art machine learning and deep learning algorithms. Experiments conducted on four real and representative datasets show the effectiveness of our methodology.

Hierarchical Representation Learning in Graph Neural Networks With Node Decimation Pooling.

In graph neural networks (GNNs), pooling operators compute local summaries of input graphs to capture their global properties, and they are fundamental for building deep GNNs that learn hierarchical representations. In this work, we propose the Node Decimation Pooling (NDP), a pooling operator for GNNs that generates coarser graphs while preserving the overall graph topology. During training, the GNN learns new node representations and fits them to a pyramid of coarsened graphs, which is computed offline in a preprocessing stage. NDP consists of three steps. First, a node decimation procedure selects the nodes belonging to one side of the partition identified by a spectral algorithm that approximates the MAXCUT solution. Afterward, the selected nodes are connected with Kron reduction to form the coarsened graph. Finally, since the resulting graph is very dense, we apply a sparsification procedure that prunes the adjacency matrix of the coarsened graph to reduce the computational cost in the GNN. Notably, we show that it is possible to remove many edges without significantly altering the graph structure. Experimental results show that NDP is more efficient compared to state-of-the-art graph pooling operators while reaching, at the same time, competitive performance on a significant variety of graph classification tasks.

Graph Neural Networks With Convolutional ARMA Filters.

Popular graph neural networks implement convolution operations on graphs based on polynomial spectral filters. In this paper, we propose a novel graph convolutional layer inspired by the auto-regressive moving average (ARMA) filter that, compared to polynomial ones, provides a more flexible frequency response, is more robust to noise, and better captures the global graph structure. We propose a graph neural network implementation of the ARMA filter with a recursive and distributed formulation, obtaining a convolutional layer that is efficient to train, localized in the node space, and can be transferred to new graphs at test time. We perform a spectral analysis to study the filtering effect of the proposed ARMA layer and report experiments on four downstream tasks: semi-supervised node classification, graph signal classification, graph classification, and graph regression. Results show that the proposed ARMA layer brings significant improvements over graph neural networks based on polynomial filters.

Estimation of Excess Mortality and Years of Life Lost to COVID-19 in Norway and Sweden between March and November 2020.

We estimate the weekly excess all-cause mortality in Norway and Sweden, the years of life lost (YLL) attributed to COVID-19 in Sweden, and the significance of mortality displacement. We computed the expected mortality by taking into account the declining trend and the seasonality in mortality in the two countries over the past 20 years. From the excess mortality in Sweden in 2019/20, we estimated the YLL attributed to COVID-19 using the life expectancy in different age groups. We adjusted this estimate for possible displacement using an auto-regressive model for the year-to-year variations in excess mortality. We found that excess all-cause mortality over the epidemic year, July 2019 to July 2020, was 517 (95%CI = (12, 1074)) in Norway and 4329 [3331, 5325] in Sweden. There were 255 COVID-19 related deaths reported in Norway, and 5741 in Sweden, that year. During the epidemic period of 11 March-11 November, there were 6247 reported COVID-19 deaths and 5517 (4701, 6330) excess deaths in Sweden. We estimated that the number of YLL attributed to COVID-19 in Sweden was 45,850 [13,915, 80,276] without adjusting for mortality displacement and 43,073 (12,160, 85,451) after adjusting for the displacement accounted for by the auto-regressive model. In conclusion, we find good agreement between officially recorded COVID-19 related deaths and all-cause excess deaths in both countries during the first epidemic wave and no significant mortality displacement that can explain those deaths.

Reservoir Computing Approaches for Representation and Classification of Multivariate Time Series.

Classification of multivariate time series (MTS) has been tackled with a large variety of methodologies and applied to a wide range of scenarios. Reservoir computing (RC) provides efficient tools to generate a vectorial, fixed-size representation of the MTS that can be further processed by standard classifiers. Despite their unrivaled training speed, MTS classifiers based on a standard RC architecture fail to achieve the same accuracy of fully trainable neural networks. In this article, we introduce the reservoir model space, an unsupervised approach based on RC to learn vectorial representations of MTS. Each MTS is encoded within the parameters of a linear model trained to predict a low-dimensional embedding of the reservoir dynamics. Compared with other RC methods, our model space yields better representations and attains comparable computational performance due to an intermediate dimensionality reduction procedure. As a second contribution, we propose a modular RC framework for MTS classification, with an associated open-source Python library. The framework provides different modules to seamlessly implement advanced RC architectures. The architectures are compared with other MTS classifiers, including deep learning models and time series kernels. Results obtained on the benchmark and real-world MTS data sets show that RC classifiers are dramatically faster and, when implemented using our proposed representation, also achieve superior classification accuracy.

Intervention Fatigue is the Primary Cause of Strong Secondary Waves in the COVID-19 Pandemic.

As of November 2020, the number of COVID-19 cases was increasing rapidly in many countries. In Europe, the virus spread slowed considerably in the late spring due to strict lockdown, but a second wave of the pandemic grew throughout the fall. In this study, we first reconstruct the time evolution of the effective reproduction numbers R(t) for each country by integrating the equations of the classic Susceptible-Infectious-Recovered (SIR) model. We cluster countries based on the estimated R(t) through a suitable time series dissimilarity. The clustering result suggests that simple dynamical mechanisms determine how countries respond to changes in COVID-19 case counts. Inspired by these results, we extend the simple SIR model for disease spread to include a social response to explain the number X(t) of new confirmed daily cases. In particular, we characterize the social response with a first-order model that depends on three parameters ν1,ν2,ν3. The parameter ν1 describes the effect of relaxed intervention when the incidence rate is low; ν2 models the impact of interventions when incidence rate is high; ν3 represents the fatigue, i.e., the weakening of interventions as time passes. The proposed model reproduces typical evolving patterns of COVID-19 epidemic waves observed in many countries. Estimating the parameters ν1,ν2,ν3 and initial conditions, such as R0, for different countries helps to identify important dynamics in their social responses. One conclusion is that the leading cause of the strong second wave in Europe in the fall of 2020 was not the relaxation of interventions during the summer, but rather the failure to enforce interventions in the fall.

Determination of the Edge of Criticality in Echo State Networks Through Fisher Information Maximization.

It is a widely accepted fact that the computational capability of recurrent neural networks (RNNs) is maximized on the so-called "edge of criticality." Once the network operates in this configuration, it performs efficiently on a specific application both in terms of: 1) low prediction error and 2) high short-term memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in RNNs. It is proved that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and requires the analytic form of the probability density function ruling the system behavior. This paper takes advantage of a recently developed nonparametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks (ESNs), a particular class of recurrent networks. The considered control parameters, which indirectly affect the ESN performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.

Investigating Echo-State Networks Dynamics by Means of Recurrence Analysis.

In this paper, we elaborate over the well-known interpretability issue in echo-state networks (ESNs). The idea is to investigate the dynamics of reservoir neurons with time-series analysis techniques developed in complex systems research. Notably, we analyze time series of neuron activations with recurrence plots (RPs) and recurrence quantification analysis (RQA), which permit to visualize and characterize high-dimensional dynamical systems. We show that this approach is useful in a number of ways. First, the 2-D representation offered by RPs provides a visualization of the high-dimensional reservoir dynamics. Our results suggest that, if the network is stable, reservoir and input generate similar line patterns in the respective RPs. Conversely, as the ESN becomes unstable, the patterns in the RP of the reservoir change. As a second result, we show that an RQA measure, called , is highly correlated with the well-established maximal local Lyapunov exponent. This suggests that complexity measures based on RP diagonal lines distribution can quantify network stability. Finally, our analysis shows that all RQA measures fluctuate on the proximity of the so-called edge of stability, where an ESN typically achieves maximum computational capability. We leverage on this property to determine the edge of stability and show that our criterion is more accurate than two well-known counterparts, both based on the Jacobian matrix of the reservoir. Therefore, we claim that RPs and RQA-based analyses are valuable tools to design an ESN, given a specific problem.

Multiplex visibility graphs to investigate recurrent neural network dynamics.

A recurrent neural network (RNN) is a universal approximator of dynamical systems, whose performance often depends on sensitive hyperparameters. Tuning them properly may be difficult and, typically, based on a trial-and-error approach. In this work, we adopt a graph-based framework to interpret and characterize internal dynamics of a class of RNNs called echo state networks (ESNs). We design principled unsupervised methods to derive hyperparameters configurations yielding maximal ESN performance, expressed in terms of prediction error and memory capacity. In particular, we propose to model time series generated by each neuron activations with a horizontal visibility graph, whose topological properties have been shown to be related to the underlying system dynamics. Successively, horizontal visibility graphs associated with all neurons become layers of a larger structure called a multiplex. We show that topological properties of such a multiplex reflect important features of ESN dynamics that can be used to guide the tuning of its hyperparamers. Results obtained on several benchmarks and a real-world dataset of telephone call data records show the effectiveness of the proposed methods.

Prediction of telephone calls load using Echo State Network with exogenous variables.

We approach the problem of forecasting the load of incoming calls in a cell of a mobile network using Echo State Networks. With respect to previous approaches to the problem, we consider the inclusion of additional telephone records regarding the activity registered in the cell as exogenous variables, by investigating their usefulness in the forecasting task. Additionally, we analyze different methodologies for training the readout of the network, including two novel variants, namely ν-SVR and an elastic net penalty. Finally, we employ a genetic algorithm for both the tasks of tuning the parameters of the system and for selecting the optimal subset of most informative additional time-series to be considered as external inputs in the forecasting problem. We compare the performances with standard prediction models and we evaluate the results according to the specific properties of the considered time-series.