Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects.

Self-supervised learning (SSL) has recently achieved impressive performance on various time series tasks. The most prominent advantage of SSL is that it reduces the dependence on labeled data. Based on the pre-training and fine-tuning strategy, even a small amount of labeled data can achieve high performance. Compared with many published self-supervised surveys on computer vision and natural language processing, a comprehensive survey for time series SSL is still missing. To fill this gap, we review current state-of-the-art SSL methods for time series data in this article. To this end, we first comprehensively review existing surveys related to SSL and time series, and then provide a new taxonomy of existing time series SSL methods by summarizing them from three perspectives: generative-based, contrastive-based, and adversarial-based. These methods are further divided into ten subcategories with detailed reviews and discussions about their key intuitions, main frameworks, advantages and disadvantages. To facilitate the experiments and validation of time series SSL methods, we also summarize datasets commonly used in time series forecasting, classification, anomaly detection, and clustering tasks. Finally, we present the future directions of SSL for time series analysis.

Investigation on the Effectiveness of Augmented Reality Memory Training Game for Chinese Older Adults: A Randomized Controlled Trial.

Objective: To evaluate the effectiveness of augmented reality (AR) game based on n-back training paradigm as a training tool for working memory (WM) of Chinese healthy older adults. Materials and Methods: One hundred eighteen older adults self-assessed as healthy were included in this study. Individuals were randomly divided into an intervention group (n = 57) and a control group (n = 61). Interventions, consisting of a 30-minute AR game-based training and a 30-minute health science program, were administered three times per week for 4 weeks, whereas the control group was required to view a 60-minute health science program three times per week for 4 weeks. Tests, Digit Span, Corsi Block-Tapping Task (CBT), and Stroop Color and Word Test (SCWT), were conducted for all participants before and after the experiment, and the game accuracy rate of the intervention group before and after intervention was recorded. Results: There was a statistically significant difference in terms of both CBT indicators, CBT forward span (z = -2.835, P = 0.005) and CBT backward span (z = 3.285, P = 0.001), and the SCWT indicator of Stroop Words Test (SW) (z = -1.894, P = 0.048) in the two groups. The intervention group showed significant improvements in the game accuracy of both medium level (z = -3.535, P < 0.05) and of high level (z = -3.953, P < 0.05). In addition, differences were observed in subgroup analysis in the accuracy of medium level (H = 6.218, P < 0.05) and high level (H = 8.002, P < 0.05) among older people with different levels of education. Conclusion: AR game based on n-back training paradigm could improve WM of Chinese older adults, showing potential for wider promotion and adoption.

Ricci Curvature-Based Graph Sparsification for Continual Graph Representation Learning.

Memory replay, which stores a subset of historical data from previous tasks to replay while learning new tasks, exhibits state-of-the-art performance for various continual learning applications on the Euclidean data. While topological information plays a critical role in characterizing graph data, existing memory replay-based graph learning techniques only store individual nodes for replay and do not consider their associated edge information. To this end, based on the message-passing mechanism in graph neural networks (GNNs), we present the Ricci curvature-based graph sparsification technique to perform continual graph representation learning. Specifically, we first develop the subgraph episodic memory (SEM) to store the topological information in the form of computation subgraphs. Next, we sparsify the subgraphs such that they only contain the most informative structures (nodes and edges). The informativeness is evaluated with the Ricci curvature, a theoretically justified metric to estimate the contribution of neighbors to represent a target node. In this way, we can reduce the memory consumption of a computation subgraph from O(dL) to O(1) and enable GNNs to fully utilize the most informative topological information for memory replay. Besides, to ensure the applicability on large graphs, we also provide the theoretically justified surrogate for the Ricci curvature in the sparsification process, which can greatly facilitate the computation. Finally, our empirical studies show that SEM outperforms state-of-the-art approaches significantly on four different public datasets. Unlike existing methods, which mainly focus on task incremental learning (task-IL) setting, SEM also succeeds in the challenging class incremental learning (class-IL) setting in which the model is required to distinguish all learned classes without task indicators and even achieves comparable performance to joint training, which is the performance upper bound for continual learning.

Hierarchical Prototype Networks for Continual Graph Representation Learning.

Despite significant advances in graph representation learning, little attention has been paid to the more practical continual learning scenario in which new categories of nodes (e.g., new research areas in citation networks, or new types of products in co-purchasing networks) and their associated edges are continuously emerging, causing catastrophic forgetting on previous categories. Existing methods either ignore the rich topological information or sacrifice plasticity for stability. To this end, we present Hierarchical Prototype Networks (HPNs) which extract different levels of abstract knowledge in the form of prototypes to represent the continuously expanded graphs. Specifically, we first leverage a set of Atomic Feature Extractors (AFEs) to encode both the elemental attribute information and the topological structure of the target node. Next, we develop HPNs to adaptively select relevant AFEs and represent each node with three levels of prototypes. In this way, whenever a new category of nodes is given, only the relevant AFEs and prototypes at each level will be activated and refined, while others remain uninterrupted to maintain the performance over existing nodes. Theoretically, we first demonstrate that the memory consumption of HPNs is bounded regardless of how many tasks are encountered. Then, we prove that under mild constraints, learning new tasks will not alter the prototypes matched to previous data, thereby eliminating the forgetting problem. The theoretical results are supported by experiments on five datasets, showing that HPNs not only outperform state-of-the-art baseline techniques but also consume relatively less memory. Code and datasets are available at https://github.com/QueuQ/HPNs.

Improved inter-residue contact prediction via a hybrid generative model and dynamic loss function.

Protein contact maps represent spatial pairwise inter-residue interactions, providing a protein's translationally and rotationally invariant topological representation. Accurate contact map prediction has been a critical driving force for improving protein structure determination. Contact maps can also be used as a stand-alone tool for varied applications such as prediction of protein-protein interactions, structure-aware thermal stability or physicochemical properties. We develop a novel hybrid contact map prediction model, CGAN-Cmap, that uses a generative adversarial neural network embedded with a series of modified squeeze and excitation residual networks. To exploit features of different dimensions, we introduce two parallel modules. This architecture improves the prediction by increasing receptive fields, surpassing redundant features and encouraging more meaningful ones from 1D and 2D inputs. We also introduce a new custom dynamic binary cross-entropy loss function to address the input imbalance problem for highly sparse long-range contacts in proteins with insufficient homologs. We evaluate the model's performance on CASP 11, 12, 13, 14, and CAMEO test sets. CGAN-Cmap outperforms state-of-the-art models, improving precision of medium and long-range contacts by at least 3.5%. As a direct assessment between our model and AlphaFold2, the leading available protein structure prediction model, we compare extracted contact maps from AlphaFold2 and predicted contact maps from CGAN-Cmap. The results show that CGAN-Cmap has a mean precision higher by 1% compared to AlphaFold2 for most ranges of contacts. These results demonstrate an efficient approach for highly accurate contact map prediction toward accurate characterization of protein structure, properties and functions from sequence.

Efficient robust conditional random fields.

Conditional random fields (CRFs) are a flexible yet powerful probabilistic approach and have shown advantages for popular applications in various areas, including text analysis, bioinformatics, and computer vision. Traditional CRF models, however, are incapable of selecting relevant features as well as suppressing noise from noisy original features. Moreover, conventional optimization methods often converge slowly in solving the training procedure of CRFs, and will degrade significantly for tasks with a large number of samples and features. In this paper, we propose robust CRFs (RCRFs) to simultaneously select relevant features. An optimal gradient method (OGM) is further designed to train RCRFs efficiently. Specifically, the proposed RCRFs employ the l1 norm of the model parameters to regularize the objective used by traditional CRFs, therefore enabling discovery of the relevant unary features and pairwise features of CRFs. In each iteration of OGM, the gradient direction is determined jointly by the current gradient together with the historical gradients, and the Lipschitz constant is leveraged to specify the proper step size. We show that an OGM can tackle the RCRF model training very efficiently, achieving the optimal convergence rate [Formula: see text] (where k is the number of iterations). This convergence rate is theoretically superior to the convergence rate O(1/k) of previous first-order optimization methods. Extensive experiments performed on three practical image segmentation tasks demonstrate the efficacy of OGM in training our proposed RCRFs.

Biologically inspired feature manifold for scene classification.

Biologically inspired feature (BIF) and its variations have been demonstrated to be effective and efficient for scene classification. It is unreasonable to measure the dissimilarity between two BIFs based on their Euclidean distance. This is because BIFs are extrinsically very high dimensional and intrinsically low dimensional, i.e., BIFs are sampled from a low-dimensional manifold and embedded in a high-dimensional space. Therefore, it is essential to find the intrinsic structure of a set of BIFs, obtain a suitable mapping to implement the dimensionality reduction, and measure the dissimilarity between two BIFs in the low-dimensional space based on their Euclidean distance. In this paper, we study the manifold constructed by a set of BIFs utilized for scene classification, form a new dimensionality reduction algorithm by preserving both the geometry of intra BIFs and the discriminative information inter BIFs termed Discriminative and Geometry Preserving Projections (DGPP), and construct a new framework for scene classification. In this framework, we represent an image based on a new BIF, which combines the intensity channel, the color channel, and the C1 unit of a color image; then we project the high-dimensional BIF to a low-dimensional space based on DGPP; and, finally, we conduct the classification based on the multiclass support vector machine (SVM). Thorough empirical studies based on the USC scene dataset demonstrate that the proposed framework improves the classification rates around 100% relatively and the training speed 60 times for different sites in comparing with previous gist proposed by Siagian and Itti in 2007.