Class-Incremental Learning: A Survey.

Deep models, e.g., CNNs and Vision Transformers, have achieved impressive achievements in many vision tasks in the closed world. However, novel classes emerge from time to time in our ever-changing world, requiring a learning system to acquire new knowledge continually. Class-Incremental Learning (CIL) enables the learner to incorporate the knowledge of new classes incrementally and build a universal classifier among all seen classes. Correspondingly, when directly training the model with new class instances, a fatal problem occurs - the model tends to catastrophically forget the characteristics of former ones, and its performance drastically degrades. There have been numerous efforts to tackle catastrophic forgetting in the machine learning community. In this paper, we survey comprehensively recent advances in class-incremental learning and summarize these methods from several aspects. We also provide a rigorous and unified evaluation of 17 methods in benchmark image classification tasks to find out the characteristics of different algorithms empirically. Furthermore, we notice that the current comparison protocol ignores the influence of memory budget in model storage, which may result in unfair comparison and biased results. Hence, we advocate fair comparison by aligning the memory budget in evaluation, as well as several memory-agnostic performance measures. The source code is available at https://github.com/zhoudw-zdw/CIL_Survey/.

Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks.

New classes arise frequently in our ever-changing world, e.g., emerging topics in social media and new types of products in e-commerce. A model should recognize new classes and meanwhile maintain discriminability over old classes. Under severe circumstances, only limited novel instances are available to incrementally update the model. The task of recognizing few-shot new classes without forgetting old classes is called few-shot class-incremental learning (FSCIL). In this work, we propose a new paradigm for FSCIL based on meta-learning by LearnIng Multi-phase Incremental Tasks (Limit), which synthesizes fake FSCIL tasks from the base dataset. The data format of fake tasks is consistent with the 'real' incremental tasks, and we can build a generalizable feature space for the unseen tasks through meta-learning. Besides, Limit also constructs a calibration module based on transformer, which calibrates the old class classifiers and new class prototypes into the same scale and fills in the semantic gap. The calibration module also adaptively contextualizes the instance-specific embedding with a set-to-set function. Limit efficiently adapts to new classes and meanwhile resists forgetting over old classes. Experiments on three benchmark datasets (CIFAR100, miniImageNet, and CUB200) and large-scale dataset, i.e., ImageNet ILSVRC2012 validate that Limit achieves state-of-the-art performance.

Generalized Knowledge Distillation via Relationship Matching.

The knowledge of a well-trained deep neural network (a.k.a. the "teacher") is valuable for learning similar tasks. Knowledge distillation extracts knowledge from the teacher and integrates it with the target model (a.k.a. the "student"), which expands the student's knowledge and improves its learning efficacy. Instead of enforcing the teacher to work on the same task as the student, we borrow the knowledge from a teacher trained from a general label space - in this "Generalized Knowledge Distillation (GKD)," the classes of the teacher and the student may be the same, completely different, or partially overlapped. We claim that the comparison ability between instances acts as an essential factor threading knowledge across tasks, and propose the RElationship FacIlitated Local cLassifiEr Distillation (stance-label confidence between models, ReFilled requires the teacher to reweight the hard tuples pushed forward by the student and then matches the similarity comparison levels between instances. An embedding-induced classifier based on the teacher model supervises the student's classification confidence and adaptively emphasizes the most related supervision from the teacher. ReFilled demonstrates strong discriminative ability when the classes of the teacher vary from the same to a fully non-overlapped set w.r.t. the student. It also achieves state-of-the-art performance on standard knowledge distillation, one-step incremental learning, and few-shot learning tasks.

Revisiting Unsupervised Meta-Learning via the Characteristics of Few-Shot Tasks.

Meta-learning has become a practical approach towards few-shot image classification, where "a strategy to learn a classifier" is meta-learned on labeled base classes and can be applied to tasks with novel classes. We remove the requirement of base class labels and learn generalizable embeddings via Unsupervised Meta-Learning (UML). Specifically, episodes of tasks are constructed with data augmentations from unlabeled base classes during meta-training, and we apply embedding-based classifiers to novel tasks with labeled few-shot examples during meta-test. We observe two elements play important roles in UML, i.e., the way to sample tasks and measure similarities between instances. Thus we obtain a strong baseline with two simple modifications - a sufficient sampling strategy constructing multiple tasks per episode efficiently together with a semi-normalized similarity. We then take advantage of the characteristics of tasks from two directions to get further improvements. First, synthesized confusing instances are incorporated to help extract more discriminative embeddings. Second, we utilize an additional task-specific embedding transformation as an auxiliary component during meta-training to promote the generalization ability of the pre-adapted embeddings. Experiments on few-shot learning benchmarks verify that our approaches outperform previous UML methods and achieve comparable or even better performance than its supervised variants.

Few-Shot Learning with a Strong Teacher.

Few-shot learning (FSL) aims to generate a classifier using limited labeled examples. Many existing works take the meta-learning approach, constructing a few-shot learner (a meta-model) that can learn from few-shot examples to generate a classifier. The performance is measured by how well the resulting classifiers classify the test (\ie, query) examples of those tasks. In this paper, we point out two potential weaknesses of this approach. First, the sampled query examples may not provide sufficient supervision for meta-training the few-shot learner. Second, the effectiveness of meta-learning diminishes sharply with the increasing number of shots. We propose a novel meta-training objective for the few-shot learner, which is to encourage the few-shot learner to generate classifiers that perform like strong classifiers. Concretely, we associate each sampled few-shot task with a strong classifier, which is trained with ample labeled examples. The strong classifiers can be seen as the target classifiers that we hope the few-shot learner to generate given few-shot examples, and we use the strong classifiers to supervise the few-shot learner. We validate our approach in combinations with many representative meta-learning methods. More importantly, with our approach, meta-learning based FSL methods can consistently outperform non-meta-learning based methods at different numbers of shots.

Heterogeneous Few-Shot Model Rectification With Semantic Mapping.

There still involve lots of challenges when applying machine learning algorithms in unknown environments, especially those with limited training data. To handle the data insufficiency and make a further step towards robust learning, we adopt the learnware notion Z.-H. Zhou, "Learnware: On the future of machine learning," Front. Comput. Sci., vol. 10, no. 4 pp. 589-590, 2016 which equips a model with an essential reusable property-the model learned in a related task could be easily adapted to the current data-scarce environment without data sharing. To this end, we propose the REctiFy via heterOgeneous pRedictor Mapping (ReForm) framework enabling the current model to take advantage of a related model from two kinds of heterogeneous environment, i.e., either with different sets of features or labels. By Encoding Meta InformaTion (Emit) of features and labels as the model specification, we utilize an optimal transported semantic mapping to characterize and bridge the environment changes. After fine-tuning over a few labeled examples through a biased regularization objective, the transformed heterogeneous model adapts to the current task efficiently. We apply ReForm over both synthetic and real-world tasks such as few-shot image classification with either learned or pre-defined specifications. Experimental results validate the effectiveness and practical utility of the proposed ReForm framework.

Learning Multiple Local Metrics: Global Consideration Helps.

Learning distance metric between objects provides a better measurement for their relative comparisons. Due to the complex properties inside or between heterogeneous objects, multiple local metrics become an essential representation tool to depict various local characteristics of examples. Different from existing methods building more than one local metric directly, however in this paper, we emphasize the effect of the global metric when generating those local ones. Since local metrics can be considered as types of amendments which describe the biases towards localities based on some commonly shared characteristic, it is expected that the performance of every single local metric for a specified locality can be "lifted" when learning with the global jointly. Following this consideration, we propose the Local metrIcs Facilitated Transformation (Lift) framework, where an adaptive number of local transformations are constructed with the help of their global counterpart. Generalization analyses not only reveal the relationship between the global and local metrics but also indicate when and why the framework works theoretically. In the implementation of Lift, locality anchored centers assist the decomposition of multiple local views, and a diversity regularizer is proposed to reduce the redundancy among biases. Empirical classification comparisons reveal the superiority of the Lift idea. Numerical and visualization investigations on different domains validate its adaptability and comprehensibility as well.

What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

Linkages are essentially determined by similarity measures that may be derived from multiple perspectives. For example, spatial linkages are usually generated based on localities of heterogeneous data. Semantic linkages, however, can come from even more properties, such as different physical meanings behind social relations. Many existing metric learning models focus on spatial linkages but leave the rich semantic factors unconsidered. We propose a Unified Multi-Metric Learning framework to exploit multiple types of metrics with respect to overdetermined similarities between linkages. In , a type of combination operator is introduced for distance characterization from multiple perspectives, and thus can introduce flexibilities for representing and utilizing both spatial and semantic linkages. Besides, we propose a uniform solver for , and the theoretical analysis reflects the generalization ability of as well. Extensive experiments on diverse applications exhibit the superior classification performance and comprehensibility of . Visualization results also validate its ability on physical meanings discovery.