11institutetext: National Chiao Tung University, Taiwan
11email: {raygoah.cs07g<EMAIL_ADDRESS>11email: {wpeng,
<EMAIL_ADDRESS>
# Class-incremental Learning with
Rectified Feature-Graph Preservation

Cheng-Hsun Lei Both authors contributed equally to the paper Yi-Hsin Chen*
Wen-Hsiao Peng  Wei-Chen Chiu

###### Abstract

In this paper, we address the problem of distillation-based class-incremental
learning with a single head. A central theme of this task is to learn new
classes that arrive in sequential phases over time while keeping the model’s
capability of recognizing seen classes with only limited memory for preserving
seen data samples. Many regularization strategies have been proposed to
mitigate the phenomenon of catastrophic forgetting. To understand better the
essence of these regularizations, we introduce a feature-graph preservation
perspective. Insights into their merits and faults motivate our weighted-
Euclidean regularization for old knowledge preservation. We further propose
rectified cosine normalization and show how it can work with binary cross-
entropy to increase class separation for effective learning of new classes.
Experimental results on both CIFAR-100 and ImageNet datasets demonstrate that
our method outperforms the state-of-the-art approaches in reducing
classification error, easing catastrophic forgetting, and encouraging evenly
balanced accuracy over different classes. Our project page is at :
https://github.com/yhchen12101/FGP-ICL.

##  1 Introduction

Class-incremental learning [1] is a difficult yet practical problem for visual
recognition models. The task requires the model with a single classification
head to learn new classes that arrive sequentially while preserving its
knowledge on the old ones. It is worth noting that class-incremental learning
is different from task-incremental learning [2, 3, 4, 5, 6], which learns to
handle multiple tasks (i.e. groups of classes, possibly from different
datasets) given sequentially. In particular, task-incremental learning has a
unique assumption that the task indices of classes are known during both
training and test time. Thus, the model typically aims to learn a shared
feature extractor across tasks under the multi-head setup (i.e. each task has
its own head). Even though there are some works [3, 7, 8] addressing class-
incremental learning from the perspective of task-incremental learning, the
differences stemmed from the aforementioned assumption clearly distinguish
these works from typical class-incremental learning. This paper focuses on the
more difficult yet practical setup to learn a single-head classifier without
task identification.

Generally, class-incremental learning is faced with two major challenges: (1)
the model needs to be updated on-the-fly upon the arrival of training data of
new classes, and (2) there is only limited memory space (also known as
experience reply buffer) for retaining partially the training data of
previously learned classes. It is shown in [1] that naively fine-tuning the
model on newly received training data of incoming classes without considering
the previously learned classes often suffers from _catastrophic forgetting_
[9]; that is, the classification accuracy on old classes may decline
dramatically when the model attempts to learn new classes.

Several prior works based on deep learning models have been proposed to deal
with catastrophic forgetting. They can roughly be categorized into three
types: the parameter-based, distillation-based, and generative-model-based
approaches. The parameter-based methods aim to restrict the update on network
parameters that are highly sensitive to the recognition accuracy of old
classes [2, 3, 10, 11], and to conserve the usage of network parameters in
learning new classes [12, 13, 14]. Identifying and regularizing those
important network parameters can be prohibitively expensive, especially with
large learning models. By contrast, the distillation-based methods leverage
the idea of knowledge distillation [15], requiring that the model updated with
the training data of new classes should produce similar predictions to the old
model over the previously learned classes [1, 16, 17, 18, 19, 4, 20, 21]. They
typically need a divergence measure to be defined that measures the similarity
between the old and new predictions. In particular, this class of methods is
often influenced by the amount of data available in the experience reply
buffer on which the similarity measurement is conducted. The generative-model-
based methods replace the experience replay buffer with deep generative models
[22], so that these generative models can later be utilized to generate
synthetic training data for old classes while updating the model to learn new
classes [23, 24, 25]. Obviously, their capacity for learning the distribution
of old data crucially affects the quality of generated training data.

This work presents a new geometry-informed attempt at knowledge distillation
for class-incremental learning. Instead of devising new divergences for
similarity measurement, our work stresses preserving the geometric structure
of the feature space. Specifically, we first (1) analyze the objectives used
in the existing distillation-based algorithms from a feature-graph
preservation perspective. By inspection of their merits and faults, we then
(2) propose weighted-Euclidean regularization to preserve the knowledge of
previously learned classes while providing enough flexibility for the model to
learn new classes. Lastly, we (3) introduce rectified cosine normalization to
learn discriminative feature representations, together with the objective of
binary cross-entropy.

##  2 Related Work

#### Parameter-based.

Parameter-based methods attempt to identify the network parameters that have a
profound impact on the classification accuracy of previously learned classes
once altered. These important parameters are encouraged to be fixed during the
incremental learning in order to resolve the issue of catastrophic forgetting
(e.g. MAS [2], EWC [3], SI [10], and RWalk [11]). Other works such as Rahaf et
al. [12], Piggy-back [13], and Packnet [14] further advance freeing up the
remaining parameters selectively in learning new classes. This is achieved by
learning a weighting mask, pruning the network, or adding specifically-
designed regularization. Because separate parts of the network can be
activated for different tasks, parameter-based methods often find application
in task-incremental learning. They however incur heavy computation in
identifying and regularizing important network parameters.

#### Generative-model-based.

Generative-model-based methods (e.g. [23, 24, 25]) turn to generative models
as a means of keeping the training data of old classes. Since the experience
replay buffer has a limited size, the training data of previously learned
classes can only be retrained partially. This leads to an imbalanced
distribution of training data between the new and old classes. To overcome
this issue, generative-model-based methods employ generative models to
generate synthetic training data for old classes in the course of incremental
learning. It is worth noting that the use, storage, and training of these
generative models can cause a considerable complexity increase.

#### Distillation-based.

Knowledge distillation [15] is originally proposed to distill the knowledge
from a teacher network (which is typically a network with larger capacity) and
transfer it to a student network (which is a smaller network than the
teacher), by encouraging the student to have a similar posterior distribution
to the teacher’s. This concept is adapted to preserve the model’s knowledge of
old classes for class-incremental learning. LwF [4] is the first work to apply
knowledge distillation to incremental learning, introducing a modified cross-
entropy loss function. iCaRL [1] uses binary cross-entropy as a new
distillation loss, provides a mechanism for selecting old-class exemplars, and
classifies images with the _nearest-mean-of-exemplars_ criterion. Castro et
al. [17] proposes a framework, termed End-to-End in this paper, that adopts
the ordinary cross-entropy as the distillation loss in learning end-to-end the
feature representation and the classifier. In particular, it introduces
balanced fine-tuning as an extra training step to deal with the data imbalance
between the old and new classes. BIC [20] proposes a bias correction method to
address the data imbalance, achieving excellent accuracy on large datasets.
Hou19 [18] proposes several designs, including the cosine normalization, the
less-forget constraint, and the inter-class separation, to encourage the
classifier to treat old and new classes more uniformly. Based on Hou19’s
training strategies, Liu et al. [21], termed Mnemonics, further optimize the
parameterized exemplars by a bi-level optimization program. Both Hou19 and
Mnemonics rely on a well-pretrained feature extractor to obtain superior
performance on incremental learning.

##  3 Proposed Method

Figure 1: Illustration of the class-incremental learning process: (1)
selecting exemplars, (2) adding the data samples of new classes, and (3)
updating the model. Figure 2: Knowledge distillation from a bipartite feature-
graph preservation perspective, with the bottom row and top row denoting the
feature vectors of data samples and the class embeddings respectively.
Minimizing the distillation loss is viewed as the structure (i.e. edge
strength) preservation of such a bipartite graph during the model update.

This section presents our strategies – namely, weighted-Euclidean
regularization and rectified cosine normalization – for dealing with
catastrophic forgetting and effective new class learning in the scenario of
class-incremental learning.

Fig. 1 illustrates the learning of data features and class embeddings (i.e.
the classifier’s weight vectors) from one incremental learning phase $n$ to
the next $n+1$. As shown, the data of different classes arrive sequentially in
different phases. We denote the old data that have arrived before phase $n+1$
as $X_{old}$ and the new data arriving in phase $n+1$ as $X_{new}$. Assuming
we already have a model capable of classifying the classes $C_{old}$ in the
$X_{old}$, we now aim to train the model to additionally learn the new classes
$C_{new}$ in $X_{new}$. However, there is limited memory for storing data
samples; we can only preserve some exemplars chosen from the seen samples
before phase $n+1$. Thus, the training is based on the dataset
$X_{all}=X_{new}\cup X_{exem}$, where $X_{exem}$ is composed of the exemplars
of $C_{old}$ (i.e $X_{exem}$ is a tiny subset of $X_{old}$) and the number of
exemplars that can be kept is constrained by the memory size.

###  3.1 Mitigating Catastrophic Forgetting

To motivate the need for our weighted-Euclidean regularization, we introduce a
feature-graph preservation perspective on knowledge distillation. This new
perspective reveals the essence of existing distillation-based regularization
schemes in how they tackle catastrophic forgetting and highlights the striking
feature of our method.

#### Knowledge Distillation in Class-Incremental Learning.

We begin by revisiting the prior works from iCaRL [1] and End-to-End [17]. To
prevent class-incremental learning from catastrophic forgetting, both models
introduce a distillation loss that requires the model after the gradient
update (termed the new model) to produce a similar prediction on $C_{old}$ to
that made by the model before the update (termed the old model), for any data
in $X_{all}$. In symbols, for a data sample $x_{k}\in X_{all}$, iCaRL [1]
computes the distillation loss $\mathcal{L}_{dist\\_bce}(x_{k})$ to be the sum
of the binary cross-entropy between the old prediction $p^{*}_{i|k}$ and the
new prediction $p_{i|k}$ over different classes $i\in C_{old}$:

$\displaystyle\mathcal{L}_{dist\\_bce}(x_{k})=\sum_{i\in
C_{old}}\left\\{1\times\left[-p^{*}_{i|k}\log
p_{i|k}-(1-p^{*}_{i|k})\log(1-p_{i|k})\right]\right\\},$ (1)

where $p_{i|k}=\sigma(a_{i|k})$ is the sigmoid output of the activation
$a_{i|k}=w^{T}_{i}f_{k}+b_{i}$ evaluated based on the new model, with
$w_{i},b_{i}$ representing the weight vector and bias of the $i$-th classifier
respectively, and $f_{k}$ denoting the feature vector of $x_{k}$; and
$p^{*}_{i|k}=\sigma(a^{*}_{i|k})$ is computed similarly yet with the
activation $a^{*}_{i|k}=w^{*T}_{i}f^{*}_{k}+b^{*}_{i}$ evaluated with the old
model. In contrast, the Eq. (3) in the End-to-End [17] paper minimizes the KL-
divergence111This amounts to minimizing the cross-entropy
$\sum_{i}\\{-p^{*}_{i|k}\times\log p_{i|k}\\}$ between $p^{*}_{i|k}$ and
$p_{i|k}$ over $i\in C_{old}$ since $p^{*}_{i|k}$ is not affected by the model
update. between $p^{*}_{i|k}$ and $p_{i|k}$ over $i\in C_{old}$ for preserving
knowledge learned in previous phases:

$\displaystyle\mathcal{L}_{dist\\_KL}(x_{k})=\sum_{i\in
C_{old}}\left\\{p^{*}_{i|k}\times\left[\log p^{*}_{i|k}-\log
p_{i|k}\right]\right\\},$ (2)

where $p_{i|k}=softmax(a_{i|k})$ (respectively
$p^{*}_{i|k}=softmax(a^{*}_{i|k})$) is the softmax output of the activation
$a_{i|k}/T$ (respectively, $a^{*}_{i|k}/T$) attenuated by a temperature
parameter $T$.

Close examination of Eqs. (1) and (2) above suggests a more general
distillation loss of the form

$\displaystyle\mathcal{L}_{dist}(x_{k})=\sum_{i\in
C_{old}}\left\\{\gamma_{i|k}\times\mathcal{D}(p^{*}_{i|k},p_{i|k})\right\\},$
(3)

with $\mathcal{D}(p^{*}_{i|k},p_{i|k})$ measuring the discrepancy between the
old and new predictions (i.e. $p^{*}_{i|k}$ and $p_{i|k}$) for class $i$, and
$\gamma_{i|k}$ weighting the contribution of
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$ to the resulting loss
$\mathcal{L}_{dist}(x_{k})$. Note that a non-zero discrepancy value
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$, be it positive or negative, signals a
change in the model’s prediction. By Eq. (3), iCaRL [1] is seen to have
$\gamma_{i|k}=1$ and $\mathcal{D}(p^{*}_{i|k},p_{i|k})$ the same as the binary
cross-entropy between $p^{*}_{i|k}$ and $p_{i|k}$, while End-to-End [17] sets
$\gamma_{i|k}$ and $\mathcal{D}(p^{*}_{i|k},p_{i|k})$ to $p^{*}_{i|k}$ and
$(\log p^{*}_{i|k}-\log p_{i|k})$, respectively.

#### Knowledge Distillation as Feature-Graph Preservation.

A graph interpretation can help us understand better what we are observing
from Eq. (3). We first note that Eq. (3) is to be optimized towards zero for
all $x_{k}$ in $X_{all}$ and is concerned with the change in relations between
the feature vector $f_{k}$ of certain $x_{k}\in X_{all}$ and the class
embeddings $w_{i},\forall i\in C_{old}$. Their relations are seen to be
quantified by $p_{i|k}$; specifically, the higher the value of $p_{i|k}$, the
more strongly related the $f_{k}$ and $w_{i}$ is. This relationship, as
depicted in Fig. 2, can be visualized as a bipartite graph, where the feature
vectors $f_{k}$ of all $x_{k}\in X_{all}$ form a set of vertices that is
disjoint from another set of vertices comprising $w_{i},i\in C_{old}$. The
graph is bipartite because Eq. (3) does not concern the relations among
vertices in each of these sets, and each $f_{k}$ is connected to all the
$w_{i}$ with edge weights specified by $p_{i|k}$. We then view the
optimization of the distillation loss over $x_{k}\in X_{all}$ towards zero as
the structure preservation of such a bipartite graph when the learning
progresses from one phase to another. This perspective is motivated by the
fact that $\mathcal{D}(p^{*}_{i|k},p_{i|k})$ in Eq. (3) reflects how the
strength of an edge changes as a result of the model update in a learning
phase. In particular, edge-adaptive preservation is achieved when
$\gamma_{i|k}$ is assigned non-uniformly.

To shed light on the rationale of our design, we now delve deeper into iCaRL
[1] and End-to-End [17] in terms of their edge weighting scheme
$\gamma_{i|k}$, edge strength specification $p_{i|k}$, and edge discrepancy
measure $\mathcal{D}(p^{*}_{i|k},p_{i|k})$. First, iCaRL [1] applies equal
weighting to all edges on the bipartite graph by having
$\gamma_{i|k}=1,\forall i,k$, suggesting that the strength of every edge
should be preserved equally after the model update. In contrast, End-to-End
[17] attaches more importance to stronger edges from the viewpoint of the old
model by having $\gamma_{i|k}=p^{*}_{i|k}$. Second, using sigmoid activation
in Eq. (1), iCaRL [1] evaluates the edge strength $p_{i|k}$ as a function of
the similarity between $f_{k}$ and $w_{i}$ alone, whereas the softmax
activation (cp. Eq. (2)) in End-to-End [17] renders $p_{i|k}$ dependent on the
relative similarities between $f_{k}$ and all the $w_{i},i\in C_{old}$. As
such, End-to-End [17] imposes the additional constraint
$\sum_{i}\gamma_{i|k}=\sum_{i}p^{*}_{i|k}=1$ on edge weighting, implying a
dependent weighting scheme across edges connecting $f_{k}$ and $w_{i},i\in
C_{old}$. Third, the discrepancy measure $\mathcal{D}(p^{*}_{i|k},p_{i|k})$
involves $p^{*}_{i|k}$ and $p_{i|k}$, which in turn depends on the activations
$a^{*}_{i|k}$ and $a_{i|k}$, respectively. These activations reflect
explicitly the similarity, or the geometrical relation, between $f_{k}$ and
$w_{i}$ in the feature space. When viewed as a function of $a_{i|k}$, the
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$ in both iCaRL [1] and End-to-End [17] is
non-symmetrical with respect to $a^{*}_{i|k}$; that is, the same deviation of
$a_{i|k}$ from $a^{*}_{i|k}$ yet in different signs causes
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$ to vary differently. This implies that
their distillation losses may favor some non-symmetrical geometry variations
in the feature space.

#### Weighted-Euclidean Regularization.

In designing the distillation loss, we deviate from the divergence-based
criterion to adopt a feature-graph preservation approach. This is motivated by
our use of the nearest-mean-of-exemplars classifier, which stresses the
geometric structure of the feature space. From the above observations, we
conjecture that good regularization for knowledge distillation in the feature
space should possess three desirable properties, including (1) prioritized
edge preservation, (2) independent edge strength specification, and (3)
$a^{*}_{i|k}$-symmetric discrepancy measure. The first allows more flexibility
in adapting feature learning to new classes while preserving old knowledge, by
prioritizing stronger edges. The second dispenses with the constraint that
$\sum_{i}\gamma_{i|k}=\sum_{i}p^{*}_{i|k}=1$, since the sum-to-one constraint
is essential to a cross-entropy interpretation in End-to-End [17], but is not
necessary when $p_{i|k}^{*}$ is regarded as an edge weighting $\gamma_{i|k}$
from our feature-graph viewpoint. The third is to penalize equally the
positive and negative deviation from $a^{*}_{i|k}$ for a geometrically-
symmetric preservation of the graph in the feature space. To this end, our
weighted-Euclidean regularization chooses $a_{i|k}$, $p_{i|k}$, $\gamma_{i|k}$
and $D(p^{*}_{i|k},p_{i|k})$ to be

$\displaystyle a_{i|k}=-\frac{\|\bar{w_{i}}-\bar{f_{k}}\|_{2}^{2}}{2}$
$\displaystyle,\qquad
p_{i|k}=\frac{1}{Z}\exp\\{a_{i|k}\\},\qquad\gamma_{i|k}=p^{*}_{i|k}$ (4)
$\displaystyle D(p^{*}_{i|k},p_{i|k})$ $\displaystyle=\left(\log
p^{*}_{i|k}-\log p_{i|k}\right)^{2}=\left(a^{*}_{i|k}-a_{i|k}\right)^{2}$

where $\bar{w_{i}},\bar{f_{k}}$ are the class embedding and feature vector
with rectified cosine normalization (in Sec. 3), $Z$ is a normalization
constant, and $a^{*}_{i|k},p^{*}_{i|k}$ are the activation and edge strength
evaluated with the old model. In particular, we choose the activation
$a_{i|k}=-\|\bar{w_{i}}-\bar{f_{k}}\|_{2}^{2}/2$ to match the criterion of the
nearest-mean-of-exemplars classification [1] at test time. It then follows
from Eq. (3) that our distillation loss has a form of

$\mathcal{L}_{dist\\_wE}(x_{k})=\sum_{i\in
C_{old}}\left\\{\exp(-\frac{\|\bar{w^{*}_{i}}-\bar{f^{*}_{k}}\|^{2}_{2}}{2})\times\left(\|\bar{w^{*}_{i}}-\bar{f^{*}_{k}}\|^{2}_{2}-\|\bar{w_{i}}-\bar{f_{k}}\|^{2}_{2}\right)^{2}\right\\},$
(5)

where constant terms have been omitted for brevity. Extensive experiments in
Sec. 4.3 validate the deceptively simple design of our distillation loss and
confirms its superior performance to the seemingly more principled divergence-
based methods (e.g. End-to-End [17]).

###  3.2 Learning New Classes

Learning new classes in the context of incremental learning is often faced
with data imbalance. Each new class usually has more training data than an old
class since only a limited amount of old data can be stored (as exemplars in
the memory). To address this issue, we adopt a two-pronged approach: (1)
minimizing binary cross-entropy as the classification loss and (2) applying
_rectified cosine normalization_ to increase class separation in the feature
space.

#### Classification Loss.

In the literature [17, 18, 1], there are two main choices of the
classification loss: cross-entropy and binary cross-entropy. The former is
mostly used for multi-class classification while the latter is popular for
multi-label classification. In the context of class-incremental learning (a
kind of multi-class classification), the study in [18] shows that the model
trained with cross-entropy tends to bias in favor of new classes by increasing
their class embeddings $w_{i}$ and bias terms $b_{i}$, to take advantage of
the imbalanced data distribution for lower classification errors (i.e. the
model would quickly adapt to new classes without being able to maintain well
the knowledge of old classes). This is because cross-entropy adopts a softmax
activation function, which determines the class prediction based on the
relative strength of the class activations, thereby creating the
aforementioned short cut. With the aim of achieving more evenly distributed
classification accuracy, we choose the stricter binary cross-entropy to
leverage its property that the activation of each class is weighted absolutely
rather than relatively in determining the class prediction. In symbols, it is
evaluated as

$\displaystyle\mathcal{L}_{bce}(x_{k})=-\sum_{i\in
C_{all}}\Big{[}\delta_{y_{i}=y_{k}}\log(\sigma(a_{i|k}))+\delta_{y_{i}\neq
y_{k}}\log(1-\sigma(a_{i|k}))\Big{]},$ (6)

where $y_{k}$ is the label of the data sample $x_{k}$, $C_{all}=C_{old}\cup
C_{new}$, and $\delta$ is an indicator function.

(a) $\text{rectified-}CN$

(b) $CN$

Figure 3:  A toy example based on MNIST for showcasing the benefit of our
rectified cosine normalization (denoted as $\text{rectified-}CN$) with respect
to the typical cosine normalization (denoted as $CN$). The colorized dots in
(a) (respectively (b)) are the t-SNE visualization of the feature
representations for images of 10 digit classes, which are obtained from the
classifier trained with adopting our $\text{rectified-}CN$ (respectively
$CN$); while the gray dots are the features from the classifier of using $CN$
(respectively $\text{rectified-}CN$).

#### Rectified Cosine Normalization.

In [18], they additionally adopt cosine normalization to address data
imbalance by normalizing the class embedding $w_{i}$ and the feature vector
$f_{k}$ in evaluating the activation ${a}_{i|k}=\bar{w}^{T}_{i}\bar{f_{k}}$,
where $\bar{w}_{i},\bar{f_{k}}$ are the normalized class embedding and feature
vector. Here we further develop a rectified cosine normalization technique,
which is empirically shown to encourage greater separation between
incrementally-learned classes in the feature space than the typical cosine
normalization. Our rectified cosine normalization is implemented by (1)
augmenting, during training with Eq. (6), every class embedding $w_{i}$ with a
learnable bias $b_{i}$ and every feature vector $f_{k}$ with a constant 1 for
separate normalization; and by (2) involving, at test time, only the feature
vector $f_{k}$ (without augmentation) in the nearest-mean-of-exemplars
classification [1]. Step 1 gives rise to an augmented feature space of
$W_{i}=(w_{i},b_{i})$ and $F_{k}=(f_{k},1)$, with the activation222This
activation is to be distinguished from the one
$a_{i|k}=-\|\bar{w_{i}}-\bar{f_{k}}\|_{2}^{2}/2$ for weighted-Euclidean
regularization. evaluated as $a_{i|k}=\bar{W}^{T}_{i}\bar{F_{k}}$ where the
bar indicates normalization.

Here we use MNIST dataset [26], which is composed of images of handwritten
digits, to showcase the benefit of our rectified cosine normalization. We
construct the classifier by a shallow network (including 2 convolutional
layers followed by 3 fully connected layers with an output size of 3, which
implies that the feature vector $f$ is three-dimensional), and train it with
whole 10 digit classes. Fig. 3 shows the t-SNE visualization of the feature
representations $f$ learned from both the model variants of using our
rectified cosine normalization and the typical cosine normalization. It is
clear to see that our proposed normalization scheme better encourages the
separation between classes. In Sec. 4.3, we present an ablation study to
demonstrate the effectiveness of our rectified cosine normalization in
improving incremental accuracy.

###  3.3 Objective Function

We combine the binary cross-entropy classification loss, rectified cosine
normalization, and the weighted-Euclidean regularization to build our overall
objective:

$\displaystyle\mathcal{L}(\mathbf{x})=\frac{1}{N}\sum^{N}_{k=1}\mathcal{(}{L}_{bce}(x_{k})+\lambda\mathcal{L}_{dist\\_wE}(x_{k})),$
(7)

where ${a}_{i|k}=\bar{W}^{T}_{i}\bar{F_{k}}$ in $\mathcal{L}_{bce}(x_{k})$,
$\\{x_{1},x_{2},\ldots,x_{N}\\}$ is a mini-batch drawn from $X_{all}$, and
$\lambda$ is the hyper-parameter used to balance $\mathcal{L}_{bce}$ against
$\mathcal{L}_{dist\\_wE}$. Inspired by [27], we choose $\lambda$ as follows in
recognizing that: in the early stages of incremental learning, when the
relative size of old classes to new classes is small, the weighted-Euclidean
regularization should be de-emphasized as the feature extractor may not be
well learned yet. Note that the ratio $|C_{old}|/|C_{all}|$ is updated
constantly along the incremental learning process.

$\displaystyle\lambda=\lambda_{base}\sqrt{\frac{\left|C_{old}\right|}{\left|C_{all}\right|}},$
(8)

where $\lambda_{base}=0.1$ is set empirically.

##  4 Experimental Results

#### Datasets and Baselines.

We follow the same experiment protocol as prior works. Two datasets, CIFAR-100
[28] and ImageNet [29] with 100 classes selected randomly, are used for
experiments. The data samples of these 100 classes comprise a data stream that
arrives in a class-incremental manner in 10 phases, each adding 10 new classes
to those already received in previous phases. Particularly, we adopt two
training scenarios. One is to train the model from scratch, i.e. the model
starts with random initialization and learns 100 classes incrementally. The
other follows [18] to train the model with the first 50 classes in order to
have a reasonably good feature extractor to begin with. Then the remaining 50
classes are split evenly and learned in 5 sequential phases. The baselines
include iCaRL [1], End-to-End [17], BIC [20], Hou19 [18], and Mnemonics [21].

#### Metrics.

The evaluation basically adopts the same common metrics used in prior works,
including incremental accuracy and average incremental accuracy [17, 18, 1].
Additionally, we introduce phase accuracy. Incremental accuracy [17], also
known simply as accuracy in [1], is the accuracy for classifying all the seen
classes at the end of each training phase. It is the most commonly used metric
but has the limitation of showing only one single accuracy value over the seen
classes as a whole without giving any detail of how the model performs on
separate groups of classes (learned incrementally in each phase). Average
incremental accuracy simply takes the average of the incremental accuracy
values obtained from the first training phase up to the current phase. The
results are given in parentheses for the last phase in Fig. 4. Phase accuracy
is evaluated at the end of the entire incremental training to present the
average classification accuracy on separate groups of classes, where each
group includes several classes that are added into the training at the same
phase. It provides a breakdown look at whether the model would favor some
groups of classes over the others as a consequence of catastrophic forgetting.

#### Implementation Details.

For a fair comparison, we follow the baselines to use a 32-layer
(respectively, 18-layer) ResNet [30] as the feature extractor for CIFAR-100
(respectively, ImageNet), and a fully connected layer as the linear
classifier. In particular, we remove the last ReLU layer for rectified cosine
normalization. The memory size is fixed at 1000 or 2000, and the exemplars of
old classes are chosen by the herd selection [31] proposed in iCaRL [1]. At
test time, we apply the same _nearest-mean-of-exemplars_ classification
strategy as iCaRL.

###  4.1 Incremental Accuracy Comparison

Fig. 4 presents incremental accuracy against the number of classes learned
over the course of incremental learning, while Table 1 summarizes the results
at the end of the entire training. Notice that we use five random orderings of
classes to build up class-incremental phases, where the metrics are evaluated
and averaged over these five orderings. Please also note that the figures
showing the experimental results are better viewed in color and zoomed in for
details.

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 4: Incremental accuracy on CIFAR-100 (top row) and ImageNet (bottom
row), for memory sizes $M=1K,2K$ and training scenarios “FS” and “from 50”.
Average incremental accuracy of each method is shown in parentheses.

We see that on CIFAR-100, when learning from scratch, our model outperforms
all the baselines in almost every training phase, showing a significant boost
in incremental accuracy especially with memory size 1000. In comparison, the
recent state-of-the-arts from Hou19 [18] and Mnemonics [21] perform
significantly worse as they rely heavily on a well-learned feature extractor
(which is hard to obtain while being trained from scratch). When starting from
50 classes, ours performs the best with memory size 1000, and comparably to
both Hou19 [18] and Mnemonics [21] with memory size 2000. Furthermore, our
model shows higher average incremental accuracy than the baselines in nearly
every setting, as shown in parentheses in Fig. 4. On ImageNet, our model
performs comparably to End-to-End [17] and BIC [20] when learning from scratch
with $M=2K$ and is superior to the competing methods by a large margin when
learning from 50 classes. To sum up, our method shows consistently higher
incremental accuracy than the other baselines on different datasets with
varied characteristics.

Table 1: Comparison of incremental accuracy at the end of training Training scenario | from scratch |  | from 50 classes
---|---|---|---
Dataset | CIFAR |  | ImageNet |  | CIFAR |  | ImageNet
Memory size | 1K | 2K |  | 1K | 2K |  | 1K | 2K |  | 1K | 2K
iCaRL [1] | 38.8 | 43.1 |  | 52.9 | 55.5 |  | 48.5 | 52.4 |  | 57.2 | 60.7
End-to-End [17] | 40.0 | 46.0 |  | 50.4 | 56.7 |  | 45.8 | 53.0 |  | 52.1 | 58.9
BIC [20] | 38.6 | 44.0 |  | 51.5 | 56.1 |  | 46.9 | 52.4 |  | 55.3 | 59.7
Hou19 [18] | 32.7 | 39.9 |  | 27.1 | 34.4 |  | 51.0 | 54.5 |  | 55.2 | 60.1
Mnemonics [21] | 35.9 | 40.9 |  | 34.6 | 39.4 |  | 52.0 | 54.2 |  | 59.8 | 61.0
Ours | 44.7 | 47.2 |  | 55.1 | 57.0 |  | 52.8 | 54.0 |  | 61.9 | 62.6

###  4.2 Phase Accuracy Comparison

Fig. 5 presents the phase accuracy for different methods to compare their
effectiveness in preserving knowledge of old classes. Generally, balanced
phase accuracy is desirable. It is important to point out that there is a
fundamental trade-off between incremental accuracy and phase accuracy. For a
fair comparison, we particularly choose the training scenario “from 50”, where
the baselines perform more closely to our method in terms of incremental
accuracy evaluated at the end of the entire training. Shown in parentheses is
the mean absolute deviation from the average of each method’s phase accuracy.
The smaller the deviation, the more balanced the classification accuracy is in
different training phases.  Our scheme is shown to achieve the minimum mean
absolute deviation in phase accuracy on CIFAR-100 and ImageNet. Remarkably,
among all the baselines, Hou19 [18] and Mnemonics [21] have the closest
incremental accuracy to ours, yet with considerable variations in phase
accuracy.

(a) $M$: 1K, from 50

(b) $M$: 2K, from 50

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

Figure 5: Phase accuracy comparison: (a)(b) are results on CIFAR-100 and
(c)(d) on ImageNet. $M$ is the memory size, and the model is pre-trained with
50 classes. The mean absolute deviation in phase accuracy is shown in
parentheses.

###  4.3 Ablation Studies

#### Binary Cross-entropy (BCE) versus Cross-entropy (CE) as Classification
Loss.

We observe the performance differences by replacing BCE in our method with CE
while keeping the other aspects (including rectified cosine normalization and
weighted-Euclidean regularization) untouched. Fig. 6 shows that BCE achieves
consistently higher incremental accuracy than CE under all the settings.
Moreover, BCE presents more evenly balanced phase accuracy whereas CE displays
a tendency of biasing towards those new classes in the last phase.

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 6: Comparison of binary cross-entropy and cross-entropy as the
classification loss on CIFAR-100 for memory sizes $M=1K,2K$ and training
scenarios “FS” and “from 50”. The results are evaluated in terms of
incremental accuracy. The bars are evaluated by five random orderings of
classes.

#### How effective is rectified cosine normalization in improving incremental
accuracy?

Fig. 7 presents results comparing rectified cosine normalization against
cosine normalization in terms of incremental accuracy. Recall that rectified
cosine normalization is proposed as a means to increase class separation and
thus classification accuracy. To see the performance differences, we train a
variant of our method by replacing solely rectified cosine normalization with
cosine normalization (while keeping the other aspects the same). From Fig. 7,
our rectified cosine normalization starts with similar incremental accuracy to
cosine normalization in early phases and gradually develops higher accuracy
along the incremental learning process, which validates our analysis in Sec.
3.2.

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

Figure 7: Comparison between rectified cosine normalization and cosine
normalization on CIFAR-100 in terms of incremental accuracy, for memory sizes
$M=1K,2K$ and training scenarios “FS” and “from 50”.

#### Efficacy of weighted-Euclidean regularization in balancing new class
learning and old knowledge preservation.

Fig. 8 compares our weighted-Euclidean regularization
$\mathcal{L}_{dist\\_wE}$ with various distillation losses (see Sec. 3.1 for
$\mathcal{L}_{dist\\_bce}$ and $\mathcal{L}_{dist\\_KL}$, and Hou19 [18] for
${L}^{G}_{dis}$) by examining their trade-offs between incremental and phase
accuracy. These variants are tested on the same base model of using BCE and
rectified cosine normalization. We see that weighted-Euclidean regularization
achieves higher incremental accuracy among the competing methods, and shows
lowest or the second-lowest mean absolute deviation (MAD) in phase accuracy in
most cases (cf. training all classes together in batch mode leads to $2.5$ MAD
in phase accuracy). These suggest that our weighted-Euclidean regularization
enables the model to strike a better balance between new class learning and
old knowledge preservation.

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 8: Comparison of various distillation losses on CIFAR-100, for memory
sizes $M=1K,2K$ and training scenarios “FS” and “from 50”, with the mean
absolute deviation in phase accuracy shown in parentheses.

#### Uniform vs. Non-uniform $\gamma_{i|k}$ Assignment.

This experiment investigates the sole benefits of prioritized edge
preservation (i.e. non-uniform assignment of $\gamma_{i|k}$) in our weighted-
Euclidean regularization. We construct a variant of our method by setting
$\gamma_{i|k}$ in Eq. (4) to $1$ and observe the performance differences. As
shown in Fig. 9, the non-uniform assignment of $\gamma_{i|k}$ in our scheme
leads to higher incremental accuracy and lower mean absolute deviations in
phase accuracy. In addition, the smaller variations in incremental accuracy
(cp. the bars in the top row of Fig. 9) suggest that our method with
prioritized edge preservation is less sensitive to the learning order of
object classes.

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 9: Comparison of uniform versus non-uniform $\gamma_{i|k}$ assignment
on CIFAR-100 for memory sizes $M=1K,2K$ and training scenario “FS” and “from
50,” with the mean absolute deviation in phase accuracy (the bottom row) shown
in parentheses. The bars in the top row are evaluated by five random orderings
of classes.

##  5 Conclusion

We introduce a feature-graph preservation perspective to study the essence of
various knowledge distillation techniques in how they preserve the structure
of the feature graph along the phases of class-incremental learning. Developed
based on three desirable properties, our weighted-Euclidean regularization is
shown to benefit the balance between new class learning and old knowledge
preservation. Furthermore, our rectified cosine normalization effectively
improves incremental accuracy due to greater class separation. With all the
techniques combined, our model consistently outperforms the state-of-the-art
baselines on CIFAR-100 and ImageNet datasets.

## References

  * [1] Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.:  icarl: Incremental classifier and representation learning.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2017)
  * [2] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.:  Memory aware synapses: Learning what (not) to forget.  In: European Conference on Computer Vision (ECCV). (2018)
  * [3] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.:  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences (PNAS) (2017)
  * [4] Li, Z., Hoiem, D.:  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2017)
  * [5] Hou, S., Pan, X., Change Loy, C., Wang, Z., Lin, D.:  Lifelong learning via progressive distillation and retrospection.  In: European Conference on Computer Vision (ECCV). (2018)
  * [6] Aljundi, R., Chakravarty, P., Tuytelaars, T.:  Expert gate: Lifelong learning with a network of experts.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2017)
  * [7] Lopez-Paz, D., Ranzato, M.:  Gradient episodic memory for continual learning.  In: Advances in Neural Information Processing Systems (NIPS). (2017)
  * [8] Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.:  Efficient lifelong learning with a-gem.  In: International Conference on Learning Representations (ICLR). (2019)
  * [9] McCloskey, M., Cohen, N.J.:  Catastrophic interference in connectionist networks: The sequential learning problem.  In: Psychology of Learning and Motivation.  (1989)
  * [10] Zenke, F., Poole, B., Ganguli, S.:  Continual learning through synaptic intelligence.  In: International Conference on Machine Learning (ICML). (2017)
  * [11] Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.:  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  In: European Conference on Computer Vision (ECCV). (2018)
  * [12] Aljundi, R., Rohrbach, M., Tuytelaars, T.:  Selfless sequential learning.  In: International Conference on Learning Representations (ICLR). (2019)
  * [13] Mallya, A., Davis, D., Lazebnik, S.:  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In: European Conference on Computer Vision (ECCV). (2018)
  * [14] Mallya, A., Lazebnik, S.:  Packnet: Adding multiple tasks to a single network by iterative pruning.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018)
  * [15] Hinton, G., Vinyals, O., Dean, J.:  Distilling the knowledge in a neural network.  In: NIPS Workshop on Deep Learning and Representation Learning. (2014)
  * [16] Belouadah, E., Popescu, A.:  Deesil: Deep-shallow incremental learning.  In: European Conference on Computer Vision (ECCV). (2018)
  * [17] Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.:  End-to-end incremental learning.  In: European Conference on Computer Vision (ECCV). (2018)
  * [18] Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.:  Learning a unified classifier incrementally via rebalancing.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2019)
  * [19] Jung, H., Ju, J., Jung, M., Kim, J.:  Less-forgetful learning for domain expansion in deep neural networks.  In: AAAI Conference on Artificial Intelligence (AAAI). (2018)
  * [20] Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.:  Large scale incremental learning.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2019)
  * [21] Liu, Y., Su, Y., Liu, A.A., Schiele, B., Sun, Q.:  Mnemonics training: Multi-class incremental learning without forgetting.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2020)
  * [22] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.:  Generative adversarial nets.  In: Advances in Neural Information Processing Systems (NIPS). (2014)
  * [23] Shin, H., Lee, J.K., Kim, J., Kim, J.:  Continual learning with deep generative replay.  In: Advances in Neural Information Processing Systems (NIPS). (2017)
  * [24] He, C., Wang, R., Shan, S., Chen, X.:  Exemplar-supported generative reproduction for class incremental learning.  In: British Machine Vision Conference (BMVC). (2018)
  * [25] Ostapenko, O., Puscas, M., Klein, T., Jahnichen, P., Nabi, M.:  Learning to remember: A synaptic plasticity driven framework for continual learning.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2019)
  * [26] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.:  Gradient-based learning applied to document recognition.  Proceedings of the IEEE (1998)
  * [27] Taitelbaum, H., Ben-Reuven, E., Goldberger, J.:  Adding new classes without access to the original training data with applications to language identification.  In: Annual Conference of the International Speech Communication Association (INTERSPEECH). (2018)
  * [28] Krizhevsky, A.:  Learning multiple layers of features from tiny images.  Technical report (2009)
  * [29] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.:  Imagenet: A large-scale hierarchical image database.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2009)
  * [30] He, K., Zhang, X., Ren, S., Sun, J.:  Deep residual learning for image recognition.  In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
  * [31] Welling, M.:  Herding dynamical weights to learn.  In: International Conference on Machine Learning (ICML). (2009)

Appendix : Class-incremental Learning
with Rectified Feature-Graph Preservation

Figure 10: Comparison of our rectified cosine normalization (denoted as
$\text{rectified-}CN$) and the typical cosine normalization (denoted as $CN$)
with different number of classes. Dots in various colors represent the learned
feature vector $f_{k}$ for class $k$, and the line segment of the same color
denotes the corresponding weight vector $w_{k}$.

##  Appendix 0.A Rectified Cosine Normalization

This section conducts a simulation to justify our rectified cosine
normalization. It is reported empirically in the main manuscript that using
rectified cosine normalization for training can better encourage the
separation between classes.

Specifically, in evaluating the binary cross-entropy loss
$\mathcal{L}_{bce}(x_{k})$, our rectified cosine normalization computes the
activation ${a}_{i|k}$ of an image $x_{k}$ for class $i$ to be
${a}_{i|k}=\bar{W}^{T}_{i}\bar{F_{k}}$, while the cosine normalization in [18]
uses ${a}_{i|k}=\bar{w}^{T}_{i}\bar{f_{k}}$. The bar indicates
$l_{2}$-normalization, $W_{i}=(w_{i},b_{i})$ is formed by the concatenation of
the classification weight vector $w_{i}$ and an additional learnable bias
$b_{i}$; likewise, $F_{k}=(f_{k},1)$ (referred to as the augmented feature
representation) concatenates the feature $f_{k}$ (referred to as the original
feature representation) of the image $x_{k}$ and a constant bias $1$. Due to
the $l_{2}$-normalization, the activation ${a}_{i|k}$ is in the range of
$[-1,1]$. We thus introduce a learnable $\eta$ to control the curvature of the
sigmoid function $\sigma(\cdot)$:

$\sigma(a_{i|k})=\frac{1}{1+exp(-\eta{a}_{i|k})}.$ (9)

Fig. 10 illustrates the benefits of our rectified cosine normalization, taking
the 2-dimensional cases as examples. In these examples, each class has only
one trainable data $f_{k}$ (or $F_{k}$) and the corresponding class embedding
$w_{k}$ (or $W_{k}$), where $f_{k},w_{k}\in R^{2}$ and $F_{k},W_{k}\in R^{3}$.
They are learned via the steepest descent algorithm. Notably, $f_{k}$ (or
$F_{k}$) is considered a parameter to be solved, with the aim of simulating
the ideal case where the feature extractor has enough capacity. The first row
presents the results of training with rectified cosine normalization in the
augmented feature space $F_{k}=(f_{k},1)$ with $f_{k}=(x,y)\in R^{2}$. Both
the learned features $F_{k}$ and the class embeddings $W_{i}=(w_{i},b_{i})$
are projected onto the $xy$-plane for visualization and comparison. The second
row corresponds to the results of using typical cosine normalization, for
which both $f_{k}$ and $w_{i}$ are learned directly in the original feature
space without augmentation. It is clear to see that in the cases with 3 and 4
classes, both rectified cosine normalization and typical cosine normalization
work well. The features $f_{k}$ point in the same direction as their class
embeddings $w_{k}$, as expected. We note that minimizing the
$\mathcal{L}_{bce}(x_{k})$ (equation (6) in the main manuscript) requires that
(1) $F_{k}$ should be separated from the class embeddings $W_{i}$ not of the
same class (i.e. $i\neq k$) by at least 90 degrees and that (2) the angle
between $F_{k}$ and $W_{k}$ of the same class should be smaller than 90
degrees and preferably be 0 degrees. These apply to $f_{k}$ and $w_{i}$
learned directly in the original feature space. When the number of classes is
small, these requirements are easy to fulfill, be it in the 2-dimensional
original feature space or in the 3-dimensional augmented feature space. They
however become difficult to hold simultaneously in the 2-dimensional feature
space when the number of classes increases beyond 4. Intriguingly, in those
cases, the feature vectors of different classes may collapse into few modes,
putting much emphasis on the first requirement due mainly to the excessive
amount of negative examples (i.e. for a given $f_{k}$, the number of class
embeddings $w_{i},i\neq k$ not of the same class is much larger than the case
$i=k$). In contrast, both requirements can still hold in the 3-dimensional
augmented feature space. We observe that this is achieved by setting all
$b_{i}$’s to negative values and having $F_{k}$ and $W_{k}$ (respectively,
$W_{i},i\neq k$) point in approximately similar directions (respectively, in
significantly different directions) when projected onto the xy-plane. The
larger degree of freedom in adapting the class embeddings $W_{i}$ to the two
requirements in the higher dimensional augmented feature space explains the
generally better separation between feature vectors $f_{k}$ with our rectified
cosine normalization in the lower dimensional original feature space.

##  Appendix 0.B Further Training Details

This section provides further training details. For a fair comparison, we
follow the settings in iCaRL [1]. On CIFAR-100, the learning rate starts from
2.0 and is divided by 5 after 49 and 63 epochs. Moreover, 70 epochs are run in
each incremental training phase. Likewise, on ImageNet the learning rate
starts from 2.0 and is divided by 5 after 20, 30, 40 and 50 epochs. Each
training phase has 60 epochs. On both datasets, the optimizer is SGD and the
batch size is 128.

##  Appendix 0.C Detail Formulation of Our Evaluation

We follow the common metrics in prior works [1, 17, 18] to evaluate the
performance of incremental learning; that is, incremental accuracy, average
incremental accuracy. Furthermore, we additionally propose a metric called
phase accuracy. The details of these metrics are described as follows.

#### Incremental Accuracy.

Let $A_{j,i}$ be the average accuracy of classes learned in phase $i$ after
training the network sequentially to phase $j$, where $i<j$. Incremental
accuracy is defined as $A_{j}=\sum_{i=1}^{j}A_{j,i}$, which is the accuracy
for classifying all the seen classes at the end of training phase $j$. It is
the most commonly-used metric, but has the limitation of showing only the
accuracy over the seen classes as a whole without giving any detail of how the
model performs on separate groups of classes (learned incrementally in each
phase).

#### Average Incremental Accuracy.

Average incremental accuracy accumulates the incremental accuracy, obtained
from each training phase, up to the current phase $j$ and then takes the
average, i.e. $\frac{1}{j}\sum_{i=1}^{j}A_{i}$.

#### Phase Accuracy.

A model that learns well incrementally should present a balanced distribution
of $A_{j,i},i\in\\{1,...,j\\}$. Thus, we propose phase accuracy to evaluate at
the end of the entire incremental training to present the classification
accuracy on separate groups of classes (a group represents a phase in this
thesis). It provides a breakdown look at whether the model would favor some
groups of classes over the others as a result of catastrophic forgetting.

##  Appendix 0.D More Ablation Experiments

###  0.D.1 Symmetric vs. Asymmetric Discrepancy Measure
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 11: Comparison of discrepancy measure
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$ on CIFAR-100 for memory sizes $M=1K,2K$ and
training scenario “FS” and “from 50,” with the mean absolute deviation in
phase accuracy shown in parentheses. The bars in the top row are evaluated by
five random orderings of classes.

We argue in Sec. 3.1 of the main manuscript that an $a^{*}_{i|k}$-symmetric
discrepancy measure is desirable. That is, $\mathcal{D}(p^{*}_{i|k},p_{i|k})$
is preferably symmetric with respect to $a^{*}_{i|k}$ when viewed as a
function of $a_{i|k}$. To single out its benefits, we change our distillation
loss $\mathcal{L}_{dist\\_wE}$ by replacing the
$\mathcal{D}(p^{*}_{i|k},p_{i|k})$ in Eq. (3) with the binary cross-entropy
(BCE) between $p^{*}_{i|k}$ and $p_{i|k}$, and setting
$\gamma_{i|k}=p^{*}_{i|k}$, while keeping the other design aspects (including
BCE as classification loss and rectified cosine normalization) untouched. Note
that these changes lead to a scheme (denoted collectively as
$\mathcal{L}_{dist\\_bce}$ with our $\gamma_{i|k}$) that differs from ours
only in adopting an asymmetric discrepancy measure with respect to
$a^{*}_{i|k}$. From the top row of Fig. 11, it is interesting to see that the
replacement of $\mathcal{D}(p^{*}_{i|k},p_{i|k})=(\log p^{*}_{i|k}-\log
p_{i|k})^{2}$ with BCE does not show much impact on incremental accuracy.
However, our $a^{*}_{i|k}$-symmetric discrepancy measure presents more
balanced phase accuracy, achieving smaller mean absolute deviations (cp. the
bottom row of Fig. 11). This highlights the fact that it allows our model to
strike a better balance between incremental accuracy and phase accuracy.

###  0.D.2 iCaRL with vs. without Rectified Cosine Normalization

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 12: Comparison of iCaRL with/without rectified cosine normalization and
our method on CIFAR-100 for memory sizes $M=1K,2K$ and training scenarios ”FS”
and ”from 50,” with the mean absolute deviation in phase accuracy shown in
parentheses. The bars in the top row are evaluated by five random orderings of
classes.

This section investigates whether our rectified cosine normalization could
also benefit iCaRL [1]. Compared in Fig. 12 are iCaRL [1] with (denoted as
Norm-iCaRL) and without (denoted as iCaRL) rectified cosine normalization.
Note that the other design aspects of iCaRL remain the same. As a benchmark,
our scheme is also presented. A comparison between Norm-iCaRL and iCaRL in the
top row of Fig. 12 reveals that rectified cosine normalization does help
improve the incremental accuracy of iCaRL [1], reducing largely the
performance gap between ours and iCaRL [1]. However, without our weighted-
Euclidean regularization, both Norm-iCaRL and iCaRL [1] exhibit higher
variations in phase accuracy than ours (cp. the bottom row of Fig. 12).

##  Appendix 0.E Adding 5 New Classes in Each Phase

Here we conduct more experiments based on almost the same setting as the one
in the main manuscript (i.e. CIFAR-100 and ImageNet datasets, two training
scenarios, memory size of 1000 and 2000, as well as the three evaluation
metrics), but now the number of new classes in each incremental phase is set
to 5 (e.g. in total 20 phases while training from scratch).

###  0.E.1 Incremental Accuracy Comparison

(a) $M$: 1K, FS

(b) $M$: 2K, FS

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

(e) $M$: 1K, FS

(f) $M$: 2K, FS

(g) $M$: 1K, from 50

(h) $M$: 2K, from 50

Figure 13: Incremental accuracy on CIFAR-100 (top row) and ImageNet (bottom row), for memory sizes $M=1K,2K$ and training scenarios “FS” and “from 50”. Average incremental accuracy of each method is shown in parentheses. Table 2: Comparison of incremental accuracy at the end of training Training scenario | from scratch |  | from 50 classes
---|---|---|---
Dataset | CIFAR |  | ImageNet |  | CIFAR |  | ImageNet
Memory size | 1K | 2K |  | 1K | 2K |  | 1K | 2K |  | 1K | 2K
iCaRL [1] | 31.8 | 37.7 |  | 42.7 | 47.9 |  | 42.5 | 49.4 |  | 47.4 | 52.8
End-to-End [17] | 29.6 | 37.7 |  | 37.2 | 46.8 |  | 39.4 | 48.6 |  | 41.9 | 51.1
BIC [20] | 28.8 | 37.8 |  | 38.9 | 46.9 |  | 40.2 | 46.6 |  | 43.3 | 51.2
Hou19 [18] | 28.1 | 36.5 |  | 23.2 | 32.4 |  | 41.5 | 48.8 |  | 44.2 | 51.4
Mnemonics [21] | 32.7 | 39.1 |  | 30.9 | 38.6 |  | 52.8 | 55.0 |  | 58.7 | 61.4
Ours | 36.1 | 40.5 |  | 44.2 | 49.8 |  | 43.4 | 48.4 |  | 49.5 | 53.6

The quantitative evaluation results in terms of incremental accuracy are
provided in Fig. 13, while Table 2 particularly summarizes the results at the
end of the entire incremental-training.

Similar to the observations we have in the main manuscript, on CIFAR-100 and
ImageNet, when learning from scratch, our model outperforms all the baselines
with memory size 1000 and 2000. Also, we find again that Hou19 [18] and
Mnemonics [21] are unable to perform well when the feature extractor is
learned from scratch. When starting from 50 classes, ours performs comparably
to Mnemonics [21] and outperforms all the other baselines on ImageNet. On
CIFAR-100, our model performs comparably to the other baselines with memory
size 2000 and achieves the second highest incremental accuracy with memory
size 1000. When starting from 50 classes, ours outperforms all the other
baselines, except Mnemonics [21], on ImageNet. On CIFAR-100, our model
performs comparably to the other baselines with memory size 2000 and achieves
the second highest incremental accuracy with memory size 1000.

###  0.E.2 Phase Accuracy Comparison

(a) $M$: 1K, from 50

(b) $M$: 2K, from 50

(c) $M$: 1K, from 50

(d) $M$: 2K, from 50

Figure 14: Phase accuracy comparison: (a)(b) are results on CIFAR-100 and
(c)(d) on ImageNet. $M$ is the memory size, and the model is pre-trained with
50 classes. The mean absolute deviation in phase accuracy is shown in
parentheses.

Fig. 14 presents the phase accuracy for different methods with the training
scenario “from 50”, where the baselines perform more closely to our method in
terms of incremental accuracy. Shown in parentheses is the mean absolute
deviation from the average of each method’s phase accuracy. The smaller the
deviation, the more balanced the classification accuracy is on different
classes. From the figure, our scheme is seen to achieve the minimum mean
absolute deviation in phase accuracy on CIFAR-100 and comparable mean absolute
deviation to iCaRL [1] and Mnemonics [21] on ImageNet.

##  Appendix 0.F Forgetting Measure

Table 3: Comparison of forgetting measure under the setting that learns 10 new classes in each phase. Training scenario | from scratch |  | from 50 classes
---|---|---|---
Dataset | CIFAR |  | ImageNet |  | CIFAR |  | ImageNet
Memory size | 1K | 2K |  | 1K | 2K |  | 1K | 2K |  | 1K | 2K
iCaRL [1] | 24.6 | 20.6 |  | 22.3 | 20.6 |  | 24.6 | 20.4 |  | 24.1 | 20.4
End-to-End [17] | 41.6 | 32.9 |  | 39.1 | 31.0 |  | 38.2 | 29.1 |  | 37.2 | 28.4
BIC [20] | 37.3 | 28.6 |  | 36.1 | 31.4 |  | 33.5 | 26.4 |  | 33.9 | 27.3
Hou19 [18] | 27.3 | 16.8 |  | 38.7 | 28.8 |  | 17.2 | 11.1 |  | 20.4 | 13.4
Mnemonics [21] | 39.9 | 33.1 |  | 38.4 | 31.7 |  | 22.8 | 18.9 |  | 15.9 | 14.0
Ours | 25.0 | 22.3 |  | 24.6 | 23.6 |  | 17.7 | 16.1 |  | 18.4 | 17.8
Table 4: Comparison of forgetting measure under the setting that learns 5 new classes in each phase Training scenario | from scratch |  | from 50 classes
---|---|---|---
Dataset | CIFAR |  | ImageNet |  | CIFAR |  | ImageNet
Memory size | 1K | 2K |  | 1K | 2K |  | 1K | 2K |  | 1K | 2K
iCaRL [1] | 34.5 | 29.4 |  | 34.1 | 30.6 |  | 32.7 | 26.7 |  | 36.3 | 30.5
End-to-End [17] | 55.6 | 44.2 |  | 56.1 | 43.8 |  | 47.4 | 36.2 |  | 49.4 | 37.9
BIC [20] | 50.4 | 40.9 |  | 54.6 | 45.7 |  | 43.2 | 35.9 |  | 48.5 | 39.9
Hou19 [18] | 61.7 | 51.6 |  | 72.5 | 63.0 |  | 46.6 | 38.2 |  | 50.3 | 41.8
Mnemonics [21] | 47.4 | 39.4 |  | 49.8 | 39.5 |  | 24.8 | 20.0 |  | 20.2 | 20.3
Ours | 28.0 | 23.9 |  | 35.6 | 30.6 |  | 26.4 | 21.8 |  | 32.1 | 27.8

In this section, we present the results of forgetting measure [11] on every
class at the end of training, in order to understand to what degree the
competing methods forget the previously learned classes. Generally, the lower
the forgetting measure, the less the catastrophic forgetting. Table 3
summarizes the results for the setting that learns 10 new classes in each
training phase. It shows that our method has low forgetting measure and
performs comparably to iCaRL [1], Hou19 [18] and Mnemonics [21].

Table 4 further provides the results for the setting of learning 5 new classes
in each phase. In this setting, the model needs to be updated more frequently
and tends to forget more easily what it has learned. On CIFAR-100, ours
achieves the lowest forgetting measure, when learning from scratch. Moreover,
it has the second lowest forgetting measure with the training scenario “from
50”. On ImageNet, iCaRL [1] performs comparably to ours, when learning from
scratch. Since Hou19 [18] and Mnemonics [21] are unable to perform well when
learning from scratch, they suffer from serious catastrophic forgetting,
showing large forgetting measure. When learning from 50 classes, ours performs
comparably to Mnemonics [21] and outperforms the other baselines.

To sum up, our method has the ability to preserve well the knowledge of old
classes under various settings, while showing high incremental accuracy in
learning new classes and more balanced phase accuracy.