# Online Continual Learning in Image Classification:
An Empirical Survey

Zheda Mai<EMAIL_ADDRESS>Ruiwen Li  Jihwan Jeong  David Quispe
Hyunwoo Kim  Scott Sanner  Department of Mechanical and Industrial
Engineering, University of Toronto, Toronto, 5 King’s College Road, ON M5S3G8,
Canada  (zheda.mai, ruiwen.li<EMAIL_ADDRESS>(jhjeong,
<EMAIL_ADDRESS>LG AI Research, 128, Yeoui-daero, Yeongdeungpo-gu,
Seoul, South Korea<EMAIL_ADDRESS>
###### Abstract

Online continual learning for image classification studies the problem of
learning to classify images from an online stream of data and tasks, where
tasks may include different classes (class incremental) or data
nonstationarity (domain incremental). One of the key challenges of continual
learning is to avoid catastrophic forgetting (CF), i.e., forgetting old tasks
in the presence of more recent tasks. Over the past few years, a large range
of methods and tricks have been introduced to address the continual learning
problem, but many have not been fairly and systematically compared under a
variety of realistic and practical settings.

To better understand the relative advantages of various approaches and the
settings where they work best, this survey aims to (1) compare state-of-the-
art methods such as Maximally Interfered Retrieval (MIR), iCARL, and GDumb (a
very strong baseline) and determine which works best at different memory and
data settings as well as better understand the key source of CF; (2) determine
if the best online class incremental methods are also competitive in domain
incremental setting; and (3) evaluate the performance of 7 simple but
effective trick such as ”review” trick and nearest class mean (NCM) classifier
to assess their relative impact. Regarding (1), we observe earlier proposed
iCaRL remains competitive when the memory buffer is small; GDumb outperforms
many recently proposed methods in medium-size datasets and MIR performs the
best in larger-scale dataset. For (2), we note that GDumb performs quite
poorly while MIR — already competitive for (1) — is also strongly competitive
in this very different (but important) incremental learning setting. Overall,
this allows us to conclude that MIR is overall a strong and versatile online
continual learning method across a wide variety of settings. Finally for (3),
we find that all tricks are beneficial, and when augmented with ”review” trick
and NCM classifier, MIR produces performance levels that bring online
continual learning much closer to its ultimate goal of matching offline
training.

###### keywords:

Incremental Learning , Continual Learning , Lifelong Learning , Catastrophic
Forgetting , Online Learning

††journal: Neurocomputing

##  1 Introduction

With the ubiquity of personal smart devices and image-related applications, a
massive amount of image data is generated daily. While image-based deep neural
networks have demonstrated exceptional advances in recent years [1],
incrementally updating a neural network with a nonstationary data stream
results in _catastrophic forgetting_ (CF) [2, 3], the inability of a network
to perform well on previously seen data after updating with recent data. For
this reason, conventional deep learning tends to focus on offline training,
where each mini-batch is sampled i.i.d from a static dataset with multiple
epochs over the training data. However, to accommodate changes in the data
distribution, such a training scheme requires entirely retraining the network
on the new dataset, which is inefficient and sometimes infeasible when
previous data are not available due to storage limits or privacy issues.

_Continual Learning_ (CL) studies the problem of learning from a non-i.i.d
stream of data, with the goal of preserving and extending the acquired
knowledge. A more complex and general viewpoint of CL is the stability-
plasticity dilemma [4, 5] where stability refers to the ability to preserve
past knowledge and plasticity denotes the fast adaptation of new knowledge.
Following this viewpoint, CL seeks to strike a balance between learning
stability and plasticity. Since CL is often used interchangeably with lifelong
learning [6, 7] and incremental learning [8, 9], for simplicity, we will use
CL to refer to all concepts mentioned above.

Most early CL approaches consider the task incremental setting [10]. In this
setting, new data arrives one task at a time, and the model can utilize task-
IDs during both training and inference time [11, 12, 13]. Hence, a common
practice in this context is to assign a separate output layer (head) for each
task; then the model just needs to classify labels within a task, which is
known as the multi-head evaluation [9]. However, the multi-head evaluation
requires additional supervisory signals at inference — namely the task-ID — to
select the corresponding head, which obviates its use when the task label is
unavailable. Moreover, this setting needs to store all the current task data
in the memory, and thus, it is not friendly for edge devices with limited
resources.

In this work, we focus on two realistic but challenging settings, known as
_Online Class Incremental_ (OCI) and _Online Class Incremental_ (ODI). In
these settings, a model needs to learn from an online stream of data, with
each sample being seen only once. The incoming data either include new classes
(class incremental) or data nonstationarity (domain incremental). In contrast
to the task incremental setting, OCI and ODI have two main differences. (1)
Single-head evaluation is adopted: the model needs to classify all labels
without task-IDs [9]. (2) The model is required to process data online, which
reduces the adaptation time and operational memory usage. These settings are
based on the practical CL desiderata proposed recently [14, 15, 16] and have
received much attention in the past year [17, 18, 19].

To keep this paper focused, we only consider the supervised classification
problem in computer vision. Although CL is also studied in reinforcement
learning [20, 21, 22] and more recently in unsupervised learning [23], the
image classification problem is still the main focus for many CL researchers.

Over the past few years, a broad range of methods and tricks have been
introduced to address CL problems, but many have not been fairly and
systematically compared under a variety of settings. To better understand the
relative advantages of different approaches and the settings where they work
best, this survey aims to do the following:

  * 1.

We fairly compare state-of-the-art methods in OCI and determine which works
best at different memory and data settings. We observe earlier proposed iCaRL
[8] remains competitive when the memory buffer is small; GDumb [24] is a
strong baseline that outperforms many recently proposed methods in medium-size
datasets, while MIR [17] performs the best in a larger-scale dataset. Also, we
experimentally and theoretically confirm that a key cause of CF is due to the
recency learning bias towards new classes in the last fully connected layer
owing to the imbalance between previous data and new data.

  * 2.

We determine if the best OCI methods are also competitive in the ODI setting.
We note that GDumb performs quite poorly in ODI, whereas MIR — already
competitive in OCI — is still strongly competitive in ODI. Overall, these
results allow us to conclude that MIR is a strong and versatile online CL
method across a wide variety of settings.

  * 3.

We evaluate the performance of 7 simple but effective tricks to assess their
relative impacts. We find that all tricks are beneficial and when augmented
with the ”review” trick [25] and a nearest class mean (NCM) classifier [8],
MIR produces performance levels that bring online CL much closer to its
ultimate goal of matching offline training.

The remainder of this paper is organized as follows. Section 2 discusses the
existing surveys in the CL community, and Section 3 formally defines the
problem, settings and evaluation metrics. In Section 4, we explain the online
continual hyperparameter tuning method we use. Then, Section 5 provides an
overview of state-of-the-art CL techniques, while Section 6 gives a detailed
description of methods that we compared in experiments. We discuss how class
imbalance results in catastrophic forgetting and introduce CL tricks that can
effectively alleviate the forgetting in Section 7. We outline our experimental
setup, comparative evaluation and key findings in Section 8. Finally, Section
9 discusses recent trends and emerging directions in CL, and we conclude in
Section 10.

##  2 Related Work

With the surge in the popularity of CL, there are multiple reviews and surveys
covering the advances of CL. The first group of surveys are not empirical.
[26] discusses the biological perspective of CL and summarizes how various
approaches alleviate catastrophic forgetting. [14] formalizes the CL problem
and outlines the existing benchmarks, metrics, approaches and evaluation
methods with the emphasis on robotics applications. They also recommend some
desiderata and guidelines for future CL research. [27] emphasizes the
importance of online CL and discusses recent advances in this setting.
Although these three surveys descriptively review the recent development of CL
and provide practical guidelines, they do not perform any empirical comparison
between methods.

In contrast, the second group of surveys on CL are empirical. For example,
[28, 10] evaluate multiple CL methods on three CL scenarios: task incremental,
class incremental and domain incremental. [16] empirically analyzes and
criticizes some common experimental settings, including the multi-head
evaluation [9] with an exclusive output layer for each task and the use of
permuted-type datasets (e.g., permuted MNIST). However, the analysis in these
three works is limited to small datasets such as MNIST and Fashion-MNIST.
Another two empirical studies on the performance of CL include [29, 30], but
only a small number of CL methods are compared. The first extensive
comparative CL survey with empirical analysis is presented in [15], which
focuses on the task incremental setting with the multi-head evaluation. In
contrast, our work addresses more practical and realistic settings, namely
Online Class Incremental (OCI) and Online Domain Incremental (ODI), which
require the model to learn online without access to task-ID at training and
inference time.

##  3 Online Class/Domain Incremental Learning

###  3.1 Problem definition and evaluation settings

We consider the supervised image classification problem with an online
(potentially infinite) non-i.i.d stream of data, following the recent CL
literature [17, 18, 27, 14]. Formally, we define a data stream of unknown
distributions $\mathcal{D}=\\{D_{1},\ldots,D_{N}\\}$ over $X\times Y$, where
$X$ and $Y$ are input and output random variables respectively, and a neural
network classifier parameterized by $\theta$, $f\mathrel{\mathop{\mathchar
58\relax}}X\mapsto\mathbb{R}^{C}$ where $C$ is the number of classes observed
so far as in [14]. At time $t$, a CL algorithm $A^{CL}$ receives a mini-batch
of samples ($\mathit{x_{t}^{i}}$, $\mathit{y_{t}^{i}}$) from the current
distribution $D_{i}$, and the algorithm only sees this mini-batch once.

An algorithm $A^{CL}$ is defined with the following signature:

$\displaystyle A_{t}^{CL}\mathrel{\mathop{\mathchar 58\relax}}\ \langle
f_{t-1},(\mathit{x_{t}},\mathit{y_{t}}),M_{t-1}\rangle\ \rightarrow\ \langle
f_{t},M_{t}\rangle$ (1)

Where:

  * 1.

$f_{t}$ is the classifier at time step $t$.

  * 2.

($\mathit{x_{t}^{i}}$, $\mathit{y_{t}^{i}}$) is a mini-batch received at time
$t$ from $D_{i}$ which contains
$\\{(\mathit{x_{tj}^{i}},\mathit{y_{tj}^{i}})\mid j\in[1,\ldots,b]\\}$ where b
is the mini-batch size.

  * 3.

$M_{t}$ is an external memory that can be used to store a subset of the
training samples or other useful data (e.g., the classifier from the previous
time step as in LwF [12]). Note that the online setting does not limit the
usage of the samples in $M$, and therefore, the classifier $f_{t}$ can use
them as many times as it wants.

Note that we assume, for simplicity, a locally i.i.d stream of data where each
task distribution $D_{i}$ is stationary as in [13, 31]; however, this
framework can also accommodate the setting in which samples are drawn
non-i.i.d from $D_{i}$ as in [32, 33], where concept drift may occur within
$D_{i}$.

The goal of $A^{CL}$ is to train the classifier $f$ to continually learn new
samples from the data stream without interfering with the performance of
previously observed samples. Note that unless the current samples are stored
in $M_{t}$, $A^{CL}$ will not have access to these sample in the future.
Formally, at time step $\tau$, $A^{CL}$ tries to minimize the loss incurred by
all the previously seen samples with only access to the current mini-batch and
data from $M_{\tau-1}$:

$\displaystyle\mathrm{min}_{\theta}\sum_{t=1}^{\tau}\mathbb{E}_{\left(\mathit{x}_{t},\mathit{y}_{t}\right)}\left[\ell\left(f_{\tau}\left(\mathit{x}_{t};\theta\right),\mathit{y}_{t}\right)\right]$
(2)

Scenario |  Difference between $D_{i-1}$ and $D_{i}$ | Task-ID | Online
---|---|---|---
$P(X_{i-1})\neq P(X_{t})$ | $P(Y_{i-1})\neq P(Y_{i})$ | $\\{Y_{i-1}\\}\neq\\{Y_{i}\\}$
Task Incremental | ✓ | ✓ | ✓ | Train & Test | No
Class Incremental | ✓ | ✓ |  | No | Optional
Domain Incremental | ✓ |  |  | No | Optional
Table 1: Three continual learning scenarios based on the difference between $D_{i-1}$ and $D_{i}$, following [28]. $P(X)$ is the input data distribution; $P(Y)$ is the target label distribution; $\\{Y_{i-1}\\}\neq\\{Y_{i}\\}$ denotes that output space are from a disjoint space which is separated by task-ID. Task | Task Incremental | Class Incremental | Domain Incremental
---|---|---|---
$D_{i-1}$ |  x: |  |  |  x: |  |  |  x: |  |
y: | Bird | Dog |  y: | Bird | Dog |  y: | Bird | Dog
task-ID(test) | i-1 | Unknown | Unknown
$D_{i}$ |  x: |  |  |  x: |  |  |  x: |  |
y: | Ship | Guitar |  y: | Ship | Guitar |  y: | Bird | Dog
task-ID(test) | i | Unknown | Unknown
Table 2: Examples of the three CL scenarios. (x, y, task-ID) represents (input
images, target label and task identity). The main distinction between task
incremental and class incremental is the availability of task-ID. The main
difference between class incremental and domain incremental is that, in class
incremental, a new task contains completely new classes, whereas domain
incremental, a new task consists of new instances with nonstationarity (e.g.,
noise) of all the seen classes.

Recently, [28, 10] have categorized the CL problem into three scenarios based
on the difference between $D_{i-1}$ and $D_{i}$. Table 1 summarizes the
differences between the three scenarios, i.e., task incremental, class
incremental and domain incremental. For task incremental, the output spaces
are separated by task-IDs and are disjoint between $D_{i-1}$ and $D_{i}$. We
denote this setting as $\\{Y_{i-1}\\}\neq\\{Y_{i}\\}$, which in turn leads to
$P(Y_{i-1})\neq P(Y_{i})$. In this setting, task-IDs are available during both
train and test times. For class incremental, mutually exclusive sets of
classes comprise each data distribution $D_{i}$, meaning that there is no
duplicated class among different task distributions. Thus $P(Y_{i-1})\neq
P(Y_{i})$, but the output space is the same for all distributions since this
setting adopts the single-head evaluation where the model needs to classify
all labels without a task-ID. Domain incremental represents the setting where
input distributions are different, while the output spaces and distribution
are the same. Note that task IDs are not available for both class and domain
incremental. Table 2 shows examples of these three scenarios. Following this
categorization, the settings we focus in this work are known as Online Class
Incremental (OCI) and Online Domain Incremental (ODI).

###  3.2 Metrics

Besides measuring the final accuracy across tasks, it is also critical to
assess how fast a model learns, how much the model forgets and how well the
model transfers knowledge from one task to another. To this end, we use five
standard metrics in the CL literature to measure performance: (1) the average
accuracy for overall performance [31]; (2) the average forgetting to measure
how much of the acquired knowledge the model has forgotten [9]; (3) the
forward transfer and (4) the backward transfer to assess the ability for
knowledge transfer [14, 13]; (5) the total running time, including training
and testing times.

Formally, we define $a_{i,j}$ as the accuracy evaluated on the held-out test
set of task $j$ after training the network from task 1 through to $i$, and we
assume there are $T$ tasks in total.

Average Accuracy can be defined as Eq. (3). When $i=T$, $A_{T}$ represents the
average accuracy by the end of training with the whole data sequence (see
example in Table 3).

$\displaystyle\text{Average Accuracy}(A_{i})=\frac{1}{i}\sum_{j=1}^{i}a_{i,j}$
(3)

Average Forgetting at task $i$ is defined as Eq. (4). $f_{i,j}$ represents how
much the model has forgot about task $j$ after being trained on task $i$.
Specifically, $\max\limits_{l\in\\{1,\cdots,k-1\\}}(a_{l,j})$ denotes the best
test accuracy the model has ever achieved on task j before learning task $k$,
and $a_{k,j}$ is the test accuracy on task $j$ after learning task $k$.

$\displaystyle\text{Average
Forgetting}(F_{i})=\frac{1}{i-1}\sum_{j=1}^{i-1}f_{i,j}$
$\displaystyle\text{where~{}}f_{k,j}=\max_{l\in\\{1,\cdots,k-1\\}}(a_{l,j})-a_{k,j},\forall
j<k$ (4)

a | $te_{1}$ | $te_{2}$ | $\dots$ | $te_{T-1}$ | $te_{T}$
---|---|---|---|---|---
$tr_{1}$ | $a_{1,1}$ | $a_{1,2}$ | $\dots$ | $a_{1,T-1}$ | $a_{1,T}$
$tr_{2}$ | $a_{2,1}$ | $a_{2,2}$ | $\dots$ | $a_{2,T-1}$ | $a_{2,T}$
$\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$ | $\dots$
$tr_{T-1}$ | $a_{T-1,1}$ | $a_{T-1,2}$ | $\dots$ | $a_{T-1,T-1}$ | $a_{T-1,T}$
$tr_{T}$ |  $a_{T,1}$ | $a_{T,2}$ | $\dots$ | $a_{T,T-1}$ | $a_{T,T}$
Table 3: Accuracy matrix example following the notations in [14]. $tr_{i}$ and
$te_{i}$ denote training and test set of task $i$. $A_{T}$ is the average of
accuracies in the box. $BWT^{+}$ is the average of accuracies in purple and
$FWT$ is the average of accuracies in green.

Positive Backward Transfer($BWT^{+}$) measures the positive influence of
learning a new task on preceding tasks’ performance (see example in Table 3).

$\displaystyle
BWT^{+}=max(\frac{\sum_{i=2}^{T}\sum_{j=1}^{i-1}\left(a_{i,j}-a_{j,j}\right)}{\frac{T(T-1)}{2}},0)$
(5)

Forward Transfer($FWT$) measures the positive influence of learning a task on
future tasks’ performance (see example in Table 3).

$\displaystyle
FWT=\frac{\sum_{i=1}^{j-1}\sum_{j=1}^{T}a_{i,j}}{\frac{T(T-1)}{2}}$ (6)

##  4 Online Continual Hyperparameter Tuning

In practice, most CL methods have to rely on well-selected hyperparameters to
effectively balance the trade-off between stability and plasticity.
Hyperparameter tuning, however, is already a challenging task in learning
conventional deep neural network models, which becomes even more complicated
in the online CL setting. Meanwhile, a large volume of CL works still tune
hyperparameters in an offline manner by sweeping over the whole data sequence
and selecting the best hyperparameter set with grid-search on a validation
set. After that, metrics are reported on the test set with the selected set of
hyperparameters. This tuning protocol violates the online CL setting where a
classifier can only make a single pass over the data, which implies that the
reported results in the CL literature may be too ideal and cannot be
reproduced in real online CL applications.

Recently, several hyperparameter tuning protocols that are useful for CL
settings have been proposed. [29] introduces a tuning protocol for two tasks
($D_{1}$ and $D_{2}$). Firstly, they determine the best combination of model
hyperparameters using $D_{1}$. Then, they tune the learning rate to be used
for learning $D_{2}$ such that the test accuracy on $D_{2}$ is maximized. [15]
proposes another protocol that dynamically determines the stability-plasticity
trade-off. When the model receives a new task, the hyperparameters are set to
ensure minimal forgetting of previous tasks. If a predefined threshold for the
current task’s performance is not met, the hyperparameters are adjusted until
achieving the threshold. While the aforementioned protocols assume the data of
a new task is available all at once, [31] presents a protocol targeting the
online CL. Specifically, a data stream is divided into two sub-streams —
$D^{CV}$, the stream for cross-validation and $D^{EV}$, the stream for final
training and evaluation. Multiple passes over $D^{CV}$ are allowed for tuning,
but a CL algorithm can only perform a single pass over $D^{EV}$ for training.
The metrics are reported on the test sets of $D^{EV}$. Since the setting of
our interest in this work is the online CL, we adopt the protocol from [31]
for our experiments. We summarize the protocol in Algorithm 1.

Input : Hyperparameter set $\mathcal{P}$

Require: $D^{CV}$ data stream for tuning, $D^{EV}$ data stream for learning &
testing

Require: $f$ classifier, $A^{CL}$ CL algorithm

1

2for  _$p\in\mathcal{P}$_ do $\triangleright$ Multiple passes over $D^{CV}$
for tuning

3 for  _$i\in\\{1,\dots,T^{CV}\\}$_ do $\triangleright$ Single pass over
$D^{CV}$ with $p$

4 for  _$B_{n}\sim D^{CV}_{i}$_ do

5 $A^{CL}(f,B_{n},p)$

6 $\text{Evaluate}(f,D^{CV}_{test})$ $\triangleright$ Store performance on
test set of $D^{CV}$

7

8$\text{Best hyperparameters, }\boldsymbol{p^{*}}\leftarrow$ based on Average
Accuracy of $D^{CV}_{test}$, see Eq.(3)

9for  _$i\in\\{1,\dots,T^{EV}\\}$_ do $\triangleright$ Learning over $D^{EV}$

10 for  _$B_{n}\sim D^{EV}_{i}$_ do $\triangleright$ Single pass over $D^{EV}$
with $p^{*}$

11 $A^{CL}(f,B_{n},\boldsymbol{p^{*}})$

12 $\text{Evaluate}(f,D^{EV}_{test})$

Report performance on $D^{EV}_{test}$

Algorithm 1 Hyperparameter Tuning Protocol

##  5 Overview of Continual Learning Techniques

Although a broad range of methods have been introduced in the past few years
to address the CL problem, the assumptions that each method makes are not
consistent due to the plethora of settings in CL. In particular, some methods
have a better ability to generalize to different CL settings because they
require less supervisory signals during both training and inference times. For
example, ER [34] was designed for the task incremental setting, but the method
can easily be used in all the other CL settings since it does not need any
additional supervisory signals. In this sense, a systematic summary of
supervisory signals that different methods demand will help understand the
generalization capacity and limitations of the methods. Furthermore, the
summary will facilitate fair comparison in the literature. On the other hand,
CL methods have typically been taxonomized into three major categories based
on the techniques they use: regularization-based, memory-based and parameter-
isolation-based [15, 26]. A clear trend in recent works, however, is to
simultaneously apply multiple techniques in order to tackle the CL problem. In
this section, we comprehensively summarize recently proposed methods based on
techniques they use and supervisory signals required at training and inference
times (see Table 4).

#### Supervisory Signals

The most important supervisory signal is the task-ID. When a task-ID is
available, a training or testing sample is given as $(x,y,t)$ instead of
$(x,y)$ where $t$ is the task-ID. For the task incremental setting, task-IDs
are available at both training and inference times. In regards to the class
incremental setting, a task-ID is not available at the inference time but can
be inferred during training as each task has disjoint class labels. On the
other hand, in the domain incremental setting, a task-ID is not available at
all times. Other supervisory signals include a natural language description of
a task or a matrix specifying the attribute values of the objects to be
recognized in the task [31].

Moreover, a method is online-able if it does not need to revisit samples it
has processed before. Hence, to be online-able, the model needs to learn
efficiently from one pass of the data. For example, the herding-based memory
update strategy proposed in iCaRL [8] needs all the samples from a class to
select a representative set; therefore, methods using this strategy are not
online-able.

#### Techniques

Regularization techniques impose constraints on the update of network
parameters to mitigate catastrophic forgetting. This is done by either
incorporating additional penalty terms into the loss function [35, 36, 37, 38]
or modifying the gradient of parameters during optimization [13, 31, 39].

Knowledge Distillation (KD) [40] is an effective way for knowledge transfer
between networks. KD has been widely adopted in CL methods [41, 12, 42, 43],
and it is often considered as one of the regularization techniques. Due to the
prevalence of KD in CL methods, we list it as a separate technique in Table 4.
One shortcoming of regularization-based techniques including KD is that it is
difficult to strike a balance between the regularization and the current
learning when learning from a long stream of data.

Memory-based techniques store a subset of samples from previous tasks for
either replay while training on new task [34, 17, 18] or regularization
purpose [44, 13, 45]. These methods become infeasible when storing raw samples
is not possible due to privacy or storage concerns.

Instead of saving raw samples, an alternative is Generative Replay which
trains a deep generative model such as GAN [46] to generate pseudo-data that
mimic past data for replay [47, 48, 49]. The main disadvantages of generative
replay are that it takes long time to train such generative models and that it
is not a viable option for more complex datasets given the current state of
deep generative models [50, 17].

Parameter-isolation (PI)-based techniques bypass interference by allocating
different parameters to each task. PI can be subdivided into Fixed
Architecture (FA) that only activates relevant parameters for each task
without modifying the architecture [51, 52, 53], and Dynamic Architecture (DA)
that adds new parameters for new tasks while keeping old parameters unchanged
[54, 55, 56]. Most previous works require task-IDs at inference time, and a
few recent methods have been introduced to predict without a task-ID [19, 57].

For a more detailed discussion of these techniques, we refer readers to the
recent CL surveys [15, 26, 14].

| Settings | Techniques
---|---|---
Methods | t-ID free(test) | t-ID free(train) | Online-able | Reg | Mem | KD | PI(FA) | PI(DA) | Generative
MIR[17], GSS[18], ER[34], CBO[58] | ✓ | ✓ | ✓ |  | ✓ |  |  |  |
GDumb[24], DER[59], MER[60]
CBRS[61], GMED[62], PRS[63]
La-MAML[64], MEFA[65]
MERLIN[66] |  | ✓ |  | ✓ |  |
CN-DPM[19], TreeCNN[67] |  |  |  |  | ✓ |
A-GEM[31], GEM[13], VCL [44] | ✓ | ✓ |  |  |  |
WA[68], BiC[42], LUCIR[69] |  | ✓ | ✓ |  |  |
IL2M[70], ILO[71] |  |  |  |
LwF-MC[8], LwM[72], DMC[73] |  |  | ✓ |  |  |
SRM[74], AQM[75] |  | ✓ |  |  |  | ✓
EWC++[9] | ✓ |  |  |  |  |
AR1[76] | ✓ |  |  | ✓ |  |
EEIL[41], iCaRL[8], MCIL[77] | ✓ | ✓ | ✗ |  | ✓ | ✓ |  |  |
SDC[78] | ✓ |  |  |  |  |
DGR[47] |  |  |  |  |  | ✓
DGM[79] |  |  |  | ✓ | ✓ | ✓
ICGAN[80], RtF[49] |  |  |  |  |  | ✓ |  |  | ✓
iTAML[81],CCG[82] | ✓ | ✗ | ✓ |  | ✓ |  | ✓ |  |
|  |  |  |  |  |  |  |  |
Table 4: Summary of recently proposed CL methods based on supervisory signals
required and techniques they use. t-ID free(test/train) means task-ID is not
required at test/train time.

##  6 Compared Methods

###  6.1 Regularization-based methods

#### Elastic Weight Consolidation (EWC)

EWC [11] incorporates a quadratic penalty to regularize the update of model
parameters that were important to past tasks. The importance of parameters is
approximated by the diagonal of the Fisher Information Matrix $F$. Assuming a
model sees two tasks A and B in sequence, the loss function of EWC is:

$\displaystyle\mathcal{L}(\theta)=\mathcal{L}_{B}(\theta)+\sum_{j}\frac{\lambda}{2}F_{j}\left(\theta_{j}-\theta_{A,j}^{*}\right)^{2}$
(7)

where $\mathcal{L}_{B}(\theta)$ is the loss for task B, $\theta^{*}_{A,j}$ is
the optimal value of $j^{th}$ parameter after learning task A and $\lambda$
controls the regularization strength. There are three major limitations of
EWC: (1) It requires storing the Fisher Information Matrix for each task,
which makes it impractical for a long sequence of tasks or models with
millions of parameters. (2) It needs an extra pass over each task at the end
of training, leading to its infeasibility for the online CL setting. (3)
Assuming the Fisher to be diagonal may not be accurate enough in practice.
Several variants of EWC are proposed lately to address these limitations [83,
9, 84]. As we use the online CL setting, we compare EWC++, an efficient and
online version of EWC that keeps a single Fisher Information Matrix calculated
by moving average. Specifically, given $F^{t-1}$ at $t-1$, the Fisher
Information Matrix at $t$ is updated as:

$\displaystyle F^{t}=\alpha F_{tmp}^{t}+(1-\alpha)F^{t-1}$ (8)

where $F_{tmp}^{t}$ is the Fisher Information Matrix calculated with the
current batch of data and $\alpha\in[0,1]$ is a hyperparameter controlling the
strength of favouring the current $F^{t}$.

#### Learning without Forgetting (LwF)

LwF [12] utilizes knowledge distillation [40] to preserve knowledge from past
tasks in the multi-head setting [9]. In LwF, the teacher model is the model
after learning the last task, and the student model is the model trained with
the current task. Concretely, when the model receives a new task ($X_{n}$,
$Y_{n}$), LwF computes $Y_{o}$, the output of old tasks for the new data
$X_{n}$. During training, LwF optimizes the following loss:

$\mathcal{L}(\theta)=\left(\lambda_{o}\mathcal{L}_{\text{KD}}\left(Y_{o},\hat{Y}_{o}\right)+\mathcal{L}_{\text{CE}}\left(Y_{n},\hat{Y}_{n}\right)+\mathcal{R}\left(\theta\right)\right)$
(9)

where $\hat{Y}_{o}$ and $\hat{Y}_{n}$ are the predicted values of the old task
and new task using the same $X_{n}$. $\mathcal{L}_{\text{KD}}$ is the
knowledge distillation loss incorporated to impose output stability of old
tasks with new data and $\mathcal{L}_{\text{CE}}$ is the cross-entropy loss
for the new task. $\mathcal{R}$ is a regularization term, and $\lambda_{o}$ is
a hyperparameter controlling the strength of favouring the old tasks over the
new task. A known shortcoming of LwF is its heavy reliance on the relatedness
between the new and old tasks. Thus, LwF may not perform well when the
distributions of the new and old tasks are different [12, 8, 56]. To apply LwF
in the single-head setting where all tasks share the same output head, a
variant of LwF (LwF.MC) is proposed in [8], and we evaluate LwF.MC in this
work.

###  6.2 Memory-based methods

A generic online memory-based method is presented in Algorithm 2. For every
incoming mini-batch, the algorithm retrieves another mini-batch from a memory
buffer, updates the model using both the incoming and memory mini-batches and
then updates the memory buffer with the incoming mini-batch. What
differentiate various memory-based methods are the memory retrieval strategy
(line 3) [17, 85], model update (line 4) [31, 13] and memory update strategy
(line 5) [61, 63, 18].

1

Input : Batch size $b$, Learning rate $\alpha$

Initialize: Memory $\mathcal{M}\leftarrow\\{\\}*M$; Parameters $\theta$;
Counter $n\leftarrow 0$

2 for  _$t\in\\{1,\dots,T\\}$_ do

3 for  _$B_{n}\sim D_{t}$_ do

4 $B_{\mathcal{M}}\\!\\!\leftarrow\\!$ MemoryRetrieval($B_{n},\\!\mathcal{M}$)

5 $\theta\leftarrow~{}\text{{ModelUpdate}}(B_{n}\cup
B_{\mathcal{M}},\theta,\alpha)$

6 $\mathcal{M}\leftarrow$ MemoryUpdate$(B_{n},\mathcal{M})$

7 $n\leftarrow n+b$

8

return $\theta$

Algorithm 2 Generic online Memory-based method

#### Averaged GEM (A-GEM)

A-GEM [31] is a more efficient version of GEM [13]. Both methods prevent
forgetting by constraining the parameter update with the samples in the memory
buffer. At every training step, GEM ensures that the loss of the memory
samples for each individual preceding task does not increase, while A-GEM
ensures that the average loss for all past tasks does not increase.
Specifically, let $g$ be the gradient computed with the incoming mini-batch
and $g_{ref}$ be the gradient computed with the same size mini-batch randomly
selected from the memory buffer. In A-GEM, if $g^{T}g_{ref}\geq 0$, $g$ is
used for gradient update but when $g^{T}g_{ref}<0$, $g$ is projected such that
$g^{T}g_{ref}=0$. The gradient after projection is:

$\displaystyle\tilde{g}=g-\frac{g^{\top}g_{ref}}{g_{ref}^{\top}g_{ref}}g_{ref}$
(10)

As we can see, A-GEM focuses on ModelUpdate in Algorithm 2, and we apply
reservoir sampling [86] in MemoryUpdate and random sampling in MemoryRetrieval
for A-GEM.

#### Incremental Classifier and Representation Learning (iCaRL)

iCaRL [8] is a replay-based method that decouples the representation learning
and classification. For representation learning, the training set is
constructed by mixing all the samples in the memory buffer and the current
task samples. The loss function includes a classification loss to encourage
the model to predict the correct labels for new classes and a KD loss to
prompt the model to reproduce the outputs from the previous model for old
classes. Note that the training set is imbalanced since the number of new-
class samples in the current task is larger than that of the old-class samples
in the memory buffer. iCaRL applies the binary cross entropy (BCE) for each
class to handle the imbalance, but BCE may not be effective in addressing the
relationship between classes. For the classifier, iCaRL uses a nearest-class-
mean classifier [87] with the memory buffer to predict labels for test images.
Moreover, it proposes a MemoryUpdate method based on the distance in the
latent feature space with the inspiration from [88]. For each class, it looks
for a subset of samples whose mean of latent features are closest (in the
Euclidean distance) to the mean of all the samples in this class. However,
this method requires all samples from every class, and therefore it cannot be
applied in the online setting. As such, we modify iCaRL to use reservoir
sampling [86], which has been shown effective for MemoryUpdate [34].

#### Experience Replay (ER)

ER refers to a simple but effective replay-based method that has been
discussed in [34, 89]. It applies reservoir sampling [86] in MemoryUpdate and
random sampling in MemoryRetrieval. Reservoir sampling ensures every streaming
data point has the same probability, ${mem_{sz}}/{n}$, to be stored in the
memory buffer, where $mem_{sz}$ is the size of the buffer and $n$ is the
number of data points observed up to now. We summarize the detail in Algorithm
3 in A. For ModelUpdate, ER simply trains the model with the incoming and
memory mini-batches together using the cross-entropy loss. Despite its
simplicity, recent research has shown that ER outperforms many specifically
designed CL approaches with and without a memory buffer [34].

#### Maximally Interfered Retrieval (MIR)

MIR [17] is a recently proposed replay-based method aiming to improve the
MemoryRetrieval strategy. MIR chooses replay samples according to the loss
increases given the estimated parameter update based on the incoming mini-
batch. Concretely, when receiving a mini-batch $B_{n}$, MIR performs a virtual
parameter update $\theta^{v}\leftarrow\text{SGD}(B_{n},\theta)$. Then it
retrieves the top-k samples from the memory buffer with the criterion
$s(x)=l\left(f_{\theta^{v}}(x),y\right)-l\left(f_{\theta}(x),y\right)$, where
$x\in M$ and $M$ is the memory buffer. Intuitively, MIR selects memory samples
that are maximally interfered (the largest loss increases) by the parameter
update with the incoming mini-batch. MIR applies reservoir sampling in
MemoryUpdate and replays the selected memory samples with new samples in
ModelUpdate.

#### Gradient based Sample Selection (GSS)

GSS [18] is another replay-based method that focuses on the MemoryUpdate
strategy111[18] proposes two gradient-based methods, and we select the more
efficient one with better performance, dubbed GSS-Greedy. Specifically, it
tries to diversify the gradient directions of the samples in the memory
buffer. To this end, GSS maintains a score for each sample in the buffer, and
the score is calculated by the maximal cosine similarity in the gradient space
between the sample and a random subset from the buffer. When a new sample
arrives and the memory buffer is full, a randomly selected subset is used as
the candidate set for replacement. The score of a sample in the candidate set
is compared to the score of the new sample, and the sample with a lower score
is more likely to be stored in the memory buffer. Algorithm 4 in A shows the
main steps of this update method. Same as ER, GSS uses random sampling in
MemoryRetrieval.

#### Greedy Sampler and Dumb Learner (GDumb)

GDumb [24] is not specifically designed for CL problems but shows very
competitive performance. Specifically, it greedily updates the memory buffer
from the data stream with the constraint to keep a balanced class distribution
(Algorithm 5 in A). At inference, it trains a model from scratch using the
balanced memory buffer only.

###  6.3 Parameter-isolation-based methods

#### Continual Neural Dirichlet Process Mixture (CN-DPM)

CN-DPM [19] is one of the first dynamic architecture methods that does not
require a task-ID. The intuition behind this method is that if we train a new
model for a new task and leave the existing models intact, we can retain the
knowledge of the past tasks. Specifically, CN-DPM is comprised of a group of
experts, where each expert is responsible for a subset of the data and the
group is expanded based on the Dirichlet Process Mixture [90] with Sequential
Variational Approximation [91]. Each expert consists of a discriminative model
(classifier) and a generative model (VAE [92] is used in this work). The goal
of CN-DPM is to model the overall conditional distribution as a mixture of
task-wise conditional distributions as the following, where $K$ is the number
of experts in the current model:

$\displaystyle p(y\mid x)=\sum_{k=1}^{K}\underbrace{p(y\mid
x,z=k)}_{\text{discriminative }}\frac{\overbrace{p(x\mid
z=k)}^{\text{generative }}{p(z=k)}}{\sum_{k^{\prime}=1}^{K}p\left(x\mid
z=k^{\prime}\right)p\left(z=k^{\prime}\right)}$ (11)

##  7 Tricks for Memory-Based Methods in the OCI Setting

Method | e(n, o) | e(n, n) | e(o, o) | e(o, n) | er(n, o) | er(o, n)
---|---|---|---|---|---|---
A-GEM | 0 | 177 | 0 | 9500 | 0% | 100%
ER | 37 | 148 | 2269 | 5852 | 20% | 72%
MIR | 54 | 113 | 2770 | 5330 | 32% | 66%
Table 5: Error analysis of CIFAR-100 by the end of training with M=5k. e(n, o)
& e(n, n) represent the number of test samples from new classes that are
misclassified as old classes and new classes, respectively. Same notation rule
is applied to e(o, o) & e(o, n).

(a) er(n, o) is the ratio of new class test samples misclassified as old
classes to the total number of new class test samples.

(b) The mean of logits for new and old classes

(c) The mean of the bias terms in the last FC layer for new and old classes

(d) The mean of the weights in the last FC layer for new and old classes

Figure 1: Error analysis for three memory-based methods(A-GEM, ER and MIR)
with 5k memory buffer on Split CIFAR-100 described in Section 8.1.

In class incremental learning, old class samples are generally not available
while training on new class samples. Although keeping a portion of old class
samples in a memory buffer has been proven effective [34, 8], the class
imbalance is still a serious problem given a limited buffer size. Moreover,
multiple recent works have revealed that class imbalance is one of the most
crucial causes of catastrophic forgetting [41, 42, 69]. To alleviate the class
imbalance, many simple but effective tricks have been proposed as the building
blocks for CL methods by modifying the loss function, post-processing, or
using different types of classifiers.

In Section 7.1 and 7.2, we perform quantitative error analysis and disclose
how class imbalance results in catastrophic forgetting. Section 7.3 explains
the compared tricks in detail.

###  7.1 Error analysis of memory-based methods

We perform quantitative error analysis for three memory-based methods with 5k
memory buffer (A-GEM, ER and MIR) on Split CIFAR-100 described in Section 8.1.
We define $e(n,o)$ and $e(n,n)$ as the number of test samples from new classes
that are misclassified as old classes and new classes, respectively. The same
notation rule is applied to $e(o,o)$ and $e(o,n)$. Also, $er(n,o)$ denotes the
ratio of new class test samples misclassified as old classes to the total
number of new class test samples. $er(o,n)$ is similarly defined.

As shown in Table 5, all methods have strong bias towards new classes by the
end of the training: A-GEM classifies all old class samples as new classes; ER
and MIR misclassify 72% and 66% old class samples as new classes,
respectively. Moreover, as we can see in Fig. 1(a), $er(o,n)$ is higher than
$er(n,o)$ most of the times along the training process for all three methods.
These phenomena are not specific to the OCI setting, and [93, 68] also found
similar results in the offline class incremental learning. Additionally, we
easily find that ER and MIR are always better than A-GEM in terms of
$er(o,n)$. This is because ER and MIR use the memory samples more directly,
namely replaying them with the new class samples. The indirect use of memory
samples in A-GEM is less effective in the class incremental setting.

###  7.2 Biased FC layer

To better understand how class imbalance affects the learning performance, we
define the following notations. The convolutional neural network (CNN)
classification model can be split into a feature extractor
$\phi(\cdot)\mathrel{\mathop{\mathchar
58\relax}}\mathbf{x}\mapsto\mathbb{R}^{d}$ where d is the dimension of the
feature vector of image $\mathbf{x}$, and a fully-connected (FC) layer with
Softmax output. The output of the FC layer is obtained with:

$\displaystyle logits(\mathbf{x})=\mathbf{W}^{T}\phi(\mathbf{x})$ (12)
$\displaystyle\mathbf{o}(\mathbf{x})=Softmax(logits(\mathbf{x})))$ (13)

where $\mathbf{W}\in\mathbb{R}^{d\times({\mathinner{\\!\left\lvert
C_{old}\right\rvert}+\mathinner{\\!\left\lvert C_{new}\right\rvert}}})$, and
$C_{old}$ and $C_{new}$ are the sets of old and new classes respectively, with
$|\cdot|$ denoting the number of classes in each set. $\mathbf{W_{i}}$
represents the weight vector for class i. For notational brevity, $\mathbf{W}$
also contains the bias terms.

We start by analyzing the mean of logits for new and old classes. As shown in
Fig. 1(b), the mean of logits for new classes is always much higher than that
for old classes, which explains the high $er(o,n)$ for all three methods. As
we can see from Eq. (12), both feature extractor $\phi(\mathbf{x})$ and FC
layer $\mathbf{W}$ may potentially contribute to the logit bias. However,
previous works have found that even a small memory buffer (implying high class
imbalance) can greatly alleviate catastrophic forgetting in the multi-head
setting where the model can utilize the task-id to select the corresponding FC
layer for each task [34, 31]. This suggests that the feature extractor is not
heavily affected by the class imbalance, and therefore we hypothesize that the
FC layer is biased.

To validate the hypothesis, we plot the means of bias terms and weights in the
FC layer for new and old classes in Fig. 1(c) and 1(d). As we can see, the
means of weights for the new classes are much higher than those for the old
classes, and the means of bias terms for the new classes are also higher than
those for the old classes most of the times. Since the biased weights and bias
terms in $\mathbf{W}$ have direct impacts on the output logits, the model has
a higher chance of predicting a sample as new classes. [93, 42, 68] have also
verified the same hypothesis but with different validation methods.

A recent work reveals how does class imbalance result in a biased $\mathbf{W}$
[93]. We denote $s_{i}=\mathbf{W_{i}}^{T}\phi(\mathbf{x})$ as the logit of
sample $\mathbf{x}$ for class $i$. The gradient of the cross-entropy loss
$L_{CE}$ with respect to $\mathbf{W_{i}}$ for the Softmax output is:

$\displaystyle\frac{\partial\mathcal{L}_{\mathrm{CE}}}{\partial\mathbf{W_{i}}}=\bigg{(}\frac{e^{s_{i}}}{\sum_{j\in
C_{old}+C_{new}}e^{s_{j}}}-\mathbbm{1}_{\\{i=y\\}}\bigg{)}\phi(\mathbf{x})$
(14)

where $y$ is the ground-truth class and $\mathbbm{1}_{\\{i=y\\}}$ is the
indicator for $i=y$. Since ReLU is often used as the activation function for
the embedding networks [94], $\phi(\mathbf{x})$ is always positive. Therefore,
the gradient is always positive for $i\neq y$. If $i$ belongs to old classes,
$i\neq y$ will hold most of the time as the new class samples significantly
outnumber the old class samples during training. Thus, the logit for class $i$
will keep being penalized during the gradient descent. As a result, the logits
for the old classes are prone to be smaller than those for the new classes,
and the model is consequently biased towards new classes.

Other than the bias related to the Softmax classifier mentioned above, another
problem induced by class imbalance is under-representation of the minority
[95, 96], where minority classes do not show a discernible pattern in the
latent feature space [63]. The under-representation introduces additional
difficulty for other classifiers apart from the Softmax classifier, such as
nearest-class-mean [87] and cosine-similarity-based classifier [97].

###  7.3 Compared tricks

To mitigate the strong bias towards new classes due to class imbalance,
multiple methods have been proposed lately [98, 99, 68, 70, 100]. For example,
LUCIR [69] proposes three tricks: cosine normalization to balance class
magnitudes, a margin ranking loss for inter-class separation, and a less-
forgetting constraint to preserve the orientation of features. BiC [42] trains
an additional linear layer to remove bias with a separate validation set. We
compare seven simple (effortless integration without additional resources) but
effective (decent improvement) tricks in this work.

#### Labels Trick (LB)

[101] proposes to consider only the outputs for the classes in the current
mini-batch when calculating the cross-entropy loss, in contrast to the common
practice of considering outputs for all the classes. To achieve this, the
outputs that do not correspond to the classes of the current mini-batch are
masked out when calculating the loss.

Although the author did not demonstrate the motivation of this trick, we can
easily find the rationale based on the analysis in Section 7.2. Masking out
all the outputs that don’t match the classes in the current mini-batch is
equivalent to changing the loss function to:

$\displaystyle\mathcal{L}_{\mathrm{CE}}(x_{i},y_{i})=-\log\left(\frac{e^{s_{y_{i}}}}{\sum_{j\in{C}_{cur}}e^{s_{j}}}\right)$
(15)

where $C_{cur}$ denotes the classes in the current mini-batch. We can see that
$\frac{\partial\mathcal{L}_{\mathrm{CE}}}{\partial s_{j}}=0$ for $j\notin
C_{cur}$, and therefore training with the current mini-batch will not overly
penalize the logits for classes that are not in the mini-batch.

#### Knowledge Distillation with Classification Loss (KDC)

KD [40] is an effective way for knowledge transfer between networks. Multiple
recent works have proposed different ways to combine the KD loss with the
classification loss [8, 41, 102]. In this part, we compare the methods from
[42]. Specifically, the loss function is given as:

$\displaystyle\mathcal{L}(\mathbf{x},y)=\lambda\mathcal{L}_{CE}(\mathbf{x},y)+(1-\lambda)\mathcal{L}_{KD}(\mathbf{x})$
(16)

where $\lambda$ is set to $\frac{\mathinner{\\!\left\lvert
C_{new}\right\rvert}}{\mathinner{\\!\left\lvert
C_{old}\right\rvert}+\mathinner{\\!\left\lvert C_{new}\right\rvert}}$. Note
that $(\mathbf{x},y)$ is from both new class data from the current task and
old class data from the memory buffer.

As shown in Table 9, however, this method does not perform well in our
experiment setting, especially with a large memory buffer. We identify
$\lambda$ as the key issue. We find that the accuracy for new class samples
becomes almost zero around the end of training because $\lambda$ is very
small. In other words, $\mathcal{L}_{KD}$ dominates the loss, and the model
cannot learn any new knowledge. Hence, we suggest setting $\lambda$ to
$\sqrt{\frac{\mathinner{\\!\left\lvert
C_{new}\right\rvert}}{\mathinner{\\!\left\lvert
C_{old}\right\rvert}+\mathinner{\\!\left\lvert C_{new}\right\rvert}}}$. We
denote the trick with this modification as KDC*.

#### Multiple Iterations (MI)

Most of the previous works only perform a single gradient update on the
incoming mini-batch in the online setup. [17] suggests performing multiple
gradient updates to maximally utilize the current mini-batch. Particularly for
replay methods, additional updates with different replay samples can improve
performance. We run 5 iterations per incoming mini-batch and retrieve
different memory samples for each iteration in this work.

#### Nearest Class Mean (NCM) Classifier

To tackle the biased FC layer, one can replace the FC layer and Softmax
classifier with another type of classifier. Nearest Class Mean classifier
(NCM) [87] is a popular option in CL [8, 78]. To make prediction for a sample
$\mathbf{x}$, NCM computes a prototype vector for each class and assigns the
class label with the most similar prototype:

$\displaystyle\mu_{y}=\frac{1}{\left|M_{y}\right|}\sum_{\mathbf{x}_{m}\in
M_{y}}\phi(\mathbf{x}_{m})$ (17) $\displaystyle
y^{*}=\underset{y=1,\ldots,t}{\operatorname{argmin}}\left\|\phi(\mathbf{x})-\mu_{y}\right\|$
(18)

In the class incremental setting, the true prototype vector for each class
cannot be computed due to the unavailability of the training data for previous
tasks. Instead, the prototype vectors can be approximated using the data in
the memory buffer. In Eq. (17), $M_{y}$ denotes the memory samples of class
$y$.

#### Separated Softmax (SS)

Since training the whole FC layer with one Softmax output layer results in
bias as explained in Section 7.2, SS [93] employs two Softmax output layers:
one for new classes and another one for old classes. The loss function can be
calculated as below:

$\displaystyle\begin{aligned} &\mathcal{L}(\mathbf{x}_{i},y_{i})\\\
=&-\log\left(\frac{e^{s_{y_{i}}}}{\sum_{j\in{C}_{old}}e^{s_{j}}}\right)\cdot\mathbbm{1}\left\\{y_{i}\in{C}_{old}\right\\}-\log\left(\frac{e^{s_{y_{i}}}}{\sum_{j\in{C}_{new}}e^{s_{j}}}\right)\cdot\mathbbm{1}\left\\{y_{i}\in{C}_{new}\right\\}\end{aligned}$
(19)

Depending on whether $y_{i}\in C_{new}$ or $C_{old}$, the corresponding
Softmax is used to compute the cross-entropy loss. We can find that
$\frac{\partial\mathcal{L}}{\partial s_{j}}=0$ for $j\in C_{old}$ and
$y_{i}\in C_{new}$. Thus, training with new class samples will not overly
penalize the logits for the old classes.

#### Review Trick (RV)

To alleviate the class imbalance, [41] proposes an additional fine-tuning step
with a small learning rate, which uses a balanced subset from the memory
buffer and the training set of the current task. A temporary distillation loss
for new classes is applied to avoid forgetting the new classes during the
fine-tuning phase mentioned above. A similar yet simplified version, dubbed
Review Trick, is applied in the winning solution in the continual learning
challenge at CVPR2020 [25]. At the end of learning the current task, the
review trick fine-tunes the model with all the samples in the memory buffer
using only the cross-entropy loss. In this work, we compare the review trick
from [25] with a learning rate 10 times smaller than the training learning
rate.

Dataset | #Task | #Train/task | #Test/task | #Class | Image Size |  Setting
---|---|---|---|---|---|---
Split MiniImageNet | 20 | 2500 | 500 | 100 | 3x84x84 | OCI
Split CIFAR-100 | 20 | 2500 | 500 | 100 | 3x32x32 | OCI
CORe50-NC | 9 |  12000$\sim$24000 |  4500$\sim$9000 | 50 | 3x128x128 | OCI
NS-MiniImageNet | 10 | 5000 | 1000 | 100 | 3x84x84 | ODI
CORe50-NI222CORe50-NI uses one test set(44972 images) for all tasks | 8 | 15000 | 44972 | 50 | 3x128x128 | ODI
Table 6: Summary of dataset statistics

##  8 Experiments

Section 8.1 explains the general setting for all experiments. Then, we focus
on the OCI setting in Section 8.2 and 8.3: we evaluate all the compared
methods and baselines in Section 8.2 and investigate the effectiveness of the
seven tricks in Section 8.3. In Section 8.4, we assess the compared methods in
the ODI setting to investigate their abilities to generalize, and Section 8.5
provides general comments on the surveyed methods and tricks.

###  8.1 Experimental setup

#### Datasets

We evaluate the nine methods summarized in Section 6 and additional two
baselines on three class incremental datasets. We also propose a new domain
incremental dataset based on Mini-ImageNet and examine the compared methods in
the ODI setting to see how well the methods can generalize to this setting.
The summary of dataset statistics is provided in Table 6.

#### Class Incremental Datasets

  * 1.

Split CIFAR-100 is constructed by splitting the CIFAR-100 dataset [103] into
20 tasks with disjoint classes, and each task has 5 classes. There are 2,500
3×32×32 images for training and 500 images for testing in each task.

  * 2.

Split MiniImageNet splits MiniImageNet dataset [104], a subset of ImageNet
[105] with 100 classes, into 20 disjoint tasks as in [34]. Each task contains
5 classes, and every class consists of 500 3×84×84 images for training and 100
images for testing.

  * 3.

CORe50-NC [106] is a benchmark designed for class incremental learning with 9
tasks and 50 classes: 10 classes in the first task and 5 classes in the
subsequent 8 tasks. Each class has around 2,398 3×128×128 training images and
900 testing images.

#### Domain Incremental Datasets

  * 1.

NonStationary-MiniImageNet (NS-MiniImageNet) The most popular domain
incremental datasets are still based on MNIST [107], such as Rotation MNIST
[13] and Permutation MNIST [11]. To evaluate the domain incremental setting in
a more practical scenario, we propose NS-MiniImageNet with three nonstationary
types: noise, blur and occlusion. The number of tasks and the strength of each
nonstationary type are adjustable in this dataset. In this survey, we use 10
tasks for each type, and each task comprises 5,000 3×84×84 training images and
1,000 testing images. As shown in Table 7, the nonstationary strength
increases over time, and to ensure a smooth distribution shift, the strength
always increases by the same constant. More details about the nonstationary
strengths used in the experiments can be found in B.

  * 2.

CORe50-NI [106] is a practical benchmark designed for assessing the domain
incremental learning with 8 tasks, where each task has around 15,000 training
images of 50 classes with different types of nonstationarity including
illumination, background, occlusion, pose and scale. There is one single test
set for all tasks, which contains 44,972 images.

Type | Task 1 | … | Task 5 | … | Task 10
---|---|---|---|---|---
Noise |  | … |  | … |
Blur |  | … |  | … |
Occlusion |  | … |  | … |
Table 7: Example images of different nonstationary types in NS-MiniImageNet.
The nonstationary strength increases over time, and to ensure a smooth
distribution shift, the strength always increases by the same constant.

#### Task Order and Task Composition

Since the task order and task composition may impact the performance [106], we
take the average over multiple runs for each experiment with different task
orders and composition to reliably assess the robustness of the methods. For
CORe50-NC and CORe50-NI, we follow the number of runs (i.e., 10), task order
and composition provided by the authors. For Split CIFAR-100 and Split
MiniImagenet, we average over 15 runs, and the class composition in each task
is randomly selected for each run.

#### Models

Similar to [34, 13], we use the reduced ResNet18 [108] as the base model for
all datasets and methods. The network is trained via the cross-entropy loss
with a stochastic gradient descent optimizer and a mini-batch size of 10. The
size of the mini-batch retrieved from the memory buffer is also set to 10,
irrespective of the size of the memory buffer as in [34]. Note that with
techniques such as transfer learning (e.g., using a pre-trained model from
ImageNet), data augmentation and deeper network architectures, it is possible
to achieve much higher performance in this setting [109]. However, since those
techniques are orthogonal to our investigation and deviate from the simpler
experimental settings of other papers we cite and compare, we do not use them
in our experiments.

#### Baselines

We compare the methods we discussed in Section 6 with two baselines:

  * 1.

Finetune greedily updates the model with the incoming mini-batch without
considering the previous task performance. The model suffers from catastrophic
forgetting and is regarded as the lower-bound.

  * 2.

Offline trains a model using all the samples in a dataset in an offline
manner. The baseline is trained for multiple epochs within each of which mini-
batches are sampled i.i.d from differently shuffled dataset. We train the
model for 70 epochs with the mini-batch size of 128.

#### Other Details

We evaluate the performance with 5 metrics we described in Section 3.2:
Average Accuracy, Average Forgetting, Forward Transfer, Backward Transfer and
Run Time. We use the hyperparameter tuning protocol described in Section 4 and
give each method similar tuning budget. The details of the implementation
including hyperparameter selection can be found in B.2.

###  8.2 Performance comparison in Online Class Incremental (OCI) setting

Method | Split CIFAR-100 | Split Mini-ImageNet | CORe50-NC
---|---|---|---
Finetune | $3.7\pm 0.3$ | $3.4\pm 0.2$ | $7.7\pm 1.0$
Offline | $49.7\pm 2.6$ | $51.9\pm 0.5$ | $51.7\pm 1.8$
EWC++ | $3.7\pm 0.4$ | $3.5\pm 0.4$ | $8.3\pm 0.3$
LwF | $7.2\pm 0.4$ | $7.6\pm 0.7$ | $7.1\pm 1.9$
Buffer Size | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k
ER | $7.6\pm 0.5$ | $17.0\pm 1.9$ | $18.4\pm 1.4$ | $6.4\pm 0.9$ | $14.5\pm 2.1$ | $15.9\pm 2.0$ | $23.5\pm 2.4$ | $27.5\pm 3.5$ | $28.2\pm 3.3$
MIR | $7.6\pm 0.5$ | $18.2\pm 0.8$ | $19.3\pm 0.7$ | $6.4\pm 0.9$ | $16.5\pm 2.1$ | $21.0\pm 1.1$ | $\mathbf{27.0\pm 1.6}$ | $\mathbf{32.9\pm 1.7}$ | $\mathbf{34.5\pm 1.5}$
GSS | $7.7\pm 0.5$ | $11.3\pm 0.9$ | $13.4\pm 0.6$ | $5.9\pm 0.7$ | $11.2\pm 0.9$ | $13.5\pm 0.8$ | $19.6\pm 3.0$ | $22.2\pm 4.4$ | $21.1\pm 3.5$
iCaRL | $\mathbf{16.7\pm 0.8}$ | $19.2\pm 1.1$ | $18.8\pm 0.9$ | $\mathbf{14.7\pm 0.4}$ | $17.5\pm 0.6$ | $17.4\pm 1.5$ | $22.1\pm 1.4$ | $25.1\pm 1.6$ | $22.9\pm 3.1$
A-GEM | $3.7\pm 0.4$ | $3.6\pm 0.2$ | $3.8\pm 0.2$ | $3.4\pm 0.2$ | $3.7\pm 0.3$ | $3.3\pm 0.3$ | $8.7\pm 0.6$ | $9.0\pm 0.5$ | $8.9\pm 0.6$
CN-DPM | $14.0\pm 1.7$ | - | - | $9.4\pm 1.2$ | - | - | $7.6\pm 0.4$ | - | -
GDumb | $10.4\pm 1.1$ | $\mathbf{22.1\pm 0.9}$ | $\mathbf{28.8\pm 0.9}$ | $8.8\pm 0.4$ | $\mathbf{21.1\pm 1.7}$ | $\mathbf{31.0\pm 1.4}$ | $15.1\pm 1.2$ | $28.1\pm 1.4$ | $32.6\pm 1.7$

Table 8: Average accuracy (end of training) for the OCI setting of Split
CIFAR-100, Split Mini-ImageNet and CORe50-NC. Replay-based methods and a
strong baseline GDumb show competitive performance across three datasets.

#### Regularization-based Methods

As shown in Table 8, EWC++ has almost the same performance as Finetune in the
OCI setting. We find that the gradient explosion of the regularization terms
is the root cause. Specifically, $\lambda$ in EWC++ controls the
regularization strength, and we need a larger $\lambda$ to avoid forgetting.
However, when $\lambda$ increases to a certain value, the gradient explosion
occurs. If we take $\theta_{j}$ in Eq. (7) as an example, the regularization
term for $\theta_{j}$ has the gradient
${\lambda}F_{j}(\theta_{j}-\theta^{*})$. Some model weights change
significantly when it receives data with new classes, and therefore the
gradients for those weights are prone to explode with a large $\lambda$. The
Huber regularization proposed lately could be a possible remedy [110].
Surprisingly, we also observe that LwF, a method relying on KD, has similar
performance as replay-based methods with a small memory buffer(1k) such as ER,
MIR and GSS in Split CIFAR-100 and even outperforms them in Split Mini-
ImageNet. In the larger and more realistic CORe50-NC, however, both EWC++ and
LwF fail. This also confirms the results of three recent studies, where [111]
shows the shortcomings of regularization-based approaches in the class
incremental setting, [112] theoretically explains why regularization-based
methods underperform memory-based methods and [113] empirically demonstrates
that KD is more useful in small-scale datasets.

(a) CIFAR-100

(b) Mini-ImageNet

(c) CORe50-NC

Figure 2: The average accuracy measured by the end of each task for the OCI
setting with a 5k memory buffer. More detailed results for different memory
buffer sizes are shown in C.1.

#### Memory-based Methods

Firstly, A-GEM does not work in this setting as it has almost the same
performance as Finetune, implying that the indirect use of the memory samples
is less efficient than the direct relay in the OCI setting. Secondly, given a
small memory buffer in Split CIFAR-100 and Mini-ImageNet, iCaRL—proposed in
2017—shows the best performance. On the other hand, other replay-based methods
such as ER, MIR and GSS do not work well because simply replaying with a small
memory buffer yields severe class imbalance. When equipped with a larger
memory buffer (5k and 10k), GDumb—a simple baseline that trains with the
memory buffer only—outperforms other methods by a large margin. Additionally,
as shown in Fig. 2(a) and Fig. 2(b), GDumb dominates the average accuracy not
only at the end of training but also at any other evaluation points along the
data stream. This raises concerns about the progress in the OCI setting in the
literature. Next, in the larger CORe50-NC dataset, GDumb is less effective
since it only relies on the memory and the memory is smaller in a larger
dataset in proportion. MIR is a robust and strong method as it exhibits
remarkable performance across different memory sizes. Also, even though GSS is
claimed to be an enhanced version of ER, ER consistently surpasses GSS across
different memory sizes and datasets, which is also confirmed by other studies
[19, 59].

#### Parameter-isolation-based methods

As one of the first dynamic-architecture methods without using a task-ID, CN-
DPM shows competitive results in Split CIFAR-100 and Mini-ImageNet but fails
in CORe5-NC. The key reason is that CN-DPM is very sensitive to
hyperparameters, and when applying CN-DPM in a new dataset, a good performance
cannot be guaranteed given a limited tuning budget.

Figure 3: Average Accuracy, Forgetting, Running Time, Forward Transfer and
Backward Transfer for the OCI setting with a 5k memory buffer. Each column
represents a metric and each row represents a dataset. In this setting, none
of the methods show any forward or backward transfer.

#### Other Metrics

We show the performance of all five metrics in Fig. 3. Generally speaking, we
find that a high average accuracy comes with low forgetting, but methods using
KD such as iCaRL and LwF, and dynamic-architecture methods such as CN-DPM have
lower forgetting. The reason for the low forgetting of these methods is
intransigence, the model’s inability to learn new knowledge [9]. For iCaRL and
LwF, KD imposes a strong regularization, which may lead to a lower accuracy on
new tasks. For CN-DPM, the inaccurate expert selector is the cause of the
intransigence [19]. Furthermore, most methods have similar running time except
for CN-DPM, GDumb and GSS. CN-DPM requires a significantly longer training
time as it needs to train multiple experts, and each expert contains a
generative model (VAE [92]) and a classifier(10-layer ResNet [108]). GDumb has
the second longest running time as it requires training the model from scratch
with the memory at every evaluation point. Lastly, we notice none of the
methods show any forward and backward transfer, which is expected since a
model tends to classify all the test samples as the current task labels due to
the strong bias in the last FC layer.

###  8.3 Effectiveness of tricks in the OCI setting

Finetune | $3.7\pm 0.3$
---|---
Offline | $49.7\pm 2.6$
Trick | A-GEM | ER | MIR
Buffer Size | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k
N/A | $3.7\pm 0.4$ | $3.6\pm 0.2$ | $3.8\pm 0.2$ | $7.6\pm 0.5$ | $17.0\pm 1.9$ | $18.4\pm 1.4$ | $7.6\pm 0.5$ | $18.2\pm 0.8$ | $19.3\pm 0.7$
LB | $5.0\pm 0.5$ | $4.6\pm 0.8$ | $4.9\pm 0.7$ | $14.0\pm 2.0$ | $19.0\pm 2.6$ | $20.4\pm 1.2$ | $\mathbf{15.1\pm 0.6}$ | $21.1\pm 0.8$ | $22.5\pm 0.9$
KDC | $8.3\pm 0.7$ | $8.8\pm 0.7$ | $7.7\pm 1.1$ | $11.7\pm 0.8$ | $10.9\pm 1.9$ | $11.9\pm 2.2$ | $12.0\pm 0.5$ | $12.3\pm 0.7$ | $11.8\pm 0.6$
KDC* | $5.6\pm 0.5$ | $5.8\pm 0.5$ | $5.8\pm 0.5$ | $12.6\pm 0.5$ | $21.2\pm 1.1$ | $24.2\pm 1.9$ | $12.4\pm 0.5$ | $20.7\pm 0.8$ | $23.2\pm 1.2$
MI | $4.0\pm 0.3$ | $4.0\pm 0.3$ | $4.0\pm 0.2$ | $8.6\pm 0.5$ | $19.7\pm 0.9$ | $26.4\pm 1.2$ | $8.5\pm 0.4$ | $17.7\pm 1.0$ | $25.9\pm 1.2$
SS | $5.0\pm 0.8$ | $5.2\pm 0.6$ | $5.1\pm 0.5$ | $12.3\pm 2.1$ | $20.9\pm 1.0$ | $23.1\pm 1.2$ | $14.0\pm 0.5$ | $21.6\pm 0.7$ | $24.5\pm 0.7$
NCM | $\mathbf{9.5\pm 0.9}$ | $11.7\pm 0.6$ | $11.5\pm 0.7$ | $\mathbf{14.6\pm 0.7}$ | $\mathbf{27.6\pm 1.0}$ | $31.0\pm 1.0$ | $13.7\pm 0.5$ | $27.0\pm 0.5$ | $30.0\pm 0.6$
RV | $4.5\pm 0.4$ | $\mathbf{22.5\pm 1.3}$ | $\mathbf{30.7\pm 1.2}$ | $12.0\pm 0.8$ | $26.9\pm 2.8$ | $\mathbf{32.0\pm 5.3}$ | $9.7\pm 0.5$ | $\mathbf{28.1\pm 0.6}$ | $\mathbf{35.2\pm 0.5}$
Best OCI | $16.7\pm 0.8$ | ${22.1\pm 0.9}$ | ${28.8\pm 0.9}$ | $16.7\pm 0.8$ | ${22.1\pm 0.9}$ | ${28.8\pm 0.9}$ | $16.7\pm 0.8$ | ${22.1\pm 0.9}$ | ${28.8\pm 0.9}$

Table 9: Performance of compared tricks for the OCI setting on Split CIFAR-100. We report average accuracy (end of training) for memory buffer with size 1k, 5k and 10k. Best OCI refers to the best performance achieved by the compared methods in Table 8.  Figure 4: Comparison of various tricks for the OCI setting on Split CIFAR-100. We report average accuracy (end of training) for memory buffer with size 1k, 5k and 10k. N/A denotes the performance of the base methods (A-GEM, ER, MIR) without any trick. Trick | Running Time(s)
---|---
| M=1k | M=5k | M=10k
N/A | 83 | 82 | 84
LB | 87 | 88 | 89
KDC | 105 | 106 | 106
KDC* | 105 | 105 | 107
MI | 328 | 324 | 325
SS | 89 | 90 | 90
NCM | 126 | 282 | 450
RV | 98 | 159 | 230
Table 10: Running time of different tricks applying to ER with different
memory sizes on Split CIFAR-100.

We evaluate Label trick (LB), Knowledge Distillation and Classification (KDC),
Multiple Iterations (MI), Separated Softmax(SS), Nearest Class Mean (NCM)
classifier, Review trick(RV) on three memory-based methods: A-GEM, ER and MIR.
The results are shown in Table 9 and Fig. 4.

Firstly, although all tricks enhance the basic A-GEM, only RV can bring A-GEM
closer to ER and MIR, which reiterates that the direct replay of the memory
samples is more effective than the gradient projection approach in A-GEM.

For replay-based ER and MIR, LB and NCM are the most effective when M=1k and
can improve the accuracy by around 100% (7.6% $\rightarrow$ 14.5% on average).
KDC, KDC* and SS have similar performance improvement effect and can boost the
accuracy by around 64% (7.6% $\rightarrow$ 12.4% on average). With a larger
memory size, NCM remains very effective, and RV becomes much more helpful.
When M=10k, RV boosts ER’s performance by 74% (18.4% $\rightarrow$ 32.0%) and
improve MIR’s performance by 82% (from 19.3% $\rightarrow$ 35.2%). Also, KDC
fails with a larger memory due to over-regularization of the KD term, and the
modified KDC* has much better performance. Compared with other tricks, MI and
RV are more sensitive to the memory size since these tricks highly depend on
the memory. Note that when equipped with NCM or RV, both ER and MIR can
outperform the best performance achieved by the compared methods (Table 8)
when M=5k or 10k.

As shown in Table 10, the running times of LB, KDC, KDC*, MI and SS do not
depend on the memory size and have a limited increase compared to the
baseline. Since NCM needs to calculate the means of all the classes in the
memory and RV requires additional training of the whole memory before
evaluation, their running times increase as the memory size grows.

To sum up, all of the tricks are beneficial to the base methods (A-GEM, ER,
MIR); NCM is a useful and robust trick across all memory sizes, while LB and
RV are more advantageous in smaller and larger memory, respectively. The
running times of NCM and RV go up with the increase in the memory size, and
other tricks only add a fixed overhead to the running time. We also get
similar results in Split Mini-ImageNet, as shown in C.

###  8.4 Performance comparison in Online Domain Incremental (ODI) setting

Method | Mini-ImageNet-Noise | Mini-ImageNet-Occlusion | Mini-ImageNet-Blur | CORe50-NI
---|---|---|---|---
Finetune | $11.1\pm 1.0$ | $13.8\pm 1.6$ | $2.4\pm 0.2$ | $14.0\pm 2.8$
Offline | $37.3\pm 0.8$ | $38.6\pm 4.7$ | $11.9\pm 1.0$ | $51.7\pm 1.8$
EWC | $12.5\pm 0.8$ | $14.8\pm 1.1$ | $2.6\pm 0.2$ | $11.6\pm 1.5$
LwF | $9.2\pm 0.9$ | $12.8\pm 0.8$ | $3.4\pm 0.4$ | $11.1\pm 1.1$
Buffer Size | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k
ER | $\mathbf{19.4\pm 1.3}$ | $21.6\pm 1.1$ | $24.3\pm 1.2$ | $\mathbf{19.2\pm 1.5}$ | $\mathbf{23.4\pm 1.4}$ | $23.7\pm 1.1$ | $5.3\pm 0.6$ | $\mathbf{8.6\pm 0.8}$ | $9.4\pm 0.7$ | $24.1\pm 4.2$ | $28.3\pm 3.5$ | $30.0\pm 2.8$
MIR | $18.1\pm 1.1$ | $\mathbf{22.5\pm 1.4}$ | $\mathbf{24.4\pm 0.9}$ | $17.6\pm 0.7$ | $22.0\pm 1.1$ | $\mathbf{23.8\pm 1.2}$ | $\mathbf{5.5\pm 0.5}$ | $8.1\pm 0.6$ | $9.6\pm 1.0$ | $\mathbf{26.5\pm 1.0}$ | $\mathbf{34.0\pm 1.0}$ | $\mathbf{33.3\pm 1.7}$
GSS | $18.9\pm 0.8$ | $21.4\pm 0.9$ | $23.2\pm 1.1$ | $17.7\pm 0.8$ | $21.0\pm 2.2$ | $23.2\pm 1.4$ | $5.2\pm 0.5$ | $7.6\pm 0.6$ | $8.0\pm 0.6$ | $25.5\pm 2.1$ | $27.2\pm 2.0$ | $25.3\pm 2.1$
A-GEM | $14.0\pm 1.3$ | $14.6\pm 0.7$ | $14.2\pm 1.4$ | $16.4\pm 0.7$ | $13.9\pm 2.6$ | $14.4\pm 2.0$ | $4.4\pm 0.4$ | $4.4\pm 0.4$ | $4.3\pm 0.5$ | $12.4\pm 1.1$ | $13.8\pm 1.2$ | $15.0\pm 2.2$
CN-DPM | $4.6\pm 0.5$ | - | - | $3.9\pm 0.8$ | - | - | $2.2\pm 0.2$ | - | - | $9.6\pm 3.9$ | - | -
GDumb | $5.4\pm 1.0$ | $12.5\pm 0.7$ | $15.2\pm 0.5$ | $5.4\pm 0.4$ | $14.2\pm 0.6$ | $20.2\pm 0.4$ | $3.3\pm 0.2$ | $7.5\pm 0.2$ | $\mathbf{10.0\pm 0.2}$ | $9.6\pm 1.5$ | $11.2\pm 2.0$ | $11.5\pm 1.7$

Table 11: The Average Accuracy (end of training) for the ODI setting of Mini-
ImageNet with three nonstationary types (Noise, Occlusion, Blur) and
CORe50-NI. Figure 5: Average Accuracy, Forgetting, Running Time, Forward
Transfer and Backward Transfer for the ODI setting with a 5k memory buffer.
Each column represents a metric and each row represents a dataset. Forward and
backward transfer are not applicable in CORe50 since it uses one test set for
all tasks.

Since most of the surveyed methods are only evaluated in the class incremental
setting in the original papers, we evaluate them in the ODI setting to
investigate their robustness and ability to generalize to other CL settings.
We assess the methods with CORe50-NI—a dataset designed for ODI—and the
proposed NS-MiniImageNet dataset consisting of three nonstationary types:
noise, occlusion, and blur (see Table 7). The average accuracy at the end of
training is summarized in Table 11.

Generally speaking, all replay-based methods (ER, MIR, GSS) show comparable
performance across three memory sizes and outperform all other methods. GDumb,
the strong baseline that dominates the OCI setting in most cases, is no longer
as competitive as the replay-based methods and fails completely in CORe50-NI.
One of the reasons is that class imbalance, the key cause of forgetting in the
OCI setting, does not exist in ODI since the class labels are the same for all
tasks. Moreover, in the ODI setting, samples in the data stream change
gradually and smoothly with different nonstationary strengths (NS-
MiniIMageNet) or nonstationary types (CORe50-NI). Learning new samples
sequentially with the replay samples(replay-based) may be more effective for
the model to adapt to the gradually changing nonstationarity than learning
only the samples in the buffer (GDumb). Additionally, the greedy memory update
strategy (see Algorithm 5 in Appendix) in GDumb is not suitable for the ODI
setting as the buffer will comprise mostly of samples in the latest tasks due
to the greedy update, and GDumb will have very limited access to samples in
the earlier tasks. Using reservoir sampling as the update strategy may
alleviate this shortcoming since reservoir sampling ensures every data point
has the same probability to be stored in the memory.

CN-DPM has terrible performance in this setting because it is very sensitive
to hyperparameters, and the method cannot find the hyperparameter set that
works in this setting within the same tuning budget as other methods.

Regarding methods without a memory buffer, the KD-based LwF underperforms EWC
and Finetune, implying KD may not be useful in the ODI setting.

Another interesting observation is that all methods, including Offline, show
unsatisfactory results in the blur scenario. The pivotal cause may be due to
the backbone model we use (ResNet18 [108]) since a recent study points out
that Gaussian blur can easily degrade the performance of ResNet [114].

In terms of other metrics, as shown in Fig. 5, we find that all methods show
positive forward transfer, and replay-based methods show much better backward
transfer than others. The first reason is that tasks in the ODI setting share
more cross-task resemblances than the OCI setting; secondly, the bias towards
the current task due to class imbalance does not happen since new tasks
contain the same class labels as old tasks. Thus, the model is able to perform
zero-shot learning (forward transfer) and improve the preceding tasks
(backward transfer).

In summary, replay-based methods exhibit more robust and surpassing
performance in the ODI setting. Considering the running time and the
performance in the larger scale dataset, MIR stands out as a versatile and
competitive method in this setting.

###  8.5 Overall comments for methods and tricks

We summarize the key findings and give comments on each method and trick based
on our findings in Table 12 and Table 13.

Method |  Comments
---|---
Regularization-based
EWC++ |  1. Ineffective in both OCI and ODI settings 2. Suffers from gradient explosion
LwF |  1. Effective on small scale datasets in OCI (achieves similar performance as replay-based methods with small memory) 2. Ineffective in ODI setting
Memory-based
ER |  1. Efficient training time over other memory-based methods 2. Better than GSS in most cases but worse than MIR, especially with a large memory buffer
MIR |  1. A versatile and competitive method in both OCI and ODI settings 2. Works better on a large scale dataset and a large memory buffer
GSS |  1. Inefficient training time 2. Worse than other memory-based methods in most cases
iCaRL |  1. Best performance (with large margins) with a small memory buffer on small scale datasets
A-GEM |  1. Ineffective in both OCI and ODI settings
GDumb |  1. Best performance with a large memory buffer on small scale datasets in OCI setting 2. Ineffective in ODI mostly due to its memory update strategy 3. Inefficient training time due to training from scratch at every inference point
Parameter-isolation-based
CN-DPM |  1. Effective when memory size is small 2. Sensitive to hyperparameters and when testing on a new dataset, it may not find a working hyperparameter set given the same tuning budget as others 3. Longest training time among compared methods
Table 12: Overall comments for compared methods Trick |  Comments
---|---
LB |  1. Effective when memory buffer is small 2. Fixed and limited training time overhead
KDC |  1. Fails because of over-regularization of knowledge distillation loss
KDC* |  1. Provides moderate improvement with fixed and acceptable training time overhead
MI |  1. Better improvement with a larger memory buffer 2. Training time increases with more iterations
SS |  1. Similar improvement as KDC* but with less training time overhead
NCM* |  1. Provides very strong improvement across different memory sizes. 2. Baselines equipped with it outperform state-of-the-art methods when the memory buffer is large 3. Inference time increases with the growth of the memory size
RV |  1. Presents very competitive improvement, especially with a larger memory buffer 2. Baselines equipped with it outperform state-of-the-art methods 3. Training time increases with the growth of the memory size but it is more efficient than NCM
Table 13: Overall comments for compared tricks

##  9 Trendy Directions in Online CL

In this section, we discuss some emerging directions in online CL that have
attracted interest and are expected to gain more attention in the future.

#### Raw-Data-Free Methods

In some applications, storing raw images is not feasible due to privacy and
security concerns, and this calls for CL methods that maintain reasonable
performance without storing raw data. Regularization [12, 11] is one of the
directions but [111] shows that this approach has theoretical limitations in
the class incremental setting and cannot be used alone to reach decent
performance. We also have empirically confirmed their claims in this work.
Generative replay [115, 47] is another direction but it is not viable for more
complex datasets as the current deep generative models still cannot generate
satisfactory images for such datasets [17, 50].

Feature replay is a promising direction where latent features of the old
samples at a given layer (feature extraction layer) are relayed instead of raw
data [65, 116, 117, 118]. Since the model changes along the training process,
to keep the latent features valid, [118] proposes to slow-down—in the limit
case, freeze—the learning of all the layers before the feature extraction
layer, while [65] proposes a feature adaptation method to map previous
features to their correct values as the model is updated. Another way is to
generate latent features with a deep generative model [117].

There are other lately proposed approaches which do not require storing the
raw data. For example, SDC [78] leverages embedding networks [119] and the
nearest class mean classifier [87]. The approach proposes a method to estimate
the drift of features during learning the current task and compensate for the
drift in the absence of previous samples. DMC [73] trains a separate model for
new tasks and combines the new and old models using publicly available
unlabeled data via a double distillation training objective. DSLDA [120]
freezes the feature extractor and uses deep Streaming Linear Discriminant
Analysis [121] to train the output layer incrementally.

With the increasing data privacy and security concerns, the raw-data-free
methods are expected to attract more research endeavour in the coming years.

#### Meta Learning

Meta-learning is an emerging learning paradigm where a neural network evolves
from multiple related learning episodes and generalizes the learned knowledge
to unseen tasks [122]. Since meta-learning builds up a potential framework to
advance CL, a lot of meta-learning based CL methods have been proposed
recently, and most of them support the online setting. MER [60] combines
experience replay with optimization based meta-learning to maximize transfer
and minimize interference based on future gradients. OML [123] is a meta-
objective that uses interference as a training signal to learn a
representation that accelerates future learning and avoid catastrophic
interference. More recently, iTAML [81] proposes to learn a task-agnostic
model that automatically predicts the task and quickly adapts to the predicted
task with meta-update. La-MAML [64] proposes an efficient gradient-based meta-
learning that incorporates per-parameter learning rates for online CL. MERLIN
[66] proposes an online CL method based on consolidation in a meta-space,
namely, the latent space that generates model weights for solving downstream
tasks. In [124], authors propose Continual-MAML, an online extension of MAML
[125], that can cope the new CL scenario they propose. We believe meta-
learning based online CL methods will continue to be popular with recent
advances in meta-learning.

#### CL in Other Areas

Although image classification and reinforcement learning are the main focuses
for most CL works, CL has drawn more and more attention in other areas. Object
detection has been another emerging topic in CL, and multiple works have been
proposed lately to tackle this problem. Most methods leverage KD [40] to
alleviate CF, and the main differences between the methods are the base object
detector and distillation parts in the network [126, 127, 128, 129]. More
recently, a meta-learning based approach is proposed to reshape model
gradients for better information share across incremental tasks [130]. A
replay-based method is introduced to address streaming object detection by
replaying compressed representation in a fixed memory buffer [131].

Beyond computer vision, CL with sequential data and recurrent neural network
(RNN) has gained attention over the past few years. Recent works have
confirmed that RNNs, including LSTMs, are also immensely affected by CF [132,
133, 134]. In [132], the authors unify GEM [13] and Net2Net [135] to tackle
forgetting in RNN. More recently, [136] shows that weight-importance based CL
in RNNs are limited and that the hypernetwork-based approaches are more
effective in alleviating forgetting. Meanwhile, [137] proposes a learning rule
to preserve network dynamics within subspaces for previous tasks and encourage
interfering dynamics to explore orthogonal subspaces when learning new tasks.
Moreover, multiple works are proposed to address general CL language learning
[138, 139] and specific language tasks, such as dialogue systems [140, 141,
142], image captioning [143], sentiment classification [144] and sentence
representation learning [145].

Recommender systems have also started to adopt CL [146, 147, 148, 149]. ADER
[148] is proposed to handle CF in session-based recommendation using the
adaptive distillation loss and replay with heading [88] technique. GraphSAIL
[149] is introduced for Graph Neural Networks based recommender systems to
preserve a user’s long-term preference during incremental model updates using
local structure distillation, global structure distillation and self-embedding
distillation.

Several works also address the deployment of CL in practice. [150] introduces
on-the-job learning, which requires a deployed model to discover new tasks,
collect training data continuously and incrementally learn new tasks without
interrupting the application. The author also uses chat-bots and self-driving
cars as the examples to highlight the necessity of on-the-job learning. [151]
presents a reference architecture for self-maintaining intelligent systems
that can adapt to shifting data distributions, cope with outliers, retrain
when necessary, and learn new tasks incrementally. [152] discusses the
clinical application of CL from three perspectives: diagnosis, prediction and
treatment decisions. [153] addresses a practical scenario where a high-
capacity server interacts with a large group of resource-limited edge devices
and proposes a Dual User-Adaptation framework which disentangles user-
adaptation into model personalization on the server and local data
regularization on the user device.

##  10 Conclusion

To better understand the relative advantages of recently proposed online CL
approaches and the settings where they work best, we performed extensive
experiments with nine methods and seven tricks in the online class incremental
(OCI) and online domain incremental (ODI) settings.

Regarding the performance in the OCI setting (see Table 8, Fig. 2 and 3), we
conclude:

  * 1.

For memory-free methods, LwF is effective in CIFAR100 and Mini-ImageNet,
showing similar performance as replay-based methods with a small memory
buffer. However, all memory-free methods fail in the larger CORe50-NC.

  * 2.

When the memory buffer is small, iCaRL shows the best performance (by large
margins) in CIFAR100 and Mini-ImageNet, followed by CN-DPM.

  * 3.

With a larger memory buffer, GDumb—a simple baseline—outperforms methods
designed specifically for the CL problem in CIFAR100 and Mini-ImageNet at the
expense of much longer training times.

  * 4.

In the larger and more realistic CORe50-NC dataset, MIR consistently surpasses
all the other methods across different memory sizes.

  * 5.

We experimentally and theoretically confirm that a key cause of CF is the bias
towards new classes in the last fully connected layer due to the imbalance
between previous data and new data [42, 68, 93].

  * 6.

None of the methods show any positive forward and backward transfer due to the
bias mentioned above.

The conclusions from our experiments for the OCI tricks (see Table 9, Fig. 4)
are as follows:

  * 1.

When the memory size is small, LB and NCM are the most effective, showing
around 64% relative improvement.

  * 2.

With a larger memory buffer, NCM remains effective, and RV becomes more
helpful, showing around 80% relative improvement.

  * 3.

When equipped with NCM or RV, both ER and MIR can outperform the best
performance of the compared methods without tricks.

  * 4.

The running times of NCM and RV increase with the growth in memory size, but
other tricks only add a fixed overhead to the running time.

For the ODI setting (see Table 11, Fig. 5), we conclude:

  * 1.

Generally speaking, all replay-based methods (ER, MIR, GSS) show comparable
performance across three memory sizes and outperform all other methods.

  * 2.

GDumb, the strong baseline that dominates the OCI setting in most cases, is no
longer effective, possibly due to its memory update strategy.

  * 3.

Other OCI methods cannot generalize to the ODI setting.

Detailed comments for compared methods and tricks can be found in Table 12 and
Table 13.

In conclusion, by leveraging the best methods and tricks identified in this
comparative survey, online CL (with a very small mini-batch) is now
approaching offline performance, bringing CL much closer to its ultimate goal
of matching offline training that opens up CL for effective deployment on edge
and other RAM-limited devices.

##  11 Acknowledgement

This research was supported by LG AI Research.

## References

  * [1] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International journal of computer vision 115 (3) (2015) 211–252.
  * [2] M. McCloskey, N. J. Cohen, Catastrophic interference in connectionist networks: The sequential learning problem, in: Psychology of learning and motivation, Vol. 24, Elsevier, 1989, pp. 109–165.
  * [3] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, Y. Bengio, An empirical investigation of catastrophic forgetting in gradient-based neural networks, arXiv preprint arXiv:1312.6211 (2013).
  * [4] G. A. Carpenter, S. Grossberg, A massively parallel architecture for a self-organizing neural pattern recognition machine, Computer vision, graphics, and image processing 37 (1) (1987) 54–115.
  * [5] M. Mermillod, A. Bugaiska, P. Bonin, The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects, Frontiers in psychology 4 (2013) 504.
  * [6] Z. Chen, B. Liu, Lifelong machine learning, Synthesis Lectures on Artificial Intelligence and Machine Learning 12 (3) (2018) 1–207.
  * [7] J. Yoon, E. Yang, J. Lee, S. J. Hwang, Lifelong learning with dynamically expandable networks, in: International Conference on Learning Representations, 2018.
URL https://openreview.net/forum?id=Sk7KsfW0-

  * [8] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, C. H. Lampert, icarl: Incremental classifier and representation learning, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2001–2010.
  * [9] A. Chaudhry, P. K. Dokania, T. Ajanthan, P. H. Torr, Riemannian walk for incremental learning: Understanding forgetting and intransigence, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 532–547.
  * [10] G. M. van de Ven, A. Tolias, Three scenarios for continual learning, ArXiv abs/1904.07734 (2019).
  * [11] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al., Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of Sciences of the United States of America 114 13 (2017) 3521–3526.
  * [12] Z. Li, D. Hoiem, Learning without forgetting, in: ECCV, Springer, 2016, pp. 614–629.
  * [13] D. Lopez-Paz, M. A. Ranzato, Gradient episodic memory for continual learning, in: Advances in Neural Information Processing Systems 30, 2017, pp. 6467–6476.
  * [14] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, N. Díaz-Rodríguez, Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges, Information Fusion 58 (2020) 52–68.
  * [15] M. De Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, T. Tuytelaars, A continual learning survey: Defying forgetting in classification tasks, arXiv preprint arXiv:1909.08383 (2019).
  * [16] S. Farquhar, Y. Gal, Towards robust evaluations of continual learning (2018).  arXiv:1805.09733.
  * [17] R. Aljundi, E. Belilovsky, T. Tuytelaars, L. Charlin, M. Caccia, M. Lin, L. Page-Caccia, Online continual learning with maximal interfered retrieval, in: Advances in Neural Information Processing Systems 32, 2019, pp. 11849–11860.
  * [18] R. Aljundi, M. Lin, B. Goujaud, Y. Bengio, Gradient based sample selection for online continual learning, in: Advances in Neural Information Processing Systems 32, 2019, pp. 11816–11825.
  * [19] S. Lee, J. Ha, D. Zhang, G. Kim, A neural dirichlet process mixture model for task-free continual learning, in: International Conference on Learning Representations, 2020.
  * [20] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, G. Wayne, Experience replay for continual learning, in: Advances in Neural Information Processing Systems 32, 2019, pp. 350–360.
  * [21] A. Nagabandi, C. Finn, S. Levine, Deep online learning via meta-learning: Continual adaptation for model-based RL, in: International Conference on Learning Representations, 2019.
  * [22] C. Kaplanis, M. Shanahan, C. Clopath, Continual reinforcement learning with complex synapses, Vol. 80 of Proceedings of Machine Learning Research, PMLR, Stockholmsmässan, Stockholm Sweden, 2018, pp. 2497–2506.
  * [23] D. Rao, F. Visin, A. Rusu, R. Pascanu, Y. W. Teh, R. Hadsell, Continual unsupervised representation learning, in: Advances in Neural Information Processing Systems, 2019, pp. 7647–7657.
  * [24] A. Prabhu, P. H. Torr, P. K. Dokania, Gdumb: A simple approach that questions our progress in continual learning, in: European Conference on Computer Vision, Springer, 2020, pp. 524–540.
  * [25] Z. Mai, H. Kim, J. Jeong, S. Sanner, Batch-level experience replay with review for continual learning, arXiv preprint arXiv:2007.05683 (2020).
  * [26] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural Networks 113 (2019) 54 – 71.
  * [27] G. I. Parisi, V. Lomonaco, Online continual learning on sequences, in: Recent Trends in Learning From Data, Springer, 2020, pp. 197–221.
  * [28] Y.-C. Hsu, Y.-C. Liu, A. Ramasamy, Z. Kira, Re-evaluating continual learning scenarios: A categorization and case for strong baselines, arXiv preprint arXiv:1810.12488 (2018).
  * [29] B. Pfülb, A. Gepperth, A comprehensive, application-oriented study of catastrophic forgetting in DNNs, in: International Conference on Learning Representations, 2019.
  * [30] R. Kemker, A. Abitino, M. McClure, C. Kanan, Measuring catastrophic forgetting in neural networks, in: AAAI, 2018.
  * [31] A. Chaudhry, M. Ranzato, M. Rohrbach, M. Elhoseiny, Efficient lifelong learning with a-GEM, in: International Conference on Learning Representations, 2019.
  * [32] A. Gepperth, B. Hammer, Incremental learning algorithms and applications, 2016.
  * [33] T. L. Hayes, R. Kemker, N. D. Cahill, C. Kanan, New metrics and experimental paradigms for continual learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 2031–2034.
  * [34] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. K. Dokania, P. H. S. Torr, M. Ranzato, On tiny episodic memories in continual learning (2019).  arXiv:1902.10486.
  * [35] S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, B.-T. Zhang, Overcoming catastrophic forgetting by incremental moment matching, in: Advances in neural information processing systems, 2017, pp. 4652–4662.
  * [36] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, T. Tuytelaars, Memory aware synapses: Learning what (not) to forget, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 139–154.
  * [37] F. Zenke, B. Poole, S. Ganguli, Continual learning through synaptic intelligence, Proceedings of machine learning research 70 (2017) 3987.
  * [38] H. Ritter, A. Botev, D. Barber, Online structured laplace approximations for overcoming catastrophic forgetting, in: Advances in Neural Information Processing Systems, 2018, pp. 3738–3748.
  * [39] X. He, H. Jaeger, Overcoming catastrophic interference using conceptor-aided backpropagation, in: International Conference on Learning Representations, 2018\.
  * [40] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531 (2015).
  * [41] F. M. Castro, M. J. Marin-Jimenez, N. Guil, C. Schmid, K. Alahari, End-to-end incremental learning, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [42] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, Y. Fu, Large scale incremental learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 374–382.
  * [43] A. Rannen, R. Aljundi, M. B. Blaschko, T. Tuytelaars, Encoder based lifelong learning, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1320–1328.
  * [44] C. V. Nguyen, Y. Li, T. D. Bui, R. E. Turner, Variational continual learning, in: International Conference on Learning Representations, 2018.
  * [45] X. Tao, X. Chang, X. Hong, X. Wei, Y. Gong, Topology-preserving class-incremental learning, in: European Conference on Computer Vision, Springer, 2020, pp. 254–270.
  * [46] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in neural information processing systems, 2014, pp. 2672–2680.
  * [47] H. Shin, J. K. Lee, J. Kim, J. Kim, Continual learning with deep generative replay, in: Advances in Neural Information Processing Systems, 2017, pp. 2990–2999.
  * [48] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, Z. Zhang, Y. Fu, Incremental classifier learning with generative adversarial networks, arXiv preprint arXiv:1802.00853 (2018).
  * [49] G. M. van de Ven, A. S. Tolias, Generative replay with feedback connections as a general strategy for continual learning, arXiv preprint arXiv:1809.10635 (2018).
  * [50] T. Lesort, H. Caselles-Dupré, M. Garcia-Ortiz, A. Stoian, D. Filliat, Generative models from the perspective of continual learning, in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–8.
  * [51] A. Mallya, S. Lazebnik, Packnet: Adding multiple tasks to a single network by iterative pruning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7765–7773.
  * [52] C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, A. A. Rusu, A. Pritzel, D. Wierstra, Pathnet: Evolution channels gradient descent in super neural networks, arXiv preprint arXiv:1701.08734 (2017).
  * [53] J. Serra, D. Suris, M. Miron, A. Karatzoglou, Overcoming catastrophic forgetting with hard attention to the task, in: International Conference on Machine Learning, 2018, pp. 4548–4557.
  * [54] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, R. Hadsell, Progressive neural networks, arXiv preprint arXiv:1606.04671 (2016).
  * [55] J. Yoon, E. Yang, J. Lee, S. J. Hwang, Lifelong learning with dynamically expandable networks, in: International Conference on Learning Representations, 2018.
  * [56] R. Aljundi, P. Chakravarty, T. Tuytelaars, Expert gate: Lifelong learning with a network of experts, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3366–3375.
  * [57] J. Rajasegaran, M. Hayat, S. H. Khan, F. S. Khan, L. Shao, Random path selection for continual learning, in: Advances in Neural Information Processing Systems, Vol. 32, Curran Associates, Inc., 2019, pp. 12669–12679.
  * [58] Z. Borsos, M. Mutnỳ, A. Krause, Coresets via bilevel optimization for continual learning and streaming, arXiv preprint arXiv:2006.03875 (2020).
  * [59] P. Buzzega, M. Boschini, A. Porrello, D. Abati, S. Calderara, Dark experience for general continual learning: a strong, simple baseline, arXiv preprint arXiv:2004.07211 (2020).
  * [60] M. Reimer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, G. Tesauro, Learning to learn without forgetting by maximizing transfer and minimizing interference, in: ICLR, 2019.
  * [61] A. Chrysakis, M.-F. Moens, Online continual learning from imbalanced data, Proceedings of Machine Learning Research (2020).
  * [62] X. Jin, J. Du, X. Ren, Gradient based memory editing for task-free continual learning, arXiv preprint arXiv:2006.15294 (2020).
  * [63] C. D. Kim, J. Jeong, G. Kim, Imbalanced continual learning with partitioning reservoir sampling, arXiv preprint arXiv:2009.03632 (2020).
  * [64] G. Gupta, K. Yadav, L. Paull, La-maml: Look-ahead meta learning for continual learning, arXiv preprint arXiv:2007.13904 (2020).
  * [65] A. Iscen, J. Zhang, S. Lazebnik, C. Schmid, Memory-efficient incremental learning through feature adaptation, arXiv preprint arXiv:2004.00713 (2020).
  * [66] K. Joseph, V. N. Balasubramanian, Meta-consolidation for continual learning, arXiv preprint arXiv:2010.00352 (2020).
  * [67] D. Roy, P. Panda, K. Roy, Tree-cnn: a hierarchical deep convolutional neural network for incremental learning, Neural Networks 121 (2020) 148–160.
  * [68] B. Zhao, X. Xiao, G. Gan, B. Zhang, S.-T. Xia, Maintaining discrimination and fairness in class incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13208–13217.
  * [69] S. Hou, X. Pan, C. C. Loy, Z. Wang, D. Lin, Learning a unified classifier incrementally via rebalancing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 831–839.
  * [70] E. Belouadah, A. Popescu, Il2m: Class incremental learning with dual memory, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 583–592.
  * [71] J. He, R. Mao, Z. Shao, F. Zhu, Incremental learning in online scenario, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13926–13935.
  * [72] P. Dhar, R. V. Singh, K.-C. Peng, Z. Wu, R. Chellappa, Learning without memorizing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5138–5146.
  * [73] J. Zhang, J. Zhang, S. Ghosh, D. Li, S. Tasci, L. Heck, H. Zhang, C.-C. J. Kuo, Class-incremental learning via deep model consolidation, in: The IEEE Winter Conference on Applications of Computer Vision, 2020, pp. 1131–1140.
  * [74] M. Riemer, T. Klinger, D. Bouneffouf, M. Franceschini, Scalable recollections for continual lifelong learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp. 1352–1359.
  * [75] L. Caccia, E. Belilovsky, M. Caccia, J. Pineau, Online learned continual compression with adaptive quantization modules, arXiv (2019) arXiv–1911.
  * [76] D. Maltoni, V. Lomonaco, Continuous learning in single-incremental-task scenarios, Neural Networks 116 (2019) 56–73.
  * [77] Y. Liu, Y. Su, A.-A. Liu, B. Schiele, Q. Sun, Mnemonics training: Multi-class incremental learning without forgetting, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12245–12254.
  * [78] L. Yu, B. Twardowski, X. Liu, L. Herranz, K. Wang, Y. Cheng, S. Jui, J. v. d. Weijer, Semantic drift compensation for class-incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6982–6991.
  * [79] O. Ostapenko, M. Puscas, T. Klein, P. Jahnichen, M. Nabi, Learning to remember: A synaptic plasticity driven framework for continual learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 11321–11329.
  * [80] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, Z. Zhang, Y. Fu, Incremental classifier learning with generative adversarial networks, arXiv preprint arXiv:1802.00853 (2018).
  * [81] J. Rajasegaran, S. Khan, M. Hayat, F. S. Khan, M. Shah, itaml: An incremental task-agnostic meta-learning approach, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13588–13597.
  * [82] D. Abati, J. Tomczak, T. Blankevoort, S. Calderara, R. Cucchiara, B. E. Bejnordi, Conditional channel gated networks for task-aware continual learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3931–3940.
  * [83] J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, R. Hadsell, Progress & compress: A scalable framework for continual learning, in: ICML, 2018.
  * [84] X. Liu, M. Masana, L. Herranz, J. Van de Weijer, A. M. Lopez, A. D. Bagdanov, Rotate your networks: Better weight consolidation and less catastrophic forgetting, in: 2018 24th International Conference on Pattern Recognition (ICPR), IEEE, 2018, pp. 2262–2268.
  * [85] Z. Mai, D. Shim, J. Jeong, S. Sanner, H. Kim, J. Jang, Adversarial shapley value experience replay for task-free continual learning, arXiv preprint arXiv:2009.00093 (2020).
  * [86] J. S. Vitter, Random sampling with a reservoir, ACM Transactions on Mathematical Software (TOMS) 11 (1) (1985) 37–57.
  * [87] T. Mensink, J. Verbeek, F. Perronnin, G. Csurka, Distance-based image classification: Generalizing to new classes at near-zero cost, IEEE transactions on pattern analysis and machine intelligence 35 (11) (2013) 2624–2637.
  * [88] M. Welling, Herding dynamical weights to learn, in: Proceedings of the 26th Annual International Conference on Machine Learning, 2009, pp. 1121–1128.
  * [89] T. L. Hayes, N. D. Cahill, C. Kanan, Memory efficient experience replay for streaming learning, in: 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 9769–9776.
  * [90] T. S. Ferguson, Bayesian density estimation by mixtures of normal distributions, in: Recent advances in statistics, Elsevier, 1983, pp. 287–302.
  * [91] D. Lin, Online learning of nonparametric mixture models via sequential variational approximation, in: Advances in Neural Information Processing Systems, 2013, pp. 395–403.
  * [92] D. P. Kingma, M. Welling, Auto-encoding variational bayes, arXiv preprint arXiv:1312.6114 (2013).
  * [93] H. Ahn, T. Moon, A simple class decision balancing for incremental learning (2020).  arXiv:2003.13947.
  * [94] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  * [95] Q. Dong, S. Gong, X. Zhu, Imbalanced deep learning by minority class incremental rectification, IEEE transactions on pattern analysis and machine intelligence 41 (6) (2018) 1367–1381.
  * [96] X. Yin, X. Yu, K. Sohn, X. Liu, M. Chandraker, Feature transfer learning for deep face recognition with under-represented data, arXiv preprint arXiv:1803.09014 (2018).
  * [97] S. Gidaris, N. Komodakis, Dynamic few-shot visual learning without forgetting, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4367–4375.
  * [98] K. Lee, K. Lee, J. Shin, H. Lee, Overcoming catastrophic forgetting with unlabeled data in the wild, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 312–321.
  * [99] H. Jung, J. Ju, J. Kim, Less-forgetful learning for domain expansion in deep neural networks, in: AAAI-18: Thirty-Second AAAI Conference on Artificial Intelligence, the Association for the Advancement of Artificial Intelligence, 2018\.
  * [100] A. Douillard, M. Cord, C. Ollion, T. Robert, Podnet: Pooled outputs distillation for small-tasks incremental learning, Springer.
  * [101] C. Zeno, I. Golan, E. Hoffer, D. Soudry, Task agnostic continual learning using online variational bayes (2018).  arXiv:1803.10123.
  * [102] K. Javed, F. Shafait, Revisiting distillation and incremental classifier learning, in: Asian Conference on Computer Vision, Springer, 2018, pp. 3–17.
  * [103] A. Krizhevsky, Learning multiple layers of features from tiny images, Tech. rep., University of Toronto (April 2009).
  * [104] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al., Matching networks for one shot learning, in: Advances in neural information processing systems, 2016, pp. 3630–3638.
  * [105] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009, pp. 248–255.
  * [106] V. Lomonaco, D. Maltoni, Core50: a new dataset and benchmark for continuous object recognition, in: Conference on Robot Learning, 2017, pp. 17–26.
  * [107] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE 86 (11) (1998) 2278–2324.
  * [108] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [109] V. Lomonaco, L. Pellegrini, P. Rodriguez, M. Caccia, Q. She, Y. Chen, Q. Jodelet, R. Wang, Z. Mai, D. Vazquez, et al., Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions, arXiv preprint arXiv:2009.09929 (2020).
  * [110] L. Liu, Z. Kuang, Y. Chen, J. H. Xue, W. Yang, W. Zhang, Incdet: In defense of elastic weight consolidation for incremental object detection, IEEE Transactions on Neural Networks and Learning Systems (2020) 1–14doi:10.1109/TNNLS.2020.3002583.
  * [111] T. Lesort, A. Stoian, D. Filliat, Regularization shortcomings for continual learning, arXiv preprint arXiv:1912.03049 (2019).
  * [112] J. Knoblauch, H. Husain, T. Diethe, Optimal continual learning has perfect memory and is np-hard, arXiv preprint arXiv:2006.05188 (2020).
  * [113] E. Belouadah, A. Popescu, Scail: Classifier weights scaling for class incremental learning, in: The IEEE Winter Conference on Applications of Computer Vision, 2020, pp. 1266–1275.
  * [114] P. Roy, S. Ghosh, S. Bhattacharya, U. Pal, Effects of degradations on deep neural network architectures, arXiv preprint arXiv:1807.10108 (2018).
  * [115] C. Wu, L. Herranz, X. Liu, J. van de Weijer, B. Raducanu, et al., Memory replay gans: Learning to generate new categories without forgetting, in: Advances in Neural Information Processing Systems, 2018, pp. 5962–5972.
  * [116] T. L. Hayes, K. Kafle, R. Shrestha, M. Acharya, C. Kanan, Remind your neural network to prevent catastrophic forgetting, in: European Conference on Computer Vision, Springer, 2020, pp. 466–483.
  * [117] X. Liu, C. Wu, M. Menta, L. Herranz, B. Raducanu, A. D. Bagdanov, S. Jui, J. van de Weijer, Generative feature replay for class-incremental learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 226–227.
  * [118] L. Pellegrini, G. Graffieti, V. Lomonaco, D. Maltoni, Latent replay for real-time continual learning, arXiv preprint arXiv:1912.01100 (2019).
  * [119] S. Chopra, R. Hadsell, Y. LeCun, Learning a similarity metric discriminatively, with application to face verification, in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 1, IEEE, 2005, pp. 539–546.
  * [120] T. L. Hayes, C. Kanan, Lifelong machine learning with deep streaming linear discriminant analysis, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 220–221.
  * [121] S. Pang, S. Ozawa, N. Kasabov, Incremental linear discriminant analysis for classification of data streams, IEEE transactions on Systems, Man, and Cybernetics, part B (Cybernetics) 35 (5) (2005) 905–914.
  * [122] T. Hospedales, A. Antoniou, P. Micaelli, A. Storkey, Meta-learning in neural networks: A survey, arXiv preprint arXiv:2004.05439 (2020).
  * [123] K. Javed, M. White, Meta-learning representations for continual learning, in: Advances in Neural Information Processing Systems, 2019, pp. 1820–1830.
  * [124] M. Caccia, P. Rodriguez, O. Ostapenko, F. Normandin, M. Lin, L. Caccia, I. Laradji, I. Rish, A. Lacoste, D. Vazquez, et al., Online fast adaptation and knowledge accumulation: a new approach to continual learning, arXiv preprint arXiv:2003.05856 (2020).
  * [125] C. Finn, P. Abbeel, S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks, in: ICML, 2017.
  * [126] K. Shmelkov, C. Schmid, K. Alahari, Incremental learning of object detectors without catastrophic forgetting, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 3400–3409.
  * [127] D. Li, S. Tasci, S. Ghosh, J. Zhu, J. Zhang, L. Heck, Rilod: near real-time incremental learning for object detection at the edge, in: Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, 2019, pp. 113–126.
  * [128] L. Chen, C. Yu, L. Chen, A new knowledge distillation for incremental object detection, in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–7.
  * [129] X. Liu, H. Yang, A. Ravichandran, R. Bhotika, S. Soatto, Multi-task incremental learning for object detection.
  * [130] K. Joseph, J. Rajasegaran, S. Khan, F. S. Khan, V. Balasubramanian, L. Shao, Incremental object detection via meta-learning, arXiv preprint arXiv:2003.08798 (2020).
  * [131] M. Acharya, T. L. Hayes, C. Kanan, Rodeo: Replay for online object detection, arXiv preprint arXiv:2008.06439 (2020).
  * [132] S. Sodhani, S. Chandar, Y. Bengio, Toward training recurrent neural networks for lifelong learning, Neural computation 32 (1) (2020) 1–35.
  * [133] M. Schak, A. Gepperth, A study on catastrophic forgetting in deep lstm networks, in: International Conference on Artificial Neural Networks, Springer, 2019, pp. 714–728.
  * [134] G. Arora, A. Rahimi, T. Baldwin, Does an lstm forget more than a cnn? an empirical study of catastrophic forgetting in nlp, in: Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association, 2019, pp. 77–86.
  * [135] T. Chen, I. Goodfellow, J. Shlens, Net2net: Accelerating learning via knowledge transfer, arXiv preprint arXiv:1511.05641 (2015).
  * [136] B. Ehret, C. Henning, M. R. Cervera, A. Meulemans, J. von Oswald, B. F. Grewe, Continual learning in recurrent neural networks with hypernetworks, arXiv preprint arXiv:2006.12109 (2020).
  * [137] L. Duncker, L. Driscoll, K. V. Shenoy, M. Sahani, D. Sussillo, Organizing recurrent network dynamics by task-computation to enable continual learning, Advances in Neural Information Processing Systems 33 (2020).
  * [138] F.-K. Sun, C.-H. Ho, H.-Y. Lee, Lamol: Language modeling for lifelong language learning, in: International Conference on Learning Representations, 2019.
  * [139] Y. Li, L. Zhao, K. Church, M. Elhoseiny, Compositional language continual learning, in: International Conference on Learning Representations, 2019.
  * [140] F. Mi, L. Chen, M. Zhao, M. Huang, B. Faltings, Continual learning for natural language generation in task-oriented dialog systems, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, 2020, pp. 3461–3474.
  * [141] S. Mazumder, B. Liu, S. Wang, N. Ma, Lifelong and interactive learning of factual knowledge in dialogues, arXiv preprint arXiv:1907.13295 (2019).
  * [142] B. Liu, S. Mazumder, Lifelong learning dialogue systems: Chatbots that self-learn on the job, arXiv preprint arXiv:2009.10750 (2020).
  * [143] R. Del Chiaro, B. Twardowski, A. Bagdanov, J. van de Weijer, Ratt: Recurrent attention to transient tasks for continual image captioning, Advances in Neural Information Processing Systems 33 (2020).
  * [144] Z. Ke, B. Liu, H. Wang, L. Shu, Continual learning with knowledge transfer for sentiment classification, in: Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2020\.
  * [145] T. Liu, L. Ungar, J. Sedoc, Continual learning for sentence representations using conceptors, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 3274–3279.
  * [146] F. Mi, B. Faltings, Memory augmented neural model for incremental session-based recommendation, arXiv preprint arXiv:2005.01573 (2020).
  * [147] F. Yuan, G. Zhang, A. Karatzoglou, X. He, J. Jose, B. Kong, Y. Li, One person, one model, one world: Learning continual user representation without forgetting, arXiv preprint arXiv:2009.13724 (2020).
  * [148] F. Mi, X. Lin, B. Faltings, Ader: Adaptively distilled exemplar replay towards continual learning for session-based recommendation, in: Fourteenth ACM Conference on Recommender Systems, 2020, pp. 408–413.
  * [149] Y. Xu, Y. Zhang, W. Guo, H. Guo, R. Tang, M. Coates, Graphsail: Graph structure aware incremental learning for recommender systems, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, 2020, pp. 2861–2868.
  * [150] B. Liu, Learning on the job: Online lifelong and continual learning., in: AAAI, 2020, pp. 13544–13549.
  * [151] T. Diethe, T. Borchert, E. Thereska, B. Balle, N. Lawrence, Continual learning in practice, arXiv preprint arXiv:1903.05202 (2019).
  * [152] C. S. Lee, A. Y. Lee, Clinical applications of continual learning machine learning, The Lancet Digital Health 2 (6) (2020) e279–e281.
  * [153] M. D. Lange, X. Jia, S. Parisot, A. Leonardis, G. Slabaugh, T. Tuytelaars, Unsupervised model personalization while preserving privacy and scalability: An open problem, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14463–14472.

##  Appendix A Algorithms

In this section, we provide the algorithms for different memory update
strategies described in Section 6.

1 procedure  __MemoryUpdate $(mem_{s}z,t,n,B)$

3 2 $j\leftarrow 0$

5 4 for  _$(\mathrm{x},y)$ in $B$_ do

7 6 $\quad M\leftarrow|\mathcal{M}|$ $\triangleright$ Number of samples
currently stored in the memory

9 8 if  _$M <\mathrm{mem}_{-}\mathrm{sz}$_ then

11 10 $\quad\mathcal{M}$.append $(\mathrm{x},y,t)$

12 else

14 13 $i=\operatorname{randint}(0,n+j)$

16 15 if  _$i <\mathrm{mem}_{-}\mathrm{sz}$_ then

18 17 $\mathcal{M}[i]\leftarrow(\mathbf{x},y,t)$ $\triangleright$ Overwrite
memory slot

20 19$\quad j\leftarrow j+1$

21

22 return $\mathcal{M}$

Algorithm 3 Reservoir sampling

1 Input : $n,M$

2 Initialize: $\mathcal{M},\mathcal{C}$

3 Receive: $(x,y)$

4 Update: $(x,y,\mathcal{M})$

65 $X,Y\leftarrow$ RandomSubset $(\mathcal{M},\mathrm{n})$

87 $g\leftarrow\nabla\ell_{\theta}(x,y);G\leftarrow\nabla_{\theta}\ell(X,Y)$

109 $c=\max_{i}\left(\frac{\left\langle
g,G_{i}\right\rangle}{\|g\|\left\|G_{i}\right\|}\right)+1$ $\triangleright$
make the score positive

1211 if  _$\operatorname{len}(\mathcal{M}) >=M$_ then

14 13 if  _$c <1$_ then

15 $\triangleright$ cosine similarity $<0$

17 16 $i\sim P(i)=\mathcal{C}_{i}/\sum_{j}\mathcal{C}_{j}$

19 18 $r\sim\text{ uniform }(0,1)$

21 20 if  _$r <\mathcal{C}_{i}/\left(\mathcal{C}_{i}+c\right)$_ then

23 22 $\quad\mathcal{M}_{i}\leftarrow(x,y);\mathcal{C}_{i}\leftarrow c$

25 24end if

27 26end if

2928else

31 30 $\mathcal{M}\leftarrow\mathcal{M}\cup\\{(x,y)\\};\mathcal{C}\cup\\{c\\}$

3332end if

Algorithm 4 GSS-Greedy

1 Init: counter $C_{0}=\\{\\},\mathcal{D}_{0}=\\{\\}$ with capacity $k.$
Online samples arrive from $\mathrm{t}=1$

32 function SAMPLE
$\left(x_{t},y_{t},\mathcal{D}_{t-1},\mathcal{Y}_{t-1}\right)\quad$
$\triangleright$ Input: New sample and past state

54 $k_{c}=\frac{k}{\left|\mathcal{Y}_{t-1}\right|}$

76 if  _$y_{t}\notin\mathcal{Y}_{t-1}$ or $C_{t-1}\left[y_{t}\right]<k_{c}$_
then

9 8 if  _$\sum_{i}C_{i} >=k$_ then

10 $\triangleright$ If memory is full, replace

12 11 $y_{r}=\operatorname{argmax}\left(C_{t-1}\right)$ $\triangleright$
Select largest class, break ties randomly

14 13 $\left(x_{i},y_{i}\right)=\mathcal{D}_{t-1}\cdot\text{ random
}\left(y_{r}\right)$ $\triangleright$ Select random sample from class $y_{r}$

16 15
$\mathcal{D}_{t}=\left(\mathcal{D}_{t-1}-\left(x_{i},y_{i}\right)\right)\cup\left(x_{t},y_{t}\right)$

18 17 $C_{t}\left[y_{r}\right]=C_{t-1}\left[y_{r}\right]-1$

20 19 else

21 $\triangleright$ If memory has space, add

23 22 $\mathcal{D}_{t}=\mathcal{D}_{t-1}\cup\left(x_{t},y_{t}\right)$

25 24 end if

27 26 $\mathcal{Y}_{t}=\mathcal{Y}_{t-1}\cup y_{t}$

29 28 $C_{t}\left[y_{t}\right]=C_{t-1}\left[y_{t}\right]+1$

3130 end if

3332 return $\mathcal{D}_{t}$

3534 end function

Algorithm 5 Greedy Balancing Sampler

##  Appendix B Experiment Details

###  B.1 Dataset Detail

The summary of dataset statistics is provided in Table 1.

The strength of each nonstationary type used in the experiments are summarized
below.

  * 1.

Noise: [0.0, 0.4, 0.8, 1.2, 1.6, 2.0, 2.4, 2.8, 3.2, 3.6]

  * 2.

Occlusion: [0.0, 0.07, 0.13, 0.2, 0.27, 0.33, 0.4, 0.47, 0.53, 0.6]

  * 3.

Blur: [0.0, 0.28, 0.56, 0.83, 1.11, 1.39, 1.67, 1.94, 2.22, 2.5]

Dataset | #Task | #Train/task | #Test/task | #Class | Image Size |  Setting
---|---|---|---|---|---|---
Split MiniImageNet | 20 | 2500 | 500 | 100 | 3x84x84 | OCI
Split CIFAR-100 | 20 | 2500 | 500 | 100 | 3x32x32 | OCI
CORe50-NC | 9 |  12000$\sim$24000 |  4500$\sim$9000 | 50 | 3x128x128 | OCI
NS-MiniImageNet | 10 | 5000 | 1000 | 100 | 3x84x84 | ODI
CORe50-NI | 8 | 15000 | 44972 | 50 | 3x128x128 | ODI
Table 1: Summary of dataset statistics

###  B.2 Implementation Details

This section describes the implementation details of each method, including
the hyperparameter grid considered for each dataset (see Table 2). As we
described in Section 4 of the main paper, the first $D^{CV}$ tasks are used
for hyperparameter tuning to satisfy the requirement that the model does not
see the data of a task more than once, and $D^{CV}$ is set to 2 in this work.

  * 1.

EWC++: We set the $\alpha$ in Eq. (8) to 0.9 as suggested in the original
paper. We tune three hyperparameters in EWC++, learning rate (LR), weight
decay(WD) and $\lambda$ in Eq. (7).

  * 2.

LwF: We set the temperature factor $T=2$ as the original paper and other CL
papers. The coefficient $\lambda$ for $\mathcal{L}_{KD}$ is set to
$\frac{\mathinner{\\!\left\lvert
C_{new}\right\rvert}}{\mathinner{\\!\left\lvert
C_{old}\right\rvert}+\mathinner{\\!\left\lvert C_{new}\right\rvert}}$
following the idea from [42] and the coefficient for $\mathcal{L}_{CE}$ is set
to $1-\lambda$.

  * 3.

ER: The reservoir sampling used in MemoryUpdate follows Algorithm 3 in A. For
MemoryRetrieval, we randomly select samples with mini-batch size of 10
irrespective of the size of the memory buffer.

  * 4.

MIR: To reduce the computational cost, MIR selects $C$ random samples from the
memory buffer as the candidate set to perform the criterion search. We tune
LR, WD as well as $C$.

  * 5.

GSS: For every incoming sample, GSS computes the cosine similarity of the new
sample gradient to $n$ gradient vectors of samples randomly drawn from the
memory buffer (see Algorithm 4 in A). Other than LR and WD, we also tune $n$.

  * 6.

iCaRL: We replace the herding-based [88] memory update method with reservoir
sampling to accommodate the online setting. We use random sampling for
MemoryRetrieval and tune LR and WD.

  * 7.

A-GEM: We use reservoir sampling for MemoryUpdate and random sampling for
MemoryRetrieval and tune LR and WD.

  * 8.

CN-DPM: CN-DPM is much more sensitive to hyperparameters than others. We need
to use different hyperparameter grids for different scenarios and datasets.
Other than LR, we tune $\alpha$, the concentration parameter controlling how
sensitive the model is to new data and classifier_chill $cc$, the parameter
used to adjust the VAE loss to have a similar scale as the classifier loss.

  * 9.

GDumb: We use batch size of 16 and 30 epochs for all memory sizes. We clip
gradient norm with max norm 10.0 and tune LR and WD.

Method | CIFAR-100 | Mini-ImageNet | CORe50-NC | NS-MiniImageNet | CORe50-NI
---|---|---|---|---|---
EWC++ | LR: [0.0001, 0.001, 0.01, 0.1]
WD: [0.0001, 0.001], $\lambda$: [0, 100, 1000]
LwF | LR: [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1]
WD: [0.0001, 0.001, 0.01, 0.1]
ER | LR: [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1]
WD: [0.0001, 0.001, 0.01, 0.1]
MIR | LR:[0.0001, 0.001, 0.01, 0.1]
WD: [0.0001, 0.001], $C$: [25, 50, 100]
GSS | LR:[0.0001, 0.001, 0.01, 0.1]
WD: [0.0001, 0.001], $n$: [10, 20, 50]
iCaRL | LR: [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1]
WD: [0.0001, 0.001, 0.01, 0.1]
A-GEM | LR: [0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1]
WD: [0.0001, 0.001, 0.01, 0.1]
CN-DPM | LR: [0.0001, 0.001, 0.01, 0.1] | [0.001, 0.005, 0.01] | [0.001, 0.01] | [0.001, 0.005, 0.01] | [0.001, 0.01]
$cc$: [0.001, 0.01, 0.1] | [0.001, 0.0015, 0.002] | [0.0005, 0.001, 0.002] | [0.0005, 0.001, 0.002] | [0.0005, 0.001, 0.002]
$\alpha$: [-100, -300, -500] | [-1200, -1000, -800] | [-1200, -1000, -800, -300] | [-15000, -5000, -500] | [-1200, -1000, -800, -300]
GDumb | LR: .001, 0.01, 0.1], WD:[0.0001, 0.000001]

Table 2:  Hyperparameter grid for the compared methods.

##  Appendix C Additional Experiments and Results

###  C.1 More Results for OCI Setting

Fig 1, 2 and 3 show the average accuracy measured by the end of each task on
Split CIFAR-100, Mini-ImageNet and CORe50-NC with three different memory
buffer sizes (1k, 5k, 10k).

Figure 1: The average accuracy measured by the end of each task for the OCI
setting on Split CIFAR-100 with three memory sizes. Figure 2: The average
accuracy measured by the end of each task for the OCI setting on Split Mini-
ImageNet with three memory sizes. Figure 3: The average accuracy measured by
the end of each task for the OCI setting on CORe50-NC with three memory sizes.

###  C.2 OCI Tricks on Split Mini-ImageNet

We evaluate the tricks described in Section 7.3 on Split Mini-ImageNet. As
shown in Table 1 and Fig. 4, we find similar results as in Split CIFAR-100
that all tricks are beneficial. LB and KDC* are most useful when the memory
buffer is small, and NCM and RV are more effective when the memory buffer is
large. One main difference is that NCM is not as effective as in CIFAR-100
with a 10k memory buffer as base methods with NCM cannot outperform the best
OCI performance.

Finetune | $3.4\pm 0.2$
---|---
Offline | $51.9\pm 0.5$
Method | A-GEM | ER | MIR
Buffer Size | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k | M=1k | M=5k | M=10k
NA | $3.4\pm 0.2$ | $3.7\pm 0.3$ | $3.3\pm 0.3$ | $6.4\pm 0.9$ | $14.5\pm 2.1$ | $15.9\pm 2.0$ | $6.4\pm 0.9$ | $16.5\pm 2.1$ | $21.0\pm 1.1$
LB | $5.8\pm 0.8$ | $5.8\pm 0.5$ | $5.4\pm 0.9$ | $14.4\pm 2.1$ | $19.3\pm 2.3$ | $22.1\pm 1.1$ | $\mathbf{17.1\pm 0.9}$ | $21.7\pm 0.7$ | $23.0\pm 0.8$
KDC | $8.0\pm 1.1$ | $7.5\pm 1.5$ | $8.2\pm 1.7$ | $12.3\pm 2.5$ | $15.4\pm 0.4$ | $14.6\pm 2.1$ | $14.3\pm 0.5$ | $15.8\pm 0.4$ | $15.5\pm 0.5$
KDC* | $5.6\pm 0.4$ | $5.5\pm 0.5$ | $5.4\pm 0.4$ | $\mathbf{16.4\pm 0.8}$ | $20.3\pm 2.5$ | $23.0\pm 3.1$ | $16.4\pm 0.6$ | $25.1\pm 0.8$ | $26.1\pm 0.9$
MI | $3.5\pm 0.2$ | $3.7\pm 0.2$ | $3.6\pm 0.3$ | $6.4\pm 0.6$ | $16.3\pm 1.3$ | $24.1\pm 1.3$ | $6.6\pm 0.6$ | $15.2\pm 1.1$ | $22.0\pm 1.9$
SS | $5.7\pm 0.9$ | $6.2\pm 0.8$ | $5.7\pm 0.8$ | $12.5\pm 1.9$ | $20.5\pm 2.1$ | $24.1\pm 1.1$ | $14.2\pm 1.0$ | $21.9\pm 0.8$ | $24.7\pm 0.9$
RV | $4.1\pm 0.2$ | $\mathbf{19.9\pm 3.7}$ | $\mathbf{25.5\pm 4.7}$ | $11.4\pm 0.6$ | $\mathbf{32.1\pm 0.8}$ | $\mathbf{36.3\pm 1.5}$ | $9.1\pm 0.5$ | $\mathbf{29.9\pm 0.7}$ | $\mathbf{37.3\pm 0.5}$
NCM | $\mathbf{10.2\pm 0.4}$ | $11.7\pm 1.5$ | $13.0\pm 0.5$ | $14.2\pm 0.7$ | $26.7\pm 0.7$ | $28.2\pm 0.6$ | $13.6\pm 0.6$ | $26.4\pm 0.7$ | $28.6\pm 0.4$
Best OCI | $14.7\pm 0.4$ | ${21.1\pm 1.7}$ | ${31.0\pm 1.4}$ | $14.7\pm 0.4$ | ${21.1\pm 1.7}$ | ${31.0\pm 1.4}$ | $14.7\pm 0.4$ | ${21.1\pm 1.7}$ | ${31.0\pm 1.4}$

Table 1: Performance of compared tricks for the OCI setting on Split Mini-
ImageNet. We report average accuracy (end of training) for memory buffer with
size 1k, 5k and 10k. Best OCI refers to the best performance from the compared
methods in Table 8.  Figure 4: Comparison of various tricks for the OCI
setting on Split Mini-ImageNet. We report average accuracy (end of training)
for memory buffer with size 1k, 5k and 10k.

###  C.3 More Results for ODI Setting

Fig. 5, 6 and 7 show the average accuracy measured by the end of each task on
Mini-ImageNet-Noise, Mini-ImageNet-Occlusion and CORe50-NI with three
different memory buffer sizes (1k, 5k, 10k).

Figure 5: The average accuracy measured by the end of each task for the ODI
setting on Mini-ImageNet-Noise with three memory sizes. Figure 6: The average
accuracy measured by the end of each task for the ODI setting on Mini-
ImageNet-Occlusion with three memory sizes. Figure 7: The average accuracy