# Heterogeneous Continual Learning

Divyam Madaan1,2 , Hongxu Yin1, Wonmin Byeon1, Jan Kautz1, Pavlo Molchanov1

1NVIDIA, 2New York University
<EMAIL_ADDRESS>{dannyy, wbyeon, jkautz<EMAIL_ADDRESS>Work
done during an internship at NVIDIA.

###### Abstract

We propose a novel framework and a solution to tackle the continual learning
(CL) problem with changing network architectures. Most CL methods focus on
adapting a single architecture to a new task/class by modifying its weights.
However, with rapid progress in architecture design, the problem of adapting
existing solutions to novel architectures becomes relevant. To address this
limitation, we propose _Heterogeneous Continual Learning (HCL)_ , where a wide
range of evolving network architectures emerge continually together with novel
data/tasks. As a solution, we build on top of the distillation family of
techniques and modify it to a new setting where a weaker model takes the role
of a teacher; meanwhile, a new stronger architecture acts as a student.
Furthermore, we consider a setup of limited access to previous data and
propose _Quick Deep Inversion (QDI)_ to recover prior task visual features to
support knowledge transfer. QDI significantly reduces computational costs
compared to previous solutions and improves overall performance. In summary,
we propose a new setup for CL with a modified knowledge distillation paradigm
and design a quick data inversion method to enhance distillation. Our
evaluation of various benchmarks shows a significant improvement on accuracy
in comparison to state-of-the-art methods over various networks architectures.

##  1 Introduction

Over the past decade, we have seen a plethora of innovations in deep neural
networks (DNNs) that have led to remarkable improvement in performance on
several applications [40, 35, 34]. AlexNet [24] was the first breakthrough
showing the potential of deep learning on the ImageNet benchmark [12], it was
followed by various architectural advances such as VGG [46], Inception [48],
ResNet and its variants [15, 58, 53], efficient architectures [18, 49], and
applications of Transformer [52] in computer vision [14, 32]; however, all
these architectures are trained from scratch and compared on ImageNet
classification [12]. In real-world scenarios, the dataset size is not
constant, yet storage of billions of data instances and retraining the model
representations from scratch is computationally expensive and infeasible.
Further, access to old datasets is often not available due to data privacy.
Thus, it is crucial to transfer the knowledge learned by the previous model
representations to the recent state-of-the-advances without forgetting
knowledge and accessing prior datasets.

Continual learning (CL) [50] is a common way to train representations on a
data stream without forgetting the learned knowledge. However, all prior CL
techniques [59, 1, 43, 41, 42, 4, 33] continually adapt a fixed representation
structure – a fully-connected network with two hidden layers for MNIST [26]
and standard ResNet-18 [15] for all the other datasets (see Figure 1). While
recent works investigate the effect of depth and width [37] on continual
learning, to our knowledge, no work has focused on the real problem of
learning on sequentially arriving tasks with changing network architectures.
Moreover, most works assume the weight transfer with the same architecture
[43, 56], in case of the previous model being available only as black box,
these methods are not applicable.

Figure 1: Illustration of standard and heterogenous continual learning.
Standard CL updates the same representational structure while preserving
previous information with incoming tasks. On the contrary, the goal of HCL is
to continuously evolve the network architecture while maintaining the accuracy
of new data instances.

In this work, we argue that maintaining the same network structure during
training is not a realistic assumption for practical applications such as
autonomous driving, clinical applications, and recommendation systems.
Instead, constant upgrades to stronger models with the best-known architecture
are vital for the best customer experience and competitive advantage, and all
streams of previous data are not only large but difficult to store, transfer,
manage, and protect against security breach. As a motivating example, consider
the tasks of clinical diagnosis and segmentation using medical images, where
it is essential to continually update the model on arriving data to adapt to
the changing environment. However, continually adapting the old model
architectures can potentially hurt the current and future task performance.
Also, it is often not allowed to retain the previous patients’ data to update
to the new model due to privacy concerns. Based on this motivation, our goal
is to continually update the state-of-the-art deep learning architectures
without storing the previous data while preserving the knowledge from
previously learned representations.

We propose a novel CL framework called _Heterogeneous Continual Learning
(HCL)_ to learn continual representations on a data stream without any
restrictions on the training configuration or network architecture of the
representations (see Figure 1). In particular, we consider the continual
learner as a stream of learners, where we train each learner with the same or
different backbone architecture. To this end, we consider a diverse range of
heterogeneous architectures including LeNet [27], ResNet and its variants [15,
58, 53], and recent architectures such as RegNet [54] and vision transformers
[32]. However, due to the assumption of the same architectural structure
across different tasks, conventional regularization [30, 59, 44, 1] and
architectural methods [43, 56, 29] are not directly applicable for HCL.

To continually learn heterogeneous representations, we revisit knowledge
distillation [16] – a method to transfer the knowledge learned across
different networks. While knowledge distillation has been formulated for
continual learning [30, 41, 4], prior methods do not outperform the experience
replay based methods [41, 42, 2, 4]. Moreover, the distillation pipeline has
not been considered for network evolution. This work shows that knowledge
distillation can outperform state-of-the-art methods with and without replay
examples in standard and heterogeneous continual learning settings with proper
modifications. Specifically, we incorporate _consistency_ and _augmentation_
[3] into the knowledge distillation paradigm for CL and low-temperature
transfer with label-smoothing [6] for learning the network representations.
Our solution comes at the cost of storing and reusing only the most recent
model.

The continual learner might not have access to the prior data. Therefore,
motivated by DeepInversion (DI) [55], we consider _data-free continual
learning_. In particular, DI optimizes random noise to generate class-
conditioned samples; however, it requires many steps to optimize a single
batch of instances. This increases its computational cost and slows CL. In
response to this, we propose _Quick Deep Inversion (QDI)_ , which utilizes
current task examples as the starting point for synthetic examples and
optimizes them to minimize the prediction error on the previous tasks.
Specifically, this leads to an interpolation between the current task and
previous task instances, which promotes current task adaptation while
minimizing catastrophic forgetting. We compare our proposed method on various
CL benchmarks against state-of-the-art methods, where it outperforms them
across all settings with a wide variety of architectures. In summary, the
contributions of our work are:

  * •

We propose a novel CL framework called _Heterogeneous Continual Learning
(HCL)_ to learn a stream of different architectures on a sequence of tasks
while transferring the knowledge from past representations.

  * •

We revisit knowledge distillation and propose _Quick Deep Inversion (QDI)_ ,
which inverts the previous task parameters while interpolating the current
task examples with minimal additional cost.

  * •

We benchmark existing state-of-the-art solutions in the new setting and
outperform them with our proposed method across a diverse stream of
architectures for both task-incremental and class-incremental CL.

##  2 Related work

Replay-based approaches. These approaches formulate different criterion [41,
42, 2, 4] to select a representative subset that is revisited in future tasks.
While most approaches [41, 42, 2] store the input examples, Buzzega et al. [4]
retain the logits to preserve the knowledge from the past. However, all these
methods are limited in practical scenarios due to privacy concerns and
excessive memory requirements to store the data from previous tasks. Our
approach does not strictly require a buffer to store data or representation,
and we also do not assume that the learners’ architecture is fixed.

Regularization methods. These methods penalize the change in the
representations by adding a regularization term in the loss [30, 59, 44, 1].
Li & Hoiem [30] minimizes the change in the output representations, and Zenke
et al. [59] estimates the important parameters and restricts their change
during training. However, the regularization cannot be enforced when the
architectures are different across tasks.

Architectural methods. They consider the inclusion of task-dependent
representations to prevent catastrophic forgetting during continual learning
[43, 56, 29, 10, 57]. Rusu et al. [43] instantiates a new column for the new
task and require inputs from both columns and Yoon et al. [57] decomposes each
layer parameters to task-specific and task-shared parameters. The objective of
these methods is to reduce forgetting and the network expansion is restricted
to the same structure. In contrast, we do not make any assumption on the
network structure and incorporate the recent advances in deep learning to
train the continual learner without any knowledge of prior data or its
architecture. To the best of our knowledge, this is the first work that
explores continual learning with different representational structures while
adapting to the current task and minimizing forgetting.

Optimal architectures for continual learning. Prior works also investigate
most optimal architecture for continual learning. Lee et al. [28] formulated a
data-driven layer-based transfer function to learn the layers that should be
transferred to mitigate forgetting. Mirzadeh et al. [36] showed that
increasing the width mitigates forgetting, whereas increasing the depth
negatively affects forgetting, and Mirzadeh et al. [38] investigated the
effect of various components in an architecture. Carta et al. [5] proposed Ex-
Model Continual Learning to learn the continual learner from a stream of
trained models; however, their proposed scheme is not directly applicable to
the HCL setting as the architecture of their learner is fixed and it requires
an extra buffer to store the past expert and CL representations.

##  3 Heterogeneous Continual Learning

In real-world scenarios, the data distribution continuously evolves with time;
thus, adapting the network representations to the distribution shifts is
essential while preserving the knowledge on the past sequence of data.
Meanwhile, deep learning techniques are improving rapidly, providing better
model architectures and techniques with higher performance on various tasks.
However, prior CL methods [59, 1, 43, 41, 42, 4] either optimize the
performance on a single architecture [15] or focus on finding the optimal
architecture for CL [36, 38].

Figure 2: Average accuracy comparison between standard CL – ResNet18 and LeNet
trained with buffer, ours with adaptive architectures without buffer on
CIFAR-10 dataset with two tasks containing five classes each.

To motivate further, we provide a simple experiment on CIFAR-10 split into two
tasks demonstrating the benefit of adapting to novel neural architectures in
Figure 2. With the new ability to switch the network architecture, (e.g., from
LeNet to ResNet) the performance surpasses standard CL setup that is limited
by a rigid but weak architecture (HCL vs. CL LeNet). More appealingly, it even
approximates the stronger model deployed throughout the CL process (HCL vs. CL
ResNet), while alleviating the requirement of replay buffer to store prior
data experiences (HCL is data-free vs. CL with buffer), as we will show next
in methodology and experiment sections.

###  3.1 Problem setup

The problem of standard CL considers the learning objective where the
continual learner $f_{W_{t}}:\mathcal{X}_{t}\rightarrow\mathcal{Y}_{t}$ learns
a sequence of $T$ tasks by updating the fixed representation structure for
each task. Each task $t\in T$ contains training data
$\mathcal{D}_{t}=\\{x_{i},y_{i}\\}_{i=1}^{N_{t}}\sim\mathcal{X}_{t}\times\mathcal{Y}_{t}$,
which is composed of $N_{t}$ identically and independently distributed
examples. More formally, we minimize the following objective:

${\underset{W_{t}}{\text{minimize}}}\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbb{E}_{(x_{i},y_{i})\sim\mathcal{D}_{t}}[\ell\left(f_{W_{t}}(x_{i}),y_{i}\right)],$
(1)

where $\ell:\mathcal{Y}_{t}\times\mathcal{Y}_{t}\rightarrow\mathbb{R}_{\geq
0}$ is the task-specific loss function. In this work, we consider the
continual learner as a stream of architectures
$\\{f^{1}_{W_{1}},\ldots,f^{t}_{W_{t}}\\}$, where the learner can completely
change the backbone architecture to incorporate recent architectural
developments [15, 14, 32, 51] in order to improve model performance.

However, when the architectures are different, there is no natural knowledge
transfer mechanism, and the network parameters are initialized randomly, which
makes this problem challenging compared to standard continual learning.

Particularly, each architectural representation
$f^{t}_{W_{t}}\leavevmode\nobreak\ :\leavevmode\nobreak\
\mathcal{X}_{t}\rightarrow\mathcal{Y}_{t},\leavevmode\nobreak\
\forall{t\in\\{1,\ldots,T\\}}$ is trained on task distribution
$\mathcal{D}_{t}$ and our objective is to train these stream of networks on a
sequence of tasks without forgetting the knowledge of the previous set of
tasks. Additionally, when the network structure remains the same, we want to
transfer the learned representations sequentially to train incoming tasks.
Overall, the learning objective of the HCL is:

${\underset{W_{t}}{\text{minimize}}}\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbb{E}_{(x_{i},y_{i})\sim\mathcal{D}_{t}}[\ell\left(f^{t}_{W_{t}}(x_{i}),y_{i}\right)],$
(2)

For notation simplicity, we discard $W_{t}$ in $f^{t}_{W_{t}}$ for the rest of
the paper. In this work, we focus on the continual learner that does not rely
on task identifiers during training and uses constant memory following prior
works [4, 11]. We consider task-incremental (task-IL) and class-incremental
learning (class-IL) during inference but do not use task labels during
training. In task-IL, we provide the task identity to select the
classification head, whereas class-IL uses a shared head across all classes.
Consequently, class-IL is more challenging due to an equal weight for all the
tasks and prone to higher forgetting and lower accuracy than task-IL; however,
recent works [9] also raise concerns regarding the practicality of class-IL in
real-world scenarios.

###  3.2 Proposed method

We address the problem of HCL with our method summarized in Figure 3. As in
the standard CL, new task data and an optional buffer are used to directly
train a new model (stronger student) initialized from scratch with a task
loss. Additionally, a knowledge transfer data (KDA) is collected given new
task data, (optional) buffer and (optional) synthesised samples by quick deep
inversion. The KDA supports network transfer via knowledge distillation
between old and new models. To describe our method, we first revisit knowledge
distillation [16] in Section 3.2.1 and propose a training paradigm that
outperforms state-of-the-art methods. Then, we introduce quick deep inversion
in Section 3.2.2 to recover past visual features promptly. It inverts the
previous task model to generate synthetic examples

Figure 3: Proposed method to perform continual learning with neural network
architecture change.

####  3.2.1 Revisiting knowledge distillation

Due to a difference in model representations and the absence of the training
data used to train previous models, we concentrate on knowledge distillation
(KD) [16] for HCL. However, our goal is orthogonal to standard KD; instead of
distilling the knowledge from a large model (or ensemble) to a smaller
(weaker) model for efficient deployment we distill from the latter to the
former. Our motivation is to improve model performance by transferring to a
state-of-the-art model.

While KD has been adopted in CL [30, 41, 17, 13, 4], it is (i) limited to
homogeneous architectures across different tasks, and (ii) considered a weak
baseline in comparison to replay-based methods. Contrarily, we focus on HCL
and propose a modified paradigm inspired by the recent advances in KD to
improve the performance and lift both limitations.

Label smoothing. Prior works [39, 45] investigating the compatibility between
label smoothing (LS) and knowledge distillation have derived opposite
conclusions. Recently, Chandrasegaran et al. [6] highlighted that LS-trained
teacher with a low-temperature transfer is essential to achieve students with
higher performance. This inspired us to adapt label smoothing to produce
better soft targets for KD, which improves the knowledge transfer from a
smaller to a larger model architecture reducing the forgetting for both the
standard CL and HCL scenarios.

Knowledge distillation. To improve the KD paradigm for HCL, we incorporate
_consistency_ and _augmentation_ [3]. Specifically, fixed teaching and
distillation without augmentation leads to overfitting of the larger model on
the current task performance. On the contrary, consistent teaching using the
same input views for the student and teacher and augmentation improves
generalization while reducing forgetting. In contrast to standard knowledge
distillation, we use the current task instances for distillation, as the
previous task data is often unavailable in real-world applications due to data
privacy and legal restrictions.

Additionally, our method is applicable to task-free continual learning during
training as it does not require task identity. In addition, we do not use
warmup before the current task as we find that it is not essential to improve
model performance. We conduct ablation experiments to justify the design
choices of every component in Table 2. Our overall objective for the lifelong
learner is:

$\small{\underset{W_{t}}{\text{minimize}}}\leavevmode\nobreak\
\leavevmode\nobreak\
\mathbb{E}_{(x_{i},y_{i})\sim\mathcal{D}_{t}}[\ell\left(f^{t}(x_{i}),y^{t}_{i}(\psi)\right)]+\alpha\cdot\text{KL}\left(p^{t}_{i}(\tau),p^{t-1}_{i}(\tau)\right),$
(3)

where $y^{t}_{i}(\psi)=y(1-\psi)+\psi/C$, $\psi$ denotes the mixture parameter
to interpolate the hard targets to uniform distribution defined using $\psi$,
$C$ is the number of classes, $p^{t}(\tau)$ and $p^{t-1}(\tau)$ denote the
temperature-scaled output probabilities for current task model and past-task
model, respectively, with $\tau$ as the corresponding temperature. $\alpha$ is
the hyper-parameter that controls the strength of KD loss.

####  3.2.2 Enhancing data efficiency

To further improve our data-free HCL paradigm, we extend DeepInversion (DI)
[55], which optimizes images $\tilde{x}$ initialized with Gaussian noise to
improve knowledge transfer from the prior tasks given only the trained models
and no data. The objective of the optimization is to excite particular
features, or, as in our work, classes from previous tasks. The optimization is
guided by proxy image priors: (i) total variation $\mathcal{L}_{tv}$, (ii)
$\mathcal{L}_{\ell_{2}}$ of the generated samples and (iii) feature
distribution regularization $\mathcal{L}_{\text{feature}}$. DI requires many
steps to generate examples prior to every task, which increases the
computational cost of CL method and, therefore, becomes less appealing.

To tackle this limitation, we propose _Quick Deep Inversion (QDI)_ that
initializes the synthetic examples with the current task data prior to the
optimization. There is no reason to believe that randomly sampled Gaussian
noise is an optimal starting point for generating past task examples. We
optimize the current task examples such that features will fall to the
manifold learned by the previous model and the domain shift is minimized. As a
result such imags are classified as past task classes. QDI generates examples
${\tilde{x}}_{\text{prior}}\\{f^{t-1},k\\}$ to approximate features from all
prior tasks $\\{1,\ldots,t-1\\}$ by inverting the last model $f^{t-1}$ with
$k$ optimization steps:

$\displaystyle{\tilde{x}}_{\text{prior}}\\{f^{t-1},k\\}=\underset{\tilde{x}}{\text{argmin}}\leavevmode\nobreak\
\leavevmode\nobreak\ \mathcal{L}(f^{t-1}(\tilde{x}),\tilde{y})$
$\displaystyle+\alpha_{tv}\mathcal{L}_{tv}(\tilde{x})+\alpha_{\ell_{2}}\mathcal{L}_{\ell_{2}}(\tilde{x})+\alpha_{\text{feature}}\mathcal{L}_{\text{feature}},$
(4)

where the synthesized examples are optimized towards prior classes
$\tilde{y}\sim\mathcal{Y}_{\\{1,...,t-1\\}}$ to minimize forgetting.
$\alpha_{tv},\alpha_{\ell_{2}},\alpha_{\text{feature}}$ denote the hyper-
parameters that determine the strength of individual losses. To improve
synthesis speed, we initialize $\tilde{x}$ with the current task input image
$x_{t}$, which provides a $4\times$ speed-up and leads to more natural images:

$\displaystyle\tilde{x}_{\text{prior}}\\{f^{t-1},0\\}=\tilde{x}_{\text{prior},\>k=0}=x_{t}.$

The initialized $\tilde{x}_{\text{prior}}$ can then optimized using Equation
3.2.2, regularized for realism by the target model $f^{t-1}$, hence quickly
unveiling previous task visual features (see Figure 4) on top of the current
task image through

$\displaystyle\small\mathcal{L}_{\text{feature}}=\sum_{l\in
L}\Big{[}d\Big{(}\mu_{l}(\tilde{x}_{\text{prior}}),\mathbb{E}\big{[}\mu_{l}(x)|x\sim{\mathcal{X}_{\\{1,...,t-1\\}}}\big{]}\Big{)}$
$\displaystyle+\>d\Big{(}\sigma_{l}(\tilde{x}_{\text{prior}}),\mathbb{E}\big{[}\sigma_{l}(x)|x\sim{\mathcal{X}_{\\{1,...,t-1\\}}}\big{]}\Big{)}\Big{]}.$
(5)

$d(\cdot,\cdot)$ denotes the distance metric for feature regularization. In
this paper, we use the mean-squared distance. Akin to Yin et al. [55], we
assume Gaussian distribution for feature maps and focus on batch-wise mean
$\mu_{l}(x)$ and variance $\sigma_{l}(x)$ for the layer $l$. We note that
these statistics are implicitly captured through the batch normalization in
$f^{t-1}$ without storing input data for all previous tasks $\\{1,...,t-1\\}$:
$\mathbb{E}\big{[}\mu_{l}(x)|x\sim{\mathcal{X}_{\\{1,...,t-1\\}}}\big{]}\simeq\text{BN}_{l}(\text{running
mean})$,
$\mathbb{E}\big{[}\sigma_{l}(x)|x\sim{\mathcal{X}_{\\{1,...,t-1\\}}}\big{]}\simeq\text{BN}_{l}(\text{running
var}).$ Otherwise, we approximate them using values calculated with post-
convolution feature maps given a current task batch to the target model,
$f^{t-1}$, leveraging its feature extraction capability for previous tasks.
This efficient QDI allows us to use it for continual learning with minimal
additional cost to existing methods. and the learning objective in Equation 3
can be updated as:

$\displaystyle{\underset{W_{t}}{\text{minimize}}}\leavevmode\nobreak\
\leavevmode\nobreak\
\mathbb{E}_{(x_{i},y_{i})\sim\mathcal{D}_{t}}[\ell\left(f^{t}(x_{i}),y^{t}_{i}(\psi)\right)]$
$\displaystyle+\>\alpha\cdot\text{KL}\left(p^{t}_{i}(\tau),p^{t-1}_{i}(\tau)\right)+\>\beta\cdot\text{KL}\left(\tilde{p}^{t}_{i}(\tau),\tilde{p}^{t-1}_{i}(\tau)\right),$
(6)

where $\tilde{p}^{t}(\tau)$ and $\tilde{p}^{t-1}(\tau)$ are the output
probabilities of the generated examples scaled with temperature $\tau$ using
the current and past task-models, $\beta$ is the hyper-parameter to control
the strength of QDI knowledge distillation loss. QDI utilizes the generated
examples to enhance knowledge distillation with a four times speedup to prior
data-inversion methods [55] due to the current task being a better prior than
pixel-wise Gaussian to learn the generated data.

##  4 Experiments

###  4.1 Experimental setup

Datasets and architectures. In this subsection, we provide the details about
the datasets and the architectures, where we consider various architectural
innovations in the ImageNet challenge [12] for HCL.

  1. 1.

Split CIFAR-10 [23]. It consists of CIFAR-10 sized $32\times 32$ split into
five tasks, where each task contains two different classes. We consider a
different architecture for each task in HCL: LeNet [25], ResNet [15], DenseNet
[20], SENet [19], and RegNet [54].

  2. 2.

Split CIFAR-100 [23]. It is a split of CIFAR-100, where the 100 object classes
are divided into 20 tasks. We consider nine architectures in the HCL: LeNet
[25], AlexNet [24], VGG [46] with batch normalization [21], InceptionNet [48],
ResNet [15], ResNeXt [53], DenseNet [20], SENet [19] for two tasks each and
RegNet [54] for last four tasks.

  3. 3.

Split Tiny-ImageNet. It is a variant of ImageNet [12] that consists of images
sized $64\times 64$ from 200 classes. It consists of 10 tasks, where each task
contains 20 classes. The set of architectures in HCL include LeNet [25],
ResNet [15], ResNeXt [53], SENet [19], and RegNet [54] architectures, where
each architecture is used for two consecutive tasks.

Method | $\mathcal{B}$ | Split CIFAR-10 | Split CIFAR-100 | Split Tiny-ImageNet
---|---|---|---|---
|  | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$ | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$ | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$
Standard Continual Learning
Finetune | – | 62.93 ($\pm$ 1.41) | 41.45 ($\pm$ 2.39) | 40.13 ($\pm$ 3.60) | 53.64 ($\pm$ 3.65) | 19.31 ($\pm$ 0.71) | 59.50 ($\pm$ 0.81)
SI [59]∗ | – | 68.05 ($\pm$ 5.91) | 38.76 ($\pm$ 0.89) | – | – | 36.32 ($\pm$ 0.13) | –
LwF [30]∗ | – | 63.29 ($\pm$ 2.35) | 32.56 ($\pm$ 0.56) | – | – | 15.85 ($\pm$ 0.58) | –
DI [55] | – | 89.11 ($\pm$ 3.70) | 7.12 ($\pm$ 2.43) | 49.64 ($\pm$ 0.59) | 35.29 ($\pm$ 0.75) | 44.29 ($\pm$ 1.88) | 22.42 ($\pm$ 3.23)
KD (Ours) | – | 95.01 ($\pm$ 0.79) | 0.87 ($\pm$ 0.35) | 87.13 ($\pm$ 0.45) | 4.26 ($\pm$ 0.41) | 65.48 ($\pm$ 0.12) | 7.33 ($\pm$ 2.03)
KD w/ QDI (Ours) | – | 95.46 ($\pm$ 0.15) | 0.37 ($\pm$ 0.09) | 88.30 ($\pm$ 0.17) | 2.00 ($\pm$ 0.15) | 66.79 ($\pm$ 0.45) | 7.28 ($\pm$ 0.49)
ICARL [41]∗ | ✓ | 88.99 ($\pm$ 2.13) | 2.63 ($\pm$ 3.48) | – | – | 28.19 ($\pm$ 1.47) | –
A-GEM [7] | ✓ | 87.09 ($\pm$ 0.84) | 12.54 ($\pm$ 1.23) | 63.88 ($\pm$ 1.23) | 29.98 ($\pm$ 1.40) | 22.44 ($\pm$ 0.24) | 55.43 ($\pm$ 0.71)
ER [42] | ✓ | 91.39 ($\pm$ 0.41) | 5.99 ($\pm$ 0.43) | 67.06 ($\pm$ 1.01) | 25.31 ($\pm$ 1.27) | 21.03 ($\pm$ 2.41) | 57.88 ($\pm$ 2.63)
DER [4] | ✓ | 92.45 ($\pm$ 0.26) |  5.79 ($\pm$ 0.20) | 67.74 ($\pm$ 0.88) | 24.77 ($\pm$ 0.94) | 30.97 ($\pm$ 0.14) | 45.25 ($\pm$ 0.48)
DER++ [4] | ✓ | 92.16 ($\pm$ 0.68) |  5.96 ($\pm$ 0.74) | 69.03 ($\pm$ 0.74) | 23.06 ($\pm$ 0.69) | 33.63 ($\pm$ 0.10) | 40.66 ($\pm$ 0.83)
KD w/ Buffer (Ours) | ✓ | 95.81 ($\pm$ 0.03) | 0.73 ($\pm$ 0.34) | 80.35 ($\pm$ 0.53) | 10.76 ($\pm$ 0.65) | 69.07 ($\pm$ 0.26) | 4.37 ($\pm$ 0.17)
Multitask∗ | ✓ | 98.31 ($\pm$ 0.12) | N/A | 93.89 ($\pm$ 0.78) | N/A | 82.04 ($\pm$ 0.10) | N/A
Heterogeneous Continual Learning
Finetune | – | 61.99 ($\pm$ 1.98) | 42.15 ($\pm$ 2.94) | 27.08 ($\pm$ 1.20) | 64.54 ($\pm$ 1.45) | 14.83 ($\pm$ 0.53) | 56.31 ($\pm$ 0.60)
DI [55] | – | 81.16 ($\pm$ 2.77) | 15.84 ($\pm$ 3.01) | 38.33 ($\pm$ 1.44) | 47.08 ($\pm$ 1.37) | 31.69 ($\pm$ 1.99) | 32.42 ($\pm$ 3.63)
KD (Ours) | – | 93.13 ($\pm$ 0.35) | 3.97 ($\pm$ 0.72) | 68.97 ($\pm$ 0.72) | 21.45 ($\pm$ 0.91) | 52.86 ($\pm$ 2.78) | 11.73 ($\pm$ 2.59)
KD w/ QDI (Ours) | – | 90.30 ($\pm$ 1.27) | 6.34 ($\pm$ 0.15) | 76.44 ($\pm$ 1.36) | 12.31 ($\pm$ 0.99) | 53.39 ($\pm$ 1.67) | 13.66 ($\pm$ 1.18)
A-GEM [7] | ✓ | 76.70 ($\pm$ 0.67) | 24.31 ($\pm$ 1.03) | 35.37 ($\pm$ 1.29) | 56.41 ($\pm$ 2.19) | 15.79 ($\pm$ 0.60) | 54.10 ($\pm$ 1.00)
ER [42] | ✓ | 81.78 ($\pm$ 2.15) | 17.57 ($\pm$ 2.86) | 52.59 ($\pm$ 1.03) | 36.41 ($\pm$ 0.54) | 27.82 ($\pm$ 1.43) | 39.66 ($\pm$ 1.24)
DER [4] | ✓ | 79.92 ($\pm$ 0.51) | 19.99 ($\pm$ 0.64) | 53.01 ($\pm$ 0.61) | 37.33 ($\pm$ 0.58) | 25.27 ($\pm$ 1.34) | 40.81 ($\pm$ 0.57)
DER++ [4] | ✓ | 81.33 ($\pm$ 0.76) | 18.79 ($\pm$ 0.75) | 54.81 ($\pm$ 1.65) | 35.20 ($\pm$ 1.63) | 29.74 ($\pm$ 1.40) | 36.79 ($\pm$ 2.16)
KD w/ Buffer (Ours) | ✓ | 93.95 ($\pm$ 0.22) | 2.49 ($\pm$ 0.44) | 73.98 ($\pm$ 0.13) | 16.40 ($\pm$ 0.18) | 54.84 ($\pm$ 0.87) | 10.82 ($\pm$ 0.80)

Table 1: Accuracy and forgetting with task-IL on standard CL and HCL. The best
results are highlighted in bold. $\mathcal{B}$ denotes replay-buffer,
$\mathcal{A}_{T},\mathcal{F}_{T}$ denote average accuracy and forgetting after
the completion of training.∗ denotes the methods whose numbers were used from
Buzzega et al. [4] and $-$ indicates the unavailability of results. All other
experiments are over three independent runs.

Baselines. We compare our designed KD paradigm (Equation 3), with QDI
(Equation 3.2.2) agains standard fine-tuning (Finetune), SI [59], LwF [30],
ICARL [41] A-GEM [7], ER [8], DER and DER++ [4], DI [55] and multi task
training (Multitask), where all tasks are trained jointly with ResNet18 – most
prominent architecture in CL literature. For a fair comparison, we extend our
method with the classification loss using Buffer in Equation 3. The buffer-
size for replay-based method is fixed to $200$ for all the methods that use
exemplar or generative replay. We use the same set of augmentations for
training all the methods.

##### Evaluation.

To evaluate and compare our method in both standard CL and HCL setting, we use
the average accuracy and forgetting as two metrics.

  1. 1.

Average accuracy. It is the average of the accuracy of all tasks after the
completion of training with the sequence of $T$ tasks. More formally, average
accuracy $\mathcal{A}_{T}=\frac{1}{T}\sum_{t=1}^{T}a_{T,t}$, where $a_{i,j}$
is the accuracy of task $j$ after completion of task $i$.

  2. 2.

Average forgetting. It is the averaged difference in accuracy between the
accuracy after training and the maximum accuracy for each task. More formally,
average forgetting $\mathcal{F}_{T}\leavevmode\nobreak\ =\leavevmode\nobreak\
\frac{1}{T-1}\sum_{t=1}^{T}\underset{i\in\\{1,\ldots,T\\}}{\max}\left(a_{i,t}-a_{T,t}\right)$

Model | Task-IL | Class-IL
---|---|---
KD | 83.43 ($\pm$ 0.42) | 21.11 ($\pm$ 0.19)
$+$ AUG-KD | 91.76 ($\pm$ 2.03) | 34.11 ($\pm$ 1.07)
$+$ LS | 93.13 ($\pm$ 0.35) | 30.21 ($\pm$ 0.11)
$+$Warmup | 91.64 ($\pm$ 1.37) | 28.26 ($\pm$ 0.28)
$+$ TI | 90.14 ($\pm$ 1.33) | 22.04 ($\pm$ 0.27)

Table 2:  Ablation showin $\mathcal{A}_{t}$ to measure the effect of various
components with Split-CIFAR10. The inclusion of distillation with augmented
images (aug-kd) and label-smoothing (ls) significantly improve performance,
whereas warmup and task-identity (ti) degrade performance. Blue and red colors
denote the components that were included and excluded from the method
respectively.

Model | Task-IL | Class-IL |
---|---|---|---
CE | 91.06 ($\pm$ 0.63) | 20.99 ($\pm$ 0.27) |
MSE | 89.68 ($\pm$ 0.99) | 20.09 ($\pm$ 0.26) |
KL | 93.13 ($\pm$ 0.35) | 30.21 ($\pm$ 0.11) |

Table 3: Ablation showing $\mathcal{A}_{t}$ using validation set to measure
the effect of different distance metrics. We measure the average accuracy and
forgetting for HCL on Split-CIFAR10 dataset.

Model | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$ |
---|---|---|---
Finetune |  55.47 ($\pm$ 2.06) |  77.68 ($\pm$ 3.68) |
Der++ |  69.05 ($\pm$ 2.56) |  51.99 ($\pm$ 1.91) |
KD (Ours) |  88.77 ($\pm$ 0.55) |  10.95 ($\pm$ 1.34) |
Multitask |  88.71 ($\pm$ 0.40) | – |

Table 4: Transformer analysis. Average accuracy and forgetting for HCL with
ResNet18 to Swin-T with two tasks on CIFAR-10.

Figure 4:  Visualization of QDI and comparison to prior art. Top: prior
feature synthesis from current task data. Bottom: samples for DI that start
from $\mathcal{N}(0,1)$ and QDI from $x_{t}$ in pairs (left – optimization
start point; right – optimization convergence point) for ResNet18 on CIFAR-10
dataset.

###  4.2 Quantitative results

Task-incremental continual learning. First, we compare the performance of our
proposed method with other baselines for task-IL on standard and heterogeneous
continual learning settings. In Table 1, we find that the performance of all
the methods drop significantly by going from the SCL to the HCL setting.
Contrary, our proposed methods achieve the lowest drop across all methods;
moreover, it obtains the best performance with and without buffer across all
considered scenarios. Additionally, we show that in standard CL and HCL, QDI
leads to a $\sim 40\%$ and $\sim 22\%$ absolute improvement in average
accuracy over state-of-the-art data-free methods for Split CIFAR-100 and Tiny-
ImageNet. Interestingly, we find that distillation with synthetic examples
generated by QDI obtains comparable or better performance to the replay buffer
highlighting the flexibility of data-free continual learning.

Moreover, QDI improves synthesis speed by $4\times$ for generating the
synthetic examples in comparison to DI [55] for each task. All variants of our
method achieve the least forgetting in comparison to the baselines across all
datasets for both the standard CL and HCL scenarios. Our proposed training
scheme is especially beneficial when past-task data is unavailable due to
privacy concerns. It allows the adaptation to the new network structure for
the new task by utilizing the saved model checkpoints. It enables training
with different network structures without storing any past-task instances.

##### Class-incremental continual learning.

We also show the validation of our proposed method on class-IL in the
supplementary material, where task-labels are not provided during training or
inference. We observe that all our methods outperform the baselines with and
without buffer in average accuracy and forgetting. Specifically, we show that
ours with QDI and Buffer examples show a five to ten percent improvement in
average accuracy for Split CIFAR-100 and Tiny-ImageNet datasets. However, we
find that buffer is an important component of our method for class-IL; it is
consistent to the observations in prior works [55, 5]. Further, we observe
that the synthetic examples generated by QDI lead to unstable training for
CIFAR-10, but improve the performance for other datasets. We believe this is
an outcome of the model’s dependence on domain over semantics [47] and further
investigation of data-free continual learning techniques for class-IL would be
an interesting direction for future work.

###  4.3 Additional analysis

Ablation studies. To dissect the contribution of different components in our
proposed method for HCL, we isolate and study them on Split CIFAR-10 in Table
2. First, we find that knowledge distillation with augmented instances, in the
contrast to a common setting of precomputed logits, improves the performance
by $\sim 8\%$ and $\sim 13\%$ for task-IL and class-IL respectively due to the
better generalization of current task representations. Next, we observe that
label-smoothing improves the task-IL performance while hurting the class-IL
average accuracy highlighting the presence of a trade-off between both the
settings. Additionally, we measure the effect of warmup and task-identity for
HCL following prior works [22, 30] for standard CL, where we find that warm-up
of the linear head before current-task training and inclusion of task-identity
hurt the performance for HCL.

Further, we conduct an ablation on different distance metrics used for KD in
Equation 3 for HCL. While, a modified version of cross entropy (CE) or mean-
square distance (MSE) are commonly used for knowledge distillation in CL [30,
41, 4], we find the KL-divergence (KL) is more stable and often obtains better
performance in Table 3 for both class-IL and task-IL. We believe that the
ablation of these design choices will provide critical insights towards
building future methods for standard and heterogeneous CL.

Transfer to vision transformer. To show the flexibility of our framework
beyond convolutional architectures, we design a two task experiment with
CIFAR-10 consisting of five classes each. The first task is trained with
ResNet18 [15] followed by Swin-T [32] on the second task. Table 4 shows the
comparison of our proposed method with only KD against Finetune, DER++ trained
with HCL and Multitask learned with CIFAR-10 on Swin-T [32]. First, we remark
that Swin-T (Multitask: $88.71\pm 0.40$) achieves lower accuracy in comparison
to ResNet18 (Multitask: $98.31\pm 0.12$) because ViTs are data-hungry [31].
Second, we show that our proposed method with evolving architectures obtains
performance similar to the multitask baseline. Lastly, our proposed method
obtains better average accuracy and lower forgetting over the evaluated
baselines showing the benefit of training with multiple architectures.

Visualization of generated examples. We visualize the generated examples by DI
and QDI in the HCL setting for ResNet18 architecture with CIFAR-10 dataset in
Figure 4. To explain the interplay of the generated examples initialized with
the current task instances and the past task examples, we consider the example
of a current task (e.g., a cat or bird) to invert the visual features of the
past task (car). We observe that the input-features swiftly generated by QDI
have a combination of both classes and superior in realism; in contrast, DI
only contains the past-task features and requires more optimization steps from
random noise for full convergence. We believe this interpolation also helps in
current task-training while also alleviating catastrophic forgetting.

##  5 Conclusion

In this work, we propose a novel continual learning setting named
_Heterogeneous Continual Learning (HCL)_ , where the lifelong learner learns
on a sequence of tasks while adopting state-of-the-art deep-learning
techniques and architectures. We benchmark the state-of-the-art CL solutions
in this new setting and observe a large degradation in performance. To tackle
this limitation, we revisit knowledge distillation and propose a modified
paradigm inspired by the recent advances in knowledge distillation.
Additionally, we propose _Quick Deep Inversion (QDI)_ that generates synthetic
examples using current task instances to enhance knowledge distillation
performance in the data-free continual learning setup at a negligible
additional cost. Our experimental evaluation shows that without tuning the
training configurations, we achieve a significant improvement for both the
standard and heterogeneous continual learning scenarios across all datasets,
highlighting our proposed approach’s efficacy in real-world applications.

Limitations. Although the proposed approach is applicable for general deep-
learning architectures, we found some limitations with the current
approach/setup. First, in some cases, the recent architectures can be sub-
optimal and degrade the model performance due to their training configuration,
model size, or hyper-parameter mismatch. We note that in this work, we did not
tune the training configuration or hyper-parameters for each model for fair
comparisons. However, model specific hyper-parameter tuning can further
improve model performance. Second, our work is not yet applicable to
unsupervised CL with heterogeneous architectures, where the learner is
expected to learn on unlabelled data with changing architectures because model
inversion and KD with unsupervised CL is not a trivial extension due to the
absence of class-dependent information to generate class-conditioned examples.

## References

  * Ahn et al. [2019] Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon.  Uncertainty-based continual learning with adaptive regularization.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Aljundi et al. [2019] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio.  Gradient based sample selection for online continual learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Beyer et al. [2022] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Buzzega et al. [2020] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara.  Dark experience for general continual learning: a strong, simple baseline.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Carta et al. [2022] Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu.  Ex-model: Continual learning from a stream of trained models.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , 2022.
  * Chandrasegaran et al. [2022] Keshigeyan Chandrasegaran, Ngoc-Trung Tran, Yunqing Zhao, and Ngai-Man Cheung.  Revisiting label smoothing and knowledge distillation compatibility: What was missing?  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2022.
  * Chaudhry et al. [2019a] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019a.
  * Chaudhry et al. [2019b] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato.  Continual learning with tiny episodic memories.  _arXiv preprint arXiv:1902.10486_ , 2019b.
  * Cossu et al. [2022] Andrea Cossu, Gabriele Graffieti, Lorenzo Pellegrini, Davide Maltoni, Davide Bacciu, Antonio Carta, and Vincenzo Lomonaco.  Is class-incremental enough for continual learning?  _Frontiers in Artificial Intelligence_ , 2022.
  * Dai et al. [2020] Xiaoliang Dai, Hongxu Yin, and Niraj K Jha.  Incremental learning using a grow-and-prune paradigm with efficient neural networks.  _IEEE Transactions on Emerging Topics in Computing_ , 2020.
  * De Lange et al. [2021] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009.
  * Dhar et al. [2019] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa.  Learning without memorizing.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hou et al. [2019] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Hu et al. [2018] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Ioffe & Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2015.
  * Jung et al. [2016] Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim.  Less-forgetting learning in deep neural networks.  _arXiv preprint arXiv:1607.00122_ , 2016.
  * Krizhevsky [2012] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  _University of Toronto_ , 05 2012.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2012.
  * LeCun et al. [1989] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural Computation_ , 1989.
  * LeCun [1998] Yann LeCun.  The mnist database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_ , 1998.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * Lee et al. [2021] Seungwon Lee, Sima Behpour, and Eric Eaton.  Sharing less is more: Lifelong learning in deep networks with selective layer transfer.  In _Proceedings of the 38th International Conference on Machine Learning_ , 2021.
  * Li et al. [2019] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2019.
  * Li & Hoiem [2016] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016.
  * Liu et al. [2021a] Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco De Nadai.  Efficient training of visual transformers with small-size datasets.  _arXiv preprint arXiv:2106.03746_ , 2021a.
  * Liu et al. [2021b] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021b.
  * Madaan et al. [2022] Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang.  Representational continuity for unsupervised continual learning.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2022.
  * McGrath et al. [2021] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik.  Acquisition of chess knowledge in alphazero.  _arXiv preprint arXiv:2111.09259_ , 2021.
  * Mehta et al. [2021] Sachin Mehta, Marjan Ghazvininejad, Srinivasan Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi.  Delight: Deep and light-weight transformer.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Mirzadeh et al. [2022a] Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Wide neural networks forget less catastrophically.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2022a.
  * Mirzadeh et al. [2022b] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Wide neural networks forget less catastrophically.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2022b.
  * Mirzadeh et al. [2022c] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Architecture matters in continual learning.  _arXiv preprint arXiv:2202.00275_ , 2022c.
  * Müller et al. [2019] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  _Neurips_ , 2019.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2021.
  * Rebuffi et al. [2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Rolnick et al. [2019] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Rusu et al. [2016] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Schwarz et al. [2018] Jonathan Schwarz, Jelena Luketina, Wojciech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell.  Progress & compress: A scalable framework for continual learning.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2018.
  * Shen et al. [2021] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides.  Is label smoothing truly incompatible with knowledge distillation: An empirical study.  In _iclr_ , 2021.
  * Simonyan & Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015.
  * Smith et al. [2021] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Always be dreaming: A new approach for data-free class-incremental learning.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Proceedings of the AAAI National Conference on Artificial Intelligence (AAAI)_ , 2017.
  * Tan & Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2019.
  * Thrun [1996] Sebastian Thrun.  Is learning the n-th thing any easier than learning the first?  In _Neurips_ , 1996.
  * Tolstikhin et al. [2021] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  Mlp-mixer: An all-mlp architecture for vision, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 30, 2017.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Xu et al. [2022] Jing Xu, Yu Pan, Xinglin Pan, Steven Hoi, Zhang Yi, and Zenglin Xu.  Regnet: self-regulated network for image classification.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022\.
  * Yin et al. [2020] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz.  Dreaming to distill: Data-free knowledge transfer via DeepInversion.  In _Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Yoon et al. [2018] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2018.
  * Yoon et al. [2020] Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang.  Scalable and order-robust continual learning with additive parameter decomposition.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * Zagoruyko & Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2017.

##### Organization.

In the supplementary material, we provide the implementation details and
hyper-parameter configurations in Appendix A. Further, we show the results for
class-IL continual learning and additional visualization of examples generated
by QDI in Table B.5.

##  Appendix A Experimental Details

We follow the base hyper-parameter setup from Buzzega et al. [4] for SCL and
HCL experiments. We use an SGD optimizer for experiments with base learning
rate as $0.03$ for all the models. For each new task a total of $200$ epochs
are utilized to train the current model and we use the average accuracy metric
on the validation set to store the best-model for evaluation for each task.
Batch size is set as $32$ and training is conducted on one NVIDIA V100 GPU of
$16$G for CIFAR-10 experiment and $32$G for Split CIFAR-100 and Tiny-ImageNet
datasets. For knowledge distillation methods, we use $\alpha$ and $\beta$
equal to $1.0$ for Split CIFAR-10, $3.0$ for Split CIFAR-100 and Tiny-ImageNet
and $\beta$ equal to $0.3$ with buffer. QDI previous class target labels are
evenly sampled across previous tasks. Optimization hyper-parameters for QDI
are provided next for three datasets, each shared across all networks during
the training run:

  * •

Split CIFAR-10. Optimization using Adam optimizer of learning rate $0.005$,

$\alpha_{tv}=0.001,\alpha_{\ell_{2}}=0,\alpha_{\text{feature}}=0.1$

for $0.5$K iterations.$d(\cdot,\cdot)$ is MSE loss.

  * •

Split CIFAR-100. Optimization using Adam with learning rate $0.03$,

$\alpha_{tv}=0.003,\alpha_{\ell_{2}}=0.003,\alpha_{\text{feature}}=0.2$

for $0.5$K iterations. $d(\cdot,\cdot)$ is MSE loss.

  * •

Split Tiny-ImageNet. Optimization based on Adam optimizer of learning rate
$0.05$,

$\alpha_{tv}=0.001,\alpha_{\ell_{2}}=0.05,\alpha_{\text{feature}}=0.5$

for $0.5$K iterations. $d(\cdot,\cdot)$ is MSE loss.

For each dataset we found the QDI hyper-parameters based on a validation set
obtainend by randomly sampling $10\%$ of the training set. All results in main
paper are on the test set, with $3$ independent runs of the hyperset from
random seeds for means and standard deviations.

##  Appendix B Additional Results

Class-IL continual learning. Table B.5 shows the results for class-IL
continual learning.

Method | $\mathcal{B}$ | Split CIFAR-10 | Split CIFAR-100 | Split Tiny-ImageNet
---|---|---|---|---
|  | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$ | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$ | $\mathcal{A}_{T}\leavevmode\nobreak\ (\uparrow)$ | $\mathcal{F}_{T}\leavevmode\nobreak\ (\downarrow)$
Standard Continual Learning
Finetune | – | 19.60 ($\pm$ 0.03) | 96.66 ($\pm$ 0.12) | 6.93 ($\pm$ 1.13) | 60.53 ($\pm$ 1.11) | 6.78 ($\pm$ 0.14) | 40.67 ($\pm$ 0.92)
DI [55] | – | 22.72 ($\pm$ 1.02) | 77.65 ($\pm$ 0.30) | 7.21 ($\pm$ 0.75) | 63.90 ($\pm$ 2.87) | 7.28 ($\pm$ 2.05) | 51.70 ($\pm$ 0.63)
SI [59]∗ | – | 19.48 ($\pm$ 0.17) | 95.78 ($\pm$ 0.64) | – | – | 6.58 ($\pm$ 0.31) | –
LwF [30]∗ | – | 19.61 ($\pm$ 0.05) | 96.69 ($\pm$ 0.25) | – | – | 8.46 ($\pm$ 0.22) | –
KD (Ours) | – | 22.73 ($\pm$ 0.75) | 87.00 ($\pm$ 2.29) | 16.77 ($\pm$ 0.63) | 80.65 ($\pm$ 0.99) | 20.86 ($\pm$ 0.14) | 42.38 ($\pm$ 0.95)
KD w/ QDI (Ours) | – | 19.75 ($\pm$ 0.03) | 0.08 ($\pm$ 0.02) | 22.53 ($\pm$ 1.15) | 16.50 ($\pm$ 2.72) | 24.21 ($\pm$ 0.52) | 36.58 ($\pm$ 1.56)
ICARL [41]∗ | ✓ | 49.02 ($\pm$ 3.20) | 28.72 ($\pm$ 0.49) | – | – | 7.53 ($\pm$ 0.79) | –
A-GEM [7] | ✓ | 21.98 ($\pm$ 0.56) | 92.18 ($\pm$ 1.98) | 5.04 ($\pm$ 0.12) | 91.93 ($\pm$ 0.22) | 7.49 ($\pm$ 0.12) | 72.04 ($\pm$ 0.34)
ER [42] | ✓ | 48.56 ($\pm$ 1.40) | 58.11 ($\pm$ 1.61) | 9.65($\pm$ 0.95) | 85.20 ($\pm$ 1.27) | 10.70 ($\pm$ 0.27) | 83.99 ($\pm$ 0.18)
DER [4] | ✓ | 66.08 ($\pm$ 1.18) | 27.40 ($\pm$ 2.16) | 19.01 ($\pm$ 0.74) | 65.06 ($\pm$ 0.15) | 10.01 ($\pm$ 1.52) | 65.66 ($\pm$ 3.60)
DER++ [4] | ✓ | 67.23 ($\pm$ 1.36) | 26.13 ($\pm$ 0.28) | 21.38 ($\pm$ 0.48) | 56.17 ($\pm$ 3.37) | 8.23 ($\pm$ 0.31) | 68.51 ($\pm$ 1.17)
KD w/ Buffer (Ours) | ✓ | 70.34 ($\pm$ 1.07) | 16.32 ($\pm$ 0.18) | 25.19 ($\pm$ 0.15) | 40.53 ($\pm$ 0.73) | 29.21 ($\pm$ 0.74) | 16.48 ($\pm$ 0.70)
Multitask∗ | – | 92.20 ($\pm$ 0.15) | N/A | 70.32 ($\pm$ 0.48) | N/A | 59.99 ($\pm$ 0.19) | N/A
Heterogeneous Continual Learning
Finetune | – | 21.45 ($\pm$ 0.75) | 91.24 ($\pm$ 2.33) | 5.27 ($\pm$ 0.23) | 72.99 ($\pm$ 1.94) | 7.90 ($\pm$ 0.13) | 57.23 ($\pm$ 0.11)
DI [55] | – | 20.67 ($\pm$ 1.10) | 70.97 ($\pm$ 2.13) | 6.20 ($\pm$ 1.40) | 73.70 ($\pm$ 1.49) | 6.39 ($\pm$ 0.60) | 51.14 ($\pm$ 0.87)
KD (Ours) | – | 30.21 ($\pm$0.11) | 27.54 ($\pm$ 1.91) | 12.94 ($\pm$ 1.13) | 58.89 ($\pm$ 1.64) | 14.18 ($\pm$ 0.35) | 46.91 ($\pm$ 0.59)
KD w/ QDI (Ours) | – | 33.89 ($\pm$ 3.53) | 31.73 ($\pm$ 3.51) | 15.86 ($\pm$1.51) | 32.73 ($\pm$0.77) | 15.38 ($\pm$1.67) | 37.27 ($\pm$1.31)
ER [7] | ✓ | 38.77 ($\pm$ 1.99) | 69.61 ($\pm$ 2.20) | 7.43 ($\pm$ 0.36) | 82.85 ($\pm$ 0.20) | 7.59 ($\pm$ 0.12) | 61.77 ($\pm$ 0.31)
A-GEM [7] | ✓ | 19.67 ($\pm$ 0.41) | 89.26 ($\pm$ 2.89) | 4.62 ($\pm$ 0.02) | 88.78 ($\pm$ 0.84) | 6.93 ($\pm$ 0.11) | 63.93 ($\pm$ 0.73)
DER [4] | ✓ | 44.13 ($\pm$ 0.98) | 57.64 ($\pm$ 4.76) | 10.11 ($\pm$ 0.65) | 80.92 ($\pm$ 1.35) | 8.11 ($\pm$ 0.26) | 56.73 ($\pm$ 0.81)
DER++ [4] | ✓ | 48.82 ($\pm$ 1.75) | 50.11 ($\pm$ 2.74) | 10.97 ($\pm$ 0.55) | 73.62 ($\pm$ 0.86) | 8.88 ($\pm$ 0.61) | 55.54 ($\pm$ 2.13)
KD w/ Buffer (Ours) | ✓ | 65.40 ($\pm$ 0.96) | 10.13 ($\pm$ 2.19) | 21.00 ($\pm$ 1.01) | 38.49 ($\pm$ 1.39) | 18.77 ($\pm$ 0.91) | 8.76 ($\pm$ 0.79)

Table B.5: Accuracy and forgetting with class-IL on standard CL and HCL. The
best results are highlighted in bold. $\mathcal{B}$ denotes replay-buffer,
$\mathcal{A}_{T},\mathcal{F}_{T}$ denote average accuracy and forgetting after
the completion of training.∗ denotes the methods whose numbers were used from
Buzzega et al. [4] and $-$ indicates the unavailability of results. All other
experiments are over three independent runs.

Visualization of synthesized examples. We provide extra visualization of QDI
samples for the CIFAR-100 and Tiny-ImageNet datasets as in Fig. B.5. It can be
observed that the proposed method scales across datasets with high fidelity in
synthesized samples. More interestingly, the optimization step can find very
close proxy of the current task semantics in older domains and dream out
visual features of high perceptual realism.

|  |  |
---|---|---|---
starting | synthesized | starting | synthesized
CIFAR-100 | Tiny-ImageNet

Figure B.5: More QDI visualization for CIFAR-100 and Tiny-ImageNet datasets.
Best viewed in color.