# Forget-free Continual Learning with Soft-Winning SubNetworks

Haeyong Kang, Jaehong Yoon, Sultan Rizky Madjid, Sung Ju Hwang, and Chang D.
Yoo∗ Email<EMAIL_ADDRESS>Corresponding Author.

###### Abstract

Inspired by _Regularized Lottery Ticket Hypothesis (RLTH)_ , which states that
competitive smooth (non-binary) subnetworks exist within a dense network in
continual learning tasks, we investigate two proposed architecture-based
continual learning methods which sequentially learn and select adaptive
binary- (WSN) and non-binary Soft-Subnetworks (SoftNet) for each task. WSN and
SoftNet jointly learn the regularized model weights and task-adaptive non-
binary masks of subnetworks associated with each task whilst attempting to
select a small set of weights to be activated (winning ticket) by reusing
weights of the prior subnetworks. Our proposed WSN and SoftNet are inherently
immune to catastrophic forgetting as each selected subnetwork model does not
infringe upon other subnetworks in Task Incremental Learning (TIL). In TIL,
binary masks spawned per winning ticket are encoded into one N-bit binary
digit mask, then compressed using Huffman coding for a sub-linear increase in
network capacity to the number of tasks. Surprisingly, in the inference step,
SoftNet generated by injecting small noises to the backgrounds of acquired WSN
(holding the foregrounds of WSN) provides excellent forward transfer power for
future tasks in TIL. SoftNet shows its effectiveness over WSN in regularizing
parameters to tackle the overfitting, to a few examples in Few-shot Class
Incremental Learning (FSCIL).

###### Index Terms:

Continual Learning (CL), Task Incremental Learning (TIL), Few-shot Class
Incremental Learning (FSCIL), Regularized Lottery Ticket Hypothesis (RLTH),
Wining SubNetworks (WSN), Soft-Subnetwork (SoftNet)

##  1 Introduction

Continual Learning (CL), also known as Lifelong Learning [69, 61, 82, 23], is
a learning paradigm where a series of tasks are learned sequentially. The
principle objective of continual learning is to replicate human cognition,
characterized by the ability to learn new concepts or skills incrementally
throughout one’s lifespan. An optimal continual learning system could
facilitate a positive forward and backward transfer, leveraging the knowledge
gained from previous tasks to solve new ones while also updating its
understanding of previous tasks with the new knowledge. However, building
successful continual learning systems is challenging due to the occurrence of
catastrophic forgetting or catastrophic interference [53], a phenomenon where
the performance of the model on previous tasks significantly deteriorates when
it learns new tasks. This can make it challenging to retain the knowledge
acquired from previous tasks, ultimately leading to a decline in overall
performance. To tackle the catastrophic forgetting problem in continual
learning, numerous approaches have been proposed, which can be broadly
classified as follows: (1) Regularization-based methods [38, 7, 34, 70, 55]
aim to keep the learned information of past tasks during continual training
aided by sophisticatedly designed regularization terms, (2) Rehearsal-based
methods [59, 60, 8, 9, 62] utilize a set of real or synthesized data from the
previous tasks and revisit them, and (3) Architecture-based methods [49, 64,
44, 72, 36, 37] propose to minimize the inter-task interference via newly
designed architectural components.

|  |  |
---|---|---|---
(a) Fixed Backbone (Piggyback, SupSup) |  (b) Biased Transfer (PackNet, CLNP) |  (c) Selective Reuse Expansion beyond Dense Network (APD) |  (d) Selective Reuse Expansion within Network (WSN)
Figure 1: Concept Comparison: (a) Piggyback [49], and SupSup [72] find the
optimal binary mask on a fixed backbone network a given task (b) PackNet [50]
and CLNP [19] forces the model to reuse all features and weights from previous
subnetworks which causes bias in the transfer of knowledge (c) APD [78]
selectively reuse and dynamically expand the dense network (d) Our WSN
selectively reuse and dynamically expand subnetworks within a dense network.
Green edges are reused weights.

Despite the remarkable success of recent works on rehearsal- and architecture-
based continual learning, a majority of recent methods request external memory
as new tasks arrive, making the model difficult to scale to larger and more
complex tasks. Rehearsal-based CL requires additional storage to store the
replay buffer or generative models, and architecture-based methods leverage
additional model capacity to account for new tasks. These trends lead to an
essential question: how can we build a memory-efficient CL model that does not
exceed the backbone network’s capacity or even requires a much smaller
capacity? Several studies have shown that deep neural networks are over-
parameterized [17, 21, 43] and thus removing redundant/unnecessary weights can
achieve on-par or even better performance than the original dense network.
More recently, Lottery Ticket Hypothesis (LTH) [18] demonstrates the existence
of sparse subnetworks, named _winning tickets_ , that preserve the performance
of a dense network. However, searching for optimal winning tickets during
continual learning with iterative pruning methods requires repetitive pruning
and retraining for each arriving task, which could be more practical.
Furthermore, leveraged by Regularized Lottery Ticket Hypothesis (RLTH) [37],
subnetworks exemplified that it could overfit a few task data, potentially
limiting their effectiveness on new tasks or datasets.

To tackle the issues of external replay buffer, capacity, and over-fitting, we
suggest a novel regularized CL method which finds the high-performing
Regularized Winning SubNetwork referred to as Soft-Subnetwork (SoftNet) [37]
given tasks without the need for retraining and rewinding. As a baseline of
SoftNet, non-regularized Winning Subnetworks referred to as WSN [36] are
restated, as shown in Figure 1 (d). Also, we set previous pruning-based CL
approaches [49, 72] (see Figure 1 (a)) to baselines of architectures, which
obtain task-specific subnetworks given a pre-trained backbone network. Our WSN
incrementally learns model weights and task-adaptive binary masks (the
subnetworks) within the neural network. To allow the forward transfer when a
model learns on a new task, we reuse the learned subnetwork weights for the
previous tasks, however selectively, as opposed to using all the weights [50]
(see Figure 1 (b)), that may lead to biased transfer. Further, the WSN
eliminates the threat of catastrophic forgetting during continual learning by
freezing the subnetwork weights for the previous tasks and does not suffer
from the negative transfer, unlike [80] (see Figure 1 (c)), whose subnetwork
weights for the previous tasks can be updated when training on the new tasks.

LTH often leverages the magnitudes of the weights as a pruning criterion to
find the optimal subnetworks. However, in CL, relying only on the weight
magnitude may be ineffective since the weights are shared across classes, and
thus training on the new tasks will change the weights trained for previous
tasks (reused weights). The update of reused weights will trigger an avalanche
effect where weights selected to be part of the subnetworks for later tasks
will always be better in the eyes of the learner, which will result in the
catastrophic forgetting of the knowledge for the prior tasks. Thus, in CL, the
learner must train on the new tasks without changing the reused weights. To
find the optimal subnetworks, we decouple the information of the learning
parameter and the network structure into two separate learnable parameters,
namely, _weights_ and _weight scores_. The weight scores are binary masks with
the same shapes as the weights. Now, subnetworks are found by selecting the
weights with the top-$k$ percent weight ranking scores. More importantly,
decoupling the weights and the structure allows us to find the optimal
subnetwork online without iterative retraining, pruning, and rewinding. There
is one more thing to consider to find the optimal subnetworks induced by
binary masks in CL. According to CL tasks, the subnetworks tend to overfit the
few samples, i.e., Few-Shot Class Incremental Learning (FSCIL). Therefore, we
could find regularized subnetworks yielded by smooth (soft) masks. To this
end, the proposed methods are designed to jointly learn the weights and the
structure of the optimal regularized subnetworks, whose overall size is
smaller than a dense network.

Our contributions can be summarized as follows:

  * •

Inspired by Regularized Lottery Ticket Hypothesis (RLTH), we propose novel
forget-free continual learning methods referred to as WSN and SoftNet, which
learn a compact subnetwork for each task while keeping the weights selected by
the previous tasks intact.

  * •

Our proposed WSN and SoftNet do not perform explicit pruning for learning the
subnetwork. Our methods eliminate catastrophic forgetting and enable forward
transfer from previous tasks to new ones in Task Incremental Learning (TIL).

  * •

Our WSN obtains compact subnetworks using Huffman coding with a sub-linear
increase in the network capacity, outperforming existing continual learning
methods regarding accuracy-capacity trade-off and forward / backward transfer
in TIL.

  * •

Our SoftNet trains two different types of subnetworks for solving the FSCIL
problem, alleviating the continual learner from forgetting previous sessions
and overfitting simultaneously, outperforming strong baselines on public
benchmark tasks.

##  2 Related Works

Continual Learning. Continual learning [53, 69, 41, 45], also known as
lifelong learning, is the challenge of learning a sequence of tasks
continuously while utilizing and preserving previously learned knowledge to
improve performance on new tasks. Several major approaches have been proposed
to tackle the challenges of continual learning, such as catastrophic
forgetting. One such approach is regularization-based methods [38, 7, 34, 70,
55], which aim to reduce catastrophic forgetting by imposing regularization
constraints that inhibit changes to the weights or nodes associated with past
tasks. Rehearsal-based approaches [59, 8, 9, 62, 16] store small data
summaries to the past tasks and replay them during training to retain the
acquired knowledge. Some methods in this line of work [66, 2] accommodate the
generative model to construct the pseudo-rehearsals for previous tasks.
Architecture-based approaches [49, 64, 44, 72, 36, 37] use the additional
capacity to expand [75, 80] or isolate [61] model parameters, preserving
learned knowledge and preventing forgetting. Both rehearsal and architecture-
based methods have shown remarkable efficacy in suppressing catastrophic
forgetting but require additional capacity for the task-adaptive parameters
[72] or the replay buffers.

Pruning-based Continual Learning. While most works aim to increase the
performance of continual learners by adding memory, some researchers have
focused on building memory and computationally efficient continual learners by
using pruning-based constraints. CLNP [19] is one example of a method that
selects important neurons for a given task using $\ell_{1}$ regularization to
induce sparsity and freezes them to maintain performance. Neurons that are not
selected are reinitialized for future task training. Another method, Piggyback
[49], trains task-specific binary masks on the weights given a pre-trained
model. However, this method does not allow for knowledge transfer among tasks,
and its performance highly depends on the quality of the backbone model. HAT
[64] proposes task-specific learnable attention vectors to identify important
weights per task. The masks are formulated to layerwise cumulative attention
vectors during continual learning. A recent method, LL-Tickets [11], shows a
sparse subnetwork called lifelong tickets that performs well on all tasks
during continual learning. The method searches for more prominent tickets from
current ones if the obtained tickets cannot sufficiently learn the new task
while maintaining performance on past tasks. However, LL-Tickets require
external data to maximize knowledge distillation with learned models for prior
tasks, and the ticket expansion process involves retraining and pruning steps.
WSN [36] is another method that jointly learns the model and task-adaptive
binary masks on subnetworks associated with each task. It attempts to select a
small set of weights activated (winning ticket) by reusing weights of the
prior subnetworks.

Soft-subnetwork. Recent studies have shown that context-dependent gating of
sub-spaces [25], parameters [50, 26, 52], or layers [63] of a single deep
neural network is effective in addressing catastrophic forgetting during
continual learning. Moreover, combining context-dependent gating with
constraints that prevent significant changes in model weights, such as SI [82]
and EWC [38], can lead to further performance improvements, as shown by Masse
et al. [51]. Flat minima, which can be seen as acquiring sub-spaces, have also
been proposed to address catastrophic forgetting. Previous studies have
demonstrated that a flat minimizer is more robust to random perturbations [29,
30, 33]. In a recent study by Shi et al. [65], obtaining flat loss minima in
the base session, which refers to the first task session with a sufficient
number of training instances, was found to be necessary to alleviate
catastrophic forgetting in Few-Shot Class Incremental Learning (FSCIL). They
achieved this by shifting the model weights on the obtained flat loss contour.
Our work investigates the performance of two proposed architecture-based
continual learning methods: WSN / Soft-Subnetworks (SoftNet). We select sub-
networks [18, 46, 81, 85, 71, 58, 36, 14] and optimize the regularized sub-
network parameters in a sub-space [37] in Task Incremental Learning (TIL) and
Few-Shot Class Incremental Learning (FSCIL) settings.

##  3 Soft-Winning SubNetworks

In this section, we present our pruning-based continual learning methods,
Winning SubNetworks (WSN) [36] and Soft-Winning SubNetworks (SoftNet) [37]. In
WSN, the neural network searches for the task-adaptive winning tickets and
updates only the weights that have not been trained on the previous tasks.
After training on each task, the subnetwork parameters of the model are frozen
to ensure that the proposed method is inherently immune to catastrophic
forgetting. The WSN method is designed to selectively transfer previously
learned knowledge to future tasks (i.e., forward transfer), which can
significantly reduce the training time needed for convergence during
sequential learning. This feature is especially critical for large-scale
learning problems where a continual learner trains on multiple tasks
sequentially, leading to significant time and computational savings.
Originally, SoftNet was proposed to address the issues of forgetting previous
sessions and overfitting a few samples of new sessions. To achieve this, the
method trains two types of subnetworks concurrently.

|  |  |
---|---|---|---
(a) selected weights $\hat{\bm{\theta}}_{t-1}$ at prior task |  (b) forward pass using reused weights |  (c) backward pass only on non-used weights |  (d) selected weights $\hat{\bm{\theta}}_{t}$ with subsets of reused weights
Figure 2: An illustration of Winning SubNetworks (WSN): (a) The top-c% weights
$\hat{\bm{\theta}}_{t-1}$ at prior task are obtained, (b) In the forward pass
of a new task, WSN reuses weights selected from prior tasks, (c) In the
backward pass, WSN updates only non-used weights, and (d) after several
iterations of (b) and (c), we acquire again the top-c% weights
$\hat{\bm{\theta}}_{t}$ including subsets of reused weights for the new task.

Problem Statement. Consider a supervised learning setup where $T$ tasks arrive
to a learner sequentially. We denote that
$\mathcal{D}_{t}=\\{\textbf{x}_{i,t},y_{i,t}\\}_{i=1}^{n_{t}}$ is the dataset
of task $t$, composed of $n_{t}$ pairs of raw instances and corresponding
labels. We assume a neural network $f(\cdot;\bm{\theta})$, parameterized by
the model weights $\bm{\theta}$ and standard continual learning scenario aims
to learn a sequence of tasks by solving the following optimization procedure
at each step $t$:

$\bm{\theta}^{\ast}=\operatorname*{minimize}_{\bm{\theta}}\frac{1}{n_{t}}\sum^{n_{t}}_{i=1}\mathcal{L}(f(\textbf{x}_{i,t};\bm{\theta}),y_{i,t}),$
(1)

where $\mathcal{L}(\cdot,\cdot)$ is a classification objective loss such as
cross-entropy loss. $\mathcal{D}_{t}$ for task $t$ is only accessible when
learning task $t$. Note rehearsal-based continual learning methods allow
memorizing a small portion of the dataset to replay. We further assume that
task identity is given in the training and testing stages.

Continual learners frequently use over-parameterized deep neural networks to
ensure enough capacity for learning future tasks. This approach often leads to
the discovery of subnetworks that perform as well as or better than the
original network. Given the neural network parameters $\bm{\theta}$, the
binary attention mask $\mathbf{m}^{*}_{t}$ that describes the optimal
subnetwork for task $t$ such that $|\mathbf{m}^{*}_{t}|$ is less than the
model capacity $c$ follows as:

$\begin{split}\mathbf{m}^{*}_{t}&=\underset{\mathbf{m}_{t}\in\\{0,1\\}^{|\bm{\theta}|}}{\operatorname*{minimize}}\frac{1}{n_{t}}\sum^{n_{t}}_{i=1}\mathcal{L}\big{(}f(\textbf{x}_{i,t};\bm{\theta}\odot\mathbf{m}_{t}),y_{i,t}\big{)}-\mathcal{J}\\\
&\quad\quad\quad\quad\quad\quad\text{subject to\leavevmode\nobreak\
}|\mathbf{m}^{*}_{t}|\leq c,\end{split}$ (2)

where task loss
$\mathcal{J}=\mathcal{L}\big{(}f(\textbf{x}_{i,t};\bm{\theta}),y_{i,t}\big{)}$
and $c\ll|\bm{\theta}|$ (used as the selected proportion $\%$ of model
parameters in the following section). In the optimization section, we describe
how to obtain $\mathbf{m}^{*}_{t}$ using a single learnable weight score
$\mathbf{s}$ that is subject to updates while minimizing task loss jointly for
each task.

###  3.1 Winning SubNetworks (WSN)

Let each weight be associated with a learnable parameter we call weight score
$\mathbf{s}$, which numerically determines the importance of the weight
associated with it; that is, a weight with a higher weight score is seen as
more important. We find a sparse subnetwork $\hat{\bm{\theta}}_{t}$ of the
neural network and assign it as a solver of the current task $t$. We use
subnetworks instead of the dense network as solvers for two reasons: (1)
Lottery Ticket Hypothesis [18] shows the existence of a subnetwork that
performs as well as the whole network, and (2) subnetwork requires less
capacity than dense networks, and therefore it inherently reduces the size of
the expansion of the solver.

Motivated by such benefits, we propose a novel Winning SubNetworks (WSN111WSN
code is available at https://github.com/ihaeyong/WSN.git), which is the joint-
training method for continual learning that trains on task - while selecting
an important subnetwork given the task $t$ as shown in Fig. 2. The
illustration of WSN explains how to acquire binary weights within a dense
network step by step. We find $\hat{\bm{\theta}}_{t}$ by selecting the $c$%
weights with the highest weight scores $\mathbf{s}$, where $c$ is the target
layerwise capacity ratio in %. A task-dependent binary weight represents the
selection of weights $\mathbf{m}_{t}$ where a value of $1$ denotes that the
weight is selected during the forward pass and $0$ otherwise. Formally,
$\mathbf{m}_{t}$ is obtained by applying a indicator function
$\mathbbm{1}_{c}$ on $\mathbf{s}$ where $\mathbbm{1}_{c}(s)=1$ if $\mathbf{s}$
belongs to top-$c\%$ scores and $0$ otherwise. Therefore, the subnetwork
$\hat{\bm{\theta}}_{t}$ for task $t$ is obtained by
$\hat{\bm{\theta}}_{t}=\bm{\theta}\odot\mathbf{m}_{t}$.

###  3.2 Soft-Subnetworks (SoftNet)

Several works have addressed overfitting issues in continual learning from
different perspectives, including NCM [31], BiC [73], OCS [79], and FSLL [52].
To mitigate the overfitting issue in subnetworks, we use a simple yet
efficient method named _SoftNet_ proposed by [37]. The following new paradigm,
referred to as _Regularized Lottery Ticket Hypothesis_ [37] which is inspired
by the _Lottery Ticket Hypothesis_ [18] has become the cornerstone of SoftNet:

Regularized Lottery Ticket Hypothesis (RLTH). A randomly-initialized dense
neural network contains a regularized subnetwork that can retain the prior
class knowledge while providing room to learn the new class knowledge through
isolated training of the subnetwork.

SoftNet. Based on RLTH, we propose a method, referred to as Soft-SubNetworks
(SoftNet222SoftNet code is available at https://github.com/ihaeyong/SoftNet-
FSCIL.git). SoftNet jointly learns the randomly initialized dense model, and
soft mask $\bm{m}\in[0,1]^{|\bm{\theta}|}$ on Soft-subnetwork on each task
training; the soft mask consists of the major part of the model parameters
$m=1$ and the minor ones $m<1$ where $m=1$ is obtained by the top-$c\%$ of
model parameters and $m<1$ is obtained by the remaining ones
($100-\text{top-}c\%$) sampled from the uniform distribution $U(0,1)$. Here,
it is critical to select minor parameters $m<1$ in a given dense network.

(a) epoch=70, $c=10.0\%$ | (b) epoch=100, $c=10.0\%$
---|---
Figure 3: Loss landscapes of Dense Network, HardNet (WSN), and SoftNet:
Subnetworks provide a more flat global minimum than dense neural networks. To
demonstrate the loss landscapes, we trained a simple three-layered, fully
connected model (fc-4-25-30-3) on the Iris Flower dataset (which is three
classification problem) for 100 epochs [22, 15].

###  3.3 Convergence of Subnetworks

Convergences of HardNet (WSN) / SoftNet. To interpret the convergence of
SoftNet, we follow the Lipschitz-continuous objective gradients [5, 4]: the
objective function of dense networks $R:\mathbb{R}^{d}\rightarrow\mathbb{R}$
is continuously differentiable and the gradient function of $R$, namely,
$\nabla R:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$, Lipschitz continuous with
Lipschitz constant $L>0$, i.e.,

$\begin{split}||\nabla R(\bm{\theta})-\nabla
R(\bm{\theta}^{\prime})||_{2}&\leq
L||\bm{\theta}-\bm{\theta}^{\prime}||\;\;\\\ &\text{ for all
}\\{\bm{\theta},\bm{\theta}^{\prime}\\}\subset\mathbb{R}^{d}.\end{split}$ (3)

Proposition. Subnetwork achieves a faster rate than dense networks. To prove
this, following the same formula, we define the Lipschitz-continuous objective
gradients of subnetworks as follows:

$\begin{split}||\nabla R(\bm{\theta}\odot\bm{m})-\nabla
R(\bm{\theta}^{\prime}\odot\bm{m})||_{2}&\leq
L||(\bm{\theta}-\bm{\theta}^{\prime})\odot\bm{m}||\\\ &\text{ for all
}\\{\bm{\theta},\bm{\theta}^{\prime}\\}\subset\mathbb{R}^{d}.\end{split}$ (4)

where $\bm{m}$ is a binary mask. In comparision of Eq. 3 and 4, we use the
theoretical analysis [77] where subnetwork achieve a faster rate of
$R(\bm{\theta}\odot\bm{m})=\mathcal{O}(1/||\bm{m}||^{2}_{1})$ at most. The
comparison is as follows:

$\begin{split}\frac{||\nabla R(\bm{\theta}\odot\bm{m})-\nabla
R(\bm{\theta}^{\prime}\odot\bm{m})||_{2}}{||(\bm{\theta}-\bm{\theta}^{\prime})\odot\bm{m}||}&<\\\
\frac{||\nabla R(\bm{\theta})-\nabla
R(\bm{\theta}^{\prime})||_{2}}{||\bm{\theta}-\bm{\theta}^{\prime}||}&\leq
L\end{split}$ (5)

The smaller the value is, the flatter the solution (loss landscape) has. The
equation is established from the relationship $R(\bm{\theta}\odot\bm{m})\ll
R^{\ast}(\bm{\theta})$, where $R^{\ast}(\bm{\theta}$ denotes the best possible
loss achievable by convex combinations of all parameters despite
$||(\bm{\theta}-\bm{\theta}^{\prime})\odot\bm{m}||<||\bm{\theta}-\bm{\theta}^{\prime}||$.

Corollary. Furthermore, we have the following inequality if
$||R(\bm{\theta}\odot\bm{m}_{hard})-R(\bm{\theta}\odot\bm{m}_{soft})||\simeq
0$ and $||\bm{m}_{hard}||<||\bm{m}_{soft}||$:

$\begin{split}&\frac{||\nabla R(\bm{\theta}\odot\bm{m}_{hard})-\nabla
R(\bm{\theta}^{\prime}\odot\bm{m}_{hard})||_{2}}{||(\bm{\theta}-\bm{\theta}^{\prime})\odot\bm{m}_{hard}||}\geq\\\
&\frac{||\nabla R(\bm{\theta}\odot\bm{m}_{soft})-\nabla
R(\bm{\theta}^{\prime}\odot\bm{m}_{soft})||_{2}}{||(\bm{\theta}-\bm{\theta}^{\prime})\odot\bm{m}_{soft}||}\end{split}$
(6)

where the equality holds iff $||\bm{m}_{hard}||=||\bm{m}_{soft}||$. We prepare
the loss landscapes of Dense Network, HardNet (WSN), and SoftNet as shown in
Figure 3 as an example to support the above subnetwork’s inequality.

###  3.4 Optimization of Winning SubNetworks (WSN)

To jointly learn the model weights and task-adaptive binary masks of
subnetworks associated with each task, given an objective
$\mathcal{L}(\cdot)$, we optimize $\bm{\theta}$ and $\mathbf{s}$ with:

$\operatorname*{minimize}_{\bm{\theta},\mathbf{s}}\mathcal{L}(\bm{\theta}\odot\mathbf{m}_{t};\mathcal{D}_{t}).$
(7)

However, this vanilla optimization procedure presents two problems: (1)
updating all $\bm{\theta}$ when training for new tasks will cause interference
to the weights allocated for previous tasks, and (2) the indicator function
always has a gradient value of $0$; therefore, updating the weight scores
$\mathbf{s}$ with its loss gradient is not possible. To solve the first
problem, we selectively update the weights by allowing updates only on the
weights not selected in the previous tasks. To do that, we use an accumulate
binary mask $\mathbf{M}_{t-1}=\operatorname*{\vee}_{i=1}^{t-1}\mathbf{m}_{i}$
when learning task $t$, then for an optimizer with learning rate $\eta,$ the
$\bm{\theta}$ is updated as follows:

$\bm{\theta}\leftarrow\bm{\theta}-\eta\left(\frac{\partial\mathcal{L}}{\partial\bm{\theta}}\odot(\mathbf{1}-\mathbf{M}_{t-1})\right),$
(8)

effectively freezing the weights of the subnetworks selected for the previous
tasks. To solve the second problem, we use Straight-through Estimator [28, 3,
58] in the backward pass since $\mathbf{m}_{t}$ is obtained by top-$c\%$
scores. Specifically, we ignore the derivatives of the indicator function and
update the weight score as follows:

$\quad\mathbf{s}\leftarrow\mathbf{s}-\eta\left(\frac{\partial\mathcal{L}}{\partial\mathbf{s}}\right).$
(9)

The use of separate weight scores $\mathbf{s}$ as the basis for selecting
subnetwork weights makes it possible to reuse some of the weights from
previously chosen weights $\bm{\theta}\odot\mathbf{m}_{t}$ in solving the
current task $t$, which can be viewed as transfer learning. Likewise,
previously chosen weights that are irrelevant to the new tasks are not
selected; instead, weights from the set of not-yet-chosen weights are selected
to meet the target network capacity for each task, which can be viewed as
finetuning from tasks $\\{1,...,t-1\\}$ to task $t$. Our WSN optimizing
procedure is summarized in pseudo algorithm 1.

Algorithm 1 Winning SubNetworks (WSN) for TIL.

0: $\\{\mathcal{D}_{t}\\}_{t=1}^{\mathcal{T}}$, model weights $\bm{\theta}$,
score weights $\mathbf{s}$, binary mask
$\mathbf{M}_{0}=\mathbf{0}^{|\bm{\theta}|}$, layer-wise capacity $c$

1: Randomly initialize $\bm{\theta}$ and $\mathbf{s}$.

2: for task $t=1,...,\mathcal{T}$ do

3: for batch $\mathbf{b}_{t}\sim\mathcal{D}_{t}$ do

4: Obtain soft-mask $\mathbf{m}_{t}$ of the top-$c\%$ scores $\mathbf{s}$ at
each layer

5: Compute
$\mathcal{L}\left(\bm{\theta}\odot\mathbf{m}_{t};\mathbf{b}_{t}\right)$

6:
$\bm{\theta}\leftarrow\bm{\theta}-\eta\left(\frac{\partial\mathcal{L}}{\partial\bm{\theta}}\odot(\mathbf{1}-\mathbf{M}_{t-1})\right)$
$\triangleright$ Weight update

7:
$\mathbf{s}\leftarrow\mathbf{s}-\eta(\frac{\partial\mathcal{L}}{\partial\mathbf{s}})$
$\triangleright$ Weight score update

8: end for

9:
$\mathbf{M}_{t}\leftarrow\mathbf{M}_{t-1}\mathbin{{\vee}\mspace{-2.9mu}}\mathbf{m}_{t}$
$\triangleright$ Accumulate binary mask

10: end for

###  3.5 Winning Ticket Encoding

A binary mask is needed to store task-specific weights for each task in WSN.
However, the main issue is that as the number of tasks increases in deep-
learning models, the number of binary masks required also increases. We use a
compression algorithm to compress all binary task masks to address this
capacity issue and achieve forget-free continual learning. To accomplish this,
we first convert a sequence of binary masks into a single accumulated decimal
mask and then convert each integer into an ASCII code to represent a unique
symbol. Next, we apply Huffman encoding [32], a lossless compression
algorithm, to the symbols. We empirically observed that Huffman encoding
compresses 7-bit binary maps with a compression rate of approximately 78% and
decompresses them without any bit loss. Furthermore, experimental results
demonstrate that the compression rate is sub-linearly increasing with the size
of binary bits.

###  3.6 SoftNet via Complementary Winning Tickets

Similar to WSN’s optimization discussed in Section 3.4, let each weight be
associated with a learnable parameter we call weight score $\bm{s}$, which
numerically determines the importance of the associated weight. In other
words, we declare a weight with a higher score more important. At first, we
find a subnetwork ${\bm{\theta}}^{\ast}=\bm{\theta}\odot\bm{m}^{\ast}_{t}$ of
the dense neural network and then assign it as a solver of the current session
$t$. The subnetworks associated with each session jointly learn the model
weight $\bm{\theta}$ and binary mask $\bm{m}_{t}$. Given an objective
$\mathcal{L}_{t}$, we optimize $\bm{\theta}$ as follows:

$\bm{\theta}^{\ast},\bm{m}^{\ast}_{t}=\operatorname*{minimize}_{\bm{\theta},\bm{s}}\mathcal{L}_{t}(\bm{\theta}\odot\bm{m}_{t};\mathcal{D}_{t}).$
(10)

where $\bm{m}_{t}$ is obtained by applying an indicator function
$\mathbbm{1}_{c}$ on weight scores $\bm{s}$. Note $\mathbbm{1}_{c}(s)=1$ if
$s$ belongs to top-c$\%$ scores and $0$ otherwise.

In the optimization process for FSCIL, however, we consider two main problems:
(1) Catastrophic forgetting: updating all $\bm{\theta}\odot\bm{m}_{t-1}$ when
training for new sessions will cause interference with the weights allocated
for previous tasks; thus, we need to freeze all previously learned parameters
$\bm{\theta}\odot\bm{m}_{t-1}$; (2) Overfitting: the subnetwork also
encounters overfitting issues when training an incremental task on a few
samples, as such, we need to update a few parameters irrelevant to previous
task knowledge., i.e., $\bm{\theta}\odot(\bm{1}-\bm{m}_{t-1})$.

To acquire the optimal subnetworks that alleviate the two issues, we define a
soft-subnetwork by dividing the dense neural network into two parts-one is the
major subnetwork $\bm{m}_{\text{major}}$, and another is the minor subnetwork
$\bm{m}_{\text{minor}}$. The defined Soft-SubNetwork (SoftNet) follows as:

$\bm{m}_{\text{soft}}=\bm{m}_{\text{major}}\oplus\bm{m}_{\text{minor}},$ (11)

where $\bm{m}_{\text{major}}$ is a binary mask and $\bm{m}_{\text{minor}}\sim
U(0,1)$ and $\oplus$ represents an element-wise summation. As such, a soft-
mask is given as $\bm{m}_{t}^{\ast}\in[0,1]^{|\bm{\theta}|}$ in Eq.10. In the
all-experimental FSCIL setting, $\bm{m}_{\text{major}}$ maintains the base
task knowledge $t=1$ while $\bm{m}_{\text{minor}}$ acquires the novel task
knowledge $t\geq 2$. Then, with base session learning rate $\alpha,$ the
$\bm{\theta}$ is updated as follows:
$\bm{\theta}\leftarrow\bm{\theta}-\alpha\left(\frac{\partial\mathcal{L}}{\partial\bm{\theta}}\odot\bm{m}_{\text{soft}}\right)$
effectively regularize the weights of the subnetworks for incremental
learning. The subnetworks are obtained by the indicator function that always
has a gradient value of $\bm{0}$; therefore, updating the weight scores
$\bm{s}$ with its loss gradient is impossible. We update the weight scores by
using Straight-through Estimator [28, 3, 58] in the backward pass.
Specifically, we ignore the derivatives of the indicator function and update
the weight score
$\bm{s}\leftarrow\bm{s}-\alpha\left(\frac{\partial\mathcal{L}}{\partial\bm{s}}\odot\bm{m}_{\text{soft}}\right)$,
where $\bm{m}_{\text{soft}}=\bm{1}$ for exploring the optimal subnetwork for
base session training. Our Soft-subnetwork optimizing procedure is summarized
in Algorithm 2. Once a single soft-subnetwork $\bm{m}_{\text{soft}}$ is
obtained in the base session, then we use the soft-subnetwork for the entire
new sessions without updating.

Algorithm 2 Soft-Subnetworks (SoftNet) for FSCIL.

0: $\\{\mathcal{D}^{t}\\}_{t=1}^{\mathcal{T}}$, model weights $\bm{\theta}$,
and score weights $\bm{s}$, layer-wise capacity $c$

1: // Training over base classes $t=1$

2: Randomly initialize $\bm{\theta}$ and $\bm{s}$.

3: for epoch $e=1,2,\cdots$ do

4: Obtain softmask $\bm{m}_{\text{soft}}$ of $\bm{m}_{major}$ and
$\bm{m}_{minor}\sim U(0,1)$ at each layer

5: for batch $\bm{b}_{t}\sim\mathcal{D}^{t}$ do

6: Compute
$\mathcal{L}_{base}\left(\bm{\theta}\odot\bm{m}_{\text{soft}};\bm{b}_{t}\right)$
by Eq. 10

7:
$\bm{\theta}\leftarrow\bm{\theta}-\alpha\left(\frac{\partial\mathcal{L}}{\partial\bm{\theta}}\odot\bm{m}_{\text{soft}}\right)$

8:
$\bm{s}\leftarrow\bm{s}-\alpha\left(\frac{\partial\mathcal{L}}{\partial\bm{s}}\odot\bm{m}_{\text{soft}}\right)$

9: end for

10: end for

11: // Incremental learning $t\geq 2$

12: Combine the training data $\mathcal{D}^{t}$

13: and the exemplars saved in previous few-shot sessions

14: for epoch $e=1,2,\cdots$ do

15: for batch $\bm{b}_{t}\sim\mathcal{D}^{t}$ do

16: Compute
$\mathcal{L}_{m}\left(\bm{\theta}\odot\bm{m}_{\text{soft}};\bm{b}_{t}\right)$
by Eq. 12

17:
$\bm{\theta}\leftarrow\bm{\theta}-\beta\left(\frac{\partial\mathcal{L}}{\partial\bm{\theta}}\odot\bm{m}_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}minor}}\right)$

18: end for

19: end for

19: model parameters $\bm{\theta}$, $\bm{s}$, and $\bm{m}_{\text{soft}}$.

###  3.7 Soft-SubNetwork for Incremental Learning

We now describe the overall procedure of our soft-pruning-based incremental
learning/inference method under the following FSCIL settings. This includes
the training phase with a normalized informative measurement, as outlined in
prior work [65], and the inference phase.

Few-shot Class Incremental Learning (FSCIL) aims to learn new sessions with
only a few examples continually. A FSCIL model learns a sequence of $T$
training sessions $\\{\mathcal{D}^{1},\cdots,\mathcal{D}^{T}\\}$, where
$\mathcal{D}^{t}=\\{z_{i}^{t}=(\bm{x}_{i}^{t},y_{i}^{t})\\}^{n_{t}}_{i}$ is
the training data of session $t$ and $\bm{x}_{i}^{t}$ is an example of class
$y_{i}^{t}\in\mathcal{O}^{t}$. In FSCIL, the base session $\mathcal{D}^{1}$
usually contains a large number of classes with sufficient training data for
each class. In contrast, the subsequent sessions ($t\geq 2$) will only contain
a small number of classes with a few training samples per class, e.g., the
$t^{\mathrm{th}}$ session $\mathcal{D}^{t}$ is often presented as a N-way
K-shot task. In each training session $t$, the model can access only the
training data $\mathcal{D}^{t}$ and a few examples stored in the previous
session. When the training of session $t$ is completed, we evaluate the model
on test samples from all classes
$\mathcal{O}=\bigcup_{i=1}^{t}\mathcal{O}^{i}$, where
$\mathcal{O}^{i}\bigcap\mathcal{O}^{j\neq i}=\emptyset$ for $\forall i,j\leq
T$.

Base Training $(t=1)$. In the base learning session, we optimize the soft-
subnetwork parameter $\bm{\theta}$ (including a fully-connected layer as a
classifier) and weight score $\bm{s}$ with cross-entropy loss jointly using
the training examples of $\mathcal{D}^{1}$.

Incremental Training $(t\geq 2)$. In the incremental few-shot learning
sessions $(t\geq 2)$, leveraged by $\bm{\theta}\odot\bm{m}_{\text{soft}}$, we
fine-tune few minor parameters $\bm{\theta}\odot\bm{m}_{\text{minor}}$ of the
soft-subnetwork to learn new classes. Since $\bm{m}_{\text{minor}}<\bm{1}$,
the soft-subnetwork alleviates the overfitting of a few samples. Furthermore,
instead of Euclidean distance [65], we employ a metric-based classification
algorithm with cosine distance to finetune the few selected parameters. In
some cases, Euclidean distance fails to give the real distances between
representations, especially when two points with the same distance from
prototypes do not fall in the same class. In contrast, representations with a
low cosine distance are located in the same direction from the origin,
providing a normalized informative measurement. We define the loss function
as:

$\begin{split}&\mathcal{L}_{m}(z;\bm{\theta}\odot\bm{m}_{soft})=\\\
&-\sum_{z\in\mathcal{D}}\sum_{o\in\mathcal{O}}\mathbbm{1}(y=o)\log\left(\frac{e^{-d(\bm{p}_{o},f(\bm{x};\;\bm{\theta}\odot\bm{m}_{soft}))}}{\sum_{o_{k}\in\mathcal{O}}e^{-d(\bm{p}_{o_{k}},f(\bm{x};\;\bm{\theta}\odot\bm{m}_{soft}))}}\right)\end{split}$
(12)

where $d\left(\cdot,\cdot\right)$ denotes cosine distance, $\bm{p}_{o}$ is the
prototype of class $o$, $\mathcal{O}=\bigcup_{i=1}^{t}\mathcal{O}^{i}$ refers
to all encountered classes, and
$\mathcal{D}=\mathcal{D}^{t}\bigcup\mathcal{P}$ denotes the union of the
current training data $\mathcal{D}^{t}$ and the exemplar set
$\mathcal{P}=\left\\{\bm{p}_{2}\cdots,\bm{p}_{t-1}\right\\}$, where
$\mathcal{P}_{t_{e}}\left(2\leq t_{e}<t\right)$ is the set of saved exemplars
in session $t_{e}$. Note that the prototypes of new classes are computed by
$\bm{p}_{o}=\frac{1}{N_{o}}\sum_{i}\mathbbm{1}(y_{i}=o)f(\bm{x}_{i};\bm{\theta}\odot\bm{m}_{soft})$
and those of base classes are saved in the base session, and $N_{o}$ denotes
the number of the training images of class $o$. We also save the prototypes of
all classes in $\mathcal{O}^{t}$ for later evaluation.

Inference for Incremental Soft-Subnetwork. In each session, the inference is
also conducted by a simple nearest class mean (NCM) classification algorithm
[54, 65] for fair comparisons. Specifically, all the training and test samples
are mapped to the embedding space of the feature extractor $f$, and Euclidean
distance $d_{u}(\cdot,\cdot)$ is used to measure the similarity between them.
The classifier gives the $k$th prototype index
$o_{k}^{\ast}=\arg\min_{o\in\mathcal{O}}d_{u}(f(\bm{x};\bm{\theta}\odot\bm{m}_{soft}),\bm{p}_{o})$
as output.

##  4 Experiments

We now validate our method on several benchmark datasets against relevant
continual learning baselines on Task-Incremental Learning (TIL) and Few-shot
Class Incremental Learning (FSCIL).

###  4.1 Task-incremental Learning (TIL)

First, we consider task-incremental continual learning with a multi-head
configuration for all experiments in the paper. We follow the experimental
setups in recent works [62, 78, 16].

Datasets and architectures. We use six different popular sequential datasets
for CL problems with five different neural network architectures as follows:
1) Permuted MNIST (PMNIST): A variant of MNIST [42] where each task has a
deterministic permutation to the input image pixels. 2) 5-Datasets: A mixture
of 5 different vision datasets [62]: CIFAR-10 [39], MNIST [42], SVHN [56],
FashionMNIST [74], and notMNIST [6]. 3) Omniglot Rotation: An OCR images
dataset composed of 100 tasks, each including 12 classes. We further
preprocess the raw images by generating their rotated version in
$90\degree,\leavevmode\nobreak\ 180\degree,\text{\leavevmode\nobreak\
and\leavevmode\nobreak\ }270\degree$, followed by [78]. 4) CIFAR-100 Split
[39]: A visual object dataset constructed by randomly dividing 100 classes of
CIFAR-100 into ten tasks with ten classes per task. 5) CIFAR-100 Superclass:
We follow the setting from [78] that divides CIFAR-100 dataset into 20 tasks
according to the 20 superclasses, and each superclass contains five different
but semantically related classes. 6) TinyImageNet [67]: A variant of ImageNet
[40] containing 40 of 5-way classification tasks with the image size by
$64\times 64\times 3$.

We use two-layered MLP with $100$ neurons per layer for PMNIST, variants of
LeNet [42] for the experiments on Omniglot Rotation and CIFAR-100 Superclass
experiments. Also, we use a modified version of AlexNet similar to [64, 62]
for the CIFAR-100 Split dataset and a reduced ResNet-18 similar to [9, 62] for
5-Datasets. For TinyImageNet, we also use the same network architecture as
[20, 16], which consists of 4 Conv layers and three fully connected layers.

Baselines. We compare our WSN with strong CL baselines; regularization-based
methods: HAT [64] and EWC [38], rehearsal-based methods: GPM [62], and a
pruning-based method: PackNet [50] and SupSup [72]. PackNet and SupSup is set
to the baseline to show the effectiveness of re-used weights. We also compare
with naive sequential training strategy, referred to as FINETUNE. Multitask
Learning (MTL) and Single-task Learning (STL) are not a CL method. MTL trains
on multiple tasks simultaneously, and STL trains on single tasks
independently.

We summarize the architecture-based baselines as follows:

  1. 1.

PackNet [50]: iterative pruning and network re-training architecture for
packing multiple tasks into a single network.

  2. 2.

SupSup [72]: finding supermasks (subnetworks) within a randomly initialized
network for each task in continual learning.

  3. 3.

WSN / SoftNet (ours): jointly training model and finding task-adaptive
subnetworks of novel/prior parameters for continual learning. Note that, in
the inference step, SoftNet is acquired empirically by injecting small noises
$U(0,\text{1e-3})$ to the backgrounds $\bm{0}$ of acquired task WSN while
holding the foregrounds $\bm{1}$ of WSN.

Table I: Performance comparison of the proposed method and baselines on
various benchmark datasets. We report the mean and standard deviation of the
average accuracy (ACC), average capacity (CAP), and average forward / backward
transfer (FWT / BWT) across five independent runs. The best results are
highlighted in bold. Values with $\dagger$ and $\ast$ denote reported
performances from [62] and [78]. We consider PackNet [50] and SupSup [72] as
the baselines.

Method | Permuted MNIST | 5 Datasets | Omniglot Rotation
---|---|---|---
| ACC (%) | CAP (%) | FWT / BWT | ACC (%) | CAP (%) | FWT / BWT | ACC (%) | CAP (%) | FWT / BWT
STL |  97.37 ($\pm$ 0.01) |  1,000.0 | \- / - |  93.44 ($\pm$ 0.12) |  500.0 | \- / - |  82.13 ($\pm$ 0.08)∗ |  10,000.0 | \- / -
FINETUNE |  78.22 ($\pm$ 0.84) |  100.0 |  \- / -0.21 ($\pm$ 0.01) |  80.06 ($\pm$ 0.74) |  100.0 |  \- / -0.17 ($\pm$ 0.01) |  44.48 ($\pm$ 1.68) |  100.0 |  \- / -0.45 ($\pm$ 0.02)
EWC [38] |  92.01 ($\pm$ 0.56) |  100.0 |  \- / -0.03 ($\pm$ 0.00) |  88.64 ($\pm$ 0.26)† |  100.0† |  \- / -0.04 ($\pm$ 0.01)† |  68.66 ($\pm$ 1.92)∗ |  100.0∗ | \- / -
HAT [64] | - | - | \- / - |  91.32 ($\pm$ 0.18)† |  100.0† |  \- / -0.03 ($\pm$ 0.00)† | - | - | \- / -
GPM [62] |  94.96 ($\pm$ 0.07) |  100.0 |  \- / -0.02 ($\pm$ 0.01) |  91.22 ($\pm$ 0.20)† |  100.0 |  \- / -0.01 ($\pm$ 0.00)† |  85.24 ($\pm$ 0.37) |  100.0 |  \- / -0.01 ($\pm$ 0.00)
PackNet [50] |  96.37 ($\pm$ 0.1) |  96.38 |  -9.52 ($\pm$ 0.6) / 0.0 |  92.81 ($\pm$ 0.1) |  82.86 |  -9.73 ($\pm$ 0.5) / 0.0 |  30.70 ($\pm$ 1.5) |  399.2 |  -9.02 ($\pm$ 0.8) / 0.0
SupSup [72] |  96.31 ($\pm$ 0.1) |  122.89 ($\pm$ 0.1) |  -9.63 ($\pm$ 0.7) / 0.0 |  93.28 ($\pm$ 0.2) |  104.27 ($\pm$ 0.2) |  -9.52 ($\pm$ 0.6) / 0.0 |  58.14 ($\pm$ 2.4) |  407.12 ($\pm$ 0.2) |  -8.87 ($\pm$ 0.6) / 0.0
WSN, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 3\%$ |  94.84 ($\pm$ 0.1) |  19.87 ($\pm$ 0.2) |  -10.25 ($\pm$ 0.0) / 0.0 |  90.57 ($\pm$ 0.7) |  12.11 ($\pm$ 0.1) |  -9.62 ($\pm$ 0.6) / 0.0 |  80.68 ($\pm$ 2.6) |  75.87 ($\pm$ 1.2) |  -8.33 ($\pm$ 0.1) / 0.0
WSN, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 5\%$ |  95.65 ($\pm$ 0.0) |  26.49 ($\pm$ 0.2) |  -9.96 ($\pm$ 0.7) / 0.0 |  91.61 ($\pm$ 0.2) |  17.26 ($\pm$ 0.3) |  -9.86 ($\pm$ 0.4) / 0.0 | 87.28 ($\pm$ 0.7) |  79.85 ($\pm$ 1.2) |  -8.42 ($\pm$ 0.1) / 0.0
WSN, $c=10\%$ |  96.14 ($\pm$ 0.0) |  40.41 ($\pm$ 0.6) |  -10.14 ($\pm$ 1.4) / 0.0 |  92.67 ($\pm$ 0.1) |  28.01 ($\pm$ 0.3) |  -9.33 ($\pm$ 0.4) / 0.0 |  83.10 ($\pm$ 1.6) |  83.08 ($\pm$ 1.6) |  -8.29 ($\pm$ 0.1) / 0.0
WSN, $c=30\%$ | 96.41 ($\pm$ 0.1) |  77.73 ($\pm$ 0.4) |  -9.52 ($\pm$ 0.4) / 0.0 |  93.22 ($\pm$ 0.3) |  62.30 ($\pm$ 0.7) |  -9.68 ($\pm$ 0.6) / 0.0 |  81.89 ($\pm$ 1.2) |  102.2 ($\pm$ 0.9) |  -8.34 ($\pm$ 0.1) / 0.0
WSN, $c=50\%$ |  96.24 ($\pm$ 0.1) |  98.10 ($\pm$ 0.3) |  -9.39 ($\pm$ 0.8) / 0.0 | 93.41 ($\pm$ 0.1) |  86.10 ($\pm$ 0.6) |  -9.51 ($\pm$ 0.4) / 0.0 |  79.80 ($\pm$ 2.2) |  121.2 ($\pm$ 0.5) |  -8.33 ($\pm$ 0.1) / 0.0
WSN, $c=70\%$ |  96.29 ($\pm$ 0.0) |  102.56 ($\pm$ 0.1) |  -9.67 ($\pm$ 0.7) / 0.0 |  92.38 ($\pm$ 0.5) |  98.73 ($\pm$ 0.4) |  -9.53 ($\pm$ 0.4) / 0.0 |  79.02 ($\pm$ 2.5) |  140.6 ($\pm$ 0.3) |  -8.32 ($\pm$ 0.1) / 0.0
SoftNet, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 3\%$ |  94.84 ($\pm$ 0.1) |  19.87 ($\pm$ 0.2) |  23.76 ($\pm$ 1.8) / 0.0 |  90.57 ($\pm$ 0.7) |  12.11 ($\pm$ 0.1) |  36.08 ($\pm$ 7.1) / 0.0 |  80.68 ($\pm$ 2.6) |  75.87 ($\pm$ 1.2) |  51.52 ($\pm$ 4.3) / 0.0
SoftNet, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 5\%$ |  95.65 ($\pm$ 0.0) |  26.49 ($\pm$ 0.2) |  23.44 ($\pm$ 2.0) / 0.0 |  91.61 ($\pm$ 0.2) |  17.26 ($\pm$ 0.3) |  40.19 ($\pm$ 4.6) / 0.0 | 87.28 ($\pm$ 0.7) |  79.85 ($\pm$ 1.2) |  56.54 ($\pm$ 4.1) / 0.0
SoftNet, $c=10\%$ |  96.14 ($\pm$ 0.0) |  40.41 ($\pm$ 0.6) |  22.12 ($\pm$ 1.6) / 0.0 |  92.67 ($\pm$ 0.1) |  28.01 ($\pm$ 0.3) |  38.06 ($\pm$ 9.6) / 0.0 |  83.10 ($\pm$ 1.6) |  83.08 ($\pm$ 1.6) |  54.14 ($\pm$ 3.5) / 0.0
SoftNet, $c=30\%$ | 96.41 ($\pm$ 0.1) |  77.73 ($\pm$ 0.4) |  22.12 ($\pm$ 1.6) / 0.0 |  93.22 ($\pm$ 0.3) |  62.30 ($\pm$ 0.7) |  40.83 ($\pm$ 5.7) / 0.0 |  81.89 ($\pm$ 1.2) |  102.2 ($\pm$ 0.9) |  53.03 ($\pm$ 1.5) / 0.0
SoftNet, $c=50\%$ |  96.24 ($\pm$ 0.1) |  98.10 ($\pm$ 0.3) |  21.91 ($\pm$ 1.9) / 0.0 | 93.41 ($\pm$ 0.1) |  86.10 ($\pm$ 0.6) |  40.81 ($\pm$ 7.2) / 0.0 |  79.80 ($\pm$ 2.2) |  121.2 ($\pm$ 0.5) |  50.15 ($\pm$ 0.7) / 0.0
SoftNet, $c=70\%$ |  96.29 ($\pm$ 0.0) |  99.21 ($\pm$ 0.1) |  20.57 ($\pm$ 1.2) / 0.0 |  92.38 ($\pm$ 0.5) |  98.73 ($\pm$ 0.4) |  41.04 ($\pm$ 5.2) / 0.0 |  79.02 ($\pm$ 2.5) |  140.6 ($\pm$ 0.3) |  49.62 ($\pm$ 1.7) / 0.0
MTL |  96.70 ($\pm$ 0.02)† |  100.0 | \- / - |  91.54 ($\pm$ 0.28)† |  100.0 | \- / - |  81.23 ($\pm$ 0.52) |  100.0 | \- / -

Experimental settings. As we directly implement our method from the official
code of [62], we provide the values for HAT and GPM reported in [62]. For
Omniglot Rotation and Split CIFAR-100 Superclass, we deploy the proposed
architecture in multi-head settings with hyperparameters as reported in [78].
All our experiments run on a single-GPU setup of NVIDIA V100. We provide more
details of the datasets, architectures, and experimental settings, including
the hyperparameter configurations for all methods in Appendix A.

(a) 5-Dataset - ResNet-18 | (b) Omniglot Rotation - LeNet
---|---
Figure 4: Accuracy over Total Capacity Usage: The performances of Hard-WSN
gets better than others as total model capacities increase, and then,
saturated approximately at 80 %.

Performance metrics. We evaluate all methods based on the following four
metrics:

  1. 1.

Accuracy (ACC) measures the average of the final classification accuracy on
all tasks: $\mathrm{ACC}=\frac{1}{T}\sum_{i=1}^{T}A_{T,i}$, where $A_{T,i}$ is
the test accuracy for task $i$ after training on task $T$.

  2. 2.

Capacity (CAP) measures the total percentage of non-zero weights plus the
prime masks for all tasks as follows:
$\mathrm{Capacity}=(1-\mathcal{S})+\frac{(1-\alpha)T}{32}$, where we assume
all task weights of 32-bit precision. $\mathcal{S}$ is the sparsity of
$\mathbf{M}_{T}$ and the average compression rate $\alpha\approx 0.78$ that we
acquired through 7bit Huffman encoding, which depends on the size of bit-
binary maps; the compression rate $\alpha$ also depends on compression methods
typically.

  3. 3.

Forward Transfer (FWT) measures how much the representations that we learned
so far help learn new tasks, namely:
$FWT=\frac{1}{T}\sum_{i=2}^{T}A_{i-1,i}-R_{i}$ where $R_{i}$ is the accuracy
of a randomly initialized network on task $i$.

  4. 4.

Backward Transfer (BWT) measures the forgetting during continual learning.
Negative BWT means that learning new tasks causes the forgetting on past
tasks: $\mathrm{BWT}=\frac{1}{T-1}\sum_{i=1}^{T-1}A_{T,i}-A_{i,i}$.

###  4.2 Few-shot Class Incremental Learning (FSCIL)

We introduce experimental setups - Few-Shot Class Incremental Learning (FSCIL)
settings to provide soft-subnetworks’ effectiveness. We empirically evaluate
and compare our soft-subnetworks with state-of-the-art methods and vanilla
subnetworks in the following subsections.

Datasets. To validate the effectiveness of the soft-subnetwork, we follow the
standard FSCIL experimental setting. We randomly select 60 classes as the base
and 40 as new classes for CIFAR-100 and miniImageNet. In each incremental
learning session, we construct 5-way 5-shot tasks by randomly picking five
classes and sampling five training examples for each class.

Baselines. We mainly compare our SoftNet with architecture-based methods for
FSCIL: FSLL [52] that selects important parameters for each session, and
HardNet, representing a binary subnetwork. Furthermore, we compare other FSCIL
methods such as iCaRL [59], Rebalance [31], TOPIC [68], IDLVQ-C [10], and F2M
[65]. We also include a joint training method [65] that uses all previously
seen data, including the base and the following few-shot tasks for training as
a reference. Furthermore, we fix the classifier re-training method (cRT) [35]
for long-tailed classification trained with all encountered data as the
approximated upper bound.

Experimental settings. The experiments are conducted with NVIDIA GPU RTX8000
on CUDA 11.0. We also randomly split each dataset into multiple sessions. We
run each algorithm ten times for each dataset and report their mean accuracy.
We adopt ResNet18 [24] as the backbone network. For data augmentation, we use
standard random crop and horizontal flips. In the base session training stage,
we select top-$c\%$ weights at each layer and acquire the optimal soft-
subnetworks with the best validation accuracy. In each incremental few-shot
learning session, the total number of training epochs is $6$, and the learning
rate is $0.02$. We train new class session samples using a few minor weights
of the soft-subnetwork (Conv4x layer of ResNet18) obtained by the base session
learning. We specify further experiment details in Appendix.

##  5 Results On Task-Incremental Learning

###  5.1 Comparisons with Baselines

We evaluate our algorithm on three-standard benchmark datasets: Permuted
MNIST, 5-Datasets, and Omniglot Rotation. We set PackNet and SupSup to
baselines as non-reused weight methods and compared WSN with the baselines,
including other algorithms as shown in Table I. Our WSN outperformed all the
baselines in three measurements, achieving the best average accuracy of
96.41%, 93.41%, and 87.28% while using the least capacity compared to the
other existing methods, respectively. Moreover, our WSN was proved to be a
forget-free model like PackNet. Compared with PackNet, however, WSN showed the
effectiveness of reused weights for continual learning and its scalability in
the Omniglot Rotation experiments with the least capacities by a large margin.
Figure 4 provides the accuracy over total capacity usage. Our WSN’s accuracy
is higher than PackNet’s, approximately at 80% total capacity usage on
5-Dataset, and WSN outperformed others at 80% total capacity usage on Omniglot
Rotation. The lower performances of PackNet might attribute to Omniglot
Rotation dataset statistics since, regardless of random rotations, tasks could
share visual features in common such as circles, curves, and straight lines.
Therefore, non-reused methods might be brutal to train a new task model unless
prior weights were transferred to the current model, a.k.a. forward transfer
learning. To show the WSN’s power of forward transferring knowledge, SoftNet
was prepared by initializing zero-part of masks to small random perturbations.
Then, we observed that, in the inference step, SoftNet showed great power to
transfer knowledge while maintaining WSN’s performances of ACC and CAP.

###  5.2 Comparisons with the SOTA in TIL

Table II: Performance comparisons of the proposed method and other state-of-
the-art including baselines - PackNet [50] and SupSup [72] \- on various
benchmark datasets. We report the mean and standard deviation of the average
accuracy (ACC), average capacity (CAP), and average forward / backward
transfer (FWT / BWT) across $5$ independent runs with five seeds under the
same experimental setup [16]. The best results are highlighted in bold. Also,
$\dagger$ denotes results reported from [16].

Method | CIFAR-100 Split | CIFAR-100 Superclass | TinyImageNet
---|---|---|---
| ACC (%) | CAP (%) | FWT / BWT (%) | ACC (%) | CAP (%) | FWT / BWT (%) | ACC (%) | CAP (%) | FWT / BWT (%)
La-MaML [20] | 71.37 ($\pm$ 0.7)† | 100.0 | \- / -5.39 ($\pm$ 0.5)† | 54.44 ($\pm$ 1.4)† | 100.0 | \- / -6.65 ($\pm$ 0.9)† | 66.90 ($\pm$ 1.7)† | 100.0 | \- / -9.13 ($\pm$ 0.9)†
GPM [62] | 73.18 ($\pm$ 0.5)† | 100.0 | \- / -1.17 ($\pm$ 0.3)† | 57.33 ($\pm$ 0.4)† | 100.0 | \- / -0.37 ($\pm$ 0.1)† | 67.39 ($\pm$ 0.5)† | 100.0 | \- / 1.45 ($\pm$ 0.2)†
FS-DGPM [16] | 74.33 ($\pm$ 0.3)† | 100.0 | \- / -2.71 ($\pm$ 0.2)† | 58.81 ($\pm$ 0.3)† | 100.0 | \- / -2.97 ($\pm$ 0.4)† | 70.41 ($\pm$ 1.3)† | 100.0 | \- / -2.11 ($\pm$ 0.9)†
PackNet [50] | 72.39 ($\pm$ 0.4) |  96.38 ($\pm$ 0.0) |  0.12 ($\pm$ 0.6) / 0.0 | 58.78 ($\pm$ 0.5) | 126.65 ($\pm$ 0.0) |  0.56 ($\pm$ 0.8) / 0.0 | 55.46 ($\pm$ 1.2) | 188.67 ($\pm$ 0.0) |  -0.44 ($\pm$ 0.5) / 0.0
SupSup [72] | 75.47 ($\pm$ 0.3) | 129.00 ($\pm$ 0.1) |  0.06 ($\pm$ 0.5) / 0.0 | 61.70 ($\pm$ 0.3) | 162.49 ($\pm$ 0.0) |  -0.50 ($\pm$ 0.6) / 0.0 | 59.60 ($\pm$ 1.1) | 214.52 ($\pm$ 0.9) |  -0.82 ($\pm$ 0.6) / 0.0
WSN, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 3\%$ | 72.23 ($\pm$ 0.3) |  18.56 ($\pm$ 0.3) |  0.13 ($\pm$ 0.2) / 0.0 | 54.99 ($\pm$ 0.7) |  22.30 ($\pm$ 0.2) |  0.15 ($\pm$ 0.9) / 0.0 | 68.72 ($\pm$ 1.6) |  37.19 ($\pm$ 0.2) | -0.49 ($\pm$ 0.3) / 0.0
WSN, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 5\%$ | 73.56 ($\pm$ 0.2) |  25.09 ($\pm$ 0.4) |  0.05 ($\pm$ 0.5) / 0.0 | 57.99 ($\pm$ 1.3) |  27.37 ($\pm$ 0.3) |  0.59 ($\pm$ 0.6) / 0.0 | 71.22 ($\pm$ 0.9) |  41.98 ($\pm$ 0.5) | -0.73 ($\pm$ 0.3) / 0.0
WSN, $c=10\%$ | 75.46 ($\pm$ 0.1) |  39.87 ($\pm$ 0.6) |  0.07 ($\pm$ 0.3) / 0.0 | 60.45 ($\pm$ 0.4) |  38.55 ($\pm$ 0.2) | -0.35 ($\pm$ 0.5) / 0.0 |  71.96 ($\pm$ 1.4) |  48.65 ($\pm$ 3.0) | -0.63 ($\pm$ 0.4) / 0.0
WSN, $c=30\%$ | 77.12 ($\pm$ 0.3) |  80.26 ($\pm$ 1.5) | -0.02 ($\pm$ 0.5) / 0.0 | 61.47 ($\pm$ 0.3) |  63.47 ($\pm$ 1.3) | -0.49 ($\pm$ 0.5) / 0.0 | 70.92 ($\pm$ 1.3) |  73.44 ($\pm$ 2.4) | -0.32 ($\pm$ 0.1) / 0.0
WSN, $c=50\%$ | 77.46 ($\pm$ 0.4) |  99.13 ($\pm$ 0.5) | -0.04 ($\pm$ 0.3) / 0.0 |  61.79 ($\pm$ 0.2) |  80.93 ($\pm$ 1.6) | -0.26 ($\pm$ 0.7) / 0.0 | 69.06 ($\pm$ 0.8) |  92.03 ($\pm$ 1.8) | -0.33 ($\pm$ 0.1) / 0.0
WSN, $c=70\%$ |  78.08 ($\pm$ 0.3) | 105.77 ($\pm$ 0.2) | -0.06 ($\pm$ 0.1) / 0.0 | 61.24 ($\pm$ 0.2) |  97.40 ($\pm$ 1.0) | -0.16 ($\pm$ 1.2) / 0.0 | 67.18 ($\pm$ 1.6) | 108.82 ($\pm$ 1.3) | -0.37 ($\pm$ 0.4) / 0.0
SoftNet, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 3\%$ | 72.23 ($\pm$ 0.3) |  18.56 ($\pm$ 0.3) |  52.07 ($\pm$ 0.5) / 0.0 | 55.00 ($\pm$ 0.7) |  22.30 ($\pm$ 0.2) |  34.88 ($\pm$ 1.1) / 0.0 | 68.72 ($\pm$ 1.6) |  37.19 ($\pm$ 0.2) | 49.98 ($\pm$ 1.4) / 0.0
SoftNet, $c=\leavevmode\nobreak\ \leavevmode\nobreak\ 5\%$ | 73.57 ($\pm$ 0.2) |  25.09 ($\pm$ 0.4) | 51.70 ($\pm$ 0.5) / 0.0 | 57.99 ($\pm$ 1.3) |  27.37 ($\pm$ 0.3) | 34.76 ($\pm$ 1.1) / 0.0 | 71.22 ($\pm$ 0.9) |  41.98 ($\pm$ 0.5) |  50.83 ($\pm$ 0.6) / 0.0
SoftNet, $c=10\%$ | 75.47 ($\pm$ 0.1) |  39.87 ($\pm$ 0.6) | 51.58 ($\pm$ 0.6) / 0.0 | 60.46 ($\pm$ 0.4) |  38.55 ($\pm$ 0.2) | 32.21 ($\pm$ 0.9) / 0.0 |  71.97 ($\pm$ 1.4) |  48.65 ($\pm$ 3.0) | 48.86 ($\pm$ 1.6) / 0.0
SoftNet, $c=30\%$ | 77.12 ($\pm$ 0.3) |  80.26 ($\pm$ 1.5) | 51.64 ($\pm$ 0.9) / 0.0 | 61.47 ($\pm$ 0.3) |  63.47 ($\pm$ 1.3) | 32.04 ($\pm$ 1.3) / 0.0 | 70.92 ($\pm$ 1.4) |  73.44 ($\pm$ 2.4) | 47.54 ($\pm$ 1.0) / 0.0
SoftNet, $c=50\%$ | 77.46 ($\pm$ 0.4) |  99.13 ($\pm$ 0.5) | 52.06 ($\pm$ 1.3) / 0.0 |  61.80 ($\pm$ 0.2) |  80.93 ($\pm$ 1.6) | 30.40 ($\pm$ 0.7) / 0.0 | 69.06 ($\pm$ 0.8) |  92.03 ($\pm$ 1.8) | 47.80 ($\pm$ 1.1) / 0.0
SoftNet, $c=70\%$ |  78.08 ($\pm$ 0.3) |  105.77 ($\pm$ 0.2) | 54.07 ($\pm$ 0.5) / 0.0 | 61.24 ($\pm$ 0.2) |  97.40 ($\pm$ 1.0) | 30.94 ($\pm$ 1.4) / 0.0 | 67.19 ($\pm$ 1.6) | 108.82 ($\pm$ 1.3) | 48.27 ($\pm$ 0.9) / 0.0
Multitask |  79.75 ($\pm$ 0.4)† | 100.0 | \- / - |  61.00 ($\pm$ 0.2)† | 100.0 | \- / - |  77.10 ($\pm$ 1.1)† | 100.0 | \- / -

(a) Per Task Accuracy over $c$ | (b) Model Capacity
---|---
Figure 5: WSN’s Performances and Compressed Capacities: Sequence of TinyImageNet Dataset Experiments: (a) Setting $c=10\%$ shows generalized performances over others and (b) Within 40 tasks, the 7-bits compressed model increase its capacity the least. (a) A Diagram of Capacities | (b) Capacities over $c$ | (c) Acc. of Reused Weights | (d) Acc. w/o Reused Weights
---|---|---|---
Figure 6: Layer-wise Analysis of WSN on TinyImageNet Dataset Experiments: (a)
Weights reusability within a dense network, (b) Capacities except to binary
maps are determined by $c=10\%$, (c) The most significant forgetting occurs
from weights without reused per task; the significance of weights reused for
all tasks gets lower, and from task 7, all used weights seem to be enough to
infer all tasks, and (d) In the inference step, we inspected a layer-wise
forgetting caused by removing (setting masking value 1 to 0) reused weights
per task on the trained model and observed performance drops significantly at
Conv1 layer.

We use a multi-head setting to evaluate our WSN algorithm under the more
challenging visual classification benchmarks. The WSN’s performances are
compared with others w.r.t three measurements on three major benchmark
datasets as shown in Table II. Our WSN outperformed all state-of-the-art,
achieving the best average accuracy of 76.38%, 61.79%, and 71.96%. WSN is also
a forget-free model (BWT = ZERO) with the least model capacity in these
experiments. Note that we assume the model capacities are compared based on
the model size without extra memory, such as samples. We highlight that our
method achieves the highest accuracy, the lowest capacity, and backward
transfer on all datasets. Figure 5 shows the process of performance and
compressed capacity changing with the number of tasks on the TinyImageNet
datasets, where the “Average Progressive Capacity” metric is defined as the
average capacity (the proportion of the number of network weights used for any
one of the tasks) after five runs of the experiment with different seed
values. Furthermore, we consistently showed progressively improved
performances of WSN than others on CIFAR-100 Split datasets as shown in Figure
7. The increasing number of reused weights (see Figure 9) could explain the
progressively improved performances as shown in Figure 8. Other forward-
transferring results are depicted in Appendix.

###  5.3 Forget-Free Performance and Model Capacity

We prepare results on performance and bit-map compressed capacity on the
TinyImageNet dataset as shown in Figure 5 \- “c=0.1” and “c=0.1+7bit-Huffman”
refers respectively to using 10% of network weights and no compression on the
binary mask and the latter refers to the same with 7bit-Huffman encoding. In
Figure 5 (a), using initial capacity, $c=0.1$ shows better performances over
others. With fixed $c=0.1$, the bit-wise Huffman compression rate delivers
positive as the number of tasks increases, as shown in Figure 5 (b). The most
interesting part is that the average compression rate increases as the number
of bits to compress increases, and the increasing ratio of reused weights
(symbols with a high probability in the Huffman encoding) might affect the
compression rate (symbols with a small probability might be rare in the
Huffman tree, where the infrequent symbols tend to have long bit codes). We
investigated how the compression rate is related to the total model capacity.
The more bits the binary mask compression does, the less the model capacity to
save is required. This shows that within 40 tasks, N-bit Huffman compressed
capacities are sub-linearly increasing as binary map capacities increase
linearly. The 7-bit Huffman encoding is enough to compress binary maps without
exceeding the model capacity, even though the compression rate $\alpha$
depends on compression methods typically.

###  5.4 Catastrophic Forgetting From WSN’s Viewpoint

We interpreted how reused weights affect the inference performances on the
TinyImageNet dataset as shown in Figure 6. We divide all used weights for
solving sequential tasks into specific sets for more precise interpretability.
All used weights (a) within a trained dense network are separated as follows:
all used represents all activated sets of weights up to task $t-1$. per task
represents an activated set of weights at task $t$. new per task represents a
new activated set of weights at task $t$. reused per task represents an
intersection set of weights per task and all used weights. reused for all
tasks represents an intersection set of weights reused from task $1$ up to
task $t-1$.

First, our WSN adaptively reuses weights to solve the sequential tasks. In
Figure 6 (b), initial and progressive task capacity start from $c$ value; the
proportion of reused weights per task decreases in the early task learning
stage. However, it tends to be progressively saturated to $c=0.1$ since the
number of the new activated set of weights decreases, and the proportion of
reused weights for all prior tasks tends to decrease. In other words, WSN uses
diverse weights to solve the sequential tasks within all used weights rather
than depending on the reused weights for all prior tasks as the number of
tasks increases.

Second, our WSN provides a stepping stone for forget-free continual learning.
Regarding the benefits of WSN, in Figure 6 (c), we interpret the importance of
three types of weights through an ablation study. The additional evaluations
were performed by the acquired task binary masks and trained models to
investigate the importance of reused weights in each layer, where the ”w/”
refers to ”with reused network weights” and the ”w/o” refers to ”without
reused network weights.” Model forgetting occurred from the performances
without using weights reused per task severely. The most significant weights
were weights reused per task, the subset of all used weights; the importance
of the weights reused for all prior tasks decreases as the number of tasks
increases since its capacity gets small relatively, as shown in Figure 6 (b).
Moreover, in Figure 6 (d), we inspected layer-wise forgetting caused by
removing weights reused per task of network layers; the performance
sensitivities were quite diverse. In particular, we observed that the most
performance drops at the Conv1 layer.

Figure 7: Comparisons on CIFAR100-Split with [16]: It was difficult to determine the superiority of WSN only by comparing the performances in the early stage, however, it shows the progressively better performances of WSN than others. (a) WSN, $c=5\%$ | (b) SoftNet, $c=5\%$
---|---
(c) WSN, $c=70\%$ | (d) SoftNet, $c=70\%$
Figure 8: Average Forward Transfer Matrix on CIFAR-100 Split. WSN / SoftNet
with $c=5\%$ and $c=70\%$, respectively.

Finally, our WSN reuses weights if the knowledge from previous tasks is
already enough to solve the task at hand and employs a few new weights
otherwise. Specifically, from task 7, all used weights seem enough to infer
all tasks since the weights reused per task catch up with the task
performances. For more generalized forget-free continual learning, the model
should consider the layer-wise sensitivity of weights reused per task when
selecting weights reused for all prior tasks. These analyses might broadly
impact other machine learning fields, such as transfer, semi-supervised, and
domain adaptation.

Table III: Computational efficiency of WSN compared with PackNet and SupSup.
We report model training time in hours.

Method | Permuted MNIST | 5 Dataset | Omniglot Rotation
---|---|---|---
| ACC (%) | CAP (%) | Tr. TIME (h) | ACC (%) | CAP (%) | Tr. TIME (h) | ACC (%) | CAP (%) | Tr. TIME (h)
PackNet | 96.37 ($\pm$ 0.04) | 96.38 ($\pm$ 0.00) | 0.49 ($\pm$ 0.03) | 92.81 ($\pm$ 0.12) | 82.86 ($\pm$ 0.00) | 3.38 ($\pm$ 0.11) | 30.70 ($\pm$ 1.50) | 399.2 ($\pm$ 0.00) | 7.30 ($\pm$ 0.01)
SupSup | 96.31 ($\pm$ 0.09) | 122.89 ($\pm$ 0.07) | 0.48 ($\pm$ 0.06) | 93.28 ($\pm$ 0.21) | 104.27 ($\pm$ 0.21) | 3.20 ($\pm$ 0.01) | 58.14 ($\pm$ 2.42) | 407.12 ($\pm$ 0.17) | 6.92 ($\pm$ 0.03)
WSN (best) |  96.41 ($\pm$ 0.07) | 77.73 ($\pm$ 0.36) | 0.35 ($\pm$ 0.02) |  93.41 ($\pm$ 0.13) |  86.10 ($\pm$ 0.57) | 3.02 ($\pm$ 0.03) |  87.28 ($\pm$ 0.72) |  79.85 ($\pm$ 1.19) | 6.33 ($\pm$ 0.04)
Method | CIFAR-100 Split | CIFAR-100 Superclass | TinyImageNet
| ACC (%) | CAP (%) | Tr. TIME (h) | ACC (%) | CAP (%) | Tr. TIME (h) | ACC (%) | CAP (%) | Tr. TIME (h)
PackNet | 72.39 ($\pm$ 0.37) | 96.38 ($\pm$ 0.00) | 1.04 ($\pm$ 0.19) | 58.78 ($\pm$ 0.52) | 126.65 ($\pm$ 0.00) | 0.46 ($\pm$ 0.01) | 55.46 ($\pm$ 1.22) | 188.67 ($\pm$ 0.00) | 1.39 ($\pm$ 0.03)
SupSup | 75.47 ($\pm$ 0.30) | 129.00 ($\pm$ 0.03) | 0.79 ($\pm$ 0.14) | 61.70 ($\pm$ 0.31) | 162.49 ($\pm$ 0.00) | 0.37 ($\pm$ 0.00) | 59.60 ($\pm$ 1.05) | 214.52 ($\pm$ 0.89) | 0.92 ($\pm$ 0.00)
WSN (best) |  76.38 ($\pm$ 0.34) |  99.13 ($\pm$ 0.48) | 0.71 ($\pm$ 0.09) |  61.79 ($\pm$ 0.23) |  80.93 ($\pm$ 1.58) | 0.36 ($\pm$ 0.00) |  71.96 ($\pm$ 1.41) |  48.65 ($\pm$ 3.03) | 0.89 ($\pm$ 0.00)

(a) PackNet | (b) WSN, $c=5\%$
---|---
(c) WSN, $c=10\%$ | (d) WSN, $c=50\%$
Figure 9: Task-wise Binary Map Correlations on Omniglot Rotation (Top row) and
TinyImageNet (Bottom row). WSN reuses the weights, while PackNet does not. WSN
with $c=50\%$ reuses task weights more than the case of $c=10\%$ as shown in
higher correlation results than others.

###  5.5 Sparse Binary Maps

We prepared task-wise binary mask correlations to investigate how WSN reuses
weights over sequential tasks. As shown in Figure 9 (a) and (b), WSN tends to
progressively transfer weights used for prior tasks to weights for new ones
compared with PackNet. Figure 9 (c) and (d) showed that the tendency of reused
weights differs according to the $c$. This result might suggest that more
sparse reused binary maps lead to generalization than others.

##  6 Results On Few-shot CIL

Table IV: Classification accuracy of ResNet18 on CIFAR-100 for 5-way 5-shot
incremental learning. Underbar denotes the comparable results with FSLL [52].
$\ast$ denotes the results reported from [65].

Method | sessions |  The gap with cRT
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
cRT [65] | 65.18 | 63.89 | 60.20 | 57.23 | 53.71 | 50.39 | 48.77 | 47.29 | 45.28 | -
iCaRL [59]∗ | 66.52 | 57.26 | 54.27 | 50.62 | 47.33 | 44.99 | 43.14 | 41.16 | 39.49 | -5.79
Rebalance [31]∗ | 66.66 | 61.42 | 57.29 | 53.02 | 48.85 | 45.68 | 43.06 | 40.56 | 38.35 | -6.93
FSLL [52]∗ | 65.18 | 56.24 | 54.55 | 51.61 | 49.11 | 47.27 | 45.35 | 43.95 | 42.22 | -3.08
iCaRL [59] | 64.10 | 53.28 | 41.69 | 34.13 | 27.93 | 25.06 | 20.41 | 15.48 | 13.73 | -31.55
Rebalance [31] | 64.10 | 53.05 | 43.96 | 36.97 | 31.61 | 26.73 | 21.23 | 16.78 | 13.54 | -31.74
TOPIC [12] | 64.10 | 55.88 | 47.07 | 45.16 | 40.11 | 36.38 | 33.96 | 31.55 | 29.37 | -15.91
F2M [65] | 64.71 | 62.05 | 59.01 | 55.58 | 52.55 | 49.96 | 48.08 | 46.28 | 44.67 | -0.61
FSLL [52] | 64.10 | 55.85 | 51.71 | 48.59 | 45.34 | 43.25 | 41.52 | 39.81 | 38.16 | -7.12
HardNet (WSN), $c=50\%$ | 64.80 | 60.77 | 56.95 | 53.53 | 50.40 | 47.82 | 45.93 | 43.95 | 41.91 | -3.37
HardNet (WSN), $c=80\%$ | 69.65 | 64.60 | 60.59 | 56.93 | 53.60 | 50.80 | 48.69 | 46.69 | 44.63 | -0.65
HardNet (WSN), $c=99\%$ | 71.95 | 66.83 | 62.75 | 59.09 | 55.92 | 53.03 | 50.78 | 48.52 | 46.31 | +1.03
SoftNet, $c=50\%$ | 69.20 | 64.18 | 60.01 | 56.43 | 53.11 | 50.62 | 48.60 | 46.51 | 44.61 | -0.67
SoftNet, $c=80\%$ | 70.38 | 65.04 | 60.94 | 57.26 | 54.13 | 51.58 | 49.52 | 47.36 | 45.16 | -0.12
SoftNet, $c=99\%$ | 72.62 | 67.31 | 63.05 | 59.39 | 56.00 | 53.23 | 51.06 | 48.83 | 46.63 | +1.35

Table V: Classification accuracy of ResNet18 on miniImageNet for 5-way 5-shot
incremental learning. Underbar denotes the comparable results with FSLL [52].
$\ast$ denotes the results reported from [65].

Method | sessions |  The gap with cRT
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
cRT [65] | 67.30 | 64.15 | 60.59 | 57.32 | 54.22 | 51.43 | 48.92 | 46.78 | 44.85 | -
iCaRL [59]∗ | 67.35 | 59.91 | 55.64 | 52.60 | 49.43 | 46.73 | 44.13 | 42.17 | 40.29 | -4.56
Rebalance [31]∗ | 67.91 | 63.11 | 58.75 | 54.83 | 50.68 | 47.11 | 43.88 | 41.19 | 38.72 | -6.13
FSLL [52]∗ | 67.30 | 59.81 | 57.26 | 54.57 | 52.05 | 49.42 | 46.95 | 44.94 | 42.87 | -1.11
iCaRL [59] | 61.31 | 46.32 | 42.94 | 37.63 | 30.49 | 24.00 | 20.89 | 18.80 | 17.21 | -27.64
Rebalance [31] | 61.31 | 47.80 | 39.31 | 31.91 | 25.68 | 21.35 | 18.67 | 17.24 | 14.17 | -30.68
TOPIC [12] | 61.31 | 50.09 | 45.17 | 41.16 | 37.48 | 35.52 | 32.19 | 29.46 | 24.42 | -20.43
IDLVQ-C [10] | 64.77 | 59.87 | 55.93 | 52.62 | 49.88 | 47.55 | 44.83 | 43.14 | 41.84 | -3.01
F2M [65] | 67.28 | 63.80 | 60.38 | 57.06 | 54.08 | 51.39 | 48.82 | 46.58 | 44.65 | -0.20
FSLL [52] | 66.48 | 61.75 | 58.16 | 54.16 | 51.10 | 48.53 | 46.54 | 44.20 | 42.28 | -2.57
HardNet (WSN), $c=50\%$ | 65.13 | 60.37 | 56.12 | 53.17 | 50.17 | 47.74 | 45.34 | 43.35 | 42.13 | -2.72
HardNet (WSN), $c=80\%$ | 69.73 | 64.46 | 60.42 | 57.09 | 54.09 | 51.18 | 48.76 | 46.81 | 45.66 | +0.81
HardNet (WSN), $c=90\%$ | 64.68 | 59.80 | 55.70 | 52.82 | 50.01 | 47.30 | 45.17 | 43.34 | 42.09 | -2.76
SoftNet, $c=50\%$ | 72.83 | 67.23 | 62.82 | 59.41 | 56.44 | 53.55 | 50.92 | 48.99 | 47.60 | +2.75
SoftNet, $c=80\%$ | 76.63 | 70.13 | 65.92 | 62.52 | 59.49 | 56.56 | 53.71 | 51.72 | 50.48 | +5.63
SoftNet, $c=90\%$ | 77.00 | 70.38 | 65.94 | 62.45 | 59.32 | 56.25 | 53.76 | 51.75 | 50.39 | +5.54
SoftNet, $c=97\%$ | 77.17 | 70.32 | 66.15 | 62.55 | 59.48 | 56.46 | 53.71 | 51.68 | 50.24 | +5.39

###  6.1 Results and Comparisons

We compared SoftNet with the architecture-based FSLL and HardNet (WSN)
methods. We pick FSLL as an architecture-based baseline since it selects
important parameters for acquiring old/new class knowledge. The architecture-
based results on CIFAR-100 and miniImageNet are presented in Table IV and
Table V respectively. The performances of HardNet show the effectiveness of
the subnetworks that go with less model capacity compared to dense networks.
To emphasize our point, we found that ResNet18, with approximately $50\%$
parameters, achieves comparable performances with FSLL on CIFAR-100 and
miniImageNet. In addition, the performances of ResNet20 with $30\%$ parameters
(HardNet) are comparable with those of FSLL on CIFAR-100.

Experimental results are prepared to analyze the overall performances of
SoftNet according to the sparsity and dataset as shown in Figure 10. As we
increase the number of parameters employed by SoftNet, we achieve performance
gain on both benchmark datasets. The performance variance of SoftNet’s
sparsity seems to depend on datasets because the performance variance on
CIFAR-100 is less than that on miniImageNet. In addition, SoftNet retains
prior session knowledge successfully in both experiments as described in the
dashed line, and the performances of SoftNet ($c=60.0\%$) on the new class
session (8, 9) of CIFAR-100 than those of SoftNet ($c=80.0\%$) as depicted in
the dashed-dot line. From these results, we could expect that the best
performances depend on the number of parameters and properties of datasets.

|
---|---
(a) CIFAR-100 |  (b) miniImageNet
Figure 10: Classification accuracy of SoftNet on CIFAR-100 and miniImageNet
for 5-way 5-shot FSCIL: the overall performance depends on capacity $c$ and
the softness of subnetwork. Note that solid( ), dashed( ), and dashed-dot( )
lines denote overall, base, and novel class performances respectively.

Our SoftNet outperforms the state-of-the-art methods and cRT, which is used as
the approximate upper bound of FSCIL [65] as shown in Table IV and Table V.
Moreover, Figure 11 represents the outstanding performances of SoftNet on
CIFAR-100 and miniImageNet. SoftNet provides a new upper bound on each
dataset, outperforming cRT, while HardNet (WSN) provides new baselines among
pruning-based methods.

Figure 11: Comparision of subnetworks (HardNet (WSN) and SoftNet) with state-
of-the-art methods.

###  6.2 Considerations From SoftNet

Through extensive experiments, we deduce the following conclusions for
incorporating our method in the few-shot class incremental learning regarding
architectures.

Comparisions of HardNet (WSN) and SoftNet. Furthermore, increasing the number
of network parameters leads to better overall performance in both subnetworks
types, as shown in Figure 12 and Figure 13. Subnetworks, in the form of
HardNet and SoftNet, tend to retain prior (base) session knowledge denoted in
dashed ( ) line, and HardNet seems to be able to classify new session class
samples without continuous updates stated in dashed-dot ( ) line. From this,
we could expect how much previous knowledge HardNet learned at the base
session to help learn new incoming tasks (Forward Transfer). The overall
performances of SoftNet are better than HardNet since SoftNet improves both
base/new session knowledge by updating minor subnetworks. Subnetworks have a
broader spectrum of performances on miniImageNet (Figure 13) than on CIFAR-100
(Figure 12). This could be an observation caused by the dataset complexity -
i.e., if the miniImagenet dataset is more complex or harder to learn for a
subnetwork or a deep model as such subnetworks need more parameters to learn
miniImageNet than the CIFAR-100 dataset.

(a) $c=10\%$ | (b) $c=30\%$ | (c) $c=80\%$
---|---|---
Figure 12: Performances of HardNet (WSN) v.s. SoftNet on CIFAR-100 for 5-way 5-shot FSCIL: the overall performance depends on capacity $c$ and the softness of subnetwork. Note that solid( ), dashed( ), and dashed-dot( ) lines denote overall, base, and novel class performances respectively. (a) $c=10\%$ | (b) $c=30\%$ | (c) $c=80\%$
---|---|---
Figure 13: Performances of HardNet (WSN) v.s. SoftNet on miniImageNet for 5-way 5-shot FSCIL: the overall performance depends on capacity $c$ and the softness of the subnetwork. Note that solid( ), dashed( ), and dashed-dot( ) lines denote overall, base, and novel class performances respectively. (a) HardNet (WSN) with $c=30\%$ @Session2 | (b) SoftNet with $c=30\%$ @Session2
---|---
Figure 14: t-SNE Plots of HardNet (WSN) v.s. SoftNet on miniImageNet for 5-way
5-shot FSCIL: t-SNE plots represent the embeddings of the even-numbered test
class samples and compare one another. Note Session1 Class
Set:$\\{0,\cdots,59\\}$ and Session2 Novel Class Set:$\\{60,\cdots,64\\}$.

Smoothness of SoftNet. SoftNet has a broader spectrum of performances than
HardNet on miniImageNet. $20\%$ of minor subnet might provide a smoother
representation than HardNet because the performance of SoftNet was the best
approximately at $c=80\%$. We could expect that model parameter smoothness
guarantees quite competitive performances from these results. To support the
claim, we prepared the loss landscapes of a dense neural network, HardNet, and
SoftNet on two Hessian eigenvectors [76] as shown in Fig. 3. We observed the
following points through simple experiments. From these results, we can expect
how much knowledge the specified subnetworks can retain and acquire on each
dataset. The loss landscapes of Subnetworks (HardNet and SoftNet) were flatter
than those of dense neural networks. The minor subnet of SoftNet helped find a
flat global minimum despite random scaling weights in the training process.

Moreover, we compared the embeddings using t-SNE plots as shown in Figure 14.
In t-SNE’s 2D embedding spaces, the overall discriminative of SoftNet is
better than that of HardNet in terms of base class set and novel class set.
This $70\%$ of minor subnet affects SoftNet positively in base session
training and offers good initialized weights in novel session training.

Preciseness. Regarding fine-grained and small-sized CUB200-2011 FSCIL settings
as shown in Appendix Table, HardNet (WSN) also shows comparable results with
the baselines, and SoftNet outperforms others as denoted in Table IX. In this
FSCIL setting, we acquired the best performances of SoftNet through the
specific parameter selections. As of now, our SoftNet achieves state-of-the-
art results on the three datasets.

###  6.3 Additional Comparisons with SOTA

Comparisons with SOTA. We compare SoftNet with the following state-of-art-
methods on TOPIC class split [68] of three benchmark datasets - CIFAR100
(Appendix, Table. 7), miniImageNet (Appendix, Table. 8), and CUB-200-2011
(Appendix, Table. 9). We summarize the current FSCIL methods such as CEC [83],
LIMIT [84], MetaFSCIL [13], C-FSCIL [27], Subspace Reg. [1], Entropy-Reg [47],
and ALICE [57]. Leveraged by regularized backbone ResNet, SoftNet outperformed
all existing current works on CIFAR100, miniImageNet. On CUB-200-201, the
performances of SoftNet were comparable with those of ALICE and LIMIT,
considering that ALICE used class/data augmentations and LIMIT added an extra
multi-head attention layer.

##  7 Conclusion

Inspired by _Regularized Lottery Ticket Hypothesis (RLTH)_ , which states that
competitive smooth (non-binary) subnetworks exist within a dense network in
continual learning tasks, we investigated the performances of the proposed two
architecture-based continual learning methods referred to as Winning
SubNetworks (WSN) which sequentially learns and selects an optimal binary-
subnetwork (WSN) and an optimal non-binary Soft-Subnetwork (SoftNet) for each
task, respectively. Specifically, WSN and SoftNet jointly learned the
regularized model weights and task-adaptive non-binary masks of subnetworks
associated with each task whilst attempting to select a small set of weights
to be activated (winning ticket) by reusing weights of the prior subnetworks.
The proposed WSN and SoftNet were inherently immune to catastrophic forgetting
as each selected subnetwork model does not infringe upon other subnetworks in
Task Incremental Learning (TIL). In TIL, binary masks spawned per winning
ticket were encoded into one N-bit binary digit mask, then compressed using
Huffman coding for a sub-linear increase in network capacity to the number of
tasks. Surprisingly, we observed that in the inference step, SoftNet generated
by injecting small noises to the backgrounds of acquired WSN (holding the
foregrounds of WSN) provides excellent forward transfer power for future tasks
in TIL. Softnet showed its effectiveness over WSN in regularizing parameters
to tackle the overfitting, to a few examples in Few-shot Class Incremental
Learning (FSCIL).

## References

  * [1] Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya, and Jacob Andreas.  Subspace regularizers for few-shot class incremental learning.  arXiv preprint arXiv:2110.07059, 2021.
  * [2] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia.  Online continual learning with maximal interfered retrieval.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [3] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  CoRR, 2013.
  * [4] Léon Bottou, Frank E Curtis, and Jorge Nocedal.  Optimization methods for large-scale machine learning.  Siam Review, 60(2):223–311, 2018.
  * [5] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe.  Convex optimization.  Cambridge university press, 2004.
  * [6] Yaroslav Bulatov.  notmnist dataset.  2011\.
  * [7] Arslan Chaudhry, Naeemullah Khan, Puneet K Dokania, and Philip HS Torr.  Continual learning in low-rank orthogonal subspaces.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [8] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient lifelong learning with a-gem.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [9] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato.  Continual learning with tiny episodic memories.  arXiv preprint arXiv:1902.10486, 2019.
  * [10] Kuilin Chen and Chi-Guhn Lee.  Incremental few-shot learning via vector quantization in deep embedded space.  In International Conference on Learning Representations, 2020.
  * [11] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang.  Long live the lottery: The existence of winning tickets in lifelong learning.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [12] Ali Cheraghian, Shafin Rahman, Pengfei Fang, Soumava Kumar Roy, Lars Petersson, and Mehrtash Harandi.  Semantic-aware knowledge distillation for few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2534–2543, 2021.
  * [13] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang.  Metafscil: A meta-learning approach for few-shot class incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14166–14175, 2022.
  * [14] Daiki Chijiwa, Shin’ya Yamaguchi, Atsutoshi Kumagai, and Yasutoshi Ida.  Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks.  In Advances in Neural Information Processing Systems, 2022.
  * [15] Belur V Dasarathy.  Nosing around the neighborhood: A new system structure and classification rule for recognition in partially exposed environments.  IEEE Transactions on Pattern Analysis and Machine Intelligence, (1):67–71, 1980.
  * [16] Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng.  Flattening sharpness for dynamic gradient projection memory benefits continual learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2021\.
  * [17] Misha Denil, Babak Shakibi, Laurent Dinh, Marc Aurelio Ranzato, and Nando de Freitas.  Predicting parameters in deep learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2013\.
  * [18] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
  * [19] Siavash Golkar, Michael Kagan, and Kyunghyun Cho.  Continual learning via neural pruning.  arXiv preprint arXiv:1903.04476, 2019.
  * [20] Gunshi Gupta, Karmesh Yadav, and Liam Paull.  La-maml: Look-ahead meta learning for continual learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [21] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
  * [22] Peter Hart.  The condensed nearest neighbor rule (corresp.).  IEEE transactions on information theory, 14(3):515–516, 1968.
  * [23] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick.  Neuroscience-inspired artificial intelligence.  Neuron, 95(2):245–258, 2017.
  * [24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [25] Xu He and Herbert Jaeger.  Overcoming catastrophic interference using conceptor-aided backpropagation.  In International Conference on Learning Representations, 2018.
  * [26] Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu.  Task agnostic continual learning via meta learning.  arXiv preprint arXiv:1906.05201, 2019.
  * [27] Michael Hersche, Geethan Karunaratne, Giovanni Cherubini, Luca Benini, Abu Sebastian, and Abbas Rahimi.  Constrained few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9057–9067, 2022.
  * [28] Geoffrey Hinton.  Neural networks for machine learning, 2012.
  * [29] Geoffrey E Hinton and Drew Van Camp.  Keeping the neural networks simple by minimizing the description length of the weights.  In Proceedings of the sixth annual conference on Computational learning theory, pages 5–13, 1993.
  * [30] Sepp Hochreiter and Jürgen Schmidhuber.  Simplifying neural nets by discovering flat minima.  Advances in neural information processing systems, 7, 1994.
  * [31] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 831–839, 2019.
  * [32] David A Huffman.  A method for the construction of minimum-redundancy codes.  Proceedings of the IRE, 40(9):1098–1101, 1952.
  * [33] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio.  Fantastic generalization measures and where to find them.  arXiv preprint arXiv:1912.02178, 2019.
  * [34] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon.  Continual learning with node-importance based adaptive group sparse regularization.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [35] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  arXiv preprint arXiv:1910.09217, 2019.
  * [36] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo.  Forget-free continual learning with winning subnetworks.  In International Conference on Machine Learning, pages 10734–10750. PMLR, 2022.
  * [37] Haeyong Kang, Jaehong Yoon, Sultan Rizky Hikmawan Madjid, Sung Ju Hwang, and Chang D Yoo.  On the soft-subnetwork for few-shot class incremental learning.  arXiv preprint arXiv:2209.07529, 2022.
  * [38] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  2017\.
  * [39] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [40] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [41] Abhishek Kumar and Hal Daume III.  Learning task grouping and overlap in multi-task learning.  In Proceedings of the International Conference on Machine Learning (ICML), 2012.
  * [42] Yann LeCun.  The mnist database of handwritten digits.  1998\.
  * [43] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.  Pruning filters for efficient convnets.  arXiv preprint arXiv:1608.08710, 2016.
  * [44] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In Proceedings of the International Conference on Machine Learning (ICML), 2019.
  * [45] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  * [46] Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu.  Hierarchical representations for efficient architecture search, 2017.
  * [47] Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang.  Few-shot class-incremental learning via entropy-regularized data-free replay.  arXiv preprint arXiv:2207.11213, 2022.
  * [48] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [49] Arun Mallya, Dillon Davis, and Svetlana Lazebnik.  Piggyback: Adapting a single network to multiple tasks by learning to mask weights.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [50] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [51] Nicolas Y Masse, Gregory D Grant, and David J Freedman.  Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization.  Proceedings of the National Academy of Sciences, 115(44):E10467–E10475, 2018.
  * [52] Pratik Mazumder, Pravendra Singh, and Piyush Rai.  Few-shot lifelong learning.  arXiv preprint arXiv:2103.00991, 2021.
  * [53] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [54] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka.  Distance-based image classification: Generalizing to new classes at near-zero cost.  IEEE transactions on pattern analysis and machine intelligence, 35(11):2624–2637, 2013.
  * [55] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh.  Linear mode connectivity in multitask and continual learning.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [56] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
  * [57] Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell.  Few-shot class-incremental learning from an open-set perspective.  In European Conference on Computer Vision, pages 382–397. Springer, 2022.
  * [58] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari.  What’s hidden in a randomly weighted neural network?  In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [59] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [60] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro.  Learning to learn without forgetting by maximizing transfer and minimizing interference.  arXiv preprint arXiv:1810.11910, 2018.
  * [61] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [62] Gobinda Saha, Isha Garg, and Kaushik Roy.  Gradient projection memory for continual learning.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [63] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In International Conference on Machine Learning, pages 4548–4557. PMLR, 2018.
  * [64] Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou.  Overcoming catastrophic forgetting with hard attention to the task.  In Proceedings of the International Conference on Machine Learning (ICML), 2018.
  * [65] Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu.  Overcoming catastrophic forgetting in incremental few-shot learning by finding flat minima.  Advances in Neural Information Processing Systems, 34, 2021.
  * [66] Hanul Shin, Jung Kwon Lee, Jaehon Kim, and Jiwon Kim.  Continual learning with deep generative replay.  In Advances in Neural Information Processing Systems (NeurIPS), 2017\.
  * [67] Stanford.  Available online at http://cs231n.stanford.edu/tiny-imagenet-200.zip.  CS 231N, 2021.
  * [68] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong.  Few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12183–12192, 2020.
  * [69] Sebastian Thrun.  A Lifelong Learning Perspective for Mobile Robot Control.  Elsevier, 1995.
  * [70] Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh.  Functional regularisation for continual learning with gaussian processes.  In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  * [71] Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari.  Discovering neural wirings.  Advances in Neural Information Processing Systems, 32, 2019.
  * [72] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi.  Supermasks in superposition.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [73] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.  Large scale incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 374–382, 2019.
  * [74] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv, 2017.
  * [75] Ju Xu and Zhanxing Zhu.  Reinforced continual learning.  In Advances in Neural Information Processing Systems (NeurIPS), 2018\.
  * [76] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney.  Pyhessian: Neural networks through the lens of the hessian.  In 2020 IEEE international conference on big data (Big data), pages 581–590. IEEE, 2020.
  * [77] Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu.  Good subnetworks provably exist: Pruning via greedy forward selection.  In International Conference on Machine Learning, pages 10820–10830. PMLR, 2020.
  * [78] Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sung Ju Hwang.  Scalable and order-robust continual learning with additive parameter decomposition.  In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  * [79] Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang.  Online coreset selection for rehearsal-based continual learning.  In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
  * [80] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [81] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin.  Drawing early-bird tickets: Towards more efficient training of deep networks.  arXiv preprint arXiv:1909.11957, 2019.
  * [82] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In International Conference on Machine Learning, pages 3987–3995. PMLR, 2017.
  * [83] Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu.  Few-shot incremental learning with continually evolved classifiers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12455–12464, 2021.
  * [84] Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan.  Few-shot class-incremental learning by sampling multi-phase tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [85] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski.  Deconstructing lottery tickets: Zeros, signs, and the supermask.  In Advances in Neural Information Processing Systems (NeurIPS), 2019\.
  * [86] Kai Zhu, Yang Cao, Wei Zhai, Jie Cheng, and Zheng-Jun Zha.  Self-promoted prototype refinement for few-shot class-incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6801–6810, 2021.

|  Haeyong Kang (S’05) received the M.S. degree in Systems and Information
Engineering from University of Tsukuba in 2007. From April 2007 to October
2010, he worked as an associate research engineer at LG Electronics. With
working experiences at Korea Institute of Science and Technology (KIST) and
the University of Tokyo, He is currently pursuing the Ph.D at School of
Electrical Engineering, KAIST. His current research interests include unbiased
machine learning and continual learning.
---|---
|  Jaehong Yoon He received the B.S. and M.S. degrees in Computer Science from
Ulsan National Institute of Science and Technology (UNIST), and received the
Ph.D. degree in the School of Computing from Korea Advanced Institute of
Science and Technology (KAIST). He is currently working as a postdoctoral
research fellow at KAIST. His current research interests include efficient
deep learning, on-device learning, and learning with real-world data.
---|---
|  Sultan Rizky Madjid received a B.S. degree in Electrical Engineering with a
double major in Mechanical Engineering from KAIST in 2021 and an M.S. degree
in Electrical Engineering from KAIST in 2023. His research interests include
model compression, sparse representations in deep learning, and continual
learning.
---|---
|  Sung Ju Hwang He received the B.S. degree in Computer Science and
Engineering from Seoul National University. He received the M.S. and Ph.D.
degrees in Computer Science from The University of Texas at Austin. From
September 2013 to August 2014, he was a postdoctoral research associate at
Disney Research. From September 2013 to December 2017, he was an assistant
professor in the School of Electric and Computer Engineering at UNIST. Since
2017, he has been on the faculty at the Korea Advanced Institute of Science
and Technology (KAIST), where he is currently a KAIST Endowed Chair Professor
in the Kim Jaechul School of Artificial Intelligence and School of Computing
at KAIST.
---|---
|  Chang D. Yoo (Senior Member, IEEE) He received the B.S. degree in
Engineering and Applied Science from the California Institute of Technology,
the M.S. degree in Electrical Engineering from Cornell University, and the
Ph.D. degree in Electrical Engineering from the Massachusetts Institute of
Technology. From January 1997 to March 1999, he was Senior Researcher at Korea
Telecom (KT). Since 1999, he has been on the faculty at the Korea Advanced
Institute of Science and Technology (KAIST), where he is currently a Full
Professor with tenure in the School of Electrical Engineering and an Adjunct
Professor in the Department of Computer Science. He also served as Dean of the
Office of Special Projects and Dean of the Office of International Relations.
---|---

##  Appendix A Experimental Details of WSN for TIL

We followed similar experimental setups (architectures and hyper-parameters)
described in [62] for baseline comparisons and explained in [16] for SOTA
comparisons.

###  A.1 Architecture Details

All the networks for our experiments are implemented in a multi-head setting.
Two-layered MLP: In conducting the PMNIST experiments, we are following the
exact setup as denoted by [62] fully-connected network with two hidden layers
of 100 [48].

Reduced ResNet18: In conducting the 5-Dataset experiments, we use a smaller
version of ResNet18 with three times fewer feature maps across all layers as
denoted by [48].

Modified LeNet: In conducting the Omniglot Rotation and CIFAR-100 Superclass
experiments, we use a large variant of LeNet as the base network with
64-128-2500-1500 neurons based on [78].

Modified AlexNet: In conducting the split CIFAR-100 dataset, we use a modified
version of AlexNet similar to [64, 62].

4 Conv layers and 3 Fully connected layers: For TinyImageNet, we use the same
network architecture as [20, 16].

###  A.2 List of Hyperparameters

Table VI: List of hyperparameters for the baselines and our WSN. Here, ’lr’
and ’optim’ represents (initial) learning rate and optimizer used for
training. We represent PMNIST as ’perm’, 5-Datasets as ’5data’, Omniglot
Rotation as ’omniglot’, CIFAR-100 Split as ’cifar100-split’, and CIFAR-100
Superclass as ’cifar100-sc’.

Methods | Hyperparameters
---|---
EWC | lr : 0.03 (perm)
| optim : sgd
| regularization coefficient : 1000 (perm)
GPM | lr : 0.01(perm, omniglot), 0.1 (5data)
| optim : sgd
|  $n_{s}$ : 300 (perm), 100 (5data), 125 (omniglot)
PackNet | lr = 0.001 (perm, 5data, omniglot)
| optim : adam
|  $c$ : 0.1 (perm), 0.2 (5data), 0.02 (omniglot)
WSN (ours) | lr = 0.001
| (perm, 5data, omniglot,
| cifar100-split, cifar100-sc, tinyimagenet)
| optim : adam

Table VI details the hyperparameter setup for the baselines and our approach.
$n_{s}$ in GPM denotes the number of random training examples sampled from the
replay buffer to construct the representation matrix for each architecture
layer.

(a) $c=5\%$ | (b) $c=10\%$
---|---
(c) $c=30\%$ | (d) $c=50\%$
Figure 15: Average Binary Map Correlation on Sequence of PMNIST Experiments: the binary maps get overlapped with prior ones as the number of tasks increases. (a) $c=5\%$ | (b) $c=10\%$
---|---
(c) $c=30\%$ | (d) $c=50\%$
Figure 16: Average Binary Map Correlation on Sequence of 5-Dataset Experiments: the binary maps get overlapped with prior ones as the number of tasks increases. (a) WSN, $c=5\%$ | (b) SoftNet, $c=5\%$
---|---
(c) WSN, $c=70\%$ | (d) SoftNet, $c=70\%$
Figure 17: Average Forward Transfer Matrix on Permuted MNIST. WSN / SoftNet with c = 5% and c = 70%, respectively (a) WSN, $c=5\%$ | (b) SoftNet, $c=5\%$
---|---
(c) WSN, $c=70\%$ | (d) SoftNet, $c=70\%$
Figure 18: Average Forward Transfer Matrix on 5-Dataset. WSN / SoftNet with c
= 5% and c = 70%, respectively

##  Appendix B Experimental Details of SoftNet for FSCIL

###  B.1 Datasets

The following datasets are utilized for comparisons with current state-of-the-
art:

CIFAR-100 In CIFAR-100, each class contains $500$ images for training and
$100$ images for testing. Each image has a size of $32\times 32$. Here, we
follow an identical FSCIL procedure as in [65], dividing the dataset into a
base session with 60 base classes and eight novel sessions with a 5-way 5-shot
problem on each session.

miniImageNet miniImageNet consists of RGB images from 100 different classes,
where each class contains $500$ training images and $100$ test images of size
$84\times 84$. Initially proposed for few-shot learning problems, miniImageNet
is part of a much larger ImageNet dataset. Compared with CIFAR-100, the
miniImageNet dataset is more complex and suitable for prototyping. The setup
of miniImageNet is similar to that of CIFAR-100. To proceed with our
evaluation, we follow the procedure described in [65], incorporating 60 base
classes and eight novel sessions through 5-way 5-shot problems.

CUB-200-2011 CUB-200-2011 contains $200$ fine-grained bird species with
$11,788$ images with varying images for each class. To proceed with
experiments, we split the dataset into $6,000$ training images, and $6,000$
test images as in [68]. During training, We randomly crop each image to size
$224\times 224$. We fix the first 100 classes as base classes, utilizing all
samples in these respective classes to train the model. On the other hand, we
treat the remaining 100 classes as novel categories split into ten novel
sessions with a 10-way 5-shot problem in each session.

###  B.2 Comparisons with SOTA

We compare SoftNet with the following state-of-art-methods on TOPIC class
split [68] of three benchmark datasets - CIFAR100 (Table VII), miniImageNet
(Table VIII), and CUB-200-2011 (Table IX).

Table VII: Classification accuracy of ResNet18 on CIFAR-100 for 5-way 5-shot
incremental learning with the same class split as in TOPIC [12]. ∗ denotes the
results reported from [65]. † represents our reproduced results.

Method | sessions |  The gap with cRT
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
cRT [65]∗ | 72.28 | 69.58 | 65.16 | 61.41 | 58.83 | 55.87 | 53.28 | 51.38 | 49.51 |
TOPIC [12] | 64.10 | 55.88 | 47.07 | 45.16 | 40.11 | 36.38 | 33.96 | 31.55 | 29.37 | -20.14
CEC [83] | 73.07 | 68.88 | 65.26 | 61.19 | 58.09 | 55.57 | 53.22 | 51.34 | 49.14 | -0.37
F2M [65] | 71.45 | 68.10 | 64.43 | 60.80 | 57.76 | 55.26 | 53.53 | 51.57 | 49.35 | -0.16
LIMIT [84] | 73.81 | 72.09 | 67.87 | 63.89 | 60.70 | 57.77 | 55.67 | 53.52 | 51.23 | +1.72
MetaFSCIL [13] | 74.50 | 70.10 | 66.84 | 62.77 | 59.48 | 56.52 | 54.36 | 52.56 | 49.97 | +0.46
ALICE [57] | 79.00 | 70.50 | 67.10 | 63.40 | 61.20 | 59.20 | 58.10 | 56.30 | 54.10 | +4.59
Entropy-Reg [47] | 74.40 | 70.20 | 66.54 | 62.51 | 59.71 | 56.58 | 54.52 | 52.39 | 50.14 | +0.63
C-FSCIL [27] | 77.50 | 72.45 | 67.94 | 63.80 | 60.24 | 57.34 | 54.61 | 52.41 | 50.23 | +0.72
FSLL [52] | 64.10 | 55.85 | 51.71 | 48.59 | 45.34 | 43.25 | 41.52 | 39.81 | 38.16 | -11.35
HardNet (WSN), $c=50\%$ | 78.35 | 74.12 | 70.13 | 65.88 | 62.74 | 59.56 | 57.98 | 56.31 | 54.32 | +4.81
HardNet (WSN), $c=80\%$ | 79.27 | 75.38 | 71.11 | 66.68 | 63.32 | 60.06 | 58.16 | 56.40 | 54.31 | +4.80
SoftNet, $c=50\%$ | 79.88 | 75.54 | 71.64 | 67.47 | 64.45 | 61.09 | 59.07 | 57.29 | 55.33 | +5.82
SoftNet, $c=80\%$ | 80.33 | 76.23 | 72.19 | 67.83 | 64.64 | 61.39 | 59.32 | 57.37 | 54.94 | +5.43

Table VIII: Classification accuracy of ResNet18 on miniImageNet for 5-way
5-shot incremental learning with the same class split as in TOPIC [12]. ∗
denotes results reported from [65].

Method | sessions |  The gap with cRT
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
cRT [65]∗ | 72.08 | 68.15 | 63.06 | 61.12 | 56.57 | 54.47 | 51.81 | 49.86 | 48.31 | -
TOPIC [12] | 61.31 | 50.09 | 45.17 | 41.16 | 37.48 | 35.52 | 32.19 | 29.46 | 24.42 | -23.89
IDLVQ-C [10] | 64.77 | 59.87 | 55.93 | 52.62 | 49.88 | 47.55 | 44.83 | 43.14 | 41.84 | -6.47
CEC [83] | 72.00 | 66.83 | 62.97 | 59.43 | 56.70 | 53.73 | 51.19 | 49.24 | 47.63 | -0.68
F2M [65] | 72.05 | 67.47 | 63.16 | 59.70 | 56.71 | 53.77 | 51.11 | 49.21 | 47.84 | -0.43
LIMIT [84] | 73.81 | 72.09 | 67.87 | 63.89 | 60.70 | 57.77 | 55.67 | 53.52 | 51.23 | +2.92
MetaFSCIL [13] | 72.04 | 67.94 | 63.77 | 60.29 | 57.58 | 55.16 | 52.90 | 50.79 | 49.19 | +0.88
ALICE [57] | 80.60 | 70.60 | 67.40 | 64.50 | 62.50 | 60.00 | 57.80 | 56.80 | 55.70 | +7.39
C-FSCIL [27] | 76.40 | 71.14 | 66.46 | 63.29 | 60.42 | 57.46 | 54.78 | 53.11 | 51.41 | +3.10
Entropy-Reg [47] | 71.84 | 67.12 | 63.21 | 59.77 | 57.01 | 53.95 | 51.55 | 49.52 | 48.21 | -0.10
Subspace Reg. [1] | 80.37 | 71.69 | 66.94 | 62.53 | 58.90 | 55.00 | 51.94 | 49.76 | 46.79 | -1.52
FSLL [52] | 66.48 | 61.75 | 58.16 | 54.16 | 51.10 | 48.53 | 46.54 | 44.20 | 42.28 | -6.03
HardNet (WSN), $c=80\%$ | 78.70 | 72.55 | 68.26 | 64.45 | 61.74 | 58.93 | 55.99 | 54.09 | 52.74 | +4.43
HardNet (WSN), $c=87\%$ | 79.17 | 73.05 | 69.16 | 65.43 | 62.61 | 59.31 | 56.73 | 54.69 | 53.47 | +5.16
HardNet (WSN), $c=90\%$ | 79.15 | 72.03 | 68.76 | 65.32 | 62.00 | 58.21 | 56.52 | 53.66 | 53.07 | +4.76
SoftNet, $c=80\%$ | 79.37 | 74.31 | 69.89 | 66.16 | 63.40 | 60.75 | 57.62 | 55.67 | 54.34 | +6.03
SoftNet, $c=87\%$ | 79.77 | 75.08 | 70.59 | 66.93 | 64.00 | 61.00 | 57.81 | 55.81 | 54.68 | +6.37
SoftNet, $c=90\%$ | 79.72 | 74.25 | 70.00 | 66.35 | 63.19 | 60.04 | 57.36 | 55.38 | 54.14 | +5.83

Table IX: Classification accuracy of ResNet18 on CUB-200-2011 for 10-way
5-shot incremental learning (TOPIC class split [68]). ∗ denotes results
reported from [65]. † represents our reproduced results.

Method | sessions |  The gap with cRT
---|---|---
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11
cRT [65]∗ | 77.16 | 74.41 | 71.31 | 68.08 | 65.57 | 63.08 | 62.44 | 61.29 | 60.12 | 59.85 | 59.30 | -
TOPIC [12] | 68.68 | 62.49 | 54.81 | 49.99 | 45.25 | 41.40 | 38.35 | 35.36 | 32.22 | 28.31 | 26.28 | -34.80
SPPR [86] | 68.68 | 61.85 | 57.43 | 52.68 | 50.19 | 46.88 | 44.65 | 43.07 | 40.17 | 39.63 | 37.33 | -21.97
CEC [83] | 75.85 | 71.94 | 68.50 | 63.50 | 62.43 | 58.27 | 57.73 | 55.81 | 54.83 | 53.52 | 52.28 | -7.02
F2M [65] | 77.13 | 73.92 | 70.27 | 66.37 | 64.34 | 61.69 | 60.52 | 59.38 | 57.15 | 56.94 | 55.89 | -3.41
LIMIT [84] | 75.89 | 73.55 | 71.99 | 68.14 | 67.42 | 63.61 | 62.40 | 61.35 | 59.91 | 58.66 | 57.41 | -1.89
MetaFSCIL [13] | 75.90 | 72.41 | 68.78 | 64.78 | 62.96 | 59.99 | 58.30 | 56.85 | 54.78 | 53.82 | 52.64 | -6.66
ALICE [57] | 77.40 | 72.70 | 70.60 | 67.20 | 65.90 | 63.40 | 62.90 | 61.90 | 60.50 | 60.60 | 60.10 | -0.02
Entropy-Reg [47] | 75.90 | 72.14 | 68.64 | 63.76 | 62.58 | 59.11 | 57.82 | 55.89 | 54.92 | 53.58 | 52.39 | -6.91
FSLL [52] | 72.77 | 69.33 | 65.51 | 62.66 | 61.10 | 58.65 | 57.78 | 57.26 | 55.59 | 55.39 | 54.21 | -6.87
HardNet (WSN), $c=88\%$ | 76.89 | 73.40 | 69.77 | 66.15 | 64.00 | 60.98 | 59.56 | 58.05 | 56.05 | 55.84 | 55.20 | -4.10
HardNet (WSN), $c=90\%$ | 77.23 | 73.62 | 70.20 | 66.36 | 64.32 | 61.40 | 59.86 | 58.28 | 56.36 | 55.88 | 55.30 | -4.00
HardNet (WSN), $c=93\%$ | 77.76 | 73.97 | 70.41 | 66.60 | 64.47 | 61.35 | 59.80 | 58.18 | 56.17 | 55.73 | 55.18 | -4.12
SoftNet, $c=88\%$ | 78.14 | 74.61 | 71.28 | 67.46 | 65.14 | 62.39 | 60.84 | 59.17 | 57.41 | 57.12 | 56.64 | -2.66
SoftNet, $c=90\%$ | 78.07 | 74.58 | 71.37 | 67.54 | 65.37 | 62.60 | 61.07 | 59.37 | 57.53 | 57.21 | 56.75 | -2.55
SoftNet, $c=93\%$ | 78.11 | 74.51 | 71.14 | 62.27 | 65.14 | 62.27 | 60.77 | 59.03 | 57.13 | 56.77 | 56.28 | -3.02