11institutetext: Computer Vision Center, Barcelona, Spain 22institutetext:
Universitat Autonoma de Barcelona, Barcelona, Spain 33institutetext: IDEAS
NCBR, Warsaw, Poland 44institutetext: MICC, University of Florence, Florence,
Italy
44email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Exemplar-free Continual Representation Learning via Learnable Drift
Compensation

Alex Gomez-Villa 1122 Dipam Goswami 1122 Kai Wang Corresponding author11
Andrew D. Bagdanov 44 Bartlomiej Twardowski 112233 Joost van de Weijer 1122

###### Abstract

Exemplar-free class-incremental learning using a backbone trained from scratch
and starting from a small first task presents a significant challenge for
continual representation learning. Prototype-based approaches, when
continually updated, face the critical issue of semantic drift due to which
the old class prototypes drift to different positions in the new feature
space. Through an analysis of prototype-based continual learning, we show that
forgetting is not due to diminished discriminative power of the feature
extractor, and can potentially be corrected by drift compensation. To address
this, we propose Learnable Drift Compensation (LDC), which can effectively
mitigate drift in any moving backbone, whether supervised or unsupervised. LDC
is fast and straightforward to integrate on top of existing continual learning
approaches. Furthermore, we showcase how LDC can be applied in combination
with self-supervised CL methods, resulting in the first exemplar-free semi-
supervised continual learning approach. We achieve state-of-the-art
performance in both supervised and semi-supervised settings across multiple
datasets. Code is available at https://github.com/alviur/ldc.

###### Keywords:

Continual Learning Semi-Supervised Learning Self-Supervised Learning Exemplar-
Free Class Incremental Learning

##  1 Introduction

The standard supervised learning paradigm assumes that data are available in a
single training session. This belies, however, the reality of many real-world
scenarios in which machine learning methods must deal with data that arrive in
a continuous and non-stationary stream of tasks. These shifting distributions
lead to catastrophic forgetting if approached naively (for instance, with
continuing stochastic gradient descent) [37]. Continual Learning (CL)
considers the design of algorithms capable of learning from such streams of
non-stationary data.

Several techniques have been proposed for continually training models, such as
regularization [31, 27, 54, 2], exemplar replay [43, 5, 51, 23, 11], and
growing architectures [46]. One group of methods focuses on the more difficult
exemplar-free continual learning scenario where the storing of exemplars is
prohibited [59, 60, 18, 41, 52]. This setting is relevant in applications
where storing previous-task samples is not feasible due to memory limitations,
legal constraints arising from new data privacy regulations (like GDPR), or
when handling sensitive data such as medical imaging where patient identity
must be protected.

One of the main limitations of exemplar-free CL methods is that, as the
backbone is updated during the training of new tasks, the network experiences
semantic feature drift [52]. This feature drift results in catastrophic
forgetting and a significant drop in performance. As a consequence, most
exemplar-free methods are evaluated in Warm Start settings (starting from a
large first task or from a pretrained network) to start from a strong backbone
and thus avoid significant semantic drift of backbone network [52, 18, 59, 58,
60, 38, 23, 51, 1]. In fact, a recent paper [41] showed that a simple strategy
combined with a frozen backbone after learning the first task can lead to very
competitive results in this Warm Start settings. In this paper, we focus on
the exemplar-free Cold Start setting [33] in which we train a network from
scratch, starting with a small task, and continually update the
backbone.111This setting is predominant in exemplar-based continual learning
but hardly explored for exemplar-free methods.

Class Prototype (CP) accumulation is among the most successful approaches to
exemplar-free continual learning [18, 25, 38, 40]. CP methods store class
prototypes computed using training data and then use them to classify using a
distance classifier in feature space. Class Prototype methods are attractive
since they can be agnostically applied to any feature extractor, such as
multilayer perceptrons, convolutional neural networks, or transformers. CP
methods have been successfully applied in Warm Start settings [18, 38, 41] but
are expected to struggle in Cold Start settings due to significant semantic
drift.

In this article, we aim to apply class prototype methods in Cold Start
settings. The main challenge in this setting is the semantic drift of the
feature backbone, due to which class prototypes become progressively
invalidated since the mean position of each class moves with each new task and
cannot be corrected without access to old data in exemplar-free settings.
However, in an initial analysis of feature drift (see Fig. 1), we observe that
an oracle that perfectly compensates for feature drift leads to a significant
recovery of performance. This is an important observation since it shows that
semantic drift has not diminished the discriminative power of the feature
representations. Consequently, we show that if drift is correctly estimated,
feature drift compensation can correct for most of the performance drop
resulting from it. Existing drift estimation methods [52, 49] assume that the
drift can locally be approximated with a translation. Instead, we propose a
simple Learnable Drift Compensation (LDC) approach to update old class
prototypes which has not prior assumptions on the nature of the drift. The
learnable compensation maps the previous task features to the current feature
space via a forward projector network. This compensation can be learned from
only the stored old class prototypes and does not require class labels. Owing
to the label-agnostic nature of LDC, we propose the first semi-supervised
exemplar-free approach, which uses the labeled data only to compute prototypes
and all available data to learn a robust forward projector. The main
contributions of this work are as follows:

  * •

We show that a large part of forgetting in prototype-based methods can be
addressed by prototype compensation. Very little actual forgetting of features
crucial for previous tasks is occurring, which motivates the need to develop
drift compensation methods for continual representation learning.

  * •

We propose a new exemplar-free continual learning method, which we call
Learnable Drift Compensation (LDC), that is based on class prototype
accumulation, can compensate the drift of a moving backbone, and is fast and
easy to add on top of CP-based methods.

  * •

We experimentally demonstrate that LDC is better than previous drift
compensation methods for supervised exemplar-free CL setup. We show that LDC
can be applied to any self-supervised CL method to create the first, to the
best of our knowledge, exemplar-free semi-supervised CL method.

##  2 Related Work

Continual Learning [37, 27] addresses the issue of catastrophic forgetting
when training model on a non-i.i.d stream of data [36]. Various methods [36,
57] have been proposed to tackle this problem, including parameter isolation
methods [46, 47, 35], regularization techniques [2, 27, 31], and rehearsal
strategies [43, 7]. In this paper, we focus on class-incremental learning
[50], where the learner has no access to the task oracle during inference,
without the need to store any exemplars.

Exemplar-free Class Incremental Learning. In this paper, we consider the
challenging exemplar-free CIL setting. PASS [59] aligns prototypes of new
classes with those of previous tasks via augmentation with noise. SSRE [61]
focuses on continually expanding the model’s representations to accommodate
new classes. FeTrIL [41] translates old prototype features using the
difference between old and new prototypes to create pseudo-features of old
data in the new space. In FeCAM [18], the heterogeneous distribution of
features in class-incremental learning is exploited using anisotropic
Mahalanobis distance instead of Euclidean metric.

Continual Representation Learning.

CURL [42] performs continual unsupervised representation learning and is based
on a variational autoencoder whose latent space can be sampled to replay
samples of old classes. CURL is applicable only to small-sized image datasets,
requires task boundaries, and is evaluated only on simple classification task.
Using self-supervised learning to enhance the learning process of a sequence
of supervised tasks was introduced in prior works [53, 59]. Their aim is not
to learn directly from unlabeled data, but rather to leverage self-supervised
learning to enrich the feature representation further. Similarly, several
studies [14, 6, 32, 4, 39] have used pre-trained models with self-supervision
to enhance incremental average classification accuracy through techniques like
data augmentation, distillation, and even exemplars. Furthermore, models are
trained in self-supervised way in the class-incremental learning (CL) setting
without relying on exemplars, as seen in [13, 15].

Continual unsupervised learning can be straightforwardly extended to address
the challenge of semi-supervised continual learning, where training phases
abstain from utilizing labels entirely, similar to the approach in CaSSLe
[13]. However, a limitation of this approach is mitigated by NNCSL [26], which
builds upon the PAWS [3] for continual learning. Nonetheless, it is worth
noting that these methods typically rely on exemplars to achieve satisfactory
performance.

Drift estimation. An obstacle to exemplar-free methods is the drift of
backbone features, which can result in catastrophic forgetting. Semantic drift
compensation (SDC) [52] addresses _drift compensation_ by computing the drift
of features using new task data and approximating the drift of old class
prototypes based on this information. Subsequent work [49] extended SDC by
also considering feature drift. Elastic feature consolidation (EFC) [33]
regularizes the drift in a direction highly relevant to the previous task
using the empirical feature matrix. Recently, Adversarial drift compensation
[19] was proposed to generate adversarial samples from new task data which
behaves as pseudo-exemplars and are used for estimating drift of old classes.
Our method shares a similar objective of estimating feature backbone drift,
but we propose a different approach by suggesting the use of a dedicated
network for drift compensation.

Memory efficient CIL through feature adaptation (MEA) [24] addresses the
problem of feature drift. It stores the features after the backbone and
corresponding labels. A learned mapping is applied to translate stored
features into the new space to compensate for drift. Unlike our method, MEA
requires labels of features, making it unsuitable for unsupervised and semi-
supervised settings. Additionally, MEA stores an extensive amount of features
for all old classes, leading to higher memory requirements.

##  3 Preliminaries

###  3.1 The Continual Learning Problem

We consider a class-incremental learning setup where a model learns the
dataset $D$ split into a sequence of tasks $T=\\{t_{1},...,t_{s}\\}$. During
the training of each task, the model only has access to the data in the split
$t$ (exemplar-free setup). Each task contains a set of $m$ classes
$C^{t}=\\{c^{t}_{1},c^{t}_{2},...,c^{t}_{m}\\}$ and there is no overlap
between classes of different tasks: $C^{t}\cap C^{s}=\emptyset$ for $t\neq s$.

Samples inside a task $t$ can contain different data depending on the learning
method. For supervised learning, each sample is a pair $(x_{i},y_{i})$ where
$x_{i}$ is an image of class $y_{i}\in C^{t}$. In the semi-supervised setting,
$L$ samples have an assigned label $y_{i}$ and $U$ samples are unlabeled
images $x_{i}$ (usually $L_{t}<<U_{t}$).

###  3.2 Motivation

####  3.2.1 Not all forgetting is catastrophic.

Semantic feature drift, as described in prior work [52], manifests when a
model learns sequentially. In continual learning, the prototypes computed for
task $t$ may lose relevance for future tasks because the position of class $c$
after training on task $t$ differs from that in task $t-1$. This phenomenon is
independent of any learning paradigm (whether supervised or unsupervised) and
impacts all non-frozen models during incremental training.

To demonstrate the impact of feature drift on Class-Prototype (CP) methods, we
conduct experiments using a ResNet-18 backbone trained in a Cold Start setting
on 10-task CIFAR-100 and evaluate the performance of classes from the initial
3 tasks after training on all tasks. Fig. 1(a) illustrates the backbone
trained with the supervised continual Learning without Forgetting (LwF)
strategy [31], while Fig. 1(b) shows the backbone trained with the
unsupervised continual learning (CaSSLe) method [13]. We use a Nearest Class
Mean (NCM) [43] classifier to predict the labels of the test set for the
classes encountered up to each task $t$.

In Fig. 1, we show the NCM accuracy using four types of prototypes:

  * •

Naive prototypes ($p_{naive}$): These are prototypes without any update or
correction. More specifically, prototypes of classes $C^{t}$ are computed at
the end of $t$ and used as is afterward.

  * •

Corrected prototypes ($p_{corrected}$): At the end of each task, all stored
prototype positions are corrected using LDC (see Section 3.3), without the use
of exemplars. This correction strategy is the main focus of this paper.

  * •

Oracle prototypes ($p_{oracle}$): The true position of all old class
prototypes is computed by utilizing all the training data from old tasks at
the end of $t$.

  * •

Jointly trained prototypes ($p_{joint}$): The prototypes are computed using a
model which is based on incremental joint training. This serves as the upper
bound.

We observe that while a substantial gap in performance exists between the
$p_{naive}$ and $p_{joint}$ accuracy in the two plots, which is generally
referred to as catastrophic forgetting, a major part of this forgetting can be
mitigated using $p_{oracle}$. This indicates that a significant amount of
forgetting can be attributed to the drift in class prototypes which can be
reduced by correcting the position of old prototypes. For example, for the
case of task 1 and 2 (red lines) we can observe that the features that are
discriminative for the task 1&2 are present in the backbone and have not been
significantly impaired by the learning of subsequent tasks. However,
regularization methods (like LwF and CaSSLe) are not enough for mitigating
forgetting. This motivates us to explore drift compensation. We will
demonstrate that correcting the prototypes with our method LDC improves the
performance of $p_{naive}$ to $p_{corrected}$ by big margins (around 38% for
tasks 1&2&3 in unsupervised setting).

Figure 1: Last task accuracy of class-prototype accumulation strategies using
an NCM classifier in the 10-task CIFAR-100 scenario. The figure row shows
regularization-based methods (a) LwF and (b) CaSSLe. Here we show result for
two settings: Tasks 1&2 shows how the performance on the first two tasks
evolves while incrementally training all tasks. Analogously, Tasks 1&2&3 shows
how the performance on the first three tasks evolves over training of all ten
tasks.

####  3.2.2 Limitations of Semantic Drift Compensation.

The closest methods to LDC rely on SDC. However, this method has an important
weakness: it assumes a fixed local transformation for drift (namely locally it
can be captured by translations). If semantic drift follows a different type
of transformation, such as scaling, this assumption will lead to errors in the
estimated drift.

To illustrate this point, Fig. 2 presents a toy example showcasing
translations, rotations, and scaling applied to three sample distributions. In
this experiment, we aim to estimate the true mean of $C_{1}$ after it has
drifted by using the transformation observed in $D_{t_{2}}$ as a reference.
When only translations are applied (first row), SDC perfectly estimates the
distribution’s drift. However, under transformations that include rotations
and scaling (second row), SDC incorrectly approximates the true mean of the
distributions.

Figure 2: Feature drift estimation after applying random translations,
rotations and scaling on three sample 2D distributions. We aim to estimate the
true mean of $C_{1}$ at the end of $t_{2}$ using $D_{t_{2}}$. SDC assumes
locally that transformations can be captured by translations. LDC can handle
rotation and scaling in feature space. Note that LDC (cyan) more accurately
approximates the real distribution mean (green).

###  3.3 Learnable Drift Compensation

Figure 3: Learnable Drift Compensation. We train the model on the current task
data using regularization-based, self-supervised or supervised continual
representation learning methods. After training the new feature extractor
$f_{t}$, we learn a forward projector $p_{F}$ by minimizing the mean squared
error between the projected features from $f_{t-1}$ and the features from
$f_{t}$. We use the learned projector to compensate for the drift of the old
class prototypes in the new feature space.

LDC is applied at the end of each training session $t$. Let $f_{\theta}$ be a
feature extractor parameterized by $\theta$. To correct the semantic drift, we
learn a forward projector $p_{F}^{t}$ to map the features from the old feature
space of $f_{\theta}^{t-1}$ to the new feature space of $f_{\theta}^{t}$. In
order to learn the drift in the feature space, we use the data of the current
task $D_{t}$ and minimize the mean squared error (MSE) between the projected
features of $f_{\theta}^{t-1}$ and $f_{\theta}^{t}$. Hence, the loss function
to learn $p_{F}^{t}$ is:

$\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}(p_{F}^{t}(f_{\theta}^{t-1}(x_{i}))-f_{\theta}^{t}(x_{i}))^{2},$
(1)

where $f_{\theta}^{t}$ and $f_{\theta}^{t-1}$ are frozen. After this
optimization, the old prototypes $P_{t-1}^{c}$ for all classes seen till task
$t-1$ are updated using Eq. (2):

$P_{t}^{c}=p_{F}^{t}(P_{t-1}^{c}).$ (2)

This process is repeated each time $f_{\theta}$ is trained in a new task as
shown in Fig. 3. Hence, if we perform class-incremental learning in $T$ tasks,
$T-1$ projectors are trained. We do not need to store any old images or
features, or use any labels of the current data; we simply store only the old
class prototypes and update them using the learned current task projector. The
projector $p_{F}^{t}$ can be any trainable mapping function. However, we found
that a linear layer performs best in this task (see section 4.4). We
summarized the LDC algorithm in the supplementary material.

Nearest Class Mean (NCM) Classifier. NCM classifier [43, 25, 18, 52] has been
found to be very effective for classification in CL settings. We use a NCM
classifier on top of the trained feature extractor $f_{\theta}^{t}$ which
classifies the test images $x$ based on their distance to all the class
prototypes $P_{t}^{c}$ as follows:

$y^{\ast}=\operatorname*{argmin}\limits_{y=1,\dots,Y^{t}}\|f_{\theta}^{t}(x)-P_{t}^{y}\|,$
(3)

where $Y^{t}$ is the set of classes seen up to task $t$ and
$P_{t}^{y}=p_{F}^{t}(P_{t-1}^{y})\,\forall y\in Y^{t-1}$.

###  3.4 Continual Training Strategies

Our proposed method for drift compensation is general and can be combined with
many existing regularization methods commonly used in CL. LDC can be applied
at the end of training session $t$ with the only requirement being access to
the old feature extractor $f_{t-1}$ and the old class prototypes
$P_{t-1}^{c}$. A projector $p_{F}^{t}$ must be optimized using Eq. (1) with
$D_{t}$, and then old prototypes are updated using Eq. (2) and stored along
with current ones. Finally, a new training session begins on $D_{t+1}$ using
any CL strategy (e.g., LwF). It is important to note that our method is
independent of the CL strategy used to retain knowledge in the backbone, as
LDC is a prototype correction method, not a CL approach. Here, we explain how
to extend several existing methods with LDC.

Supervised CL. Regularization-based methods retain knowledge while learning
new concepts by preventing weights or activations from drifting far from the
old model. For instance, LwF [31] aims to match the output of the previous
model softmax layer $h_{\theta}^{t-1}(x)$ with the current model outputs
$h_{\theta}^{t}(x)$ using knowledge distillation [22] as follows:

$\displaystyle\mathcal{L}=\mathcal{L}_{ce}(h_{t}(x),y)+\lambda\mathcal{L}_{ce}(h_{t-1}(x),h_{t}(x)),$
(4)

where $\mathcal{L}_{ce}$ refers to the cross-entropy loss and $\lambda$
denotes the regularization strength. After the training session, the
prototypes $p_{t}^{c}$ of $D_{t}$ are computed using the labels and added to
the prototype pool for NCM classification. This work conducts experiments
using LwF as it is a well-established and flexible method in the CL community.
However, any exemplar-free supervised CL method can replace the LwF method.
Regularization-based methods are particularly well-suited for LDC as they
already maintain a copy of the old model at each task.

Self-supervised CL. Exemplar-free unsupervised CL methods are based on a
projected regularization strategy at the feature level. These approaches
perform self-supervised learning (SSL) in $D_{t}$ while preserving similarity
between old-fixed $f_{t-1}(D_{t})$ and new-projected features
$p(f_{t}(D_{t}))$. PFR [15] project features after the backbone and CaSSLe
[13] after a post-backbone layer. POCON [16] uses two networks, one expert
network performs SSL in $D_{t}$ and the other network (main) distills the
knowledge from the expert and the old main network using projected features
from the current main network.

The CP accumulation strategy cannot be directly applied in exemplar-free self-
supervised CL due to the absence of labels during training. However, if labels
become available after training, we can compute prototypes, store them, and
even correct them as in the supervised case. After the SSL CL session, we
train a projector using the frozen SSL feature extractors and correct the old
prototype positions accordingly.

Semi-supervised CL. If some labels are accessible during training, we operate
within the semi-supervised learning setup. Here, we propose using LDC with
exemplar-free self-supervised CL methods, which to the best of our knowledge
firstly achieve the exemplar-free continual semi-supervised learning. To
implement this, we follow a similar procedure as in the self-supervised CL
case but utilize the available labels to compute prototypes. It is important
to note that the performance of LDC remains independent of the number of
available labels, meaning that the primary source of error would stem from the
computation of the prototype, which typically exhibits low variability.

##  4 Experimental Results

###  4.1 Datasets

We use several publicly available datasets for our experiments. CIFAR-100 [29]
contains 100 classes of $32\times 32$ images with 500 training and 100 testing
images per class. Tiny-ImageNet [30] contains 200 classes of $64\times 64$
images with a total of 100K training and 10K testing images. ImageNet100 is a
subset of one hundred classes from ImageNet [45] containing $224\times 224$
images with a total of 130K training and 5K testing images. Stanford Cars [28]
consists of 196 finegrained categories of cars and has 8144 images for
training and also for testing. We equally split all the datasets into 10
tasks, which is different from the conventional Warm Start settings [59, 60,
18, 41, 52].

###  4.2 Supervised Continual Learning

Baseline methods. Since the existing methods were originally proposed for Warm
Start settings, we re-implement all methods in Table 1 using PyCIL [57]. We
implement a standard distillation baseline method, LwF [31] and also use NCM
[43] classification naively. We use SDC [52] with the models trained with LwF.
Note that SDC was originally proposed in Warm Start settings with feature
distillation. Here, we adapt SDC for Cold Start settings with logits
distillation (LwF). We include recent methods like PASS [59], FeTrIL [41],
FeCAM [18] and EFC [33] in our comparison. While LwF, PASS and EFC are
continual representation learning methods, FeCAM and FeTrIL freeze the model
after the first task and continually learn the classifier. We do not compare
to the methods [34, 61, 48, 60] proposed in Warm Start settings since they
showed poor performance compared to FeTrIL and FeCAM. Following recent
practice [56, 38, 55, 20] of using pre-trained ViTs for CIL, we use a ViT-B/16
[9] model pretrained on ImageNet21k [44] and present results with NCM, SDC and
LDC in Table 2.

Table 1: Results in supervised settings for 10 tasks on CIFAR-100, Tiny-Imagenet, and ImageNet100. The mean and standard deviation using 5 different seeds are reported. Best results are in bold and second-best underlined. $\dagger$: results excerpted from [33]. Method | CIFAR-100 | Tiny-ImageNet | ImageNet100
---|---|---|---
$A_{last}$ | $A_{inc}$ | $A_{last}$ | $A_{inc}$ | $A_{last}$ | $A_{inc}$
LwF+NCM[43] | $40.5\pm 2.7$ | $56.2\pm 3.6$ | $28.6\pm 1.1$ | $44.2\pm 1.3$ | $44.0\pm 1.9$ | $64.4\pm 0.4$
LwF+SDC[52] | $40.6\pm 1.8$ | $56.2\pm 3.0$ | $29.5\pm 0.8$ | $43.8\pm 1.3$ | $42.6\pm 1.9$ | $\underline{65.3}\pm 0.6$
PASS [59] | $37.8\pm 0.2$ | $52.3\pm 0.1$ | $31.2\pm 0.4$ | $45.3\pm 0.7$ | $38.7\pm 0.5$ | $55.5\pm 0.5$
FeTrIL [41] | $37.0\pm 0.6$ | $52.1\pm 0.5$ | $24.4\pm 0.6$ | $38.5\pm 1.1$ | $40.7\pm 0.5$ | $56.3\pm 0.8$
FeCAM [18] | $33.1\pm 0.9$ | $48.1\pm 1.3$ | $24.9\pm 0.5$ | $38.6\pm 1.3$ | $42.4\pm 0.9$ | $57.9\pm 1.5$
EFC [33] † | $\underline{43.6}\pm 0.7$ | $\underline{58.6}\pm 0.9$ | $\underline{34.1}\pm 0.8$ | $\textbf{48.0}\pm 0.6$ | $\underline{47.3}\pm 1.4$ | $59.9\pm 1.4$
LwF+LDC | $\textbf{45.4}\pm 2.8$ | $\textbf{59.5}\pm 3.9$ | $\textbf{34.2}\pm 0.7$ | $\underline{46.8}\pm 1.1$ | $\textbf{51.4}\pm 1.2$ | $\textbf{69.4}\pm 0.6$
Table 2: Performance on Stanford Cars (10 tasks of 20 classes each) with pre-trained ViT-B/16. Method | $t1$ | $t3$ | $t5$ | $t7$ | $t10$
---|---|---|---|---|---
FT$+$NCM | $85.8$ | $73.3$ | $63.2$ | $58.0$ | $53.4$
FT+SDC | $85.8$ | $73.5$ | $63.5$ | $57.8$ | $53.3$
FT+LDC | $85.8$ | $76.0$ | $71.3$ | $67.0$ | 62.9

Training procedure. We train a ResNet-18 [21] using an SGD optimizer. For all
datasets and methods we train the first task for 200 epochs and the
incremental tasks for 100 epochs. In the first task for CIFAR-100, we use a
learning rate of 0.1, momentum of 0.9, weight decay of 0.0005, and the
learning rate is reduced by 10 after 60, 120, and 160 epochs following the
default settings of PyCIL. In the incremental tasks, we use a learning rate of
0.05, which is reduced by 10 after 45 and 90 epochs. For LwF models, we use
$\lambda=10$. For LDC, we learn a linear layer using Adam optimizer and a
learning rate of 0.001 for 20 epochs using all the data in the current task.
We report all details about experimental settings in the supplementary
materials. For table 2, we use the same settings and hyperparameters as [55]
for finetuning in new tasks and learn a two-layer MLP using Adam optimizer and
a learning rate of 0.005 for 100 epochs.

Results. We report the last task accuracy as well as the average incremental
accuracy with $5$ random seeds and class sequences in Table 1. We observe that
while the recently proposed methods like PASS [59], FeTrIL [41], and FeCAM
[18] performed well in Warm Start settings, LwF with an NCM classifier
outperformed these methods in Cold Start settings. While SDC was effective in
the originally proposed Warm Start settings, we see that with higher feature
drift in our settings, SDC does not improve the performance in most cases. We
show that LDC with LwF outperforms all these baselines significantly across
all datasets. In CIFAR-100, LDC outperforms SDC by $4.8\%$ and by $3.3\%$ on
last task and incremental accuracy respectively. For Tiny-ImageNet, LDC
performs similar to the recent EFC method which is also designed for the cold
start setting. However, on the more challenging ImageNet100 LDC improves over
EFC by a large margin of almost 4% on last task accuracy. Finally, using pre-
trained ViTs on Stanford Cars, we observe that LDC improves over SDC by 9.6%
after the last task, see Table 2.

Table 3: Results in semi-supervised settings for $10$ tasks on CIFAR-100. The mean and standard deviation using $3$ different seeds are reported. The best results for the proposed setting are in bold. Method | Ex Free | Semi-Sup |  | 100% |
---|---|---|---|---|---
LwF+NCM | ✓ | ✗ |  | 40.5$\pm$2.7 |
PASS | ✓ | ✗ |  | 37.8$\pm$0.2 |
|  |  | 0.8% | 5% | 25%
PASS | ✓ | ✓ | 5.91 | 15.13 | 15.84
NNCSL (500) | ✗ | ✓ | 27.4$\pm$0.51 | 31.4$\pm$0.40 | 35.3$\pm$0.30
NNCSL (0) | ✓ | ✓ |  8.7$\pm$0.63 |  9.0$\pm$0.42 | 9.32$\pm$0.39
PFR+naive | ✓ | ✓ | 20.5$\pm$0.44 | 24.8$\pm$0.35 | 25.7$\pm$0.12
CaSSLe+naive | ✓ | ✓ | 19.4$\pm$0.57 | 23.2$\pm$0.21 | 23.6$\pm$0.13
POCON+naive | ✓ | ✓ | 20.1$\pm$0.23 | 21.7$\pm$0.27 | 22.4$\pm$0.31
PFR+SDC | ✓ | ✓ | 16.2$\pm$0.66 | 22.5$\pm$0.57 | 24.8$\pm$0.12
CaSSLe+SDC | ✓ | ✓ | 22.1$\pm$0.69 | 25.8$\pm$0.09 | 26.5$\pm$0.04
POCON+SDC | ✓ | ✓ | 21.8$\pm$0.82 | 25.7$\pm$0.23 | 26.7$\pm$0.09
PFR+LDC | ✓ | ✓ | 18.6$\pm$0.20 | 29.3$\pm$0.12 | 33.4$\pm$0.06
CaSSLe+LDC | ✓ | ✓ |  27.0$\pm$0.60 |  32.8$\pm$0.30 |  35.0$\pm$0.21
POCON+LDC | ✓ | ✓ | 26.1$\pm$0.74 | 31.1$\pm$0.62 | 33.8$\pm$0.25

###  4.3 Semi-supervised Continual Learning

Baseline methods. Since the proposed method is the first exemplar-free semi-
supervised CL method, we can only compare it against drift correction methods
(SDC). Nevertheless, we present results for the most close approaches. The
naive approach achieves the lowest possible accuracy, no prototype correction
is performed. SDC [52] is used as a comparison for drift correction. PASS [59]
is not a semi-supervised method, but we present our results with all labels
(as a reference) and with reduced training data corresponding to the proposed
label sets. NNCSL [26] is a semi-supervised method that uses exemplars; we
present here results with the same number of exemplars proposed by the authors
(as a reference) and a reduced version. Removing exemplars from exemplar-based
methods or reducing data to comply with our setup is not fair, and we present
these results as a reference to showcase the performance of our approach. We
use the code and hyperparameters provided by the authors.

Training procedure. We train a ResNet-18 [21] using PFR [15], CaSSLe [13], and
POCON [16] as SSL CL strategies. We use the same code, training procedure, and
hyperparameters provided by the authors in each method. At the end of each
task, we compute the prototypes using the available labels: $0.8\%$, $5\%$,
$25\%$ for CIFAR-100 and $1\%$, $5\%$, $25\%$ for ImageNet100. For LDC, we
train a linear projector using the Adam optimizer (learning rate $5e-3$) for
100 epochs using all the data in the task. We report the mean last task
accuracy in the test set using an NCM classifier.

Results. Table 3 presents the results of 10 task split of CIFAR-100. LDC
presents the best results for all the pure semi-supervised exemplar-free
settings. For $5\%$ and $25\%$ settings, it is reaching the same accuracy as
the exemplar-based method NNCSL. Furthermore, LDC is only $3\%$ below the
performance of the fully supervised version of PASS. LDC surpasses SDC by at
least $7\%$ in all the base SSL CL methods in the pure semi-supervised
exemplar-free setting.

Table 4 presents the results of 10 task split of ImageNet100. As in CIFAR-100,
LDC performs best for the proposed setting, surpassing SDC by at least $7\%$
in all the SSL CL backbones. However, in this dataset, the replay-based method
NNCSL is better with the same number of exemplars proposed by the authors.
Nevertheless, a reduction in the number of exemplars in NNCSL leads to a
significant performance drop, ending below our proposed exemplar-free method.

Table 4: Results in semi-supervised settings for $10$ tasks on ImageNet100. The mean and standard deviation using $3$ different seeds are reported. The best results for the proposed setting are in bold. Method | Ex Free | Semi-Sup |  | 100% |
---|---|---|---|---|---
LwF+NCM | ✓ | ✗ |  | 44.0$\pm$1.9 |
PASS | ✓ | ✗ |  | 38.7$\pm$0.5 |
|  |  | 1% | 5% | 25%
PASS | ✓ | ✓ | 1.76 | 10.36 | 23.9
NNCSL (5120) | ✗ | ✓ | 35.4$\pm$0.21 | 41.6$\pm$0.12 | 47.1$\pm$0.08
NNCSL (500) | ✗ | ✓ | 9.93$\pm$0.22 | 15.1$\pm$0.15 | 16.6$\pm$0.1
PFR+naive | ✓ | ✓ | 19.6$\pm$0.16 | 21.8$\pm$0.38 | 22.0$\pm$0.12
CaSSLe+naive | ✓ | ✓ | 25.37$\pm$0.27 | 26.9$\pm$0.25 | 27.36$\pm$0.11
POCON+naive | ✓ | ✓ | 24.2$\pm$0.32 | 26.1$\pm$0.22 | 25.8$\pm$0.20
PFR+SDC | ✓ | ✓ | 17.2$\pm$0.55 | 19.1$\pm$0.16 | 19.8$\pm$0.08
CaSSLe+SDC | ✓ | ✓ | 22.2$\pm$0.03 | 25.9$\pm$0.23 | 26.6$\pm$0.01
POCON+SDC | ✓ | ✓ | 21.8$\pm$0.08 | 25.1$\pm$ 0.30 | 25.7$\pm$0.03
PFR+LDC | ✓ | ✓ | 26.5$\pm$0.09 | 32.4$\pm$0.35 | 33.5$\pm$0.07
CaSSLe+LDC | ✓ | ✓ |  29.5$\pm$0.58 |  33.9$\pm$0.15 |  35.8$\pm$0.39
POCON+LDC | ✓ | ✓ | 28.3$\pm$0.37 | 33.2$\pm$0.30 | 34.0$\pm$0.28

###  4.4 Ablation Studies

In this section we report on a range of ablation studies for LDC. Unless
specified, the training and evaluation procedure are the same as in Sections
4.2 and 4.3. We provide more ablation experiments on comparison of LDC with
NME and using stored features for learning the projector, in the supplementary
materials.

Architecture of projector. To ablate the complexity of the proposed projector
$p_{F}$, Table LABEL:tab:projector presents experiments for LwF+LDC(supervised
CL) and CaSSLe+LDC (semi-supervised CL - $25\%$ of labels). We show LDC
performance with a linear layer, with and without bias, with an activation
layer ReLU after the linear layer and also using a two-layer MLP. We observe
that a single linear layer works better in both settings.

Table 5: Ablation study for different complexities of the proposed projector $p_{F}$. Method | Linear | Linear + bias | Linear + ReLu | MLP
---|---|---|---|---
LwF + LDC | 45.4$\pm$2.8 | 45.2$\pm$3.0 | 41.2$\pm$2.1 | 43.7$\pm$2.3
CaSSLe + LDC | 35.0$\pm$0.2 | 34.3$\pm$0.2 | 29.1$\pm$0.5 | 34.9$\pm$0.4

Exemplars versus LDC for prototype correction. We compare the performance of
LDC against Nearest-Mean of Exemplars (NME) which directly uses exemplars for
old class prototype compensation and is explored in exemplar-based methods
[10, 43]. While it is easy to predict the updated positions of the old
prototypes using original samples of old classes by NME, it is very
challenging without exemplars in our settings. We compare the last task
accuracy of LDC (same setting as in Table 1, with LwF+NCM) with NME which
stores a memory of old class samples. We show in Fig. 4 that LDC is on par
with using 20 exemplars per class for NME for CIFAR-100 and Tiny-ImageNet. For
ImageNet100, LDC is marginally less than NME with 20 exemplars per class. This
demonstrates that LDC is able to effectively correct the prototypes without
access to exemplars and yet is on par with NME using exemplars.

Figure 4: Comparison of LDC with NME for varying memory size in the supervised
settings on CIFAR-100, Tiny-ImageNet and ImageNet100.

Drift Compensation Analysis. In order to visualize the drift compensation
using LDC, Fig. 5 presents a comparison between SDC and LDC. We update all the
old prototypes using both SDC and LDC and then compute the cosine distance of
the updated prototype to the oracle prototype (computed using all old data for
analysis) in the new feature space. We plot the distributions over all old
class prototype distances after alternate tasks till the lats task. We observe
that SDC struggles to estimate the prototypes even after the first increment
(task 2) and is centered around distance of 0.1. As we move to the last task,
SDC gets even worse with a higher deviation which implies increasing distance
of updated prototypes from their oracle positions. On the other hand, LDC is
more robust and the distribution is centered between 0 and 0.05 even after the
last task, with much lower deviation. This analysis suggests that the LDC
predicts the old prototypes very close to their oracle positions and thus
better tracks the prototypes position when moving from one feature space to
another.

Figure 5: Drift compensation analysis using SDC and LDC in supervised settings
on CIFAR-100. We compute the cosine distance between the updated and oracle
prototypes. We plot the distributions of the cosine distances for all old
classes using both methods after alternate tasks. While SDC fails to estimate
the prototypes closer to the oracle (with increasing distance for many
classes), LDC predictions are very close to the oracle prototypes (mean of
distributions using LDC is close to 0 even after the last task and has low
standard deviation).

##  5 Conclusions

We showed that performance degradation in exemplar-free CP accumulation
methods is largely due to feature drift. We proposed a simple yet effective
prototype correction method called Learnable Drift Compensation (LDC). LDC
learns a projector that maps between the feature spaces of the consecutive
tasks using only the data available at each task along with current and
previous model. The trained projector is used at the end of each task to
correct the stored old prototype positions. We demonstrate the efficacy of LDC
in correcting the drift of any moving backbone by using it on top of already
established exemplar-free CL strategies. Extensive experiments on CIFAR-100,
Tiny-ImageNet, and ImageNet100 show that LDC outperforms previous drift
correction methods in supervised and semi-supervised settings.

Limitations. While the learned projection $p_{f}$ between tasks is effective
for correcting drift, it’s not flawless. The gap between $p_{corrected}$ and
$p_{oracle}$ in Fig. 1 highlights this difference. Without exemplars, $p_{f}$
only learns the projection from current task data. Consequently, due to biases
in the current data, updates to some old class prototypes may not be perfect.
We would like to explore methods to further minimize or eliminate this gap in
our future work.

Acknowledgments. We acknowledge projects TED2021-132513B-I00 and
PID2022-143257NB-I00, financed by MCIN/AEI/10.13039/501100011033 and FSE+ by
the European Union NextGenerationEU/PRTR, and by ERDF A Way of Making Europa,
and the Generalitat de Catalunya CERCA Program. Bartłomiej Twardowski
acknowledges the grant RYC2021-032765-I.

## References

  * [1] Liu, Xialei, Wu, C., Menta, M., Herranz, L., Raducanu, B., Bagdanov, A.D., Jui, S., van de Weijer, J.: Generative feature replay for class-incremental learning. In: Conference on Computer Vision and Pattern Recognition Workshops (2020)
  * [2] Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: European Conference on Computer Vision (2018)
  * [3] Assran, M., Caron, M., Misra, I., Bojanowski, P., Joulin, A., Ballas, N., Rabbat, M.: Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8443–8452 (2021)
  * [4] Caccia, L., Pineau, J.: Special: Self-supervised pretraining for continual learning. In: International Workshop on Continual Semi-Supervised Learning. pp. 91–103. Springer (2022)
  * [5] Castro, F.M., Marín-Jiménez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: European Conference on Computer Vision (2018)
  * [6] Cha, H., Lee, J., Shin, J.: Co2l: Contrastive continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9516–9525 (2021)
  * [7] Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. In: International Conference on Learning Representations (2019)
  * [8] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [9] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021)
  * [10] Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Podnet: Pooled outputs distillation for small-tasks incremental learning. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XX 16. pp. 86–102. Springer (2020)
  * [11] Douillard, A., Ramé, A., Couairon, G., Cord, M.: Dytox: Transformers for continual learning with dynamic token expansion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9285–9295 (2022)
  * [12] Fini, E., da Costa, V.G.T., Alameda-Pineda, X., Ricci, E., Alahari, K., Mairal, J.: Self-supervised models are continual learners. In: CVPR (2022)
  * [13] Fini, E., Da Costa, V.G.T., Alameda-Pineda, X., Ricci, E., Alahari, K., Mairal, J.: Self-supervised models are continual learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9621–9630 (2022)
  * [14] Gallardo, J., Hayes, T.L., Kanan, C.: Self-supervised training enhances online continual learning. arXiv preprint arXiv:2103.14010 (2021)
  * [15] Gomez-Villa, A., Twardowski, Bartlomiej, Yu, Lu, Bagdanov, A.D., van de Weijer, J.: Continually learning self-supervised representations with projected functional regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3867–3877 (2022)
  * [16] Gomez-Villa, A., Twardowski, B., Wang, K., van de Weijer, J.: Plasticity-optimized complementary networks for unsupervised continual learning. In: IEEE/CVF Winter Conference on Applications of Computer Vision (2024)
  * [17] Gomez-Villa, A., Twardowski, B., Yu, L., Bagdanov, A.D., van de Weijer, J.: Continually learning self-supervised representations with projected functional regularization. In: CVPRW (2021)
  * [18] Goswami, D., Liu, Y., Twardowski, B., van de Weijer, J.: Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. Advances in Neural Information Processing Systems 36 (2024)
  * [19] Goswami, D., Soutif-Cormerais, A., Liu, Y., Kamath, S., Twardowski, B., van de Weijer, J.: Resurrecting old classes with new data for exemplar-free continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
  * [20] Goswami, D., Twardowski, B., Van De Weijer, J.: Calibrating higher-order statistics for few-shot class-incremental learning with pre-trained vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2024)
  * [21] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Conference on Computer Vision and Pattern Recognition (2016)
  * [22] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning Workshop (2014)
  * [23] Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: International Conference on Computer Vision (2019)
  * [24] Iscen, A., Zhang, J., Lazebnik, S., Schmid, C.: Memory-efficient incremental learning through feature adaptation. In: European Conference on Computer Vision (2020)
  * [25] Janson, P., Zhang, W., Aljundi, R., Elhoseiny, M.: A simple baseline that questions the use of pretrained-models in continual learning. In: NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications (2022)
  * [26] Kang, Z., Fini, E., Nabi, M., Ricci, E., Alahari, K.: A soft nearest-neighbor framework for continual semi-supervised learning. arXiv preprint arXiv:2212.05102 (2022)
  * [27] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. National Academy of Sciences (2017)
  * [28] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: IEEE International Conference on Computer Vision Workshops. pp. 554–561 (2013)
  * [29] Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep., University of Toronto (2009)
  * [30] Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N (2015)
  * [31] Li, Z., Hoiem, D.: Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017)
  * [32] Madaan, D., Yoon, J., Li, Y., Liu, Y., Hwang, S.J.: Representational continuity for unsupervised continual learning. In: ICLR (2022)
  * [33] Magistri, S., Trinci, T., Soutif-Cormerais, A., van de Weijer, J., Bagdanov, A.D.: Elastic feature consolidation for cold start exemplar-free incremental learning. In: International Conference on Learning Representations (2024), https://openreview.net/forum?id=7D9X2cFnt1
  * [34] Malepathirana, T., Senanayake, D., Halgamuge, S.: Napa-vq: Neighborhood-aware prototype augmentation with vector quantization for continual learning. In: International Conference on Computer Vision (ICCV) (2023)
  * [35] Mallya, A., Lazebnik, S.: Packnet: Adding multiple tasks to a single network by iterative pruning. In: Conference on Computer Vision and Pattern Recognition (2018)
  * [36] Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A.D., Van De Weijer, J.: Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(5), 5513–5533 (2022)
  * [37] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)
  * [38] McDonnell, M.D., Gong, D., Parvaneh, A., Abbasnejad, E., van den Hengel, A.: Ranpac: Random projections and pre-trained models for continual learning. Advances in Neural Information Processing Systems 36 (2024)
  * [39] Ostapenko, O., Lesort, T., Rodríguez, P., Arefin, M.R., Douillard, A., Rish, I., Charlin, L.: Foundational models for continual learning: An empirical study of latent replay. arXiv preprint arXiv:2205.00329 (2022)
  * [40] Panos, A., Kobe, Y., Reino, D.O., Aljundi, R., Turner, R.E.: First session adaptation: A strong replay-free baseline for class-incremental learning. In: International Conference on Computer Vision (ICCV) (2023)
  * [41] Petit, G., Popescu, A., Schindler, H., Picard, D., Delezoide, B.: Fetril: Feature translation for exemplar-free class-incremental learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3911–3920 (2023)
  * [42] Rao, D., Visin, F., Rusu, A., Pascanu, R., Teh, Y.W., Hadsell, R.: Continual unsupervised representation learning. In: Advances in Neural Information Processing Systems (2019)
  * [43] Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation learning. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [44] Ridnik, T., Ben-Baruch, E., Noy, A., Zelnik-Manor, L.: Imagenet-21k pretraining for the masses. In: Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2021)
  * [45] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International journal of computer vision (2015)
  * [46] Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv (2016)
  * [47] Serra, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. In: International Conference on Machine Learning (2018)
  * [48] Shi, W., Ye, M.: Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-incremental learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1772–1781 (2023)
  * [49] Toldo, M., Ozay, M.: Bring evanescent representations to life in lifelong class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16732–16741 (2022)
  * [50] van de Ven, G.M., Tolias, A.S.: Three scenarios for continual learning. In: NeurIPS Continual Learning Workshop (2018)
  * [51] Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large scale incremental learning. In: Conference on Computer Vision and Pattern Recognition (2019)
  * [52] Yu, L., Twardowski, Bartlomiej, Liu, Xialei, Herranz, L., Wang, K., Jui, S., Weijer, J.v.d.: Semantic drift compensation for class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6982–6991 (2020)
  * [53] Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230 (2021)
  * [54] Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: International Conference on Machine Learning (2017)
  * [55] Zhang, G., Wang, L., Kang, G., Chen, L., Wei, Y.: Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
  * [56] Zhou, D.W., Sun, H.L., Ning, J., Ye, H.J., Zhan, D.C.: Continual learning with pre-trained models: A survey. arXiv preprint arXiv:2401.16386 (2024)
  * [57] Zhou, D.W., Wang, F.Y., Ye, H.J., Zhan, D.C.: Pycil: a python toolbox for class-incremental learning. SCIENCE CHINA Information Sciences (2023)
  * [58] Zhu, F., Cheng, Z., Zhang, X.y., Liu, C.l.: Class-incremental learning via dual augmentation. Advances in Neural Information Processing Systems 34, 14306–14318 (2021)
  * [59] Zhu, F., Zhang, X.Y., Wang, C., Yin, F., Liu, C.L.: Prototype augmentation and self-supervision for incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5871–5880 (2021)
  * [60] Zhu, K., Zhai, W., Cao, Y., Luo, J., Zha, Z.J.: Self-sustaining representation expansion for non-exemplar class-incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9296–9305 (2022)
  * [61] Zhu, K., Zheng, K., Feng, R., Zhao, D., Cao, Y., Zha, Z.J.: Self-organizing pathway expansion for non-exemplar class-incremental learning. In: International Conference on Computer Vision (ICCV) (2023)

## Supplementary Material

##  Appendix 0.A Learnable Drift Compensation pseudocode

At the end of each training session $t$

input : $D_{t}$;$f_{\theta}^{t-1}$;$f_{\theta}^{t}$;Prototype pool of old task
classes $p^{C}$

Phase 1 : Train feature extractor $f_{\theta}^{t}$ in $D_{t}$ using any
learning strategy

Phase 2 : Train forward projector $p_{F}^{t}$ using $D_{t}$;
$f_{\theta}^{t-1}$; $f_{\theta}^{t}$

begin

freeze(_$f_{\theta}^{t-1}$ ,$f_{\theta}^{t}$_)

for  _minibatch in training_batches_ do

minimize(_MSE(_$p_{F}^{t}(f_{\theta}^{t-1}(\textnormal{{minibatch}}))$
,$f_{\theta}^{t}(\textnormal{{minibatch}})$_)_)

end for

end

Phase 3 : Update old prototypes with $p_{F}^{t}$

begin

for  _prototype in $p^{C}$_ do

$p^{\textnormal{{prototype}}}=p_{F}^{t}(p^{\textnormal{{prototype}}})$

end for

end

Algorithm 1 Learnable Drift Compensation

##  Appendix 0.B Training Settings

###  0.B.1 Supervised CL settings

We implemented existing methods like LwF [31] (with NCM [43] and SDC [52]),
PASS [59], FeTrIL [41] and FeCAM [18] and optimized them for small-start
settings. We use the PyCIL [57] framework. Here we share the details of the
settings for reproducibility.

For CIFAR-100, we use the CIFAR-10 policy augmentations from PyCIL and use it
for all the methods to have a fair comparison. For all datasets, we follow the
common practice of using random crop and random horizontal flip for the
training set.

LwF: For all datasets in the first task, we follow PyCIL and use a learning
rate of 0.1, momentum of 0.9, weight decay of 0.0005 and train with a batch
size of 128 for 200 epochs. The learning rate is reduced by 10 after 60 , 120
and 180 epochs. For all new tasks in CIFAR-100 and ImageNet100, we use a
learning rate of 0.05. For TinyImageNet, we use a learning rate of 0.001. We
train for 100 epochs and reduce the learning rate by 10 after 45 and 90
epochs. The temperature scale is set to 2. The LwF regularization strength is
set to 10 for CIFAR-100 and TinyImageNet and 5 for ImageNet100. We use the
models trained with LwF and use them with a NCM classifier and SDC. For FeTrIL
and FeCAM, we train the first task in the same way as LwF.

SDC: For SDC [52], we use $\sigma=0.3$, which is the standard deviation of the
Gaussian kernel for estimating the weights of the drift vectors. For
ImageNet100, we set $\sigma=1.0$.

PASS: Following the PyCIL implementation of PASS, we set $\lambda_{fkd}=10$
and $\lambda_{proto}=10$.

FeTrIL: For all incremental tasks, we follow the same settings as the FeTrIL
implementation from PyCIL [57].

FeCAM: FeCAM trains the first task and use the frozen model for all subsequent
tasks. FeCAM use the stored prototypes and distribution statistics (covariance
matrix) for all old classes. Following the implementation from [18], we use
the covariance shrinkage hyperparameters of (1,1) and 0.5 as the Tukey’s
normalization value.

###  0.B.2 Self-supervised CL settings.

We use the existing CL exemplar-free methods namely PFR [17], CaSSLE [12], and
POCON [16]. We utilize the code provided for each method. Here we include
additional details of the settings for reproducibility.

For each dataset, we follow the image augmentation pipeline of SimCLR [8]. All
the methods use Barlow Twins [53] as base self-supervised learning strategy.

PFR: We conduct our experiments with $500$ epochs per task on CIFAR100 and
$400$ for ImageNet100. As for CL hyperparameters, $\lambda=25$ for both
datasets; the learning rate, optimizer, and training schedules are the same as
proposed by the authors.

CaSSLe: We use the same training epochs as PFR. We strictly follow the
hyperparameters shared by the authors.

POCON: For both datasets, we employ the configuration CopyOP. The
hyperparameters setup is the same as the authors proposed for the 10-task
partition of each dataset.

##  Appendix 0.C Additional Ablation Experiments

Feature Storage. In this work, we update only prototypes (one sample mean per
class), but one can argue that storing a set of features per class is better
than a single class mean. In Table LABEL:tab:feat_stats, instead of projecting
a prototype per class, we project $N$ stored features and use them to compute
the class prototype at the classification stage. The results show that storing
and projecting features do not present a significant advantage over projecting
the class means.

Table 6: Accuracy comparison storing and projecting $N$ features per class against storing and projecting only class prototypes. Method | Mean | $\boldsymbol{N=5}$ | $\boldsymbol{N=50}$ | $\boldsymbol{N=100}$ | $\boldsymbol{N=500}$
---|---|---|---|---|---
CaSSLe + LDC | $35.0\pm 0.2$ | $21.1\pm 0.01$ | $32.9\pm 0.03$ | $33.9\pm 0.02$ | $35.05\pm 0.04$


11institutetext: Worcester Polytechnic Institute, Worcester MA 01609, USA

# Invariant representation learning for sequential recommendation

Xiaofan Zhou 11

###### Abstract

Sequential recommendation involves automatically recommending the next item to
users based on their historical item sequence. While most prior research
employs RNN or transformer methods to glean information from the item
sequence—generating probabilities for each user-item pair and recommending the
top items—these approaches often overlook the challenge posed by spurious
relationships. This paper specifically addresses these spurious relations. We
introduce a novel sequential recommendation framework named Irl4Rec. This
framework harnesses invariant learning and employs a new objective that
factors in the relationship between spurious variables and adjustment
variables during model training. This approach aids in identifying spurious
relations. Comparative analyses reveal that our framework outperforms three
typical methods, underscoring the effectiveness of our model. Moreover, an
ablation study further demonstrates the critical role our model plays in
detecting spurious relations.

###### Keywords:

Sequential recommendation Invariant learning spurious relation.

##  1 Introduction

A Sequential Recommendation System (SRS) predicts user-item interactions by
analyzing a user’s historical interaction sequence [18]. Distinct from other
recommendation systems, SRS is tailored to discern both long-term and short-
term patterns within this history, meticulously capturing the underlying
nuances within these sequences. In recent times, state-of-the-art techniques
like GRU4Rec [7], SASRec [9], and Bert4Rec [17] have taken the forefront in
sequential recommendation tasks. Notably, these methods adapt encoders,
traditionally employed for content understanding in domains such as visual
classification, object recognition, and text classification. This adaptation
underscores the potency of deep learning in the realm of sequential
recommendation.

However, the journey of SRS isn’t without hurdles. One persisting challenge is
the issue of spurious relations [2], which are irrelevant properties when
viewed from a recommendation lens. Such relations cause a difficulty to
extract accurate user preference extraction from historical sequences, leading
to skewed predictions. For instance, consider a user who has purchased a
series of ornate cups and a unique tea table. An SRS might falter in
discerning whether the user’s intent was to decorate their room or simply to
facilitate gatherings with friends. In another scenario, if a user watched two
superhero movies tagged as “Hero”, “Handsome”, and “Fly”, an SRS heavily
influenced by these tags might inappropriately recommend “Harry Potter” over
“Wonder Woman”. Eliminating such preference-independent yet behavior-
correlated factors from multimedia representations becomes paramount.

In response to this challenge, our research introduces a solution anchored in
a causal graph perspective. We conceptualize the original sequence as a blend
of two distinct variables: confounder variables and true preference variables.
While confounder variables represent potential triggers for spurious
relations, the true preference variables capture authentic user inclinations.
Our approach, therefore, revolves around a newly designed objective loss,
aiming to enhance the model’s accuracy. Preliminary experiments have
substantiated the efficacy of our method.

##  2 Related Work

###  2.1 Sequential Recommendation

Your content is well-structured and provides a good overview of the Sequential
Recommendation System and its evolution. I’ve revised your text for better
flow and clarity:

The Sequential Recommendation System (SRS) is tailored to predict forthcoming
items by scrutinizing a user’s prior interaction sequences. Earlier methods
for sequential recommendations, such as Sequential Pattern Mining and Markov
Chain models, offer contrasting mechanisms. Sequential Pattern Mining focuses
on uncovering frequent patterns within sequence data and then harnesses these
insights to craft subsequent recommendations [21]. Markov Chain-based
recommendation systems, on the other hand, lean on Markov Chain models to map
transitions between user-item interactions within sequences [6, 5].

With the advent of deep learning, there’s been a surge in integrating Deep
Neural Networks and Transformer-based approaches into sequential
recommendation systems. RNNs (Recurrent Neural Networks) stand out in this
cohort due to their inherent prowess in sequence modeling. For instance,
GRU4Rec employs Gated Recurrent Units (GRU) to absorb information from past
items [7]. In a different stride, Bert4Rec leverages deep bidirectional self-
attention mechanisms to capture user behavioral patterns [17]. SASRec [9],
while echoing the sequence modeling strength of RNNs, distinctively employs an
attention mechanism to base its predictions on select actions, offering a
nuanced understanding of long-term semantics.

Despite the sophistication of modern techniques, many overlook a critical
issue: the inadvertent neglect of spurious relations within the data. While
delving deeply into historical sequences, these systems often bypass the
spurious relation to the data. In our research, we advocate for the
integration of invariant learning. By constructing a causal graph, we employ
an encoder to identify the spurious variables within historical item
sequences. Recognizing the interplay between spurious variables, adjustment
variables, and targets, we introduce a novel objective. This strategy
specifically addresses the challenge posed by spurious relations in historical
sequences.

###  2.2 Invariant learning

Invariant learning, as delineated in various studies [2, 4, 15], operates on
the premise that observed data exhibits heterogeneity, implying that such data
emerges from a multitude of diverse environments. These environments
inherently possess varying data distributions. The overarching objective of
invariant learning is to seize representations that consistently predict
across these diverse environments. [12] undertook a meticulous analysis of the
foundational assumptions behind existing invariant learning methodologies.
Their study not only provided a theoretical relaxation of prior invariance
assumptions but also proffered a relevant solution. [1] furthered this
discourse by substantiating that integrating a form of the information
bottleneck constraint with invariance can effectively address the predominant
shortcomings of IRM-based techniques [3]. Their proposed solution seamlessly
marries invariant learning with the information bottleneck principle. [15]
introduced a pragmatic, easily implementable weighting method aimed at
capturing invariance to bolster generalization. Meanwhile, [11] provided a
heuristic analysis of IRM’s pitfalls and suggested a novel invariance penalty.
This penalty finds its roots in a renewed exploration of the data
representation’s Gramian matrix.

##  3 Method

###  3.1 Problem definition

The primary objective of this paper is to predict the next item, $c_{n+1}$,
that a user might purchase based on their previous shopping or viewing
history, denoted as $S^{u}=[c_{1},c_{2},\ldots,c_{n}]$. Here, $c_{i}$
represents the $i$-th item the user purchased, and $n$ denotes the maximum
history length. In the following sections, we will elaborate on the proposed
methods.

Figure 1: Causal graph of recommendation

###  3.2 Overview

In this work, we introduce a novel framework designed specifically to address
the issue of spurious relations in sequential recommendation. This framework
can be bifurcated into two integral components:

  1. 1.

Adjustment Encoder $f_{\theta}$: This encoder captures the users’ genuine
preferences. Techniques such as Recurrent Neural Networks or self-supervised
methods (e.g., GRU, BERT) are employed to extract these true user preferences.

  2. 2.

Confounder Encoder $f_{\phi}$: Parallel in structure to the adjustment
encoder, the confounder encoder is tasked with extracting user representations
that might lead to spurious correlations when discerning preferences from
historical interactions. It is pivotal to note that the confounder encoder is
exclusively employed during the training phase to negate these inadvertent
correlations.

By synergizing the representations derived from both the confounder and
adjustment encoders, we formulate an objective function tailored to optimize
the task. Subsequent sections will delve into each of the aforementioned
components in meticulous detail.

###  3.3 Sequential recommendation Model

The sequential recommendation model is employed within both the confounder
encoder and the adjustment encoder. The sequential recommendation model
comprises three parts: item embedding parts, sequence encoder layer, and next-
item prediction layers.

##### Item Embedding Parts

Firstly, all items are embedded into the same space to produce an item
embedding matrix $M\in R^{|I|\times d}$ [10], where $|I|$ denotes the item set
and $d$ signifies the dimension of the embedding. Given the input history
sequence $S^{u}=[c_{1},c_{2},...,c_{n}]$, the sequence’s embedding $S^{u}$ is
initialized to $e^{u}\in R^{n\times d}$ where
$e^{u}=[m_{s_{1}}+p_{1},m_{s_{2}}+p_{2},...,m_{s_{n}}+p_{n}]$. Here,
$m_{s_{i}}\in R^{d}$ is the item’s embedding at position $k$ in the sequence,
$p_{k}\in R^{d}$ is the position’s embedding, and $n$ is the sequence length.

##### Sequence Encoder Layer

The sequence encoder layer derives the representation of $e^{u}$ using a deep
neural network (e.g., Bert4Rec) [17]. This is defined as $f_{\theta}$, where
$\theta$ represents the model’s parameters. The output representation
$H^{u}\in R^{n\times d}$ is calculated as:

$H^{u}=f_{\theta}(e^{u})$

Given our task is predicting the next item, we employ the final vector $h^{n}$
in $H^{u}=[h_{1},h_{2},...,h_{n}]$ as the representation of historical items.

##### Next-Item Prediction Layers

Lastly, we determine the probability of each item as $\hat{y}=h^{u}_{n}M^{T}$.
In our model, the BPR loss is optimized to heighten the probability of a
correct prediction given user $u$ and a pair of positive and negative items
$i$ and $j$:

$\displaystyle BPR_{uij}=-ln\sigma(\hat{x}_{ui}-\hat{x}_{uj})$ (1)

Where $\sigma$ is the logistic sigmoid function:

$\displaystyle\sigma(x)=\frac{1}{1+e^{-x}}$ (2)

###  3.4 Spurious relation elimination part

Figure 2: (a) During the training phase, the model derives a specific
embedding vector from variables $x$, consisting of an adjustment component $t$
and a spurious relation component $s$. (b) In the test phase, the spurious
relation component $s$ is discarded, and only the adjustment component $t$ is
employed in the recommendation process.

As depicted in Fig. 2, the primary concept of our proposed framework is
elaborated. In the training phase, we impose constraints on the model to yield
a specialized representation vector from variables $x$. This vector
encompasses two distinct components: the spurious relation component $s$,
accounting for the indirect effect, and the adjustment component $t$,
responsible for the direct effect. We anticipate the spurious relation
component $s$ to encapsulate the entire deviations observed in the feedback
label due to the spurious relation, while the adjustment component $t$
discerns the ideal feedback label portion that arises from genuine user
preferences. During the test phase, the spurious relation component $s$ is
disregarded, and only the adjustment component $t$ is used to enhance
recommendation accuracy.

From Fig 2, we can discern that for a more precise recommendation, we must
satisfy the following conditions at a minimum:

  1. 1.

To avert the influence of the spurious relation, the adjustment component
shouldn’t overfit to variables $x$.

  2. 2.

Owing to the role of the direct effect, the adjustment component should
accurately predict the label $y$.

  3. 3.

The adjustment component $t$ and the spurious relation $s$ should remain as
independent as possible to ensure a clear distinction, i.e., $s\perp t$.

  4. 4.

Because of the indirect effect’s role, the adjustment component $t$ should
also predict label $y$ to some degree.

We have deliberately not constrained the relationship between the adjustment
component $t$ and the variables $x$ since the dependency level between them
hinges on the feedback data’s inherent bias. Arbitrarily optimizing this might
lead to undesirable outcomes. Drawing inspiration from information theory, we
can formulate the required objective function based on the aforementioned
analysis:

$\displaystyle\mathcal{L}:=\underbrace{\beta
I(t;x)}_{(1)}-\underbrace{I(t;y)}_{(2)}+\underbrace{\gamma
I(t;s)}_{(3)}-\underbrace{\alpha I(s;y)}_{(4)}$ (3)

Where:

  * •

Term (1) is a compression term, characterizing the mutual information between
variables $x$ and the adjustment embedding $t$.

  * •

Term (2) is an accuracy term, detailing the performance of the adjustment
embedding $t$.

  * •

Term (3) acts as a de-confounder penalty term, illustrating the dependency
magnitude between the spurious embedding $s$ and the adjustment embedding $t$.

  * •

Term (4) is another accuracy term, considering the potential benefits accrued
from the spurious embedding $s$.

Here, $\beta$, $\gamma$, and $\alpha$ are the respective weight parameters. By
optimizing $\mathcal{L}$, we aim to obtain the desired spurious and adjustment
embeddings, subsequently pruning the spurious relationship. Equation (4)
stands as a vital optimization objective to steer the model towards the
spurious relation embedding vector. Introducing further reasonable targets,
grounded in Equation (4), could assist in securing a more precise adjustment
embedding vector.

Despite the provided equation, calculating $\mathcal{L}$ remains a challenge
due to its inherent complexity. We seek a more tractable solution by
considering an upper bound.

Using the chain rule of mutual information, we express the deconfounder
penalty term in $\mathcal{L}$ as:

$\displaystyle I(t;s)=I(t;y)-I(t;y|t)+I(t;s|y)$ (4)

Inspecting the term $I(t;s|y)$ in Equation (4), we realize that the
distribution of $t$ depends solely on $x$ (which in turn is affected by $y$).
This gives us:

$\displaystyle H(t|y,s)=H(t|y)$ (5)

where $H(|)$ represents the bpr loss. Using properties of mutual information,
we deduce:

$\displaystyle I(t;s|y)=H(t|y)-H(t|y,s)=0$ (6)

Substituting Equation (5) into Equation (4), we get:

$\displaystyle I(t;s)=I(t;y)-I(t;y|s)$ (7)

Given the complexity of the term $I(t;y|s)$ in Equation (6), we simplify
further using the concept of conditional entropy:

$\displaystyle I(t;s)=I(t;y)-H(y|s)+H(y|t,s)$ (8)

By integrating the findings from Equations (3) and (7), we can redefine
$\mathcal{L}$ from Equation (3) as:

$\displaystyle\mathcal{L}$ $\displaystyle=\beta I(t;x)-I(t;y)+\gamma
I(t;s)-\alpha I(s;y)$ (9) $\displaystyle=\beta
I(t;x)-I(t;y)+\gamma[I(t;y)-H(y\mid s)+H(y\mid t,s)]-\alpha I(s;y)$
$\displaystyle=\beta I(t;x)-(1-\gamma)I(t;y)-\gamma H(y\mid s)+\gamma H(y\mid
t,s)-\alpha I(s;y)$

We can find that only the compression term I (t;x) is related to the variables
x in Equation (8). To optimize it directly, we describe a simple and precise
expression of this mutual information using a method similar to that in [8,
20]. First, based on the relationship between mutual information and Kullback-
Leibler (KL) divergence, the compression term I (t; x) can be calculated as
follows:

$\displaystyle I(t;x)$
$\displaystyle=\mathbb{E}_{x}[D_{\mathrm{KL}}(p(t|x)\|p(t))]$ (10)
$\displaystyle=\sum_{x}p(x)\sum_{t}p(t|x)\log p(t|x)-\sum_{t}p(t)\log p(t)$

However, the marginal probability $p(t)=xp(t|x)p(x)$ is usually difficult to
be calculated in practice. We use variational approximation to address this
issue, i.e., we use a variational distribution $q(t)$ instead of $p(t)$.
According to Gibbs’ inequality, we know that the KL divergence is non-
negative. Therefore, we can derive an upper bound of Equation (9),

$\begin{array}[]{ll}&D_{\mathrm{KL}}(p(t)\|q(t))\geq 0\\\
&\Rightarrow-\sum_{t}p(t)\log p(t)\leq-\sum_{t}p(t)\log q(t)\\\ &\Rightarrow
D_{\mathrm{KL}}(p(t\mid x)\|p(t))\leq D_{\mathrm{KL}}(p(t\mid
x)\|q(t)).\end{array}$ (11)

Similar to most previous works[14], we can assume that the posterior $p(t|x)$
is a Gaussian distribution(i.e.,$p(t|x)=N(\mu(x),diag{\sigma^{2}(x)})$),where
$\mu(x)$ is the encoded embedding of the variables $x$ and $diag{\sigma 2(x)}$
is the diagonal matrix indicating the variance. Through the re-
parameterization trick, the embedding $t$ can be generated according to
$t=\mu(x)+\epsilon\dot{\sigma}(x)$, where $\epsilon~{}N(0,I)$. Obviously, if
we fix $\sigma(x)$ to be an all-zero matrix, $t$ will reduce to a
deterministic embedding. On the other hand, the prior $q(t)$ is assumed to be
a standard Gaussian variational distribution, i.e., $q(t)=N(0,I)$. Finally, we
can rewrite the above upper bound,

$\displaystyle D_{\mathrm{KL}}(p(t\mid
x)\|q(t))=\frac{1}{2}\|\mu(x)\|_{2}^{2}+\frac{1}{2}\sum_{d}\left(\sigma_{d}^{2}-\log\sigma_{d}^{2}-1\right),$
(12)

where $\sigma_{d}^{2}$ is an element in $diag{\sigma^{2}(x)}$, i.e.,
$diag{\sigma^{2}(x)}={\sigma_{d}^{2}}_{d=1}^{D}$ . This means that for a
deterministic embedding $t$, we can optimize this upper bound by directly
applying the $l_{2}$-norm regularization on the embedding vector $t$, which is
equivalent to optimizing the compression term $I(t;x)$. Note that the
compression term in previous works acts on the entire biased representation
$t^{*}$ , and we only compress the unbiased component $t$ of the
representation.

Finally, as we have:

$I(t;y)=H(y)-H(y|t)$ (13)

and since $H(y)$ is a positive constant and can be ignored, we can deduce the
following inequality:

$I(t;y)\geq-H(y|t)$ (14)

This logic also applies to the mutual information $I(s;y)$ in Equation (8).

Given these relationships, we can express the loss function, which we denote
as $\mathcal{L}$, in the following manner:

$\displaystyle\mathcal{L}$ $\displaystyle=\beta
I(t;\text{x})-(1-\gamma)I(t;y)-\gamma H(y\mid s)+\gamma H(y\mid t,s)-\alpha
I(s;y)$ (15)
$\displaystyle\leq\beta\|\mu(\text{x})\|_{2}^{2}+(1-\gamma)H(y\mid
t)-(\gamma-\alpha)H(y\mid s)+\gamma H(y\mid t,s).$ (16)

Upon simplifying $\mathcal{L}$, we arrive at a more tractable solution:

$\hat{\mathcal{L}}=\underbrace{(1-\gamma)H(y\mid
t)}_{\text{(a)}}-\underbrace{(\gamma-\alpha)H(y\mid
s)}_{\text{(b)}}+\underbrace{\gamma H(y\mid
t,s)}_{\text{(c)}}+\underbrace{\beta\|\mu(\text{x})\|_{2}^{2}}_{\text{(d)}}$
(17)

##  4 Experiment Design

###  4.1 Research Questions

We are seek to deal with the following questions::(RQ1)How does the proposed
framework perform in real-world recommendation scenarios? (RQ2) What is the
role of each term in equation (14)?

Table 1: Statistic information of experimented datasets Datasets | #users | #items | #actions | avg.length | sparsity
---|---|---|---|---|---
Sports | 33598 | 18357 | 296337 | 8.3 | 99.95%
Beauty | 22363 | 12101 | 198502 | 8.8 | 99.93%

###  4.2 Dataset

To validate the efficiency of our approach, we tested the model on two
benchmark datasets from the real-world: Amazon Beauty and Amazon Sports. These
datasets are derived from user reviews on amazon.com, one of the world’s
leading e-commerce platforms. Our experiments focus on two specific sub-
categories: Amazon-Beauty and Amazon-Sports. Following the preprocessing steps
outlined by [22], interactions involving users and items with fewer than five
engagements are excluded. The characteristics of the curated datasets are
detailed in Table 1.

###  4.3 Baseline

We compared our approach against several general sequential models:

  * •

GRU4Rec [7] employs the Gated Recurrent Unit (GRU) for sequential
recommendation.

  * •

SASRec [10] introduced the attention mechanism to sequential recommendation,
yielding notable performance.

  * •

BERT4Rec [17] leverages deep bidirectional self-attention to discern potential
relationships between items and their sequences in a Cloze task.

###  4.4 Metrics

To evaluate, the data is partitioned into training, validation, and testing
subsets based on timestamps provided in the datasets [10]. Specifically, the
last item is designated for testing, the penultimate for validation, and all
preceding items for training. Following [13, 19], we rank the complete item
set without employing negative sampling. To ensure comprehensive model
assessment, we employ two prevalent evaluation metrics: Hit Ratio @k (HR@k)
and Normalized Discounted Cumulative Gain @k (NDCG@k), where
$k\in\\{5,10,20\\}$. While the HR metric checks if the actual result is ranked
within the top k items, the NDCG metric offers a position-sensitive ranking
assessment.

###  4.5 Implementation Details

For optimization, the Adam optimizer was employed, with hyper-parameters fine-
tuned on the validation set. To ensure parity in comparisons, the batch size
was fixed at 256, the embedding dimension at 64, and the maximum history
length at 20 across all methodologies. The learning rate oscillated within the
range $[1e-3,5e-4,1e-4]$, and weight decay was adjusted between
$[1e-4,1e-6,1e-8,0]$. Additional hyper-parameters specific to the baselines
were tuned within ranges recommended by their respective authors. The
performance of the three base models refers from [16]. For transparency and
replicability, the source codes have been made publicly
accessible111https://github.com/THUwangcy/ReChorus.

##  5 Experiment Result

###  5.1 RQ1

We evaluated the efficacy of our model, Irl4Rec, against prominent models,
including GRU4Rec, SASRec, and BERT4Rec. For consistent and unbiased
assessment, all models underwent identical preprocessing and were evaluated
using two metrics: NDCG@k and Hit Ratio@k (HR@k), with $k\in[5,10,20]$. The
results, delineated in Table 2, yield several insightful observations:Irl4Rec
outperforms the other three models, solidifying its superior capabilities in
the given context. Our proposed structure exhibits enhanced performance
particularly with BERT4Rec. This can be attributed to the BERT encoder’s
prowess in gleaning pertinent information from historical data. When
juxtaposing the improvements in BERT4Rec and GRU4Rec, it’s evident that for
the beauty dataset, the enhancements in BERT4Rec surpassed those in GRU4Rec.
Conversely, for the sports dataset, our Irl4Rec displayed a more significant
boost when paired with GRU4Rec. Overall, the standout performance of Irl4Rec
can be traced back to its adeptness at discerning and leveraging spurious
relationships inherent in the datasets. "In our evaluation of the SASRec
model, we found evidence of our proposed models’ efficiency across different
datasets. While there was a decline in the NDCG@10 metric for the sports
dataset, the overall experimental results still indicate that our models are
efficient.

Table 2: Performance comparisons of proposed model and base model(sport) Model | HR@5 | HR@10 | HR@20 | NDCG@5 | NDCG@10 | NDCG@20
---|---|---|---|---|---|---
GRU4Rec | 0.0126 | 0.0203 | 0.0316 | 0.0082 | 0.0106 | 0.0135
GRU4Rec+ | 0.0164 | 0.0250 | 0.0394 | 0.0107 | 0.0135 | 0.0171
improvement(%) | 30.16 | 23.15 | 25.08 | 30.49 | 27.36 | 26.67
BERT4Rec | 0.0217 | 0.0359 | 0.0604 | 0.0143 | 0.0190 | 0.0251
BERT4Rec+ | 0.0260 | 0.0385 | 0.0566 | 0.0172 | 0.0213 | 0.0258
improvement(%) | 19.81 | 7.24 | -6.29 | 20.27 | 10.94 | 2.79
SASRec | 0.0214 | 0.0333 | 0.0500 | 0.0144 | 0.0177 | 0.0224
SASRec+ | 0.0248 | 0.0369 | 0.0549 | 0.0207 | 0.0168 | 0.0252
improvement(%) | 15.89 | 10.81 | 9.8 | 43.75 | -5.08 | 12.5
Table 3: Performance comparisons of proposed model and base model(beauty) Model | HR@5 | HR@10 | HR@20 | NDCG@5 | NDCG@10 | NDCG@20
---|---|---|---|---|---|---
GRU4Rec | 0.0168 | 0.0289 | 0.0461 | 0.0103 | 0.0142 | 0.0185
GRU4Rec+ | 0.0203 | 0.0334 | 0.0559 | 0.0119 | 0.0164 | 0.0218
improvement(%) | 20.83 | 15.57 | 21.26 | 15.53 | 15.49 | 17.84
BERT4Rec | 0.0360 | 0.0601 | 0.0984 | 0.0216 | 0.0300 | 0.0391
BERT4Rec+ | 0.0458 | 0.0661 | 0.0935 | 0.0302 | 0.0367 | 0.0435
improvement(%) | 27.22 | 9.98 | -4.97 | 39.81 | 22.33 | 11.25
SASRec | 0.0377 | 0.0624 | 0.0894 | 0.0241 | 0.0342 | 0.0386
SASRec+ | 0.0441 | 0.0656 | 0.0937 | 0.0299 | 0.0368 | 0.0439
improvement(%) | 30.86 | 5.12 | 4.81 | 24.06 | 7.60 | 1373

###  5.2 RQ2(Ablation Study)

In our ablation study, we adopted NDCG@20 and HR@20 as evaluation metrics and
centered our experiments on the BERT4Rec model, given its recent prominence
and frequent application in sequential recommendation tasks. The results
reveal several key observations: For the beauty dataset, our model
consistently surpasses other approaches in all three segments of the ablation
study, marking a new state-of-the-art benchmark. In contrast, with the sports
dataset, term (2) doesn’t significantly impact the predictions. This
discrepancy could be attributed to the sports dataset’s larger volume when
compared to the beauty dataset, thereby heightening the likelihood of
overfitting. Omitting term (1) leads to a discernible decline in performance,
emphasizing its integral role. This outcome aligns with expectations since
term (1) functions as the training objective for adjustment variables, which
are pivotal in the recommendation schema. In summation, every constituent of
our proposed framework collectively converges to realize the maximal
performance gains.

Table 4: Ablation study(on Bert4Rec) | sport | beauty
---|---|---
BERT4Rec+ | HR@20 | NDCG@20 | HR@20 | NDCG@20
raw | 0.0550 | 0.0252 | 0.0937 | 0.0439
w/o (1) | 0.0436 | 0.0184 | 0.0776 | 0.0344
w/o (2) | 0.0553 | 0.0256 | 0.0935 | 0.0435
w/o (3) | 0.0550 | 0.0246 | 0.0908 | 0.0413

##  6 CONCLUSION AND FUTURE WORK

In this study, we present an innovative approach to Top-k sequential
recommendation with a particular focus on addressing the challenge posed by
spurious relations. To this end, we introduce a causal graph methodology and
unveil the Irl4Rec model. Irl4Rec is rooted in the principles of invariant
learning. It not only emphasizes the intricate relationship between spurious
relation variables and adjustment variables but also proposes a novel
objective to address the issue at hand. Our experiments, conducted on Amazon’s
sport and beauty datasets, attest to Irl4Rec’s prowess in enhancing the
efficacy of existing methods. This marked improvement can be primarily
attributed to the adept integration of invariant learning coupled with our
novel objective.

However, Irl4Rec is not without its limitations. Its integration with
contrastive learning — a leading technique in sequential recommendation —
remains a challenge. As we chart our future research direction, our ambition
is to devise strategies that effectively handle spurious relations in
sequential history items, all the while harnessing the power of contrastive
learning during training. Ultimately, we aspire to refine Irl4Rec such that it
seamlessly integrates with a broader spectrum of sophisticated models.

####  6.0.1 Acknowledgements

Please place your acknowledgments at the end of the paper, preceded by an
unnumbered run-in heading (i.e. 3rd-level heading).

## References

  * [1] K. Ahuja, E. Caballero, D. Zhang, J.-C. Gagnon-Audet, Y. Bengio, I. Mitliagkas, and I. Rish.  Invariance principle meets information bottleneck for out-of-distribution generalization, 2022.
  * [2] K. Ahuja, K. Shanmugam, K. R. Varshney, and A. Dhurandhar.  Invariant risk minimization games, 2020.
  * [3] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz.  Invariant risk minimization, 2020.
  * [4] P. Bühlmann.  Invariance, causality and robustness.  2020\.
  * [5] S. Feng, X. Li, Y. Zeng, G. Cong, and Y. M. Chee.  Personalized ranking metric embedding for next new poi recommendation.  In IJCAI’15 Proceedings of the 24th International Conference on Artificial Intelligence, pages 2069–2075. ACM, 2015.
  * [6] F. Garcin, C. Dimitrakakis, and B. Faltings.  Personalized news recommendation with context trees.  In Proceedings of the 7th ACM Conference on Recommender Systems, pages 105–112, 2013.
  * [7] B. Hidasi and A. Karatzoglou.  Recurrent neural networks with top-k gains for session-based recommendations.  In Proceedings of the 27th ACM international conference on information and knowledge management, pages 843–852, 2018.
  * [8] A. Jaiswal, R. Brekelmans, D. Moyer, G. V. Steeg, W. AbdAlmageed, and P. Natarajan.  Discovery and separation of features for invariant representation learning, 2019.
  * [9] W.-C. Kang and J. McAuley.  Self-attentive sequential recommendation.  In 2018 IEEE international conference on data mining (ICDM), pages 197–206. IEEE, 2018.
  * [10] W.-C. Kang and J. McAuley.  Self-attentive sequential recommendation, 2018.
  * [11] K. Khezeli, A. Blaas, F. Soboczenski, N. Chia, and J. Kalantari.  On invariance penalties for risk minimization, 2021.
  * [12] M. Koyama and S. Yamaguchi.  When is invariance useful in an out-of-distribution generalization problem ?, 2021.
  * [13] W. Krichene and S. Rendle.  On sampled metrics for item recommendation.  In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1748–1757, 2020.
  * [14] D. Liang, R. G. Krishnan, M. D. Hoffman, and T. Jebara.  Variational autoencoders for collaborative filtering, 2018.
  * [15] J. Liu, Z. Hu, P. Cui, B. Li, and Z. Shen.  Heterogeneous risk minimization.  In International Conference on Machine Learning, pages 6804–6814. PMLR, 2021.
  * [16] X. Qin, H. Yuan, P. Zhao, J. Fang, F. Zhuang, G. Liu, Y. Liu, and V. Sheng.  Meta-optimized contrastive learning for sequential recommendation.  In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, jul 2023\.
  * [17] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang.  Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer.  In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1441–1450, 2019.
  * [18] S. Wang, L. Hu, Y. Wang, L. Cao, Q. Z. Sheng, and M. Orgun.  Sequential recommender systems: challenges, progress and prospects.  arXiv preprint arXiv:2001.04830, 2019.
  * [19] X. Wang, X. He, M. Wang, F. Feng, and T.-S. Chua.  Neural graph collaborative filtering.  In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, jul 2019\.
  * [20] Z. Wang, X. Chen, R. Wen, S.-L. Huang, E. E. Kuruoglu, and Y. Zheng.  Information theoretic counterfactual learning from missing-not-at-random feedback, 2020.
  * [21] G.-E. Yap, X.-L. Li, and P. S. Yu.  Effective next-items recommendation via personalized sequential pattern mining.  In Database Systems for Advanced Applications: 17th International Conference, DASFAA 2012, Busan, South Korea, April 15-19, 2012, Proceedings, Part II 17, pages 48–64. Springer, 2012.
  * [22] K. Zhou, H. Wang, W. X. Zhao, Y. Zhu, S. Wang, F. Zhang, Z. Wang, and J.-R. Wen.  S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization.  In Proceedings of the 29th ACM international conference on information & knowledge management, pages 1893–1902, 2020.