# Continual Zero-Shot Learning through Semantically Guided Generative Random
Walks

Wenxuan Zhang1 Paul Janson1,2 Kai Yi1 Ivan Skorokhodov1 Work done during
internship at KAUST Mohamed Elhoseiny1
KAUST1 University of Moratuwa2
{wenxuan.zhang, remond.janson, kai.yi, ivan.skorokhodov,
<EMAIL_ADDRESS>
###### Abstract

Learning novel concepts, remembering previous knowledge, and adapting it to
future tasks occur simultaneously throughout a human’s lifetime. To model such
comprehensive abilities, continual zero-shot learning (CZSL) has recently been
introduced. However, most existing methods overused unseen semantic
information that may not be continually accessible in realistic settings. In
this paper, we address the challenge of continual zero-shot learning where
unseen information is not provided during training, by leveraging generative
modeling. The heart of the generative-based methods is to learn quality
representations from seen classes to improve the generative understanding of
the unseen visual space. Motivated by this, we introduce generalization-bound
tools and provide the first theoretical explanation for the benefits of
generative modeling to CZSL tasks. Guided by the theoretical analysis, we then
propose our learning algorithm that employs a novel semantically guided
Generative Random Walk (GRW) loss. The GRW loss augments the training by
continually encouraging the model to generate realistic and characterized
samples to represent the unseen space. Our algorithm achieves state-of-the-art
performance on AWA1, AWA2, CUB, and SUN datasets, surpassing existing CZSL
methods by 3-7%. The code has been made available here https://github.com/wx-
zhang/IGCZSL

.

##  1 Introduction

Figure 1: Semantically guided generative random walk (GRW): At each time step,
new classes are added to the seen classes space, and the random walk starts
from each seen class center (in green) and transitions through generated
samples of hallucinated classes (in orange), then the landing probability
distribution over the seen classes is predicted. The GRW loss encourages the
generated samples from the hallucinated classes to be distinguishable from the
seen classes by encouraging the landing probability over seen classes starting
from any seen center to be uniformly distributed, and hence hard to classify
to any seen class.

Researchers have devoted significant effort to developing AI learners to mimic
human cognition. One such endeavor is zero-shot learning (ZSL), which aims to
identify unseen classes without accessing any of their images during training.
However, human zero-shot learning abilities improve dynamically over time. As
individuals acquire more knowledge of seen tasks, they become better at
recognizing unseen tasks. To evaluate the zero-shot learning in such a dynamic
seen-unseen distribution, the continual zero-shot learning problem (CZSL) has
been proposed [53]. CZSL emulates the continuous learning process of a human’s
life, where the model continually sees more classes from the unseen world and
is evaluated on both seen and unseen classes. This CZSL skill, may it get
maturely developed to the world scale, has the potential to accelerate
research in species discovery, for example, as known species grow continually,
but close to 90% of the species are not yet discovered[55].

Generative models (e.g., GANs[26]) have made significant progress in producing
photorealistic images by learning high-dimensional probability distributions.
This ability motivated researchers to adapt GANs to ZSL to generate missing
data of unseen classes conditioning on unseen semantic information, known as
generative-based ZSL. Training the classifier on synthetic unseen samples can
reduce model prediction bias towards seen classes and thus achieves
competitive zero-shot learning performance [38, 58, 42]. Some CZSL works
directly adopt this framework continually, known as transductive continual
zero-shot learning [25, 36]. However, in CZSL, the unseen world changes
dynamically and unexpectedly, making it unrealistic to use prior knowledge
about unseen classes[53]. When we do not assume access to unseen semantic
information in the CZSL setting, which is known as _inductive continual zero-
shot learning_ , most existing methods struggle to perform well, as we show in
our experiments. Furthermore, the theoretical understanding of how zero-shot
learning benefits from synthetic data is limited, which poses an obstacle to
developing purely inductive continual zero-shot methods. Recent analyses of
training generative models with synthetic data [8] provide a possible avenue
for developing the desired theoretical explanation. This led us to develop a
generalization-bound tool to understand the learning mechanism in generative-
based CZSL and further develop inductive methods based on it.

In our analysis, we have identified it is crucial to reduce the distance
between the generated and actual visual space of unseen classes. This requires
the model to generate realistic samples to represent unseen space to augment
the training of the classifier. However, the lack of ground truth semantic
descriptions for unseen classes and the lack of previously seen classes data
often leads to the generated samples collapsing to the seen classes. A similar
problem has been addressed in generating novel style artworks, where GAN
training is augmented to encourage the generated styles to deviate from
existing art style classes [15, 51, 27, 30, 28, 31]. Drawing inspiration from
the improved feature representation achieved by generative models in producing
novel art, and the connection between the ability to generate novel styles in
art generation and to generate samples to represent the unseen space in
generative-based CZSL, we propose a purely inductive, Generative Random Walk
(GRW) loss, guided only by semantic descriptions of seen classes.

In each continual learning task, we first hallucinate some classes by
interpolating on or sampling from a learnable dictionary based on the current
and previous classes, with the belief that the realistic classes, both seen
and unseen, should be relatable to each other [16, 17]. We then generate
samples from the hallucinated classes. To prevent the generated samples of
hallucinated classes from collapsing to the seen classes, we apply the GRW
loss, as illustrated in Figure 1. We perform a random walk starting from the
seen class and moving through generated examples of hallucinated classes for
$R$ steps, as described in detail later in Section 5.2.2. The GRW loss
encourages high transition probabilities to the realistic unseen space by
deviating from the visual space of the seen classes and avoiding less
realistic areas. The resulting representations are both realistic and
distinguishable from seen classes, which enhances the generative understanding
of unseen classes. This approach is particularly effective when the model is
updated continually, as it enables the model to use the newly learned
knowledge to improve further the generated examples of hallucinated classes.
Our contributions lie in

  * •

We provide a theoretical analysis of continual zero-shot learning. This
analysis guides us to use proper signals to make up for the missing unseen
information. We present these generalization-bound tools for the analysis in
Section 4.

  * •

Guided by the analysis, we develop a method for purely inductive continual
zero-shot learning; described in detail in Section 5. Our method, ICGZSL,
first provides two ways to hallucinate classes, _i.e_. interpolation of two
seen classes and learning a dictionary based on the seen classes. Then, we
integrate our introduced semantically guided Generative Random Walk (GRW) loss
to generate distinguishable and realistic samples to represent unseen classes.

  * •

We performed comprehensive experiments (Section 6) that demonstrate the
effectiveness of our approach. Specifically, our model achieves state-of-the-
art results on standard continual zero-shot learning benchmarks, AWA1, AWA2,
CUB, SUN, and performs often better than transductive methods

##  2 Related Works

Inductive and Transductive Zero-Shot Learning. There are varying degrees of
accessibility to unseen information in zero-shot learning. Transductive
methods use both unlabeled samples and attributes of unseen classes during
training [44, 48]. Semantically transductive methods, on the other hand, only
use attributes of unseen classes in training [63, 60]. In the inductive
setting, however, no unseen information is allowed to be used(_e.g_., [69, 16,
41, 64]). This can result in a bias towards seen classes [45]. Generative
methods, such as those used by [69, 41, 16], can produce unseen samples using
only seen class information during training to solve this issue. For example,
[16] relate zero-shot learning to human creativity to generate images that
deviate from seen classes during training. [7] used unlabeled samples from
out-of-distribution data to gather knowledge about unseen data. [54] utilize
two variational autoencoders to generate latent representations for visual and
semantic modalities in a shared latent space. In contrast, our approach
focuses on investigating the relationship between the generated samples of
hallucinated classes and the seen classes, which leads to GRW loss.

Continual Learning.  The majority of continual learning works aim to tackle
the problem of catastrophic forgetting, where the data representation becomes
biased towards the most recent task in sequential learning. Regularization-
based methods [39, 4], structure-based methods [49, 13], and replay-based
methods [52, 65] have been proposed to resolve this problem. More recently,
research has explored forward transfer in continual learning, with the belief
that as knowledge accumulates, higher next-task transferability, as measured
by zero-shot assessment, should be attained. Their evaluation space either
includes the next task [40] or the whole class space [12]. However, compared
to our setting, [40] did not evaluate the model in a generalized manner, and
[12] only paid attention to the seen accuracy.

Continual Zero-shot learning.  [11] introduced A-GEM for continual learning,
which was later applied to deal with zero-shot tasks sequentially, laying the
foundation for the initial work on CZSL. [53] proposed the inductive CZSL
scenario and demonstrated that a class-based normalization approach can
improve performance in continual zero-shot learning. Both [21] and [24]
explore the CZSL problem, but rely on unseen class descriptions to train a
classifier before inference. [36] proposed a generative adversarial approach
with a cosine similarity-based classifier that supports the dynamic addition
of classes without requiring unseen samples for training. Their approach also
relies on unseen class descriptions for seen-unseen deviation, making it a
semantically transductive method. This motivated us to explore a purely
inductive method for handling seen-unseen deviation and improving the realism
of unseen samples.

##  3 Problem Setup and Notations

###  3.1 Formulation

We start by defining our problem and notations. A labelled dataset is defined
as a tuple
${\bm{D}}=\\{({\bm{x}},{\bm{a}},y)|y=f({\bm{x}}),({\bm{x}},{\bm{a}},y)\sim\mathcal{D}\\}$,
where $\mathcal{D}$ represents the data distribution. Each data point is a
tuple of image feature ${\bm{x}}\in\mathbb{R}^{d_{x}}$, class attribute
${\bm{a}}\in\mathbb{R}^{d_{a}}$, and a class label $y$. Here $d_{x}$ is the
dimension of the visual feature space, and $d_{a}$ is the dimension of the
attribute space. Each distribution has a specific labeling function $f$. Our
goal is to learn a model $\hat{f}$ on top of ${\bm{D}}$ to estimate $f$. We
study the continual zero-shot learning setting proposed by [53], where we seek
to learn the model $\hat{f}$ on a stream of tasks. In each task $t$, the model
is learned on the seen dataset ${\bm{D}}^{t}_{s}$, and is evaluated on both
the seen distribution $\mathcal{D}^{t}_{s}$ and unseen distribution
$\mathcal{D}^{t}_{u}$. Moreover, we assume that the set of seen class and
unseen class are disjoint, that is $\mathcal{D}_{s}\cap\mathcal{D}_{u}=\phi$.
This procedure is illustrated in the bottom part of Figure 2.

We use generative models as the backbone. During the training time, the model
$\hat{f}$ is trained on the seen dataset ${{\bm{D}}}_{s}$ as well as the
synthesized dataset ${{\bm{D}}}_{h}$. ${{\bm{D}}}_{h}$ is generated by
conditioning on hallucinated attributes ${{\bm{a}}}_{h}$ and prior
$\mathcal{Z}\sim\mathcal{N}(0,1)$. The labeling function ${f}_{h}$ of the
generated dataset is a look-up table of the generated features
${{\bm{x}}}\in{{\bm{X}}}_{h}$ and the corresponding attribute condition
${{\bm{a}}}_{h}$.

###  3.2 Notations.

In our theoretical analysis, we use the following notations: 1) We discuss the
relationship between the three types of variables, namely, real seen sample,
real unseen samples, and generated samples from the hallucinated classes. To
specify the variables related to these types of samples, we use subscripts
$\cdot_{s},\cdot_{u},\cdot_{h}$ respectively, _e.g_., $f_{s},f_{u},f_{h}$ ; 2)
We denote the values and model empirically computed by a variable with a hat,
_e.g_., $\hat{f}$; 3) We use superscripts $\cdot^{t}$ or $\cdot^{1:t}$ to
indicate that a variable is for task $t$ or for tasks $1:t$ respectively,
_e.g_., $f^{t}_{s},f_{s}^{1:t}$; 4) ${\bm{D}}$ is used for the empirical
sample set, and $\mathcal{D}$ is used for the distribution; 5) We use $N_{s}$
and $N_{u}$ to denote the number of seen and unseen classes.

In practice, the unseen information, _i.e_.,
${\bm{a}}_{u},{\bm{D}}_{u},N_{u}$, is not available. Therefore, we hallucinate
some classes denoted by ${\bm{a}}_{h}$ and generate samples
${\bm{D}}_{h}=\\{({\bm{x}}_{h},{\bm{a}}_{h})\\}$ by conditioning on these
attributes. We use $N_{h}$ to represent the number of hallucinated classes.
Additionally, we do not have access to all the previous data , so
${\bm{X}}^{1:t}$ refers to the current samples as well as the previous ones in
the buffer. We also use generated seen samples $\cdot_{sg}$ for GAN training.

##  4 Theoretical Analysis

As mentioned in the introduction, we propose using hallucinated classes to
represent the unseen space. By training our model on synthetic samples
generated from these classes, we improve the model’s generalization ability to
the actual unseen classes during the testing time of continual zero-shot
learning. In this section, we quantify the model’s generalization ability by
measuring the distance between the synthetic samples that represent the unseen
space and the actual unseen samples. Additionally, we explain our motivation
for using a random walk-based method to reduce this distance when no
information about the unseen space is available.

###  4.1 Generalization Bound Inductive Continual Zero-Shot Learning

In this section, we present a generalization bound for a continual zero-shot
learning algorithm. Given the entire training distribution, a learning
algorithm can output an optimal hypothesis $h$ that estimates the ground truth
labeling function $f$. However, since the learning algorithm can only be
trained on a finite sample from the training set, it outputs an empirical
hypothesis $\hat{h}$ to estimate the ground truth labeling function. We define
the generalization error [33] for these two types of hypotheses. We define the
actual risk,

$\small\epsilon(h,f)=\mathbb{E}_{({\bm{x}},{\bm{a}})\sim\mathcal{D}}[\mathbbm{1}_{f({\bm{x}})\neq
h({\bm{x}},{\bm{a}})}]\enspace,$ (1)

which measures the expected probability of a disagreement between the ground
truth and the optimal hypothesis. We also define the empirical risk on the
finite sample set ${\bm{D}}$,

$\small\hat{\epsilon}(\hat{h},f)=\frac{1}{|{\bm{D}}|}\sum_{({\bm{x}},{\bm{a}})\in{\bm{D}}}\mathbbm{1}_{f({\bm{x}})\neq\hat{h}({\bm{x}},{\bm{a}})}\enspace,$
(2)

which measures the probability of a disagreement between the ground truth and
the empirical hypothesis.

In a continual zero-shot learning algorithm, given a training set
${\bm{D}}^{t}_{s}$, the algorithm outputs $\hat{h}$ to estimate
$f^{1:t}_{s}\cup f_{u}$ instead of the ground truth labeling function $f_{s}$.
To begin our analysis, we propose a distance measure between the generated
unseen distribution111In the transductive setting, the unseen distribution is
generated by conditioning on the unseen semantic information. In our work, we
utilize generated samples from hallucinated classes to represent the generated
unseen distribution. and the real unseen distribution
$\bar{d}_{GDB}({\bm{D}}_{h},{\bm{D}}_{u})$ as follows:

######  Definition 4.1 (Empirical Generative distance).

Given the training set ${\bm{D}}_{s}$ and the synthetic set ${\bm{D}}_{h}$ ,
the ground truth labeling functions $f_{s}$, $f_{h}$, and $f_{u}$, and the
optimal hypothesis $\hat{h}^{*}=\operatorname*{arg\,min}_{h\in
H}\hat{\epsilon}_{s}(h,f_{s})+\hat{\epsilon}_{h}(h,f_{h})$ obtained by
training the model on ${\bm{D}}_{h}$ and ${\bm{D}}_{s}$, we can define the
distance between ${\bm{D}}_{h}$ and ${\bm{D}}_{u}$ as follows:

$\small\bar{d}_{GDB}({\bm{D}}_{h},{\bm{D}}_{u})=|\hat{\epsilon}(\hat{h}^{*},f_{u})-\hat{\epsilon}(\hat{h}^{*},f_{h})|\enspace.$
(3)

Our proposed $\bar{d}_{GDB}$ is a feasible distance measure that satisfies the
properties of a pseudo-metric. In the following, we present our generalization
bound following [8] for the continual zero-shot learning algorithm, which
shows how the generalization ability of the zero-shot learning algorithm is
mainly influenced by this distance.

######  Theorem 4.2 (Generalization bound of the generative-based CZSL).

Given the CZSL procedure described in section 3.1, with confidence $1-\delta$
the risk on the unseen distribution is bounded by

$\footnotesize\begin{split}\epsilon(h,f_{u}^{t})\leq&\hat{\epsilon}(\hat{h}^{*},f_{s}^{1:t})+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{s}^{1:t},\mathcal{D}^{t}_{u})+\bar{\lambda}\\\
&+\frac{1}{2}\bar{d}_{GDB}({\bm{D}}^{t}_{u},{\bm{D}}^{t}_{h})+C(\frac{1}{m}+\frac{1}{\delta})\end{split}$
(4)

where $\hat{h}^{*}=\operatorname*{arg\,min}_{h\in
H}\sum_{i=1}^{t}\hat{\epsilon}(h,f_{s}^{i})+\hat{\epsilon}(h,f^{t}_{h})$,
$\bar{\lambda}=\hat{\epsilon}(\hat{h}^{*},f_{s}^{1:t})+\hat{\epsilon}(\hat{h}^{*},f^{t}_{h})$.

In Equation 24, measurement $d_{\mathcal{H}\Delta\mathcal{H}}$ [6] is used to
quantify the difference between two distributions for domain adaptation based
on the type of model, and is fixed for a specific problem. $\bar{\lambda}$ and
$\hat{\epsilon}_{s}(h,f_{s})$ are highly depended on the optimization
algorithm. However, if we hallucinate a diverse set of classes, $f_{u}$ can be
compactly supported by $f_{h}$. If we further generate realistic samples for
each of the hallucinated classes, the optimal solution trained on the
synthetic set, $\hat{h}^{*}=\operatorname*{arg\,min}_{h\in
H}\hat{\epsilon}_{s}(h,f_{s})+\hat{\epsilon}_{h}(h,f_{h})$, should perform
well on the real unseen dataset. This can lead to a reduction in
$\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})$ in Equation 3. We will discuss this
further in the following section. The detailed derivation of this theorem can
be found in Appendix A.1.

###  4.2 Reducing the bound using Markov Chain.

To reduce $\bar{d}_{GDB}({\bm{D}}_{h},{\bm{D}}_{u})$ in Equation 24, we need
to decrease the difference between ${\bm{D}}_{u}$ and ${\bm{D}}_{h}$. One
approach proposed by [16] is to hallucinate ${\bm{a}}_{h}$ as a compact
support of ${\bm{a}}_{u}$. Once we have achieved this, we can further generate
high-quality samples to increase
$\mathbb{P}[{\bm{D}}_{u}\subset{\bm{D}}_{h}]$, where the probability is taken
over all possible generations.

To quantify the probability value
$\mathbb{P}[{\bm{D}}_{u}\subset{\bm{D}}_{h}]$, we follow the approach of [57]
and view the generations as nodes in a Markov chain. We define the transition
probability between two states as the probability with which one sample is
classified as another. Then, we can bound
$\mathbb{P}[{\bm{D}}_{u}\subset{\bm{D}}_{h}]$ by the self-transition
probability using a generalization bound. When the self-transition probability
is the same in two sets of generations, we prefer the one with higher
diversity quantified by DDP, as suggested by [32] and [14].

For detailed explanations, please refer to Appendix A.2. Here, we provide an
informal statement.

######  Statement 4.3.

Finding generated samples from hallucinated classes to “carefully” increase
the determinant and the diagonal entries of the transition matrix of the above
described Markov Chain can reduce $\bar{d}_{GDB}$.

We can now design an algorithm that first hallucinates classes and then
generates diverse samples from these classes to represent the unseen space
that follows Statement 4.3. However, the transition matrix of the Markov chain
described above is intractable to compute in practice. To quantify the
transition probability, we adapt the random walk framework [29, 5] originally
used in semi-supervised few-shot learning to generative zero-shot learning
with a few yet important changes. Please refer to Appendix B.4 the relation
between our work and previous work.

We also make the following two adjustments to the statement to encourage the
generated samples from hallucinated classes to be consistently realistic like
the real samples. Firstly, we represent the transition matrix among
hallucinated classes (noted as
${\bm{P}}^{X_{h}X_{h}}\in\mathbb{R}^{N_{h}\times N_{h}}$) in the seen class
space using a congruent transformation
${\bm{P}}^{{\bm{C}}_{s}X_{h}}{\bm{P}}^{X_{h}X_{h}}{\bm{P}}^{X_{h}C_{s}}$,
where ${\bm{P}}^{{\bm{C}}_{s}X_{h}}\in\mathbb{R}^{N_{s}\times N_{h}}$ is the
transition probability matrix from seen prototypes to generated samples from
hallucinated classes, and ${\bm{P}}^{X_{h}C_{s}}$ is the opposite. Secondly,
hallucinating a compact support of unseen class attributes and encouraging the
transition matrix to be diagonal requires a huge number of generations. To
reduce this number, we encourage the generated samples of hallucinated classes
to have a "relatable deviation" to the seen classes. The relationship between
the two types of samples is that both should be realistic. This means that the
transition matrix ${\bm{P}}^{X_{h}X_{h}}$ may not be strictly diagonal, and
our goal is to reduce the non-diagonal entries, _i.e_., to reduce the
transition probability between different generated samples of hallucinated
classes. We further repeat the transition among generated samples of
hallucinated classes to further reduce the non-diagonal entries.

In conclusion, our transitions start from the seen prototypes to generated
samples of hallucinated classes for ${R}$ steps and back to seen prototypes,
the transition matrix of which is
${\bm{P}}^{{\bm{C}}_{s}X_{h}}({{\bm{P}}^{X_{h}X_{h}}})^{R}{\bm{P}}^{X_{h}C_{s}}\in\mathbb{R}^{N_{s}\times
N_{s}}$. To encourage“relatable deviation” of the generated samples of
hallucinated classes from seen classes, we aim to reduce the non-diagonal
entries of the transition matrix , as detailed later in Section 5. This
approach intuitively prevents the generations from being attracted by any seen
classes, and theoretically can reduce the distance $\bar{d}_{GDB}$.
Intentionally, this method also transfers knowledge between seen and
hallucinated classes, which is useful for generating realistic images.

##  5 Generative-based Inductive CZSL Approach

Method overview. Generative-based inductive CZSL algorithms adopt generative
models as their architecture, where seen samples are used to train the
classifier to correctly classify seen classes, and generators are trained to
generate realistic samples. At the same time, the generator is encouraged to
synthesize samples to represent unseen classes to train the classifier to
perform classification on these samples. In our work, we can only hallucinate
some classes to represent the actual unseen space and generate samples from
the hallucinated classes. As guided by our analysis, the key point of
inductive zero-shot learning is to generate realistic and diverse samples from
hallucinated classes that are deviated from the real seen space. We introduce
the generative model backbone in Section 5.1, and how we generate the
abovementioned samples in Section 5.2. The overall procedure is shown in
Algorithm 1 in Appendix.

###  5.1 Generative-based CZSL baseline

Figure 2: The discriminator embeds attributes ${\bm{a}}_{s}^{1:t}$ into the
real feature space to perform classification with real samples $x_{s}^{t}$,
while the generator produces features ${\bm{x}}^{t}_{h}$ and
${\bm{x}}^{t}_{sg}$ conditioning on the corresponding attributes. The real-
fake loss and classification loss encourage the generated sample distribution
consistent with the real samples. Then the inductive loss applied to the
generated feature space, which encourages the characterization of generated
samples from hallucinated classes, can reduce the bias towards the current
seen classes of the classifier and improve continual zero-shot learning
performacne.

We follow [36] as our baseline. The model contains a generator
$G({\bm{a}},{\bm{z}}):\mathbb{R}^{d_{a}+d_{z}}\to\mathbb{R}^{d_{x}}$ and a
discriminator $D({\bm{a}}):\mathbb{R}^{d_{a}}\to\mathbb{R}^{d_{x}}$. The
generator takes the semantic information (denoted by ${\bm{a}}$) and the prior
(denoted by ${\bm{z}}$) sampled from a standard normal distribution
${\mathcal{Z}}$ as input and outputs visual features. Discriminator projects
semantic information ${\bm{a}}$ into visual space. The conditional adversarial
training can be illustrated by the discriminator loss and generator loss as:

$\small\begin{split}\small\mathcal{L}_{D}&=-\mathcal{L}_{\text{real-
fake}}+\lambda_{\text{cls}}\mathcal{L}_{\text{classification}}+\lambda_{\text{rd}}\mathcal{R}_{D},\,\,\,\,\\\
\mathcal{L}_{G}&=\mathcal{L}_{\text{real-
fake}}+\lambda_{\text{cls}}\mathcal{L}_{\text{classification}}+\mathcal{L}_{\text{inductive}}+\lambda_{\text{rg}}\mathcal{R}_{G}.\\\
\end{split}$ (5)

As shown in Figure 2, we use $\mathcal{L}_{\text{real-fake}}$ to denote the
GAN loss that discriminates between the real and fake samples for the current
task, and $\mathcal{L}_{\text{classification}}$ to denote the entropy loss
based on cosine similarity that is used to perform the classification of all
seen classes up to the current task. The equations for
$\mathcal{L}_{\text{real-fake}}$ and $\mathcal{L}_{\text{classification}}$ are
shown below

$\footnotesize\begin{split}\mathcal{L}_{\text{real-
fake}}&=\mathbb{E}_{({\bm{x}},{\bm{a}})\sim{\bm{D}}^{t}_{s}}\big{[}\log\langle{\bm{x}},D({\bm{a}})\rangle\big{]}\\\
&-\mathbb{E}_{{\bm{z}}\sim\mathcal{Z},({\bm{a}},{\bm{x}})\sim{\bm{D}}^{t}_{s}}\big{[}\log\langle
G(z,{\bm{a}}),D({\bm{a}})\rangle\big{]}\\\
\mathcal{L}_{\text{classification}}&=\mathbb{E}_{({\bm{x}},y)\sim{\bm{D}}^{1:t}_{s}}\big{[}L_{e}(\langle{\bm{x}},D({\bm{A}}_{s}^{1:t})\rangle,y)\big{]}\\\
&+\mathbb{E}_{{{\bm{z}}\sim\mathcal{Z},({\bm{x}},{\bm{a}},y)\sim{\bm{D}}^{1:t}_{s}}}\big{[}L_{e}(\langle
G({\bm{z}},{\bm{a}}),D({\bm{A}}_{s}^{1:t})\rangle,y)\big{]}\end{split}$ (6)

where $\langle\cdot,\cdot\rangle$ represents the cosine similarity,
${\bm{A}}_{s}^{1:t}$ is a matrix of attributes of seen classes up to the
current task, and $L_{e}$ is the cross-entropy loss. In practice,
${\bm{D}}^{1:t}_{s}$ consists of the current samples and previous samples in
the buffer. We follow [36] for regularization terms
$\mathcal{R}_{D},\mathcal{R}_{G}$ and $\lambda_{c,rd,rg}$. See Appendix B.1
for more details about the baseline algorithm.
$\mathcal{L}_{\text{inductive}}$ with its corresponding $\lambda_{i}$ is the
main component to improve inductive continual zero-shot learning, which will
be described in detail in section 5.2.

###  5.2 Inductive Loss

####  5.2.1 Hallucinate Attributes

To begin our method, we first hallucinate classes to represent the unseen
space. During this procedure, we aim to generate a diverse and compact set of
attributes without using any information from the unseen test set.

Interpolation-based method. When the attributes are distributed uniformly in
the attribute space, which can be compactly supported by the seen attribute,
we use interpolation-based method. To hallucinate the attributes at every
mini-batch, we use an interpolation-based method that was introduced by [16].
Hallucinated attributes are generated using the formula
${\bm{a}}_{ug}=\alpha{\bm{a}}_{s_{1}}+(1-\alpha){\bm{a}}_{s_{2}}$, where
$\alpha$ is drawn from the uniform a distribution $\mathcal{U}(0.2,0.8)$, and
${\bm{a}}_{s_{1}}$ and ${\bm{a}}_{s_{2}}$ are two randomly chosen seen
attributes. The sample interval is chosen to be $(0.2,0.8)$ to ensure that the
interpolated attributes are not too close to the seen attributes.

Dictionary-based method. We further propose to learn an attribute dictionary
containing $N_{s}^{t}$ attribute vectors during training. The use of a
learnable dictionary allows the attributes to change more freely in accordance
with the loss function. The dictionary is randomly initialized by
interpolating seen attributes, and during the computation of GRW loss, we
randomly pick attributes from it. This approach is particularly useful for
classification at a finer level, where the attributes are more specific.

If the hallucinated class can accurately represent the actual unseen space,
which is only accessible during the test time, then the model will have good
generalization ability on the test set. We visualize the hallucinated classes
to examine if this assumption holds. Please refer to Appendix B.2 for the
visualization of our hallucinated attributes.

Dataset | AWA1 |  | AWA2 |  | CUB |  | SUN
---|---|---|---|---|---|---|---
Metric | mSA | mUA | mHA |  | mSA | mUA | mHA |  | mSA | mUA | mHA |  | mSA | mUA | mHA
EWC (cl) [34] | 29.4 | 9.0 | 13.8 |  | 30.8 | 10.5 | 15.8 |  | 12.2 | 0.8 | 1.3 |  | 11.6 | 2.6 | 4.1
A-GEM (cl) [11] | 64.2 | 3.9 | 7.2 |  | 65.8 | 6.7 | 11.9 |  | 14.4 | 0.4 | 0.8 |  | 8.6 | 3.0 | 4.2
Tf-GZSL (tr) [22] | 70.8 | 27.4 | 37.9 |  | 78.6 | 28.7 | 41.1 |  | 46.3 | 30.8 | 35.3 |  | 15.3 | 30.7 | 18.7
DVGR (tr) [25] | 65.1 | 28.5 | 38.0 |  | 73.5 | 28.8 | 40.6 |  | 44.9 | 14.6 | 21.7 |  | 22.4 | 10.7 | 14.5
A-CGZSL (tr) [24] | 71.0 | 24.3 | 35.8 |  | 70.2 | 25.9 | 37.2 |  | 34.3 | 12.4 | 17.4 |  | 17.2 | 6.3 | 9.7
BD-CGZSL (tr) [36] | 62.9 | 29.9 | 39.0 |  | 68.1 | 33.9 | 42.9 |  | 19.8 | 17.2 | 17.8 |  | 27.5 | 15.9 | 20.0
CN-CZSL (in) [53] | - | - | - |  | 33.6 | 6.4 | 10.8 |  | 44.3 | 14.8 | 22.7 |  | 22.2 | 8.2 | 12.5
BD-CGZSL-in (in) [36] | 62.1 | 31.5 | 40.5 |  | 67.7 | 32.9 | 42.3 |  | 37.8 | 9.1 | 14.4 |  | 34.9 | 14.9 | 20.8
CARNet (in) [23] | 67.6 | 27.4 | 37.0 |  | - | - | - |  | 42.4 | 12.4 | 18.8 |  | 31.5 | 15.9 | 20.9
ours \+ interpolation | 67.0 | 34.2 | 43.4 |  | 71.1 | 34.9 | 44.5 |  | 42.2 | 22.7 | 28.4 |  | 36.0 | 21.6 | 26.8
ours \+ dictionary | 67.1 | 33.5 | 41.6 |  | 70.2 | 35.1 | 44.6 |  | 42.4 | 23.6 | 28.8 |  | 36.5 | 21.8 | 27.1
Table 1: Our proposed method achieves state-of-the-art results when compared
with traditional continual learning method (cl) recent inductive (in) methods
and even shows competitive results in mHA with recent semantic transductive
methods (tr).

####  5.2.2 Improve Generation Quality by Inductive Loss

As we discussed in Section 4.2, we use GRW loss to improve the generation
quality such that the generated samples are realistic, diverse and
characterized. To encourage diversity of the samples generated from the
hallucinated attributes, we firstly generate only one sample for each
hallucinated attribute. And then, we perform a random walk to compute the
transition probability using generated seen samples ${\bm{X}}_{sg}$ and
generated samples from hallucinated samples ${\bm{X}}_{h}$. The random walk
starts from each generative seen class center
${\bm{C}}_{s}\in\mathbb{R}^{N_{s}^{1:t}\times d_{x}}$ computed by the mean of
generated seen samples from the corresponding class attributes, where
$N_{s}^{1:t}$ are the number of seen classes until step $t$. Then we take $R$
steps of transitions within generated samples of hallucinated classes
${\bm{X}}_{h}$ with the final landing probability over seen classes so far.
The transition probability matrix from seen class centers to generated samples
of hallucinated classes is defined as

$\small{\bm{P}}^{C_{s}X_{h}}=\sigma(\langle{\bm{C}}_{s},{\bm{X}}_{h}^{\top}\rangle)\enspace,$
(7)

where $\langle\cdot,\cdot\rangle$ is a similarity measure, and $\sigma(\cdot)$
is a softmax operator applied on rows. In practice, we use negative Euclidean
distance for similarity, that is, suppose ${\bm{x}}_{h}$ is the row $i$ of
${\bm{X}}_{h}$ and ${\bm{c}}$ is the class center $j$,

$\small\langle{\bm{C}}_{s},{\bm{X}}_{h}^{\top}\rangle_{i,j}=-\|{\bm{x}}_{h}-{\bm{c}}\|^{2}\enspace.$
(8)

Similarly, the transition probability matrix within generated samples of
hallucinated classes and from generated samples of hallucinated classes to
seen class centers are defined as

$\small{\bm{P}}^{X_{h}X_{h}}=\sigma(\langle{\bm{X}}_{h},{\bm{X}}_{h}^{\top}\rangle),{\bm{P}}^{X_{h}C_{s}}=\sigma(\langle{\bm{X}}_{h},{\bm{C}}_{s}^{\top}\rangle)\enspace.$
(9)

Then the random walk staring from each seen class center and transiting $R$
steps within generated samples of hallucinated classes and back to seen
centers are computed by

$\footnotesize
P^{{\bm{C}}_{s}{\bm{X}}_{h}{\bm{C}}_{s}}(R)={\bm{P}}^{C_{s}X_{h}}({{\bm{P}}^{X_{h}X_{h}}})^{R}{\bm{P}}^{X_{h}C_{s}}\enspace$
(10)

In practice, we set the diagonal values of ${\bm{P}}^{X_{h}X_{h}}$ to small
values and hope to reduce the non-diagonal values. This equals to encourage
the probability $P^{{\bm{C}}_{s}{\bm{X}}_{h}{\bm{C}}_{s}}(R)$ to be uniformly
distributed over all the seen classes. We further encourage the probability
${\bm{P}}^{C_{s}X_{h}}\in\mathbb{R}^{N_{s}^{1:t}\times N_{h}}$ to be uniformly
distributed over all the generated examples to encourage as many generations
to be visited in the random walk, and hence encourage the diversity. Hence,
our Generative Random Walk (GRW) loss is defined by

$\small{L}_{\text{GRW}}=\sum_{r=0}^{R}{\gamma^{r}L_{e}(P^{{\bm{C}}_{s}{\bm{X}}_{h}{\bm{C}}_{s}}(r),{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\mathcal{U}}})}+L_{e}({\bm{P}}_{v}(C_{s},X_{h}),{\mathcal{U}}_{v})\enspace,$
(11)

where $L_{e}(\cdot,\cdot)$ is the cross-entropy loss,
${\mathcal{U}}\in\mathbb{R}^{N_{s}^{1:t}\times N_{s}^{1:t}}$ is uniform
distribution, $R$ is the transition steps, and $\gamma$ is exponential decay.
We compute the probability that each generated point be visited by any seen
class as
$P_{v}(C_{s},X_{h})=\frac{1}{\tilde{N_{s}^{1:t}}}\sum_{i=0}^{N_{s}^{1:t}}{\bm{P}}^{C_{s}X_{h}}_{i}$,
where ${\bm{P}}^{C_{s}X_{h}}_{i}$ represents the $i^{th}$ row of the
${\bm{P}}^{C_{s}X_{h}}$ matrix. The visit loss is then defined as the cross-
entropy between $P_{v}$ and the uniform distribution
${\mathcal{U}}_{v}\in\mathbb{R}^{N_{h}}$, encouraging all the generated
examples to be visited. In addition, we empirically found that the GRW loss
can also work as a regularizer to encourage the consistency of generated seen
visual space as well, which we defined as

$\footnotesize{R}_{\text{GRW}}=\sum_{r=0}^{R}{\gamma^{r}L_{e}(P^{{\bm{C}}_{s}{\bm{X}}_{sg}{\bm{C}}_{s}}(r),{\color[rgb]{0,0.7,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.7,0}{\mathcal{I}}})}+L_{e}({\bm{P}}_{v}(C_{s},X_{sg}),{\mathcal{U}}_{v}),$
(12)

where ${\mathcal{I}}$ is identity distribution, and ${\bm{D}}_{sg}$ represents
the matrix for generated seen samples.

We numerically show that the random walk-based penalty can reduce
$\bar{d}_{GDB}$ (Def 4.1) by the relationship between $\bar{d}_{GDB}$ and
${L}_{GRW}$. Details are shown in Appendix B.3.

We also adapt the loss proposed in [16] to directly prevent the generated
unseen samples from being classified into seen classes, i.e.,

$\footnotesize
L_{\text{creativity}}=\mathbb{E}_{{\bm{z}}\sim\mathcal{Z},{\bm{a}}_{h}\sim{\bm{D}}_{h}}D_{\mathrm{KL}}\big{(}\big{\langle}G({\bm{z}},{\bm{a}}_{h}),D({\bm{A}}_{s}^{1:t})\big{\rangle}\|{\mathcal{U}}\big{)},$
(13)

where $D_{\mathrm{KL}}(\cdot\|\cdot)$ is the KL divergence,
${\bm{A}}_{s}^{1:t}\in\mathbb{R}^{N_{s}^{1:t}\times d_{a}}$ is the matrix of
seen classes attributes vectors until task $t$, ${\bm{a}}_{h}$ is hallucinated
attributes according to Section 5.2.1,
$\big{\langle}G({\bm{z}},{\bm{a}}_{ug}),D({\bm{A}}_{s}^{1:t})\big{\rangle}\in\mathbb{R}^{N_{s}^{1:t}}$
are the logits over seen classes so far for a given
$G({\bm{z}},{\bm{a}}_{h})$, ${\mathcal{U}}$ is the uniform distribution.

Inductive loss Combining Equation 11, 12 and 13 our final inductive loss is

$\small\mathcal{L}_{\text{inductive}}=\lambda_{\text{c}}L_{\text{creativity}}+\lambda_{\text{i}}{L}_{\text{GRW}}+\lambda_{\text{i}}{R}_{\text{GRW}}$
(14)

where $\lambda_{i}$ is the scaling weight for both the GRW loss term and
regularization term.

##  6 Continual Zero-Shot Learning Experiment

###  6.1 Experiment Setup

Data Stream and Benchmarks: We adopt the continual zero-shot learning
framework proposed in [53]. In this setting, a $T$-split dataset $D^{1:T}$
forms $T-1$ tasks. At time step $t$, the split $D^{1:t}$ is defined as a seen
set of tasks, and the split $D^{t+1:T}$ is an unseen set of tasks. We conduct
experiments on four widely used CGZSL benchmarks for a fair comparison: AWA1
[37], AWA2 [67], Caltech UCSD Birds 200-2011 (CUB)[59], and SUN[43]. We follow
[53, 36] for the class split in the continual zero-shot learning setting. More
details can be found in Appendix D.

Baselines, backbone, and training: We use the method proposed in [36] as the
main baseline and compare it with recent CGZSL methods in the setting we
mentioned above, including the transductive method Tf-GZSL [22], DVGR [25],
A-CGZSL [24], BD-CGZSL [36], and the inductive method CN-CZSL [53], CARNet
[23]. ‘BD-CGZSL-in’ denotes our modified inductive version of [36] by naively
removing unseen information. Following [36], we also compare our baseline with
the classical continual learning methods EWC [34] and A-GEM [11]. We use
vanilla GAN’s Generator and Discriminator, both of which are two-layer linear
networks. Image features are extracted by ResNet-101, pre-trained on ImageNet
1k. The are attributes from [62] and extracted features are used as our model
input. We use a replay buffer with a fixed size of 5k.

We run all experiments for 50 epochs and 64 batch sizes with the Adam
optimizer. We use a learning rate of 0.005 and a weight decay of 0.00001.
Results reported in Table 1 are based on one NVIDIA Tesla P100 GPU. We select
our random walk steps $R$, weight decay $\gamma$ and coefficient of inductive
loss terms $\lambda_{i}$ according to prior exploratory zero-shot learning
experiments shown in Appendix C.

Metrics: We use the mean seen accuracy, mean unseen accuracy and mean harmonic
seen/unseen accuracy [53] to measure the zero-shot learning ability. These
metrics are defined as follows,

$\footnotesize\begin{split}&\text{mSA}=\frac{1}{T}\sum_{t=1}^{T}S_{t}(D^{1:t}),\text{mUA}=\frac{1}{T-1}\sum_{t=1}^{T-1}U_{t}(D^{t+1:T})\\\
&\text{mHA}=\frac{1}{T-1}\sum_{t=1}^{T-1}H(S_{t}(D^{1:t}),U_{t}(D^{t+1:T})),\end{split}$
(15)

where $H(\cdot,\cdot)$ is the harmonic mean and $S_{t},U_{t}$ are seen and
unseen per-class accuracy using the model trained after time $t$. We also use
the backward transfer [11, 66, 53] to measure the continual learning ability,
which is defined in [53]

$\footnotesize\text{BWT}=\frac{1}{T-1}\sum_{t=1}^{T-1}(S_{T}(D^{1:t})-S_{t}(D^{1:t}))\enspace.$
(16)

Note that this should only be conducted on seen set, since part of the early
unseen set become seen set later. The BWT on unseen set cannot reflect the
knowledge retain ability of the model.

###  6.2 Results

The mean harmonic accuracy of the four benchmarks is shown in Table 1, and the
task-wise mHA of the CUB dataset is shown in Figure 3. In coarse-grained
datasets AWA1 and AWA2, our proposed learner achieves $43.4\%$ and $44.6\%$ in
mHA, respectively, surpassing all the current inductive and transductive
methods. In the fine-grained datasets and tasks with long steps (CUB, SUN),
our method achieves $28.8\%$ and $27.1\%$, surpassing all the current CZSL
methods. We observe that even though other methods have comparable mSA, they
have far lower mUA than ours. We believe that our method achieves this
improved knowledge transfer ability from seen visual space to unseen visual
space through the proposed inductive learning signals, _i.e_.,
$\mathcal{L}_{\text{inductive}}$.

Dataset |  | AWA1 | AWA2 | CUB | SUN
---|---|---|---|---|---
DVGR [25] | tr | 0.09 | 0.10 | -0.07 | -0.20
A-CGZSL [24] | tr | 0.11 | 0.05 | 0.10 | 0.005
BD-CGZSL [36] | tr | 0.18 | 0.14 | 0.13 | -0.02
CN-ZSL [53] | in | - | - | -0.04 | -0.02
BD-CGZSL-in [36] | in | 0.18 | 0.15 | 0.14 | -0.03
ours \+ interpolation | in | 0.12 | 0.10 | 0.19 | 0.01
ours \+ dictionary | in | 0.11 | 0.11 | 0.19 | 0.01
Table 2: Backward transfer of different CGZSL methods, where higher results
indicate less forgetting.

Table 2 displays the backward transfer of different continual zero-shot
algorithms, where higher results indicate better knowledge retention. Our
model exhibits a strong backward transfer capability, particularly on longer
task sequences where it is needed the most. We achieved the highest BWT score
of 0.19 on CUB. On SUN, negative BWT scores (i.e., forgetting) are observed in
most other models, but our method can still retain knowledge from the past.
These results suggest that the analysis tools we created allow us to identify
the critical factors for zero-shot learning, and the development of tools for
continual learning can improve our ability to retain information.

Figure 3: Mean Harmonic accuracy up to each task on SUN dataset. Our method
outperforms both transductive and inductive methods

###  6.3 Ablation Study

| interpolation | dictionary
---|---|---
with ${R}_{\text{GRW}}$ \+ $L_{\text{GRW}}$ | 28.4 | 28.8
\- $L_{\text{creativity}}$ | 27.72 | 27.66
w/o ${R}_{\text{GRW}}$, $L_{\text{GRW}}$ | 19.07 | 20.75
\- $L_{\text{creativity}}$ | 14.43 | 14.43
with $L_{\text{GRW}}$ | 26.73 | 27.39
Table 3: Effect of the random walk-based penalty with mH measure on CUB
dataset.

To assess the impact of our novel random walk-based penalties, $L_{GRW}$ and
$\mathcal{R}_{GRW}$, we conducted ablation experiments; see Table 3. he
results in Table 3 indicate that the improvements are mainly attributed to
$L_{GRW}$, while $\mathcal{R}_{GRW}$ contributes an additional $1\%$.
$L_{\text{creativity}}$ is also part of the inductive loss. Additionally,
removing $L\text{creativity}$ while using our GRW losses has little effect on
the performance, as shown in Table 3. More details can be found in Appendix D.

###  6.4 More CZSL settings

Our focus lies on assessing performance under varying seen/unseen class ratios
during knowledge accumulation, which is proposed in [53] and referred to as
static setting in [36]. There are other continual zero-shot learning settings
proposed in [36], such as dynamic and online settings. In the dynamic setting,
the seen and unseen classes dynamically increase, while in the online setting,
certain unseen classes are continually converted to seen classes. We find our
explored static setting a more informing benchmark for the inductive CZSL
skill, as the evaluation after every task is always performed on all classes
in the dataset and hence is more challenging. Despite this, we still provide a
comparison between our method and the baseline methods on the dynamic and
online settings; see Table 4. The results show that our method is superior to
the baseline in the dynamic and online settings in almost all datasets, and
gains the most improvements in the most challenging static setting.

| setting | AWA1 | AWA2 | CUB | SUN
---|---|---|---|---|---
BD-CGZSL | D | 56.9/49.1 | 56.4 | 16.8 | 28.0
ours + inter. | D | 60.0 | 58.8 | 32.8 | 41.6
ours + dic. | D | 59.7 | 55.5 | 31.8 | 40.2
BD-CGZSL | O | 56.9/49.1 | 53.4 | 28.4 | 33.7
ours + inter. | O | 49.6 | 48.5 | 32.3 | 39.6
ours + dic. | O | 46.1 | 47.3 | 31.2 | 39.2
Table 4: mH in dynamic setting (D) and online setting(O)

###  6.5 Replay Buffer Analysis

Some existing methods [24, 25, 36] tend to use the generative replay method
proposed by [21], where the correctly predicted seen generated features from
the previous task are stored in buffers. However, the buffer size increases
significantly over tasks since a fixed number of samples for each class is
stored, and if the model struggles to make accurate predictions for certain
classes, samples from these classes are absent in the buffer.

We empirically found that the class-balanced experience replay method proposed
by [46] can be extremely helpful. At every task, we save the class attribute
in ${\bm{A}}^{1:t}$, the class center matrix ${\bm{C}}$, and modify the buffer
with current features noted as ${\bm{D}}_{s}^{1:t}$, such that the buffer is
balanced across all the seen classes.

|  Buffer Size |  | Ours |  |  BD-CGZSL (tr)
---|---|---|---|---|---
|  | BWT | mHA |  | mHA
generative | 28.5k |  | 0.14 | 21.06 |  | 17.76
real | 10k |  | 0.17 | 28.44 |  | 27.79
real | 5k |  | 0.19 | 28.8 |  | 26.55
real | 2.5k |  | 0.08 | 26.99 |  | 26.77
Table 5: Comparison of generative replay and real replay methods on CUB [59].
Dictionary-based attribute generation is used

In this comparison on the CUB dataset, we observe in Table 5 that the method
using real replay can achieve better harmonic accuracy with a smaller buffer
size (around 1/10 of the generative replay buffer size) and comparable
backward transfer with a slightly larger buffer size (around 1/5 of the
generative replay buffer size). Moreover, the real replay-based method is not
as sensitive to the buffer size as the generative replay-based methods. It is
worth noting that DVGR, A-CGZSL, and BD-CGZSL typically use generative replay,
while only CN-CGZSL uses real replay. In addition, the last column in Table 5
shows that our proposed real replay method can also improve the harmonic
accuracy of other methods.

Figure 4: Comparison of replayed number of features per class in different
replay method at task 2 in SUN dataset

To understand the real replay and generative replay, we extend our analysis by
visualizing the distribution of buffer features across various classes in task
2 of the SUN dataset, as illustrated in Figure 4. Real replay approach
exhibits a balanced allocation of features across all classes. Conversely, the
generative replay technique displays an intriguing pattern, wherein certain
classes lack a substantial number of stored features, while others exhibit a
twofold increase. Notably, the classes with fewer stored features coincide
with instances where the model’s performance is suboptimal. This discrepancy
can be attributed to the generative replay method’s propensity to store
exclusively the accurately classified generated data. Consequently, this
uneven distribution of stored features could potentially lead to a compromised
performance in these classes during subsequent tasks.

##  7 Conclusion and Discussion

In this paper, we focus on inductive continual zero-shot learning (CZSL) to
eliminate the need of unseen information for more realistic learning systems.
To this end, we developed a framework for the theoretical analysis of
generative zero-shot learning, introducing a distance metric to measure the
ability of generated samples to represent the unseen space when the unseen
information is inaccessible during training. We also proposed a continual
zero-shot algorithm, ICGZSL, which can reduce the distance without using
unseen information during training. We conducted experiments on four popular
continual zero-shot learning benchmarks: AWA1, AWA2, CUB, and SUN. Our
approach achieved around $3\%$ higher harmonic accuracy in the small dataset
and around $7\%$ in the larger dataset compared to previous inductive and
transductive methods. These results demonstrate that unseen semantic
information is not essential when a well-analyzed seen distribution and method
are used.

However, it is important to acknowledge certain limitations in our work. While
the developed theoretical bounds and distance measures hold promise for
methodical numeric analysis, a more stringent alignment between empirical and
anticipated distance measures could substantially enhance algorithmic design.
Moreover, the consideration of multi-class classification conditions warrants
attention. Additionally, the use of a frozen backbone for image feature
extraction, though effective, encourages further exploration into continual
learning methods that facilitate viable zero-shot learning capabilities while
enabling the backbone to progressively accumulate knowledge.

## References

  * [1] Yaser S Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin.  Learning from data, volume 4.  AMLBook New York, 2012.
  * [2] Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid.  Label-embedding for image classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(7):1425–1438, 2016.
  * [3] Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele.  Evaluation of output embeddings for fine-grained image classification.  In IEEE Conference on Computer Vision and Pattern Recognition, 2015\.
  * [4] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory Aware Synapses: Learning what (not) to forget.  In European Conference on Computer Vision, 2018.
  * [5] Ahmed Ayyad, Yuchen Li, Nassir Navab, Shadi Albarqouni, and Mohamed Elhoseiny.  Semi-supervised few-shot learning with prototypical random walks.  In AAAI Workshop on Meta-Learning and MetaDL Challenge, pages 45–57, 2021.
  * [6] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan.  A theory of learning from different domains.  Machine Learning, 79(1-2):151–175, 2010.
  * [7] Shivam Chandhok, Sanath Narayan, Hisham Cholakkal, Rao Muhammad Anwer, Vineeth N Balasubramanian, Fahad Shahbaz Khan, and Ling Shao.  Structured latent embeddings for recognizing unseen classes in unseen domains.  In British Machine Vision Conference, 2021.
  * [8] Fu-Chieh Chang, Hao-Jen Wang, Chun-Nan Chou, and Edward Y. Chang.  G2R Bound: A Generalization Bound for Supervised Learning from GAN-Synthetic Data.  arXiv:1905.12313, 2019.
  * [9] Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha.  Synthesized classifiers for zero-shot learning.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 5327–5336, 2016.
  * [10] Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha.  An empirical study and analysis of generalized zero-shot learning for object recognition in the wild.  In European Conference on Computer Vision, pages 52–68, 2016.
  * [11] Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny.  Efficient Lifelong Learning with A-GEM.  In International Conference on Learning Representations, 2019.
  * [12] Arthur Douillard, Eduardo Valle, Charles Ollion, Thomas Robert, and Matthieu Cord.  Insights from the future for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3482–3491, 2021.
  * [13] Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach.  Adversarial continual learning.  In European Conference on Computer Vision, pages 386–402, 2020\.
  * [14] Mohamed Elfeki, Camille Couprie, Morgane Riviere, and Mohamed Elhoseiny.  Gdpp: Learning diverse generations using determinantal point processes.  In International conference on machine learning, pages 1774–1783, 2019.
  * [15] Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone.  Can: Creative adversarial networks, generating "art" by learning about styles and deviating from style norms.  In International Conference on Computational Creativity, 2017.
  * [16] Mohamed Elhoseiny and Mohamed Elfeki.  Creativity inspired zero-shot learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5784–5793, 2019.
  * [17] Mohamed Elhoseiny, Kai Yi, and Mohamed Elfeki.  Cizsl++: Creativity inspired generative zero-shot learning.  arXiv preprint arXiv:2101.00173, 2021.
  * [18] Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal.  Link the head to the "beak": Zero shot learning from noisy text description at part precision.  In IEEE Conference on Computer Vision and Pattern Recognition, 2017\.
  * [19] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth.  Describing objects by their attributes.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 1778–1785, 2009.
  * [20] Rafael Felix, Vijay BG Kumar, Ian Reid, and Gustavo Carneiro.  Multi-modal cycle-consistent generalized zero-shot learning.  In European Conference on Computer Vision, pages 21–37, 2018.
  * [21] Chandan Gautam, Sethupathy Parameswaran, Ashish Mishra, and Suresh Sundaram.  Generative Replay-based Continual Zero-Shot Learning.  arXiv:2101.08894, 2021.
  * [22] Chandan Gautam, Sethupathy Parameswaran, Ashish Mishra, and Suresh Sundaram.  Tf-gczsl: Task-free generalized continual zero-shot learning.  Neural Networks, 155:487–497, 2022.
  * [23] Chandan Gautam, Sethupathy Parameswaran, Vinay Verma, Suresh Sundaram, and Savitha Ramasamy.  Refinement matters: Textual description needs to be refined for zero-shot learning.  In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6127–6140, 2022.
  * [24] Subhankar Ghosh.  Adversarial training of variational auto-encoders for continual zero-shot learning.  In International Joint Conference on Neural Networks, 2021.
  * [25] Subhankar Ghosh.  Dynamic VAEs with Generative Replay for Continual Zero-shot Learning.  In CVPR workshop, 2021.
  * [26] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, volume 27, 2014.
  * [27] Aaron Hertzmann.  Can computers create art?  In Arts, volume 7, page 18, 2018.
  * [28] Aaron Hertzmann.  Visual indeterminacy in gan art.  Leonardo, 53(4):424–428, 2020.
  * [29] Philip Häusser, Alexander Mordvintsev, and Daniel Cremers.  Learning by association — a versatile semi-supervised training method for neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 626–635, 2017.
  * [30] Divyansh Jha, Hanna H. Chang, and Mohamed Elhoseiny.  Wölfflin’s affective generative analysis of visual art.  The International Conference on Computational Creativity, 2021.
  * [31] Divyansh Jha, Kai Yi, Ivan Skorokhodov, and Mohamed Elhoseiny.  Creative walk adversarial networks: Novel art generation with probabilistic random walk deviation from style norms.  In ICCC, pages 195–204, 2022.
  * [32] Byungkon Kang.  Fast determinantal point process sampling with application to clustering.  Advances in Neural Information Processing Systems, 26, 2013.
  * [33] Michael J Kearns and Umesh Vazirani.  An introduction to computational learning theory.  MIT press, 1994.
  * [34] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.
  * [35] Elyor Kodirov, Tao Xiang, and Shaogang Gong.  Semantic autoencoder for zero-shot learning.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 3174–3183, 2017.
  * [36] Hari Chandana Kuchibhotla, Sumitra S Malagi, Shivam Chandhok, and Vineeth N Balasubramanian.  Unseen classes at a later time? no problem.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [37] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.  Learning to detect unseen object classes by between-class attribute transfer.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 951–958, 2009.
  * [38] Jingjing Li, Mengmeng Jing, Ke Lu, Zhengming Ding, Lei Zhu, and Zi Huang.  Leveraging the Invariant Side of Generative Zero-Shot Learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7394–7403, 2019.
  * [39] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935–2947, 2017.
  * [40] Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan.  The clear benchmark: Continual learning on real-world imagery.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [41] Yang Liu, Jishun Guo, Deng Cai, and Xiaofei He.  Attribute attention for semantic disambiguation in zero-shot learning.  In IEEE/CVF International Conference on Computer Vision, 2019.
  * [42] Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao.  Latent embedding feedback and discriminative features for zero-shot classification.  In European Conference on Computer Vision, pages 479–495, 2020\.
  * [43] Genevieve Patterson and James Hays.  SUN attribute database: Discovering, annotating, and recognizing scene attributes.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2751–2758, 2012.
  * [44] Akanksha Paul, Narayanan C Krishnan, and Prateek Munjal.  Semantically aligned bias reducing zero shot learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7056–7065, 2019.
  * [45] Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Lim, and Xi-Zhao Wang.  A review of generalized zero-shot learning methods.  IEEE Transactions on Pattern Analysis and Machine Intelligence, PP, 07 2022.
  * [46] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In European conference on computer vision, pages 524–540, 2020\.
  * [47] Ruizhi Qiao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel.  Less is more: Zero-shot learning from online textual documents with noise suppression.  In IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [48] Shafin Rahman, Salman Khan, and Nick Barnes.  Transductive learning for zero-shot object detection.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6082–6091, 2019.
  * [49] Jathushan Rajasegaran, Munawar Hayat, Salman Khan, Fahad Shahbaz Khan, and Ling Shao.  Random path selection for incremental learning.  Advances in Neural Information Processing Systems, 2019.
  * [50] Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  Information processing & management, 24(5):513–523, 1988.
  * [51] Othman Sbai, Mohamed Elhoseiny, Antoine Bordes, Yann LeCun, and Camille Couprie.  Design: Design inspiration from generative networks.  In ECCV workshop, 2018.
  * [52] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in Neural Information Processing Systems, 30, 2017.
  * [53] Ivan Skorokhodov and Mohamed Elhoseiny.  Class normalization for (continual)? generalized zero-shot learning.  In International Conference on Learning Representations, 2021.
  * [54] Hongzu Su, Jingjing Li, Zhi Chen, Lei Zhu, and Ke Lu.  Distinguishing unseen from seen for generalized zero-shot learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7885–7894, June 2022.
  * [55] Lee Sweetlove.  Number of species on earth tagged at 8.7 million.  Nature, 23, 2011.
  * [56] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
  * [57] Twan van Laarhoven and Elena Marchiori.  Unsupervised domain adaptation with random walks on target labelings.  arXiv:1706.05335, 2017.
  * [58] Maunil Vyas, Hemanth Venkateswara, and Sethuraman Panchanathan.  Leveraging seen and unseen semantic relationships for generative zero-shot learning.  In European Conference on Computer Vision, pages 70–86, 2020.
  * [59] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [60] Wenlin Wang, Hongteng Xu, Guoyin Wang, Wenqi Wang, and Lawrence Carin.  Zero-Shot Recognition via Optimal Transport.  In IEEE Winter Conference on Applications of Computer Vision, pages 3470–3480, 2021.
  * [61] Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele.  Latent embeddings for zero-shot classification.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 69–77, 2016.
  * [62] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018\.
  * [63] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata.  Feature Generating Networks for Zero-Shot Learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5542–5551, 2018.
  * [64] Yongqin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata.  F-vaegan-d2: A feature generating framework for any-shot learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [65] Ye Xiang, Ying Fu, Pan Ji, and Hua Huang.  Incremental learning using conditional adversarial networks.  In IEEE/CVF International Conference on Computer Vision, pages 6619–6628, 2019.
  * [66] Kai Yi and Mohamed Elhoseiny.  Domain-Aware Continual Zero-Shot Learning.  arXiv:2112.12989, 2021.
  * [67] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer.  Semantic Drift Compensation for Class-Incremental Learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6980–6989, 2020.
  * [68] Li Zhang, Tao Xiang, and Shaogang Gong.  Learning a deep embedding model for zero-shot learning.  In IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [69] Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal.  A generative adversarial approach for zero-shot learning from noisy texts.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.

##  Appendix A Derivation of Theorems in Section 4.1

###  A.1 Theorem 4.2

Notation Statement in the Appendix. To involve risk between hypotheses and
between hypothesis and ground truth models, we use $\epsilon_{\cdot}(h,f)$ or
$\epsilon_{\cdot}(h,h^{*})$ to specify which space the risk is computed on.

######  Definition A.1 ($\mathcal{H}\Delta\mathcal{H}$-distance [6]).

Given two feature distributions $\mathcal{D}_{g}$ and $\mathcal{D}_{r}$, and
the hypothesis class $\mathcal{H}$, the
$\mathcal{H}\Delta\mathcal{H}$-distance between $\mathcal{D}_{g}$ and
$\mathcal{D}_{r}$ is defined as

$\begin{split}d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{g},\mathcal{D}_{r})=2\sup_{h,h^{\prime}\in\mathcal{H}}|&\mathbb{P}_{{\bm{x}}\sim\mathcal{D}_{g}}[h({\bm{x}})\neq
h^{\prime}({\bm{x}})]-\mathbb{P}_{{\bm{x}}\sim\mathcal{D}_{r}}h({\bm{x}})\neq
h^{\prime}({\bm{x}})|\enspace.\end{split}$ (17)

We first define the expected version of Definition 4.1

######  Definition A.2 (Expected Generative distance).

Given two feature distributions $\mathcal{D}_{h}$ and $\mathcal{D}_{u}$, the
ground truth labelling function $f_{h},f_{u}$, and the optimal hypothesis
$h^{*}=\operatorname*{arg\,min}_{h\in H}\epsilon(h,f_{h})+\epsilon(h,f_{s})$
of a model training on the distribution ${\mathcal{D}}_{s},{\mathcal{D}}_{h}$.
The $h^{*}\Delta f$-distance between $\mathcal{D}_{h}$ and $\mathcal{D}_{u}$
is defined as

$\begin{split}d_{h^{*}}(\mathcal{D}_{h},\mathcal{D}_{u})=&|\mathbb{P}_{({\bm{x}},{\bm{a}})\sim\mathcal{D}_{h}}[f_{h}({\bm{x}})\neq
h^{*}({\bm{x}},{\bm{a}})]\\\
&-\mathbb{P}_{({\bm{x}},{\bm{a}})\sim\mathcal{D}_{u}}[f_{u}({\bm{x}})\neq
h^{*}({\bm{x}},{\bm{a}})]|\enspace,\end{split}$ (18)

Note that the following inequality related to
$d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{g},\mathcal{D}_{r})$ holds for
any $h$ and $h^{*}$

$\begin{split}d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{g},\mathcal{D}_{r})&=2\sup_{h,h^{\prime}\in\mathcal{H}}|\mathbb{P}_{{\bm{x}}\sim\mathcal{D}_{g}}[h({\bm{x}})\neq
h^{\prime}({\bm{x}})]-\mathbb{P}_{{\bm{x}}\sim\mathcal{D}_{r}}h({\bm{x}})\neq
h^{\prime}({\bm{x}})|\\\ &\geq
2|\mathbb{E}_{{\bm{x}}\sim\mathcal{D}_{g}}[\mathbbm{1}_{h({\bm{x}})\neq
h^{*}({\bm{x}})}]-\mathbb{E}_{{\bm{x}}\sim\mathcal{D}_{r}}[\mathbbm{1}_{h({\bm{x}})\neq
h^{*}({\bm{x}})}]|\\\
&=2|\epsilon_{g}(h,h^{*})-\epsilon_{r}(h,h^{*})|\enspace.\end{split}$ (19)

######  Lemma A.3 ([1]).

For a fixed hypothesis, the actual risk can be estimated from the empirical
error with probability $1-\delta$

$\epsilon(h,f)\leq\hat{\epsilon}(h,f)+\sqrt{\frac{1}{2m}\log\frac{2}{\delta}}\enspace,$
(20)

where $\epsilon(h,f)$ is the actual risk, $\hat{\epsilon}(h,f)$ is the
empirical risk, and $m$ is the number of testing samples.

######  Proposition A.4 (Bound $d_{h^{*}}(\mathcal{D}_{u},\mathcal{D}_{h})$ by
$\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})$ ).

The distribution distance $d_{h^{*}}(\mathcal{D}_{u},\mathcal{D}_{h})$ can be
bounded by it empirical counterpart by

$d_{h^{*}}(\mathcal{D}_{u},\mathcal{D}_{h})\leq\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})+C(\frac{1}{m},\frac{1}{\delta})\enspace,$
(21)

where $C(\frac{1}{m},\frac{1}{\delta})$ is a constant term depending on the
training sample size $m$ and confidence $1-\delta$. Here $\mathcal{D}_{\cdot}$
represent the distribution, and ${\bm{D}}_{\cdot}$ represents the dataset
sampled from the corresponding distribution.

###### Proof.

Similar to Equation 19, we can write our generative distance as

$d_{h^{*}}(\mathcal{D}_{u},\mathcal{D}_{h})=2|\epsilon_{h}(h^{*},f)-\epsilon_{u}(h^{*},f)|\enspace.$
(22)

Combining Lemma A.3, we have

$\begin{split}\frac{1}{2}d_{h^{*}}(\mathcal{D}_{u},\mathcal{D}_{h})&=|\epsilon_{h}(h^{*},f)-\epsilon_{u}(h^{*},f)|\\\
&\leq|\hat{\epsilon}_{h}({h}^{*},f)-\hat{\epsilon}_{u}({h}^{*},f)|+|(\hat{\epsilon}_{h}({h}^{*},f)+\hat{\epsilon}_{u}(h^{*},f))-(\epsilon_{h}(h^{*},f)+\epsilon_{u}(h^{*},f))|\\\
&\lesssim\frac{1}{2}\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})+C(\frac{1}{m},\frac{1}{\delta})\enspace,\end{split}$
(23)

where $h^{*}=\operatorname*{arg\,min}_{h^{\prime}\in
H}\epsilon_{s}(h^{\prime},f_{s})+\epsilon_{h}(h^{\prime},f_{h})$, and
$\hat{h}^{*}=\operatorname*{arg\,min}_{h^{\prime}\in
H}\hat{\epsilon}_{s}(h^{\prime},f_{s})+\hat{\epsilon}_{h}(h^{\prime},f_{h})$.
Following the discussion of [8], we assume the optimal hypothesis
$\hat{h}^{*}$ we can achieve is very close to the global minimum when the
training sample is large, then we can estimate $h^{*}$ in Equation 23 by
$\hat{h}^{*}$. $C(\frac{1}{m},\frac{1}{\delta})$ is obtained from Lemma A.3 ∎

##### Proof of theorem 4.2

Given the CZSL procedure described in section 3.1, with confidence $1-\delta$
the risk on the unseen distribution is bounded by

$\footnotesize\begin{split}\epsilon(h,f_{u}^{t})\leq\hat{\epsilon}(\hat{h}^{*},f_{s}^{1:t})+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{s}^{1:t},\mathcal{D}^{t}_{u})+\bar{\lambda}+\frac{1}{2}\bar{d}_{GDB}({\bm{D}}^{t}_{u},{\bm{D}}^{t}_{h})\end{split}$
(24)

where $\hat{h}^{*}=\operatorname*{arg\,min}_{h\in
H}\sum_{i=1}^{t}\hat{\epsilon}(h,f_{s}^{i})+\hat{\epsilon}(h,f^{t}_{h})$,
$\bar{\lambda}=\hat{\epsilon}(\hat{h}^{*},f_{s}^{1:t})+\hat{\epsilon}(\hat{h}^{*},f^{t}_{h})$.

###### Proof.

Let $h^{*}=\operatorname*{arg\,min}_{h\in
H}\sum_{i=1}^{t}{\epsilon}(h,f_{s}^{i})+{\epsilon}(h,f^{t}_{h})$, and
$\lambda={\epsilon}({h}^{*},f_{s}^{1:t})+{\epsilon}({h}^{*},f^{t}_{h})$. We
write $\epsilon_{s}(\cdot,\cdot)$ as the union seen distribution from time
$1:t$. Then

$\begin{split}&\epsilon_{u}(h,f_{u})\\\
&=\epsilon_{s}(h,f_{s})+\epsilon_{u}(h,h^{*})-\epsilon_{s}(h,h^{*})+\epsilon_{h}(h^{*},f_{h})+\epsilon_{s}(h^{*},f_{s})-\epsilon_{h}(h^{*},f)+\epsilon_{u}(h^{*},f)\\\
&-\epsilon_{s}(h,f_{s})-\epsilon_{u}(h,h^{*})+\epsilon_{s}(h,h^{*})-\epsilon_{s}(h^{*},f_{s})-\epsilon_{u}(h^{*},f)+\epsilon_{u}(h,f_{u})\\\
&\leq\epsilon_{s}(h,f_{s})+|\epsilon_{u}(h,h^{*})-\epsilon_{s}(h,h^{*})|+|\epsilon_{h}(h^{*},f_{h})+\epsilon_{s}(h^{*},f_{s})|+|\epsilon_{h}(h^{*},f)-\epsilon_{u}(h^{*},f)|\\\
&-\epsilon_{s}(h,f_{s})+\epsilon_{s}(h,h^{*})-\epsilon_{s}(h^{*},f_{s})-\epsilon_{u}(h,h^{*})+\epsilon_{u}(h,f_{u})-\epsilon_{u}(h^{*},f)\\\
&\leq\epsilon_{s}(h,f_{s})+d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{s}^{1:t},\mathcal{D}^{t}_{u})+\lambda+d_{h^{*}}(\mathcal{D}_{h},\mathcal{D}_{u})-\epsilon_{s}(h,f_{s})+\epsilon_{s}(h,h^{*})\\\
&-\epsilon_{s}(h^{*},f_{s})-\epsilon_{u}(h,h^{*})+\epsilon_{u}(h,f_{u})-\epsilon_{u}(h^{*},f)\enspace.\end{split}$
(25)

Note that for any distribution

$\begin{split}|\epsilon_{\mathcal{D}}(h,f_{\mathcal{D}})-\epsilon_{\mathcal{D}}(h,h^{*})|&=|\mathbb{E}_{{\bm{x}}\sim\mathcal{D}}[\mathbbm{1}_{h\neq
f_{\mathcal{D}}}]-\mathbb{E}_{{\bm{x}}\sim\mathcal{D}}[\mathbbm{1}_{h\neq
h^{*}}]|\\\ &=|\mathbb{E}_{{\bm{x}}\sim\mathcal{D}}[\mathbbm{1}_{h\neq
f_{\mathcal{D}}}-\mathbbm{1}_{h\neq h^{*}}]|\\\
&\leq\mathbb{E}_{{\bm{x}}\sim\mathcal{D}}[\mathbbm{1}_{h^{*}\neq
f_{\mathcal{D}}}]=\epsilon_{\mathcal{D}}(h^{*},f_{\mathcal{D}})\enspace,\end{split}$
(26)

where the inequality holds by the triangle inequality of the characteristic
function, _i.e_., $\mathbbm{1}[a\neq b]\geq\mathbbm{1}[a\neq
c]-\mathbbm{1}[b\neq c]$ for $\forall a,b,c\in\mathbb{R}$. Equation (26) shows
that the fourth line in Equation (25) is less than or equal to zero.

Combining Equation 26, the Equation 25 can be written as

$\begin{split}\epsilon_{u}(h,f_{u})\leq&\epsilon_{s}(h,f_{s})+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{s}^{1:t},\mathcal{D}^{t}_{u})+\lambda+d_{h^{*}}(\mathcal{D}_{h},\mathcal{D}_{u})\enspace,\end{split}$
(27)

However, Equation (27) involves unknown risk and unsolvable distribution. We
combine the expected risk and the actual observed risk by Lemma A.3. Let
$h^{*}=\operatorname*{arg\,min}_{h\in
H}\sum_{i=1}^{t}\hat{\epsilon}(h,f_{s}^{i})+\hat{\epsilon}(h,f^{t}_{h})$ be
the optimal hypothesis on the training set, and
$\bar{\lambda}=\hat{\epsilon}(\hat{h}^{*},f_{s}^{1:t})+\hat{\epsilon}(\hat{h}^{*},f^{t}_{h})$,
we have $\lambda\leq\bar{\lambda}$. Together with Lemma A.3 and Proposition
A.4, we have

$\begin{split}\epsilon_{u}\leq&\sum_{i=1}^{t}\alpha^{i}(\hat{\epsilon}(h,f_{s^{i}})+\frac{1}{2}d_{\mathcal{H}\Delta\mathcal{H}}(\mathcal{D}_{s^{i}},\mathcal{D}_{u}))+\bar{\lambda}+\frac{1}{2}\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})+C(\frac{1}{m}+\frac{1}{\delta})\enspace,\end{split}$
(28)

∎

###  A.2 Explanation of Statement 4.3

Let ${\bm{D}}_{h}\sim\mathcal{D}_{h}$ be the generated unseen set we are
training on, where $\mathcal{D}_{h}$ is the empirical distribution of all
possible generations. In unsupervised domain adaptation, [57] uses random walk
to select label set for the samples who have small generalization error.
Proposition 3.2 of [57] demonstrates that the self transition probability of a
Markov chain represents an upper bound on the margin linear classifier’s
generalization error. This concept is adapted to connect our GDB bound
connected to the Markov Chain in below. In our sample generation procedure, we
generate only one sample from each class. Our discussion of this section will
be based on this. We have
$\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})\propto-\sum_{i\in
I_{u}}\mathbb{P}({\bm{a}}_{u[i]}\in{\bm{D}}_{h})$, where the probability is
taken over $\mathcal{D}_{h}$, and $I_{u}$ is the index set of unseen real
attributes. This is because the difference of the risk will be reduced if the
generations contain as many points close to ground-truth unseen ones as
possible. Consider the Markov chain with single step transition probabilities
$p_{ij}$ of jumping from node $i$ to node $j$. Each node represents a
generated sample. Let

$p_{ij}=\mathbb{P}[h({\bm{x}}_{i})=y_{j}]\enspace,$ (29)

where $h$ is the hypothesis trained on ${\bm{D}}_{h}$, and the $h$ output
predictions on the current generation’s classification space depending on the
quality of $h$, and the probability is taken over $\mathcal{D}_{h}$. We assume
the training achieves error $\epsilon$, then $h({\bm{x}}_{i})=y_{i}$ with
probability $(1-\delta)$ if the training set contains class with attribute
${\bm{a}}_{i}$. It is not hard to prove that
$\mathbb{P}({\bm{a}}_{u[i]}\in{\bm{D}}_{h})\geq p_{ii}(1-\delta)(1-\epsilon)$
by the generalization bound, since if ${\bm{a}}_{u[i]}\notin{\bm{D}}_{h}$,
$y_{u[i]}$ is not in the current generation’s classification space. It follows
that

$\begin{split}\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})\propto&-\sum_{i}\mathbb{P}({\bm{a}}_{u[i]}\subseteq{\bm{D}}_{h})\leq-\sum_{i}p_{ii}(1-\delta)(1-\epsilon)\end{split}$
(30)

Then we can release the bound $\bar{d}_{GDB}({\bm{D}}_{u},{\bm{D}}_{h})$ by
increasing $\sum_{i}p_{ii}$. Note that
$\mathbb{P}({\bm{a}}_{u[i]}\in{\bm{D}}_{h})$ can be replaced by
$\mathbb{P}(\min_{{\bm{a}}_{h[j]}\in{\bm{D}}_{h}}|{\bm{a}}_{u[i]}-{\bm{a}}_{h[j]}|<\varepsilon)$
with the robustness assumption of the model.

When two generations have the same $\sum_{i}p_{ii}$, we prefer the one having
higher diversity. The diversity of the generated set ${\bm{D}}_{h}$ can be
quantified from the perspective of determinantal point process. As mentioned
in [32] and [14], Determinantal Point Process (DPP) is a framework for
representing a probability distribution that models diversity. More
specifically, a DPP over the set $\mathcal{V}$ with $|\mathcal{V}|=N$, given a
positive-definite similarity matrix $L\in\mathbb{R}^{N\times N}$, is a
probability distribution $P_{L}$ over any $S\subseteq\mathcal{V}$ in the
following form

$P_{L}[S]\varpropto\det(L_{s})\enspace,$ (31)

where $L_{s}$ is the similarity kernel of the subset $S$222The feature
representation of the similarity space is typically normalized so the highest
eigen value is 1, and hence the determinant (multiplication of the eigen
values) is $<$ 1. Since the point process according to this probability
distribution naturally capture the notion of diversity, we hope to generate a
subset with high $P_{L}[{\bm{D}}_{h}]$ where the $\mathcal{V}$ is viewed as
$\mathcal{D}_{h}$ and the transition matrix is viewed as the similarity
kernel. One way to generate a set of unseen samples with high
$\det(L_{{\bm{D}}_{h}})$ is to encourage the diagonality of the transition
matrix, which can be achieved by promoting orthogonality of the generated
samples. Moreover, since actually $f_{h}$ is a look-up table, low $\sum_{j\neq
i}p_{ji}$ can be explained as the large dis-similarity of the generated unseen
samples from different class.

##  Appendix B More Details of Section 5

Algorithm 1 shows the overall training process. The Discriminator and
Generator are alternatively optimized. During the training of the Generator
(line 11 – 22), we propose to generate unseen attributes (line 12 for
interpolation-based method and line 12,13 for dictionary-based method) and
encourage the generations to be realistic and deviate from the seen
generations (line 19). After the training of each task, we propose to store
the current semantic information and real features in the buffer.

###  B.1 Regularization terms in Loss Function 4

We closely follow [36] for the regularization terms of the Generator and
Discriminator. The regularization term on discriminator encourages the
semantic embedding to be close to the class center, i.e., at task $t$

$\mathcal{R}^{t}_{D}=\|D({\bm{A}}^{1:t}_{s})-{{\bm{C}}}^{1:t}_{s}\|^{2}_{\mathsf{F}}\enspace,$
(32)

where ${\bm{A}}^{1:t}_{s}$ is the attribute matrix and ${{\bm{C}}}^{1:t}_{s}$
is the class mean matrix computed by seen features up to the current task.
$\|\cdot\|_{\mathsf{F}}$ is the Frobenius norm. The regularization terms on
the generator encourage the seen generations to be close to the seen class
centers and have moderately distanced to their semantic neighborhoods.
$\mathcal{R}_{G}$ is defined as

$\mathcal{R}_{G}=L_{\text{nuclear}}+L_{\text{sal}}\enspace.$ (33)

$L_{\text{nuclear}}$ is the Nuclear loss, defined as

$L_{\text{nuclear}}=\|{{\bm{C}}}^{t}_{s}-{{\bm{C}}}^{t}_{sg}\|^{2}_{\mathsf{F}}\enspace,$
(34)

where ${{\bm{C}}}^{t}_{s}$ is the class mean matrix computed by seen features
of current task, and ${{\bm{C}}}^{t}_{sg}$ is the class mean matrix computed
by generated seen features of current task. $L_{\text{sal}}$ is the
incremental bidirectional semantic alignment loss defined as

$\begin{split}L_{\text{sal}}=&\frac{1}{N_{s}^{t}}\sum_{i=1}^{N_{s}^{t}}\sum_{j\in{\bm{I}}_{i}}\|\max\\{0,\langle{{\bm{C}}}_{s[j]},{{\bm{C}}}_{sg[i]}\rangle-(\langle{\bm{A}}^{t}_{s[i]},{\bm{A}}^{t}_{s[j]}\rangle+\varepsilon)\\}\|^{2}\\\
&+\|\max\\{0,(\langle{\bm{A}}^{t}_{s[i]},{\bm{A}}^{t}_{s[j]}\rangle-\varepsilon)-\langle{{\bm{C}}}_{s[j]},{{\bm{C}}}_{sg[i]}\rangle\\}\|^{2}\enspace,\end{split}$
(35)

where $N_{s}^{t}$ is the number of current seen classes at task $t$,
${\bm{I}}_{i}$ is the neighbor set of class $i$, $\varepsilon$ is the margin
error, $\langle\cdot,\cdot\rangle$ is the cosine similarity.

###  B.2 Visualization of Attribute Distribution

In our analysis, we assume that the hallucinated attributes can effectively
represent the real unseen attributes compactly. To visualize the distribution
of these attributes, we employ the T-SNE embedding method. As shown in Figure
5, the plot illustrates the distribution of seen attributes, unseen
attributes, and hallucinated attributes across different tasks. It is
important to note that only a partial subset of the hallucinated attributes
for each task is displayed in the plot, while the actual number of
hallucinated attributes is equivalent to the number of training samples.

Figure 5: Attribute distribution T-SNE visualizations of AWA1 dataset in
different task with interpolation method

As the task progresses and the learner is exposed to more seen classes, the
hallucinated attributes become more aligned with the unseen attribute.
However, in areas where the distribution of unseen attributes is sparse (as
indicated by the blank regions in Figure 5), the hallucinated attributes are
also sparse. In such cases, the hallucinated attributes tend to describe the
potential visual space, deviating from the seen attributes, and providing
compact support for the unseen attributes. This aligns with our assumptions
and demonstrates the efficacy of our approach.

###  B.3 Numerical Verification of GRW Loss

In statement 3.3, we asserted that the GRW loss can effectively reduce
$\bar{d}_{GDB}$. To demonstrate the relationship between the GRW loss and the
bound $\bar{d}_{GDB}$, we plotted a figure using the model at different epochs
for different $\hat{h}^{*}$. In this figure, we used the difference between
the generated hallucinated samples accuracy and the test unseen accuracy to
represent $\bar{d}_{GDB}=|\hat{\epsilon}_{u}-\hat{\epsilon}_{h}|$ at a
randomly selected task. As shown in Figure 6, we observed a strong positive
correlation between the GRW loss and $\bar{d}_{GDB}$, particularly, when the
loss decreases. This finding suggests that by minimizing the GRW loss, we can
reduce the bound between the generated hallucinated space and the true unseen
space.

Figure 6: Relationship between $\bar{d}_{GDB}$ and the GRW loss in CUB dataset

###  B.4 Relation to Other Work using Random Walk

We adapt random walk modeling [5] with three key changes.

  1. 1.

Previous works such as [5, 29] have represented class prototypes or centers
using a few examples provided for each class. However, in our setting, we aim
to deviate from seen classes and facilitate knowledge transfer to unseen
classes through attributes or semantic descriptions. To achieve this, we
define the seen class centers ${\bm{C}}$ in a semantically guided way by
computing the mean of generated seen samples from their corresponding
attributes. Specifically, we define $[{\bm{C}}]_{i}$ as the mean of generated
samples from the attribute vector $a_{i}$ for class $i$, i.e.,
$[{\bm{C}}]_{i}=\text{mean }G(z,a_{i})$, where $G$ is the generator and $z$ is
the noise vector.

  2. 2.

[5, 29] use unlabeled data points to calculate the random walk, where we use
generated examples.

  3. 3.

In contrast to the few-shot learning problem where class prototypes are
computed using unlabeled examples of seen classes, our approach generates
examples from hallucinated classes. Thus, the loss functions proposed in [5,
29] aim to attract unlabeled samples to labeled samples, whereas our goal is
to push hallucinated samples away from seen samples. In [29], global
consistency is encouraged using a random walk from labeled data to unlabeled
data (represented by their ${\bm{A}}$ matrix) and back to labeled samples
(represented by their ${\bm{A}}^{\top}$ matrix). The aim is to promote the
identity distribution of paths, where the starting and ending points are of
the same class. [5] investigates a more general case where the number of
random walk steps between unlabeled classes is greater than one (represented
by their ${\bm{B}}$ matrix). In our case, as none of the generated
hallucinated samples belong to seen classes, we use the random walk approach
to encourage uniform distribution instead of identity distribution for all the
paths from seen to generated examples of hallucinated classes and back to seen
classes, represented by our
${\bm{P}}^{{\bm{C}}_{s}X_{h}}{\bm{P}}^{X_{h}X_{h}}{\bm{P}}^{X_{h}C_{s}}$
matrix. This approach provides a deviation signal that encourages the model to
learn distinct representations for seen and hallucinated classes, facilitating
better knowledge transfer to hallucinated classes.

[57] focuses on unsupervised domain adaptation, which involves doing a random
walk over all potential labeling circumstances on unlabeled target data to
identify a stationary labeling distribution. Labeling stability is defined
from the perspective of a generalization bound which can be attained through a
stationary Markov chain. We borrow the idea of using the Markov chain to
estimate the relationship between different labeling to find a stationary one
that can reduce the generalization bound. We employ the Markov chain to
estimate the relationship between different hallucinated generations and
discover a diverse one that can reduce the generalization bound. The $L_{GRW}$
loss encourages the random walk to find a highly diverse hallucinated
generation, which in turn reduces the generalization bound.

Input : Total task number $T$, training epoch $E$, random walk length $R$,
decay rate of random walk $\gamma$, and coefficients $\lambda_{c,rd,i,rg}$,
learning rate $\alpha_{G,D,Dic}$, buffer size $B$

Data : $X^{1:T}_{s}$, $y^{1:T}_{s}$, $a^{1:T}_{s}$

Initialize : Generator, Discriminator

1 for  _ $t=1:T$_ do

2 Get train loader by concatenating train set $t$ with buffer data;

3 for  _$e=1:E$_ do

4 Get $X_{s}^{t},y_{s}^{t}$ sampled from train loader. Get $a_{s}^{1:t}$ from
current train set and buffer ;

5 begin Train Discriminator

6 Generate samples conditioning on seen attributes $X^{t}_{sg}=G(z,a_{s}^{t})$
;

7 Compute real-fake loss $\mathcal{L}_{\text{real-fake}}$ in equation (5)
using real seen samples $X_{s}^{t}$, generated seen samples $X^{t}_{sg}$, and
current task attribute $a^{t}_{s}$;

8 Compute classification loss $\mathcal{L}_{\text{classification}}$ in
equation 6 using real seen samples $X_{s}^{t}$, generated seen samples
$X^{t}_{sg}$, and attributes $a^{1:t}_{s}$;

9 Compute $\mathcal{L}_{D}$ in equation 4 and update
$\theta_{D}\leftarrow\theta_{D}-\alpha_{D}\nabla\mathcal{L}_{D}$ ;

10

11 end

12

13 begin Train Generator

14 Generate $a^{t}_{ug}$ by interpolation between two random $a^{t}_{s}$ ;

15 if  _Use dictionary based method_ then

16 Initialize the dictionary with the interpolated attribute and get
$\theta_{Dic}$

17 end if

18 Generate samples conditioning on unseen attributes
$X^{t}_{ug}=G(z,a_{ug}^{t})$ ;

19 Compute the second part of real-fake loss $\mathcal{L}_{\text{real-fake}}$
in equation (5) using generated unseen samples $X^{t}_{sg}$ and current task
attribute $a^{t}_{s}$;

20 Compute the second part of classification loss
$\mathcal{L}_{\text{classification}}$ in equation 6 using generated unseen
samples $X^{t}_{sg}$ and attributes $a^{1:t}_{s}$;

21 Compute tehe inductive loss in $\mathcal{L}_{\text{inductive}}$ using
$C_{s}^{t}=\text{mean}(X^{t}_{s})$, generated seen samples $X_{sg}^{t}$, and
unseen generated samples $X_{ug}^{t}$ Compute $\mathcal{L}_{G}$ in equation 4
and update $\theta_{G}\leftarrow\theta_{G}-\alpha_{G}\nabla\mathcal{L}_{G}$ ;

22 if  _Use dictionary based method_ then

23 $\theta_{Dic}\leftarrow\theta_{Dic}-\alpha_{Dic}\nabla\mathcal{L}_{D}$

24 end if

25

26 end

27

28 end for

29 begin Replay data

30 Save $a^{t}_{s}$ to the buffer;

31 Save current real features with size $B/N_{s}^{1:t}$ per class, reduce
previous features to size $B/N_{s}^{1:t}$

32 end

33

34 end for

Algorithm 1 Training procedure of ICGZSL

##  Appendix C Zero-shot learning experiments

###  C.1 Text based zero-shot learning experiments

Text-based ZSL is more challenging because the descriptions are at the class
level and are extracted from Wikipedia, which is noisier.

##### Benchmarks:

To evaluate the efficacy of zero-shot learning (ZSL) with text descriptions as
semantic class descriptions, we conducted experiments on two well-known
benchmarks, namely Caltech UCSD Birds-2011 (CUB)[59] and North America Birds
(NAB)[56]. While CUB contains 200 classes with 11,788 images, NAB has 1011
classes with 48,562 images. To gauge the generalization capability of class-
level text zero-shot recognition, we split the benchmarks into four subsets:
CUB Easy, CUB Hard, NAB Easy, and NAB Hard. The hard splits were designed to
ensure that the unseen bird classes from super-categories do not overlap with
seen classes, following prior work [10, 69, 16].

##### Baseline and training:

We introduced a novel GRW loss ($L_{GRW}+\mathcal{R}_{GRW}$) into the
inductive zero-shot learning method GAZSL [69] and compared its performance
with other inductive zero-shot learning methods. We employed the TF-IDF[50]
representation of the input text for the text representation function
$\psi(\cdot)$, followed by an FC noise suppression layer. Our experiments were
conducted using a random walk length $R=10$, and we found that longer random
walk processes yield better performance in the ablation study. Each ZSL
experiment was executed on a single NVIDIA P100 GPU.

##### Evaluation and metrics:

During the test, the visual features of unseen classes are synthesized by the
generator conditioned on a given unseen text description $a_{u}$, i.e.
$x_{u}=G(s_{u},z)$. We generate 60 different synthetic unseen visual features
for each unseen class and apply a simple nearest neighbor classifier on top of
them. We use two metrics: standard zero-shot recognition with the Top-1 unseen
class accuracy and Seen-Unseen Generalized Zero-shot performance with Area
under Seen-Unseen curve [10].

##### Results:

Our proposed approach improves over older methods on all datasets and achieves
SOTA on both Easy and SCE(hard) splits, as shown in Table 7. We show
improvements in 0.8-1.8% Top-1 accuracy and 1-1.8% in AUC. GAZSL [69] \+ GRW
also has an improvement of around 2% over other inductive loss (GAZSL [69] \+
CIZSL [16]).

Metric | Seen-Unseen AUC (%)
---|---
Dataset | CUB | NAB
Split-Mode | Easy | Hard | Easy | Hard
ZSLNS [47] | 14.7 | 4.4 | 9.3 | 2.3
SynCfast [9] | 13.1 | 4.0 | 2.7 | 3.5
ZSLPP [18] | 30.4 | 6.1 | 12.6 | 3.5
FeatGen [63] | 34.1 | 7.4 | 21.3 | 5.6
LsrGAN (tr) [58] | 39.5 | 12.1 | 23.2 | 6.4
+GRW |  39.9+0.4 |  13.3+1.2 |  24.5+1.3 |  6.7+0.3
GAZSL (in) [69] | 35.4 | 8.7 | 20.4 | 5.8
\+ CIZSL [16] | 39.2 | 11.9 | 24.5 | 6.4
\+ GRW |  40.7 +5.3 | 13.7+5.0 | 25.8+5.4 |  7.4 +1.6
Table 6: Showing Seen-Unseen AUC results of ZSL experiments on noisy text
description-based datasets CUB and NAB(Easy and Hard Splits)

Setting | CUB-Easy | CUB-Hard
---|---|---
Top-1 Acc | SU-AUC | Top1-Acc | SU-AUC
\+ GRW ($R$=1) | 45.41 | 39.62 | 13.79 | 12.58
\+ GRW ($R$=3) | 45.11 | 39.25 | 14.21 | 13.22
\+ GRW ($R$=5) | 45.40 | 40.51 | 14.00 | 13.07
\+ GRW ($R$=10) | 45.43 | 40.68 | 15.51 | 13.70
Table 7: Ablation studies on CUB Dataset (text). Each row shows either
baseline deviation losses and GRW losses with different length on GAZSL [69]

Metric | Top-1 Accuracy (%) |  | Seen-Unseen AUC (%)
---|---|---|---
Dataset | CUB | NAB |  | CUB | NAB
Split-Mode | Easy | Hard | Easy | Hard |  | Easy | Hard | Easy | Hard
GAZSL [69] | 43.7 | 10.3 | 35.6 | 8.6 |  | 35.4 | 8.7 | 20.4 | 5.8
GAZSL [69] \+ GRW | 45.4 | 15.5 | 38.4 | 10.1 |  | 40.7 | 13.7 | 25.8 | 7.4
GAZSL [69] \+ only $L_{GRW}$ | 45.3 | 14.8 | 38.2 | 10.3 |  | 40.1 | 12.8 | 25.8 | 7.4
Table 8: Ablation study using Zero-Shot recognition on CUB & NAB datasets with
two split settings. We experiment with and without the $\mathcal{R}_{GRW}$
(second and last row). The first loss is the baseline method.

##### GRW Loss for Transductive ZSL:

To better understand how the GRW improves the consistency of generated seen
features space and generated unseen features space, we conduct experiments on
semantic transductive zero-shot learning settings. The improvements are solely
from the GRW loss with the ground truth semantic information. We choose LsrGAN
[58] as the baseline model. Our loss can also improve LsrGAN on text-based
datasets on most metrics, ranging from 0.3%-3.6%. However, as we expected, the
improvement in the purely inductive/more realistic setting is more
significant.

##### Ablation:

Table 7 shows the results of our ablation study on the random walk length. We
find that the longer random walk performs better, giving higher accuracy and
AUC scores for both easy and hard splits for CUB Dataset. With a longer random
walk process, the model could have a more holistic view of the generated
visual representation that enables better deviation of unseen classes from
seen classes.

GRW loss contains two parts, $L_{GRW}$ and $\mathcal{R}_{GRW}$. Table 8 shows
the results of our ablation study on the $\mathcal{R}_{GRW}$ in zero-shot
learning. We perform experiments both with $\mathcal{R}_{GRW}$ and without
$\mathcal{R}_{GRW}$. Training failed with NaN gradients in 5% of the times
without $\mathcal{R}_{GRW}$ but 0% with $\mathcal{R}_{GRW}$; thus, it is
important for the training stability.

###  C.2 Attribute based zero-shot learning experiments

##### Benchmarks:

We perform these experiments on the AwA2 [37], aPY [19], and SUN [43]
datasets.

##### Baseline, training, and evaluation:

We perform experiments on the widely used GBU [62] setup, where we use class
attributes as semantic descriptors. The evaluation process and training
devices are the same as text-based experiments. We use seen accuracy, unseen
accuracy, harmonic mean of seen and unseen accuracy, and top-1 accuracy as the
evaluation metrics.

##### Results:

In Table 9, we see that GRW outperforms all the existing methods on the seen-
unseen harmonic mean for AwA2, aPY, and SUN datasets. In the case of the AwA2
dataset, it outperforms all the compared methods by a significant margin,
i.e., 15.1% in harmonic mean, and is also competent with existing methods in
Top-1 accuracy while improving 4.8%. GAZSL [69]+GRW has an average relative
improvement over GAZSL [69]+CIZSL [16] and GAZSL [69] of 24.92% and 61.35% in
harmonic mean.

| Top-1 Accuracy(%) |  | Seen-Unseen H
---|---|---|---
| AwA2 | aPY | SUN |  | AwA2 | aPY | SUN
SJE [3] | 61.9 | 35.2 | 53.7 |  | 14.4 | 6.9 | 19.8
LATEM [61] | 55.8 | 35.2 | 55.3 |  | 20.0 | 0.2 | 19.5
ALE [2] | 62.5 | 39.7 | 58.1 |  | 23.9 | 8.7 | 26.3
SYNC [9] | 46.6 | 23.9 | 56.3 |  | 18.0 | 13.3 | 13.4
SAE [35] | 54.1 | 8.3 | 40.3 |  | 2.2 | 0.9 | 11.8
DEM [68] | 67.1 | 35.0 | 61.9 |  | 25.1 | 19.4 | 25.6
FeatGen [63] | 54.3 | 42.6 | 60.8 |  | 17.6 | 21.4 | 24.9
cycle-(U)WGAN [20] | 56.2 | 44.6 | 60.3 |  | 19.2 | 23.6 | 24.4
LsrGAN (tr) [58] | $60.1$ | $34.6$ | 62.5 |  | $48.7$ | $31.5$ | 44.8
\+ GRW |  63.7+3.6 |  35.5+0.9 | 64.2+1.7 |  | 49.2+0.5 | 32.7+1.2 | 46.1+1.3
GAZSL [69] | 58.9 | 41.1 | 61.3 |  | 15.4 | 24.0 | 26.7
\+ CIZSL [16] | 67.8 | 42.1 | 63.7 |  | 24.6 | 25.7 | 27.8
\+ GRW | 68.4+9.5 | 43.3+2.2 |  62.1+0.8 |  |  39.0+23.6 |  27.2+3.2 |  27.9+1.2
Table 9: Zero-Shot Recognition on class-level attributes of AwA2, aPY and SUN
datasets, showing that GRW loss can improve the performance on attribute-based
datasets.

##  Appendix D Continual zero-shot learning experiments

###  D.1 Dataset and Continual Zero-Shot Learning Setup

We display the seen and unseen class conversions in each task for each dataset
in the Table 11 to provide a better understanding of the specific
implementation of CZSL on different datasets. Covered class means the number
of unseen class converted to seen class per task.

| AWA1 | AWA2 | CUB | SUN
---|---|---|---|---
Total classes | 50 | 50 | 200 | 705
Number of tasks | 5 | 5 | 20 | 15
Initial seen classes | 10 | 10 | 10 | 47
Covered class | 10 | 10 | 10 | 47
Table 10: Seen and Unseen classes in different dataset

| AWA1 |  | AWA2 |  | CUB |  | SUN
---|---|---|---|---|---|---|---
| Inter. | Dic. |  | Inter. | Dic. |  | Inter. | Dic. |  | Inter. | Dic.
$\lambda_{\text{c}}$ | 10 | 1 |  | 1 | 10 |  | 1 | 1 |  | 1 | 1
$\lambda_{\text{i}}$ | 0.5 | 2 |  | 1 | 5 |  | 2 | 2 |  | 5 | 1
$R$ | 3 | 3 |  | 3 | 3 |  | 5 | 5 |  | 5 | 5
Table 11: The hyperparameter for Table 1

###  D.2 More Ablations

##### Random seed:

We experiment with multiple random seeds on the CUB dataset and show the
averaged mH (line) and standard deviation (shadow) in Figure 7. The random
seed mainly affects the generation part of GZSL learners. The generated data
is used directly or indirectly to train the classifier of the unseen class.
Figure 7 shows that previous models are sensitive to random seeds, but our
model is not. Previous models use the generated data as replay data or
directly train the classifier, while ours avoids these. Our method uses a non-
parametric classifier, a similarity-based classifier. During training, we pay
more attention to improving the generalization ability of our embedder
(discriminator) by encouraging the consistency between the generated visual
space and the true visual space. Plus, we store the real data in the buffer.
These all make our model more stable. Although we only reported the results of
one seed (2222) in Table 4, the figure shows that the effect of different
seeds on the results is not significant.

We also report mean and standard deviation of multiple runs of our methods in
each dataset in Table 13, 13. It shows that experiments on all the datasets
with both attribute generation methods have relatively small variance.
Although interpolation-based method has lower mean harmonic accuracy on fine-
grained dataset CUB and SUN, it is shown to be more stable with less variance
than dictionary-based method.

Figure 7: Mean harmonic accuracy at the end of each task with 5 different
random seeds on CUB [59]. Lines show the mH, and shadows show the standard
deviation.

| mSA |  | mUA |  | mHA
---|---|---|---|---|---
| Mean | Std |  | Mean | Std |  | Mean | Std
AWA1 | 65.87 | 1.19 |  | 33.77 | 1.00 |  | 42.69 | 0.57
AWA2 | 70.52 | 0.46 |  | 34.52 | 0.90 |  | 44.45 | 0.79
CUB | 42.11 | 0.88 |  | 22.10 | 0.67 |  | 27.80 | 0.53
SUN | 36.29 | 0.18 |  | 21.07 | 0.33 |  | 26.44 | 0.20
Table 12: Our method in continual zero shot learning with interpolated
attributes. Mean and variance calculated on three runs with different random
seeds.

| mSA |  | mUA |  | mHA
---|---|---|---|---|---
| Mean | Std |  | Mean | Std |  | Mean | Std
AWA1 | 66.35 | 0.28 |  | 32.75 | 0.94 |  | 41.90 | 0.91
AWA2 | 70.55 | 0.51 |  | 33.88 | 0.60 |  | 43.49 | 0.88
CUB | 42.22 | 0.30 |  | 22.78 | 0.91 |  | 28.09 | 0.68
SUN | 36.63 | 0.12 |  | 21.39 | 0.47 |  | 26.79 | 0.37
Table 13: Our method in inductive continual zero shot learning with learnable
dictionary of attributes. Mean and variance calculated using three runs with
different random seeds

###  D.3 Hyperparameters in GRW loss

Hyperparameter for Table 1: We use the validation set to tune the
hyperparameter random walk step $R$, coefficient of $L_{\text{creativity}}$
$\lambda_{\text{c}}$, and coefficient of $L_{\text{inductive}}$
$\lambda_{\text{i}}$. The hyperparameter used to report Table 1 is shown in
Table 11

Walk length $R$ and decay rate $\gamma$: We do an ablation study on the random
walk length $R$ and decay rate $\gamma$ of the GRW loss in continual zero-shot
learning experiments. Table 14 shows our method with different random walk
lengths in AWA1 dataset and CUB dataset. In the dataset AWA1, moderate lengths
give the highest mHA while in the CUB dataset higher random walk lengths
provide the best mHA. It shows that the more challenging the dataset, the more
random walk length is needed. Unlike ZSL experiments, in CZSL experiments,
knowledge is not only transferred to the unseen class space but also to the
next task. Long walk length could give the model a more holistic view of the
current task, but may harm the transformation to the next task. Therefore,
tuning the number of random walk steps is required for new datasets.

|  | Ours Interpolation |  | Ours Dictionary
---|---|---|---|---
|  | mSA | mUA | mHA |  | mSA | mUA | mHA
$R$ | 1 | 41.7 | 21.2 | 27.1 |  | 43.6 | 22.7 | 28.4
3 | 42.3 | 22.1 | 27.7 |  | 42.1 | 20.6 | 26.6
5 | 42.2 | 22.7 | 28.4 |  | 42.4 | 23.6 | 28.8

|  | Ours Interpolation |  | Ours Dictionary
---|---|---|---|---
|  | mSA | mUA | mHA |  | mSA | mUA | mHA
$R$ | 1 | 65.4 | 33.9 | 42.7 |  | 66.6 | 32.7 | 41.5
3 | 67.0 | 34.2 | 43.4 |  | 67.1 | 33.5 | 42.8
5 | 65.8 | 33.0 | 42.1 |  | 66.8 | 32.7 | 41.7

Table 14: Our method with different random walk length $R$ in AWA1 dataset
(right) and CUB dataset (left)

Decay rate $\gamma$ works as a scale factor to the GRW loss to prevent a
specific area in the probability matrix from being too close to one, resulting
in exponential growth in the multiplication results when compared to other
areas. Compared to the non-decay case when $\gamma=1$ in Table 15, the decayed
case has noticeable improvements in unseen accuracy, resulting in better
harmonic accuracy.

|  | Ours-interpolation |  | Ours-dictionary
---|---|---|---|---
|  | mSA | mUA | mH |  | mSA | mUA | mH
$\gamma$ | 0.7 | 40.97 | 21.78 | 27.26 |  | 42.22 | 22.03 | 27.47
| 1 | 40.95 | 21.21 | 27.05 |  | 42.62 | 21.6 | 27.43

|  | Ours-interpolation |  | Ours-dictionary
---|---|---|---|---
|  | mSA | mUA | mH |  | mSA | mUA | mH
| 0.7 | 66.8 | 33.42 | 42.87 |  | 66.93 | 32.41 | 41.51
$\gamma$ | 1 | 66.07 | 32.31 | 41.69 |  | 66.34 | 32.87 | 41.76

Table 15: Our method with different decay rate $\gamma$ on CUB dataset (left)
and AWA1 dataset (right)

###  D.4 Ablations on Weight of Inductive Loss

##### Inductive weight $\lambda_{i}$

We also do an ablation study on the inductive coefficient $\lambda_{i}$ in
Table 16. This factor mainly affects the proportion of inductive loss in the
overall loss. We found that our model is not sensitive to this hyperparameter.
Whether on the larger dataset CUB or the smaller dataset AWA1, the difference
of mH of different $\lambda_{i}$ on our model does not exceed 1%. Therefore,
our model does not need too much parameter tuning process.

|  | Ours-interpolation |  | Ours-dictionary
---|---|---|---|---
|  | mSA | mUA | mH |  | mSA | mUA | mH
| 0.01 | 41.81 | 20.93 | 27.01 |  | 42.8 | 23.07 | 28.51
| 0.1 | 42.32 | 21.27 | 27.11 |  | 42.73 | 21.98 | 27.85
$\lambda_{i}$ | 1 | 40.97 | 21.78 | 27.26 |  | 42.22 | 22.03 | 27.47

|  | Ours-interpolation |  | Ours-dictionary
---|---|---|---|---
|  | mSA | mUA | mH |  | mSA | mUA | mH
| 0.1 | 66.81 | 32.82 | 42.15 |  | 66.32 | 32.11 | 41.15
| 1 | 66.8 | 33.42 | 42.87 |  | 66.93 | 32.41 | 41.51
$\lambda_{i}$ | 10 | 66.38 | 33.77 | 42.92 |  | 66.47 | 31.81 | 40.89

Table 16: Our method with different inductive coefficients $\lambda_{i}$ on
CUB dataset (left) and AWA1 dataset (right)

###  D.5 Continual zero-shot learning with other common settings

Although our main research problem is inductive setting, and we think real
replay is needed, we still have an open attitude to other settings and migrate
our model naively to their setting. We show experiment results in these
settings in Table 17 and compare them with other methods.

Table 17: Comparison of our inductive loss in other common CZSL settings

| replay method | zsl setting | AWA1 | AWA2 | CUB | SUN
---|---|---|---|---|---|---
|  |  | mSA | mUA | mHA | mSA | mUA | mHA | mSA | mUA | mHA | mSA | mUA | mHA
CN-ZSL | real | in | - | - | - | 33.55 | 6.44 | 10.77 | 44.31 | 14.8 | 22.7 | 22.18 | 8.24 | 12.46
Ours-interpolation | real | in | 62.9 | 32.77 | 42.03 | 67.41 | 35.4 | 45.06 | 40.17 | 21.78 | 27.26 | 36.29 | 21.05 | 26.51
Ours-dictionary | real | in | 63.43 | 32 | 41.15 | 68.02 | 33.22 | 42.89 | 41.45 | 22.03 | 27.47 | 36.54 | 21.31 | 26.76
DVGR | generative | tr | 65.1 | 28.5 | 38 | 73.5 | 28.8 | 40.6 | 44.87 | 14.55 | 21.66 | 22.36 | 10.67 | 14.54
A-CGZSL | generative | tr | 70.16 | 25.93 | 37.19 | 70.16 | 25.93 | 37.19 | 34.25 | 12.42 | 17.41 | 17.2 | 6.31 | 9.68
BD-CGZSL | generative | tr | 67.55 | 36.04 | 47.88 | 71.37 | 38.76 | 51.6 | 31 | 23.97 | 26.01 | 30.08 | 20.07 | 23.72
Ours-interpolation | generative | tr | 62.43 | 33.03 | 42.01 | 66.84 | 34.01 | 43.77 | 32.53 | 16.66 | 21.65 | - | - | -
Ours-dictionary | generative | tr | 62.34 | 31.5 | 40.18 | 68.07 | 34.45 | 44.17 | 30 | 16.18 | 20.55 | - | - | -
BD-CGZSL-in | generative | in | 62.12 | 31.51 | 40.46 | 67.68 | 32.88 | 42.33 | 37.76 | 9.089 | 14.43 | 34.93 | 14.86 | 20.8
Ours-interpolation | generative | in | 61.43 | 34.04 | 42.18 | 67.34 | 35.29 | 44.95 | 29.78 | 16.86 | 21.06 | 30.9 | 18.4 | 22.99
Ours-dictionary | generative | in | 62.26 | 30.88 | 39.68 | 67.44 | 33.68 | 43.24 | 28.34 | 16.94 | 20.57 | 30.13 | 18.56 | 22.85

We mentioned earlier that the generative replay method has unbalanced storage
and buffer overload problems, but many models still use generative replay.
When data privacy concerns are encountered, the generative replay method may
be an alternative to the real replay method. When using the generative replay,
our model outperforms most existing methods. Our problem analysis cannot be
applied in this setting, since we believe the replayed feature should have a
balanced number in each class.

Our primary focus is on the inductive setting, but we also provide results in
the transductive setting and with generative replay. In the transductive
setting, we use the ground truth unseen attributes to generate the visual
features, and our loss works on these generations. Our method is comparable
with other transductive methods, even without carefully designing how to use
the semantic information.

Through these knots, we believe that our model has the possibility of being
migrated to other settings and is valuable for further explorations in other
settings.