# Variational Autoencoder with Disentanglement Priors for Low-Resource Task-
Specific Natural Language Generation

Zhuang Li  Lizhen Qu  Qiongkai Xu  Tongtong Wu  Tianyang Zhan  Gholamreza
Haffari

###### Abstract

In this paper, we propose a variational autoencoder with disentanglement
priors, VAE-DPrior, for conditional natural language generation with none or a
handful of task-specific labeled examples. In order to improve compositional
generalization, our model performs disentangled representation learning by
introducing a prior for the latent content space and another prior for the
latent label space. We show both empirically and theoretically that the
conditional priors can already disentangle representations even without
specific regularizations as in the prior work. We can also sample diverse
content representations from the content space without accessing data of the
seen tasks, and fuse them with the representations of novel tasks for
generating diverse texts in the low-resource settings. Our extensive
experiments demonstrate the superior performance of our model over competitive
baselines in terms of i) data augmentation in continuous zero/few-shot
learning, and ii) text style transfer in both zero/few-shot settings.

Autoencoder, Text Generation, Disentanglement Learning

##  1 Introduction

Pre-trained large-scale natural language generation (NLG) models demonstrate
their high effectiveness in producing fluent and coherent texts (Floridi &
Chiriatti, 2020). However, it is still challenging to apply them to
efficiently generate diverse texts for specific tasks with few training data
(Liu et al., 2020a), such as generating texts matching the new personality of
a virtual character without accessing the large training data of known
personalities. Herein, models are expected to understand how to combine the
representations of seen contents with the representation of the new
personality for personalized language generation. In contrast, human
intelligence exhibits systematic compositionality (Fodor & Pylyshyn, 1988) in
language, the ability to produce a huge number of novel combinations of known
linguistic constructs.

To achieve compositional generalization for new tasks in the low resource
setting, it is desirable for NLG models to i) efficiently sample diverse
content representations even when previous training data is not available, and
ii) learn how to efficiently combine sampled content representations with the
information of new tasks. The work on controllable text generation allows
generated natural texts to satisfy given attributes, such as positive
sentiment (Prabhumoye et al., 2020). However, those studies have limited
capabilities to leverage rich content information in pre-trained language
models to generate diverse texts. Another line of research (John et al.,
2018a), including their applications to text style transfer (Cheng et al.,
2020), has explored decomposing input texts into latent representations in
different subspace, at least one of which represent target attributes. Those
methods do not have the capability to sample diverse contents from their
models without accessing the original training data, so they fail to generate
a large number of diverse texts of new styles in the continuous zero or few-
shot settings. Our work, therefore, focuses on disentangled representation
learning for large pre-trained models to generate diverse and natural
sentences for new tasks in low resource settings.

In this work, we propose a deep VAE model with disentanglement priors for
controlled natural language generation, VAE-DPrior, in the zero-shot and few-
shot settings. This model performs disentangled representation learning by
introducing a latent content subspace and a latent label subspace. As the
information in the two subspace is different, we expect the representations
from the subspaces also different. Hence, we propose to use a different prior
for a different subspace. In order to feature rich modes in the content
subspace, we employ a Gaussian mixture with a large number of component
Gaussians as the content prior. In the label subspace, we adopt a label
encoder to construct latent label representations from a label name and/or a
small set of labeled examples. Our GPT2 decoder (Wang et al., 2019) based on
the prefix-tuning technique (Li & Liang, 2021) effectively combines both rich
content embeddings and label representations to generate diverse and natural
sentences. In addition, we empirically find that VAE-DPrior alleviates mode
collapse, which is a problem that prevents models from generating diverse word
sequences.

To sum up, our key contributions are two-fold:

  * •

We propose a deep VAE-based model with disentanglement priors for controlled
NLG in low resource settings, which effectively integrates multiple
controllable NLG and disentangled representation learning techniques into the
same mathematical framework.

  * •

We demonstrate the effectiveness of VAE-DPrior through extensive experiments
for two types of tasks: text style transfer and natural language understanding
with data augmentation. In each task, we consider both zero shot and few shot
settings in order to evaluate the capability of models for compositional
generalization. In all experimental configurations, our models outperform the
competitive baselines with a wide margin.

##  2 Related Work

#### Variational Autoencoder.

A large series of work learn disentangled representations based on generative
models, such as Variational Autoencoder (VAE) (Kingma & Welling, 2019). A
standard VAE minimizes the Kullback–Leibler divergence between the parametric
posterior and the true posterior. Different posterior integrates various
properties to the generative models, VQ-VAE (Oord et al., 2017) parameterizes
a posterior distribution of discrete latent random variables, Categorical-VAE
(Jang et al., 2016) and Vamp-VAE (Tomczak & Welling, 2018) constructs mixture
of posteriors on learnable pseudo-inupts.They are not capable of remembering
rich contents in NLG. An Identifiable Double VAE (Mita et al., 2021) does not
use two different priors for different subspace. It uses only one prior based
on observed random variables to remove the observed information from the
latent representations in order to achieve disentanglement. In another word,
it has no component to remember contents for continuous few shot setting. Our
work considers the posterior on multiple types of disentangle random
variables, which is potentially of more expressiveness.

Disentangled Representation Learning. Disentanglement of representations is
one of the ultimate goals of deep learning. The existing methods are either
unsupervised or supervised (Higgins et al., 2018). The unsupervised ones
mainly fall into either the framework of VAE (Burgess et al., 2018) or
Generative Adversarial Learing (GAN) (Tran et al., 2017). The recent works
have also incorporate causality theories for robustness (Hu & Li, 2021). There
are growing interests in applying disentangled representation learning in NLP
applications, such as text style transfer (John et al., 2018b) and mitigating
gender bias (Liu et al., 2020b). However, it is challenging for those NLP
approaches to work in the low resource settings because they do not store rich
content information inside models (Romanov et al., 2018).

Controllable Text Generation. Our method decomposes content and (attribute)
label, where the label could be considered as additional control signal for
text generation. We connect our work to those text style transfer (TST) and
controllable text generation (CTG). Representation disentanglement is an
important line of research in TST, which disentangles content and attribute
representations (John et al., 2018b). Many disentanglement approaches are
proposed to minimize the dependence between these two representations, such as
mutual information (Yuan et al., 2020) and orthogonality (Wei et al., 2021).
CTG controls the text generation of language models by smart prompt design (Li
& Liang, 2021; Shin et al., 2020) or training conditioned on the controllable
variables (Li et al., 2020; Hu & Li, 2021). Our work is highly aligned with
(Li et al., 2020; Li & Liang, 2021). Since (Li et al., 2020) and (Li & Liang,
2021) have similar implementations, and (Li et al., 2020) was designed for
generation conditioned on latent variables, we pick (Li et al., 2020) as one
of our baselines to compare against.

##  3 Methodology

To tackle compositional generalization in the low-resource settings, we
present the deep generative model VAE-DPrior, which is capable of disentangled
representation learning for task-specific NLG with few labeled data. Given a
phrase describing a new task $y\in{\mathcal{Y}}$ and optionally a handful of
example texts of this task
${\mathcal{D}}_{sup}=\\{{\bm{x}}_{i},y_{i}\\}_{i=1}^{m}$, the generative model
aims to generate a set of word sequences ${\mathcal{D}}_{y}$ of the new task
with $|{\mathcal{D}}_{y}|\gg m$ with the knowledge learned from previous
tasks. A new task can be a new personality category for blog classification.
Our generative model can provide synthetic data for better training of the
base learner (e.g. a text classifier) on the novel tasks. The model can also
be applied for text style transfer in the low resource settings, if an input
text is provided.

Disentangled representation learning for compositional generalization in low
resource NLG tasks poses three major challenges. First, models need to
encourage modularization by disentangling task-specific information from
content information. Hence, there should be a subspace for task-specific
information and another subspace for the remaining content information.
Second, the model should be able to memorize content prototypes and sample
rich content representations from the distribution of the content space
without experiencing mode collapse, even if the data of previous tasks is no
longer available. Third, the generated texts should feature high naturalness
and fluency.

Our model addresses the first challenge by introducing a content subspace and
a label subspace between a pre-trained BERT and a pre-trained GPT2 (see Figure
1.b). Given a new label, our model encodes the label phrase and optionally the
labeled examples from a support set into a label representation. If an input
text is given for text style transfer, we extract content representation from
the input, otherwise we sample diverse content representations from the model
to fuse with the label representation for text generation. It enforces
representation disentanglement by employing a conditional Gaussian mixture
prior for content representations and a conditional Gaussian prior for each
label. We show both empirically and theoretically that disentanglement of
representations is guaranteed if certain properties of the conditional priors
are satisfied. Our also empirically show that our model mitigates both mode
collapse and posterior collapse by employing the conditional priors and hard-
EM for learning the Gaussian mixture. Moreover, the mean vector of each
component Gaussian in the Gaussian mixture represents a distinct content
prototype so that the model can sample rich content representations
proportional to the number of component Gaussians without accessing original
training data. For the last challenge, we freeze the parameters of GPT2 during
training to encourage naturalness and fluency of generated texts (Li & Liang,
2021).

In what follows, we will present the technical framework in Sec. 3.1, followed
by introducing the VAE model, as well as the corresponding inference and
training methods.

###  3.1 The Disentangled Representation Learning Framework for Task-Specific
NLG

Figure 1: (a) A directed graphical model for disentanglement learning. (b) The
architecture for controllable NLG.

The causal relationships between latent representations and the observed texts
are illustrated by the causal graph $G=(\mathcal{V},\mathcal{E})$ in Figure
1(a). The latent representation $Z_{y}\in\mathcal{V}$ denotes the semantic
information related to a label $y\in\mathcal{Y}$, which denotes a finite
discrete space, while the latent representation $Z_{c}\in\mathcal{V}$ denotes
a content $c\in\mathcal{C}$. As an edge denotes a random variable is a direct
cause of another, both latent random variables $Z_{c}$ and $Z_{y}$ jointly
lead to $X$, which denotes the random variable for observable word sequences.
The causal graph implies a structured equation function
$X=f_{\bm{\theta}}(Z_{c},Z_{y})+U$, which projects values of $Z_{c}$ and
$Z_{y}$ and a random noise $U$ into word sequences.

The causal graph is characterized by the joint distribution
$p_{\bm{\theta}}(X,Z_{c},Z_{y}|Y,C)$, where $C$ denotes the prior knowledge of
$Z_{c}$. The marginal distribution $p_{\bm{\theta}}(X|C,Y)$ is then,

$\int
p_{\bm{\theta}}(X|Z_{c},Z_{y})p_{\bm{\theta}}(Z_{y}|Y)p_{\bm{\theta}}(Z_{c}|C)dZ_{y}dZ_{c}.$
(1)

We learn the above joint distribution in the VAE framework. Note that, the
introduction of the conditions $C$ and $Y$ makes the priors of both latent
variables conditional, which differ from vanilla VAEs that have only
unconditional priors for latent variables.

Given a dataset ${\mathcal{D}}=\\{({\bm{x}}_{i},y_{i})\\}_{i=1}^{n}$, apart
from learning the distribution $p(X|C,Y)$, a model should also know how to
construct disentangled representations for $Z_{c}$ and $Z_{y}$ respectively.
As noted in prior work (Cheng et al., 2020), a value of $Z_{y}$ should encode
the information of a label $y$ by maximizing the corresponding mutual
information $I(Z_{y},Y)$. Similarly, the model should also maximize the mutual
information $I(Z_{c},C)$ between $Z_{c}$ and the prior knowledge of content
$C$. Inspired by neural topic modelling, we encode the prior knowledge of
content into a $k$-means clusters based on bert embeddings. As such, each
cluster centroid corresponds to a topic or a content prototype. As a result,
we aim to maximize the following objective:

$\log p_{\bm{\theta}}(X|C,Y)+I(Z_{c},C)+I(Z_{y},Y)$ (2)

Maximizing the above function does not guarantee that $Z_{c}$ and $Z_{y}$ do
not share the same cause for a value of $X$. Hence, we introduce the notion of
strictly disentangled to characterize that the value of $Z_{c}$ is different
than that of $Z_{y}$ for the same $X$ after disentanglement.

######  Definition 3.1 (Strictly disentangled).

Given a distribution $p(X,Z_{c},Z_{y})$ and a function
$f_{\bm{\theta}}(Z_{c},Z_{y})$ with the parameters $\bm{\theta}$, if
${\bm{x}}_{i}=f_{\bm{\theta}}({\bm{z}}_{c,i}$, ${\bm{z}}_{y_{i}})$ and
${\bm{z}}_{c_{i}}\neq{\bm{z}}_{y_{i}}$ $\forall x_{i}$ with $p(x_{i})>0$,
$Z_{c}$ and $Z_{y}$ are strictly disentangled on $p(X,Z_{c},Z_{y})$.

We propose to encourage $Z_{c}$ and $Z_{y}$ strictly disentangled by assuming
their values are drawn from two dramatically different priors. The rationale
behind this is that when maximizing the objective (2), a latent value achieves
a high probability only in one of the priors, not both. As a result, a shared
value is unlikely assigned to both latent variables to achieve a high
likelihood. Geometrically, the idea indicates that the conditional priors that
have no or little shared regions, as indicated by Lemma 3.2.

######  Lemma 3.2 (Non-overlapped conditional priors $\rightarrow$ strictly
disentangled).

Assume
$R_{z_{c},z_{y}}=\\{{\bm{z}}_{c},{\bm{z}}_{y}|p_{\bm{\theta}}({\bm{x}},{\bm{z}}_{c},{\bm{z}}_{y}|C,y)>0\\}$
is non-empty for all data points in the dataset ${\mathcal{D}}$, let
$R_{z_{c}}=\\{{\bm{z}}_{c}|p_{\bm{\theta}}({\bm{z}}_{c}|C)>0\\}$, and
$R_{z_{y}}=\\{{\bm{z}}_{y}|p_{\bm{\theta}}({\bm{z}}_{y}|y)>0\\}$ denote the
positive support set of the corresponding random variables with the parameters
$\bm{\theta}$, if $R_{z_{c}}\cap R_{z_{y}}=\emptyset$, then $Z_{c}$ and
$Z_{y}$ are strictly disentangled on ${\mathcal{D}}$.

If priors with non-overlapped support sets are difficult to find for an
application, we can mitigate this by allowing two priors have sufficiently
different probabilities on the same values. This idea can be realized by
adding a divergence measure based regularization, such as Maximum Mean
Discrepancy $\text{MMD}(p(Z_{y}|Y),p(Z_{c}|C))$ (Gretton et al., 2012).

All above learning goals are summarized by solving the following optimization
problem.

$\displaystyle\max$ $\displaystyle\sum_{i=1}^{n}[\log
p({\bm{x}}_{i}|C,y_{i})+\text{PMI}({\bm{z}}_{c,i},C)+\text{PMI}({\bm{z}}_{y_{i}},y_{i})]$
$\displaystyle s.t.$ $\displaystyle Z_{c}\text{ and }Z_{y}\text{ are strictly
disentangled.}$ (3)

where PMI denotes the pointwise mutual information, e.g.
$\text{PMI}({\bm{z}}_{y_{i}},y_{i})=\log\frac{p(z_{y_{i}},y_{i})}{p(z_{y_{i}})p(y_{i})}$.
In what follows, we show how to apply variational techniques to derive an
evidence lower bound (ELBO) for the above optimization problem, and address
the constraint by a proper selection of priors.

###  3.2 VAE Model

The main difficulty of maximum likelihood learning for the optimization
problem (3.1) is that the marginal probability of data $p(X|C,Y)$ under the
model is intractable. We apply the variational techniques to derive the ELBO
for the optimization problem (3.1), whose constraint is removed by introducing
the priors satisfying Lemma 3.2.

More specifically, we introduce a variational posterior
$q_{\phi}(Z_{c},Z_{y}|X,C,Y)$ to approximate the true posterior
$p_{\bm{\theta}}(Z_{c},Z_{y}|X,C,Y)$, and derive the ELBO for
$p_{\bm{\theta}}(X|C,Y)$ as follows (Details in the Appendix G.1):

$\displaystyle\mathbb{E}_{q_{\bm{\phi}}(Z_{c},Z_{y}|X,C,Y)}[\log
p_{\bm{\theta}}(X,Z_{c},Z_{y}|C,Y)$ $\displaystyle-\log
q_{\bm{\phi}}(Z_{c},Z_{y}|X,C,Y)]$ (4)

We show in Appendix G.2 that the ELBO objective can be further decomposed
into:

$\displaystyle\begin{split}&\overbrace{\mathbb{E}_{q_{\bm{\phi}}(Z_{c},Z_{y}|X,C,Y)}[\log
p_{\bm{\theta}}(X|Z_{c},Z_{y})]}^{{\mathcal{L}}_{r}}\\\
&-\mathbb{D}_{\mathrm{KL}}(q_{\bm{\phi}}(Z_{c}|X,C)\|p_{\bm{\theta}}(Z_{c}|C))\\\
&-\mathbb{D}_{\mathrm{KL}}(q_{\bm{\phi}}(Z_{y}|X,Y)\|p_{\bm{\theta}}(Z_{y}|Y))\end{split}$
(5)

where the first term is referred to as the reconstruction loss
${\mathcal{L}}_{r}$, the other terms together with the mutual information
terms $I(Z_{c},C)+I(Z_{y},Y)$ constitute regularizers, which will be discussed
in Sec. 3.3. Following the convention of VAE, we refer to the network for
$q_{\bm{\phi}}(Z_{c},Z_{l}|X,C,Y)$ as inference module, the network for
$p_{\bm{\theta}}(X|Z_{c},Z_{l})$ as generator. An overview of the architecture
is depicted in Figure 1(b).

The choice of priors are critical for disentangled representation learning. In
this work, we assume both $p_{\bm{\theta}}(Z_{y}|Y)$ and
$q_{\bm{\phi}}(Z_{y}|Y)$ are simple factorized Gaussians. As such, in the
label subspace.
$p_{\bm{\theta}}(Z_{y}|Y)=\mathcal{N}(Z_{y};\bm{\mu}^{p}_{y},\lambda_{y}{\mathbf{I}})$
and
$q_{\bm{\phi}}(Z_{y}|Y)=\mathcal{N}(Z_{y};\bm{\mu}^{q}_{y},\text{diag}(\bm{\sigma}^{2})_{y})$.
In the content subspace, rich content indicates the distribution has multiple
modes, hence we employ the Gaussian mixture with a large number of component
Gaussian for the prior
$p_{\bm{\theta}}(Z_{c}|C)=\sum_{k=1}^{K}p_{\bm{\theta}}(M_{k}|C)\mathcal{N}(Z_{c};\bm{\mu}^{p}_{c,k},\lambda_{c}{\mathbf{I}})$,
where $M_{k}$ is the random variable indicating the membership to the
component Gaussian $k$. The corresponding variational distribution
$q_{\bm{\phi}}(Z_{c}|C)=\mathcal{N}(Z_{c};\bm{\mu}^{q}_{c},\text{diag}(\bm{\sigma}^{2})_{c})$
is a simple factorized Gaussian. We consider the means of Gaussians in the
label space are functions of labels, while each mean of a component Gaussian
in the content space is a function of a centroid in the $k$-means cluster
built in the pre-training stage. As the nature of constructing those
distributions make their mean vectors differ substantially from each other,
there are always sufficiently small variances $\lambda_{y}$ and $\lambda_{c}$
such that those distributions have little overlaps.

#### Inference Module.

The inference module is a transformer-based encoder augmented with an encoder
for content, and an encoder for labels. Each encoder is built on top of the
contextual embedding sequences produced by the transformers and yields latent
representations of the target type. This design is not only parameter
efficient but also leverages the strengths of a large-scale pre-trained
transformers.

Given a contextual word embedding sequence
${\mathbf{V}}=\\{{\bm{v}}_{0},...,{\bm{v}}_{u}\\}$ generated by a pre-trained
transformer (e.g. BERT (Devlin et al., 2018)), the label encoder is
implemented as a mean pooling layer followed by a linear layer for the mean
and another linear layer for the logarithm of the variance of
$q_{\bm{\phi}}(Z_{y}|Y)$. By applying the reparameterization trick (Kingma &
Welling, 2019), we compute ${\bm{z}}_{y}$ as a function of ${\bm{h}}_{y}$ and
the stochasticity is added by the product of $\bm{\sigma}_{y}$ and the
Gaussian noise $\bm{\epsilon}_{y}$ drawn from $\mathcal{N}(0,{\mathbf{I}})$.

$\displaystyle\log\bm{\sigma}_{y},{\bm{h}}_{y}$
$\displaystyle=\text{LabelEncoder}({\mathbf{V}})$ $\displaystyle{\bm{z}}_{y}$
$\displaystyle={\bm{h}}_{y}+\bm{\sigma}_{y}\odot\bm{\epsilon}_{y}$ (6)

where $\odot$ denotes the element-wise product.

We implement the content encoder in a similar manner by using the
reparameterization trick. Taking an output ${\mathbf{V}}$ from the transformer
as input, the content encoder is implemented as the mean pooling of
${\mathbf{V}}$, followed by the linear layers to compute the mean and variance
of $q_{\bm{\phi}}(Z_{c}|C)$, respectively. The latent content representation
${\bm{z}}_{c}$ is a function of ${\bm{h}}_{c}$ and a Gaussian noise
$\bm{\epsilon}_{c}$ sampled from $\mathcal{N}(0,{\mathbf{I}})$.

$\displaystyle\log\bm{\sigma}_{c},{\bm{h}}_{c}$
$\displaystyle=\text{ContentEncoder}({\mathbf{V}})$
$\displaystyle{\bm{z}}_{c}$
$\displaystyle={\bm{h}}_{c}+\bm{\sigma}_{c}\odot\bm{\epsilon}_{c}$ (7)

The main differences to the label encoder lie in the associated regularization
terms for training, which is covered in Sec 3.3.

#### Generator.

Given a pair of latent representations $({\bm{z}}_{c},{\bm{z}}_{y})$, the
generator aims to learn the conditional distribution
$p(X|{\bm{z}}_{c},{\bm{z}}_{l})$, which is further factorized into the
probability of each word in a sequence
${\bm{x}}=\\{x_{1},...,x_{|{\bm{x}}|}\\}$.

$\displaystyle
p_{{\bm{\theta}}}({\bm{x}}|{\bm{z}}_{c},{\bm{z}}_{y})=\prod_{t=1}^{|{\bm{x}}|}p_{{\bm{\theta}}}(x_{t}|{\bm{x}}_{<t},{\bm{z}}_{c},{\bm{z}}_{y})$
(8)

Although fine-tuning is the prevalent approach for adapting a large pre-
trained language model (LM) for downstream tasks, it is prone to catastrophic
forgetting when learning several new tasks. Therefore, we employ the prefix-
tuning technique (Li & Liang, 2021) to let the generator perform in the low-
resource situations. The key idea of prefix-tuning is to reuse the parameters
of a pre-trained LM and prepend a task-specific continuous prompt to the
inputs of the LM.

A continuous prompt is a sequence of continuous vectors computed by the MLP
$M_{\theta}[i,:]=\text{MLP}_{\theta}([M^{\prime}_{\theta}[i,:];{\bm{z}}_{y};{\bm{z}}_{c}])$,
where $M^{\prime}_{\theta}\in\mathbb{R}^{|M_{idx}|\times H^{\prime}}$ is a
learned matrix to encode the position information of the continuous prompt.

###  3.3 Training and Inference

Given a training corpus $\mathcal{D}=\\{{\bm{x}}_{i},y_{i}\\}_{i=0}^{n}$, the
training objective (3.1) is derived from the ELBO
${\mathcal{L}}_{\bm{\theta},\bm{\phi}}(\mathcal{D})={\mathcal{L}}_{r}+\sum_{y}{\mathcal{L}}_{y}(y)+{\mathcal{L}}_{c}$,
where ${\mathcal{L}}_{y}$ and ${\mathcal{L}}_{c}$ are the reguarlization terms
derived from the corresponding KL divergences of the ELBO and mutual
information terms.

Regularization in the Label Subspace. We derive the regularization term in the
label subspace from
$\mathbb{D}_{\mathrm{KL}}(q_{\bm{\phi}}(Z_{y}|X,Y)\|p_{\bm{\theta}}(Z_{y}|Y))$
and $I(Z_{y},Y)$. We assume both involved distributions are simple factorized
Gaussians. The key difference to most VAEs is that we assume
$p_{\bm{\theta}}(Z_{y}|Y)=\mathcal{N}(Z_{y};\bm{\mu}^{p}_{y},\lambda_{y}{\mathbf{I}})$,
rather than ${\mathcal{N}}(0,{\mathbf{I}})$. This allows us to associate its
mean with the features $\Phi(y)$ from label phrases through
$\bm{\mu}_{y}^{p}={\mathbf{W}}_{y}\Phi(l)$, where ${\mathbf{W}}_{y}$ is a
shared project matrix and $\Phi(y)$ is the mean contextual word embeddings of
label phrases in the zero-shot setting and the mean sentence embeddings using
the support set of a task in the few-shot setting. The resulting
regularization term is a L2 regularizer that enforces label representations to
be close to their prototype $\bm{\mu}_{y}^{p}$ and the logarithm of the
variance from the variation distribution.

${\mathcal{L}}_{y}(y)=-\frac{1}{\lambda_{y}}\|Z_{y}-\bm{\mu}_{y}^{p}\|^{2}+\log\bm{\sigma}_{y}^{q}$
(9)

In contrast, the corresponding regularization term in vanilla VAEs is largely
an L2 norm of $Z_{y}$. It is beneficial when adding new labels in the label
space, which is easily achieved by adding the corresponding Gaussians without
further training.

Regularization in the Content Subspace. In the content space, we derive the
regularization terms from
$\mathbb{D}_{\mathrm{KL}}(q_{\bm{\phi}}(Z_{c}|X,C)\|p_{\bm{\theta}}(Z_{c}|C))$
and $I(Z_{c},C)$, where $p_{\bm{\theta}}(Z_{c}|C)$ is a Gaussian mixture and
$q_{\bm{\phi}}(Z_{c}|X,C)$ is a simple factorized Gaussian. The key ideas are
twofold: i) utilize the Gaussian mixture to regularize the latent
representations generated by the content encoder; ii) as the content prior
contains a large amount of Gaussians, we can sample from all possible
component Gaussians for diverse contents. In contrast, vanilla VAE samples
only from a distribution with a single mode.

It is beneficial to use a $k$-means clusters as the condition $C$, where we
assume that there is a one-to-one correspondence between a Gaussian component
and a cluster in the $k$-means clusters. The rationale is that $k$-means
clusters based on sentence embeddings capture topic information (eg
semantic/syntactic/style) of texts. We expect the mean of a Gaussian component
$\mathcal{N}(Z_{c};\bm{\mu}^{p}_{c,k},\lambda_{c}{\mathbf{I}})$ is a mapping
from a certain cluster. Thus, we consider
$\bm{\mu}_{c,k}^{p}={\mathbf{W}}_{c}{\bm{c}}_{k}$, where ${\bm{c}}_{k}$ is the
centroid of a cluster $k$. During training, we keep centroids fixed and only
update their projection matrix ${\mathbf{W}}_{c}$. The $k$-means clusters are
built with BERT sentence embeddings on the source domain training data. Adding
new topics can be easily done by adding new clusters using incremental
clustering techniques.

From
$\mathbb{D}_{\mathrm{KL}}(q_{\bm{\phi}}(Z_{c}|X,C)\|p_{\bm{\theta}}(Z_{c}|C))$
and $I(Z_{c},C)$, we derive the following regularization term by using EM:

$\displaystyle\begin{split}{\mathcal{L}}_{c}=&\sum_{k=1}^{K}q_{\bm{\phi}}(M_{k}|X)\big{[}-\frac{1}{2\lambda_{c}}\|Z_{c}-\bm{\mu}_{c,k}^{p}\|^{2}\big{]}+\log\bm{\sigma}_{c}\end{split}$
(10)

The term $q_{\bm{\phi}}(M_{k}|X)$ is computed by the E-step of EM. If soft-EM
is considered, as in (Fard et al., 2020),
$q_{\bm{\phi}}(M_{k}|X)=\frac{\exp(-\text{dist}({\bm{z}}_{c},\bm{\mu}_{c,k}^{p})/\tau)}{\sum_{k^{\prime}}\exp(-\text{dist}({\bm{z}}_{c},\bm{\mu}_{c,k^{\prime}}^{p})/\tau)}$,
which denotes the probability of an example $X$ belonging to a cluster $k$
with a temperature $\tau$. In our experiments, we consider hard EM, where
$q_{\bm{\phi}}(M_{k}|X)$ is an indicator function characterizing if the
current Gaussian has the same index as the one having the minimal Euclidean
distance between its mean and a content representation ${\bm{z}}_{c}$ among
all components.

Inference. When evaluating with data augmentation tasks, for each new label,
we start with encoding its label phrase into a embedding using our label
encoder. In the few-shot setting, we also create a label embedding with the
label encoder by averaging the embeddings of text sequences in the support set
and the label phrase embedding. Then we sample a number of content embeddings
from the content prior $p_{\bm{\theta}}(Z_{c}|C)$. All combinations of label
embeddings and content embeddings are fed to the prefix encoder and the
decoder to generate word sequences. For quality control, we apply the label
encoder to each word sequence representation ${\mathbf{V}}$ generated by bert,
and measure the Euclidean distance between the outputs of the label encoder
and the corresponding label embeddings created before. We rank all generated
texts by using the distances and pick the top-$k$. For text style transfer, we
apply the same method to create label embeddings. Because the content
representations are curated directly by the content encoder, there is no need
for sampling from the content prior. The outputs are generated by feeding the
content representations and the embeddings of target styles to the prefix
encoder and the decoder.

##  4 Experiments

We evaluate our approach on continual few/zero shot text classification and
low-resource text style transfer.

###  4.1 Continual Zero/Few-shot Learning

Setting. We consider a continual few-shot learning setting similar as
(Antoniou et al., 2020). The text classification model
$\pi^{c}_{\theta}:\mathcal{X}\rightarrow\mathcal{Y}$ is trained sequentially
on $K$ distinct tasks {$\mathcal{T}^{(1)}$,
$\mathcal{T}^{(2)}$,…,$\mathcal{T}^{(K)}$}. The initial task
$\mathcal{T}^{(1)}$ includes the training and test set
$(\mathcal{D}_{train}^{(1)},\mathcal{D}_{test}^{(1)})$ while the succeed tasks
$\mathcal{T}^{(k\succ 1)}$ includes the support and test sets
$(\mathcal{D}_{sup}^{(k)},\mathcal{D}_{test}^{(k)})$, where we assume
$\mathcal{D}_{train}^{(1)}$ includes enough training data for each base class
while $\mathcal{D}_{sup}^{(k)}=\\{{\bm{x}},{\bm{y}}\\}_{i=1}^{N\times|C_{k}|}$
includes only $N$-shot instances per new class. The classes on
$\mathcal{T}^{(k)}$ are disjoint from classes of previous tasks,
$C_{1:k-1}\cap C_{k}=\emptyset$. As a conventional continual learning setting
as in (Lopez-Paz & Ranzato, 2017), a memory $\mathcal{M}_{k}$ is associated
with $\mathcal{T}^{(k)}$ to store a fixed number of training instances (either
examples selected from the support sets or the synthetic data) per each seen
task.

Upon arrival of each task, the classifier $\pi^{c}_{\theta}$ is trained on a
combined set of instances from the support set, memory set, and the augmented
examples generated by our generative model. To generate augmented examples, we
sample content from the fixed clustering obtained using the large training
data in $\mathcal{T}^{(1)}$, and sample labels from $C_{1:k}$. Note that for
our model, as long as we have the generative model and store the label
embeddings, we could regenerate examples from all old tasks. Therefore, the
memory is not necessary for our model. But for a fair comparison with other
baselines, we still assume there is fixed memory for each old task and use
only the examples from this memory to replay the classifier. We apply our data
augmentation method to EMAR (Han et al., 2020), a SOTA continual learning
approach for text classification. We follow EMAR (Han et al., 2020) for the
classifier architecture and how to update its parameters in continual
learning. We use Empathetic and TACRED for continual few-shot text
classification. Please refer to Appendix B.1 for more dataset details.

Evaluation. We use a widely adopted metric in continual learning, which
measures the performance by averaging the accuracies of the classifier on test
sets of all seen tasks
$\\{\mathcal{D}_{test}^{(1)}$,…,$\mathcal{D}_{test}^{(k)}\\}$, i.e.
$\text{ACC}_{\text{avg}}=\frac{1}{k}\sum^{k}_{i=1}acc_{i}$ (Lopez-Paz &
Ranzato, 2017). In addition, to measure the diversity of generated examples,
we calculate the average similarity scores between all pairs of examples
within each label, i.e.
$\frac{1}{|\mathcal{Y}|}\sum_{i,j}sim({\bm{x}}_{i},{\bm{x}}_{j})\mathbbm{1}[\operatorname*{arg\,max}p(y|{\bm{x}}_{i})==\operatorname*{arg\,max}p(y|{\bm{x}}_{j})]$,
where we use BLEU (Papineni et al., 2002) and word mover distance (WMD)
(Kusner et al., 2015) as the similarity functions. The lower scores indicate
more diversified examples within each label.

Baselines. We compare five data augmentation baselines: i) EDA (Wei & Zou,
2019) randomly deletes, substitutes, inserts or swaps words in the original
sentences. ii) CBERT (Wu et al., 2019) uses BERT to determine the position to
insert or substitute words. iii) RTT (Sennrich et al., 2015) augments datasets
by generating the paraphrases of the original sentences through round-trip
translation, iv) LAMBDA (Kumar et al., 2020) trains a GPT2 to generate
examples conditioned on the label text and uses a classifier to filter out
low-quality examples as in our work. v) Ex2 (Lee et al., 2021) applies T5
(Raffel et al., 2020) and extrapolation technique to increase the diversity of
generated examples and deal with the low-resource setting. vi) Optimus (Li et
al., 2020) is our backbone model which is in an auto-encoder framework that
uses BERT as the encoder and GPT2 (Radford et al., 2019) as the decoder. vii)
Casual-Lens (Hu & Li, 2021) improves the training of Optimus using an
intervention and a counterfactual losses. Both Optimus and Casual-Lens are
designed for controllable text generation. We use them for data augmentation
by assessing their ability for label-conditional generation.

Main Results and Discussions.

Methods | TACRED | Empathetic
---|---|---
0 | 1 | 5 | 0 | 1 | 5
No Aug | 11.21 | 22.02 | 36.87 | - | 14.20 | 24.07
EDA | - | 20.64 | 32.83 | - | 13.72 | 20.16
CBERT | - | 20.21 | 35.43 | - | 13.82 | 21.52
RTT | - | 24.23 | 33.60 | - | 14.06 | 20.37
LAMBDA | 16.23 | 20.16 | 32.14 | 13.39 | 14.45 | 21.95
Ex2 | 15.57 | 19.83 | 32.87 | 11.91 | 16.82 | 24.75
Optimus | 17.12 | 19.99 | 28.77 | 9.93 | 14.32 | 18.21
Casual-Lens | 9.83 | 17.17 | 25.76 | 9.72 | 11.72 | 16.49
VAE-DPrior | 28.49 | 37.43 | 42.30 | 18.16 | 24.38 | 33.86

Table 1: The $\text{ACC}_{\text{avg}}$ of the classifier across the tasks with
different data augmentation methods. ’-’ indicates that zero-shot is not
applicable to the corresponding augmentation methods.

Datasets | Metrics | EDA | CBERT | RTT | LAMBDA | Ex2 | Optimus | Casual-Lens | VAE-DPrior
---|---|---|---|---|---|---|---|---|---
TACRED | BLEU$\downarrow$ | 38.24 | 28.12 | 96.83 | 95.85 | 26.61 | 47.04 | 18.78 | 4.90
WMD$\downarrow$ | 97.91 | 96.83 | 99.93 | 99.28 | 83.24 | 94.94 | 88.24 | 86.36
Empathetic | BLEU$\downarrow$ | 31.64 | 24.97 | 96.79 | 74.82 | 44.55 | 55 12 | 9.46 | 7.20
WMD$\downarrow$ | 97.89 | 96.85 | 99.92 | 97.20 | 94.57 | 98.15 | 79.10 | 93.87

Table 2: The diversity scores of the generated examples measured with BLEU and
WMD on one-shot learning.

The results in Table 1 show that our approach VAE-DPrior outperforms all data
augmentation baselines on all zero/few-shot learning settings by significant
margins. The augmentation approaches such as EDA, CBERT and RTT generate
adversarial examples of the original sentences via manipulation of words or
paraphrasing. However, adversarial distributions are not the same as the true
distribution, thus their generated examples do not improve the continual
learning classifier. They even degrade the performance in the five-shot
setting in comparison to the classifier performance without data augmentation.

Although LAMBDA, Ex2, Optimus and Casual-Lens aim to learn the true
distribution from labeled data, we observe that they often fail to associate
generated texts with correct labels, especially for new tasks. Thus, their
performance cannot be improved given more labeled examples of new tasks. In
contrast, VAE-DPrior achieves a significantly higher degree of compositional
generalization across tasks, evident by high average accuracy of the
classifier trained on its generated examples. The performance of the
classifier further improves when our model is fed with more labeled examples
of new tasks.

We also evaluated diversity of the generated examples of all augmentation
methods in the one-shot setting, presented in Table 2. The generated sentences
from EDA, CBERT and RTT are mostly paraphrases of the original sentences.
Therefore, they cannot significantly diversify examples. LAMBDA generates data
examples conditioning on the label phrases and the first $k$ words of the
original sentence, which also lacks diversity. Ex2 enriches the diversity by
extrapolating the novel samples from the existing sentences within the novel
labels. Optimus and Casual-Lens employ a GAN (Goodfellow et al., 2014) and a
conditional GAN (Mirza & Osindero, 2014) respectively to generate diversified
latent vectors for the generation of examples with the novel labels. However,
with merely one or five sentences per label, such methods only generate a
small sample of texts with novel labels. In contrast, VAE-DPrior can composite
all seen content representations acquired from the past with representations
of new labels to generate high-quality sentences.

###  4.2 Few-shot Text-style Transfer

Setting. We follow the common non-parallel text style transfer setting as in
(Nangi et al., 2021), where each text sample ${\bm{x}}$ is associated with a
style label ${\bm{y}}$. In the few-shot setting, the style transfer model
$\pi^{s}_{\theta}:\mathcal{X}\rightarrow\mathcal{X}^{\prime}$ is pre-trained
on a training set, $\mathcal{D}_{train}$, which includes abundant training
data (e.g. more than 50 instances per style) for each base style $C_{b}$.
After pre-training, the model parameters would be frozen with only the label
embeddings updated based on a support set,
$\mathcal{D}_{sup}=\\{{\bm{x}},{\bm{y}}\\}_{i=1}^{N\times|C_{n}|}$, which
includes only $N$-shot instances for each one of $|C_{n}|$ novel styles.
Although VAE-DPrior can be easily fine-tuned on the support sets, we found the
fine-tuning brings negligible performance gain in the few-shot setting. The
test set $\mathcal{D}_{test}$ is sampled from both $\mathcal{D}_{train}$ and a
corpus $\mathcal{D}_{train}^{\prime}$ which is from the same distribution of
$\mathcal{D}_{train}$. The style transfer task is to transfer text in
$\mathcal{D}_{test}$ into the styles of $C_{n}$ in $\mathcal{D}_{sup}$. We use
two datasets, Empathetic (Rashkin et al., 2018) and Personality (Shuster et
al., 2019) for style transfer; please see Appendix B.2 for more details.

Evaluation. We use three automatic metrics, Style-transfer Accuracy, Self-WMD
and Perplexity, to evaluate the accuracy of style transfer, semantic relevance
and naturalness of the generated text, respectively. For Style-transfer
Accuracy, we train a BERT (Devlin et al., 2018) classifier on styles. The
averaged accuracy on target labels indicates the correctness of style
transfer. Self-WMD (Kusner et al., 2015) measures WMD between the original
text and the transferred text. Perplexity is estimated by a statistical
language model in English released by (Koehn et al., 2007)
111https://www.statmt.org/moses/RELEASE-4.0/models/cs-en/lm/.

Baselines. We compare five style transfer baselines: i) R-VAE-AVG (John et
al., 2018a) learns the disentangled label and content representations. ii)
R-VAE-CF (Nangi et al., 2021), on the base of R-VAE-AVG, uses a counterfactual
reasoning module to control the generation of label representations. iii) ZF
(Smith et al., 2019) is a back-translation model, which aims to deal with the
zero-shot text transfer problem. The two controllable text generation
baselines used in the continual few-shot setting, iv) Optimus and v) Casual-
Lens, are extended for style transfer as well. Please refer to the Appendix A
and their original work for the detailed style transfer implementation.

Methods | Empathetic | Personality
---|---|---
Style Accuracy$\uparrow$ | Self-WMD$\uparrow$ | Perplexity$\downarrow$ | Style Accuracy$\uparrow$ | Self-WMD$\uparrow$ | Perplexity$\downarrow$
R-VAE-AVG | 28.49 | 90.19 | 627.49 | 27.73 | 86.06 | 796.99
R-VAE-CF | 26.85 | 90.82 | 657.22 | 19.69 | 87.25 | 850.86
ZF | 25.58 | 90.05 | 943.56 | 20.92 | 83.37 | 765.58
Optimus | 27.54 | 94.03 | 829.43 | 18.25 | 94.96 | 919.58
Casual-Lens | 39.27 | 88.74 | 1157.28 | 20.52 | 89.62 | 1059.17
VAE-DPrior | 32.67 | 95.69 | 236.20 | 49.58 | 93.44 | 311.07

Table 3:  The results of one-shot style-transfer on both datasets.

Main Results and Discussions.

The results in Table 3 show that our method performs better than all baselines
in terms of all metrics except Style Accuracy on Empathetic and Self-WMD on
Personality. An ideal style transfer model should find a good balance in terms
of all three evaluation metrics. Though Casual-Lens and Optimus can achieve
the best on a single metric, they fail to perform well across all the metrics.
We inspect that Casual-Lens performs poorly on preserving the content of the
original sentence while Optimus performs poor on style transfer and basically
replicates the original sentences in Personality dataset. In contrast, VAE-
DPrior performs well across all metrics. The average ranking of VAE-DPrior
among all baselines on three metrics are 1.33 while Casual-Lens and Optimus
are only 4.33 and 3.33, respectively. Our model performs particularly well in
terms of semantic relevance and naturalness while still keeping high
accuracies of style transfer. Other methods that utilize disentanglement
learning, including R-VAE-AVG, R-VAE-CF and Casual-Lens, often perform well on
one metric while lose on the other metrics. We conjecture this is due to their
methods do not fully disentangle the representations so they can not balance
well between content preservation and style transfer.

###  4.3 Ablation Study

#### Disentanglement.

We evaluate the effectiveness of the disentanglement priors by comparing our
VAE-DPrior against the other regularizers, including MMD, GAN, HSIC and IDEL
(Cheng et al., 2020). As in Tab. 4, except for MMD, the other disentanglement
regularizers bring almost no improvement to VAE-DPrior. HSIC and GAN that
enforce independence between latent variables even hurt the performance. We
observe that the GAN-based regularizer causes mode collapse, because VAE-
DPrior with GAN tends to generate similar examples. In contrast, if we apply
the regularizers to the other types of VAEs, such as vanilla VAE, they indeed
lead to improved performance. We conjecture that the two conditional priors
alone without any disentanglement regularizers can already generate different
representations for different latent subspaces. To verify that, we apply t-SNE
to plot latent embeddings generated by VAE-DPrior. As shown in the Fig. 2 of
Appendix, the content subspace is not overlapped with the label space, and the
region of each label is also far from each other. Hence, the embeddings are
strictly disentangled.

Methods | TACRED | Empathetic
---|---|---
0 | 1 | 5 | 0 | 1 | 5
VAE-DPrior \+ no dis | 29.21 | 37.18 | 43.34 | 17.79 | 22.43 | 32.27
\+ mmd | 28.49 | 37.43 | 42.30 | 18.16 | 24.38 | 33.86
\+ gan | 25.72 | 27.97 | 35.98 | 14.41 | 17.65 | 24.17
\+ hsic | 28.86 | 28.87 | 38.62 | 16.86 | 22.24 | 27.47
\+ idel | 29.79 | 36.10 | 43.08 | 16.12 | 22.00 | 32.58

Table 4:  The $\text{ACC}_{\text{avg}}$ of VAE-DPrior with different
disentanglement losses in zero/few-shot learning.

#### Mode and Posterior Collapse.

Methods | TACRED | Empathetic
---|---|---
$\text{ACC}_{\text{avg}}\uparrow$ | BLEU$\downarrow$ | WMD$\downarrow$ | $\text{ACC}_{\text{avg}}\uparrow$ | BLEU$\downarrow$ | WMD$\downarrow$
VAE-DPrior | 37.43 | 4.90 | 86.37 | 24.38 | 7.20 | 93.87
-/+ vae-dprior (no cond) | 25.49 | 5.37 | 89.17 | 19.18 | 5.06 | 93.57
-/+ vae-dprior (gauss) | 37.05 | 4.46 | 85.06 | 20.33 | 5.22 | 91.13
-/+ autoencoder | 30.80 | 4.76 | 88.80 | 17.18 | 7.11 | 95.90
-/+ vae | 29.72 | 30.15 | 91.54 | 15.06 | 8.14 | 90.16
-/+ vq-vae | 23.76 | 20.06 | 91.48 | 20.24 | 24.92 | 95.36
-/+ c-vae | 27.79 | 12.38 | 89.94 | 22.80 | 17.83 | 94.20
-/+ vamp-vae | 21.18 | 74.62 | 97.48 | 16.86 | 66.59 | 98.28

Table 5:  The $\text{ACC}_{\text{avg}}$ and diversity scores of the NLG model
with different VAE frameworks on one-shot learning.

Classical VAEs, such as vanilla VAE, suffer from a notorious problem called
posterior collapse (Wang et al., 2020). Those models learn non-injective
mappings between latent variable values and the likelihoods, thus prevents
them from generating diversified examples. We study the problems of mode
collapse and posterior collapse by comparing VAE-DPrior with Vanilla Seq2seq
Auto-encoder,VAE, VQ-VAE (Oord et al., 2017), Categorical-VAE (Jang et al.,
2016) and Vamp-VAE (Tomczak & Welling, 2018). The results in Table 5 show that
VAE-DPrior largely outperforms other types VAEs. Our VAE-DPrior model
alleviates this problem by using a conditional prior for each subspace. It
clearly outperforms Vamp-VAE, which also introduces conditional priors. Our
method differs from Vamp-VAE by introducing priors for strictly disentangled
representations. Auto-encoder does not use any regularizers that lead to
posterior collapse, thus performs better than those VAEs from prior work. To
verify if the conditional Gaussian mixture prior mitigates mode collapse, we
replace it with a Gaussian prior conditioned on the mean sentence embeddings
of the training data (vae-dprior (gauss)) and with a Gaussian mixture with
randomly initialized and trainable means (vae-dprior (no cond)). In both
cases, the variations suffer from a significant performance drop.

Posterior collapse is caused by non-injectivity of likelihood functions. Thus
we inspect the ratio of duplicated outputs for each model by feeding them with
diverse latent variable values. The latent representations are randomly
sampled from their latent priors. We found c-vae, vq-vae and vamp-vae suffer
seriously from non-injectivity with a duplicity ratio 15.27%, 37.47% and
72.42%, respectively. vae has 0.6% duplicated outputs while the variations of
our model vae-dprior (no cond), vae-dprior (gauss) and vae-dprior all learn
one-to-one mappings between latent representations and the likelihood values.

#### Quality Control.

Methods | TACRED | Empathetic
---|---|---
$\text{ACC}_{\text{avg}}\uparrow$ | BLEU$\downarrow$ | WMD$\downarrow$ | $\text{ACC}_{\text{avg}}\uparrow$ | BLEU$\downarrow$ | WMD$\downarrow$
VAE-DPrior \+ bert + encoder | 37.43 | 4.90 | 86.37 | 24.38 | 7.20 | 93.87
-/+ random | 24.21 | 3.81 | 80.75 | 21.03 | 4.64 | 85.68
-/+ perplexity | 23.32 | 4.72 | 80.98 | 16.59 | 4.81 | 73.94
-/+ bert | 33.21 | 3.96 | 81.81 | 20.40 | 7.81 | 75.77

Table 6:  The $\text{ACC}_{\text{avg}}$ and diversity scores of VAE-DPrior
with different quality control methods on one-shot learning.

We compare our inference method (+bert + encoder) with three alternative
methods: i) only use a pre-trained BERT to encode each output text into an
embedding with mean pooling and compare it with the average BERT embeddings of
labels and support sets ( +bert); ii) select the examples with the lowest
perplexity calculated by GPT2 ( +perplexity); and iii) randomly select $k$
outputs ( + random). Table 6 shows that different methods indeed influence the
final performance of the data augmentation. Although the BLEU and WMD metrics
show that baseline filtering methods all increase the diversity, the quality
of selected examples is actually lower. But even with the worst performing
filtering method, ( +perplexity), our method can still outperform other
baselines on both datasets. Another finding is that the encoder trained with
label condition prior performs much better than its backbone model, ( +bert),
in terms of selecting high quality examples, proving that the label prior
condition could help the encoder generalize well on the novel labels.

##  5 Conclusion

In this work, we propose the first VAE model with disentanglement priors for
disentangled representation learning in low resource NLG tasks. This model
achieves compositional generalization by effectively combining rich content
representations sampled from conditional priors and task-specific
representations for new tasks. Its empirical performance outperforms the
baselines of continual zero-shot/few-shot learning and few-shot text style
transfer with a wide margin.

## References

  * Antoniou et al. (2020) Antoniou, A., Patacchiola, M., Ochal, M., and Storkey, A.  Defining benchmarks for continual few-shot learning.  _arXiv preprint arXiv:2004.11967_ , 2020.
  * Burgess et al. (2018) Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A.  Understanding disentangling in beta-vae.  _arXiv preprint arXiv:1804.03599_ , 2018.
  * Cheng et al. (2020) Cheng, P., Min, M. R., Shen, D., Malon, C., Zhang, Y., Li, Y., and Carin, L.  Improving disentangled text representation learning with information-theoretic guidance.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7530–7541, 2020.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Fard et al. (2020) Fard, M. M., Thonet, T., and Gaussier, E.  Deep k-means: Jointly clustering with k-means and learning representations.  _Pattern Recognition Letters_ , 138:185–192, 2020.
  * Floridi & Chiriatti (2020) Floridi, L. and Chiriatti, M.  Gpt-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ , 30(4):681–694, 2020.
  * Fodor & Pylyshyn (1988) Fodor, J. A. and Pylyshyn, Z. W.  Connectionism and cognitive architecture: A critical analysis.  _Cognition_ , 28(1-2):3–71, 1988.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gretton et al. (2012) Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A.  A kernel two-sample test.  _The Journal of Machine Learning Research_ , 13(1):723–773, 2012.
  * Han et al. (2020) Han, X., Dai, Y., Gao, T., Lin, Y., Liu, Z., Li, P., Sun, M., and Zhou, J.  Continual relation learning via episodic memory activation and reconsolidation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 6429–6440, 2020.
  * Higgins et al. (2018) Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A.  Towards a definition of disentangled representations.  _arXiv preprint arXiv:1812.02230_ , 2018.
  * Hu & Li (2021) Hu, Z. and Li, L. E.  A causal lens for controllable text generation.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Jang et al. (2016) Jang, E., Gu, S., and Poole, B.  Categorical reparameterization with gumbel-softmax.  _arXiv preprint arXiv:1611.01144_ , 2016.
  * John et al. (2018a) John, V., Mou, L., Bahuleyan, H., and Vechtomova, O.  Disentangled representation learning for non-parallel text style transfer.  _arXiv preprint arXiv:1808.04339_ , 2018a.
  * John et al. (2018b) John, V., Mou, L., Bahuleyan, H., and Vechtomova, O.  Disentangled representation learning for non-parallel text style transfer.  _arXiv preprint arXiv:1808.04339_ , 2018b.
  * Kingma & Welling (2019) Kingma, D. P. and Welling, M.  An introduction to variational autoencoders.  _arXiv preprint arXiv:1906.02691_ , 2019.
  * Koehn et al. (2007) Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions_ , pp. 177–180, 2007.
  * Kumar et al. (2020) Kumar, V., Choudhary, A., and Cho, E.  Data augmentation using pre-trained transformer models.  _arXiv preprint arXiv:2003.02245_ , 2020.
  * Kusner et al. (2015) Kusner, M., Sun, Y., Kolkin, N., and Weinberger, K.  From word embeddings to document distances.  In _International conference on machine learning_ , pp. 957–966. PMLR, 2015.
  * Lee et al. (2021) Lee, K., Guu, K., He, L., Dozat, T., and Chung, H. W.  Neural data augmentation via example extrapolation.  _arXiv preprint arXiv:2102.01335_ , 2021.
  * Li et al. (2020) Li, C., Gao, X., Li, Y., Peng, B., Li, X., Zhang, Y., and Gao, J.  Optimus: Organizing sentences via pre-trained modeling of a latent space.  _arXiv preprint arXiv:2004.04092_ , 2020.
  * Li & Liang (2021) Li, X. L. and Liang, P.  Prefix-tuning: Optimizing continuous prompts for generation.  _arXiv preprint arXiv:2101.00190_ , 2021.
  * Liu et al. (2020a) Liu, D., Yan, Y., Gong, Y., Qi, W., Zhang, H., Jiao, J., Chen, W., Fu, J., Shou, L., Gong, M., et al.  Glge: A new general language generation evaluation benchmark.  _arXiv preprint arXiv:2011.11928_ , 2020a.
  * Liu et al. (2020b) Liu, H., Wang, W., Wang, Y., Liu, H., Liu, Z., and Tang, J.  Mitigating gender bias for neural dialogue generation with adversarial learning.  _arXiv preprint arXiv:2009.13028_ , 2020b.
  * Lopez-Paz & Ranzato (2017) Lopez-Paz, D. and Ranzato, M.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30:6467–6476, 2017.
  * Mirza & Osindero (2014) Mirza, M. and Osindero, S.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Mita et al. (2021) Mita, G., Filippone, M., and Michiardi, P.  An identifiable double vae for disentangled representations.  In _International Conference on Machine Learning_ , pp. 7769–7779. PMLR, 2021.
  * Nangi et al. (2021) Nangi, S. R., Chhaya, N., Khosla, S., Kaushik, N., and Nyati, H.  Counterfactuals to control latent disentangled text representations for style transfer.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 40–48, 2021.
  * Oord et al. (2017) Oord, A. v. d., Vinyals, O., and Kavukcuoglu, K.  Neural discrete representation learning.  _arXiv preprint arXiv:1711.00937_ , 2017.
  * Papineni et al. (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Prabhumoye et al. (2020) Prabhumoye, S., Black, A. W., and Salakhutdinov, R.  Exploring controllable text generation techniques.  _arXiv preprint arXiv:2005.01822_ , 2020.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67, 2020\.
  * Rashkin et al. (2018) Rashkin, H., Smith, E. M., Li, M., and Boureau, Y.-L.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  _arXiv preprint arXiv:1811.00207_ , 2018.
  * Romanov et al. (2018) Romanov, A., Rumshisky, A., Rogers, A., and Donahue, D.  Adversarial decomposition of text representation.  _arXiv preprint arXiv:1808.09042_ , 2018.
  * Sennrich et al. (2015) Sennrich, R., Haddow, B., and Birch, A.  Improving neural machine translation models with monolingual data.  _arXiv preprint arXiv:1511.06709_ , 2015.
  * Shin et al. (2020) Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  _arXiv preprint arXiv:2010.15980_ , 2020.
  * Shuster et al. (2019) Shuster, K., Humeau, S., Hu, H., Bordes, A., and Weston, J.  Engaging image captioning via personality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12516–12526, 2019.
  * Smith et al. (2019) Smith, E. M., Gonzalez-Rico, D., Dinan, E., and Boureau, Y.-L.  Zero-shot fine-grained style transfer: Leveraging distributed continuous style representations to transfer to unseen styles.  _arXiv preprint arXiv:1911.03914_ , 2019.
  * Tomczak & Welling (2018) Tomczak, J. and Welling, M.  Vae with a vampprior.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 1214–1223. PMLR, 2018.
  * Tran et al. (2017) Tran, L., Yin, X., and Liu, X.  Disentangled representation learning gan for pose-invariant face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1415–1424, 2017.
  * Wang et al. (2019) Wang, C., Li, M., and Smola, A. J.  Language models with transformers.  _arXiv preprint arXiv:1904.09408_ , 2019.
  * Wang et al. (2020) Wang, Y., Blei, D., and Cunningham, J. P.  Posterior collapse and latent variable non-identifiability.  In _Third Symposium on Advances in Approximate Bayesian Inference_ , 2020.
  * Wei & Zou (2019) Wei, J. and Zou, K.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  _arXiv preprint arXiv:1901.11196_ , 2019.
  * Wei et al. (2021) Wei, Y., Shi, Y., Liu, X., Ji, Z., Gao, Y., Wu, Z., and Zuo, W.  Orthogonal jacobian regularization for unsupervised disentanglement in image generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 6721–6730, 2021.
  * Wu et al. (2019) Wu, X., Lv, S., Zang, L., Han, J., and Hu, S.  Conditional bert contextual augmentation.  In _International Conference on Computational Science_ , pp. 84–95. Springer, 2019.
  * Yuan et al. (2020) Yuan, S., Cheng, P., Zhang, R., Hao, W., Gan, Z., and Carin, L.  Improving zero-shot voice style transfer via disentangled representation learning.  In _International Conference on Learning Representations_ , 2020.


* [1] S. Grimm, Knowledge representation and ontologies, in: M. M. Gaber (Ed.), Scientific Data Mining and Knowledge Discovery - Principles and Foundations, Springer, 2010, pp. 111–137.  doi:10.1007/978-3-642-02788-8_6.
  * [2] G. Ciatto, F. Sabbatini, A. Agiollo, M. Magnini, A. Omicini, Symbolic knowledge extraction and injection with sub-symbolic predictors: A systematic literature review, ACM Computing Surveys 56 (6) (2024) 161:1–161:35.  doi:10.1145/3645103.
  * [3] G. Petasis, V. Karkaletsis, G. Paliouras, A. Krithara, E. Zavitsanos, Ontology population and enrichment: State of the art, in: G. Paliouras, C. D. Spyropoulos, G. Tsatsaronis (Eds.), Knowledge-Driven Multimedia Information Extraction and Ontology Evolution - Bridging the Semantic Gap, Vol. 6050 of Lecture Notes in Computer Science, Springer, 2011, pp. 134–166.  doi:10.1007/978-3-642-20795-2_6.
  * [4] F. Baader, I. Horrocks, U. Sattler, Description logics, in: F. van Harmelen, V. Lifschitz, B. W. Porter (Eds.), Handbook of Knowledge Representation, Vol. 3 of Foundations of Artificial Intelligence, Elsevier, 2008, pp. 135–179.  doi:10.1016/S1574-6526(07)03003-9.
  * [5] T. Berners-Lee, J. Hendler, O. Lassila, The semantic web, Scientific American 284 (5) (2001) 34–43.
URL
http://www.sciam.com/article.cfm?articleID=00048144-10D2-1C70-84A9809EC588EF21

  * [6] M. A. Musen, The Protégé project: a look back and a look forward, AI Matters 1 (4) (2015) 4–12.  doi:10.1145/2757001.2757003.
  * [7] J. Lamy, Owlready: Ontology-oriented programming in python with automatic classification and high level constructs for biomedical ontologies, Artif. Intell. Medicine 80 (2017) 11–28.  doi:10.1016/J.ARTMED.2017.07.002.
  * [8] M. Lubani, S. A. M. Noah, R. Mahmud, Ontology population: Approaches and design aspects, J. Inf. Sci. 45 (4) (2019).  doi:10.1177/0165551518801819.
  * [9] A. C. Khadir, H. Aliane, A. Guessoum, Ontology learning: Grand tour and challenges, Comput. Sci. Rev. 39 (2021) 100339.  doi:10.1016/j.cosrev.2020.100339.
  * [10] F. N. Al-Aswadi, C. H. Yong, K. H. Gan, Automatic ontology construction from text: a review from shallow to deep learning trend, Artif. Intell. Rev. 53 (6) (2020) 3901–3928.  doi:10.1007/s10462-019-09782-9.
  * [11] M. Finkelstein-Landau, E. Morin, Extracting semantic relationships between terms: Supervised vs. unsupervised methods, in: International Workshop on Ontological Engineering on the Global Information Infrastructure, Citeseer, 1999, pp. 71–80.
  * [12] E. Morin, Automatic acquisition of semantic relations between terms from technical corpora, in: Proc. of the Fifth International Congress on Terminology and Knowledge Engineering-TKE’99, 1999.
  * [13] A. Harith, S. Kim, D. E. Millard, M. J. Weal, W. Hall, P. H. Lewis, N. R. Shadbolt, Automatic ontology-based knowledge extraction and tailored biography generation from the web, IEEE Intelligent Systems 18 (1) (2003) 14–21.
  * [14] H. Tanev, B. Magnini, Weakly supervised approaches for ontology population, in: D. McCarthy, S. Wintner (Eds.), EACL 2006, 11st Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference, April 3-7, 2006, Trento, Italy, The Association for Computer Linguistics, 2006\.
URL https://aclanthology.org/E06-1003/

  * [15] H. Yoon, Y. Han, S. Park, S. Park, Ontology population from unstructured and semi-structured texts, in: Proceedings of The Sixth International Conference on Advanced Language Processing and Web Information Technology, ALPIT 2007, Luoyang, Henan, China, 22-24 August 2007, IEEE Computer Society, 2007, pp. 135–139.  doi:10.1109/ALPIT.2007.30.
  * [16] D. Maynard, Y. Li, W. Peters, NLP techniques for term extraction and ontology population, in: P. Buitelaar, P. Cimiano (Eds.), Ontology Learning and Population: Bridging the Gap between Text and Knowledge, Vol. 167 of Frontiers in Artificial Intelligence and Applications, IOS Press, 2008, pp. 107–127.
URL https://ebooks.iospress.nl/volume/proof-technology-and-computation

  * [17] D. Celjuska, M. Vargas-Vera, Ontosophie: A semi-automatic system for ontology population from text, in: International Conference on Natural Language Processing (ICON), Vol. 60, 2004.
  * [18] O. Etzioni, M. J. Cafarella, D. Downey, A. Popescu, T. Shaked, S. Soderland, D. S. Weld, A. Yates, Unsupervised named-entity extraction from the web: An experimental study, Artif. Intell. 165 (1) (2005) 91–134.  doi:10.1016/j.artint.2005.03.001.
  * [19] M. Jiang, et al., A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries, J. Am. Medical Informatics Assoc. 18 (5) (2011) 601–606.  doi:10.1136/amiajnl-2011-000163.
  * [20] A. Souili, D. Cavallucci, F. Rousselot, Natural language processing (nlp)–a solution for knowledge extraction from patent unstructured data, Procedia engineering 131 (2015) 635–643.
  * [21] C. Liu, W. Sun, W. Chao, W. Che, Convolution neural network for relation extraction, in: H. Motoda, Z. Wu, L. Cao, O. R. Zaïane, M. Yao, W. Wang (Eds.), Advanced Data Mining and Applications - 9th International Conference, ADMA 2013, Hangzhou, China, December 14-16, 2013, Proceedings, Part II, Vol. 8347 of Lecture Notes in Computer Science, Springer, 2013, pp. 231–242.  doi:10.1007/978-3-642-53917-6_21.
  * [22] D. Zeng, K. Liu, S. Lai, G. Zhou, J. Zhao, Relation classification via convolutional deep neural network, in: J. Hajic, J. Tsujii (Eds.), COLING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland, ACL, 2014, pp. 2335–2344.
URL https://aclanthology.org/C14-1220/

  * [23] A. Ayadi, A. Samet, F. de Bertrand de Beuvron, C. Zanni-Merk, Ontology population with deep learning-based NLP: a case study on the biomolecular network ontology, in: I. J. Rudas, J. Csirik, C. Toro, J. Botzheim, R. J. Howlett, L. C. Jain (Eds.), Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 23rd International Conference KES-2019, Budapest, Hungary, 4-6 September 2019, Vol. 159 of Procedia Computer Science, Elsevier, 2019, pp. 572–581.  doi:10.1016/j.procs.2019.09.212.
  * [24] J. Devlin, M. Chang, K. Lee, K. Toutanova, BERT: pre-training of deep bidirectional transformers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Association for Computational Linguistics, 2019, pp. 4171–4186.  doi:10.18653/v1/n19-1423.
  * [25] T. B. Brown, et al., Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
URL
https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

  * [26] B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, D. Roth, Recent advances in natural language processing via large pre-trained language models: A survey, ACM Computing Surveys 56 (2) (sep 2023).  doi:10.1145/3605943.
  * [27] Y. Zhu, R. Kiros, R. S. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, S. Fidler, Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, in: 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, IEEE Computer Society, 2015, pp. 19–27.  doi:10.1109/ICCV.2015.11.
  * [28] N. Carlini, et al., Extracting training data from large language models, in: M. Bailey, R. Greenstadt (Eds.), 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, USENIX Association, 2021, pp. 2633–2650.
URL https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-
extracting

  * [29] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf (2019).
  * [30] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019).
URL http://arxiv.org/abs/1907.11692

  * [31] A. Gokaslan, V. Cohen, Openwebtext corpus, http://Skylion007.github.io/OpenWebTextCorpus (2019).
  * [32] T. H. Trinh, Q. V. Le, A simple method for commonsense reasoning, CoRR abs/1806.02847 (2018).
URL http://arxiv.org/abs/1806.02847

  * [33] H. Touvron, et al., Llama: Open and efficient foundation language models, CoRR abs/2302.13971 (2023).  doi:10.48550/ARXIV.2302.13971.
  * [34] H. Touvron, et al., Llama 2: Open foundation and fine-tuned chat models, CoRR abs/2307.09288 (2023).  doi:10.48550/ARXIV.2307.09288.
  * [35] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, J. Mach. Learn. Res. 21 (2020) 140:1–140:67.
URL http://jmlr.org/papers/v21/20-074.html

  * [36] Y. Zhang, et al., Siren’s song in the AI ocean: A survey on hallucination in large language models, CoRR abs/2309.01219 (2023).  doi:10.48550/arXiv.2309.01219.
  * [37] L. Memmert, I. Cvetkovic, E. A. C. Bittner, The more is not the merrier: Effects of prompt engineering on the quality of ideas generated by GPT-3, in: T. X. Bui (Ed.), 57th Hawaii International Conference on System Sciences, HICSS 2024, Hilton Hawaiian Village Waikiki Beach Resort, Hawaii, USA, January 3-6, 2024, ScholarSpace, 2024, pp. 7520–7529.
URL https://hdl.handle.net/10125/107289

  * [38] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, X. Chen, Large language models as optimizers, CoRR abs/2309.03409 (2023).  doi:10.48550/ARXIV.2309.03409.
  * [39] J. Wei, et al., Emergent abilities of large language models, Trans. Mach. Learn. Res. 2022 (2022).
URL https://openreview.net/forum?id=yzkSU5zdwD

  * [40] R. Schaeffer, B. Miranda, S. Koyejo, Are emergent abilities of large language models a mirage?, CoRR abs/2304.15004 (2023).  doi:10.48550/ARXIV.2304.15004.
  * [41] Z. Wu, L. Qiu, A. Ross, E. Akyürek, B. Chen, B. Wang, N. Kim, J. Andreas, Y. Kim, Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks, CoRR abs/2307.02477 (2023).  doi:10.48550/ARXIV.2307.02477.
  * [42] D. Calvaresi, G. Ciatto, A. Najjar, R. Aydoğan, L. Van der Torre, A. Omicini, M. I. Schumacher, Expectation: Personalized explainable artificial intelligence for decentralized agents with heterogeneous knowledge, in: D. Calvaresi, A. Najjar, M. Winikoff, K. Främling (Eds.), Explainable and Transparent AI and Multi-Agent Systems. Third International Workshop, EXTRAAMAS 2021, Virtual Event, May 3–7, 2021, Revised Selected Papers, Vol. 12688 of Lecture Notes in Computer Science, Springer Nature, Basel, Switzerland, 2021, pp. 331–343.  doi:10.1007/978-3-030-82017-6_20.
  * [43] M. Magnini, G. Ciatto, F. Cantürk, R. Aydoǧan, A. Omicini, Symbolic knowledge extraction for explainable nutritional recommenders, Computer Methods and Programs in Biomedicine 235 (Jun. 2023).  doi:10.1016/j.cmpb.2023.107536.
  * [44] G. Wang, S. Cheng, X. Zhan, X. Li, S. Song, Y. Liu, Openchat: Advancing open-source language models with mixed-quality data, arXiv preprint arXiv:2309.11235 (2023).
  * [45] A. Q. Jiang, et al., Mistral 7b, CoRR abs/2310.06825 (2023).  doi:10.48550/ARXIV.2310.06825.
  * [46] R. Anil, et al., Gemini: A family of highly capable multimodal models, CoRR abs/2312.11805 (2023).  doi:10.48550/ARXIV.2312.11805.
  * [47] A. Q. Jiang, et al., Mixtral of experts, CoRR abs/2401.04088 (2024).  doi:10.48550/ARXIV.2401.04088.
  * [48] OpenAI, GPT-4 technical report, CoRR abs/2303.08774 (2023).  doi:10.48550/ARXIV.2303.08774.
  * [49] M. Funk, S. Hosemann, J. C. Jung, C. Lutz, Towards ontology construction with language models (2023).  arXiv:2309.09898.
  * [50] L. Chen, M. Zaharia, J. Zou, How is chatgpt’s behavior changing over time?, CoRR abs/2307.09009 (2023).  doi:10.48550/ARXIV.2307.09009.
  * [51] M. Eurich, M. J. Burtscher, The business-to-consumer lock-in effect, 2014.
URL
https://cambridgeservicealliance.eng.cam.ac.uk/system/files/documents/2014AugustPaperBusinesstoConsumerLockinEffect.pdf


# Out-of-sample scoring and automatic selection of causal estimators

Egor Kraev
Wise Plc
<EMAIL_ADDRESS>
&Timo Flesch
OSG Digital
University of Oxford
<EMAIL_ADDRESS>
&Hudson Taylor Lekunze
University of Tartu
<EMAIL_ADDRESS>
&Mark Harley
Wise Plc
<EMAIL_ADDRESS>
&Pere Planell Morell
Wise Plc
<EMAIL_ADDRESS>
###### Abstract

Recently, many causal estimators for Conditional Average Treatment Effect
(CATE) and instrumental variable (IV) problems have been published and open
sourced, allowing to estimate granular impact of both randomized treatments
(such as A/B tests) and of user choices on the outcomes of interest. However,
the practical application of such models has been hampered by the lack of a
valid way to score the performance of such models out of sample, in order to
select the best one for a given application. We address that gap by proposing
novel scoring approaches for both the CATE case and an important subset of
instrumental variable problems, namely those where the instrumental variable
is customer access to a product feature, and the treatment is the customer’s
choice to use that feature.

Being able to score model performance out of sample allows us to apply
hyperparameter optimization methods to causal model selection and tuning. We
implement that in an open source package that relies on DoWhy and EconML
libraries for implementation of causal inference models (and also includes a
Transformed Outcome model implementation), and on FLAML for hyperparameter
optimization and for component models used in the causal models.

We demonstrate on synthetic data that optimizing the proposed scores is a
reliable method for choosing the model and its hyperparameter values, whose
estimates are close to the true impact, in the randomized CATE and IV cases.
Further, we provide examples of applying these methods to real customer data
from Wise.

##  1 Introduction

Measuring impacts of actions or choices, conditional on customer
characteristics, is a crucial problem for any business. This can be used for
optimizing these choices, such as which layout to show or which promotion to
send to a particular user, or for measuring the impact of the choices we are
less able to directly control or randomize, for example of suspending a user
who has been flagged as suspicious, or of the user choosing to use a
particular product feature.

Causal inference models can help with that; yet an as yet unsolved challenge
is how, in a given application, to choose a particular model among the many
options available, and how to tune its parameters. In automated selection and
tuning of supervised machine learning models, also known as automated machine
learning (AutoML), the key concept is out-of-sample scoring, that is comparing
the models’ performance on a validation dataset that wasn’t used in training.
Here, performance is defined by a distance metric comparing model predictions
and target values. The model (including hyperparameter choices) that performs
best on the validation set is selected by the algorithm.

We extend this approach to causal inference models. This is not a trivial task
as these models estimate the impact of an action, which is not directly
observable (we can only observe an outcome of e.g. sending an email to a
customer, but we can’t _not_ send the same email to the same customer at the
same time to observe the _impact_ of sending the email). This is known as The
Fundamental Problem of Causal Inference [1].

We consider three classes of practically important problems: firstly,
randomized CATE, where the treatment assignment is fully random, such as may
be the case in an A/B test (Figure 1A). The second problem is CATE with known
confounders (Figure 1B); this can arise for example when we assign treatment
with a degree of randomness, but biased according to prior opinions on which
kind of customer responds best to treatment). Finally, we consider the special
subset of instrumental variable models where access to a product feature is
the instrumental variable, and the customer’s decision to use it is the
treatment (Figure 1C).

We introduce new ways of scoring model performance, in particular out of
sample, that are specific to the causal inference problems in question - one
score (energy distance) that is valid for Conditional Average Treatment Effect
(CATE) models under unconfoundedness and for instrumental variable models, and
another (Normalized ERUPT) that is valid for CATE problems also with observed
confounders. We test them on synthetic data and then apply them to Wise
customer behavior data.

Figure 1: Types of causal graphs we consider

We implement our approach as an open-source pip-installable Python package
AutoCausality, which capitalises on Microsoft’s FLAML package for efficient
hyperparameter tuning [2] and DoWhy and EconML packages for causal model
estimation [3, 4] 111https://github.com/transferwise/auto-causality.

For clarity, we restrict our discussion to the case of a binary treatment;
however the approach works equally well for multi-valued or continuous
treatments222The package currently only supports categorical treatments,
including multi-valued ones. However, the scoring methods we use, in
particular energy distance, are valid for continuous treatments as well, and
many of the underlying EconML estimators also support these..

##  2 Scoring methods

###  2.1 Existing methods

Perhaps the most popular score in the literature for CATE problems is the Qini
coefficient [5]. Its main limitations are firstly, that its derivation is only
applicable to randomized CATE scenarios, and secondly, that it doesn’t readily
generalize beyond binary treatments.

Another existing method is the R-scorer [6] as implemented for example in
EconML. It relies on fitting component models for treatment and outcome as
function of confounders, which firstly has high computational cost (especially
if, as we do, one defaults to AutoML for component model construction), and
secondly creates a seemingly paradoxical situation of effectively using a
model to score another model, leaving open the question of why the output of
the former model is to be considered more trustworthy than that of the latter.

A final popular family of scores is known as policy value, or ERUPT for
Estimated Response to Proposed Treatments ([7], Equation (3)). This takes an
arbitrary treatment assignment (‘policy’), specified as a function of customer
features, and the results of an existing experiment (which could use any other
treatment assignment including strictly random, as long as the propensity to
treat is strictly positive), and combines them to produce an unbiased estimate
of expected outcome had we applied that policy.

It does that by taking all data points where the desired policy coincides with
the one actually used in the experiment (we are guaranteed to find such given
a large enough sample size due to positivity of the propensity function), and
scales the outcomes of these cases with the inverse of the propensity function
to obtain an unbiased estimate (proof in [7]). This is closely related to
importance sampling [8].

The advantage of this metric is that it has the same units as the outcome
(such as monetary units or number of conversions), which makes it easier to
interpret in a business context. However, both its strength and its weakness
is that it requires us to specify a policy to be used for calculating the
score. This is a strength as it allows us to plug in business-relevant metrics
and be explicit about treatment cost (for example, ‘treat if expected impact
exceeds treatment-specific cost’), but a weakness as it requires us to make a
choice of policy, which is at odds with our desire to automate model selection
as much as possible.

###  2.2 Normalized ERUPT for CATE models

For CATE models, both and without confounding, we use a policy value score
chosen to optimize the prediction of impact heterogeneity. Let us designate by
$i_{j}$ the impact of (a binary) treatment on unit $j$, and $\bar{i}$ is its
average value. The naive approach would be to use (and let the fitter
maximize) the policy "treat iff $i_{j}>0$". However, if the impact is
heterogenous but positive throughout, this score will teach us little as it
will be identical for all broadly correct models.

Instead, we introduce Normalized ERUPT as the policy value of the policy
"treat iff $i_{j}>\bar{i}$". Thus while each individual model, when fitting,
tries to estimate the pointwise conditional impact itself, by using this score
for subsequent out-of-sample comparison of models we select for their ability
to estimate _deviations_ from the average impact, with the expectation that
the average impact estimate will also have to be accurate for this to work.

###  2.3 Energy distance

This is applicable, with slight variations, both to randomized CATE models and
to IV models.

Let us first consider the IV case: suppose we have a set of customers $X$,
where each element $x\in X$ is a feature vector for a single customer. We
randomly split them into the control set $X_{0}$ of customers who don’t get
access to a particular new product feature, and the set $X_{1}$ of customers
who get access. Of the latter, the subset $X_{1,f}$ of customers choose to use
the product feature, and the rest don’t. We further observe some outcome $y$,
such as revenue in the 30 following days, for all customers in $X$. If we want
to measure the impact of the users using the product feature, we can begin by
assuming that the product feature only impacts the outcome for those users who
actually used it. We can then treat access to it as an instrumental variable,
and usage as the treatment.

How can we compare the performance of different IV models out of sample?

We propose the following approach: let’s call the true impact of using the
product feature $dy$, which by definition may only be nonzero for the
customers in $X_{1,f}$. As we suppose the only impact of making the product
feature available is through its usage, the conditional distribution of the
corrected outcome $\hat{y}\coloneqq y-dy$ given the customer features,
$P\left(\hat{y}|x\right)$ will be identical between the sets $X_{0}$ and
$X_{1}$.

This allows us to compare IV models out of sample, by holding out a fraction
of both $X_{0}$ and $X_{1}$ as a validation set (let’s call them $X^{v}_{0}$
and $X^{v}_{1}$), training the IV models on the remaining data, and using each
of the models to calculate $\hat{y}_{m}$ on $X^{v}_{1}$ (the index $m$ goes
over the different models).

We can then introduce some measure of distributional similarity $d$, and use
it to compute the similarity between the distributions of $\hat{y}^{m}$ over
$X^{v}_{0}$ and $X^{v}_{1}$, and use that as a score for model $m$, with
higher values denoting worse models, so

$S_{m}=d(P\left(\hat{y}_{m}|x\right)~{}\text{over}~{}X^{v}_{0},P\left(\hat{y}_{m}|x\right)~{}\text{over}~{}X^{v}_{1})$

As the customers’ access was assigned randomly, the distribution of customer
features will be identical in $X^{v}_{0}$ and $X^{v}_{1}$. Therefore, if we
combine the feature vector and the corrected outcome into a single extended
feature vector, a ‘perfect’ IV model will result in the extended feature
vectors’ distributions also being identical.

The problem then becomes to define a distance between the two distributions
from which the respective sets of extended feature vectors have been sampled
(no longer conditional distributions).

The approach we use is energy distance [9], as implemented e.g. in the dcor
package.

##  3 Using scoring methods for automatic model selection and tuning

The search happens as follows: the API expects a list of the applicable
estimators that should be searched over (or chooses the default set) for one
of the problems above, as well as the dataset to be used for fitting. We
specify a hyperparameter search space for each of the relevant causal
inference models, then combine them into a hierarchical search space, first
choosing a model, then a set of hyperparameters for it. Note that the search
spaces for the FLAML component models used as part of a given causal model (if
any), are not part of this search space, they are encapsulated in the
component models.

The supplied dataset is split into train and validation (the test set is set
aside earlier), these are stored in the AutoCausality class instance and are
the same for all fits of that optimization run.

We first fit each of the chosen estimators using a default hyperparameter set
for it (taken from EconML docs) and calculate the resulting score on the
validation set; then let the FLAML tuner take the results of these runs and
search over the composite search space (models/hyperparameters) to find the
causal inference model with the best score, within the allocated time budget.

Most causal inference models use supervised ML models as components. The most
important of these is the ‘propensity function’, that is the a priori
probability that a given unit will be treated (e.g. that a customer will be
chosen to be sent an email), given its features. In all causal models we are
aware of, this is treated as a classifier, and its fitting a supervised ML
problem. We also follow that approach, defaulting to sklearn’s
`DummyClassifier(strategy=’prior’)` for the cases when we know the assignment
to be random, and FLAML’s `AutoML(task=’classification’)` for all other cases.
However, we also allow any user-supplied classifier model that supports the
sklearn `fit/predict_proba` API.

Most advanced causal inference models also require other regression and/or
classifier component models to be supplied, to represent certain relationships
in the causal graph at hand. For now, we use `DummyClassifier` when we know
the relationship in question to be random, and FLAML regressor/classifier in
all other cases. This is not a fundamental limitation of our approach, but
merely a convenience choice to outsource component model selection.

Note that inside each estimator fit, if the model in question includes a FLAML
component classifier or regressor, we do a hyperparameter search over
classifier/regressor models to fit that classifier/regressor; we thus use
FLAML twice in a nested fashion, once to fit component models, and once for
the high-level search over causal estimators and their hyperparameters.

##  4 Performance on synthetic data

To verify the effectiveness of the proposed scores as proxies for the quality
of a given model’s estimate of the causal impact, we run the above
optimization on synthetic data, to directly observe the relationship. The
details of the construction of the synthetic dataset are given in Appendix
A.1.

#### Randomized CATE

In this case, corresponding to a traditional A/B testing scenario of Figure
1A, we conduct three optimization runs, with normalized ERUPT, Qini score, and
energy score, respectively,and plot the respective score of each run against
the mean square error (MSE) of its impact estimate in the top row of Figure 2.
We see that each of the three scores provides a reliable guidance for choosing
the model with the best MSE for the synthetic dataset at hand. Figure 5
illustrates that the model chosen using the energy distance score shows the
best performance.

Figure 2: Scoring CATE estimators on synthetic data. Top row: Randomized
treatment assignment, Bottom row: fully observable confounders. Score vs MSE,
for different scores. For normalized ERUPT and Qini high values are better,
while for energy score low values are better.

#### CATE with confounding

The results for CATE with known confounders, corresponding to Figure 1B, are
shown in the bottom row of Figure 2. Note that neither the Qini score, nor the
above derivation of the energy score are strictly speaking applicable to this
case, but we wanted to assess whether they would still perform well in
practice. Unfortunately, in this case none of the scores perform that well.
However for none of the scores did the best-scoring model have one of the
worst MSEs, suggesting that even in this case the suggested scores can be
useful for rough model selection, even if not for model fine-tuning.

#### Instrumental variables

Figure 3 shows the results for the instrumental variable case, corresponding
to Figure 1C. There we compare the performance of the listed estimators using
the Wald Estimator[10] as a baseline. We see that for this synthetic dataset,
energy distance provides a reliable way of choosing the most accurate
estimator.

Figure 3: Scoring IV Estimators on synthetic data.

##  5 Performance on real data

The above methods are already seeing application across Wise. This section
presents two examples. The first is assessing the impact of making a new
product feature (Auto-conversion) available. We can assess that in two ways:
firstly, by measuring the impact of making the feature available (known as
Intent To Treat, or ITT), which we can randomize. Secondly, we can use access
to the feature as an instrumental variable to estimate the impact of the users
choosing to use the feature.

The second case is one where randomization is not possible, namely the impact
of suspending customers the likelihood of the customer doing a transaction
once reactivated.

As we don’t know the true impact when working on real data, to evaluate the
quality of the fit we compare the model scores firstly, to a baseline model
(naive propensity-score weighted averages with 1% multiplicative random noise
in the CATE case, Wald estimator in the IV case), and secondly, across
validation and test sets. If the scores across validation and test set
correlate strongly, and systematically outperform the baseline model, we
conclude that the fit was successful.

The top row of Figure 4 presents such results for the ITT analysis on customer
volumes over different time horizons after treatment. We see that the scores
on test and validation sets are indeed strongly correlated, and systematically
outperform the baseline. For reasons of space, we only show the results for
the normalized ERUPT metric.

The IV analysis is ongoing. The main challenge is that with a low conversion
rate, we need quite a large sample size to get reliable results; yet the IV
causal estimators are particularly demanding in terms of RAM required to fit
the estimator.

The final analysis is that of the impact of user suspensions (bottom row of
Figure 4), interesting as no random assignment is possible in this case. This
also shows a strong correlation between validation and test scores.

In both cases, it’s interesting that no estimator consistently outperforms all
the others, thus demonstrating the value of automated estimator selection.

Figure 4: Score consistency between test and validation.  Top: impact of
(randomized) access to product feature on volume. Bottom: Impact of suspending
customers on their transacting again after reactivation

##  6 Conclusion

Causal inference is an exciting domain that holds great promise for business
impact even in the seemingly mature areas such as analysis of A/B test
results. One of the obstacles to its wider adoption is currently the variety
of available estimators, with no clear criteria to choose between them.

Scoring causal estimators out of sample allows us to extend the power of
AutoML for estimator selection and hyperparameter tuning, to three practically
important classes of causal inference problems. This will make these methods
easier to use for the broad audience that they deserve.

## References

  * [1] J. Pearl  “Causality”, Causality: Models, Reasoning, and Inference  Cambridge University Press, 2009  URL: https://books.google.co.uk/books?id=f4nuexsNVZIC
  * [2] Chi Wang, Qingyun Wu, Markus Weimer and Eric Zhu  “FLAML: A Fast and Lightweight AutoML Library”, 2021
  * [3] Keith Battocchi et al.  “EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation”, 2019  URL: https://github.com/microsoft/EconML
  * [4] Amit Sharma and Emre Kiciman  “DoWhy: An End-to-End Library for Causal Inference”  In _arXiv preprint arXiv:2011.04216_ , 2020
  * [5] N. Radcliffe  “Using control groups to target on predicted lift: Building and assessing uplift model”, 2007  URL: https://www.semanticscholar.org/paper/Using-control-groups-to-target-on-predicted-lift%3A-Radcliffe/147b32f3d56566c8654a9999c5477dded233328e
  * [6] Alejandro Schuler, Michael Baiocchi, Robert Tibshirani and Nigam Shah  “A comparison of methods for model selection when estimating individual treatment effects” arXiv:1804.05146 [cs, stat] type: article, 2018  URL: http://arxiv.org/abs/1804.05146
  * [7] Günter J. Hitsch and Sanjog Misra  “Heterogeneous Treatment Effects and Optimal Targeting Policy Evaluation”, 2018  DOI: 10.2139/ssrn.3111957
  * [8] T. Kloek and H.. Dijk  “Bayesian Estimates of Equation System Parameters: An Application of Integration by Monte Carlo”  In _Econometrica_ 46.1 [Wiley, Econometric Society], 1978, pp. 1–19  URL: http://www.jstor.org/stable/1913641
  * [9] Gábor J. Székely and Maria L. Rizzo  “Energy statistics: A class of statistics based on distances”  In _Journal of Statistical Planning and Inference_ 143.8, 2013, pp. 1249–1272  DOI: https://doi.org/10.1016/j.jspi.2013.03.018
  * [10] Abraham Wald  “The Fitting of Straight Lines if Both Variables are Subject to Error”  In _The Annals of Mathematical Statistics_ 11.3, 1940, pp. 284–300  DOI: 10.1214/aoms/1177731868

##  Appendix A Construction of synthetic datasets

###  A.1 CATE dataset without confounders

To simulate a randomized control trial (RTC) corresponding to the graph shown
in figure 1A, we used the following data generating process (DGP):
Let $X^{Nxd}$ be the matrix of $N$ observations and $d$ covariates, $T^{nx1}$
the vector of treatment assignments and $Y^{nx1}$ the vector of outcomes. We
make the following assumptions:

  * •

binary treatments

  * •

fully random propensity to treat (unconfoundedness)

  * •

five continuous, normally distributed covariates

  * •

no interaction between treatment effects and covariates

  * •

independence of the covariates, i.e. $\Sigma=\sigma^{2}I$

  * •

no additive noise in the outcomes, i.e. $\epsilon=0$

Then, the data is generated according to the following equations:

$\displaystyle X_{i}\sim\mathcal{N}(0,\Sigma)$ $\displaystyle T_{i}\sim
Bernoulli(0.5)$ $\displaystyle Y_{i}=\tau(X_{i})T_{i}+\mu_{0}(X_{i})+\epsilon$

where $i$ indexes individual units, $\tau$ describes the following true
treatment effect, which depends linearly on all covariates:

$\tau(X_{i})=X_{i}b^{T}+e$

where $b$ is a 1xd vector of $b_{i}\sim U(0.4,0.7)$ weights for each covariate
and $e\sim\mathcal{N}(0,0.05)$ Gaussian noise. … and $\mu_{0}(x)$ describes
the following transformation of the covariates (to keep things interesting):

$\mu_{0}(X_{i})=X_{i,1}\otimes X_{i,2}+X_{i,3}+X_{i,4}\otimes X_{i,5}$

###  A.2 CATE dataset with confounders

To simulate a dataset from an observational study, i.e. with confounders,
corresponding to graph shown in figure 1B, we used the same DPG as shown above
for the RCT data, but this time with treatment assignments that were dependent
on a subset of covariates as follows:

$T_{i}\sim Bernoulli\left(clip\left(\frac{1}{1+exp(X_{i,1}\otimes
X_{i,2}+3*X_{i,3}},0.1,0.9\right)\right)$

###  A.3 Instrumental Variables

The dataset simulating an observational study with an instrumental variable
that determined who would be offered a treatment, corresponding to the graph
shown in figure 1C, was generated in a similar fashion as the one above, but
treatment allocation $T$ depended on the instrument $Z$ as follows:

$\displaystyle W\sim\;$ $\displaystyle\text{Normal}(0,\,I_{n_{w}})$ (Observed
confounders) $\displaystyle Z\sim\;$ $\displaystyle\text{Bernoulli}(p=0.5)$
(Instrument) $\displaystyle\nu\sim\;$ $\displaystyle\text{U}[0,5]$ (Unobserved
confounder) $\displaystyle C\sim\;$
$\displaystyle\text{Bernoulli}(p=0.8\cdot\text{Sigmoid}(0.4\cdot X[0]+\nu))$
(1) $\displaystyle C0\sim\;$ $\displaystyle\text{Bernoulli}(p=0.006)$ (2)
$\displaystyle\theta=$ $\displaystyle\;7.5\cdot X[2]\cdot X[8]$ (Nonlinear
effect) $\displaystyle T=\;$ $\displaystyle C\cdot Z+C0\cdot(1-Z)$ (Treatment)
$\displaystyle y\sim\;$ $\displaystyle\theta\cdot
T+2\cdot\nu+5\cdot(X[3]>0)+0.1\cdot\text{U}[0,1]$ (Outcome)

##  Appendix B Additional plots

Here we provide a couple of plots to give some additional colour to the
results. Figure 5 plots pointwise effect estimates for the unconfounded case
against the true effect values in the synthetic data, for the best model
according to each score. We see that the energy distance makes for the best
selection criterion in this case.

Figure 6 plots the estimated average effect (ATE) against the score for each
of the estimators for the ITT on real data usecase. We see that although the
treatment assignment (access to product feature) was randomized, the naive
estimator (labeled as "Dummy") significantly over-estimates the impact, but
also that different estimators with comparable Normalized ERUPT scores can
have quite different ATE estimates.

Figure 5: Randomized treatment, synthetic data. MSE between ground truth and
estimated CATE of best-fitting model on the test set. Figure 6: AutoConversion
feature: scores vs ATE estimates. Color denotes different estimators, smallest
dot represents the values on the training set, medium sized dot those on the
validation set, and the largest one those on the test set.