# Contrastive Learning for Neural Topic Model

Thong Nguyen
VinAI Research
<EMAIL_ADDRESS>
&Luu Anh Tuan
Nanyang Technological University
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Recent empirical studies show that adversarial topic models (ATM) can
successfully capture semantic patterns of the document by differentiating a
document with another dissimilar sample. However, utilizing that
discriminative-generative architecture has two important drawbacks: (1) the
architecture does not relate similar documents, which has the same document-
word distribution of salient words; (2) it restricts the ability to integrate
external information, such as sentiments of the document, which has been shown
to benefit the training of neural topic model. To address those issues, we
revisit the adversarial topic architecture in the viewpoint of mathematical
analysis, propose a novel approach to re-formulate discriminative goal as an
optimization problem, and design a novel sampling method which facilitates the
integration of external variables. The reformulation encourages the model to
incorporate the relations among similar samples and enforces the constraint on
the similarity among dissimilar ones; while the sampling method, which is
based on the internal input and reconstructed output, helps inform the model
of salient words contributing to the main topic. Experimental results show
that our framework outperforms other state-of-the-art neural topic models in
three common benchmark datasets that belong to various domains, vocabulary
sizes, and document lengths in terms of topic coherence.

##  1 Introduction

Topic models have been successfully applied in Natural Language Processing
with various applications such as information extraction, text clustering,
summarization, and sentiment analysis [1, 2, 3, 4, 5, 6]. The most popular
conventional topic model, Latent Dirichlet Allocation [7], learns document-
topic and topic-word distribution via Gibbs sampling and mean field
approximation. To apply deep neural network for topic model, Miao et al. [8]
proposed to use neural variational inference as the training method while
Srivastava and Sutton [9] employed the logistic normal prior distribution.
However, recent studies [10, 11] showed that both Gaussian and logistic normal
prior fail to capture multi-modality aspects and semantic patterns of a
document, which are crucial to maintain the quality of a topic model.

To cope with this issue, Adversarial Topic Model (ATM) [10, 11, 12, 13] was
proposed with adversarial mechanisms using a combination of generator and
discriminator. By seeking the equilibrium between the generator and
discriminator, the generator is capable of learning meaningful semantic
patterns of the document. Nonetheless, this framework has two main
limitations. First, ATM relies on the key ingredient: leveraging the
discrimination of the real distribution from the fake (negative) distribution
to guide the training. Since the sampling of the fake distribution is not
conditioned on the real distribution, it barely generates positive samples
which largely preserves the semantic content of the real sample. This limits
the behavior concerning the mutual information in the positive sample and the
real one, which has been demonstrated as key driver to learn useful
representations in unsupervised learning [14, 15, 16, 17, 18]. Second, ATM
takes random samples from a prior distribution to feed to the generator.
Previous work [19] has shown that incorporating additional variables, such as
metadata or the sentiment, to estimate the topic distribution aids the
learning of coherent topics. Relying on a pre-defined prior distribution, ATM
hinders the integration of those variables.

To address the above drawbacks, in this paper we propose a novel method to
model the relations among samples without relying on the generative-
discriminative architecture. In particular, we formulate the objective as an
optimization problem that aims to move the representation of the input (or
prototype) closer to the one that shares the semantic content, i.e., positive
sample. We also take into account the relation of the prototype and the
negative sample by forming an auxiliary constraint to enforce the model to
push the representation of the negative farther apart from the prototype. Our
mathematical framework ends with a contrastive objective, which will be
jointly optimized with the evidence lower bound of neural topic model.

Nonetheless, another challenge arises: how to effectively generate positive
and negative samples under neural topic model setting? Recent efforts have
addressed positive sampling strategies and methods to generate hard negative
samples for images [20, 21, 22, 23]. However, relevant research to adapt the
techniques to neural topic model setting has been neglected in the literature.
In this work, we introduce a novel sampling method that mimics the way human
being seizes the similarity of a pair of documents, which is based on the
following hypothesis:

######  Hypothesis 1.

The common theme of the prototype and the positive sample can be realized due
to their relative frequency of salient words.

We use the example in Fig. 1 to explain the idea of our method. Humans are
able to tell the similarity of the input with positive sample due to the
reason that the frequency of salient words such as _“league”_ and _“teams"_ is
proportional to their counterpart in the positive sample. On the other hand,
the separation between the input and the negative sample can be induced since
those words in the input do not occur in negative sample, though they both
contain words _“billions"_ and _“dollars"_ , which are not salient in the
context of the input. Based on this intuition, our method generates the
positive and negative samples for topic model by maintaining the weights of
salient entries and altering those of unimportant ones in the prototype to
construct the positive samples while performing the opposite procedure for the
negative ones. Inherently, since our method is not depended on a fixed prior
distribution to draw our samples, we are not restrained in incorporating
external variables to provide additional knowledge for better learning topics.

Figure 1: Illustration of a document with one positive and negative pair.

In a nutshell, the contributions of our paper are as follows:

  * •

We target the problem of capturing meaningful representations through modeling
the relations among samples from a new mathematical perspective and propose a
novel contrastive objective which is jointly optimized with evidence lower
bound of neural topic model. We find that capturing the mutual information
between the prototype and its positive samples provides a strong foundation
for constructing coherent topics, while differentiating the prototype from the
negative samples plays a less important role.

  * •

We propose a novel sampling strategy that is motivated by human behavior when
comparing different documents. By relying on the reconstructed output, we
adapt the sampling to the learning process of the model, and produce the most
informative samples compared with other sampling strategies.

  * •

We conduct extensive experiments in three common topic modeling datasets and
demonstrate the effectiveness of our approach by outperforming other state-of-
the-art approaches in terms of topic coherence , on both global and topic-by-
topic basis.

##  2 Related Work

Neural Topic Model (NTM) has been studied to encode a large set of documents
using latent vectors. Inspired by Variational Autoencoder, NTM inherit most
techniques from VAE-specific early works, such as reparameterization trick
[24] and neural variational inference [25]. Subsequent works attempting to
apply for topic model [9, 26, 8] focus on studying various prior
distributions, e.g. Gaussian or logistic normal. Recently, researches directly
target to improve topic coherence through formulating it as an optimizing
objective [27], incorporating contextual language knowledge [28], or passing
external information, e.g. sentiment, group of documents, as input [19].
Generating topics that are human-interpretable has become the goal of a wide
variety of latest efforts.

Adversarial Topic Model [4] is a topic modeling approach that models the
topics with GAN-based architecture. The key components in that architecture
consist of a generator projecting randomly sampled document-topic distribution
to gain the most realistic document-word distribution as possible and a
discriminator trying to distinguish between the generated and the true sample
[10, 11]. To better learn informative representations of a document, Hu et al.
[12] proposed adding two cycle-consistent constraints to encourage the
coordination between the encoder and generator.

Contrastive Framework and Sampling Techniques There are various efforts
studying contrastive method to learn meaningful representations. For visual
information, contrastive framework is applied for tasks such as image
classification [29, 30], object detection [31, 32, 33], image segmentaion [34,
35, 36], etc. Other applications different from image include adversarial
training [37, 38, 39], graph [40, 41, 42, 43], and sequence modeling [44, 45,
46]. Specific positive sampling strategies have been proposed to improve the
performance of contrastive learning, e.g. applying view-based transformations
that preserve semantic content in the image [22, 17, 18]. On the other hand,
there is a recent surge of interest in studying negative sampling methods.
Chuang et al. [20] propose a debiasing method which is to correct the fact in
false negative samples. For object detection, Jin et al. [47] employ temporal
structure of video to generate negative examples. Although widely studied,
little effort has been made to adapt contrastive techniques to neural topic
model.

In this paper, we re-formulate our goal of learning document representations
in neural topic model as a contrastive objective. The form of our objective is
mostly related to Robinson et al. [21]. However, there are two key
differences: (1) As they use the weighting factor associated with the impact
of negative sample as a tool to search for the distribution of hard negative
samples, we consider it as an adaptive parameter to control the impact of the
positive and negative sample on the learning. (2) We regard the effect of
positive sample as the main driver to achieve meaningful representations,
while they exploit the impact of negative ones. Our approach is more
applicable to topic modeling, as proven in the investigation into human
behavior of distinguishing among documents.

##  3 Methodology

###  3.1 Notations and Problem Setting

In this paper, we focus on improving the performance of neural topic model
(NTM), measured via topic coherence. NTM inherits the architecture of
Variational Autoencoder, where the latent vector is taken as topic
distribution. Suppose the vocabulary has $V$ unique words, each document is
represented as a word count vector $\mathbf{x}\in\mathbb{R}^{V}$ and a latent
distribution over $T$ topics: $\mathbf{z}\in\mathbb{R}^{T}$. NTM assumes that
$z$ is generated from a prior distribution $p(\mathbf{z})$ and x is generated
from the conditional distribution over the topic
$p_{\phi}(\mathbf{x}|\mathbf{z})$ by a decoder $\phi$. The aim of model is to
infer the document-topic distribution given the word count. In other words, it
must estimate the posterior distribution $p(\mathbf{z}|\mathbf{x})$, which is
approximated by the variational distribution
$q_{\theta}(\mathbf{z}|\mathbf{x})$ modelled by an encoder $\theta$. NTM is
trained by minimizing the following objective

$\mathcal{L}_{\text{VAE}}(\textbf{x})=-\mathbb{E}_{q_{\theta}(\mathbf{z}|\mathbf{x})}[\log
p_{\phi}(\mathbf{x}|\mathbf{z})]+\mathbb{KL}[q_{\theta}(\mathbf{z}|\mathbf{x})||p(\mathbf{z})]$
(1)

###  3.2 Contrastive objective derivation

Let $\mathcal{X}=\\{\mathbf{x}\\}$ denote the set of document bag-of-words.
Each vector $\mathbf{x}$ is associated with a negative sample $\mathbf{x}^{-}$
and a positive sample $\mathbf{x}^{+}$. We assume a discrete set of latent
classes $\mathcal{C}$, so that $(\mathbf{x},\mathbf{x}^{+})$ have the same
latent class while $(\mathbf{x},\mathbf{x}^{-})$ does not. In this work, we
choose to use the semantic dot product to measure the similarity between
prototype $\mathbf{x}$ and the drawn samples.

Our goal is to learn a mapping function
$f_{\theta}:\mathbb{R}^{V}\rightarrow\mathbb{R}^{T}$ of the encoder $\theta$
which transforms $\mathbf{x}$ to the latent distribution $\mathbf{z}$
($\mathbf{x}^{-}$ and $\mathbf{x}^{+}$ are transformed to $\mathbf{z}^{-}$ and
$\mathbf{z}^{+}$, respectively). A reasonable mapping function must fulfill
two qualities: (1) $\mathbf{x}$ and $\mathbf{x}^{+}$ are mapped onto nearby
positions; (2) $\mathbf{x}$ and $\mathbf{x}^{-}$ are projected distantly.
Regarding goal (1) as the main objective and goal (2) as the constraint
enforcing the model to learn the relations among dissimilar samples, we
specify the constrained optimization problem, in which $\epsilon$ denotes the
strength of the constraint

$\max_{\theta}\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}(\textbf{z}\cdot\textbf{z}^{+})\quad\text{subject
to}\;\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}(\textbf{z}\cdot\textbf{z}^{-})<\epsilon$
(2)

Rewriting Eq. 2 as a Lagragian under KKT conditions [48, 49], we attain:

$\mathcal{F}(\theta,\mathbf{x},\mathbf{x}^{+},\mathbf{x}^{-})=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}(\textbf{z}\cdot\textbf{z}^{+})-\alpha\cdot[\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}(\textbf{z}\cdot\textbf{z}^{-})-\epsilon]$
(3)

where the positive KKT multiplier $\alpha$ is the regularisation coefficient
that controls the effect of the negative sample on training. Eq. 3 can be
derived to arrive at the weighted-contrastive loss.

$\mathcal{F}(\theta,\mathbf{x},\mathbf{x}^{+},\mathbf{x}^{-})\geq\mathcal{L}_{\text{cont}}(\theta,\mathbf{x},\mathbf{x}^{+},\mathbf{x}^{-})=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log\frac{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})}{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})+\beta\cdot\text{exp}(\textbf{z}\cdot\textbf{z}^{-})}\right]$
(4)

where $\alpha=\exp(\beta)$. The full proof of (4) can be found in the
Appendix. Previous works [39, 35, 40, 29, 20, 50] consider the positive and
negative sample equally likely as setting $\beta=1$. In this paper, we
leverage different values of $\beta$ to guide the model concentration on the
sample which is distinct from the input. In consequence, a reasonable value of
$\beta$ will provide a clear separation among topics in the dataset. We
demonstrate our procedure to estimate $\beta$ in the following section.

Algorithm 1 Approximate $\beta$

1:Dataset $\mathcal{D}=\\{\mathbf{x}_{i}\\}_{i=1}^{N}$, model parameter
$\theta$, model $f$, total training steps $T$

2:Randomly pick a batch of $L$ samples from the training set

3:for each sample $\mathbf{x}_{l}$ in the chosen batch do

4: Draw the negative sample $\mathbf{x}^{-}_{l}$ and a positive sample
$\mathbf{x}^{+}_{l}$

5: Obtain the latent distribution associated with the drawn samples:
$\mathbf{z}_{l}^{-}=f(\mathbf{x}_{l}^{-})$,
$f(\mathbf{x}_{l}^{+})=\mathbf{z}_{l}^{+}$

6: Obtain the candidate $\beta$ value with
$\gamma_{l}=(\mathbf{z}\cdot\mathbf{z}^{+})/(\mathbf{z}\cdot\mathbf{z}^{-})$.

7:end for

8:Initialize $\beta$ as the mean of the candidate list
$\beta_{0}=\frac{1}{L}\cdot\sum_{l=1}^{L}\gamma_{l}$

9:for $t=1$ to $T$ do

10: Train the model with
$\beta_{t}=\frac{1}{2}-\frac{1}{T}\left|\frac{T}{2}-t\right|+\beta_{0}$

11:end for

###  3.3 Controlling the effect of negative sample

When choosing value of $\beta$, we need to answer the following questions: (1)
What impact does $\beta$ have on the process of training? and (2) Is it
possible to design a procedure which is data-oriented to approximate $\beta$?

Understanding the impact of $\beta$ To exemplify point (1), we study the
impact of $\beta$ on the contrastive loss presented in Section 3.2. The
gradient of the contrastive loss (4) with respect to the latent distribution
$\mathbf{z}$ would be:

$\frac{\delta\mathcal{L}_{\text{cont}}}{\delta\mathbf{z}}=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\frac{(\mathbf{z}^{+}-\mathbf{z}^{-})\cdot\text{exp}(\mathbf{z}\cdot\mathbf{z}^{-})}{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})/\beta+\text{exp}(\textbf{z}\cdot\textbf{z}^{-})}\right]$
(5)

This derivation confirms the proportionality of the gradient norm with respect
to $\beta$. As the training progresses, the update step must be carefully
controlled to avoid bouncing around the minimum or getting stuck in local
optima.

Adaptive scheduling We leverage the adaptive approach to construct a data-
oriented procedure to estimate $\beta$. Initially, the neural topic model will
consider the representation of each document equally likely. The relation of
the similarity of the positive and the prototype to the one of the negative
and the prototype can provide us with a starting viewpoint of the model.
Concretely, we store that information in the initialized value of $\beta$
which is estimated with the formula
$\beta_{0}=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[(\mathbf{z}\cdot\mathbf{z}^{+})/(\mathbf{z}\cdot\mathbf{z}^{-})\right]$.

After intialisation, to accommodate to the model learning, we continue to
adopt an adaptive strategy which keeps updating value of $\beta$ according to
the triangle scheduling procedure:
$\beta_{t}=\frac{1}{2}-\frac{1}{T}\left|\frac{T}{2}-t\right|+\beta_{0}$. We
summarize the detail of choosing $\beta$ in Algo. 1.

###  3.4 Word-based Sampling Strategy

Here we provide a technical motivation and details of our sampling method. To
choose a sample which has the same underlying topic with the input, it is
reasonable to filter out $M$ topics which hold large values in the document-
topic distribution, as they are considered to be important by the neural topic
model. Subsequently, the procedure will draw salient words in each of the
topic that will contribute the weights to the drawn samples. We call this
strategy as the topic-based sampling strategy.

However, as shown in [8], the process of topic choosing is sensitive to the
training performance and it is challenging to determine the optimal topic
number represented for every single input. Miao et al [8] implemented a stick
breaking procedure to specifically predict number of topics for each document.
Their strategy demands approximating the likelihood increase for each decision
of breaking the stick, in other word adding the number of topic that the
document denotes. Since their process takes up a considerable amount of
computation, we propose a simpler approach which is word-based to draw both
positive and negative samples.

For each document with its associated word count vector
$\mathbf{x}\in\mathcal{X}$, we form the _tf-idf_ representation
$\mathbf{x}^{\text{tfidf}}$. Then, we feed x to the neural topic model to
obtain the latent vector $\mathbf{z}$ and the reconstructed document
$\mathbf{x}^{\text{recon}}$. Our word-based sampling strategy is illustrated
in Fig. 2.

Negative sampling We select $k$ tokens $N=\\{n_{1},n_{2},…,n_{k}\\}$ that have
the highest _tf-idf_ scores. We hypothesize that these words mainly contribute
to the topic of the document. By substituting weights of chosen tokens in the
original input x with the weights of the reconstructed representation
$\mathbf{x}^{\text{recon}}$:
$\mathbf{x}^{-}_{n_{j}}=\mathbf{x}^{\text{recon}}_{n_{j}},j\in\\{1,..,k\\}$,
we enforce the negative samples $\mathbf{x}^{-}$ to have the main content
deviated from the original input $\mathbf{x}$.

Note that since the model improves its reconstruction ability as training
progresses, the weights of salient words from the reconstructed output
approach those from the original input (but not equal). The model should take
a more careful learning step to adapt to this situation. As the negative
sample controlling factor $\beta$ decays its value when converging to the
final training step, due to our adaptive scheduling approach aforementioned in
section 3.3, it is able to adapt to this phenomenon.

Positive sampling Contrary to the negative case, we select $k$ tokens
possessing the lowest _tf-idf_ scores $P=\\{p_{1},p_{2},…,p_{k}\\}$. We obtain
the positive sample which bears a resembling theme to the original input by
assigning weights of the chosen tokens in $\textbf{x}^{\text{recon}}$ to their
counterpart in $\mathbf{x}^{+}$ through
$\mathbf{x}^{+}_{p_{j}}=\mathbf{x}^{\text{recon}}_{p_{j}},j\in\\{1,..,k\\}$.
This forms a valid positive sampling procedure since modifying weights of
insignificant tokens retains the salient topics in the source document.

Figure 2: Our sampling strategy.

###  3.5 Training objective

Joint objective We jointly combine the goal of reconstructing the original
input, matching the approximate with the true posterior distribution, with the
contrastive objective specified in section 3.2.

$\begin{split}\mathcal{L}(\mathbf{x},\theta,\phi)&=-\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[\log(p_{\theta}(\mathbf{x}|\mathbf{z}))+\mathbb{KL}(q_{\theta}(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))\right]\\\
&-\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[\log\frac{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})}{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})+\beta\cdot\text{exp}(\textbf{z}\cdot\textbf{z}^{-})}\right]\end{split}$
(6)

We summarize our learning procedure in Algorithm 2.

Algorithm 2 Contrastive Neural Topic Model

1:Dataset
$\mathcal{D}=\\{\mathbf{x}_{\text{tfidf}}^{i},\mathbf{x}_{\text{BOW}}^{i}\\}_{i=1}^{N}$,
model parameter $\theta$, model $f$, push-pull balancing factor $\alpha$,
contrastive controlling weight $\gamma$

2:repeat

3: for $i=1$ to $N$ do

4: Compute $\textbf{z}^{i}$, $\mathbf{x}^{i}_{\text{recon}}$ from
$\mathbf{x}^{i}_{\text{BOW}}$;

5: Obtain top-k indices of words with smallest _tf-idf_ weights
$K_{\text{pos}}=\\{p_{1},p_{2},...,p_{k}\\}$;

6: Sample $\mathbf{x}_{\text{pos}}^{i}$ from $K^{i}_{\text{pos}}$ and
$\mathbf{x}^{i}_{\text{recon}}$;

7: Obtain top-k indices of words with largest _tf-idf_ weights
$K_{\text{neg}}=\\{n_{1},n_{2},...,n_{k}\\}$;

8: Sample $\mathbf{x}_{\text{neg}}^{i}$ from $K^{i}_{\text{neg}}$ and
$\mathbf{x}^{i}_{\text{recon}}$;

9: end for

10: Compute the loss function $\mathcal{L}$ defined in Eq. 6;

11: Update $\theta$ by gradients to minimize the loss;

12:until the training converges

##  4 Experimental Setting

In this section, we provide the experimental setups of our conducted
experiments to evaluate the performance of our proposed method. We provide the
statistics summary of the datasets in Appendix.

###  4.1 Datasets

We conduct our experiments on three readily available datasets that belong to
various domains, vocabulary sizes, and document lengths:

  * •

20Newsgroups (20NG) dataset [51] consists of about 18000 documents, each
document is a newsgroup post and associated with a newsgroup label (for
example, talk.politics.misc). Following Huynh et al. [52], we preprocess the
dataset to remove stopwords, words possessing length equal to $1$, and get rid
of words whose frequency is less than $100$. We conduct the dataset split with
$48\%$, $12\%$, $40\%$ for training, validation, and testing, respectively.

  * •

Wikitext-103 (Wiki) [53] is a version of WikiText dataset, which includes
about $28500$ articles from Good and Featured section on Wikipedia. We follow
the preprocess, keep the top $20000$ words as in [53], and use the
train/dev/test split of $70\%$, $15\%$, and $15\%$.

  * •

IMDb movie reviews (IMDb) [54] has $50000$ movie reviews for analytics. Each
review in the corpus is connected with a sentiment label, which we use as the
external variable for our topic model. Respectively, we apply the dataset
split of $50\%$, $25\%$, $25\%$ for training, validation, and testing.

For evaluation measure, we use the Normalized Mutual Pointwise Information
(NPMI) since this strongly correlates with human judgement and is popularly
applied to verify the topic quality [28]. For text classification, we use the
F1-score as the evaluation metric.

###  4.2 Baselines

We compare our method with the following state-of-the-art neural topic models
of diverse styles:

  * •

NTM [27] a Gaussian-based neural topic model proposed by (Miao et al., 2017)
inheriting the VAE architecture and utilizing neural variational inference for
training.

  * •

SCHOLAR [19] a VAE-based neural topic model learning with logistic normal
prior and is provided with a method to incorporate external variables.

  * •

SCHOLAR + BAT [28] a version of SCHOLAR model trained using knowledge
distillation where BERT model as a teacher provides contextual knowledge for
its student, the neural topic model.

  * •

W-LDA [13] a topic model which takes form of a Wasserstein auto-encoder with
Dirichlet prior approximated by minimizing Maximum Mean Discrepancy.

  * •

BATM [11] a neural topic model whose architecture is inspired by Generative
Adversarial Network. We use the version trained with bidirectional adversarial
training method and the architecture consisting of 3 components: encoder,
generator, and discriminator.

##  5 Results

###  5.1 Topic coherence

Table 1: Results measured in NPMI of neural topic models

| 20NG | IMDb | Wiki
---|---|---|---
| $T=50$ | $T=200$ | $T=50$ | $T=200$ | $T=50$ | $T=200$
NTM [27] |  0.283 $\pm$ 0.004 |  0.277 $\pm$ 0.003 |  0.170 $\pm$ 0.008 |  0.169 $\pm$ 0.003 |  0.250 $\pm$ 0.010 |  0.291 $\pm$ 0.009
W-LDA [13] |  0.279 $\pm$ 0.003 |  0.188 $\pm$ 0.001 |  0.136 $\pm$ 0.007 |  0.095 $\pm$ 0.003 |  0.451 $\pm$ 0.012 |  0.308 $\pm$ 0.007
BATM [11] |  0.314 $\pm$ 0.003 |  0.245 $\pm$ 0.001 |  0.065 $\pm$ 0.008 |  0.090 $\pm$ 0.004 |  0.336 $\pm$ 0.010 |  0.319 $\pm$ 0.005
SCHOLAR [19] |  0.319 $\pm$ 0.007 |  0.263 $\pm$ 0.002 |  0.168 $\pm$ 0.002 |  0.140 $\pm$ 0.001 |  0.429 $\pm$ 0.011 |  0.446 $\pm$ 0.009
SCHOLAR + BAT [28] |  0.324 $\pm$ 0.006 |  0.272 $\pm$ 0.002 |  0.182 $\pm$ 0.002 |  0.175 $\pm$ 0.003 |  0.446 $\pm$ 0.010 |  0.455 $\pm$ 0.007
Our model - $k=1$ |  0.327 $\pm$ 0.006 |  0.274 $\pm$ 0.003 |  0.191 $\pm$ 0.007 |  0.185 $\pm$ 0.003 |  0.455 $\pm$ 0.012 |  0.450 $\pm$ 0.008
Our model - $k=5$ |  0.328 $\pm$ 0.004 |  0.277 $\pm$ 0.003 |  0.195 $\pm$ 0.008 |  0.187 $\pm$ 0.001 |  0.465 $\pm$ 0.012 |  0.456 $\pm$ 0.004
Our model - $k=15$ |  0.334 $\pm$ 0.004 |  0.280 $\pm$ 0.003 |  0.197 $\pm$ 0.006 |  0.188 $\pm$ 0.002 |  0.497 $\pm$ 0.009 |  0.478 $\pm$ 0.006

Overall basis We evaluate our methods both at $K=50$ and $K=200$. For each
topic, we follow previous works [28, 10, 19] to pick the top $10$ words,
measure its NPMI measure and calculate in the average value. As shown in Tab.
1, our method achieves the best topic coherence on three benchmark datasets.
We surpass the baseline SCHOLAR [19], its version trained with distilled
knowledge SCHOLAR + BAT [28], and other state-of-the-art neural topic models
in both cases of $K=50$ and $K=200$. We also establish the robustness of our
improvement by conducting experiments on $5$ runs with different random seeds
and recording the mean and standard deviation. This confirms that the
contrastive framework promotes the overall quality of generated topics.

Figure 3: (left) Jensen-Shannon for aligned topic pairs of SCHOLAR and our
model. (right) The number of aligned topic pairs which our model improves upon
SCHOLAR model

Topic-by-topic basis To further evaluate the performance of our method, we
proceed to individually compare each of our topics with the aligned topic
produced by the baseline neural topic model. Following Hoyle et al. [28], we
use a variant of competitive linking to greedily approximate the optimal
weight of the bipartite graph matching. Particularly, a bipartite graph is
constructed by linking the topics of our model and the baseline one. The
weight of each link is represented as the Jensen-Shannon (JS) divergence [55,
56] between two topics. We iteratively choose the pair according to its lowest
JS score, dispense those two topics from the topic list, and repeat until the
JS score surpasses a certain threshold. Fig. 3 (left) shows the aligned scores
for three benchmark corpora. Using visual inspection, we decide to choose the
most aligned 44 topic pairs to conduct the comparison. As shown in Fig. 3
(right), our model has more topics with higher NPMI score than the baseline
model. This means that our model not only generates better topics on average
but also on the topic-by-topic basis.

###  5.2 Text classification

Table 2: Text classification employing the latent distribution predicted by neural topic models. Model | 20NG | IMDb
---|---|---
BATM [11] | 30.8 | 66.0
SCHOLAR [19] | 52.9 | 83.4
SCHOLAR + BAT [28] | 32.2 | 73.1
Our model | 54.4 | 84.2

In order to compare the extrinsic predictive performance, we use document
classification as the downstream task. We collect the latent vectors inferred
by neural topic models in $K=50$ and train a Random Forest with the number of
decision trees as $10$ and the maximum depth as $8$ to predict the class of
each document. We pick _IMDb_ and _20NG_ for our experiment. Our method
surpasses other neural topic models on the downstream text classification with
significant gaps, as shown in Tab. 2.

###  5.3 Ablation Study

Table 3: Ablation studies

| 20NG | IMDb | Wiki
---|---|---|---
| $T=50$ | $T=200$ | $T=50$ | $T=200$ | $T=50$ | $T=200$
Our method | 0.334 $\pm$ 0.004 | 0.280 $\pm$ 0.003 | 0.197 $\pm$ 0.006 | 0.190 $\pm$ 0.002 | 0.497 $\pm$ 0.009 | 0.478 $\pm$ 0.006
\- w/o positive sampling |  0.320 $\pm$ 0.004 |  0.272 $\pm$ 0.002 |  0.187 $\pm$ 0.006 |  0.182 $\pm$ 0.007 |  0.452 $\pm$ 0.012 |  0.448 $\pm$ 0.009
\- w/o negative sampling |  0.331 $\pm$ 0.002 |  0.277 $\pm$ 0.002 |  0.195 $\pm$ 0.008 |  0.188 $\pm$ 0.003 |  0.474 $\pm$ 0.010 |  0.468 $\pm$ 0.007

To verify the efficiency mimicking the human behavior in learning topic by
grasping the commonalities, we train our methods under the besting setting
with ($k=15$, with word-based sampling), but with two different objectives:
(1) Without positive sampling: model captures semantic pattern by only
distinguishing the input and the negative sample; (2) Without negative
sampling: model learns the semantic pattern by solely minimizing the
similarity the input with the positive sample. Tab. 3 demonstrates losing one
of the two views in contrastive framework degrades the quality of the topics.
We include the optimizing objective for the two approaches in the Appendix.
Remarkably, it is interesting that removing the negative objective influences
less than for the positive one. This reconfirms the soundness of our approach
to focus on the effect of positive sample, which takes inspiration from human
perspective.

##  6 Analysis

###  6.1 Effect of adaptive controlling parameter

Table 4: Results of different sampling method

| 20NG | IMDb | Wiki
---|---|---|---
| $T=50$ | $T=200$ | $T=50$ | $T=200$ | $T=50$ | $T=200$
0-sampling |  0.269 $\pm$ 0.003 |  0.231 $\pm$ 0.001 |  0.171 $\pm$ 0.005 |  0.172 $\pm$ 0.002 |  0.448 $\pm$ 0.008 |  0.429 $\pm$ 0.007
Random sampling |  0.321 $\pm$ 0.005 |  0.273 $\pm$ 0.001 |  0.183 $\pm$ 0.002 |  0.177 $\pm$ 0.001 |  0.460 $\pm$ 0.012 |  0.462 $\pm$ 0.003
Topic-based sampling - $T=1$ |  0.313 $\pm$ 0.004 |  0.270 $\pm$ 0.005 |  0.189 $\pm$ 0.002 |  0.172 $\pm$ 0.002 |  0.467 $\pm$ 0.012 |  0.464 $\pm$ 0.002
Topic-based sampling - $T=3$ |  0.322 $\pm$ 0.005 |  0.268 $\pm$ 0.002 |  0.181 $\pm$ 0.006 |  0.170 $\pm$ 0.007 |  0.450 $\pm$ 0.013 |  0.461 $\pm$ 0.008
Topic-based sampling - $T=5$ |  0.319 $\pm$ 0.001 |  0.273 $\pm$ 0.002 |  0.176 $\pm$ 0.007 |  0.170 $\pm$ 0.003 |  0.472 $\pm$ 0.007 |  0.444 $\pm$ 0.006
Our method |  0.334 $\pm$ 0.004 |  0.280 $\pm$ 0.003 |  0.197 $\pm$ 0.006 |  0.188 $\pm$ 0.002 |  0.497 $\pm$ 0.009 |  0.478 $\pm$ 0.006

Figure 4: The influence of adaptive controlling parameter $\beta$ on topic
coherence measure

We then show the relation between $\beta$, which controls the impact of our
constraint, and the topic coherence measure in Fig. 4. As shown in the figure,
adaptive weight exhibits consistent superiority over manually tuned constant
parameter. We elaborate our high performance on the triangle scheduling that
brings the self-adjustment in different training stages.

###  6.2 Random Sampling Strategy

Table 5: Significance Testing results, reporting p-value Number of Topics | 20NG | IMDb | Wiki
---|---|---|---
$T=50$ | 0.0140 | 0.0291 | 0.0344
$T=200$ | 0.0494 | 0.0012 | 0.0156

In this section, we demonstrate the effectiveness of our random sampling
strategy. We compare our performance with two other methods: (1) $0$-sampling:
we replace the weights of $k$ chosen tokens in the BoW with $0$; (2): we
create the negative samples by drawing other documents from the dataset, then
extracting the topic vector of each document; we do not perform positive
sampling in this variant. (3) Topic-based sampling: the sampling strategy we
discussed in section 3.4, we experiment with varying choices of $T$. As shown
in Tab. 4, our sampling method consistently outperforms other strategies by a
large margin. This confirms our hypothesis that topic-based sampling is
vulnerable to drawing insufficient or redundant topics and might harm the
performance.

In addition, to further evalute the statistical significance of our
outperforming over traditional random sampling method, we conduct significance
testing and report p-value in Tab. 5. As it can be seen, all of the p-values
are smaller than 0.05, which proves the statistical significance in the
improvement of our method against traditional contrastive learning.

###  6.3 Importance Measure

Table 6: Results when employing various importance measures Metrics | IMDb | 20NG | Wiki
---|---|---|---
PCA | 0.184 $\pm$ 0.004 | 0.325 $\pm$ 0.003 | 0.481 $\pm$ 0.005
SVD | 0.181 $\pm$ 0.004 | 0.313 $\pm$ 0.003 | 0.476 $\pm$ 0.014
_tf_ | 0.196 $\pm$ 0.003 | 0.332 $\pm$ 0.006 | 0.495 $\pm$ 0.008
_idf_ | 0.193 $\pm$ 0.001 | 0.334 $\pm$ 0.004 | 0.490 $\pm$ 0.009
_tf-idf_ |  0.197 $\pm$ 0.006 |  0.334 $\pm$ 0.004 |  0.497 $\pm$ 0.009

Our word-based sampling strategy employs _tf-idf_ measure to determine
important and unimportant words that have values to be superseded to form
positive and negative samples.

To have a fair judgement, we also conduct experiments with two other complex
sampling methods using Principal Component Analysis (PCA) or Singular Value
Decomposition (SVD). Specifically, we decompose the reconstructed and original
input vectors into singular values and then replace the largest/smallest
singular values of the input with the largest/smallest ones of the
reconstructed to obtain negative/positive samples, respectively. For SVD, we
choose $k=15$ largest/smallest values for substitution whereas for PCA, we
decompose the input vector onto $50$-d space in order to make it similar to
the latent space of neural topic model (number of topics $T=50$) and proceed
to substitute $k=15$ largest/smallest values as in SVD. We conducted our
experiments on 3 datasets IMDb, 20NG, and Wiki with $T=50$, and reported the
results (NPMI) in Tab. 6.

As it can be obviously seen, despite its simplicity, _tf-idf_ -based sampling
method outperforms other complicated sampling methods in our tasks.

Table 7: Some example topics on three datasets _20NG_ , _Wiki_ , and _IMDb_

Dataset | Method | NPMI | Topic
---|---|---|---
20NG | SCHOLAR | 0.259 | max bush clinton crypto pgp clipper nsa announcement air escrow
Our model | 0.543 | crypto clipper encryption nsa escrow wiretap chip proposal warrant secure
Wiki | SCHOLAR | 0.196 | airlines boeing vehicle manufactured flight skiing airline ski engine alpine
Our model | 0.564 | skiing ski alpine athletes para paralympic nordic olympic paralympics ipc
IMDb | SCHOLAR | 0.145 | hong chinese kong imagery japanese rape lynch torture violence disturbing
Our model | 0.216 | hong chinese kong japan fairy japanese sword martial fantasy magical

###  6.4 Case Studies

We randomly extract sample topic in each of three datasets to study the
quality of the generated topics and show the result in Tab. 7. Generally, the
topic words generated by our model tends to concentrate on the main topic of
the document. For example, in _20NG_ dataset, it can be seen that our words
tend to concentrate on the topic related to cryptography (_encryption_ ,
_crypto_ , etc.) and computer hardware (_chip_ , _wiretap_ , _clipper_ ,
etc.), rather than political words, e.g. _bush_ and _clinton_ generated by
SCHOLAR model. Our generated topics in Wiki is more focused on skiing, while
SCHOLAR’s topic comprises of traffic terms such as _vehicle_ , _boeing_ , and
_engine_. Similarly, the topic words in _IMDb_ generated by our model mainly
reflects the theme of Fantasy movie in _japan_ , _chinese_ , and _hong kong_ ,
while not including off-topic words such as _torture_ and _disturbing_ which
were generated by SCHOLAR model.

##  7 Conclusion

In this paper, we propose a novel method to help neural topic model learn more
meaningful representations. Approaching the problem with a mathematical
perspective, we enforce our model to consider both effects of positive and
negative pairs. To better capture semantic patterns, we introduce a novel
sampling strategy which takes inspiration from human behavior in
differentiating documents. Experimental results on three common benchmark
datasets show that our method outperforms other state-of-the-art neural topic
models in terms of topic coherence.

## References

  * [1] Y. Lu, Q. Mei, and C. Zhai, “Investigating task performance of probabilistic topic models: an empirical study of plsa and lda,” Information Retrieval, vol. 14, no. 2, pp. 178–203, 2011.
  * [2] S. Subramani, V. Sridhar, and K. Shetty, “A novel approach of neural topic modelling for document clustering,” in 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2169–2173, IEEE, 2018.
  * [3] L. A. Tuan, D. Shah, and R. Barzilay, “Capturing greater context for question generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 9065–9072, 2020.
  * [4] R. Wang, D. Zhou, and Y. He, “Open event extraction from online text using a generative adversarial network,” arXiv preprint arXiv:1908.09246, 2019\.
  * [5] M. Wang and P. Mengoni, “How pandemic spread in news: Text analysis using topic model,” arXiv preprint arXiv:2102.04205, 2021.
  * [6] T. Nguyen, A. T. Luu, T. Lu, and T. Quan, “Enriching and controlling global semantics for text summarization,” arXiv preprint arXiv:2109.10616, 2021\.
  * [7] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” the Journal of machine Learning research, vol. 3, pp. 993–1022, 2003.
  * [8] Y. Miao, E. Grefenstette, and P. Blunsom, “Discovering discrete latent topics with neural variational inference,” in International Conference on Machine Learning, pp. 2410–2419, PMLR, 2017.
  * [9] A. Srivastava and C. Sutton, “Autoencoding variational inference for topic models,” arXiv preprint arXiv:1703.01488, 2017.
  * [10] R. Wang, D. Zhou, and Y. He, “Atm: Adversarial-neural topic model,” Information Processing & Management, vol. 56, no. 6, p. 102098, 2019.
  * [11] R. Wang, X. Hu, D. Zhou, Y. He, Y. Xiong, C. Ye, and H. Xu, “Neural topic modeling with bidirectional adversarial training,” arXiv preprint arXiv:2004.12331, 2020.
  * [12] X. Hu, R. Wang, D. Zhou, and Y. Xiong, “Neural topic modeling with cycle-consistent adversarial training,” arXiv preprint arXiv:2009.13971, 2020.
  * [13] F. Nan, R. Ding, R. Nallapati, and B. Xiang, “Topic modeling with wasserstein autoencoders,” arXiv preprint arXiv:1907.12374, 2019.
  * [14] A. Blum and T. Mitchell, “Combining labeled and unlabeled data with co-training,” in Proceedings of the eleventh annual conference on Computational learning theory, pp. 92–100, 1998.
  * [15] C. Xu, D. Tao, and C. Xu, “A survey on multi-view learning,” arXiv preprint arXiv:1304.5634, 2013.
  * [16] P. Bachman, R. D. Hjelm, and W. Buchwalter, “Learning representations by maximizing mutual information across views,” arXiv preprint arXiv:1906.00910, 2019.
  * [17] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning, pp. 1597–1607, PMLR, 2020.
  * [18] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, “What makes for good views for contrastive learning,” arXiv preprint arXiv:2005.10243, 2020.
  * [19] D. Card, C. Tan, and N. A. Smith, “Neural models for documents with metadata,” arXiv preprint arXiv:1705.09296, 2017.
  * [20] C.-Y. Chuang, J. Robinson, L. Yen-Chen, A. Torralba, and S. Jegelka, “Debiased contrastive learning,” arXiv preprint arXiv:2007.00224, 2020.
  * [21] J. Robinson, C.-Y. Chuang, S. Sra, and S. Jegelka, “Contrastive learning with hard negative samples,” arXiv preprint arXiv:2010.04592, 2020.
  * [22] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton, “Big self-supervised models are strong semi-supervised learners,” arXiv preprint arXiv:2006.10029, 2020.
  * [23] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” arXiv preprint arXiv:1906.05849, 2019.
  * [24] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013.
  * [25] D. J. Rezende, S. Mohamed, and D. Wierstra, “Stochastic backpropagation and approximate inference in deep generative models,” in International conference on machine learning, pp. 1278–1286, PMLR, 2014.
  * [26] Y. Miao, L. Yu, and P. Blunsom, “Neural variational inference for text processing,” in International conference on machine learning, pp. 1727–1736, PMLR, 2016.
  * [27] R. Ding, R. Nallapati, and B. Xiang, “Coherence-aware neural topic modeling,” arXiv preprint arXiv:1809.02687, 2018.
  * [28] A. Hoyle, P. Goel, and P. Resnik, “Improving neural topic models using knowledge distillation,” arXiv preprint arXiv:2010.02377, 2020.
  * [29] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learning,” arXiv preprint arXiv:2004.11362, 2020.
  * [30] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Y. Bengio, “Learning deep representations by mutual information estimation and maximization,” arXiv preprint arXiv:1808.06670, 2018.
  * [31] E. Xie, J. Ding, W. Wang, X. Zhan, H. Xu, Z. Li, and P. Luo, “Detco: Unsupervised contrastive learning for object detection,” arXiv preprint arXiv:2102.04803, 2021.
  * [32] B. Sun, B. Li, S. Cai, Y. Yuan, and C. Zhang, “Fsce: Few-shot object detection via contrastive proposal encoding,” arXiv preprint arXiv:2103.05950, 2021\.
  * [33] E. Amrani, R. Ben-Ari, T. Hakim, and A. Bronstein, “Learning to detect and retrieve objects from unlabeled videos,” in 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3713–3717, IEEE, 2019.
  * [34] X. Zhao, R. Vemulapalli, P. Mansfield, B. Gong, B. Green, L. Shapira, and Y. Wu, “Contrastive learning for label-efficient semantic segmentation,” arXiv preprint arXiv:2012.06985, 2020.
  * [35] K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu, “Contrastive learning of global and local features for medical image segmentation with limited annotations,” arXiv preprint arXiv:2006.10511, 2020.
  * [36] T.-W. Ke, J.-J. Hwang, and S. X. Yu, “Universal weakly supervised segmentation by pixel-to-segment contrastive learning,” arXiv preprint arXiv:2105.00957, 2021.
  * [37] C.-H. Ho and N. Vasconcelos, “Contrastive learning with adversarial examples,” arXiv preprint arXiv:2010.12050, 2020.
  * [38] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, “Virtual adversarial training: a regularization method for supervised and semi-supervised learning,” IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 8, pp. 1979–1993, 2018.
  * [39] M. Kim, J. Tack, and S. J. Hwang, “Adversarial self-supervised contrastive learning,” arXiv preprint arXiv:2006.07589, 2020.
  * [40] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” Advances in Neural Information Processing Systems, vol. 33, 2020.
  * [41] F.-Y. Sun, J. Hoffmann, V. Verma, and J. Tang, “Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization,” arXiv preprint arXiv:1908.01000, 2019.
  * [42] Y. Li, C. Gu, T. Dullien, O. Vinyals, and P. Kohli, “Graph matching networks for learning the similarity of graph structured objects,” in International Conference on Machine Learning, pp. 3835–3845, PMLR, 2019.
  * [43] K. Hassani and A. H. Khasahmadi, “Contrastive multi-view representation learning on graphs,” in International Conference on Machine Learning, pp. 4116–4126, PMLR, 2020.
  * [44] L. Logeswaran and H. Lee, “An efficient framework for learning sentence representations,” arXiv preprint arXiv:1803.02893, 2018.
  * [45] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
  * [46] O. Henaff, “Data-efficient image recognition with contrastive predictive coding,” in International Conference on Machine Learning, pp. 4182–4192, PMLR, 2020.
  * [47] S. Jin, A. RoyChowdhury, H. Jiang, A. Singh, A. Prasad, D. Chakraborty, and E. Learned-Miller, “Unsupervised hard example mining from videos for improved object detection,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 307–324, 2018.
  * [48] H. W. Kuhn and A. W. Tucker, “Nonlinear programming,” in Traces and emergence of nonlinear programming, pp. 247–258, Springer, 2014.
  * [49] W. Karush, “Minima of functions of several variables with inequalities as side constraints,” M. Sc. Dissertation. Dept. of Mathematics, Univ. of Chicago, 1939.
  * [50] J. Han, M. Shoeiby, L. Petersson, and M. A. Armin, “Dual contrastive learning for unsupervised image-to-image translation,” arXiv preprint arXiv:2104.07689, 2021.
  * [51] K. Lang, “Newsweeder: Learning to filter netnews,” in Machine Learning Proceedings 1995, pp. 331–339, Elsevier, 1995.
  * [52] V. Huynh, H. Zhao, and D. Phung, “Otlda: A geometry-aware optimal transport approach for topic modeling,” Advances in Neural Information Processing Systems, vol. 33, 2020.
  * [53] S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, 2016.
  * [54] A. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142–150, 2011.
  * [55] A. K. Wong and M. You, “Entropy and distance of random graphs with application to structural pattern recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, no. 5, pp. 599–609, 1985.
  * [56] J. Lin, “Divergence measures based on the shannon entropy,” IEEE Transactions on Information theory, vol. 37, no. 1, pp. 145–151, 1991.

##  Appendix A Implementation details

In this section, we include the hyperprameter details we use in this work,
e.g. learning rate, batch size, etc. We apply different sets of
hyperparameters, with respect to the dataset the neural topic model is trained
on.

Table 8: Hyperparameter details

| 20NG | IMDb | Wiki
---|---|---|---
Learning rate | 0.002 | 0.001 | 0.002
Batch size | 200 | 200 | 500
$k$ | $\\{1,5,10,15,25,30\\}$ | $\\{1,5,10,15,25,30\\}$ | $\\{1,5,10,15,25,30\\}$

##  Appendix B Contrastive loss derivation

We provide the proof of the inequality (4) in this section.

######  Theorem 1.

Let $\mathbf{x}$ denote the word count representation of a document,
$\mathbf{x}^{+},\mathbf{x}^{-}$ denote the positive sample and negative sample
with respect to $\mathbf{x}$,
$f_{\theta}:\mathbb{R}^{V}\rightarrow\mathbb{R}^{T}$ denote the mapping
function of the encoder, $\alpha\geq 0$ denote the positive KKT multiplier,
and $\epsilon\geq 0$ denote the strength of constraint. Suppose
$\beta=\text{exp}(\alpha)$, then we have the following inequality

$\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log(\exp(\textbf{z}\cdot\textbf{z}^{+}))-\alpha\cdot(\log(\exp(\textbf{z}\cdot\textbf{z}^{-}))-\epsilon)\right]\geq\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log\frac{\exp(\textbf{z}\cdot\textbf{z}^{+})}{\exp(\textbf{z}\cdot\textbf{z}^{+})+\beta\cdot\exp(\textbf{z}\cdot\textbf{z}^{-})}\right]$
(7)

###### Proof.

We rewrite the LHS in (7)

$\begin{split}&\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log(\exp(\textbf{z}\cdot\textbf{z}^{+}))-\alpha\cdot(\log(\exp(\textbf{z}\cdot\textbf{z}^{-}))-\epsilon)\right]=\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log(\exp(\textbf{z}\cdot\textbf{z}^{+}))-\alpha\cdot(\log(\exp(\textbf{z}\cdot\textbf{z}^{-})))+\alpha\cdot\epsilon\right]\\\
&\geq\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log\left(\frac{\exp(\textbf{z}\cdot\textbf{z}^{+})}{\beta\cdot\exp(\textbf{z}\cdot\textbf{z}^{-})}\right)\right]\quad\text{as}\;\alpha,\epsilon\geq
0\\\
&\geq\mathbb{E}_{\mathbf{x}\sim\mathcal{X}}\left[\log\left(\frac{\exp(\textbf{z}\cdot\textbf{z}^{+})}{\exp(\textbf{z}\cdot\textbf{z}^{+})+\beta\cdot\exp(\textbf{z}\cdot\textbf{z}^{-})}\right)\right]\quad\text{as}\;\exp(\textbf{z}\cdot\textbf{z}^{+})>0\\\
\end{split}$

At this point, we conclude our proof. ∎

##  Appendix C Versions of loss function

We provide the description of versions of loss functions we use in this work.

Contrastive approach - Using both positive and negative samples

$\begin{split}\mathcal{L}(-\mathbf{x},\theta,\phi)&=\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[-\log(p_{\theta}(\mathbf{x}|\mathbf{z}))+\mathbb{KL}(q_{\theta}(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))\right]\\\
&-\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[\log\frac{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})}{\text{exp}(\textbf{z}\cdot\textbf{z}^{+})+\beta\cdot\text{exp}(\textbf{z}\cdot\textbf{z}^{-})}\right]\end{split}$
(8)

Contrastive approach - Using only positive sample

$\begin{split}\mathcal{L}(\mathbf{x},\theta,\phi)&=\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[-\log(p_{\theta}(\mathbf{x}|\mathbf{z}))+\mathbb{KL}(q_{\theta}(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))\right]-\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[\textbf{z}\cdot\textbf{z}^{+}\right]\end{split}$
(9)

Contrastive approach - Using only negative sample

$\begin{split}\mathcal{L}(\mathbf{x},\theta,\phi)&=\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[-\log(p_{\theta}(\mathbf{x}|\mathbf{z}))+\mathbb{KL}(q_{\theta}(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))\right]+\alpha\cdot\mathbb{E}_{\mathbf{z}\sim
q(\mathbf{z}|\mathbf{x})}\left[\textbf{z}\cdot\textbf{z}^{-}\right]\end{split}$
(10)

##  Appendix D Understanding number of chosen tokens

We demonstrate the effect of changing the number of tokens chosen for
sampling. We perform training with different choices of
$k\in\\{1,5,10,15,20,25,30\\}$ and record the topic coherence. For visibility,
we normalize them to one common scale before plotting them in Fig 5. It can be
seen that the performance initially increases as we select more tokens from
the reconstructed output to substitute for the drawn sample. However, when the
number of selected tokens $k$ grows too large, the topic coherence measure
starts decreasing as $k$ increases. We hypothesize that the overwhelming
number of substituted values can alter the semantic of the positive samples,
while producing random negative sample.

Figure 5: The influence of number of tokens chosen to construct random samples


# TFGAN: Time and Frequency Domain Based Generative Adversarial Network for
High-fidelity Speech Synthesis

###### Abstract

Recently, GAN based speech synthesis methods, such as MelGAN, have become very
popular. Compared to conventional autoregressive based methods, parallel
structures based generators make waveform generation process fast and stable.
However, the quality of generated speech by autoregressive based neural
vocoders, such as WaveRNN, is still higher than GAN. To address this issue, we
propose a novel vocoder model: TFGAN, which is adversarially learned both in
time and frequency domain. On one hand, we propose to discriminate ground-
truth waveform from synthetic one in frequency domain for offering more
consistency guarantees instead of only in time domain. On the other hand, in
contrast to the conventionally frequency-domain STFT loss approach or feature
map loss by discriminator to learn waveform, we propose a set of time-domain
loss that encourage the generator to capture the waveform directly. TFGAN has
nearly same synthesis speed as MelGAN, but the fidelity is significantly
improved by our novel learning method. In our experiments, TFGAN shows the
ability to achieve comparable mean opinion score (MOS) than autoregressive
vocoder under speech synthesis context.

Index Terms—  neural vocoder, generative adversarial network, time and
frequency domain, speech synthesis

##  1 Introduction

Neural network-based vocoders have been rapidly improved in recent years,
which have obvious advantages over the traditional parametric vocoders [1, 2]
in the aspect of naturalness of speech. Since the emergence of WaveNet [3],
the neural vocoder has developed dramatically, whose quality is closer to
human voice. However, due to autoregressive modeling method and very deep
structures, the generation efficiency of WaveNet is inherently low. To this
end, in [4], a model named WaveRNN has been proposed, which models waveform
with single GRU layer and has much smaller complexity than WaveNet. Also,
LPCNet [5] greatly increased the synthesis speed on the premise of
guaranteeing voice quality by combining linear prediction coding with neural
vocoder. What’s more, many signal processing technique has been introduced in
neural vocoder for smaller complexity, such as Multi-band WaveRNN [6] and
FeatherWave [7]. Obviously, autoregressive vocoders are easy to get good
quality, but engineered optimizations are required for achieving corresponding
inference speed. Parallel WaveNet [8] and ClariNet [9]) made steps to generate
waveform in parallel with inverse normalizing flow [10], which required to
distill from a autoregressive WaveNet model. Other flow-based models, such as
WaveGlow [11] and WaveFlow [12], removed the distillation and trained with
maximum likelihood directly, but the quality is still not good enough.
Parallel WaveGAN [13] and MelGAN [14] adopt non-autoregressive architecture
and adversarial training, they can generate waveform in parallel at inference
stage. Therefore, the synthesis speed is greatly accelerated without extra
engineered work.

Although the synthesis speed of GAN based model is quite charming, the
generated quality of speech synthesis is not satisfactory. Currently, there is
no efficient time-domain loss for generator and discriminator only model the
audio in time domain. As a result, the synthesized audio has obvious
artificial traces in the high frequency and there are discontinuous phenomena
in the phase, which lead to the problems of metallic sense and lag in hearing
sense, such as in MelGAN. What’s more, the widely adopt upsampling mechanism
with transpose convolution could introduce artifacts in breathing part of
speech. In this paper, we propose the TFGAN vocoder based on MelGAN, which
encourage generator and discriminator learn waveform both in time and
frequency domain.

We summarize the contributions of proposed TFGAN as follows: Firstly, we
propose a ResNet18 [15] based frequency discriminator to improve the reality
of phase and offer more consistency guarantees compared to only by time-domain
discriminator. Secondly, multiple time-domain waveform loss are proposed to
help generator to learn directly and reduce metallic noise caused by
overdependence of STFT Loss.

##  2 Related Work

###  2.1 MelGAN

In basic MelGAN, a stack of transposed convolutional layers are used to
upsample the mel-spectrogram to match the waveform sequence and each
transposed convolutional layer is followed by a stack of residual blocks with
dilated convolutions whose receptive field increased exponentially with the
number of layers. To solve the metallic sound problem, discriminator adopt a
multi-scale architecture which have identical network structure but operate on
different audio scales. Multiple discriminator at different scales are
motivated from the fact that audio has structure at different levels.
Specifically, with K discriminators, basic MelGAN conducts adversarial
training with objectives as:

$\begin{split}D_{loss}=\min_{D_{k}}\mathbb{E}_{x}\left[\min(0,1-D_{k}(x))\right]\\\
+\mathbb{E}_{s,z}[\min(0,1+D_{k}(G(s,z)))],{\forall}k=1,2,3,\end{split}$ (1)

$G_{loss}=\min_{G}\mathbb{E}_{s,z}[\sum_{k=1}^{3}-D_{k}(G(s,z))],$ (2)

where $x$ represents the raw waveform, $s$ represents the acoustic
features(eg., mel-spectrogram) and $z$ represents the gaussian noise vector.

In addition to the discriminator’s signal, the feature matching objective is
also been used to train the generator. This objective minimizes the L1
distance between the discriminator feature maps of real and synthesized audio.

###  2.2 Multi-resolution STFT Loss

As proposed in [13, 16, 17], STFT Loss can help model generate higher quality
speech and fast convergence. The single STFT loss contains $L_{sc}$ and
$L_{mag}$ which denote spectral convergence and log STFT magnitude loss
respectively, they are defined as follows:

$L_{sc}(x,\widetilde{x})=\frac{\left\|\left|STFT(x)\right|-\left|STFT(\widetilde{x})\right|\right\|_{F}}{\left\|\left|STFT(x)\right|\right\|_{F}},$
(3)

$L_{mag}(x,\widetilde{x})=\frac{1}{N}\left\|log\left|STFT(x)\right|-log\left|STFT(\widetilde{x})\right|\right\|_{1},$
(4)

where $\left\|\cdot\right\|_{F}$ and $\left\|\cdot\right\|_{1}$ represent the
Frobenius and $L_{1}$ norms, respectively. $\left|STFT(\cdot)\right|$
indicates the $STFT$ function to compute magnitudes and $N$ is the number of
elements in the magnitude. The multi-resolution STFT loss [13] is the sum of M
single STFT losses with different analysis parameters and the loss($L_{stft}$)
is represented as follows:

$L_{stft}(G)=\mathbb{E}_{x,\widetilde{x}}[\frac{1}{M}\sum_{m=1}^{M}\left(L_{sc}^{m}(x,\widetilde{x})+L_{mag}^{m}\left(x,\widetilde{x}\right)\right)].$
(5)

##  3 TFGAN

In order to figure out the artificial sound appear in MelGAN[14] and MB-
MelGAN[16], we model speech in both time and frequency domain. Fig. 1
illustrates the proposed TFGAN. The model consists of three trainable
components: a generator $G$ and two discriminators: time discriminator and
frequency discriminator.

Fig. 1: The architecture of proposed TFGAN

Similarly to basic MelGAN we also use s stack of transpose convolutional
layers to upsample the mel-spectrogram, but in order to remove periodic
artifacts in breathing part of speech, we improve the upsampling structure as
shown in Fig. 2. Firstly, we feed the input sequence into a sinusoidal
activation function, the results of which add to themselves. Then, in addition
to using transpose convolution, we also repeat the output of the first step by
the up-sample factor directly and following a convolutional layer. Finally the
outputs of transpose convolutional layer and repeat structure are added as the
up-sample block output. And 240x upsampling is conducted through 3 upsampling
layers with 8x, 6x and 5x factors respectively. Expanding the receptive field
appropriately is helpful to improve the quality of speech generation [16], we
increase the depth of the residual dilated convolution stack from 3 layers to
4 layers with dilation 1, 3, 9, 27.

Fig. 2: The UpSample and ResStack Block

###  3.1 Frequency Discriminator

In order to generate high quality speech without the issues(e.g., vibrations)
occurred with single time domain discriminator, we propose the Frequency
Discriminator to correct the output of generator both in the time domain and
frequency domain.

In frequency discriminator, we use the the short-time Fourier transform(STFT)
to get the characteristics of the audio in the frequency domain, then the real
and imaginary parts are fed into four F-Discriminators
($D^{(F)}_{1},D^{(F)}_{2},D^{(F)}_{3},D^{(F)}_{4}$) respectively. Each
F-Discriminator consists of two Residual Blocks, each of which contains a
stack of convolutional layers. In order to avoid the problem of gradient
disappearance caused by too many convolutional layers, $D_{1}^{(F)}$ contains
only two convolution layers, but $D_{2}^{(F)}$, $D_{3}^{(F)}$, $D_{4}^{(F)}$
have a residual connection. Basically, we adopt ResNet18 [15] as frequency
discriminator.

The time domain discriminator adopt a multi-scale architecture operating on
different audio scales which is similar to that in basic MelGAN: D1 operates
on the scale of raw audio, while D2 and D3 operate on raw audio down-sampled
by a factor of 2 and 4 respectively.

Fig. 3: The Residual Block of F-Discriminator Table 1: The parameters for Multi-resolution audio loss in time domain. | 1 | 2 | 3 | 4
---|---|---|---|---
frame length | 1 | 240 | 480 | 960
hop length | 1 | 120 | 240 | 480

###  3.2 Time domain audio loss

We find that although the multi-resolution STFT loss helps with convergence,
but its periodicity makes the synthesized speech sound artificial in high
frequency. In order to solve this problem, we design a set of loss functions
in time domain called Multi-scale dynamic loss, which can help model learn
audio in time domain directly. Multi-scale dynamic loss consists of three
parts in four resolution scales: $loss_{e}$, $loss_{p}$ and $loss_{t}$ which
are designed to capture energy, fast convergence and remove high-frequency
metallic noise respectively.

$a)$ Energy loss:

$loss_{e}=\left\|\mathbb{E}{(x^{2})}-\mathbb{E}{(\widetilde{x}^{2})}\right\|_{1},$
(6)

where $\left\|\cdot\right\|_{1}$ is the $L^{1}$ Frobenius norm, $x$ and
$\widetilde{x}$ represent the target audio signal and synthetic signal which
are processed by frames with window size and hop size showed in Table 1.

$b)$ Time loss:

$loss_{t}=\left\|\mathbb{E}{(x)}-\mathbb{E}{(\widetilde{x})}\right\|_{1},$ (7)

where $x$ and $\widetilde{x}$ represent the target audio signal and synthetic
signal processed by frames with window size and hop size showed in Table 1;

$c)$ Phase loss:

$loss_{p}=\left\|\Delta x-\Delta\widetilde{x}\right\|_{1},$ (8)

where $\Delta x$ represents the signal $x$ processed by the method of first
difference. Due to the addition of a frequency domain discriminator, the final
objective becomes:

$L^{T}(G)=loss_{e}+loss_{t}+loss_{p},$ (9)

$\begin{split}\min_{G}\mathbb{E}_{s,z}[\lambda_{1}\sum_{k=1}^{3}-D_{k}(G(s,z))]+\lambda_{2}\mathbb{E}_{s}[L_{stft}(G)]\\\
+\lambda_{3}\mathbb{E}_{s,z}^{f}[-D(stft(G(s,z)),x)]+\lambda_{4}\sum_{k=1}^{4}\mathbb{E}_{s}[L_{k}^{T}(G)].\end{split}$
(10)

As shown in Eq. 10, We also adopt the multi-resolution STFT loss to speed up
the adversarial training and stability as an auxiliary loss.

##  4 Experiments

###  4.1 Data Set

In our experiments, the training and testing data we used are ground truth
aligned(GTA) data from a well-trained Tacotron2 [18]. And the origin data is a
Mandarin corpus of 20 hours of recordings, which were recorded by a
professional broadcaster. All the recordings were down-sampled to 24 kHz
sampling rate with 16-bit format. About 19 hours of recordings were used for
model training and the rest were used for validation. We set $\lambda_{1}$,
$\lambda_{2}$, $\lambda_{3}$ as 1.0 and set $\lambda_{4}$ as 20.0 specially.

###  4.2 Experimental Setup

To demonstrate that the proposed model can synthesize high-fidelity audio, we
chose MelGAN and an autoregressive vocoder, Multi-band WaveRNN [6] as the
baseline. In the MelGAN baseline system, we used the open-source
implementation111https://github.com/kan-bayashi/ParallelWaveGAN and the
configuration was exactly the same as its original version. For WaveRNN we use
4 bands, 10-bit $\mu$-law quantization for dual softmax layers and the
dimension of FC layer is 256 [7], the preemphasis parameter is 0.85.

Generator In our proposed TFGAN, we use three upsampling and residual stack
combinations to realize $240$x upsampling, where the up-factor is (8, 6, 5).
The output channels of the first convolution is 512 and the upsampling network
are (256, 128, 64). The transposed convolution’s kernel-size in each
upsampling block is twice of the stride, which in the convolution after the
repeating layer is 1.

Discriminator We adopt ResNet18 [15] as the frequency discriminator, which
performs in frequency domain. The waveform passes through a STFT using the
hanning window whose fft size, hop length and window length are (512, 240,
512), then the real and imaginary parts of the frame-level results obtained by
the STFT are fed into the frequency discriminator respectively. The time
domain discriminator is the same as that in Multi-Band MelGAN [16], each time
domain discriminator block has 3 strided convolution with stride 4.

Training and inference In the training phase, the Adam [19] optimizer was
adopted with a learning rate of 0.0002 for both generator and discriminator.
The proposed model was trained on a single GPU with mini-batch size of 16
audio clips(24000 samples). The weights of the neural vocoders were randomly
initialized with fixed random seed and all the networks were trained with
2000k iterations.

Table 2: Mean Opinion Score (MOS) with $95\%$ confidence intervals for different models. | MOS on speech quality
---|---
MelGAN | 3.95 $\pm$ 0.05
WaveRNN | 4.34 $\pm$ 0.04
TFGAN | 4.35 $\pm$ 0.04
Ground Truth | 4.42 $\pm$ 0.03

###  4.3 Ablation study

This experiment was an ablation test on TFGAN. In order to avoid
indistinguishable error caused by too many audio samples, we use Short-Time
Objective Intelligibility(STOI) and Perceptual Evaluation of Speech
Quality(PESQ) as objective indicators to evaluate the model [20]. We conducted
four groups of ablation experiments, and the experimental results are shown in
Table 3. All the models were trained to 2 million steps. On the basis of
$B_{0}$, Multi-resolution STFT Loss and the modified upsampling structure were
added in $P_{1}$ and $P_{2}$ respectively, which all improved on the metric
PESQ. Periodic artifacts were removed in breathing part with the modification
of upsampling layer, especially under TTS context. $P_{3}$ is an improved
combination of $P_{1}$ and $P_{2}$ without the addition of a frequency
discriminator, with overall improvement in STOI and PESQ, where PESQ surpassed
WaveRNN. $P_{4}$ is our proposed model, which achieved the best in both
indices and exceeded Multi-band WaveRNN.

###  4.4 Comparison on MOS

Subjective evaluation was conducted to evaluate the perceptual quality of the
proposed TFGAN vocoder by MOS. In order to perform fair comparison, we
randomly selected 40 utterance from test set for MOS testing and 30 native
Mandarin speakers participated in the listening test. The
results222https://wavecoder.github.io/TFGAN/ of the subjective MOS evaluation
is presented in Table 2.The result show that the proposed TFGAN can generate
high quality 24 kHz speech with a slightly better MOS than MelGAN and Multi-
band WaveRNN.

Table 3: The objective result of ablation study with STOI and PESQ. The higher is better. |  | STOI | PESQ
---|---|---|---
$B_{0}$ | Baseline(MelGAN) | 0.94 | 2.77
$P_{1}$ | \+ STFT loss | 0.94 | 2.92
$P_{2}$ |  \+ Residual and sine activation | 0.94 | 2.93
$P_{3}$ | TFGAN w/o F-Discriminator | 0.95 | 3.14
$P_{4}$ | TFGAN | 0.95 | 3.24
$P_{5}$ | WaveRNN | 0.92 | 3.02
$P_{6}$ | Ground Truth | 1.00 | 4.50

##  5 Conclusion

In this work, we proposed TFGAN, an improved MelGAN neural vocoder. By the
proposed time domain audio loss, the generator could capture waveform
efficiently combining with STFT Loss. What’s more, we proposed the frequency
discriminator to score the distance between real speech and generated speech.
Also, we improved the transpose convolution based upsampling mechanism by
integrating nearest interpolation to reduce periodic artifacts. The proposed
neural vocoder has the same inference speed as MelGAN, but the fidelity of
generated speech is much improved. Our experimental results shown that TFGAN
was slightly outperformed than WaveRNN, which is a strong autoregressive
neural vocoder.

## References

  * [1] M. Morise, F. Yokomori, and K. Ozawa, “World: a vocoder-based high-quality speech synthesis system for real-time applications,” _IEICE TRANSACTIONS on Information and Systems_ , vol. 99, no. 7, pp. 1877–1884, 2016\.
  * [2] H. Kawahara, “Straight, exploitation of the other aspect of vocoder: Perceptually isomorphic decomposition of speech sounds,” _Acoustical science and technology_ , vol. 27, no. 6, pp. 349–353, 2006.
  * [3] A. Van Den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. W. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio.” in _SSW_ , 2016, p. 125.
  * [4] N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. v. d. Oord, S. Dieleman, and K. Kavukcuoglu, “Efficient neural audio synthesis,” _arXiv preprint arXiv:1802.08435_ , 2018.
  * [5] J.-M. Valin and J. Skoglund, “Lpcnet: Improving neural speech synthesis through linear prediction,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 5891–5895.
  * [6] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei _et al._ , “Durian: Duration informed attention network for multimodal synthesis,” _arXiv preprint arXiv:1909.01700_ , 2019.
  * [7] Q. Tian, Z. Zhang, H. Lu, L.-H. Chen, and S. Liu, “Featherwave: An efficient high-fidelity neural vocoder with multi-band linear prediction,” _arXiv preprint arXiv:2005.05551_ , 2020.
  * [8] A. v. d. Oord, Y. Li, I. Babuschkin, K. Simonyan, O. Vinyals, K. Kavukcuoglu, G. v. d. Driessche, E. Lockhart, L. C. Cobo, F. Stimberg _et al._ , “Parallel wavenet: Fast high-fidelity speech synthesis,” _arXiv preprint arXiv:1711.10433_ , 2017.
  * [9] W. Ping, K. Peng, and J. Chen, “Clarinet: Parallel wave generation in end-to-end text-to-speech,” _arXiv preprint arXiv:1807.07281_ , 2018.
  * [10] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling, “Improved variational inference with inverse autoregressive flow,” in _Advances in neural information processing systems_ , 2016, pp. 4743–4751.
  * [11] R. Prenger, R. Valle, and B. Catanzaro, “Waveglow: A flow-based generative network for speech synthesis,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 3617–3621.
  * [12] W. Ping, K. Peng, K. Zhao, and Z. Song, “Waveflow: A compact flow-based model for raw audio,” _arXiv preprint arXiv:1912.01219_ , 2019.
  * [13] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6199–6203.
  * [14] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and A. C. Courville, “Melgan: Generative adversarial networks for conditional waveform synthesis,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 14 910–14 921.
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [16] G. Yang, S. Yang, K. Liu, P. Fang, W. Chen, and L. Xie, “Multi-band melgan: Faster waveform generation for high-quality text-to-speech,” _arXiv preprint arXiv:2005.05106_ , 2020.
  * [17] J. Yang, J. Lee, Y. Kim, H. Cho, and I. Kim, “Vocgan: A high-fidelity real-time vocoder with a hierarchically-nested adversarial network,” _arXiv preprint arXiv:2007.15256_ , 2020.
  * [18] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan _et al._ , “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4779–4783.
  * [19] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [20] N. Adiga, V. Tsiaras, and Y. Stylianou, “On the use of wavenet as a statistical vocoder,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5674–5678.