# Preference and Concurrence Aware Bayesian Graph Neural Networks for
Recommender Systems

(October 2022)

###### Abstract

Graph-based collaborative filtering methods have prevailing performance for
recommender systems since they can capture high-order information between
users and items, in which the graphs are constructed from the observed user-
item interactions that might miss links or contain spurious positive
interactions in industrial scenarios. The Bayesian Graph Neural Network
framework approaches this issue with generative models for the interaction
graphs. The critical problem is to devise a proper family of graph generative
models tailored to recommender systems. We propose an efficient generative
model that jointly considers the preferences of users, the concurrence of
items and some important graph structure information. Experiments on four
popular benchmark datasets demonstrate the effectiveness of our proposed graph
generative methods for recommender systems.

##  1 Introduction

Recommendation systems are playing an increasingly important role in online
consumption applications in the era of the information flood. The essential
task is to predict the set of items that each user is likely to interact with
[1, 2]. To improve users’ experience, collaborative filtering (CF) [3, 4, 5,
6] is proposed to recommend similar items to users with similar interests
based on the past interactions between users and items. Later on, the
performance of CF is boosted by equipping with deep-learning models [7, 6, 8,
9, 10] that can automatically learn complex latent features and capture the
non-linear similarity between users and items.

Recently, graph neural network (GNN) models have been extensively applied in
recommender systems and achieved new state-of-the-art results [9, 11, 12, 6,
13, 14], since the user-item interactions in recommender systems can be
naturally represented as bipartite graphs and the GNN can capture the high-
order user-item interactions. However, as suggested in [15, 16], the observed
graphs from the industry can contain noise information. In recommender
systems, some edges may be misleading, where a user could click an item due to
the attractive advertisement content rather than his/her true interest in the
items, and a false negative tag might be added to some items if the user
accidentally opens an app but never has a chance to check it. GNN models on
the noisy graphs fail to consider the uncertainty of the observed graphs,
leading to sub-optimal results.

To address the uncertainty issues of the graphs, [16, 17] propose a new
training framework on Bayesian Graph Neural Networks (BGNN). The critical
module is to incorporate a graph generative model to sample ensembles of
graphs from the observed one, and the GNN is applied on both the observed
graph and the sampled graphs to generate a more robust graph representation.
To efficiently generate similar but informative graphs, they propose the Node-
Copy model, where the set of items for each user is randomly replaced with
some other set of items from other users based on the similarity of the
current user and the target user.

Despite the success of the Node-Copy model in the BGNN framework, this
critical module has limitations, and the power of BGNN is not fully exploited.
They lack the flexibility to generate novel graphs that can adapt to various
scenarios. Specifically, Node-Copy can only suggest the existing combination
of items from some existing users as the target item set. In the extreme case
that all users have only interacted with a small range of items, the Node-Copy
model can never generate a graph with the interaction between the users and
the unobserved items. Besides, this model cannot be enhanced with some
important prior beliefs about the properties of the observed graphs.

This work proposes a flexible graph generative model tailored to recommender
systems for BGNN framework and boosts performance. Specifically, we propose
that the generated graphs should share three important properties with the
observed graph, namely, _user preference_ , _item concurrence_ and _node
degree distribution_. Pursuing a similar user preference guarantees that the
generated graphs will not contain obvious false negative edges. Preserving
item concurrence turns to stick the frequently co-appeared items as a whole
and preserves valuable collaborative signals. Keeping similar node degree
distribution will generate similar graphs structurally.

Bearing the three properties in mind, we propose an efficient iterative
heuristic algorithm that independently generates the set of neighbouring items
for each user. At each iteration, the probability of sampling some item is
jointly determined by the user preference, the similarity between the item and
the sampled item set and the item degree. Through extensive experiments on
four public datasets, our proposed graph generative model consistently
outperforms the Node-Copy model and other strong baselines.

##  2 Preliminaries

Recommender systems with collaborative filtering predict a set of items for
each user that he/she may like based on past user-item interactions. We can
treat the interactions as a graph
${\mathcal{G}}=\\{{\mathcal{V}},{\mathcal{E}}\\}$, where
${\mathcal{V}}={\mathcal{U}}\cup{\mathcal{I}}$ is the union of user nodes
${\mathcal{U}}$ and item nodes ${\mathcal{I}}$, and ${\mathcal{E}}$ defines
the set of edges where $(u,i)\in{\mathcal{E}}$ if user $u$ has interacted with
item $i$. We use ${\mathcal{N}}(i)$ to denote the set of neighbor nodes for
node $i$. Since ${\mathcal{G}}$ is a bipartite graph for recommender systems,
we can represent the ${\mathcal{G}}$ with the interaction matrix
${\mathbf{R}}\in\mathbb{R}^{|{\mathcal{U}}|\times|{\mathcal{I}}|}$, where
${\textnormal{R}}_{ui}:=1$ if $(u,i)\in{\mathcal{E}}$ and otherwise
${\textnormal{R}}_{ui}:=0$. [6, 13] demonstrates that graph neural networks
achieve superior performance due to the fact that graphs can capture high-
order user-item interaction information.

As depicted in [16], the observed graphs ${\mathcal{G}}$ are noisy in
industry, and they could miss positive edges or contain false positive edges.
To solve this issue, [16] introduces the Bayesian graph neural networks (BGNN)
framework, and they incorporate a graph generative model to generate new
graphs $\hat{{\mathcal{G}}}$ based on the observed graph ${\mathcal{G}}$. The
final representation ${\bm{h}}_{i}$ of a node $i\in{\mathcal{V}}$ is defined
as the concatenation of the embedding from the original graph
${\bm{h}}_{i}^{{\mathcal{G}}}$ and the sampled graphs
${\bm{h}}_{i}^{\hat{{\mathcal{G}}}}$, i.e.,
${\bm{h}}_{i}={\bm{h}}_{i}^{\mathcal{G}}||{\bm{h}}_{i}^{\hat{{\mathcal{G}}}}$,
$\forall i\in{\mathcal{V}}$.

One of the core issues of the BGNN framework is to design a graph generative
model $p(\hat{{\mathcal{G}}}|{\mathcal{G}})$. [16, 17] propose to use the
node-copy model, where for each user, the connected item set is randomly
replaced with the item set from other users based on some predefined user-user
similarity. We argue that this approach lacks the flexibility of generating
novel graphs that extract the intrinsic nature of recommender systems and thus
is doomed to be sub-optimal. In this work, our goal is to propose a systematic
and flexible graph generative model that captures several critical factors for
recommender systems and further boosts the performance of BGNN.

##  3 Methodology

###  3.1 Design Choices for Graph Generative Models

The core problem of designing $p(\hat{{\mathcal{G}}}|{\mathcal{G}})$ is to
define $p(\hat{{\mathbf{R}}}|{\mathcal{G}})$ or
$p(\hat{{\mathbf{A}}}|{\mathcal{G}})$, where $\hat{{\mathbf{R}}}$ and
$\hat{{\mathbf{A}}}$ are the corresponding matrix representation of
$\hat{{\mathcal{G}}}$. In order to extract the intrinsic information from
recommender systems and preserve sufficient flexibility of the generative
model, we consider the model

$\displaystyle
p({\mathbf{R}}|{\mathcal{G}})=\prod_{i}p_{i}({\mathbf{R}}_{i*}|{\mathcal{G}}),$
(1)

where ${\mathbf{R}}_{i*}$ denotes the $i^{th}$ row of the matrix
${\mathbf{R}}$. Intuitively, we model the distribution of the set of items
that each user connects to, and we need to design a proper distribution
$p_{i}({\mathbf{R}}_{i*}|{\mathcal{G}})$ for each user $i$. Besides, we devise
the following three factors that the generated graphs should be consistent
with the original graphs, namely, _user preference_ , _item concurrence_ and
_node degrees_.

####  3.1.1 User preference

Although the graph generative model generates random graphs, we still want the
users to connect to the items that they have potential interests in and avoid
the obvious fake links to suppress noises. Inspired by the stochastic block
model [18, 19], we first cluster the users and items. Then we count the number
of interactions between every pair of user-item clusters in the original
graph. Specifically, for clustering users, we define the pairwise user-user
similarity $d(u,v)$ for $u,v\in{\mathcal{U}}$ as

$\displaystyle
d(u,v)=\frac{\langle{\mathbf{R}}_{u*},{\mathbf{R}}_{v*}\rangle}{|{\mathbf{R}}_{u*}+{\mathbf{R}}_{v*}|_{0}},\quad\forall
u,v\in{\mathcal{U}},$ (2)

where ${\mathbf{R}}_{u*}$ denotes the $u^{th}$ row of matrix ${\mathbf{R}}$,
$\langle\cdot,\cdot\rangle$ denotes dot product and $|\cdot|_{0}$ denotes
$l_{0}$ norm. Then we adopt DB-scan [20], an efficient spectral cluster method
[21], to cluster the users. We adopt a similar approach for items. Denotes
$c_{u}$ as the cluster index for user $u$ and $c_{i}$ as the cluster index for
item $i$. For each pair of user $u$ and item $i$, the user preference of user
$u$ over item $i$ is related to $e(c_{u},c_{i})$ given by

$\displaystyle
e(c_{u},c_{i})=\sum_{v:c_{v}=c_{u}}\sum_{j:c_{j}=c_{i}}{\textnormal{R}}_{vj}.$
(3)

####  3.1.2 Item Concurrence

In recommender systems, certain combinations of items could be frequently
observed in different users’ interactions. To name a few, a user who bought a
TV might probably buy a TV antenna; a user who bought flower seeds is very
likely to buy a flower pot; a user who bought a pen could be interested in
buying a bottle of ink. In our graph generative model, we incorporate such an
item concurrence signal. Specifically, we define a matrix
${\mathbf{S}}^{\mathcal{G}}\in\mathbb{R}^{|{\mathcal{I}}|\times|{\mathcal{I}}|}$
to measure the concurrence of items on the graph ${\mathcal{G}}$ by

$\displaystyle{\textnormal{S}}^{\mathcal{G}}_{ij}=\begin{cases}\frac{\langle{\mathbf{R}}_{*i},{\mathbf{R}}_{*j}\rangle}{|{\mathbf{R}}_{*i}+{\mathbf{R}}_{*j}|_{0}},\quad&\forall
i\neq j,\\\ 0,\quad&\text{otherwise}.\end{cases}$ (4)

where ${\mathbf{R}}_{*i}$ denotes the $i^{th}$ column of matrix
${\mathbf{R}}$, $\langle\cdot,\cdot\rangle$ denotes dot product and
$|\cdot|_{0}$ denotes $l_{0}$ norm. We enforce that
$\mathbb{E}({\mathbf{S}}^{\hat{{\mathcal{G}}}})$ should be close to
${\mathbf{S}}^{\mathcal{G}}$. Although we take a similar definition for the
item-item concurrent matrix $S_{ij}$ with the item-item similarity measure
$d(i,j)$ defined in Sec. 3.1.1, we argue that these are just a well-performed
setting from our experiments. However, we can take different definitions for
$S_{ij}$ and $d(i,j)$ for potential improvement.

####  3.1.3 Node Degree

Node degree is one of the common statistics [22] to characterize the structure
of graphs. We aim to keep a consistent structure for the generated graphs in
that the node degree of each node should be similar to it from the original
graph. Specifically, in the original graph ${\mathcal{G}}$, the node degree
for a node $i\in{\mathcal{V}}$ is

$\displaystyle d^{\mathcal{G}}_{i}=\sum_{j}{\textnormal{A}}_{ij}.$ (5)

In the sampled graph $\hat{{\mathcal{G}}}$, we enforce that

$\displaystyle\mathbb{E}(d^{\hat{{\mathcal{G}}}}_{i})=d^{\mathcal{G}}_{i},\quad\forall
i\in{\mathcal{V}}.$ (6)

###  3.2 Preference and Concurrence (PECO) Aware Bipartite Graph Generation

We propose an efficient iterative heuristic algorithm to implement the
sampling procedure for each user $u$ that takes the design choices in Sec. 3.1
into account. Specifically, we need to sample a set of items for each user. To
control the deviation of the sampled graph from the original graph, we
initialize the set $\hat{S}_{u}$ of items by uniformly randomly sampling a
proportion $r$ of items from its original item set, where $r\in[0,1]$ is some
tune-able hyper-parameter. Then we iteratively sample one item and add it to
$\hat{S}_{u}$ until $|\hat{S}_{u}|=|{\mathcal{N}}(u)|$. The probability of
sampling item $i$ is defined as

$\displaystyle p_{u}(i)\propto q_{u}(i)+\alpha\cdot s_{\hat{S}_{u}}(i),$ (7)

where $q_{u}(i)$ defines a preference probability, $s_{\hat{S}_{u}}(i)$
defines the concurrence score of the item $i$ against the sampled set
$\hat{S}_{u}$, and $\alpha$ is some trade-off hyper-parameter controlling how
much item concurrence signal we want to add in the sampled graphs.
Specifically, $q_{u}(i)$ is a normalized value of

$\displaystyle q_{u}(i)\propto\frac{e(c_{u},c_{i})}{|\\{j|c_{j}=c_{i}\forall
j\in{\mathcal{I}}\\}|}d_{i}^{\mathcal{G}},$ (8)

where $|\\{j|c_{j}=c_{i}\forall j\in{\mathcal{I}}\\}|$ is the size of cluster
$c_{i}$, $e(c_{u},c_{i})$ is defined in (3) and $d_{i}^{\mathcal{G}}$ is the
node degree of item $i$ in graph ${\mathcal{G}}$.
$s_{\hat{S}_{u}}(i)=\frac{1}{|\hat{S}_{u}|}\sum_{j\in\hat{S}_{u}}{\textnormal{S}}_{ij}$
is the similarity measure between item $i$ and the sampled set $\hat{S}_{u}$.
To sample a graph $\hat{{\mathcal{G}}}=\\{\hat{{\mathcal{V}}},\hat{gE}\\}$, we
initialize the graph with the same nodes as the original graph
${\mathcal{G}}$. We sample edges for each of the users and combine all the
edges as the edge set $\hat{{\mathcal{E}}}$. Algorithm 1 presents the detailed
procedure.

Algorithm 1 PECO Bipartite Graph Generation

Initialize $\hat{{\mathcal{G}}}=\\{\hat{{\mathcal{V}}},\hat{{\mathcal{E}}}\\}$
with $\hat{{\mathcal{V}}}\leftarrow{\mathcal{V}}$ and
$\hat{{\mathcal{E}}}\leftarrow\emptyset$

For $u$ in $\hat{{\mathcal{U}}}$

Initialize $\hat{S}_{u}$ with $r|{\mathcal{N}}(u)|$ uniformly randomly sampled
items from ${\mathcal{N}}(u)$

While $|\hat{S}_{u}|<|{\mathcal{N}}(u)|$

sample $i$ according to (7)

$\hat{S}_{u}\leftarrow\hat{S}_{u}\cup\\{i\\}$

$\hat{{\mathcal{E}}}\leftarrow\hat{{\mathcal{E}}}\cup\\{(u,i)|\forall
i\in\hat{S}_{u}\\}$

return $\hat{{\mathcal{G}}}$

##  4 Experiments

###  4.1 Experiment Setting

Datasets. We evaluate and compare our proposed algorithms with baselines on
four public datasets of various domains, sizes and connection densities:
Amazon-Beauty, MovieLens-1m, yelp2018, and Amazon-CDs. Amazon-Beauty and
Amazon-CDs are subsets of Amazon-review, a popular dataset for product
recommendations. MovieLens-1m contains ratings for movies. The yelp dataset is
a subset of Yelp’s businesses, reviews, and user data. For each user, we
reserve 20% of the ground-truth interacted items as validation and testing
sets respectively, and adopt the remaining 60% data as the training set. Table
1 summarizes the statistics of all the datasets.

Table 1: Statistics of public datasets. Dataset | # Users | # Items | # Interacts | Density
---|---|---|---|---
Amazon-Beauty | 7068 | 3750 | 70506 | 0.299%
MovieLens-1m | 6034 | 3247 | 574631 | 2.932%
yelp2018 | 45919 | 45538 | 930030 | 0.044%
Amazon-CDs | 43169 | 35648 | 777426 | 0.051%

Evaluation Metrics. For all experiments, we evaluate the recommendation
accuracy by recall and NDCG for the 20 top-rated items.

Baselines. To demonstrate the effectiveness, we compare our proposed
algorithms with the following methods:

  1. 1.

Multi-Graph Convolution Collaborative Filtering: Multi-GCCF [13]

  2. 2.

Baseline of graph convolutional neural network: LightGCN [6]

  3. 3.

Bayesian graph neural network with node-copying graph-generative model: Node-
Copy [16]

Note that LightGCN is one of the SOTA models for GNN-based recommender systems
[23, 24, 11, 12, 25]. Multi-GCCF is the backbone GNN model for both the BGNN
and our PECO.

Hyper-parameters. Table 2 shows the important hyper-parameters we adopt for
different datasets. We select those hyper-parameters based on the best
validation performance.

###  4.2 Discussion on the Results

Main results. In Table 3, we list the performances of 4 trials with different
random seeds. We can see that our proposed PECO outperforms all the other
algorithms across all datasets. Specifically, both Node-Copy and PECO are
based on Bayesian graph neural networks (BGNN), and they consistently perform
better than their backbone model Multi-GCCF, which indicates the effectiveness
of the BGNN. Besides, our PECO robustly outperforms Node-Copy, since our
generative model covers a wider span of graph generative models and our
proposed design choices can effectively capture the characteristics of
recommender systems. Interestingly, from the hyper-parameter settings in Table
2, the optimal setting of $\alpha$ is different from dataset to dataset. The
MovieLens-1m dataset does not benefit from the item concurrence signal while
the Amazon-Beauty dataset favours a generative model with a strong item
concurrence signal. Our design of the generative model is flexible to adapt to
the different datasets and boost the final performance.

Statistics of sampled graphs. We analyze how the generated graphs preserve the
proposed design choices in Sec. 3.1. Figure 1 depicts the item degree
distribution for different graphs. Note that our PECO will always result in
the exact user degree distribution since we sample the same number of item
nodes for each user as in the observed graph.

Table 2: Important hyper-parameter settings for PECO. Dataset | learning rate | # epochs | $\alpha$ | r
---|---|---|---|---
Amazon-Beauty | 0.01 | 500 | 1000 | 0
MovieLens-1m | 0.001 | 500 | 0 | 0
yelp2018 | 0.002 | 500 | 100 | 0.5
Amazon-CDs | 0.0025 | 400 | 10 | 0
Table 3: The overall performance comparison. Amazon-Beauty | Recall@20 | NDCG@20
---|---|---
Multi-GCCF | 0.1194 | 0.0654
LightGCN | 0.1109 | 0.0621
Node-Copy | 0.1247 | 0.0701
PECO | 0.1265 | 0.0706
MovieLens-1m | Recall@20 | NDCG@20
Multi-GCCF | 0.2212 | 0.2239
LightGCN | 0.2238 | 0.2319
Node-Copy | 0.2223 | 0.2258
PECO | 0.2245 | 0.2280
yelp2018 | Recall@20 | NDCG@20
Multi-GCCF | 0.0804 | 0.0481
LightGCN | 0.0717 | 0.0429
Node-Copy | 0.0812 | 0.0485
PECO | 0.0820 | 0.0489
Amazon-CDs | Recall@20 | NDCG@20
Multi-GCCF | 0.1135 | 0.0683
LightGCN | 0.1055 | 0.0636
Node-Copy | 0.1167 | 0.0705
PECO | 0.1172 | 0.0708
Fig. 1: The item degree distribution. Item indices are sorted according to the
item degree in the original graph.

##  5 Conclusion

In this work, we propose a bipartite graph generation algorithm that exploits
the preferences of users and the concurrence relationships of items. We
applied it to Bayesian Graph Neural Networks replacing the node copying graph
generative model. From our extensive experiments, our proposed method
demonstrated consistent recommendation accuracy improvement over the node-
copying algorithm and other benchmarks for four public benchmark datasets.
Potential future research directions include more sophisticated node
classification algorithms and learning the trade-off parameter $\alpha$.

## References

  * [1] Paul Covington, Jay Adams, and Emre Sargin,  “Deep neural networks for youtube recommendations,”  in Proceedings of the 10th ACM conference on recommender systems, 2016, pp. 191–198.
  * [2] Paul Resnick and Hal R Varian,  “Recommender systems,”  Communications of the ACM, vol. 40, no. 3, pp. 56–58, 1997.
  * [3] Yehuda Koren, Robert Bell, and Chris Volinsky,  “Matrix factorization techniques for recommender systems,”  Computer, vol. 42, no. 8, pp. 30–37, 2009.
  * [4] Yehuda Koren,  “Factorization meets the neighborhood: a multifaceted collaborative filtering model,”  in Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, 2008, pp. 426–434.
  * [5] Thomas Hofmann,  “Latent semantic models for collaborative filtering,”  ACM Transactions on Information Systems (TOIS), vol. 22, no. 1, pp. 89–115, 2004.
  * [6] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang,  “Lightgcn: Simplifying and powering graph convolution network for recommendation,”  in Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, 2020, pp. 639–648.
  * [7] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He,  “Deepfm: a factorization-machine based neural network for ctr prediction,”  in Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2017, pp. 1725–1731.
  * [8] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst,  “Convolutional neural networks on graphs with fast localized spectral filtering,”  Advances in neural information processing systems, vol. 29, 2016\.
  * [9] Will Hamilton, Zhitao Ying, and Jure Leskovec,  “Inductive representation learning on large graphs,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [10] Max Welling and Thomas N Kipf,  “Semi-supervised classification with graph convolutional networks,”  in J. International Conference on Learning Representations (ICLR 2017), 2016.
  * [11] Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui,  “Graph neural networks in recommender systems: a survey,”  ACM Computing Surveys (CSUR), 2020.
  * [12] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec,  “Graph convolutional neural networks for web-scale recommender systems,”  in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 974–983.
  * [13] Jianing Sun, Yingxue Zhang, Chen Ma, Mark Coates, Huifeng Guo, Ruiming Tang, and Xiuqiang He,  “Multi-graph convolution collaborative filtering,”  in 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 2019, pp. 1306–1311.
  * [14] Ishaan Kumar, Yaochen Hu, and Yingxue Zhang,  “Eflec: Efficient feature-leakage correction in gnn based recommendation systems,”  in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 1885–1889.
  * [15] Yingxue Zhang, Soumyasundar Pal, Mark Coates, and Deniz Ustebay,  “Bayesian graph convolutional neural networks for semi-supervised classification,”  in Proceedings of the AAAI conference on artificial intelligence, 2019, vol. 33, pp. 5829–5836.
  * [16] Jianing Sun, Wei Guo, Dengcheng Zhang, Yingxue Zhang, Florence Regol, Yaochen Hu, Huifeng Guo, Ruiming Tang, Han Yuan, Xiuqiang He, and Mark J. Coates,  “A framework for recommending accurate and diverse items using bayesian graph convolutional neural networks,”  Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
  * [17] Florence Regol, Soumyasundar Pal, Jianing Sun, Yingxue Zhang, Yanhui Geng, and Mark Coates,  “Node copying: A random graph model for effective graph sampling,”  Signal Processing, vol. 192, pp. 108335, 2022.
  * [18] Paul W. Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt,  “Stochastic blockmodels: First steps,”  Social Networks, vol. 5, no. 2, pp. 109–137, 1983.
  * [19] Wenzhe Li, Sungjin Ahn, and Max Welling,  “Scalable mcmc for mixed membership stochastic blockmodels,”  in Artificial Intelligence and Statistics. PMLR, 2016, pp. 723–731.
  * [20] Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu,  “Dbscan revisited, revisited: why and how you should (still) use dbscan,”  ACM Transactions on Database Systems (TODS), vol. 42, no. 3, pp. 1–21, 2017.
  * [21] Jialu Liu and Jiawei Han,  “Spectral clustering,”  in Data clustering, pp. 177–200. Chapman and Hall/CRC, 2018.
  * [22] Xiaojie Guo and Liang Zhao,  “A systematic survey on deep generative models for graph generation,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [23] Rianne van den Berg, Thomas N Kipf, and Max Welling,  “Graph convolutional matrix completion,”  arXiv preprint arXiv:1706.02263, 2017.
  * [24] Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua,  “Neural graph collaborative filtering,”  in Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, 2019, pp. 165–174.
  * [25] Jiani Zhang, Xingjian Shi, Shenglin Zhao, and Irwin King,  “Star-gcn: Stacked and reconstructed graph convolutional networks for recommender systems,”  in IJCAI, 2019.
  * [26] Nidhi Kashyap and Archana Patel,  “An efficient and scalable density-based clustering algorithm for normalize data,”  Procedia Computer Science, vol. 92, pp. 136–141, 12 2016.

We appreciate the insightful and thoughtful feedback from the reviewers. We
are encouraged that they find our work to be an important research problem
(R1, 3, 4), a novel task (R1, 3), with correct technical measures (R1, 3, 4)
and valid experiments (R1). We address the reviewers’ comments below.

——-Reply to Reviewer #1——-

Comment 1: DB-scan can be replaced by other unsupervised classification
methods like K-Means. DB-scan turns out to outperform other clustering
algorithms with spectral feature embedding as the input.

Comment 2: Although the improvement seems marginal, it is observed
consistently across different datasets. We analyzed the standard deviation and
found that the improvement was statistically significant.

Dataset | Algorithms | Recall@20 | Recall std | NDCG@20 | NDCG std
---|---|---|---|---|---
Amazon-Beauty | Multi-GCCF | 0.1194 | 0.0016 | 0.0654 | 0.0005
| LightGCN | 0.1109 | 0.0016 | 0.0621 | 0.0014
| Node-Copy | 0.1247 | 0.0005 | 0.0701 | 0.0004
| PECO | 0.1265 | 0.0013 | 0.0706 | 0.0007
MovieLens-1m | Multi-GCCF | 0.2212 | 0.0004 | 0.2239 | 0.0007
| LightGCN | 0.2238 | 0.0006 | 0.2319 | 0.0003
| Node-Copy | 0.2223 | 0.0018 | 0.2258 | 0.0012
| PECO | 0.2245 | 0.0009 | 0.2280 | 0.0005
yelp2018 | Multi-GCCF | 0.0804 | 0.0002 | 0.0481 | 0.0002
| LightGCN | 0.0717 | 0.0003 | 0.0429 | 0.0003
| Node-Copy | 0.0812 | 0.0003 | 0.0485 | 0.0001
| PECO | 0.0820 | 0.0003 | 0.0489 | 0.0001
Amazon-CDs | Multi-GCCF | 0.1135 | 0.0010 | 0.0683 | 0.0009
| LightGCN | 0.1055 | 0.0007 | 0.0636 | 0.0004
| Node-Copy | 0.1167 | 0.0005 | 0.0705 | 0.0005
| PECO | 0.1172 | 0.0004 | 0.0708 | 0.0002

Comment #3: Being noisy does not necessarily mean that the nodes have extra
connections, as there could be missing links. Our empirical experiments showed
that maintaining the node degree improves prediction performance.

——-Reply to Reviewer #3——-

Comment 1: The definitions of user-user similarity in (2) and item concurrence
in (3) adopt the Jaccard index, a common practice of regarding the number of
common neighbours of two nodes divided by the total number of their neighbours
as the score of similarity. Using the Jaccard index matches the design of our
sampling algorithm, i.e., it captures the concurrence of the nodes. We will
give more justifications in our next edition.

Comment 2: Although the spectral clustering method is computationally
expensive in general, we selected DB-scan due to its efficiency.[26]

Comment 3: The random sampling decides the order of adding items according to
the probabilities in equations 7 and 8. Even if some items with larger
probabilities are not selected because of a full set, it will not influence
the overall performance as indicated by empirical experiments since
statistically, more important items with larger probabilities will be selected
more frequently in expectation.

Comment 4: Please refer to Reviewer #1 Comment 2. We will add the standard
deviation in the revised version.

——-Reply to Reviewer #4——-

Comment #1: We proposed a new graph sampling algorithm with several advantages
over other methods and consistently outperforms them on four popular benchmark
datasets. Our proposed algorithm captures the item concurrence relationship,
which is important as some items tend to appear together, such as pen and ink.
In addition, we proposed an approximation algorithm to reduce the computation
complexity of the full model without degrading the performance.

Comment #2: Industrial data contains false positive edges and missing
potentially positive edges. In the recommendation scenarios, users might click
some content by mistake or ignore his/her interested content by chance, etc.
These are some concrete examples of the noises. We will describe it more
clearly in the revised version,

Comment #3: Although the improvement seems marginal, it is observed
consistently across different datasets. As requested by Reviewer #3, we
analyzed the standard deviation and found that the improvement was
statistically significant (cf the Reviewer#1 Comment 2). We did not add too
much ablation study in the current version due to the page limit, but we
provided sufficient information for reproducing our results. Besides, all the
proposed modules were empirically verified to be effective. We could devise
and add more results in a revised version.


# Do Neural Topic Models Really Need Dropout?
Analysis of the Effect of Dropout in Topic Modeling

Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal
Indian Association for the Cultivation of Science, Jadavpur, Kolkata-700032,
India
{ adhyasuman30, avisheklahiri2014, debarshisanyal<EMAIL_ADDRESS>
###### Abstract

Dropout is a widely used regularization trick to resolve the overfitting issue
in large feedforward neural networks trained on a small dataset, which
performs poorly on the held-out test subset. Although the effectiveness of
this regularization trick has been extensively studied for convolutional
neural networks, there is a lack of analysis of it for unsupervised models and
in particular, VAE-based neural topic models. In this paper, we have analyzed
the consequences of dropout in the encoder as well as in the decoder of the
VAE architecture in three widely used neural topic models, namely,
contextualized topic model (CTM), ProdLDA, and embedded topic model (ETM)
using four publicly available datasets. We characterize the dropout effect on
these models in terms of the quality and predictive performance of the
generated topics.

##  1 Introduction

Dropout Hinton et al. (2012) is used while training neural networks, by
stochastically dropping out the activation of neurons to prevent complex co-
adaptations of feature vectors Baldi and Sadowski (2013). The working of
dropout is attributed to the implicit averaging over an ensemble of neural
networks Labach et al. (2019); Warde-Farley et al. (2014). It has been shown
to be effective on supervised learning tasks to prevent overfitting Srivastava
et al. (2014).

As the volume of digital documents significantly increases with time,
organizing them manually is becoming quite an inconvenient task. Because of
the ability of topic models to learn a thematic structure from a set of
documents in an unsupervised manner and label the documents with their
corresponding dominant topics, the significance of topic models is enormous in
this area Hall et al. (2008); Adhya and Sanyal (2022). But in the traditional
topic models, not only the computation cost of the approximate posterior is
very high but also for a small change in the modeling assumption, re-
derivation of the inference method is needed. With greater flexibility and
scalability than traditional topic models, a class of Neural Topic Models
(NTMs) aim to leverage the potential of neural networks using the AEVB Kingma
and Welling (2014) based inference technique. Following Zhao et al. (2021), we
refer to this class of models as VAE-NTMs where the training objective is to
maximize the log-likelihood of the reconstruction of the input document while
minimizing the KL-divergence of the learned posterior distribution of the
latent space from a known prior distribution.

An earlier study by Ha et al. (2019) of the dropout effect on two traditional
topic models LDA Blei et al. (2003) and BTM Yan et al. (2013) shows that the
correct choice of the dropout rate not only decreases the learning time of the
models but also significantly improves the predictive performance and
generalization for short texts. However, the study does not consider neural
topic models.

In this work, we propose the use of dropout on VAE-NTMs as a hyperparameter in
order to achieve much better performance in terms of topic coherence, topic
diversity, and topic quality. We test this proposition on a range of standard
VAE-NTM architectures. To the best of our knowledge, there has been no other
study focusing specifically on the use of dropout in neural topic models. We
have made our analysis publicly
available111https://github.com/AdhyaSuman/NTMs_Dropout_Analysis.

In summary, our contributions are as follows:

  1. 1.

We comprehensively show both quantitatively and qualitatively that topic
quality undergoes a massive improvement with either very low or zero dropout
settings in both the encoder and the decoder of a VAE-NTM.

  2. 2.

We show that for VAE-NTMs the systematic choice of low dropout rates can lead
to a significant improvement in downstream tasks like document classification.

  3. 3.

We study the dependence of dropout on the length of the input documents.

  4. 4.

We present an empirical analysis for the increase in performance of VAE-NTMs
with a decrease in dropout.

##  2 Task Formulation

Given a corpus $\\{D_{1},D_{2},\ldots,D_{N}\\}$ of $N$ documents with
vocabulary $\\{w_{1},w_{2},...,w_{V}\\}$ of $V$ words, topic models describe a
document $D_{i}$ as a distribution over $K$ topics
$\\{\bm{\mathbf{\beta}}_{1},\bm{\mathbf{\beta}}_{2},...,\bm{\mathbf{\beta}}_{K}\\}$,
where an individual topic $\bm{\mathbf{\beta}}_{k}$ is a distribution over
$V$-words.

###  2.1 VAE Framework in Neural Topic Models

Figure 1: VAE framework in neural topic models.

Given an input sample $\bm{\mathbf{x}}$, a VAE encoder learns the approximate
posterior distribution $q_{W}(\bm{\mathbf{z}}|\bm{\mathbf{x}})$ where $W$ is
the encoder’s weights that are to be learned and $\bm{\mathbf{z}}$ is a latent
variable. Given a sample $\bm{\mathbf{z}}\thicksim
q_{W}(\bm{\mathbf{z}}|\bm{\mathbf{x}})$, the VAE decoder learns the likelihood
$p_{W^{\prime}}(\bm{\mathbf{x}}|\bm{\mathbf{z}})$ where $W^{\prime}$ is the
learnable decoder’s weights.

In VAE-NTMs the input to the encoder is a document representation (e.g., bag-
of-words) $\bm{\mathbf{x}}_{V\times 1}$. The encoder then returns the Gaussian
parameters $\left(\bm{\mathbf{\mu}}_{K\times 1},\bm{\mathbf{\Sigma}}_{K\times
1}\right)$ that approximate the true posterior where $K$ is the dimension of
latent (topic) space, $\bm{\mathbf{\mu}}_{K\times 1}$ is the mean, and
$\bm{\mathbf{\Sigma}}_{K\times 1}$ is the diagonal covariance matrix. Upon
taking these Gaussian parameters as input, the decoder samples a latent
representation $\bm{\mathbf{z}}_{K\times 1}$ from
$\mathcal{N}(\bm{\mathbf{\mu}}_{K\times 1},\bm{\mathbf{\Sigma}}_{K\times 1})$
using the reparametrization trick as follows:

$\bm{\mathbf{z}}_{K\times 1}=\bm{\mathbf{\mu}}_{K\times
1}+\bm{\mathbf{\Sigma}}^{\frac{1}{2}}_{K\times
1}\odot\bm{\mathbf{\epsilon}}_{K\times 1}$

where $\bm{\mathbf{\epsilon}}_{K\times
1}\thicksim\mathcal{N}(\bm{\mathbf{0}},\bm{\mathbf{I}})$ and $\odot$
represents the element-wise product. Then the document-topic distribution
vector ($\bm{\mathbf{\theta}}_{K\times 1}$) is generated such that
$\bm{\mathbf{\theta}}_{K\times 1}=\sigma(\bm{\mathbf{z}}_{K\times 1})$ where
$\sigma(\cdot)$ is a softmax function. The input document-term distribution
vector is reconstructed with the product of $\bm{\mathbf{\theta}}_{K\times 1}$
and $\bm{\mathbf{\beta}}_{K\times V}$, the topic-word matrix, in the following
manner:

${\tilde{\bm{\mathbf{x}}}}_{V\times
1}=\begin{cases}\bm{\mathbf{\beta}}^{T}\bm{\mathbf{\theta}}&\text{if
}\bm{\mathbf{\beta}}\text{ is normalized.}\\\
\sigma\left(\bm{\mathbf{\beta}}^{T}\bm{\mathbf{\theta}}\right)&\text{if
}\bm{\mathbf{\beta}}\text{ is unnormalized.}\end{cases}$

As shown in Figure 1, in the encoder, dropout is applied with probability
$E_{p}$ on the output of the hidden layer(s) of the multi-layer feed-forward
neural network (FFNN). This output is then fed to two separate layers to get
the approximate posterior $q_{W}(z|x)$. In the decoder, dropout is applied
with probability $D_{p}$ on the document-topic distribution vector
($\bm{\mathbf{\theta}}_{K\times 1}$), just before the reconstruction process.

###  2.2 Task Description

The goal is to measure the effect that dropout has on the performance of VAE-
NTMs by varying the dropout rates from $0.0$ to $0.6$ in steps of $0.1$, in
both the encoder and the decoder. We have chosen $0.6$ as the upper bound of
the dropout rates for our experiments because it is the highest dropout rate
used in any VAE-NTMs that we have considered as a baseline in this work. We
measure performance using: topic coherence, topic diversity, and topic
quality. We use NPMI Lau et al. (2014); Röder et al. (2015) to measure topic
coherence. Topic diversity Dieng et al. (2020) shows the uniqueness of topics.
Topic quality is the product of coherence and diversity Dieng et al. (2020).
As the automated topic model measures do not always accurately capture the
quality of the topics Hoyle et al. (2021), we also perform a manual evaluation
of the topics and study their predictive performance on the document
classification task.

Figure 2: Topic quality and NPMI for different topic models with optimal
dropout rate and default dropout rate.

##  3 Empirical Study

We perform all experiments in OCTIS Terragni et al. (2021), which is an
integrated framework for topic modeling.

###  3.1 Datasets

We have used four publicly available datasets in our experiments. Among them,
20NG222http://qwone.com/~jason/20Newsgroups/ and BBC Greene and Cunningham
(2006) are already available in OCTIS in the pre-processed format while we
added Wiki40B Guo et al. (2020) and AllNews Zhu et al. (2018) datasets
further. The statistical descriptions of these datasets are mentioned in Table
1. Each corpus is split into train/valid/test sets in the ratio $70\colon
15\colon 15$. The validation set is used for early stopping.

Dataset #Docs Avg. #words |Vocab| 20NG $16309$ $48.02$ $1612$ BBC $2225$
$120.12$ $2949$ Wiki40B $24774$ $541.08$ $2000$ AllNews $49754$ $229.53$
$2000$

Table 1: Statistics of the used datasets.

###  3.2 Models

We use the following three VAE-NTMs: CTM Bianchi et al. (2021) which
incorporates the contextualized documents embeddings with the neural topic
models; ProdLDA Srivastava and Sutton (2017) which, unlike LDA, relaxes the
simplex constraint over the topic-word matrix; (ETM) Dieng et al. (2020) which
incorporates word-embeddings in topic modeling to increase robustness in
presence of stopwords.

Model 20NG BBC Wiki40B AllNews CTM $(0.2,\>0.2)$ $(0.0,\>0.0)$ $(0.0,\>0.0)$
$(0.2,\>0.1)$ $(0.0,\>0.1)$ ProdLDA $(0.6,\>0.6)$ $(0.1,\>0.1)$ $(0.0,\>0.0)$
$(0.1,\>0.1)$ $(0.1,\>0.1)$ ETM $(0.5,\>0.0)$ $(0.0,\>0.0)$ $(0.1,\>0.0)$
$(0.0,\>0.0)$ $(0.1,\>0.0)$

Table 2: For each of the datasets, the optimal dropout rates of all the models
considering the highest topic quality are mentioned in the $({E_{p}},{D_{p}})$
format in the second through last columns. The default dropout rate is also
specified for each model in the first column.

For each of the four datasets, we compute the dropout rate that optimizes the
topic quality of each model on that dataset. We train all three topic models
for topic-count $K\in\\{20,50,100\\}$ with 30 epochs while keeping all
hyperparameter values, except dropout, the same as in their original
implementations. To ensure robustness, we average scores over 10 independent
runs of each model. For comparison, we use the default dropout rates for each
model as mentioned in the original papers that proposed the corresponding
model. In Table 2, we show the default and the optimal dropout rates.

Figure 3: Topic qualities on 20NG for
$(E_{p},D_{p})\in[0.0,\;0.6]\times[0.0,\;0.6]$ with a increment of $0.1$.

###  3.3 Results and Analysis

####  3.3.1 Quantitative Evaluation of Topic Quality

In Figure 2, we compare, for each dataset and each model, the topic quality
and the NPMI respectively between the dropout-optimized model that gives the
highest topic quality and the model with default dropout rates as mentioned in
Table 2.

On 20NG, the topic quality score for (CTM, ProdLDA, ETM) is improved from
$(0.056,-0.051,0.004)$ to $(0.065,0.039,0.009)$ by optimizing the dropout
rate. For CTM, the increase in performance is around $16.07\%$ whereas for the
other two models it is over $100\%$. This is because the original
implementation of CTM already uses a relatively low dropout rate, i.e., $0.2$,
for both the encoder and the decoder. The other two models show a significant
increase in performance due to their large dropout in the baseline models.

Figure 3 shows that the topic quality on the 20NG dataset for the VAE-NTMs
generally produces better results on keeping the dropout rate for both the
encoder and the decoder either to be zero or close to it, especially values
like $\\{0.0,0.1\\}$. Similar results have been found for the other datasets.
Based on these observations, the topic quality is found to reduce with an
increase in dropout rates in the encoder and decoder.

####  3.3.2 Qualitative Evaluation of Topic Quality

To qualitatively evaluate the models, we trained all of them for a topic count
of $100$ on the 20NG dataset. We then aligned the topics for each pair of
(optimal-dropout model, default-dropout model) for all three different models.
We followed a two-step strategy for topic alignment. For a given pair of
models, namely, one with optimal dropout and another with default dropout,
with topics lists $P$ and $Q$, respectively, we first construct a similarity
matrix of the topic lists using Rank-biased Overlap Webber et al. (2010) (RBO)
which computes the similarity between two ordered lists by taking into
consideration the rank of the individual elements. For example, for $100$
topics, we get a matrix, $\mathbf{A}=(a_{ij})_{1\leq i,j\leq 100}$ such that,
$a_{i,j}=\operatorname{RBO}\big{(}P[i],Q[j]\big{)}$. The RBO score lies in
$[0,1]$, where $0$ represents no overlap and $1$ implies exact overlap. In the
final step, we iteratively select the pair of topics for which the similarity
score is maximum and simultaneously exclude these two topics from further
consideration, i.e. if $\big{(}P[i_{1}],Q[j_{1}]\big{)}$ and
$\big{(}P[i_{2}],Q[j_{2}]\big{)}$ are two selected pairs then $\left(i_{1}\neq
i_{2}\land j_{1}\neq j_{2}\right)$.

In Table 3 we show the top words from aligned topics of all the models.
‘$\ast$’ marked models have dropout optimized to give the highest topic
quality while others use the default dropout rates as mentioned in Table 2. We
see that dropout-optimized models output more interpretable topics.

Model Topics CTM* ${(0.0,0.0)}$ monitor, card, video, port, vga, apple,
connector, serial, slot, output firearm, weapon, dangerous, military, license,
file, state, gun, police, issue christian, truth, scripture, exist, belief,
accept, understand, word, human, doctrine CTM ${(0.2,0.2)}$ card, monitor,
video, offer, sale, upgrade, mouse, vga, port, parallel firearm, dangerous,
license, weapon, section, file, division, device, manufacture, carry
interpretation, truth, scripture, christian, agree, moral, understand, human,
faith, claim ProdLDA* ${(0.1,0.1)}$ window, driver, mode, run, mouse, session,
server, program, manager, install car, engine, buy, company, vehicle, make,
brake, tire, dealer, road signal, voltage, output, circuit, noise, power,
switch, wire, connector, degree ProdLDA ${(0.6,0.6)}$ line, window, gun, read,
space, run, statement, datum, drive, make make, battery, engine, homosexual,
assault, reason, place, single, large, attempt voltage, damn, signal, usual,
label, hour, bio, leg, bullet, hundred ETM* ${(0.0,0.0)}$ version, software,
program, file, include, image, application, set, server, support armenian,
turkish, village, people, israeli, population, muslim, kill, russian, genocide
system, run, work, window, problem, include, set, good, support, information
ETM ${(0.5,0.0)}$ file, application, set, program, support, image, display,
list, version, bit armenian, turkish, village, israeli, population, muslim,
genocide, son, land, jewish work, call, system, window, problem, bit, set,
run, support, good

Table 3: Some selected topics among 100 topics from 20NG. ‘*’ indicates models
with optimal dropout. The dropout rate is mentioned in the $(E_{p},D_{p})$
format. The more related words in a topic are highlighted in bold while less
related ones are italicized.

####  3.3.3 Effect of Dataset Length

Among the input datasets on which we have experimented, the 20NG dataset
contains relatively short texts, while the others contain longer texts. Ha et
al. (2019) find that their dropout methods are not effective on long texts.
But here we see that the performance of all VAE-NTMs decreases uniformly with
the increase in the dropout rate, irrespective of the length of the dataset.

####  3.3.4 Document Classification

Figure 4: Accuracy for different topic models with optimal dropout and default
dropout from Table 2.

We test the predictive performance of the topics produced by the models on a
document classification task. We train the models on 20NG and BBC corpora for
$K$ topics using the training subset. We represent each document as a
$K$-dimensional document-topic vector and train an SVM, which is then tested
on the test subset. We average the accuracy scores over $K\in\\{20,50,100\\}$.
Figure 4 shows that accuracy increases when we use the optimized dropout
rates.

##  4 Theoretical Understanding of Results

Our experiments show that by tuning the dropout carefully, we can achieve a
significant improvement in the performance of VAE-NTMs. Therefore, we argue
that the dropout rate should be treated as an important hyperparameter and
carefully selected based on the choice of the model as well as the dataset,
especially in the case of VAE-NTMs. More precisely, in most cases, low dropout
rates in the encoder and the decoder lead to higher performance than that
achieved for higher dropout rates.

Standard dropout and other types of dropout have been extensively used in
supervised learning techniques Srivastava et al. (2014); Wu and Gu (2015);
Tompson et al. (2015); Devries and Taylor (2017); Cai et al. (2019). The main
prerogative of using dropout in the supervised scenario is to introduce noise
while training so that the model can recognize the outliers in the testing
phase. The drop in performance with high dropout that we see in our
experiments is perhaps due to the fact that we are trying to learn a
generative model of the data. Dropout makes the model robust against
perturbations in the input data and thereby also prevents it from learning the
characteristics of the input distribution accurately. This is probably why we
see a drop in topic coherence and quality. In the case of document
classification, if the topic model is trained with a high dropout, the
document-topic vectors are of poor quality and the classifier gets trained on
these vectors; this results in poor accuracy on the test documents. This
setting is different from the usual supervised learning of neural classifiers
where dropout is introduced directly in the classifier to prevent overfitting.
We intend to analyze these aspects in more depth in the future.

##  5 Conclusion

We present a detailed study of the effect of the dropout rate on VAE-NTMs. We
find that the model performance generally reduces with the increase in dropout
rate in the encoder as well as the decoder.

## Limitations

The following limitations are known and should be considered when applying the
results of this work or relying on them in future studies: (1) Other variants
of dropout can be applied to the VAE-NTMs. (2) Analysis of the dropout effect
may be done for other VAE-NTMs as well. (3) Other downstream tasks may be
formulated for further analysis.

## Acknowledgments

This work is partially supported by the SERB-DST Project CRG/2021/000803
sponsored by the Department of Science and Technology, Government of India at
Indian Association for the Cultivation of Science, Kolkata.

## References

  * Adhya and Sanyal (2022) Suman Adhya and Debarshi Kumar Sanyal. 2022.  What does the Indian Parliament discuss? an exploratory analysis of the question hour in the Lok Sabha.  In _Proceedings of the LREC 2022 workshop on Natural Language Processing for Political Sciences_ , pages 72–78, Marseille, France. European Language Resources Association.
  * Baldi and Sadowski (2013) Pierre Baldi and Peter J Sadowski. 2013.  Understanding dropout.  In _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc.
  * Bianchi et al. (2021) Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021.  Pre-training is a hot topic: Contextualized document embeddings improve topic coherence.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 759–766. Association for Computational Linguistics.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent Dirichlet allocation.  _Journal of Machine Learning Research_ , 3(Jan):993–1022.
  * Cai et al. (2019) Shaofeng Cai, Jinyang Gao, Meihui Zhang, Wei Wang, Gang Chen, and Beng Chin Ooi. 2019.  Effective and efficient dropout for deep convolutional neural networks.  _CoRR_ , abs/1904.03392.
  * Devries and Taylor (2017) Terrance Devries and Graham W. Taylor. 2017.  Improved regularization of convolutional neural networks with cutout.  _CoRR_ , abs/1708.04552.
  * Dieng et al. (2020) Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020.  Topic modeling in embedding spaces.  _Transactions of the Association for Computational Linguistics_ , 8:439–453.
  * Greene and Cunningham (2006) Derek Greene and Pádraig Cunningham. 2006.  Practical solutions to the problem of diagonal dominance in kernel document clustering.  In _Proceedings of the 23rd International Conference on Machine Learning_ , ICML ’06, page 377–384, New York, NY, USA. Association for Computing Machinery.
  * Guo et al. (2020) Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020.  Wiki-40B: Multilingual language model dataset.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2440–2452, Marseille, France. European Language Resources Association.
  * Ha et al. (2019) Cuong Ha, Van-Dang Tran, Linh Ngo Van, and Khoat Than. 2019.  Eliminating overfitting of probabilistic topic models on short and noisy text: The role of dropout.  _International Journal of Approximate Reasoning_ , 112:85–104.
  * Hall et al. (2008) David Hall, Daniel Jurafsky, and Christopher D. Manning. 2008.  Studying the history of ideas using topic models.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 363–371, Honolulu, Hawaii. Association for Computational Linguistics.
  * Hinton et al. (2012) Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012.  Improving neural networks by preventing co-adaptation of feature detectors.  _CoRR_ , abs/1207.0580.
  * Hoyle et al. (2021) Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Lee Boyd-Graber, and Philip Resnik. 2021.  Is automated topic model evaluation broken? the incoherence of coherence.  In _Advances in Neural Information Processing Systems_.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-encoding variational bayes.  In _Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014_.
  * Labach et al. (2019) Alex Labach, Hojjat Salehinejad, and Shahrokh Valaee. 2019.  Survey of dropout methods for deep neural networks.  _CoRR_ , abs/1904.13310.
  * Lau et al. (2014) Jey Han Lau, David Newman, and Timothy Baldwin. 2014.  Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality.  In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 530–539, Gothenburg, Sweden. Association for Computational Linguistics.
  * Röder et al. (2015) Michael Röder, Andreas Both, and Alexander Hinneburg. 2015.  Exploring the space of topic coherence measures.  In _Proceedings of the Eighth ACM International Conference on Web Search and Data Mining_ , WSDM ’15, page 399–408, New York, NY, USA. Association for Computing Machinery.
  * Srivastava and Sutton (2017) Akash Srivastava and Charles Sutton. 2017.  Autoencoding variational inference for topic models.  In _Proceedings of the 5th International Conference on Learning Representations, ICLR 2017_.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958.
  * Terragni et al. (2021) Silvia Terragni, Elisabetta Fersini, Bruno Giovanni Galuzzi, Pietro Tropeano, and Antonio Candelieri. 2021.  OCTIS: Comparing and optimizing topic models is simple! In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 263–270. Association for Computational Linguistics.
  * Tompson et al. (2015) Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. 2015\.  Efficient object localization using convolutional networks.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015_ , pages 648–656. IEEE Computer Society.
  * Warde-Farley et al. (2014) David Warde-Farley, Ian J. Goodfellow, Aaron C. Courville, and Yoshua Bengio. 2014\.  An empirical analysis of dropout in piecewise linear networks.  In _2nd International Conference on Learning Representations, ICLR 2014_.
  * Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel. 2010.  A similarity measure for indefinite rankings.  _ACM Transactions on Information Systems (TOIS)_ , 28(4):1–38.
  * Wu and Gu (2015) Haibing Wu and Xiaodong Gu. 2015.  Towards dropout training for convolutional neural networks.  _Neural Networks_ , 71:1–10.
  * Yan et al. (2013) Xiaohui Yan, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2013.  A biterm topic model for short texts.  In _Proceedings of the 22nd International Conference on World Wide Web_ , WWW ’13, page 1445–1456, New York, NY, USA. Association for Computing Machinery.
  * Zhao et al. (2021) He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray L. Buntine. 2021.  Topic modelling meets deep neural networks: A survey.  _CoRR_ , abs/2103.00498.
  * Zhu et al. (2018) Qile Zhu, Zheng Feng, and Xiaolin Li. 2018.  GraphBTM: Graph enhanced autoencoded variational inference for biterm topic model.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4663–4672, Brussels, Belgium. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Datasets

We run our experiments on the following datasets:

  * •

20NewsGroups (20NG)is a dataset of $18,846$ documents from $20$ different
newsgroups posts. The 20NG dataset is present in OCTIS, so it is already in
pre-processed form. All the documents of this dataset have their corresponding
category type as the document labels. The details about these categories are
mentioned in Table 4.

  * •

BBC News (BBC) Greene and Cunningham (2006) is a dataset of news articles from
BBC. It is also accessible from OCTIS in pre-processed form. The documents of
this dataset are categorized into 5 different categories which are tech,
business, entertainment, sports, and politics. The details of these categories
are mentioned in Table 5.

  * •

Wiki40B Guo et al. (2020) is a Wikipedia text dataset in 40+ languages,
available in TensorFlow dataset format. In our experiment, we take a sample of
$24,774$ English documents from this dataset.

  * •

All the News (AllNews) Zhu et al. (2018) dataset consists of $50,001$ news
articles from $15$ news publishers.

#No. Label #Docs %Docs 1. misc.forsale 861 5.28 2. comp.windows.x 883 5.41 3.
soc.religion.christian 920 5.64 4. talk.religion.misc 521 3.19 5. rec.autos
822 5.04 6. sci.med 866 5.31 7. talk.politics.misc 689 4.22 8.
talk.politics.mideast 828 5.08 9. sci.electronics 867 5.32 10.
rec.sport.hockey 843 5.17 11. rec.sport.baseball 787 4.83 12.
talk.politics.guns 808 4.95 13. sci.crypt 883 5.41 14. comp.sys.mac.hardware
838 5.14 15. comp.sys.ibm.pc.hardware 891 5.46 16. comp.graphics 836 5.13 17.
comp.os.ms-windows.misc 828 5.08 18. alt.atheism 689 4.22 19. sci.space 856
5.25 20. rec.motorcycles 793 4.86

Table 4: 20NG labels with corresponding document counts and percentage of
documents.

#No. Label #Docs %Docs 1. tech 401 18.02 2. business 510 22.92 3.
entertainment 386 17.35 4. sport 511 22.97 5. politics 417 18.74

Table 5: BBC labels with corresponding document counts and percentage of
documents.

###  A.2 Pre-processing Steps

Using OCTIS, we convert each document to lowercase, remove the punctuations,
lemmatize it, filter the vocabulary with the most frequent 2000 terms, filter
words with less than 3 characters, and filter documents with less than 3
words.

###  A.3 Topic Evaluation Metrics

  1. 1.

Coherence metric: This measures how much the top words of the topics are
relevant. Topic coherence $(\operatorname{TC})$ for $K$ topics each of which
contains $n$ top words can be calculated as:

$\operatorname{TC}=\frac{1}{K}\sum_{k=1}^{K}\frac{1}{\prescript{n\mkern-0.5mu}{}{C}_{2}}\sum_{i=1}^{n}\sum_{j=i+1}^{n}f\left(w_{i}^{(k)},w_{j}^{(k)}\right)$

Here, $f(\cdot,\cdot)$ is the Normalized Pointwise Mutual Information or NPMI
Lau et al. (2014) of the words $w_{i}^{(k)}$ and $w_{j}^{(k)}$ appearing in
topic $k$:

$f(w_{i}^{(k)},w_{j}^{(k)})=\frac{\log{\frac{p(w_{i}^{(k)},w_{j}^{(k)})+\epsilon}{p(w_{i}^{(k)})p(w_{j}^{(k)})}}}{-\log{\left(p(w_{i}^{(k)},w_{j}^{(k)})+\epsilon\right)}}$

where, $p(w_{i}^{(k)},w_{j}^{(k)})$ is the probability of the co-occurrence of
the words $w_{i}^{(k)}$ and $w_{j}^{(k)}$ in a boolean sliding window in topic
$k$ and $p(w_{i}^{(k)})$ and $p(w_{j}^{(k)})$ represents the probability of
the occurrence of the individual words in topic $k$. $\epsilon$ is a small
positive constant that is used to avoid zero in the $\log(\cdot)$ function.

  2. 2.

Diversity metric: This measures how much the generated topics are different
from each other. To measure the diversity score we have used the metric Topic
Diversity (TD) Dieng et al. (2020) which is defined as the proportion of the
number of unique words appearing across all topics. It ranges between $[0,1]$
where a value close to $0$ implies repetitive topics and a value near $1$
represents more diversification in the topics.

  3. 3.

Topic quality: This is an overall metric that is defined as the product of the
two metrics NPMI and TD.

In our experiments, we take the top 10 words for each topic (i.e., $n=10$) to
compute NPMI and TD scores.

###  A.4 Detailed Results

The detailed results of our experiments are given in Tables 6, 7, and 8. An
asterisk ($\ast$) against a model in the above tables indicates that it is
trained with the optimal dropout rate, and the absence of an asterisk
indicates that the default dropout rate is used. The default dropout rate for
CTM is taken from Bianchi et al. (2021), for ProdLDA from Srivastava and
Sutton (2017), and for ETM from Dieng et al. (2020).

Model Topic quality for each dataset 20NG BBC Wiki40B AllNews CTM* 0.0652
0.0392 0.0961 0.0958 CTM 0.0556 0.0234 0.0884 0.0854 ProdLDA* 0.0392 0.0254
0.0910 0.0891 ProdLDA -0.0507 -0.0887 0.0336 0.0202 ETM* 0.0092 0.0011 0.0502
0.0276 ETM 0.0036 -0.0003 0.0304 0.0181

Table 6: Topic Quality values for different VAE-NTMs with optimal dropout rate
and default dropout rate (see Table 2). ‘*’ indicates models with optimal
dropout.

Model NPMI for each dataset 20NG BBC Wiki40B AllNews CTM* 0.0896 0.0623 0.1219
0.1218 CTM 0.0774 0.0458 0.1152 0.1153 ProdLDA* 0.0513 0.0367 0.1162 0.1166
ProdLDA -0.0907 -0.1293 0.0662 0.0498 ETM* 0.0331 0.0100 0.0841 0.0605 ETM
0.0183 -0.0033 0.0662 0.0494

Table 7: NPMI values for different VAE-NTMs with optimal dropout rate and
default dropout rate (see Table 2). ‘*’ indicates models with optimal dropout.

Model TD for each dataset 20NG BBC Wiki40B AllNews CTM* 0.7283 0.6295 0.7883
0.7871 CTM 0.7175 0.51 0.7671 0.7409 ProdLDA* 0.7644 0.6902 0.7829 0.7640
ProdLDA 0.5594 0.6861 0.5071 0.4061 ETM* 0.2776 0.1108 0.5973 0.4561 ETM
0.1949 0.0902 0.4599 0.3659

Table 8: Topic Diversity values for different VAE-NTMs with optimal dropout
rate and default dropout rate (see Table 2). ‘*’ indicates models with optimal
dropout.

(a) BBC

(b) Wiki40B

(c) AllNews

Figure 5: Change in topic quality for
$(E_{p},D_{p})\in[0.0,\;0.6]\times[0.0,\;0.6]$ with a increment of $0.1$.

(a) 20NG

(b) BBC

Figure 6: Accuracy scores for $(E_{p},D_{p})\in[0.0,\;0.6]\times[0.0,\;0.6]$
(step $=0.1$) in document classification task.