for Topic Modeling on Microtexts with Word Vectors

1]Trung Trinh
1]Tho Quan
1]Trung Mai

Trinh et al

[1]Faculty of Computer Science and Engineering, , Ho Chi Minh, Vietnam

Tho Quan, Faculty of Computer Science and Engineering, , Ho Chi Minh, Vietnam.

[Summary]Most of the information on the Internet is represented in the form of microtexts, which are short text snippets such as news headlines or tweets.
These sources of information are abundant, and mining these data could uncover meaningful insights.
Topic modeling is one of the popular methods to extract knowledge from a collection of documents; however, conventional topic models such as latent Dirichlet allocation (LDA) are unable to perform well on short documents, mostly due to the scarcity of word co-occurrence statistics embedded in the data.
The objective of our research is to create a topic model that can achieve great performances on microtexts while requiring a small runtime for scalability to large datasets.
To solve the lack of information of microtexts, we allow our method to take advantage of word embeddings for additional knowledge of relationships between words.
For speed and scalability, we apply autoencoding variational Bayes, an algorithm that can perform efficient black-box inference in probabilistic models.
The result of our work is a novel topic model called the , which is a distribution that takes into account word vectors and is parameterized by a neural network architecture.
For optimization, the model is trained to approximate the posterior distribution of the original LDA model.
Experiments show the improvements of our model on microtexts as well as its runtime advantage.

§ INTRODUCTION

The ubiquity of microtexts, which is due to the emergence of social media sites such as Facebook and Twitter, has become an increasingly valuable asset for mining information about the real world. In health care, by monitoring information posted by users on social networks, one can observe the status of public health <cit.>. nEmesis <cit.> is a system deployed in combination with social media to prevent foodborne illness. On a broader scale, social platform data can be used to observe public ideas <cit.> and offer emergency service <cit.>. The RSC system <cit.> is a system that monitors temporal human activities. More sophisticated mining tasks can be employed to detect fake news <cit.> or to detect themes on social media <cit.>.

Many of these applications could be distilled to inferring topics from these sources of information. Probabilistic models such as probabilistic latent semantic analysis (PLSA) <cit.> and latent Dirichlet allocation (LDA) <cit.> have been successfully applied to long texts. These models operate on the assumption that each document comprises a small number of topics, each of which in turn consists of a subset of words. The topic proportions of each document as well as the distribution over the vocabulary of each topic are then learned from the corpus using statistical methods such as Gibbs sampling <cit.> or variational inference <cit.>. The effectiveness of these models strongly depends on the patterns of word cooccurrences within the corpus, which is fully and correctly represented in a large corpus of long texts. The sparsity of microtexts, on the other hand, has proven to be inadequate in presenting the relationship between words. Thus, microtexts remain challenging for these conventional topic modeling methods. One way to alleviate this problem is to introduce additional information that could help the model to uncover the true semantic relationships between words. Such methods include utilizing search results for matching similar text snippets <cit.>, using a  knowledgebase for microtext conceptualization <cit.>, and leveraging auxiliary long texts to enhance microtext clustering performance <cit.>.

Recently, the introduction of word embeddings <cit.> has led to improvements in many natural language processing (NLP) tasks due to their ability to capture semantic relationships between words in a distributed fashion. In this model, each word is represented using a dense vector that contains the semantic and syntactic information about that word, and similar words tend to stay close to each other in the embedding space. One could expect that these latent features of words could be used to improve the performance of topic modeling on microtexts. In fact, there have been some studies that incorporate word vectors with available topic modeling methods such as LDA and have shown promising results <cit.>. While the improvements are remarkable, these papers use Gibb sampling as the method to infer the parameters of the model, and although it has been proven theoretically to produce the best results compared to other methods, it requires a long time to converge.

With the advancement of deep learning in recent years, these methods have been used to replace the traditional mean field approximator in variational inference, which has led to the invention of variational autoencoders (VAEs) <cit.>. Most studies on VAEs employ the Gaussian distribution since the reparameterization trick (RT) for the Gaussian distribution is readily available <cit.>. To approximate the Dirichlet prior of LDA, <cit.> used a logistic normal distribution to approximate the Laplace approximation of the Dirichlet prior on the softmax basis. Their study provides evidence that VAEs could be effective for topic modeling, showing better results than traditional LDA with Gibbs sampling while enjoying much faster convergence.

The recent development in RT allows more distributions to be used in VAEs <cit.>. This has opened up new possibilities for VAEs to better approximate complex probabilistic models, which include LDA. The experiment in <cit.> indicates that using the Dirichlet prior directly in VAE produces lower perplexity than that of <cit.>, and the model also has the ability to further fine-tune the parameters of the Dirichlet prior, which has proven to greatly improve the performance of LDA <cit.>.

In this paper, we introduce a new VAE model that can leverage additional information from word embeddings to perform topic modeling on microtexts. Our motivation originates from the need of an algorithm that is effective in detecting latent topics in a corpus of short texts, which is achieved through the assistance of word embeddings, and has a good performance in order to scale up to a large dataset by using amortized variational inference with a neural network. However, this approach poses two major challenges:

* One must design a model that can appropriately use word vectors to their advantage.

* The original VAE approach only supports approximating the Gaussian distribution. Meanwhile, the topics generated for each document by LDA follow the Dirichlet distribution, and the relationships between each topic and each word are categorical.

To overcome these difficulties, we first posit a probability distribution, denoted q, that factors into two conditional distributions: the first one is the topic distribution of each word conditioned on its embedding and current context; the second one is the topic distribution of each document based on all the topic assignments of all the words in that document. We then approximate this probability distribution using a neural network with a purposely designed architecture.
The resulting network is a 2-layered nested structure of latent variables, which we aptly name  (). To find the parameter of the neural network, we minimize the Kullback-Leibler divergence between q and the posterior probability distribution of the LDA model by maximizing the corresponding evidence lower bound (ELBO) function, as normally done in the variational inference method. We use the Gumbel-Softmax trick <cit.> and the technique described by <cit.>, respectively, to optimize the parameters of the the word-to-topic distribution (a categorical distribution) and the document-to-topic distribution (a Dirichlet distribution) in q using stochastic gradient descent.

Compared to other LDA-based approaches for topic modeling, our approach enjoys the following advantages.

* Since word vectors are encoded with rich semantic information, we can potentially generate more meaningful topics. Moreover, the word vectors can be further fine-tuned in the training process through backpropagation.

* Using a neural network to parameterize the approximator in the variational inference setting gives us a shorter convergence time (since neural networks allow concurent processing) and smaller memory requirements (by using mini-batch gradient descent for training).

Indeed, experiments on various datasets of microtexts show that our model produces much better results on document clustering tasks and competitive results on topic coherence evaluation compared to other methods, while requiring a much smaller amount of time for convergence. Our implementation is available at

§ RELATED WORKS

With the ability to infer latent topics from a collection of documents, topic modeling has many interesting applications in various fields: detecting themes in historical newspapers <cit.>, studying scholarly literature <cit.>, and analyzing biological data <cit.>.
In terms of social media, the topic model has been utilized for identifying influential users on Twitter <cit.>, detecting communities <cit.>, and comparing topics of interest between different regions <cit.>.
Even though popular methods such as LDA <cit.> have been successfully applied to collections of long texts, microtexts remain a huge challenge for these models.
The sparsity of microtexts provides insufficient information on word cooccurrences required to infer the topic-to-word and document-to-topic distributions.
Therefore, various solutions have been proposed to overcome this problem.

One way to solve the lack of signals from microtexts is to aggregate them into a large document using some heuristics.
This method is more commonly used for social media data, by leveraging the structure of the network and the relationship between the data.
<cit.> explore the author-based aggregate scheme, i.e., combining tweets of each user to form a document.
Such aggregation is also used in <cit.>.
<cit.>, further introducing other aggregate schemes such as pooling tweets containing similar hashtags (Hashtag-based pooling), pooling tweets posted within the same hour to detect major events (temporal pooling), etc.
These methods have been shown to improve the performance of LDA on tweet data, but they depend on the specific properties of the dataset, which means they cannot be generalized to other use cases of microtext-based topic modeling.

Another direction is to invent new topic models that are more suitable for the characteristics of microtexts.
<cit.> study the effectiveness of the Dirichlet multinomial mixture model on microtext clustering.
<cit.> introduce the biterm topic model whose generative process is specifically designed to adapt to the sparseness of microtexts.
<cit.> propose a model with a two-phase generative process, where the first phase is similar to the LDA model while the second phase assumes that each text snippet is derived from a hidden pseudo-document.
While these models are more effective than LDA on microtexts, they can only use the limited information provided by the training corpus, which may not truly reflect the semantic relationships between words within the vocabulary.

To improve the results of topic models on microtexts, auxiliary information could be provided to augment or act as an alternative to the word cooccurrence statistics from the current corpus.
<cit.> use additional information on word similarities from thesauri and dictionaries to help place synonyms into the same topic.
<cit.> employ a probabilistic knowledgebase to improve short text comprehension.
<cit.> develop an extension to the LDA model called dual LDA, where the target topics of a short text dataset are jointly learned with the supporting topics from an external collection of long texts.
Other simpler approaches include training a topic model on a very large and universal corpus and then using the trained model to infer topics on a microtext corpus <cit.>.

Word embedding <cit.> is a distributed representation of words and encodes the semantic relationships between them, which could be useful in aiding topic modeling.
Some studies on using word embeddings in topic modeling have resulted in better performances on microtexts.
<cit.> combine word embeddings with a text aggregation method and the Markov random field regularized model.
<cit.> integrate LDA with word vectors via adding new components to the original model.
However, these papers only explore Gibbs sampling as the training method, which is slow and can not scale up to a large dataset, rendering them impractical for real use cases.

Variational inference with the mean field approximator could be used to train the LDA model <cit.>, which is faster than Gibbs sampling.
<cit.> replace the mean field approximator with the neural network, resulting in the VAE.
<cit.> present a VAE model for topic modeling based on LDA that has a much faster training and inference time while maintaining a competitive performance to the original LDA model trained using conventional methods.
However, this model suffers from the component collapsing problem when training on microtexts.

We take inspiration from those previous works and create a new model that employs a neural network architecture to approximate the posterior distribution of LDA and can benefit from word embeddings for knowledge of semantic relationships between words.
We believe that using auxiliary information for assistance is the most general solution to tackle the problems of constructing a topic model on microtexts.
This method not only works better on microtexts but could also enjoy improvements on long texts.
We choose the stochastic variational inference with a neural network approximator because we want our model to scale up to large datasets without needing a long time to converge and consuming a large amount of memory.

§ BACKGROUND

§.§ Word Embeddings

Word embeddings encode the information of each word using a dense vector.
Two popular unsupervised methods for training word embeddings are Word2Vec <cit.> and GloVe <cit.>.
Word2Vec uses a shallow neural network consisting of an input layer, a projection layer and an output layer to predict neighboring words.
There are two versions of Word2Vec: continuous bag-of-words (CBOW) and skip-gram.
The CBOW model attempts to predict a word based on its context, i.e., its surrounding words, while the skip-gram model does the inverse, predicting context words based on the current word input.

Instead of learning the relationships between one word and its neighbors similar to Word2Vec, the GloVe method trains a log-bilinear regression model directly on the matrix of the global word cooccurrence statistics of a corpus.
Its objective is to approximate the log probability of the cooccurrence of two words using the dot product of their word vectors.

Both Word2Vec and GloVe produce vectors that can encapsulate semantic relationships between words, which are usually encoded in the differences between two vectors.
For example, $\text{vec(Vietnam)} - \text{vec(Hanoi)} \approx \text{vec(France)} - \text{vec(Paris)} \approx  \text{vec(Germany)} - \text{vec(Berlin)}$.
Likewise, words that are near each other in the embedding space usually have certain similarities in their meanings.

§.§ Latent Dirichlet Allocation

The generative process of LDA

each document $d$
Sample a document-to-topic distribution $\theta_d \sim \text{Dirichlet}(\alpha)$
each word $i$ in document $d$
Sample a topic $z_{d,i} \sim \text{Categorical}(\theta_d)$
Sample a word $w_{d,i} \sim \text{Categorical}(\beta_{z_{d,i}})$

The LDA graphical model.

LDA <cit.> is a probabilistic generative model popularly used to extract latent topics from a collection of documents. The model assumes that each document is a mixture of topics and each topic is a probability distribution over a fixed vocabulary. The generative process of LDA is described in Algorithm <ref> and depicted in Figure <ref>, where:

$\bullet$ $\beta_k$ represents the parameters of the categorical distribution over the vocabulary of topic $k$ and $K$ is the number of topics;

$\bullet$ $\theta_{d}$ represents the parameters of the categorical distribution over topics of document $d$ and $D$ is the number of documents;

$\bullet$ $z_{d,i}$ is the topic assignment for the $i$-th word of document $d$;

$\bullet$ $w_{d,i}$ is the $i$-th word of document $d$ and $N_d$ is the number of words in document $d$.

$\bullet$ $\alpha$ is the parameter of the Dirichlet prior distribution.

Subsequently, the joint probability distribution of LDA is defined as:

\begin{equation}
\begin{split}
p(w,z,\theta|\alpha,\beta) = \prod_{d} p(\theta_d|\alpha) \prod_{i} p(z_{d,i}|\theta_d) p(w_{d,i}|\beta_{z_{d,i}})
\end{split}
\end{equation}

§.§ Variational Inference

Probabilistic generative models such as LDA require Bayesian inference methods to induce the values of their latent variables from the corresponding posterior distribution. Recall that in Bayes' theorem, $p(z|x)$ can be evaluated as

\begin{equation}
\label{eq:bayes}
\begin{split}
p(z|x) = \frac{p(x|z)p(z)}{p(x)}.
\end{split}
\end{equation}

However, such a distribution is usually intractable, leading to the employment of approximation methods. One such method is variational inference, where a tractable distribution $q$ is used to approximate the true distribution $p$. To find $q$ that most resembles $p$, we find $q$ that minimizes the Kullback-Leibler (KL) divergence from $q$ to $p$. Given a model with evidence $x$ and a set of latent variables $z$, the KL divergence from $q(z)$ to the posterior distribution $p(z|x)$ is defined as:

\begin{equation}
\label{eq:kl_vi}
\begin{split}
KL(q(z)||p(z|x)) &= -\mathbb{E}_{q(z)}[\log \frac{p(z|x)}{q(z)}] \\
&= \mathbb{E}_{q(z)}[\log q(z) - \log p(z|x)]
\end{split}
\end{equation}

Since directly minimizing this function requires knowing how to calculate $p(z|x)$, which is intractable in the first place, we instead maximize the ELBO function:

\begin{equation}
\begin{split}
\text{ELBO} &= \mathbb{E}_{q(z)}[\log p(z,x) - \log q(z)] \\
&= \mathbb{E}_{q(z)}[\log p(x) + \log p(z|x) - \log q(z)] \\
&= \log p(x) - \mathbb{E}_{q(z)}[\log q(z) - \log p(z|x)] \\
&= \log p(x) - KL(q(z)||p(z|x))
\end{split}
\end{equation}

The name ELBO stems from the fact that the term is always smaller or equal to the log probability of the evidence, i.e., $\log p(x) \geq \text{ELBO}$ since $KL(q(z)||p(z|x))$ $\geq 0$. Because $\log p(x)$ is a constant with regard to the observed data $x$ and the parameters of $q$, maximizing the ELBO corresponds to minimizing the $KL(q(z)||p(z|x))$.

The chosen approximate posterior $q$ usually comes from the mean field family, where each latent variable is assumed to come from a distribution with its own parameters. The optimization process requires deriving the updating rules by hand analytically for the coordinate descent algorithm. The analytical solution only exists if the model is conjugate. LDA is one such model because of the conjugacy between the Dirichlet and the multinomial distributions. Hence, the largest limitation of mean field variational inference is that it can only be used with conjugate models.

§.§ Variational Autoencoder and Reparameterization Tricks

An example VAE with a Gaussian posterior and a Gaussian decoder.

One way to mitigate the limitation of mean field variational inference is to use a neural network to parameterize the approximate posterior $q$. This idea is explored in <cit.>, which gives rise to a new class of models called the VAE. Essentially, the VAE consists of two parts: the encoder $q_{\phi}(z|x)$ and the decoder $p_{\theta}(x|z)$, where $\phi$ and $\theta$ denote the parameters of the encoder and decoder respectively. The encoder role is to map each of the input $x$ to its corresponding latent variables $z$, and the decoder role is to reconstruct $x$ from $z$. Ideally, we want the encoder $q_{\phi}(z|x)$ to act as if it is the true posterior $p_{\theta}(z|x)$ as much as possible. That goal is achieved by minimizing the KL-divergence between the two distributions, similar to the variational inference method. Under this setting, Equation (<ref>) is rewritten as:

\begin{equation}
\label{eq:kl_vae}
\begin{split}
KL(q_{\phi}(z|x)||p_{\theta}(z|x))=\mathbb{E}_{q_{\phi}(z|x)}[\log q_{\phi}(z|x) - \log p_{\theta}(z|x)]
\end{split}
\end{equation}

and the new ELBO is:

\begin{equation}
\label{eq:elbo_vae}
\begin{split}
\text{ELBO} &= \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(z,x) - \log q_{\phi}(z|x)] \\
&= -KL(q_{\phi}(z|x)||p_{\theta}(z)) + \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)]
\end{split}
\end{equation}

Figure <ref> depicts a VAE whose latent variables $z$ are assumed to be generated from a Gaussian distribution.

The KL term in Equation (<ref>) usually has a closed-form expression. However, samplings need to be done in order to approximate the expectation term. There is a direct formula to calculate the gradients of the expectation with respect to the variational parameters, but this method yields gradients with very high variance, making training impossible. Therefore, both papers introduce an alternative called the RT. This method uses a differentiable and invertible function $g_\phi$ such that $\varepsilon = g_{\phi}(z)$ and $z = g_{\phi}^{-1}(\varepsilon)$. Here, $\varepsilon \sim p(\varepsilon)$ is called a noise variable, and this transformation helps remove the dependence of the sampling process on the variational parameters. Instead of sampling directly from $q_\phi(z|x)$, we now draw $\varepsilon$ from its distribution and calculate the corresponding $z$. If $q_\phi(z|x)$ happens to be a Gaussian distribution $\mathcal{N}(\mu,\sigma)$, we can choose $p(\varepsilon) = \mathcal{N}(0,1)$ and $g_{\phi}(z)=(z-\mu)/\sigma$, i.e., $z=\mu + \sigma \varepsilon$. Unfortunately, finding a pair of $p(\varepsilon)$ and a function $g_\phi$ is not a trivial task for other distributions, such as the Dirichlet or categorical distributions used in LDA. One universal solution for every continuous distribution is to consider $p(\varepsilon) = \text{Uniform}(0,1)$ and $g_{\phi}(z)=F_\phi(z|x)$ where $F_\phi(z|x)$ is the cumulative distribution function of $q_\phi(z|x)$; however, to calculate $z$, we need to find the inverse of $F_\phi(z|x)$, which could be a very complicated process. The work in <cit.> proposes a novel way to alleviate this problem called implicit reparameterization, which in contrast to the method discussed so far does not need to find the inverse of $g_\phi$. This solution enables the RT to be used with a variety of continuous distributions, including the Dirichlet distribution. For discrete cases such as the categorical distribution, we use the Gumbel-Softmax <cit.> as a continuous approximation, which also permits backpropagation to update the distribution parameters.

§ NESTED VARIATIONAL AUTOENCODER () FOR TOPIC MODELING ON MICROTEXT WITH WORD VECTORS

In this section, we introduce , a novel VAE architecture for topic modeling with word embeddings. Our approach differs from other works on combining word embeddings and LDA since instead of replacing or extending components of the original LDA model with elements that are compatible with word vectors <cit.>, we propose a variational approximation q to the posterior of the LDA model that considers word embeddings as one of its parameters. In the following subsections, we will introduce the formulation of q, the derivation of the ELBO as the objective function and the translation of q to a neural network architecture. We will denote $V$, $K$, $D$ as the vocabulary size, the number of topics and the word embedding size, respectively.

§.§ The proposed distribution q

An example of word embeddings with two topic vectors: Car and Animal. Here, the word Jaguar is a homonym which could mean an animal or a name of a car manufacturer.

To design the variational distribution q, we make two assumptions:

* A coherent topic should be a group of words whose vectors are close to each other in the embedding space.

* From the embedding and the context of a word, we could decide which topic that the word belongs to.

To realize both assumptions, we parameterize the word-to-topic distribution - a categorical distribution - using word embeddings and a set of vector representations of the topic (one vector for each topic). More specifically, for word $i$, topic $t$ and document $d$, the log probability of the unnormalized word-to-topic distribution is a sum of two factors: the first one, denoted $s_{d,i}^t$, is the dot product between the word embedding $\omega_i$ and the topic embedding $\rho_t$; the second one, denoted $c_d^t$, is a function $g_t$ that receives the context representation of that word in document $d$ and outputs the coefficient of topic $t$:

\begin{equation}
\label{eq:word2topic}
\begin{split}
s_{d,i}^t &= \omega_{i}^d \cdot \rho_t \\
c_d^t &= g_t(\mean{\omega^{d}}) \\
\log \Tilde{q}(z_{d,i} = t | \rho_t, \omega^d) &= \pi_{d,i}^t = s_{d,i}^t + c_d^t
\\
q(z_{d,i} = t | \rho_t, \omega^d) &= \frac{\exp(\pi_{d,i}^t)}{\sum_t \exp(\pi_{d,i}^t)}
\end{split}
\end{equation}

where $z_{d,i}$ is the topic assigned to word $i$ in document $d$, and $\omega^d$ is the matrix of all the word embeddings within document $d$, $\mean{\omega^{d}}$ is the mean of all the word vectors in document $d$. Here, we represent the context of a word in a document as a the mean of word vectors in that document since we want the representation to be agnostic to the document's length. This strategy works well in practice.

The intuition behind this formulation is as follows:

* For $s_{d,i}^t$, since a topic should be a group of neighboring word embeddings, the dot product between each word vector in the topic and the topic vector should be large. Therefore, this term will put a high probability on a topic whose vector is near the vector of the current word. This property is illustrated in Figure <ref>.

* To account for homonyms, we introduce the term $c_d^t$, since these words have different meanings in different contexts and thus will belong to different topics. For example, the word Jaguar in Figure <ref> could indicate an animal or a car manufacturer depending on the current context.

In practice, we use the Gumbel-Softmax estimator to approximate the categorical distribution. Therefore, Equation (<ref>) is rewritten as:

\begin{equation}\label{word2topic_gumbel}
\begin{split}
q(z_{d,i} = t | \rho_t, \omega^d) = \frac{\exp(\pi_{d,i}^t/\tau)}{\sum_t \exp(\pi_{d,i}^t/\tau)} = \mu_{d,i}^t
\end{split}
\end{equation}

where $\tau$ is called the temperature. This term controls how much the continuous approximation resembles the real categorical distribution, with $\tau$ approaching $0$ resulting in a more discrete-like sample. When training, we anneal the temperature from $1$ to a predefined minimum value.

With each word in a document now assigned a topic based on the distribution defined in Equation (<ref>), we calculate the topic proportion of document $d$ as $q(\theta_d|z_d) = Dirichlet(\nu_d)$, where $\nu_d$ is defined as:

\begin{equation}
\begin{split}
\nu_d = \texttt{softplus}(\eta_d * a + b)
\end{split}
\end{equation}

\begin{equation}\label{eta}
\begin{split}
\eta_d=[\eta_{d,0},\eta_{d,1},...,\eta_{d,K-1}]
\end{split}
\end{equation}

is a $K$-dimension vector and $\eta_{d,i}$ is the number of words in document $d$ assigned the $i$-th topic, $a$ and $b$ are scalars that map $\eta_d$ to the appropriate values for the parameters of the Dirichlet distribution, and $\texttt{softplus}=\log (1+\exp(x))$. The use of $\texttt{softplus}$ is to make sure that $\nu_d$ is always positive, a requirement for Dirichlet parameters.

Overall, the variational posterior q is defined as:

\begin{equation}
\label{q_definition}
\begin{split}
q(\theta, z|\rho,\omega)&=\prod_d q(\theta_d|z_d) q(z_d|\rho,\omega^d)\\
&=\prod_d q(\theta_d|z_d)\prod_i q(z_{d,i} = t | \rho_t, \omega^d)
\end{split}
\end{equation}

§.§ Variational objective

With the definition of the variational posterior in Equation (<ref>), we can write the variational objective function. Since all the documents in a corpus are generated independently of each other, we derive the objective function for one document and the sum of all such functions as the final objective function of the corpus. The ELBO of document $d$ is defined as:

\begin{equation}\label{elbo_temp}
\begin{split}
\mathcal{L}_d  &= \mathbb{E}_{q(\theta_d,z_d|\rho,\omega^d)}[\log p(w_d,\theta_d,z_d)|\alpha,\beta) - \log q(\theta_d,z_d|\rho,\omega^d)]\\
&= \mathbb{E}_{q(\theta_d,z_d|\rho,\omega^d)}[\log p(\theta_d|\alpha) + \log p(z_d|\theta_d) + \log p(w_d|z_d,\beta)
- \log q(\theta_d|z_d) - \log q(z_d|\rho,\omega^d)]
\\
&= \mathbb{E}_{q(z_d|\rho,\omega^d)}[-\log q(z_d|\rho,\omega^d) - KL_{Dir}(q(\theta_d|z_d)||p(\theta_d|\alpha)) + \log p(w_d|z_d,\beta) + \mathbb{E}_{q(\theta_d|z_d)}[\log p(z_d|\theta_d)]]
\end{split}
\end{equation}

Combining Equation (<ref>), (<ref>) and (<ref>) we have:

\begin{equation}
\label{elbo}
\mathcal{L}_d = - \sum_i w_{d,i}(\sum_t \mu_{d,i}^{t}\log\mu_{d,i}^{t}) + \mathbb{E}_{q(z_d|\rho,\omega^d)}[-KL_{Dir}(q(\theta_d|z_d)||p(\theta_d|\alpha)) + \sum_i w_{d,i}\log \beta^{z_{d,i}}_{i} + \mathbb{E}_{q(\theta_d|z_d)}[\sum_t \eta_{d,t}\log \theta_{d,t}]]
\end{equation}

$\bullet$ Each $w_{d,i}$ indicates the count of word $i$ in document $d$.

$\bullet$ Each $z_{d,i}$ denotes the topic assignment of word $i$ in document $d$ and $z_d=[z_{d,0},z_{d,1},...,z_{d,V-1}]$.

$\bullet$ $\omega^d \in \mathbb{R}^{V \times D}$ is the matrix of all word embeddings of document $d$.

$\bullet$ $\rho \in \mathbb{R}^{K \times D}$ is the matrix of all the topic vectors.

$\bullet$ $\alpha$ represents the parameters of the Dirichlet prior in the LDA model.

$\bullet$ $\beta \in \mathbb{R}^{K \times V}$ is the matrix containing the parameters of all the topic-to-word distributions.

$\bullet$ Each $\beta^{t}_{i}$ is the probability that topic $t$ generates word $i$.

$\bullet$ $KL_{Dir}$ is the KL-divergence between two Dirichlet distributions, which can be calculated analytically.

Here, we treat each document $d$ as a bag-of-words following the standard practice in training LDA using variational inference.

To optimize Equation (<ref>), we have to use the RT twice: one for the word-to-topic distribution $q(z_d|\rho,\omega^d)$ and one for the document-to-topic distribution $q(\theta_d|z_d)$. For the word-to-topic distribution, a categorical distribution, we use the RT introduced in <cit.>, and for the document-to-topic distribution, a Dirichlet distribution, we use the RT introduced in <cit.>.

§.§ Neural network architecture

An overview of the neural network architecture of the .

The neural network representation of the posterior distribution $q(\theta,z|\rho,\omega)$. Each dotted line originated from a vector and points to the corresponding factor in the formulas introduced in Section <ref>. Each circle depicts a simple operation, and those with an orange background are operations performed on elements across the first dimension of a matrix. Red vectors are samples drawn from a distribution. ReLU is the function $f(x)=max(0, x)$. Linear is the function $f(x)=ax+b$ where $a$ and $b$ are scalars. A parallelogram depicts a distribution from which we draw samples. $K$ is the number of topics, $D$ is the size of the word embeddings and $V$ is the vocabulary size. Here, $K=3$, $D=4$ and $V=5$. Blue vectors indicate the input of the model.

From the equations presented in Section <ref>, it is straightforward to translate the variational distribution to a neural network representation. An overview of the architecture is illustrated in Figure <ref>, and the direct mapping between the formula and the architecture is depicted in Figure <ref>.

As seen in the figure, our model receives the embeddings of words within a document as its inputs. In practice, these embeddings are parts of the model's parameters, which gives us the ability to further fine-tune the word embeddings to suit the dataset, should we choose to do so.

Similar to <cit.>, we found the usage of batch normalization (BN) to be essential to achieving the desire performance of the model, as training without BN frequently leads to component collapsing. This phenomenon happens because early in the training process, the KL-divergence between two Dirichlet distributions dominates the loss term, which leads to the model converging to a local minimum where most of the components in the posterior are inactive; hence, the resulting topics are all similar to each other. Moreover, in addition to adding BN between fully connected layers as in Figure <ref>, we also utilize BN in the calculation of the $\beta$ term in Equation (<ref>):

\begin{align}
\beta = \texttt{Softmax}(\texttt{Transpose}(\texttt{Batch-Norm}(\texttt{Transpose}(\Tilde{\beta}))))
\end{align}

where $\Tilde{\beta} \in \mathbb{R}^{K \times V}$. Here, we apply BN so that for each topic, the non-normalized probability $\Tilde{\beta}_{t}$ will have a stable mean and variance, which we found necessary for convergence. We hypothesize that such stability provided by BN helps all the topic-to-word distributions to learn at the same rate, i.e., to have gradients with similar magnitudes, which makes the model converge to a more favorable local minimum and avoid component collapsing. Indeed, when measuring the gradients of the topic-to-word distributions, we found the model without BN at the $\beta$ matrix experiencing a divergence - where dominating topics have larger gradients, while other topics' gradients approach zero - and the model with BN having its corresponding gradients stay close to each other. A detailed study will be included in Section <ref>.

§ EXPERIMENTS AND RESULTS

In this section, we evaluate the  by measuring its topic coherence and its performance on the document clustering task.

Topic coherence indicates how related the words that are assigned to each topic are, which closely resembles how humans evaluate a topic model.
The document clustering task directly compares the clustering results to the ground truth labels by considering each class as a cluster.
To demonstrate the advantages of our model, we compare its performance and runtime against the original LDA model and other topic models that utilize word embeddings. To study the significance of word embeddings with respect to the topic model's quality, we compare the results on different sets of pretrained word embeddings. Since our focus is on performing a topic model on a corpus of microtexts, we conduct our experiments mostly on datasets with the average text length smaller than 20 words. We also present the hyperparameters and the training process required for the  to reach its optimal performance. Finally, we present a detailed study regarding the importance of BN to the model's performance.
Note that for each experiment in the subsequent sections, we ran it $10$ times and reported the average score.

§.§ Experimental Setup

§.§.§ Pretrained word embeddings

We use two state-of-the-art sets of pretrained word embeddings in our experiments:

* Google word vectors[https://code.google.com/archive/p/word2vec/]: these are 300-dimensional embeddings trained on a subset of the Google News corpus that contains approximately 100 billions words using the Word2Vec framework <cit.>. We denote this set of vectors as w2v.

* Stanford word vectors[https://nlp.stanford.edu/projects/glove/]: these are 300-dimensional embeddings trained on the Common Crawl dataset that contains 42 billions tokens using the GloVe method <cit.>. We denote this set of vectors as glove.

§.§.§ Datasets

Statistics of the datasets. #g: number of ground truth labels. #d: number of documents. #w/d: the average length of a document. V: the size of the vocabulary.

Dataset          #g     #d       #w/d     V

N20              20      18820     103.3     19572

N20short         20      1794      13.6      6377

N20small         20      400       88        8157

TMN              7       32597     18.3      13428

TMNtitle         7       32503     4.9       6347

Web Snippets     8       12335     14.5      7314

Twitter          4       2546      4.9       1402

We use four datasets: the 20-Newsgroups dataset, the TagMyNews dataset <cit.>, the Sander Twitter corpus and the Web Snippets dataset <cit.> to evaluate our model.

For the 20-Newsgroups and TagMyNews datasets, we use the preprocessed version and their derivation provided by <cit.>. These include the full version of the 20-Newsgroups dataset and the TagMyNews dataset, denoted N20 and TMN respectively; the N20short dataset contains all the documents from the N20 dataset having less than 21 words; the N20small dataset is balanced and contains 400 randomly selected documents from the original N20 dataset; and the TMNtitle dataset consists of only news titles from the TMN dataset.

For the Sander Twitter corpus[http://www.sananalytics.com/lab/index.php], we download the 5513 tweets using their Tweet IDs. There are 400 non-downloadable tweets. We closely follow the preprocessing method for this dataset as presented in <cit.>. After the preprocessing, there are 2546 remaining tweets.

For the Web Snippets[http://jwebpro.sourceforge.net/data-web-snippets.tar.gz]
dataset, we remove stop words using the list of stop words from the Stanford CoreNLP[https://github.com/stanfordnlp/CoreNLP],
as well as any words that are not contained in the Stanford and Google pretrained vectors. We also eliminate words that appear less than 3 times in the corpus. Finally, we remove any document whose length is zero after the preprocessing.

Table <ref> summarizes the statistics of all the datasets.

§.§.§ Baseline models

We use the following models as baselines in our experiments:

* LDA: the original LDA model introduced in <cit.>. Here, we consider using Gibbs sampling <cit.> as the inference method.

* LFLDA: a model introduced in <cit.> that incorporates word vectors to improve the topic model result on short text by using a mixture of the original Dirichlet multinomial and a latent feature component as the topic-to-word distribution. For the experiments, we use the code provided by the authors[https://github.com/datquocnguyen/LFTM] and the recommended settings in the paper, which includes setting the mixture weight $\lambda$ to $0.6$, the Dirichlet prior $\alpha$ and $\beta$ to $0.1$ and $0.01$, respectively, the number of initial sampling iterations to $1500$ and the number of iterations involving word vectors to $500$.

§.§.§ Training process and hyperparameters

The hyperparameters for each dataset. #epochs: number of training epochs. BS: batch size. #burn-in epochs: number of epochs before training $\alpha$. Min $\tau$: minimum temperature.

Dataset           #epochs     BS     c]@l@#burn-in
epochs     Min $\tau$     Layers     c]@l@Train word

N20              64                                                            256            32                                                                                0.5                                                               128,128     Yes

N20short         256                                                           256            128                                                                                0.7                                                               128,128     No

N20small         256                                                           200            128                                                                                0.7                                                               128         No

TMN              128                                                           256            64                                                                                0.7                                                               128,128     No

TMNtitle         128                                                           256            64                                                                                0.7                                                               128,128     No

Web Snippets     64                                                            256            32                                                                                0.7                                                               128,128     No

Twitter          128                                                           256            64                                                                                0.7                                                               128,128     No

To train the , we use the Adam Optimizer <cit.> with $\beta_1 = 0$ and $\beta_2 = 0.99$. We use a low momentum value since it allows the topic placement of a word to change quickly during the training process. From our experiments, we are able to confirm that using a low momentum allows our model to reach a lower perplexity as well as improve both topic coherence scores and document clustering results. We set the learning rate to $8e-3$ since it allows our model to converge quickly.

When training, we slowly increase the learning rate from $0$ to $8e-3$ while decreasing the temperature $\tau$ of the Gumbel-Softmax from $1$ to a minimum value - which is $0.7$ for small datasets or datasets of microtexts and $0.5$ for large datasets of long texts. This process takes place in the first epoch, which acts as a warm-up period for the model.

We set the initial value of the Dirichlet prior $\alpha$ to $0.1$, and optimize this value after a certain number of epochs, which is usually one-half the total amount of epochs. We find this greatly improves our model performance, especially on the document clustering task, which is not surprising since the study in <cit.> confirmed the importance of the Dirichlet prior to the quality of the topic model.

We use a batch size of $200$ for the N20small dataset and of $256$ for other datasets. We choose the number of epochs for each dataset that guarantees the convergence of our model. We use one fully connected layer of 128 units for the N20small dataset and two fully connected layers of 128 units for other datasets.

On large datasets of long texts, such as the N20 dataset, we also allow the training of word embeddings, which resulted in an additional boost in the model performance.

Table <ref> presents the hyperparameters for each dataset.

§.§ Topic coherence

NPMI scores (higher is better) for the N20short, TMN, TMNtitle, Twitter and Web Snippets dataset.

Dataset                       1cc]@c@Number of
topics     1cc]@c@LFLDA
glove     1cc]@c@LFLDA
w2v     1cc]@c@N-VAE
glove     1cc]@c@N-VAE
w2v     1cLDA

4*N20short     6                                                                                -3.70$\pm$0.37                                                                -3.86$\pm$1.01                                                              -4.29$\pm$1.30                                                                -3.62$\pm$1.03                                                     -3.62$\pm$0.74

20                                                                                -5.13$\pm$0.66                                                                -4.68$\pm$0.71                                                     -5.23$\pm$0.91                                                                -5.57$\pm$1.00                                                              -5.18$\pm$0.68

40                                                                                -6.42$\pm$0.32                                                                -6.37$\pm$0.41                                                     -7.62$\pm$0.64                                                                -6.95$\pm$0.99                                                              -6.69$\pm$0.24

80                                                                                -8.04$\pm$0.34                                                                -7.88$\pm$0.20                                                              -9.01$\pm$0.66                                                                -7.14$\pm$1.05                                                     -8.86$\pm$0.28
4*TMN          7                                                                                3.83$\pm$1.14                                                        3.60$\pm$0.74                                                               3.77$\pm$0.63                                                                 3.72$\pm$1.44                                                               3.47$\pm$0.91

20                                                                                3.74$\pm$0.54                                                                 3.97$\pm$0.52                                                      2.35$\pm$1.39                                                                 3.16$\pm$1.14                                                               3.20$\pm$0.70

40                                                                                2.83$\pm$0.41                                                                 3.00$\pm$0.55                                                      -3.43$\pm$1.27                                                                -2.34$\pm$1.38                                                              2.86$\pm$0.32

80                                                                                2.01$\pm$0.31                                                        2.00$\pm$0.18                                                               -7.34$\pm$0.57                                                                -7.07$\pm$1.18                                                              1.61$\pm$0.33
4*TMNtitle     7                                                                                -0.44$\pm$0.95                                                                -0.71$\pm$0.81                                                              -0.17$\pm$0.98                                                       -0.43$\pm$1.51                                                              -0.94$\pm$0.69

20                                                                                0.10$\pm$0.58                                                        -0.02$\pm$0.59                                                              -3.81$\pm$1.04                                                                -2.79$\pm$1.76                                                              0.01$\pm$0.52

40                                                                                -0.39$\pm$0.31                                                                -0.32$\pm$0.28                                                     -7.48$\pm$1.74                                                                -5.46$\pm$0.97                                                              -0.62$\pm$0.24

80                                                                                -1.29$\pm$0.10                                                       -1.31$\pm$0.34                                                              -7.50$\pm$0.73                                                                -5.47$\pm$0.87                                                              -1.60$\pm$0.28
4*Twitter      4                                                                                -2.50$\pm$0.54                                                                -2.70$\pm$0.94                                                              -1.99$\pm$1.56                                                       -2.06$\pm$1.73                                                              -3.09$\pm$0.96

20                                                                                -6.47$\pm$0.57                                                                -6.30$\pm$0.40                                                              -5.52$\pm$0.79                                                                -5.17$\pm$1.03                                                     -6.45$\pm$0.53

40                                                                                -7.41$\pm$0.19                                                                -7.48$\pm$0.38                                                              -6.98$\pm$0.73                                                                -5.43$\pm$0.35                                                     -7.92$\pm$0.41

80                                                                                -8.74$\pm$0.33                                                                -8.36$\pm$0.25                                                              -6.17$\pm$0.58                                                                -6.16$\pm$0.67                                                     -9.43$\pm$0.19
4*snippets     8                                                                                0.93$\pm$1.02                                                                 0.67$\pm$1.23                                                               2.14$\pm$1.01                                                        1.91$\pm$1.07                                                               0.98$\pm$0.91

20                                                                                1.62$\pm$0.61                                                                 1.92$\pm$1.38                                                               2.21$\pm$1.32                                                                 2.98$\pm$0.75                                                      1.66$\pm$0.74

40                                                                                -0.07$\pm$0.48                                                                0.18$\pm$0.44                                                               -3.97$\pm$0.73                                                                0.96$\pm$0.66                                                      -0.19$\pm$0.77
80                                                                                -1.54$\pm$0.34                                                                -1.54$\pm$0.19                                                              -8.57$\pm$1.30                                                                -1.39$\pm$0.59                                                     -1.90$\pm$0.43

We examine the quality of each topic produced by our model via measuring how semantically coherent its top words are.

Based on the survey done in <cit.>, we use normalized pointwise mutual information (NPMI) as the metric for quantitative analysis of topic coherence. This metric is introduced in <cit.> and is proven empirically to have a strong correlation with human evaluation. The NPMI is defined as:

\begin{equation}\label{eq:npmi}
\text{NPMI}(w_i, w_j) = \sum_{i=0}^{N-2}\sum_{j=i+1}^{N-1} -\frac{\log \frac{P(w_i,w_j)}{P(w_i)P(w_j)}}{\log P(w_i, w_j)}
\end{equation}

where the probabilities are collected using a sliding window of 10 words on an external corpus. For each topic, its NPMI score is calculated using its top-15 words.
We use Palmetto[ https://github.com/dice-group/Palmetto], the tool provided by the authors of <cit.> to measure topic coherence using Wikipedia as the external corpus. For each dataset, we calculate the coherence of each topic and use the average score of all the topics as the model's coherence.

Table <ref> presents the NPMI scores from various models with different numbers of topics. Since the LFLDA model requires a long time to train, our time and resource limitations only permit us to conduct experiments on $5$ datasets: N20short, TMN, TMNtitle, Twitter and Web Snippets. The table shows that our model produces competitive results, frequently having the highest NPMI scores when combined with Word2Vec.

§.§ Document clustering

NMI scores (higher is better) for the N20, N20small, N20short, TMN, TMNtitle, Twitter and Web Snippets dataset.

Dataset                       1cc]@c@Number of
topics     1cc]@c@LFLDA
glove     1cc]@c@LFLDA
w2v     1cc]@c@N-VAE
glove     1cc]@c@N-VAE
w2v     1cLDA
4*N20          6                                                                                0.522$\pm$0.003                                                      0.500$\pm$0.008                                                             0.519$\pm$0.014                                                               0.513$\pm$0.019                                                             0.516$\pm$0.009

20                                                                                0.596$\pm$0.012                                                               0.563$\pm$0.009                                                             0.632$\pm$0.014                                                      0.620$\pm$0.012                                                             0.582$\pm$0.009

40                                                                                0.550$\pm$0.010                                                               0.535$\pm$0.008                                                             0.624$\pm$0.012                                                      0.617$\pm$0.012                                                             0.557$\pm$0.007

80                                                                                0.507$\pm$0.003                                                               0.505$\pm$0.004                                                             0.613$\pm$0.004                                                      0.600$\pm$0.013                                                             0.515$\pm$0.003
4*N20small     6                                                                                0.436$\pm$0.019                                                               0.406$\pm$0.023                                                             0.437$\pm$0.019                                                      0.403$\pm$0.022                                                             0.376$\pm$0.016

20                                                                                0.504$\pm$0.020                                                               0.519$\pm$0.014                                                             0.533$\pm$0.023                                                      0.492$\pm$0.017                                                             0.474$\pm$0.013

40                                                                                0.527$\pm$0.013                                                               0.548$\pm$0.017                                                             0.560$\pm$0.011                                                      0.514$\pm$0.012                                                             0.513$\pm$0.009

80                                                                                0.576$\pm$0.006                                                               0.585$\pm$0.009                                                    0.585$\pm$0.012                                                      0.554$\pm$0.017                                                             0.563$\pm$0.008
4*N20short     6                                                                                0.290$\pm$0.011                                                               0.269$\pm$0.012                                                             0.301$\pm$0.025                                                      0.252$\pm$0.032                                                             0.224$\pm$0.012

20                                                                                0.294$\pm$0.007                                                               0.296$\pm$0.012                                                             0.353$\pm$0.013                                                      0.302$\pm$0.017                                                             0.248$\pm$0.009

40                                                                                0.299$\pm$0.009                                                               0.309$\pm$0.008                                                             0.375$\pm$0.015                                                      0.311$\pm$0.015                                                             0.268$\pm$0.010

80                                                                                0.338$\pm$0.008                                                               0.340$\pm$0.005                                                             0.367$\pm$0.011                                                      0.294$\pm$0.018                                                             0.308$\pm$0.009
4*TMN          7                                                                                0.448$\pm$0.017                                                               0.446$\pm$0.014                                                             0.487$\pm$0.031                                                      0.467$\pm$0.034                                                             0.429$\pm$0.026

20                                                                                0.403$\pm$0.004                                                               0.399$\pm$0.006                                                             0.467$\pm$0.011                                                      0.457$\pm$0.011                                                             0.400$\pm$0.009

40                                                                                0.356$\pm$0.004                                                               0.355$\pm$0.005                                                             0.464$\pm$0.013                                                      0.452$\pm$0.013                                                             0.354$\pm$0.004

80                                                                                0.324$\pm$0.004                                                               0.325$\pm$0.003                                                             0.474$\pm$0.015                                                      0.447$\pm$0.031                                                             0.319$\pm$0.006
4*TMNtitle     7                                                                                0.322$\pm$0.015                                                               0.321$\pm$0.012                                                             0.400$\pm$0.026                                                      0.367$\pm$0.028                                                             0.317$\pm$0.013

20                                                                                0.280$\pm$0.004                                                               0.279$\pm$0.006                                                             0.395$\pm$0.020                                                      0.361$\pm$0.025                                                             0.269$\pm$0.005

40                                                                                0.235$\pm$0.006                                                               0.239$\pm$0.005                                                             0.393$\pm$0.028                                                      0.362$\pm$0.028                                                             0.227$\pm$0.006

80                                                                                0.209$\pm$0.003                                                               0.210$\pm$0.002                                                             0.372$\pm$0.017                                                      0.341$\pm$0.018                                                             0.196$\pm$0.003
4*Twitter      4                                                                                0.260$\pm$0.022                                                               0.263$\pm$0.031                                                             0.262$\pm$0.032                                                               0.274$\pm$0.031                                                    0.234$\pm$0.021

20                                                                                0.184$\pm$0.005                                                               0.192$\pm$0.007                                                             0.249$\pm$0.011                                                               0.251$\pm$0.025                                                    0.178$\pm$0.008

40                                                                                0.175$\pm$0.006                                                               0.180$\pm$0.004                                                             0.256$\pm$0.018                                                               0.276$\pm$0.019                                                    0.162$\pm$0.007

80                                                                                0.163$\pm$0.006                                                               0.167$\pm$0.004                                                             0.240$\pm$0.027                                                               0.267$\pm$0.027                                                    0.156$\pm$0.007
4*snippets     8                                                                                0.523$\pm$0.035                                                               0.516$\pm$0.032                                                             0.628$\pm$0.034                                                               0.630$\pm$0.033                                                    0.490$\pm$0.039

20                                                                                0.433$\pm$0.011                                                               0.430$\pm$0.015                                                             0.590$\pm$0.014                                                               0.597$\pm$0.020                                                    0.424$\pm$0.017

40                                                                                0.372$\pm$0.006                                                               0.381$\pm$0.007                                                             0.573$\pm$0.010                                                               0.574$\pm$0.015                                                    0.364$\pm$0.008

80                                                                                0.346$\pm$0.005                                                               0.349$\pm$0.007                                                             0.587$\pm$0.021                                                      0.579$\pm$0.018                                                             0.336$\pm$0.011

Purity scores (higher is better) for the N20, N20small, N20short, TMN, TMNtitle, Twitter and Web Snippets dataset.

Dataset                       1cc]@c@Number of
topics     1cc]@c@LFLDA
glove     1cc]@c@LFLDA
w2v     1cc]@c@N-VAE
glove     1cc]@c@N-VAE
w2v     1cLDA
4*N20          6                                                                                0.295$\pm$0.001                                                      0.291$\pm$0.002                                                             0.293$\pm$0.004                                                               0.288$\pm$0.011                                                             0.293$\pm$0.002

20                                                                                0.604$\pm$0.031                                                      0.569$\pm$0.021                                                             0.602$\pm$0.032                                              0.584$\pm$0.028                                                             0.573$\pm$0.019

40                                                                                0.632$\pm$0.017                                                               0.616$\pm$0.017                                                             0.652$\pm$0.024                                                      0.634$\pm$0.027                                                             0.639$\pm$0.017

80                                                                                0.638$\pm$0.007                                                               0.638$\pm$0.006                                                             0.664$\pm$0.020                                                      0.652$\pm$0.018                                                             0.646$\pm$0.005
4*N20small     6                                                                                0.235$\pm$0.008                                                               0.229$\pm$0.005                                                             0.252$\pm$0.008                                                      0.237$\pm$0.015                                                             0.232$\pm$0.011

20                                                                                0.427$\pm$0.022                                                               0.439$\pm$0.015                                                             0.451$\pm$0.023                                                      0.408$\pm$0.025                                                             0.408$\pm$0.017

40                                                                                0.492$\pm$0.022                                                               0.516$\pm$0.024                                                             0.524$\pm$0.026                                                      0.465$\pm$0.024                                                             0.477$\pm$0.015

80                                                                                0.579$\pm$0.011                                                               0.595$\pm$0.016                                                    0.586$\pm$0.017                                                               0.543$\pm$0.028                                                             0.559$\pm$0.018
4*N20short     6                                                                                0.287$\pm$0.017                                                               0.278$\pm$0.007                                                             0.297$\pm$0.028                                                      0.269$\pm$0.025                                                             0.265$\pm$0.020

20                                                                                0.358$\pm$0.011                                                               0.366$\pm$0.010                                                             0.375$\pm$0.022                                                      0.330$\pm$0.026                                                             0.320$\pm$0.017

40                                                                                0.380$\pm$0.015                                                               0.395$\pm$0.013                                                             0.400$\pm$0.028                                                      0.345$\pm$0.017                                                             0.353$\pm$0.014

80                                                                                0.425$\pm$0.015                                                               0.429$\pm$0.009                                                    0.373$\pm$0.014                                                               0.326$\pm$0.021                                                             0.395$\pm$0.009
4*TMN          7                                                                                0.658$\pm$0.034                                                               0.658$\pm$0.020                                                             0.675$\pm$0.040                                                      0.662$\pm$0.040                                                             0.645$\pm$0.037

20                                                                                0.722$\pm$0.007                                                               0.716$\pm$0.012                                                             0.754$\pm$0.016                                                               0.758$\pm$0.015                                                    0.725$\pm$0.013

40                                                                                0.719$\pm$0.008                                                               0.720$\pm$0.008                                                             0.766$\pm$0.014                                                      0.755$\pm$0.011                                                             0.719$\pm$0.007

80                                                                                0.725$\pm$0.006                                                               0.725$\pm$0.004                                                             0.741$\pm$0.025                                                      0.701$\pm$0.042                                                             0.716$\pm$0.008
4*TMNtitle     7                                                                                0.584$\pm$0.026                                                               0.579$\pm$0.020                                                             0.634$\pm$0.030                                                      0.603$\pm$0.038                                                             0.581$\pm$0.016

20                                                                                0.623$\pm$0.012                                                               0.619$\pm$0.015                                                             0.662$\pm$0.029                                                      0.629$\pm$0.026                                                             0.610$\pm$0.010

40                                                                                0.600$\pm$0.008                                                               0.611$\pm$0.007                                                             0.654$\pm$0.042                                                      0.622$\pm$0.041                                                             0.593$\pm$0.011

80                                                                                0.601$\pm$0.004                                                               0.598$\pm$0.004                                                             0.632$\pm$0.025                                                      0.588$\pm$0.035                                                             0.582$\pm$0.005
4*Twitter      4                                                                                0.606$\pm$0.030                                                               0.615$\pm$0.039                                                    0.594$\pm$0.029                                                               0.603$\pm$0.034                                                             0.592$\pm$0.026

20                                                                                0.626$\pm$0.014                                                               0.637$\pm$0.013                                                             0.665$\pm$0.016                                                      0.651$\pm$0.025                                                             0.611$\pm$0.015

40                                                                                0.633$\pm$0.009                                                               0.640$\pm$0.008                                                             0.661$\pm$0.024                                                               0.669$\pm$0.029                                                    0.616$\pm$0.009

80                                                                                0.630$\pm$0.007                                                               0.636$\pm$0.006                                                             0.654$\pm$0.040                                                               0.659$\pm$0.037                                                    0.624$\pm$0.011
4*snippets     8                                                                                0.741$\pm$0.036                                                               0.730$\pm$0.045                                                             0.790$\pm$0.044                                                               0.796$\pm$0.041                                                    0.707$\pm$0.042

20                                                                                0.725$\pm$0.013                                                               0.724$\pm$0.015                                                             0.849$\pm$0.020                                                               0.853$\pm$0.023                                                    0.721$\pm$0.022

40                                                                                0.696$\pm$0.009                                                               0.709$\pm$0.010                                                             0.855$\pm$0.017                                                               0.856$\pm$0.015                                                    0.686$\pm$0.012

80                                                                                0.701$\pm$0.010                                                               0.703$\pm$0.012                                                             0.866$\pm$0.016                                                      0.847$\pm$0.015                                                             0.685$\pm$0.015

We measure to what extent the clustering result of the topic model agrees with the ground truth label. After calculating the topic proportion for each document using a topic model, we consider a document to belong to the topic with the highest probability in that document. We then calculate the similarity between clusters produced by the topic model and the ground truth label using two metrics: purity and normalized mutual information (NMI). For details on these two metrics, please refer to <cit.>, Section 16.3.

Table <ref> and <ref>, respectively presents the NMI and purity scores from various models with different numbers of topics. Note that for the N20 and N20small dataset, we use the results of the LFLDA-glove and LFLDA-w2v models reported in <cit.>. The table shows that the  outperforms other models most of the time, and using GloVe will frequently lead to better scores than using Word2Vec. Even on the N20 dataset, the additional information from the word vector is still useful and allows our model to reach a much better score than the original LDA model.

§.§ Runtime

Runtime of the LFLDA and the  on the N20short dataset with various numbers of topics.

Dataset      Number of topics     LFLDA (s)      (s)

N20short     6                1266          40

20               2007          48

40               3247          61

80               4068          73

We compare the runtime between the LFLDA and the . We ran each model on the N20short dataset using a machine with 24 vCPUs and 90 GB of RAM rented on the Google Compute Engine[ https://cloud.google.com/compute/]. Even though a GPU could be utilized to speed up the training process of our model, we only use the CPU for fairness. The results are shown in table <ref>. Overall, our model requires less than $1/30$th the training time of the LFLDA to reach the desired performance. This is the largest advantage of our model, enabling training on a large corpus while requiring only a small amount of time.

§.§ Effect of Batch Normalization

The value of $\alpha$

$\beta$'s gradients

The last FC layer's weights' gradients

No BN between the FC layers and at the $\beta$ matrix

The value of $\alpha$

$\beta$'s gradients

The last FC layer's weights' gradients

BN only at the $\beta$ matrix and none between the FC layers.

The value of $\alpha$

$\beta$'s gradients

The last FC layer's weights' gradients

BN only between the FC layers and none at the $\beta$ matrix.

The value of $\alpha$

$\beta$'s gradients

The last FC layer's weights' gradients

BN between the FC layers and at the $\beta$ matrix.

To provide a deeper understanding of the training process of our model, we investigate the effect of BN on the 's performance. In the model's architecture, we use BN at two components: between the fully connected (FC) layers and at the $\beta$ matrix. We carry out the experiment on the N20short dataset with 6 topics, where we alternately add and remove the BN at each component such that there are ultimately $4$ possible combinations. We plot the value of the Dirichlet parameters $\alpha$, the gradients of the topic-to-word distribution $\beta$ and the gradients of the weights of the last FC layer for each experiment.

Figure <ref>, <ref>, <ref> and <ref> depict the graphs of each combination of the 4 experiments. From the graph, we can observe that the removal of BN at the $\beta$ matrix leads to the divergence of the gradients between topics, where topics are divided into two groups: one group enjoys a high gradient while the other group obtains an increasingly smaller gradient. Therefore, the effect of BN at the $\beta$ matrix is to maintain a similar learning speed between topics in order to prevent the component collapsing phenomenon from happening. We can also witness the component collapsing phenomenon by observing the $\alpha$: its values diverge in Figure <ref> and <ref> and remain similar in Figure <ref> and <ref>.

On the other hand, the BN between the FC layers has the effect of allowing a larger gradient update at the weights of these layers. Figure <ref> and <ref> show that without the BN at these layers, their gradient becomes extremely small, which means that the model fails to learn anything at all. By contrast, the gradients in Figure <ref> and <ref> have a much larger magnitude throughout the training process.

We hypothesize that these phenomena occur due to the saturation of the softmax function during the training period. At the $\beta$ matrix, we use softmax to transform the original weights into a legitimate probability distribution. The training process may leads to some topics unfortunately having their softmax at the $\beta$ matrix saturated quickly, resulting in a diminishing gradient. The same event could happen with the Gumbel-Softmax distribution, where the saturation of the softmax means that the FC layers have very small gradients during backpropagation. Both of these catastrophic phenomena could be remedied by BN. Finally, BN is shown to smooth out the loss landscape <cit.>, which allows the usage of a large learning rate and makes the model more robust against changes in hyperparameters.

§.§ Discussion

Overall, the  frequently returns better results than the original LDA model as well as the LFLDA on both the topic coherence evaluation and document clustering task. In terms of the significance of the word embedding, we found Word2Vec helps the model to reach better topic coherence scores than GloVe, while GloVe is evidently more beneficial to the document clustering task. This is actually quite logical, since the NPMI metric used in the topic coherence evaluation has a window-based nature similar to the Word2Vec. The NPMI employs a sliding window over tokens in an external corpus (Wikipedia) to calculate the cooccurrences between pairs of words, while the Word2Vec training method emphasizes capturing the relationships between a word and its neighbors, i.e., local co-occurrences. This is not to posit that the NPMI is a well-rounded metric for topic coherence, since it fails to take into account words that obviously come from the same topic but rarely occur near each other, such as names of competing brands, e.g., Lamborghini and Toyota. Conversely, the GloVe training method considers the cooccurrences within the entire document (global cooccurrences) and the indirect relationships between words, which is better in identifying words that belong to the same topic and grouping them together. In fact, table 3 in <cit.> suggests that GloVe is better than Word2Vec in word similarity tasks even when it is trained on a smaller corpus. This, in turn, leads to more words being included in one topic, which means more documents having the same label are assigned to the same cluster, reflected through a high agreement between the clustering result of the  and the ground truth label.

We consider how BN assists in the training process of our model. We conclude that the effect of BN is two-fold: first, it helps solve the component collapsing problem of the model caused by the rapid saturation of the softmax function; and second, it permits the employment of high learning rates which speeds up convergence.

Finally, the major advantage of our model is its speed. Our model can extract topics from a corpus using only a fraction of the time required by the LFLDA, and the time gap increases with a larger number of topics. Our model not only succeeds in securing a time advantage but also consumes less memory since it is trained using mini batches instead of loading the entire corpus into the memory, which makes it extremely useful for inducing topics of a large dataset of microtexts, e.g., posts on social media.

§ CONCLUSION

In this paper, we propose , a model that can be combined with pretrained word vectors to induce latent topics from a dataset of microtexts.
We present a distribution q that takes into account word embeddings to approximate the joint distribution of LDA using the variational inference method.
We express this distribution using a neural network for a faster convergence speed and a smaller memory footprint so that our model can operate on large datasets.
Experiments show the improved performance and runtime of our model in deriving latent topics from microtexts compared to other methods.
These advantages make our model suitable for real-world use cases, such as categorizing a large collection of comments on social networks.

§ ACKNOWLEDGMENTS

§.§ Conflict of interest

We declare that we have no conflict of interest  for the work presented in this paper.

, Ng

Blei, DM.
, Ng, AY.
Jordan, MI.

Latent Dirichlet Allocation Latent Dirichlet
Journal of Machine Learning Research3993–1022.

Bouma, G.

Normalized (pointwise) mutual information in collocation
extraction Normalized (pointwise) mutual information in collocation
Proceedings of GSCL31–40.

, Zaheer

Das, R.
, Zaheer, M.
Dyer, C.

Gaussian LDA for Topic Models with Word
Embeddings Gaussian LDA for Topic Models with Word
Proceedings of the 53rd Annual Meeting of the
Association for Computational Linguistics and the 7th International
Joint Conference on Natural Language Processing (Volume 1: Long
Papers) Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint
Conference on Natural Language Processing (Volume 1: Long
Papers) ( 795–804).
Beijing, ChinaAssociation for Computational

Ferraz Costa
, Yamaguchi
, Juci Machado Traina
, Traina

Ferraz Costa

Ferraz Costa, A.
, Yamaguchi, Y.
, Juci Machado Traina, A.
, Traina, C., Jr.
Faloutsos, C.

RSC: Mining and Modeling Temporal Activity in
Social Media RSC: Mining and Modeling Temporal Activity in
Social Media.
Proceedings of the 21th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining Proceedings
of the 21th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining ( 269–278).
New York, NY, USAACM.

, Mohamed

Figurnov, M.
, Mohamed, S.
Mnih, A.

Implicit Reparameterization Gradients Implicit
Reparameterization Gradients.
arXiv:1805.08498 [cs, stat].
arXiv: 1805.08498

Goldstone, A.
Underwood, T.

The Quiet Transformations of Literary Studies: What
Thirteen Thousand Scholars Could Tell Us The quiet transformations of
literary studies: What thirteen thousand scholars could tell us.
New Literary History45359-384.

Griffiths, TL.
Steyvers, M.

Finding scientific topics Finding scientific
Proceedings of the National Academy of
Sciences101suppl 15228–5235.

Hofmann, T.

Probabilistic Latent Semantic Indexing
Probabilistic Latent Semantic Indexing.
Proceedings of the 22Nd Annual International ACM
SIGIR Conference on Research and Development in Information
Retrieval Proceedings of the 22nd Annual International ACM SIGIR
Conference on Research and Development in Information Retrieval  ( 50–57).
New York, NY, USAACM.
event-place: Berkeley, California, USA

Hong, L.
Davison, BD.

Empirical Study of Topic Modeling in Twitter
Empirical Study of Topic Modeling in Twitter.
Proceedings of the First Workshop on Social
Media Analytics Proceedings of the First Workshop on Social
Media Analytics ( 80–88).
New York, NY, USAACM.

, Gu

Jang, E.
, Gu, S.
Poole, B.

Categorical Reparameterization with
Gumbel-Softmax Categorical Reparameterization with
arXiv:1611.01144 [cs, stat].
arXiv: 1611.01144

, Liu
, Zhao
, Yu

Jin, O.
, Liu, NN.
, Zhao, K.
, Yu, Y.
Yang, Q.

Transferring Topical Knowledge from Auxiliary
Long Texts for Short Text Clustering Transferring Topical
Knowledge from Auxiliary Long Texts for Short Text
Proceedings of the 20th ACM International
Conference on Information and Knowledge Management Proceedings of
the 20th ACM International Conference on Information and Knowledge
Management ( 775–784).
New York, NY, USAACM.
event-place: Glasgow, Scotland, UK

Kennedy, H.
Moss, G.

Known or knowing publics? Social media data mining and
the question of public agency Known or knowing publics? Social media data
mining and the question of public agency.
Big Data & Society222053951715611145.

Kingma, DP.
Ba, J.

Adam: A Method for Stochastic Optimization
Adam: A Method for Stochastic Optimization.
arXiv:1412.6980 [cs].
arXiv: 1412.6980

Kingma, DP.
Welling, M.

Auto-Encoding Variational Bayes Auto-Encoding
Variational Bayes.

Lazard, SEBJWG., A.J.
Suran, M.

Detecting themes of public concern: a text mining
analysis of the Centers for Disease Control and Prevention's Ebola
live Twitter chat Detecting themes of public concern: a text mining
analysis of the Centers for Disease Control and Prevention's Ebola
live Twitter chat.
American journal of infection

, Tang
, Dong
, Yao

Liu, L.
, Tang, L.
, Dong, W.
, Yao, S.
Zhou, W.

An overview of topic modeling and its current
applications in bioinformatics An overview of topic modeling and its
current applications in bioinformatics.

, Mnih

Maddison, CJ.
, Mnih, A.
Teh, YW.

The Concrete Distribution: A Continuous
Relaxation of Discrete Random Variables The Concrete
Distribution: A Continuous Relaxation of Discrete Random
arXiv:1611.00712 [cs, stat].
arXiv: 1611.00712

, Schütze

Manning, CD.
, Schütze, H.
Raghavan, P.

Introduction to information retrieval Introduction to
information retrieval.
Cambridge University Press.

, Sanner
, Buntine

Mehrotra, R.
, Sanner, S.
, Buntine, W.
Xie, L.

Improving LDA Topic Models for Microblogs via
Tweet Pooling and Automatic Labeling Improving LDA Topic
Models for Microblogs via Tweet Pooling and Automatic
Proceedings of the 36th International ACM SIGIR
Conference on Research and Development in Information Retrieval
Proceedings of the 36th International ACM SIGIR Conference on
Research and Development in Information Retrieval (  889–892).
New York, NY, USAACM.

, Chen
, Corrado

, Chen

Mikolov, T.
, Chen, K.
, Corrado, G.
Dean, J.

Efficient Estimation of Word Representations in
Vector Space Efficient Estimation of Word Representations in
Vector Space.
arXiv:1301.3781 [cs].
arXiv: 1301.3781

, Sutskever
, Chen
, Corrado

, Sutskever

Mikolov, T.
, Sutskever, I.
, Chen, K.
, Corrado, GS.
Dean, J.

Distributed Representations of Words and Phrases
and their Compositionality Distributed Representations of Words and
Phrases and their Compositionality.
CJC. Burges, L. Bottou, M. Welling, Z. Ghahramani   KQ. Weinberger (), Advances in Neural
Information Processing Systems 26 Advances in Neural
Information Processing Systems 26 ( 3111–3119).
Curran Associates, Inc.

Mimno, D.

Computational Historiography: Data Mining in a Century
of Classics Journals Computational historiography: Data mining in a century
of classics journals.
J. Comput. Cult. Herit.513:1–3:19.

Newman, DJ.
Block, S.

Probabilistic topic decomposition of an
eighteenth-century American newspaper Probabilistic topic decomposition
of an eighteenth-century American newspaper.
Journal of the American Society for Information Science
and Technology576753-767.

, Billingsley
, Du

Nguyen, DQ.
, Billingsley, R.
, Du, L.
Johnson, M.

Improving Topic Models with Latent Feature
Word Representations Improving Topic Models with Latent Feature
Word Representations.
Transactions of the Association for Computational

Pandey, R.
Purohit, H.

CitizenHelper-Adaptive: Expert-Augmented
Streaming Analytics System for Emergency Services and
Humanitarian Organizations CitizenHelper-Adaptive:
Expert-Augmented Streaming Analytics System for Emergency
Services and Humanitarian Organizations.
2018 IEEE/ACM International Conference on
Advances in Social Networks Analysis and Mining (ASONAM) 2018
IEEE/ACM International Conference on Advances in Social Networks
Analysis and Mining (ASONAM) ( 630-633).

Paul, MJ.
, Sarker, A.
, Brownstein, JS.
, Nikfarjam, A.
, Scotch, M.
, Smith, KL.
Gonzalez, G.

Social media mining for public health monitoring and
surveillance Social media mining for public health monitoring and
Biocomputing 2016 Biocomputing 2016 (  468–479).
WORLD SCIENTIFIC.

, Socher

Pennington, J.
, Socher, R.
Manning, C.

Glove: Global Vectors for Word Representation
Glove: Global Vectors for Word Representation.
Proceedings of the 2014 Conference on Empirical
Methods in Natural Language Processing (EMNLP) Proceedings of the
2014 Conference on Empirical Methods in Natural Language
Processing (EMNLP) ( 1532–1543).
Association for Computational Linguistics.
event-place: Doha, Qatar

, Buntine
, Narayanamurthy
, Caetano

Petterson, J.
, Buntine, W.
, Narayanamurthy, SM.
, Caetano, TS.
Smola, AJ.

Word Features for Latent Dirichlet Allocation
Word Features for Latent Dirichlet Allocation.
JD. Lafferty, CKI. Williams, J. Shawe-Taylor, RS. Zemel  A. Culotta (), Advances in
Neural Information Processing Systems 23 Advances in Neural
Information Processing Systems 23 ( 1921–1929).
Curran Associates, Inc.

Phan, X.
, Nguyen, C.
, Le, D.
, Nguyen, L.
, Horiguchi, S.
Ha, Q.

A Hidden Topic-Based Framework toward Building
Applications with Short Web Documents A Hidden Topic-Based
Framework toward Building Applications with Short Web
IEEE Transactions on Knowledge and Data

, Chen
, Wang

Qiang, J.
, Chen, P.
, Wang, T.
Wu, X.

Topic Modeling over Short Texts by Incorporating
Word Embeddings Topic Modeling over Short Texts by
Incorporating Word Embeddings.
J. Kim, K. Shim, L. Cao, JG. Lee, X. Lin  YS. Moon (), Advances in Knowledge Discovery and
Data Mining Advances in Knowledge Discovery and Data Mining  ( 363–374).
ChamSpringer International Publishing.

, Kit
, Ge

Quan, X.
, Kit, C.
, Ge, Y.
Pan, SJ.

Short and Sparse Text Topic Modeling via
Self-aggregation Short and Sparse Text Topic Modeling via
Proceedings of the 24th International Conference on
Artificial Intelligence Proceedings of the 24th International
Conference on Artificial Intelligence ( 2270–2276).
AAAI Press.

, Mohamed

Rezende, DJ.
, Mohamed, S.
Wierstra, D.

Stochastic Backpropagation and Approximate
Inference in Deep Generative Models Stochastic Backpropagation
and Approximate Inference in Deep Generative Models.
Proceedings of the 31st International Conference on
International Conference on Machine Learning - Volume 32
Proceedings of the 31st International Conference on International
Conference on Machine Learning - Volume 32 (  II–1278–II–1286).
event-place: Beijing, China

Riddell, AB.
, Belgum, K.
, Boes, T.
, Koepnick, L.
, Kontje, T.
, Mellmann, K.
et al.

How to Read 22,198 Journal Articles: Studying
the History of German Studies with Topic Models How to Read
22,198 Journal Articles: Studying the History of German Studies
with Topic Models.
Distant Readings: Topologies of German Culture
in the Long Nineteenth Century Distant Readings: Topologies of
German Culture in the Long Nineteenth Century ( 91–114).
Boydell & Brewer.

, Both

Röder, M.
, Both, A.
Hinneburg, A.

Exploring the Space of Topic Coherence Measures
Exploring the Space of Topic Coherence Measures.
Proceedings of the Eighth ACM International
Conference on Web Search and Data Mining Proceedings of the
Eighth ACM International Conference on Web Search and Data
Mining ( 399–408).
New York, NY, USAACM.
event-place: Shanghai, China

Sadilek, A.
, Kautz, H.
, DiPrete, L.
, Labus, B.
, Portman, E.
, Teitel, J.
Silenzio, V.

Deploying nEmesis: Preventing Foodborne Illness
by Data Mining Social Media Deploying nEmesis: Preventing
Foodborne Illness by Data Mining Social Media.
AI Magazine38137–48.

Sahami, M.
Heilman, TD.

A Web-based Kernel Function for Measuring the
Similarity of Short Text Snippets A Web-based Kernel Function
for Measuring the Similarity of Short Text Snippets.
Proceedings of the 15th International Conference on
World Wide Web Proceedings of the 15th International Conference
on World Wide Web ( 377–386).
New York, NY, USAACM.
event-place: Edinburgh, Scotland

, Tsipras
, Ilyas

Santurkar, S.
, Tsipras, D.
, Ilyas, A.
Madry, A.

How Does Batch Normalization Help
Optimization? How Does Batch Normalization Help
S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi  R. Garnett (), Advances in Neural
Information Processing Systems 31 Advances in Neural
Information Processing Systems 31 ( 2483–2493).
Curran Associates, Inc.

, Sliva
, Wang
, Tang

Shu, K.
, Sliva, A.
, Wang, S.
, Tang, J.
Liu, H.

Fake News Detection on Social Media: A
Data Mining Perspective Fake News Detection on Social
Media: A Data Mining Perspective.
SIGKDD Explor. Newsl.19122–36.

, Wang
, Wang
, Li

Song, Y.
, Wang, H.
, Wang, Z.
, Li, H.
Chen, W.

Short Text Conceptualization Using a
Probabilistic Knowledgebase Short Text Conceptualization Using a
Probabilistic Knowledgebase.
Twenty-Second International Joint Conference on
Artificial Intelligence. Twenty-Second International Joint
Conference on Artificial Intelligence.

Srivastava, A.
Sutton, C.

Autoencoding Variational Inference For Topic
Models Autoencoding Variational Inference For Topic
arXiv:1703.01488 [stat].
arXiv: 1703.01488

, Lamrani

Tamimi, I.
, Lamrani, EK.
Kamili, ME.

Community Detection Through Topic Modeling in
Social Networks Community Detection Through Topic Modeling in
Social Networks.
E. Sabir, A. García Armada, M. Ghogho  M. Debbah  (), Ubiquitous Networking Ubiquitous networking  ( 70–80).
ChamSpringer International Publishing.

, Newman

Teh, YW.
, Newman, D.
Welling, M.

A Collapsed Variational Bayesian Inference
Algorithm for Latent Dirichlet Allocation A Collapsed
Variational Bayesian Inference Algorithm for Latent Dirichlet
Proceedings of the 19th International Conference on
Neural Information Processing Systems Proceedings of the 19th
International Conference on Neural Information Processing
Systems ( 1353–1360).
Cambridge, MA, USAMIT Press.

, Ferragina

Vitale, D.
, Ferragina, P.
Scaiella, U.

Classification of Short Texts by Deploying
Topical Annotations Classification of Short Texts by Deploying
Topical Annotations.
R. Baeza-Yates . (), Advances in
Information Retrieval Advances in Information Retrieval (  376–387).
Springer Berlin Heidelberg.

, Mimno

Wallach, HM.
, Mimno, DM.
McCallum, A.

Rethinking LDA: Why Priors Matter Rethinking
LDA: Why Priors Matter.
Y. Bengio, D. Schuurmans, JD. Lafferty, CKI. Williams  A. Culotta (), Advances in
Neural Information Processing Systems 22 Advances in Neural
Information Processing Systems 22 ( 1973–1981).
Curran Associates, Inc.

, Lim
, Jiang

Weng, J.
, Lim, EP.
, Jiang, J.
He, Q.

TwitterRank: Finding Topic-sensitive Influential
Twitterers Twitterrank: Finding Topic-sensitive Influential
Proceedings of the Third ACM International
Conference on Web Search and Data Mining Proceedings of the
Third ACM International Conference on Web Search and Data
Mining ( 261–270).
New York, NY, USAACM.

, Guo
, Lan

Yan, X.
, Guo, J.
, Lan, Y.
Cheng, X.

A biterm topic model for short texts A biterm topic
model for short texts.
WWW 2013 - Proceedings of the 22nd International
Conference on World Wide Web Www 2013 - Proceedings of the 22nd
International Conference on World Wide Web ( 1445-1456).

, Torget

Yang, TI.
, Torget, AJ.
Mihalcea, R.

Topic Modeling on Historical Newspapers Topic
Modeling on Historical Newspapers.
Proceedings of the 5th ACL-HLT Workshop on
Language Technology for Cultural Heritage, Social Sciences, and
Humanities Proceedings of the 5th ACL-HLT Workshop on Language
Technology for Cultural Heritage, Social Sciences, and
Humanities ( 96–104).
Stroudsburg, PA, USAAssociation for Computational

J. Yin

J. Yin

Yin, J.
Wang, J.

A Dirichlet Multinomial Mixture Model-based
Approach for Short Text Clustering A Dirichlet Multinomial
Mixture Model-based Approach for Short Text Clustering.
Proceedings of the 20th ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining Proceedings
of the 20th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining ( 233–242).
New York, NY, USAACM.

Z. Yin
, Cao
, Han
, Zhai

Z. Yin

Yin, Z.
, Cao, L.
, Han, J.
, Zhai, C.
Huang, T.

Geographical Topic Discovery and Comparison
Geographical Topic Discovery and Comparison.
Proceedings of the 20th International Conference on
World Wide Web Proceedings of the 20th International Conference
on World Wide Web ( 247–256).
New York, NY, USAACM.