Topic modeling of text documents is one of the most important tasks in representation learning. In this work, we propose iTM-VAE, which is a Bayesian nonparametric (BNP) topic model with variational auto-encoders. On one hand, as a BNP topic model, iTM-VAE potentially has infinite topics and can adapt the topic number to data automatically. On the other hand, different with the other BNP topic models, the inference of iTM-VAE is modeled by neural networks, which has rich representation capacity and can be computed in a simple feed-forward manner. Two variants of iTM-VAE are also proposed in this paper, where iTM-VAE-Prod models the generative process in products-of-experts fashion for better performance and iTM-VAE-G places a prior over the concentration parameter such that the model can adapt a suitable concentration parameter to data automatically. Experimental results on 20News and Reuters RCV1-V2 datasets show that the proposed models outperform the state-of-the-arts in terms of perplexity, topic coherence and document retrieval tasks. Moreover, the ability of adjusting the concentration parameter to data is also confirmed by experiments. Probabilistic topic models focus on discovering the abstract "topics" that occur in a collection of documents and representing a document as a weighted mixture of the discovered topics. Classical topic models, the most popular being LDA BID2 , have achieved success in a range of applications, such as information retrieval BID41 , document understanding BID2 , computer vision BID32 and bioinformatics BID34 . A major challenge of topic models is that the inference of the distribution over topics does not have a closed-form solution and must be approximated, using either MCMC sampling or variational inference. Hence, any small change to the model requires re-designing a new inference method tailored for it. Moreover, as the model grows more expressive, the inference becomes increasingly complex, which becomes the bottleneck to discover the latent semantic structures of complicated data. Hence, black-box inference methods BID31 BID26 BID16 BID33 , which require only limited knowledge from the models and can be flexibly applied to new models, is desirable for topic models.Among all the black-box inference methods, Auto-Encoding Variational Bayes (AEVB) BID16 BID33 ) is a promising one for topic models. AEVB contains an inference network that can map a document directly to a variational posterior without the need for further local variational updates on test data, and the Stochastic Gradient Variational Bayes (SGVB) estimator allows efficient approximate inference for a broad class of posteriors, which makes topic models more flexible. Hence, an increasing number of work has been proposed recently to combine topic models with AEVB, such as BID23 BID37 BID5 BID24 .Deciding the number of topics is another challenge for topic models. One option is to use model selection, which trains models with different topic numbers and selects the best on the validation set. Bayesian nonparametric (BNP) topic models, however, side-step this issue by making the number of topics adaptive to data. For example, BID39 proposed Hierarchical Dirichlet Process (HDP), which models each document with a Dirichlet Process (DP) and all DPs for documents in a corpus share a base distribution that is itself also from a DP. HDP extends LDA in that it can adapt the number of topics to data. Hence, HDP has potentially an infinite number of topics and allows the number to grow as more documents are observed. Unlike the black-box inference based models, traditionally, one needs to redesign the inference methods when there are some changes in the generative process of HDP BID39 BID40 BID11 .In this work, we make progress on this problem by proposing an infinite Topic Model with Variational Auto-Encoders (iTM-VAE), which is a Bayesian nonparametric topic model with AEVB. Coupling Bayesian nonparametric techniques with deep neural networks, iTM-VAE is able to capture the uncertainty regarding to the number of topics, and the inference can be conducted in a simple feed-forward manner. More specifically, iTM-VAE uses a stick-breaking process BID35 to generate the mixture weights for a countably infinite set of topics, and use neural networks to approximate the variational posteriors. The main contributions of the paper are:• We propose iTM-VAE, which, to our best knowledge, is the first Bayesian nonparametric topic model equipped with AEVB.• We propose iTM-VAE-Prod whose distribution over words is a product of experts rather than a mixture of multinomials.• We propose iTM-VAE-G, which helps the model to adjust the concentration parameter to data automatically.• The experimental results show that iTM-VAE and its two variants outperform the state-ofthe-art models on two challenging benchmarks significantly. In this paper, we propose iTM-VAE, which, to our best knowledge, is the first Bayesian nonparametric topic model that is modeled by Variational Auto-Encoders. Specifically, a stick-breaking prior is used to generate the mixture weights of countably infinite topics and the Kumaraswamy distribution is exploited such that the model can be optimized by AEVB algorithm. Two variants of iTM-VAE are also proposed in this work. One is iTM-VAE-Prod, which replaces the mixture of multinomials assumption of iTM-VAE with a product of experts for better performance. The other one is iTM-VAE-G which places a Gamma prior on the concentration parameter of the stick-breaking process such that the model can adapt the concentration parameter to data automatically. The advantage of iTM-VAE and its variants over the other Bayesian nonparametric topics models is that the inference is performed by feed-forward neural networks, which is of rich representation capacity and requires only limited knowledge of the data. Hence, it is flexible to incorporate more information sources to the model, and we leave it to future work. Experimental results on two public benchmarks show that iTM-VAE and its variants outperform the state-of-the-art baselines significantly. Table 4 : Top 10 words of topics learned by iTM-VAE-Prod without cherry picking.As shown in Table 4 , iTM-VAE-Prod can learn topics that are diverse and of high quality. One possible reason is that the stick-breaking prior for the document-specific π encourages the model to learn sparse representation, and the model can adjust the number of topics according to the data. Thus the topics can be sufficiently trained and of high diversity. The comparison of representation sparsity is illustrated in FIG5 (a).In contrast, the topics learned by ProdLDA BID37 lack diversity. As we listed in Table 5 , there are a lot of redundant topics. As a result, the latent representation learned by ProdLDA is of poor discriminative power. Figure 4: (a) Representation sparsity of different models on 20News. We sample one topic assignment π for each document, sort and then average across the test set. 9 (b) The TSNE-visualization of the representation learned by by iTM-VAE-Prod. (c) The TSNE-visualization of the representation learned by ProdLDA BID37 with the best topic coherence on 20News (K = 50).
A Bayesian Nonparametric Topic Model with Variational Auto-Encoders which achieves the state-of-the-arts on public benchmarks in terms of perplexity, topic coherence and retrieval tasks. This paper constructs an infinite Topic Model with Variational Auto-Encoders by combining Nalisnick & Smith's stick-breaking variational auto-encoder with latent Dirichlet allocation and several inference techniques used in Miao.