We consider the problem of topic modeling in a weakly semi-supervised setting. In this scenario, we assume that the user knows a priori a subset of the topics she wants the model to learn and is able to provide a few exemplar documents for those topics. In addition, while each document may typically consist of multiple topics, we do not assume that the user will identify all its topics exhaustively.
      
 Recent state-of-the-art topic models such as NVDM, referred to herein as Neural Topic Models (NTMs), fall under the variational autoencoder framework. We extend NTMs to the weakly semi-supervised setting by using informative priors in the training objective. After analyzing the effect of informative priors, we propose a simple modification of the NVDM model using a logit-normal posterior that we show achieves better alignment to user-desired topics versus other NTM models. Topic models are probabilistic models of data that assume an abstract set of topics underlies the data generating process BID0 . These abstract topics are often not only useful as feature representations for downstream tasks, but also for exploring and analyzing a corpus. Topic models are used to explore natural scenes in images BID2 BID9 , genetics data BID12 , and numerous text corpora BID11 BID10 .While latent Dirichlet allocation (LDA) serves as the classical benchmark for topic models, recent state-of-the-art topic models such as NVDM BID7 fall under the variational autoencoder (VAE) framework BID5 , which we refer to as Neural Topic Models (NTMs). NTMs leverage the flexibility of deep learning to fit an approximate posterior using variational inference. This posterior can then be used to efficiently predict the topics contained in a document. NTMs have been shown to model documents well, as well as associate a set of meaningful top words with each topic BID8 .Often, the top words associated with each extracted topic only approximately match the user's intuition. Therefore, the user may want to guide the model towards learning topics that better align with natural semantics by providing example documents. To our knowledge, supervision has been explored in more classical LDA models, but has not been explored yet in the NTM literature. Labeling the existence of topics in each document across a corpus is prohibitively expensive. Hence, we focus on a weak form of supervision. Specifically, we assume a user may identify the existence of a single topic in a document. Furthermore, if a user does not specify the existence of a topic, it does not mean the topic does not appear in the document. The main contribution of our work is an NTM with the ability to leverage minimal user supervision to better align topics to desired semantics. Topic models describe documents (typically represented as bags-of-words) as being generated by a mixture of underlying abstract topics. Each topic is represented as a distribution over the words in a vocabulary so that each document can be expressed as a mixture of different topics and its words drawn from the associated mixture distribution. A Neural Topic Model is a topic model constructed according to the variational autoencoder (VAE) paradigm. The generative process for a document x given a latent topic mixture υ is υ ∼ p(υ), x ∼ p θ (x|υ), where p(υ) is the prior and θ are learned parameters. The marginal likelihood p θ (x) and posterior p θ (υ|x) are in general intractable so a standard approach is to introduce an approximate posterior, q φ (υ|x), and seek maximization of the evidence lower bound (ELBO): ELBO(x;θ,φ) . DISPLAYFORM0 (The reparamterization trick BID5 lets us train this model via stochastic optimization. The most commonly used reparameterization is for sampling from a Gaussian with diagonal covariance, e.g., υ = µ(x) + σ(x) , ∼ N (0, 1) where µ(x) and σ(x) are neural networks. As motivated in the introduction, the topics extracted from unsupervised topic modeling can be challenging to interpret for a user (see Table 1 ). It is often the case that a user has a set of "ground truth" topics in mind that they would like the model to align with. We focus on the setting where a user is presented a subset of documents and identifies at least a single topic for each of the documents.(U) rf harvesting energy chains documents (S) recommendation retrieval item document items consumption compression shortest texts retrieval filtering documents collaborative preferences engine Table 1 : Upon scanning topics extracted by an unsupervised (U) NVDM for the AAPD ArXiv dataset, the user recognizes a few words relevant to the topic "information retrieval". Supervision (S) leveraging exemplar documents containing the topic helps the model to better align the topic to the desired semantics. We emphasize two challenging components in our setting. First, it is semi-supervised in the sense that only a small subset of documents is labeled by the user. Second, the supervision is weak in the sense that the user does not necessarily identify all the topics that are present in a document. We define "ground truth" word rankings for each topic using a relative chi-square association metric, a common measure of word association. Specifically, we compute the chi-square statistics between the occurrence of each word and occurrence of each document class using the ground truth topic labels. We then divide the chi-square statistic for each word by the average of its chi-square statistics over all labeled topics, giving us a relative importance that is discriminative across topics. The top-10 words for each topic are then the words with highest relative importance. To evaluate alignment of topics extracted by an NTM to these "ground truth" topics, we compute the average normalized point-wise mutual information (NPMI) BID1 ) between each word in the NTM's topics and each word in the "ground truth" topics-a higher score indicating the NTM's extracted words often co-occur with the ground truth's across Wikipedia. We refer to this score as pairwise-NPMI (P-NPMI). We additionally report the average NPMI computed between all word pairs within each NTM topic (NPMI) as a measure of topic coherence as well as NPMI separately averaged over just supervised topics (S-NPMI) and unsupervised topics (U-NPMI). Our approach focuses on extending the NTM to the semi-supervised setting. Our method is based on modifying the prior, p i (υ i ), of a VAE to incorporate user label information, where p i (υ i ) is the prior distribution over the latent variables for the i th document. As discussed above, we assume the user provides partial binary labels indicating the existence of topics in a document (1=exists, 0=unlabeled). We would like to encourage the posterior samples of the model, υ i ∼ q φ (υ i |x i ), to match the true presence of topics in a document. In the case of a Gaussian posterior (NVDM), a natural interpretation of υ i is as the logits of the probabilities that each topic exists. Therefore, to recover probabilities, we simply sigmoid the outputs of our encoder. Given this interpretation for semi-supervision, we set p i (υ ik ) to be N (µ ik , 1), where DISPLAYFORM0 and p k+ is the class-prior probability, i.e. the probability that any given document contains topic k. Note that p k+ can be estimated from PU data BID3 . In experiments, we instead use the mean class-prior probability for all topics (p + ) and are still able to obtain high performance suggesting this approach is robust to estimation error. In order to extract topics from the model, we condition the generative distribution, p θ (x|υ), on the logits of each of the possible approximate onehot vectors, e.g., υ 1 =logit([0.9999, 0.0001, . . .]) for topic 1. After conditioning on υ k , we observe the mean of the conditional distribution, ξ k . We then subtract from each ξ k the mean of ξ over the topics, i.e.,ξ k = ξ k −ξ. The indices corresponding to the top-10 values ofξ k indicate the top-10 words associated with each topic k. We also explore using two other posteriors. One is the Concrete / Gumbel-softmax approximation to the Bernoulli BID4 BID6 which more closely matches our goal of modeling the existence of topics in documents. The differentiable posterior q φ (υ|x) = Concrete(π, τ ) can be sampled with Equations (3), (4), and (5) : DISPLAYFORM1 where τ is a hyperparameter such that υ ∼ Ber(π) as τ → 0, sigmoid(r) = (1 + e −r ) −1 , and DISPLAYFORM2 The prior is p(υ ik ) ∼ Concrete(π ik , τ ) where π ik is the prior probability defined in Equation (6). The log-density of Concrete(π, τ ) is given in Section C.3.2 of BID4 and we use the Bernoulli-D KL as an approximation. The other posterior we consider is a logit-normal distribution, for which the logit(υ) is normally distributed. In FIG1 , we fix the prior distributions to Ber(π p =0.9999) for the Bernoulli model and N (µ p =logit(0.9999), σ p =1) for NVDM to signify the presence of a topic with high certainty. The x-axis, p, denotes the posterior probability: π q for the Bernoulli model and sigmoid(µ q ) for NVDM/logitnormal. The cross entropy loss is −π p log p−(1−π p ) log(1− p) where π p is the prior probability. The posterior variance of the NVDM is assumed to match the prior variance in this figure. It is clear that NVDM D KL strictly provides the strongest loss signal for topic prediction. On the other hand, the Bernoulli model generally achieves better topic coherence (NPMI). We expect this is due to the cleaner decoding process relative to NVDM. For example, the Bernoulli decoding computes the word distribution for the first out of K topics as the softmax of 1·W 1 +. . .+0·W K +b where W ∈ R |V |×K and b ∈ R |V | are the decoder weight matrix and bias respectively, |V | is the size of the vocabulary, and W k refers to the k th row of W . In contrast, NVDM decodes with 9.2 · W 1 + . . . − 9.2 · W K + b where 9.2 ≈ logit(0.9999) = −logit(0.0001).Therefore, we propose NVDM-σ with a logit-normal posterior. The intuition is to retain both the strong supervision signal of NVDM as well as the clean decoding used by the Bernoulli model. We train and evaluate these models on three multi-label datasets: BibTeX, a tagging dataset containing BibTeX metadata, Delicious, a document classification dataset, and AAPD, a dataset extracted from the abstracts and subjects of computer science ArXiv papers. We vary three experimental variables. We consider the percentage of topics supervised to be either 10%, 50%, or 100%. For example, for Delicious with 20 known topics, we consider supervising 2, 10, and all 20 dimensions of υ. We also consider the number of provided labeled samples per supervised topic. For example, in one experiment, when supervising 10% of topics for Delicious, we consider providing 3 or 10 example documents per topic. Lastly, we vary the number of labels given per document. As described in Section 3, the user is not required to provide all the topics contained in each document. We consider documents with only 1 label, up to half the maximum number of labels (5 for Delicious), and fullylabeled (up to 11). Using 10% topic supervision, 10 labeled documents per topic, and 1 label per document as an experimental baseline, we vary each experimental variable independently resulting in six different settings for each of three datasets. Note each setting is still weakly semi-supervised. In general, all models see a rise in NPMI from informative priors. We observe that NPMI rises on average by 12% relative to the unsupervised setting and by 15% beyond that to the fully supervised case. This demonstrates that weak labels are capable of providing a significant boost in performance. Top-10 Words (First Iteration) Top-10 Words (Fully Trained) NVDM transformation software attributes np necessary word allocation string greedy distance (0.230) correlation author posterior oracle alternating jointly tree sub trees latent Bernoulli provides received game white numbers greedy performance procedure allocation distance (0.219) distinct increasingly effectiveness loss gives jointly speed report existing learn NVDM-σ rf outperforms produces availability minimal equation allocation distance string greedy jointly (0.239) correcting increased formalism centrality correction word sub tree learn Table 2 : The supervised topic for AAPD is Computation and Language and the P-NPMI scores are under the model names. Only 3 documents are provided for 10% of topics with 1 label per document. Table 2 shows successful alignment for supervised topics. Only three documents are labeled for each topic; this positive result is important as labeling numerous documents is prohibitively expensive for commercial applications. Table 3 shows NVDM-σ does best in terms of all NPMI metrics. Table 3 : Average performance (with standard deviation in parentheses) relative to best score in each experimental setting. Specifically, for each experimental setting and each metric, we choose the best score among the 3 models and normalize the scores according to it. We then average these relative scores over the experimental settings. Note NVMD-σ achieves high mean performance with relatively low variance. In this work, we proposed supervising Neural Topic Models with weak supervision via informative priors and explored a variety of model posteriors. A careful analysis of their KL divergences and decoding mechanisms led us to an NTM with logit-normal posterior which best aligned extracted topics to desired user semantics. We use a feedforward neural network with 2 hidden layers followed by a linear transformation for the encoder. The first layer is 3K neurons wide and the second is 2K neurons wide where K is the size given to the latent space. Each hidden layer uses sigmoid nonlinearities. The final layer is linear and outputs a vector of size 2K for the parameters of the Gaussian posterior. The mean is obtained from the first K entries of this vector and the standard deviation, σ, is obtained from the last K entries by applying a softplus, log(1 + e x ). Instead of directly modeling the probability π in Equation (5) for the Bernoulli posterior, we interpret the output of the final linear layer of the encoder as logit(π). We set the dimensionality of the latent-space υ equal to 50 for 20NG and the number of ground truth topics for all other datasets. The decoder always consists of a linear layer followed by a softmax to obtain the multinomial probabilities for each word in the vocabulary. We set τ = 1 for the Gumbel-softmax sampler. We train with Adadelta, rescale gradients by 0.
We propose supervising VAE-style topic models by intelligently adjusting the prior on a per document basis. We find a logit-normal posterior provides the best performance. A flexible method of weakly supervising a topic model to achieve better alignment with user intuition.