Topics as Entity Clusters: Entity-based Topics from Large Language
  Models and Graph Neural Networks
Topic models aim to reveal latent structures within a corpus of text,
typically through the use of term-frequency statistics over bag-of-words
representations from documents. In recent years, conceptual entities --
interpretable, language-independent features linked to external knowledge
resources -- have been used in place of word-level tokens, as words typically
require extensive language processing with a minimal assurance of
interpretability. However, current literature is limited when it comes to
exploring purely entity-driven neural topic modeling. For instance, despite the
advantages of using entities for eliciting thematic structure, it is unclear
whether current techniques are compatible with these sparsely organised,
information-dense conceptual units. In this work, we explore entity-based
neural topic modeling and propose a novel topic clustering approach using
bimodal vector representations of entities. Concretely, we extract these latent
representations from large language models and graph neural networks trained on
a knowledge base of symbolic relations, in order to derive the most salient
aspects of these conceptual units. Analysis of coherency metrics confirms that
our approach is better suited to working with entities in comparison to
state-of-the-art models, particularly when using graph-based embeddings trained
on a knowledge base.
Manuel V. Loureiro, Steven Derby, Tri Kurniawan Wijaya
# Topics as Entity Clusters: Entity-based Topics from Language Models and Graph Neural Networks

###### Abstract

Topic models aim to reveal the latent structure behind a corpus, typically conducted over a bag-of-words representation of documents. In the context of topic modeling, most vocabulary is either irrelevant for uncovering underlying topics or contains strong relationships with relevant concepts, impacting the interpretability of these topics. Furthermore, their limited expressiveness and dependency on language demand considerable computation resources. Hence, we propose a novel approach for cluster-based topic modeling that employs conceptual entities. Entities are language-agnostic representations of real-world concepts rich in relational information. To this end, we extract vector representations of entities from (i) an encyclopedic corpus using a language model; and (ii) a knowledge base using a graph neural network. We demonstrate that our approach consistently outperforms other state-of-the-art topic models across coherency metrics and find that the explicit knowledge encoded in the graph-based embeddings provides more coherent topics than the implicit knowledge encoded with the contextualized embeddings of language models.

## 1 Introduction

Following the seminal work of Blei et al. (2003), topic models have since become the _de facto_ method for extracting and elucidating prominent themes from corpora. Traditionally, the semantic content of a document is composed of document-term frequencies or latently through a mixture of distributions of topics, common with probabilistic generative models such as _Latent Dirichlet Allocation_ (LDA). Here, individual topics are represented by salient lexical constituents such as words that depict some subjects of the corpora Blei et al. (2003); Blei and Lafferty (2006); Li and McCallum (2006); Teh et al. (2006); Crain et al. (2012). In recent years, the field of _Natural Language Processing_ (NLP) has seen a trend toward continuous vector representations of words, which look to capture the paradigmatic relationship between concepts by learning distributional co-occurrence patterns in text. For example, large-scale language models such as _BERT_Devlin et al. (2018) have explored robust contextualized representations that can explain an array of linguistic phenomena and implicit real-world knowledge Peters et al. (2018); Tenney et al. (2019); Petroni et al. (2019); Rogers et al. (2020), making them highly advantageous for topic modeling Sia et al. (2020); Bianchi et al. (2021).

Despite their successes, it becomes evident that certain limitations emerge from conventional topic modeling due to the superfluous nature and limited expressiveness of word-level tokens. These methods rely on data-driven techniques -- while ignoring real-world knowledge -- to uncover statistical patterns and infer relevant lexical items, which results in topics with limited guarantees of interpretability. Furthermore, in a multilingual setting, these models require expansive, resource-intensive lexicons that may not produce a desirable set of shared, language-free universal topics Ni et al. (2009); Boyd-Graber and Blei (2009).

To overcome these challenges, in this paper we focus on entities; they are distinct, free form human-derived concepts that are represented through encyclopedic-based definitions and a number of key relational attributes, which offer a better alternative for topic modeling Chemudugunta et al. (2008); Andrzejewski et al. (2009); Andryerjewski et al. (2011); Allahyari and Kochut (2016). We supersede word-level topic modeling with real-world entities, as these are both rich in conceptual information and language-agnostic. We demonstrate that by considering purely entity-level units in the text, it is possible to construct topics that are both interpretable to humans and founded on a rich set of prior knowledge. We pursue this approach using two sources to represent entities: (1) contextualized text represen
tations constructed from entity definitions and (2) structured graph data extracted from a knowledge base that we use to train a graph neural network to learn node embeddings. Furthermore, we propose _Topics as Entity Clusters_ (TEC), a novel topic modeling algorithm that can discover meaningful and highly informative topics by clustering either type of entity vectors or a combination of both. We successfully verify that, through the experimental procedure, our approach outperforms a number of state-of-the-art topic models on a range of metrics across numerous datasets.

## 2 Literature Review

Previous research has attempted to represent topic models using entities. For instance, Newman et al. (2006) proposed representing documents with salient entities obtained using _Named Entity Recognition_ (NER) instead of using the words directly. Others have attempted to capture the patterns among words, entities, and topics, either by expanding LDA Blei et al. (2003) or more complex Bayesian topic models -- see Alghamdi and Alfalqi (2015), Chauhan and Shah (2021) and Vaysansky and Kumar (2020) for a general overview -- by describing entities using words Newman et al. (2006); Kim et al. (2012); Hu et al. (2013).

### Word embeddings

Researchers have also found success by capitalizing on contemporary work in distributional semantics, integrating embedding lookup tables into their frameworks to represent words and documents. For instance, _lda2vec_Moody (2016) combines embeddings with topic models by embedding word, document, and topic vectors into a common representation space. Concurrently, Van Gysel et al. (2016) introduce an unsupervised model that learns unidirectional mappings between latent vector representations of words and entities. Using a shared embedding space for words and topics, Dieng et al. (2020) instead present the _Embedded Topic Model_ (ETM), which merges traditional topic models with the neural-based word embeddings of Mikolov et al. (2013)1.

Footnote 1: We do not compare our model against ETM because to do so requires us to pick the entity embeddings to be used in place of word embeddings.

### Neural topic models

In recent years, researchers have also looked to incorporate modern deep learning techniques that utilize contextualized representations in contrast to more traditional static embeddings Zhao et al. (2021). Srivastava and Sutton (2016) propose _ProdLDA_, a neural variational inference method for LDA that explicitly approximates the Dirichlet prior. Other models, however, such as _Neural Variational Document Model_ (NVDM) Miao et al. (2017), employ a multinomial factor model of documents that uses amortized variational inference to learn document representations. Bianchi et al. (2021) expand on ProdLDA presenting _CombinedTM_, which improves the model with contextualized embeddings.

### Knowledge extraction

More related to our work, Piccardi and West (2021) -- leveraging the self-referencing nature of Wikipedia -- define a cross-lingual topic model in which documents are represented by extracted and densified bags-of-links. The adoption of large-scale lexical resources has recently gained popularity in NLP as a way to directly inject knowledge into the model Gillick et al. (2019); Sun et al. (2020); Liu et al. (2020), which further motivates our research.

### Clustering

Clustering techniques have also proved effective for topic modeling. For instance, Sia et al. (2020) introduce clustering to generate coherent topic models from word embeddings, lowering complexity and producing better runtimes compared to traditional topic modeling approaches. Thompson and Mimno (2020) experiment with different pretrained contextualized embeddings and demonstrate that clustering contextualized representations at the token level is indistinguishable from a Gibbs sampling state for LDA. These findings were also recently corroborated by Zhang et al. (2022) who cluster sentence embeddings and extract top topic words using TF-IDF to produce more coherent and diverse topics than neural topic models. In contrast to our work, none of these works considers the expressiveness of entities.

## 3 Topics as Entity Clusters

In this section, we describe the steps necessary to perform topic modeling with entities and the novel approach for extracting salient entities to represent topics. We present an overview of the model in Figure 1.
### Entity representation

We explore methods to encode entities for cluster-based topic modeling. Broadly, we construct expressive entity representations from two sources of information: _Implicit Knowledge_ from a large pre-trained language model and _Explicit Knowledge_ extracted directly from a knowledge graph.

**Language embeddings.** Language models are used to construct document representations and depict knowledge obtained implicitly through a considerable amount of unsupervised learning (Petroni et al., 2019). To encode the entities, we first extract their definitions from an encyclopedic corpus before using these descriptions to build sentence embeddings to represent each entity -- for example, utilizing Reimers and Gurevych (2019). Using the text description of the entity rather than the entity alone as a query for these unsupervised models elicits a stronger response due to their highly contextualized nature (Ethayarajh, 2019).

**Graph embeddings.** Another advantage of using entities from lexical resources such as a knowledge base is that they provide a systematic framework for organizing and describing curated relationships between concepts. Similar to a semantic network, these entities exhibit a complex structure that provides meaningful information about their content, provided in the form of a directed graph. For instance, the triplet _<Petra, Culture, Nabataean kingdom>_ contains intricate encyclopedic knowledge about the city of Petra that can be difficult to learn with less specialized corpora. Language models may fail to adequately capture this relationship due to the abstract notion of the concept. Hence, to effectively capture these human-curated and refined factual relationships, we employ the graph neural network _node2vec_(Grover and Leskovec, 2016) to encode information about the sophisticated semantic structure between these entities.

**Combining Approaches.** In this work, we balance the contribution of the language model and our graph neural network. For some normalized language and graph embeddings \(\hat{E}_{LM}\in\mathbb{R}^{d_{LM}}\) and \(\hat{E}_{G}\in\mathbb{R}^{d_{G}}\), respectively, we weight their contributions using the following concatenation function,

\[\hat{E}=\left[\sqrt{\frac{1}{1+\alpha}}\cdot\hat{E}_{LM}^{\mathsf{T}},\sqrt{ \frac{\alpha}{1+\alpha}}\cdot\hat{E}_{G}^{\mathsf{T}}\right]^{\mathsf{T}} \tag{1}\]

where \(\alpha\in\mathbb{R}\) is the scalar ratio of embedding weights and \(\hat{E}\in\mathbb{R}^{d_{LM}+d_{G}}\) is our final embedding used in entity clustering. We take the square root to guarantee that the final embedding is normalized similarly to the input embeddings.

### Entity clustering

Independent of the specific method, we represent entities in an embedding space. Clustering allow us to define centroids which we interpret as topic centroids. Therefore, we model topics to have representations in a shared embedding space with entities. To this effect, we apply K-Means to the set of entities contained in a corpus, using the implementation available in FAISS (Johnson et al., 2021).

Figure 1: Overview of Topics as Entity Clusters (TEC). The top half illustrates the processing of entity embeddings, topic centroids and top entities per topic, while the bottom half inferencing the top topics per document.
### Entity extraction

We adopt a two-stage approach to extract entities, which allows us to represent text as a language-agnostic collection of entity identifiers arranged in order of appearance.

**Pattern matching**. We first extract candidate entities by finding language-specific text patterns in the original text. Inspired by Mendes et al. (2011) and Daiber et al. (2013), we use the deterministic Aho-Corasick algorithm (Aho and Corasick, 1975) due to its speed and effectiveness in extracting text patterns. The only language-specific components are the preprocessing components, such as lemantizers, that increase the number of relevant entity matches. These preprocessing components are independent of each other. Consequently, we can expand the model to additional languages without compromising the performance of the others.

**Disambiguation**. Since text patterns could represent multiple entities -- for example, acronyms of organizations or people sharing the same name -- we perform disambiguation and entity filtering. For each textual pattern and its corresponding set of entities, we choose the entity that best fits the text. We embed the text using using the same model used to derive the language embeddings and calculate their cosine similarity. We choose the best candidate based on the highest score if it is above a set similarity threshold. Otherwise, we discard it.

### Topic inference

Topic inference requires the representation of documents in the same embedding space as entities and topic centroids. To accomplish this, we extract entities as described in Section 3.3. We then obtain the document representation by calculating the weighted average of those entity embeddings. With \(K\) representing the number of topics, we can now measure the Euclidean distances \(\textbf{d}=\left[d_{1},d_{2},...,d_{K}\right]^{\mathsf{T}}\) of the document to the topic centroids. Documents are assumed to contain a share of all topics. We infer the topic weight contribution \(\textbf{w}=\left[w_{1},w_{2},...,w_{K}\right]^{\mathsf{T}}\) to the document using the inverse distance squared weighted interpolation2 (Shepard, 1968):

Footnote 2: If we consider the embedding of a document as an interpolation of topic centroids, squaring the distances yields more weight to the closest topic centroids.

\[w_{i}=\frac{d_{i}^{-2}}{\sum_{j=1}^{K}d_{j}^{-2}}\quad,\forall\,i\in\left\{1,...\,,K\right\}. \tag{2}\]

### Reranking top entities

A list of highly descriptive entities, weighted by their importance, can be used to express the theme of a topic. However, the closest entities to topic centroids are not necessarily the most descriptive as that does not consider entity co-occurrences in the corpus. In Algorithm 1, we propose a novel inference-based method to rerank top entities, which assigns the entity frequency of a document to the top topic centroid, as measured by **w**.

We start by assigning entities to topics based on their distances weighted by a small value, \(\epsilon\) (Lines 1-3). This ensures all topics have top entities. We follow by inferring the top topic for each document and updating the top entities in that topic using the document entity frequency. The update is proportional to the inference score, \(\max\left(\textbf{w}\right)\), as it represents the degree of confidence in the inference. (Lines 4-10). To increase topic diversity, we only update the top topic. Lastly, we calculate the relative frequencies to obtain the top entities per topic (Lines 11-13).

```
Input: Number of topics \(K\), number of top entities  per topic \(N\), small initialization weight \(\epsilon\),  documents \(Docs\), all entity identifiers in the  corpus \(entities\), entity embeddings \(\hat{E}\) Output: Lists of top entities per topic \(topEntities\),  each element is a list of pairs \(\left(entityId,frequency\right)\) for\(topicId\in\left\{1,...,K\right\}\)do \(topEntities[topicId]\leftarrow\)  ClosestEntities(\(topicId,\hat{E},N,\epsilon\))
1 end for\(doc\in Docs\)do \(\textbf{w}\), \(entityFrequency\leftarrow\) TopicInference(\(doc\)) // Section 3.4 \(topTopic\leftarrow\) argmax(\(\textbf{w}\)) for\(entityId\inentities\)do \(topEntities\)\([topTopic]\leftarrow\) \(\max\left(\textbf{w}\right)\cdotentityFrequency[entityId]\)
2 end for
3
4 end for
5
6 end for
7for\(topicId\in\left[1,...,K\right]\)do \(topEntities[topicId]\leftarrow\)  RelativeFrequency(\(topEntities[topicId]\))
8 end for
```

**Algorithm 1**Reranking top entities

## 4 Experiments

We study the performance of TEC and qualitatively compare it to other state-of-the-art topic models using a set of corpora preprocessed into lists of entity identifiers. By contrasting the top entities and measuring results across several coherency metrics, we can infer the quality of each topic model.
In summary, we find that TEC produces significantly more coherent topics. These gains are more pronounced when using graph embeddings.

### Entity extractor

We build the entity extractor using Wikidata3 as the source of our knowledge base and Wikipedia4 as the encyclopedic corpus. Wikidata currently has more than 97 million entities, most of which would be a long tail of entities in a topic model therefore we restrict the entity extractor to only include the top one million entities, as ranked by QRank5 - a public domain project that ranks page views across Wikimedia projects. Out of these entities, we select those matching at least one predicate-object pair from lists of preselected objects for predicates "instance of", "subclass of", and "facet of". We generate the entity embeddings used in disambiguation using _SBERT6_. Entities are matched to Wikipedia articles using Wikidata identifiers.

Footnote 3: Wikidata JSON dump downloaded on March 24, 2022.

Footnote 4: Collected with Beautiful Soup on March 28, 2022.

Footnote 5: QRank downloaded on March 24, 2022.

Footnote 6: We use paraphrase-multilingual-mpnet-base-v2.

Footnote 7: _CC-News_ available at Hugging Face.

Footnote 8: _zcyrek_ available on GitHub.

### Corpora

We evaluate all models on various corpora: _Wikipedia_, _CC-News7_, and _MLSUM_(Scialom et al., 2020); Table 1 contains a statistics summary. The _Wikipedia_ corpus consists of a sample of preprocessed documents, each matching an entity in the vocabulary. _CC-News_ consists of monolingual news articles written in English. _MLSUM_ is a collection of news articles written in German, Spanish, French, Russian, and Turkish.

Footnote 7: _CC-News_ available at Hugging Face.

We preprocess the documents according to Section 3.3. The language-specific components for documents in English, German, Spanish and French are _spaCy_ lemmatizers (Honnibal and Montani, 2017), for documents in Russian we use _pymorphy2_(Korobov, 2015), and for documents in Turkish we use _zeyrek8_.

Footnote 8: _zcyrek_ available on GitHub.

### Models

We start by comparing our approach with LDA (Blei et al., 2003) due to its pervasiveness in topic model literature. Specifically, we use the _Mallet_ implementation of LDA (McCallum, 2002). On top of that, we compare using other state-of-the-art topic models from the literature.

**NVDM-GSM.**_Neural Variational Document Model_ (NVDM) is a neural network-based topic model that discovers topics through variational inference training, proposing a number of ways to construct topic distributions, such as a _Gaussian Softmax_ (GSM) function (Miao et al., 2017).

**ProdLDA.** Similar to NVDM-GSM, this model is an autoencoder trained to reconstruct the input embeddings with variational inference-based training (Srivastava and Sutton, 2016).

**CombinedTM.** This model is a direct extension to ProdLDA that includes pre-trained contextualized embeddings from a pretrained language model (Bianchi et al., 2021). In this case, the authors extract contextual vectors for documents using _SBERT_.

**WikiPDA.** We also consider the Wikipedia-based Polyglot Dirichlet Allocation model, an LDA model trained on entities extracted from Wikipedia (Piccardi and West, 2021). WikiPDA has its own preprocessing method.

### Metrics

Topic models produce subjective results, so we calculate different measures to understand model performance. We use topic coherence measures to estimate the relationship between top entities of a topic (Roder et al., 2015).

\[C_{f_{t}}=\frac{1}{T}\sum_{t\in\{1..T\}}\left[\frac{2}{N(N-1)}\sum_{ \begin{subarray}{c}i\in\{1..N\}\\ j\in\{1..i-1\}\end{subarray}}f_{t}(w_{i},w_{j})\right] \tag{3}\]

All coherence metrics are calculated using Eq. 3, as implemented in _gensim_(Rehurek and Sojka, 2011), over the top most relevant \(N\) entities for all topics \(t\in\{1..T\}\), with \(N=10\). The specific element \(f_{t}\) changes for each measure.

**Coherence UCI.**Newman et al. (2010) present a coherence measure that averages the _Pointwise Mutual Information_ (PMI, Eq. 4) of all entity pairs in a topic using a sliding window of entities:

\[\mathrm{PMI}(w_{i},w_{j})=\log\left(\frac{p(w_{i},w_{j})}{p(w_{i})p(w_{j})} \right). \tag{4}\]

\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Corpus** & Vocabulary & Documents & 
\begin{tabular}{c} Avg. Entities \\ per Document \\ \end{tabular} \\ \hline WIKIPEDIA & 359,507 & 359,507 & 44.62 \\ CC-NEWS & 94,936 & 412,731 & 13.97 \\ MLSUM & 89,383 & 661,422 & 11.71 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the corpora.
\begin{table}
\begin{tabular}{l|c} \hline \hline
**Model** & **Sample Topic** \\ \hline \hline LDA & United Nations (Q1065) \(|\) Teenage Mutant Ninja Turtles (Q12296099) \(|\) Miles Davis (Q93341) \\  & \(|\) Star Trek (Q1092) \(|\) United Nations Security Council (Q37470) \(|\) United Nations Relief and Works Agency for Palestine Refugees in the Near East (Q846656) \(|\) public health (Q189603) \(|\) Dizzy Gillespie (Q49575) \(|\) Greenpeace (Q81307) \(|\) John Coltrante (Q7346) \\ NVDM-GSM & bitcoin (Q131723) \(|\) Apple Inc. (Q312) \(|\) Halloween [film franchise] (Q1364022) \(|\) Fisker Inc. [automaker] (Q1420893) \(|\) IBM (Q37156) \(|\) Michael Myers (Q1426891) \(|\) Yakuza [video game series] (Q2594935) \(|\) Facebook (Q355) \(|\) cryptocurrency (Q13479982) \(|\) Vancouver (Q234053) \\ ProdLDA & Paul McCartney (Q2599) \(|\) Maxim Gorky (Q12706) \(|\) Lucy-Jo Hudson (Q1394969) \(|\) Bob Dylan (Q392) \(|\) sport utility vehicle (Q192152) \(|\) FIFA World Cup (Q19317) \(|\) sedan (Q190578) \(|\) American football (Q41323) \(|\) concept car (Q850270) \(|\) racing automobile (Q673687) \\ CombinedTM & vocalist (Q2643890) \(|\) United States of America (Q30) \(|\) music interpreter (Q3153559) \(|\) England (Q21) \(|\) Ryuichi Sakamoto (Q345494) \(|\) human rights (Q8458) \(|\) David Tennant (Q214601) \(|\) Harry Potter (Q76164749) \(|\) Comedian (Q2591461) \(|\) Aoni Production (Q1359479) \\ WikiPDA & a cappedla (Q185298) \(|\) X-Men (Q128452) \(|\) Marvel Comics (Q173496) \(|\) To Be [music album] \\  & (Q17025795) \(|\) The Allman Brothers Band (Q507327) \(|\) proton-proton chain reaction (Q223073) \(|\) features of the Marvel Universe (Q5439694) \(|\) Features of the Marvel Chematic Universe (Q107088537) \(|\) Uncanny X-Men (Q1399747) \(|\) member of parliament (Q486839) \\ \hline TEC \(E_{LM}\left(\alpha=0\right)\) & Google (Q95) \(|\) Amazon (Q3884) \(|\) Microsoft (Q2283) \(|\) open source (Q39162) \(|\) Apple Inc. (Q312) \(|\) Facebook (Q355) \(|\) Meta Platforms (Q380) \(|\) Cisco Systems (Q173395) \(|\) Salesforce.com (Q941127) \(|\) Citrix Systems (Q916196) \\ TEC \(E_{G}\left(\alpha=\infty\right)\) & Mike Tyson (Q79031) \(|\) World Boxing Organization (Q830940) \(|\) International Boxing Federation (Q742944) \(|\) Floyd Mayweather (Q318204) \(|\) World Boxing Association (Q725676) \(|\) Tyson Party (Q1000592) \(|\) Manye Pacquique (Q486359) \(|\) World Boxing Council (Q724450) \(|\) Evander Hylofield (Q313451) \(|\) Joe Frazier (Q102301) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Example topics using WIKIPEDIA corpus for models trained with 300 topics. Each topic is represented by its top 10 entities.

\begin{table}
\begin{tabular}{l|r r r r r} \hline \hline
**Model** & \multicolumn{4}{c}{**WIKIPEDIA**} \\ \hline \hline
**Model** & \multicolumn{2}{c}{**C\({}_{\mathbf{NPMI}}\)**} & \multicolumn{2}{c}{**C\({}_{\mathbf{UCI}}\)**} & \multicolumn{2}{c}{**U\({}_{\mathbf{Mass}}\)**} & \multicolumn{2}{c}{**TD**} & \multicolumn{2}{c}{**TQ**} \\ \hline \hline
**Number of Topics \(\times\)100** & & & & & \\ \hline LDA & \(-0.05\) (\(0.01\)) & \(-4.72\) (\(0.21\)) & \(-11.23\) (\(0.23\)) & \(\mathbf{0.98}\) (\(0.00\)) & \(-0.05\) (\(0.01\)) \\ NVDM-GSM & \(0.06\) (\(0.02\)) & \(-2.66\) (\(0.36\)) & \(-9.17\) (\(0.31\)) & \(0.87\) (\(0.02\)) & \(0.05\) (\(0.02\)) \\ ProdLDA & \(-0.16\) (\(0.03\)) & \(-6.55\) (\(0.44\)) & \(-12.91\) (\(0.52\)) & \(0.62\) (\(0.16\)) & \(-0.10\) (\(0.03\)) \\ CombinedTM & \(-0.10\) (\(0.02\)) & \(-5.94\) (\(0.35\)) & \(-11.54\) (\(0.49\)) & \(0.22\) (\(0.03\)) & \(-0.02\) (\(0.00\)) \\ WikiPDA & \(0.08\) (\(0.01\)) & \(0.37\) (\(0.14\)) & \(\mathbf{-3.60}\) (\(0.11\)) & \(0.73\) (\(0.01\)) & \(0.06\) (\(0.00\)) \\ \hline TEC \(E_{LM}\left(\alpha=0\right)\) & \(0.18\) (\(0.01\)) & \(0.66\) (\(0.18\)) & \(-5.91\) (\(0.35\)) & \(0.95\) (\(0.00\)) & \(0.17\) (\(0.01\)) \\ TEC \(\alpha=1^{\frac{1}{2}}\) & \(0.21\) (\(0.01\)) & \(1.10\) (\(0.21\)) & \(-4.79\) (\(0.27\)) & \(0.95\) (\(0.01\)) & \(0.20\) (\(0.01\)) \\ TEC \(\alpha=1\) & \(0.21\) (\(0.01\)) & \(1.10\) (\(0.17\)) & \(-4.82\) (\(0.21\)) & \(0.96\) (\(0.01\)) & \(0.21\) (\(0.01\)) \\ TEC \(\alpha=2\) & \(0.22\) (\(0.01\)) & \(1.26\) (\(0.17\)) & \(-4.67\) (\(0.23\)) & \(0.97\) (\(0.00\)) & \(0.22\) (\(0.01\)) \\ TEC \(E_{G}\left(\alpha=\infty\right)\) & \(\mathbf{0.24}\) (\(0.01\)) & \(\mathbf{1.67}\) (\(0.15\)) & \(-4.94\) (\(0.26\)) & \(0.97\) (\(0.00\)) & \(\mathbf{0.23}\) (\(0.01\)) \\ \hline \hline
**Number of Topics \(\times\)300** & & & & & \\ \hline LDA & \(0.09\) (\(0.01\)) & \(-2.00\) (\(0.19\)) & \(-9.42\) (\(0.22\)) & \(\mathbf{0.97}\) (\(0.00\)) & \(0.09\) (\(0.01\)) \\ NVDM-GSM & \(0.06\) (\(0.02\)) & \(-2.31\) (\(0.37\)) & \(-9.23\) (\(0.39\)) & \(0.69\) (\(0.03\)) & \(0.04\) (\(0.02\)) \\ ProdLDA & \(-0.14\) (\(0.02\)) & \(-5.82\) (\(0.37\)) & \(-13.28\) (\(0.36\)) & \(0.44\) (\(0.17\)) & \(-0.06\) (\(0.03\)) \\ CombinedTM & \(-0.13\) (\(0.03\)) & \(-6.20\) (\(0.47\)) & \(-13.01\) (\(0.38\)) & \(0.15\) (\(0.03\)) & \(-0.02\) (\(0.00\)) \\ WikiPDA & \(0.06\) (\(0.01\)) & \(-0.30\) (\(0.13\)) & \(-5.72
\begin{table}
\begin{tabular}{l|r r r r r} \hline \hline \multicolumn{5}{c}{**MCSUM**} \\ \hline
**Model** & \multicolumn{1}{c}{\(\mathbf{C_{NPMI}}\)} & \multicolumn{1}{c}{\(\mathbf{C_{UCI}}\)} & \multicolumn{1}{c}{\(\mathbf{U_{Mass}}\)} & \multicolumn{1}{c}{**TD**} & \multicolumn{1}{c}{**TQ**} \\ \hline \multicolumn{5}{l}{**Number of Topics \(\times\)100**} \\ \hline LDA & \(-0.02\) (0.01) & \(-3.89\) (0.26) & \(-10.49\) (0.28) & **0.96** (0.00) & \(-0.02\) (0.01) \\ NVDM-GSM & \(0.08\) (0.01) & \(-1.19\) (0.32) & \(-7.44\) (0.64) & \(0.59\) (0.09) & \(0.04\) (0.01) \\ ProdLDA & \(-0.21\) (0.02) & \(-6.79\) (0.42) & \(-12.95\) (0.42) & \(0.36\) (0.04) & \(-0.08\) (0.01) \\ CombinedTM & \(-0.25\) (0.01) & \(-7.54\) (0.24) & \(-12.67\) (0.56) & \(0.25\) (0.09) & \(-0.06\) (0.02) \\ \hline TEC \(E_{LM}\) (\(\alpha=0\)) & \(0.16\) (0.01) & \(0.27\) (0.25) & \(-6.80\) (0.23) & \(0.79\) (0.01) & \(0.13\) (0.01) \\ TEC \(\alpha=\nicefrac{{1}}{{2}}\) & \(\mathbf{0.24}\) (0.01) & \(1.48\) (0.16) & \(-5.49\) (0.16) & \(0.82\) (0.01) & \(0.19\) (0.01) \\ TEC \(\alpha=1\) & \(\mathbf{0.24}\) (0.01) & \(1.45\) (0.17) & \(-5.58\) (0.15) & \(0.82\) (0.01) & \(0.19\) (0.01) \\ TEC \(\alpha=2\) & \(\mathbf{0.24}\) (0.01) & \(\mathbf{1.53}\) (0.15) & **-5.45** (0.19) & \(0.82\) (0.01) & \(\mathbf{0.20}\) (0.01) \\ TEC \(E_{G}\) (\(\alpha=\infty\)) & \(\mathbf{0.24}\) (0.01) & \(1.46\) (0.20) & \(-5.62\) (0.19) & \(0.83\) (0.01) & \(\mathbf{0.20}\) (0.01) \\ \hline \multicolumn{5}{l}{**Number of Topics \(\times\)300**} \\ \hline LDA & \(0.03\) (0.01) & \(-3.06\) (0.17) & \(-10.79\) (0.18) & **0.88** (0.00) & \(0.02\) (0.01) \\ NVDM-GSM & \(0.13\) (0.01) & \(-0.45\) (0.21) & \(-7.00\) (0.28) & \(0.42\) (0.04) & \(0.06\) (0.01) \\ ProdLDA & \(-0.16\) (0.01) & \(-5.54\) (0.13) & \(-12.01\) (0.16) & \(0.17\) (0.01) & \(-0.03\) (0.00) \\ CombinedTM & \(-0.20\) (0.01) & \(-6.44\) (0.15) & \(-12.13\) (0.17) & \(0.12\) (0.01) & \(-0.02\) (0.00) \\ \hline TEC \(E_{LM}\) (\(\alpha=0\)) & \(0.14\) (0.01) & \(-0.24\) (0.14) & \(-8.37\) (0.14) & \(0.74\) (0.01) & \(0.10\) (0.01) \\ TEC \(\alpha=\nicefrac{{1}}{{2}}\) & \(0.22\) (0.01) & \(1.17\) (0.12) & \(-6.84\) (0.14) & \(0.76\) (0.00) & \(0.16\) (0.01) \\ TEC \(\alpha=1\) & \(0.22\) (0.01) & \(1.22\) (0.13) & \(-6.79\) (0.12) & \(0.76\) (0.01) & \(\mathbf{0.17}\) (0.01) \\ TEC \(\alpha=2\) & \(0.22\) (0.01) & \(\mathbf{1.27}\) (0.08) & **-6.75** (0.12) & \(0.76\) (0.01) & \(\mathbf{0.17}\) (0.01) \\ TEC \(E_{G}\) (\(\alpha=\infty\)) & \(\mathbf{0.23}\) (0.01) & \(1.26\) (0.09) & \(-6.77\) (0.13) & \(0.76\) (0.01) & \(\mathbf{0.17}\) (0.00) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results on MLSUM corpus for all topic models. We record the results on five metrics, including \(\mathbf{C_{NPMI}}:\) Normalized pointwise mutual information, more correlated with humans, \(\mathbf{U_{Mass}}:\) How often a word appears with another against how often it appears on its own \(\mathbf{TD}:\) (Topic Diversity) the ratio of unique entities to total entities and \(\mathbf{TQ}:\) (Topic Quality) Topic Diversity \(\times\)\(\mathbf{C_{NPMI}}\). The results are reported as averages (95% confidence interval) based on 10 random experimental runs. Our model outperforms all baselines across all metrics except for \(\mathbf{TD}\).

\begin{table}
\begin{tabular}{l|r r r r} \hline \hline \multicolumn{5}{c}{**MLSUM**} \\ \hline
**Model** & \multicolumn{1}{c}{\(\mathbf{C_{NPMI}}\)} & \multicolumn{1}{c}{\(\mathbf{C_{UCI}}\)} & \multicolumn{1}{c}{\(\mathbf{U_{Mass}}\)} & \multicolumn{1}{c}{**TD**} & \multicolumn{1}{c}{**TQ**} \\ \hline
**Number of Topics \(\times\)100** & & & & \\ \hline LDA & \(-0.02\) (0.01) & \(-3.89\) (0.26) & \(-10.49\) (0.28) & **0.96** (0.00) & \(-0.02\) (0.01) \\ NVDM-GSM & \(0.08\) (0.01) & \(-1.19\) (0.32) & \(-7.44\) (0.64) & \(0.59\) (0.09) & \(0.04\) (0.01) \\ ProdLDA & \(-0.21\) (0.02) & \(-6.79\) (0.42) & \(-12.95\) (0.42) & \(0.36\) (0.04) & \(-0.08\) (0.01) \\ CombinedTM & \(-0.25\) (0.01) & \(-7.54\) (0.24) & \(-12.67\) (0.56) & \(0.25\) (0.09) & \(-0.06\) (0.02) \\ \hline TEC \(E_{LM}\) (\(\alpha=0\)) & \(0.16\) (0.01) & \(0.27\) (0.25) & \(-6.80\) (0.23) & \(0.79\) (0.01) & \(0.13\) (0.01) \\ TEC \(\alpha=\nicefrac{{1}}{{2}}\) & \(\mathbf{0.24}\) (0.01) & \(1.48\) (0.16) & \(-5.49\) (0.16) & \(0.82\) (0.01) & \(0.19\) (0.01) \\ TEC \(\alpha=1\) & \(\mathbf{0.24}\) (0.01) & \(1.45\) (0.17) & \(-5.58\) (0.15) & \(0.82\) (0.01) & \(0.19\) (0.01) \\ TEC \(\alpha=2\) & \(\mathbf{0.24}\) (0.01) & \(\mathbf{1.53}\) (0.15) & **-5.45** (0.19) & \(0.82\) (0.01) & \(\mathbf{0.20}\) (0.01) \\ TEC \(E_{G}\) (\(\alpha=\infty\)) & \(\mathbf{0.24}\) (0.01) & \(1.46\) (0.20) & \(-5.62\) (0.19) & \(0.83\) (0.01)
**Coherence NPMI.**Bouma (2009) proposes an alternative coherence measure, where the above elements are substituted by _Normalized PMI_ (NPMI, Eq. (5)) as it was found that these have higher correlation to human topic coherence ratings:

\[\mathrm{NPMI}(w_{i},w_{j})=\frac{\mathrm{PMI}(w_{i},w_{j})}{-\log\left(p(w_{i},w _{j})\right)}. \tag{5}\]

**Coherence UMass.**Mimno et al. (2011) suggests the asymmetrical coherence measure _UMass_ (Eq. 6), which is also calculated based on intrinsic entity co-occurrences conditioned to top entity occurrences:

\[\mathrm{UMass}(w_{i},w_{j})=\log\left(\frac{p(w_{i},w_{j})}{p(w_{j})}\right). \tag{6}\]

**Topic diversity and quality.** Topic diversity (**TD**) is the ratio between the number of unique entities and the total number of entities, considering the top 25 entities per topic (Dieng et al., 2020). Topic quality (**TQ**) is the product of topic coherence, as measured by \(\mathrm{C_{NPMI}}\), and topic diversity.

### Experiments specifications

For each combination of model, corpus, and the number of topics -- 100 and 300 --, we compute metrics over 10 runs and present both the averages and 95% confidence interval range in Tables 3, 4 and 5. We use sequential seeds for the sake of reproducibility. We use implementation defaults for all models, including TEC, with the exceptions of NVDM-GSM, where we run 100 epochs, and ProdLDA and CombinedTM, that we run each for 250 epochs. We report metrics for the epoch with higher \(\mathrm{C_{NPMI}}\).

We run the experiments in a shared Linux machine with 72 CPU cores, 256GB RAM and use a Tesla V100-SXM2-16GB GPU.

### Qualitative results

We present exemplar topics for the different models in Table 2. Using visual inspection, we find cases where some top entities do not match the general topic theme. These must be attributed to limitations in the model as they all share the same preprocessed corpora, with the exception of WikiPDA. Overall these issues seem less prevalent with TEC. Particularly for ProdLDA and CombinedTM, we also find unrelated entities that linger across many topics, with the lingering entities varying between runs. We also find topics covering multiple themes, such as the ones resulting from LDA and WikiPDA.

### Quantitative results

For all combinations of corpora and number of topics, TEC achieves better \(\mathbf{C_{NPMI}}\), \(\mathbf{C_{UCI}}\), \(\mathbf{U_{Mass}}\) and \(\mathbf{TQ}\) when compared to the other models. \(\mathbf{U_{Mass}}\) has a single exception where WikiPDA performs better for 100 topics.

As opposed to word-based preprocessing, entity extraction results in sparser representations of corpora, and, for that reason, we observe significantly worse results to those presented in the original topic model papers.

Documents generally assume the reader has background knowledge on the subject. Models like LDA, NVDM-GSM, ProdLDA and WikiPDA learn based on entity co-occurrences. Relationships that are not explicit are neglected, justifying their lack of performance in comparison to TEC. WikiPDA considers the relationship across entities during its preprocessing however training is based on LDA so the same limitations apply to it. CombinedTM uses implicit knowledge, but much like ProdLDA, seems to be affected by component collapse as can be verified by their low \(\mathbf{TD}\) scores -- a state where variational autoencoders can get stuck in a poor local optimum, due to the choice in objective function, that results in topics being similar (Masada, 2022).

The results suggest that both embedding types are valuable sources of knowledge to use with topic models. Models using graph-based embeddings perform significantly better than models using embeddings obtained with language models, and we only find a few circumstances where some combination of both embeddings produces better results than graph-based embeddings alone.

## 5 Conclusions

We explore entity-based topic models based on the clustering of vector representations of entities. TEC internally represents documents using language-agnostic entity identifiers, which results in a single set of topics shared across languages and allows it to extend to new languages without sacrificing the performance of the existing languages.

Our results suggest that the implicit knowledge provided by language models is superior to the state-of-the-art in terms of coherence and quality. Nevertheless, these results are surpassed by the explicit knowledge encoded in graph-based embeddings, using human contributed Wikidata knowledge base as a source.
## 6 Limitations

TEC assumes that documents contain entities, yet this is not necessarily the case. The proposed model is specifically valuable for entity-rich applications such as news articles. A potential solution we are interested in exploring in the future is to train a self-supervised model to generate word embeddings using the bag-of-words as input and the document embedding as the target. It results in word embeddings having a representation in the shared embedding space.

We produce graph-based embeddings using _node2vec_ -- a shallow neural network that may be unable to learn deeper, more complex relationships between entities. We believe that our results can improve if we obtain embeddings using a multi-layer graph neural network with unsupervised training. Furthermore, while our approach outperforms other models on a range of metrics, they still lag behind when it comes to topic diversity. Finding a way to improve the diversity of the topics while preserving their intrinsic performance could make for important future work.

Lastly, updating the knowledge base will force the retraining of the model, which does not currently guarantee a direct relationship between former and new topics. It requires additional research as this can be a hindrance for some applications.
