\begin{table}[]
\begin{tabular}{lll}
\hline
coherence (NPMI) & diversity & topics \\ \hline
-0.25 & 0.97 & 10 \\
-0.27 & 0.97 & 20 \\
-0.32 & 0.96 & 25 \\
-0.32 & 0.97 & 15 \\
-0.35 & 1.00 & 5 \\
-0.38 & 1.00 & 3 \\
-0.38 & 0.97 & 50 \\ \hline
\end{tabular}
\caption{
\label{ctm_training}
Selecting CTM topic model by evaluating CTM coherence (NPMI) and diversity on different topic numbers parameters.
}
\end{table}
\subsection{Topic modeling as clustering and MDS}
In order to address the issues presented above, we propose using abstractive MDS as an approach to topic labeling. Topic modeling can be reframed as a set of two tasks: (1) finding meaningful clusters for documents \citep{sia_tired_2020, zhang_is_2022} and (2) performing MDS on those individual clusters to find meaningful topic labels. In this framework, LDA \cite{blei_latent_2003} uses document-word distributions to construct clusters and word lists drawn from those clusters as a form of MDS. Since we are looking at abstractive MDS that moves beyond word lists, we propose that the \textit{topic representation be a sentence or paragraph} but there is no reason why an abstractive MDS can't be trained to generate phrases or even word lists (see \citet{alokaili_automatic_2020}) since word lists may still be appropriate in some situations.

\subsection{Evaluation}
Topic model evaluation is challenging (see \citet{chang_reading_2009, hoyle_is_2021, doogan_topic_2021}). Traditional metrics like coherence (NPMI), perplexity, and diversity scores are studied in the context of topic word lists and validated with correlation to human ratings of the utility or coherence of those topic word lists. Since we suggest developing abstractive topic representations, we want a way to compare various forms of both abstractive and extractive topic representations presented by the model. Since we are treating representation as a summarization task and this task includes measures that work across extractive and abstractive settings, we suggest that we start with standard summarization metrics such as overlap metrics like Rogue (as used in \citet{cui_topic-guided_2021} or semantic metrics such as BERTScore \citep{zhang_is_2022} (as used in \citet{alokaili_automatic_2020}).

\section{Case study: how has a scientific document been cited?}
To evaluate our proposed method, we chose topic modeling over scientific documents as a setting. While several methods exist for determining citation intent function \citep{basuki_sdcf_2022, nicholson_scite_2021} and the relationship between two papers \citep{luu_explaining_2021}, there is very little work on topic models over citations (for some representative work on "citation summary" see \citet{elkiss_blind_2008, wang_generating_2021, zou_citation_2021}). Topic representations of citations are interesting for characterizing trends in how a paper has been cited or helping researchers identify relevant citations to read among potentially thousands of other citations. In this work, we treat topic labels as a "citation intent" label and use the proposed approach to understand the utility of MDS for topic modeling in this setting.

For this study, we used scite.ai \citep{nicholson_scite_2021} to extract in-text passages which contained citations (citation statements) to the paper \cite{lau_machine_2014}, a well known paper that introduces the NPMI metric in topic modeling. This resulted in 183 citation statements which is the corpus we will use for topic modeling.

In order to identify meaningful groups of clusters we use contextualized topic models (CTM) \citep{bianchi_pre-training_2021} since this method uses contextualized word embeddings (we used SPECTER for constructing embeddings \citep{cohan-etal-2020-specter}). We selected CTM since we still get word lists as topic labels which we used for evaluation. In order to select the number of topics hyperparameter, we trained CTM several times steadily increasing the number of topics from 3 to 50 and selected the best model according to coherence (NPMI) resulting in a 10 topic model (see Appendix \ref{ctm} for more details) over 183 citation statements.

The models selected for generating abstractive MDS are outlined in Table \ref{models_table}. All MDS models used are based on the longformer architecture \citep{beltagy_longformer} and used beam search (5 beams) with greedy decoding.

We randomly sampled 3 topics to explore their representations. As an example, table \ref{topic_evaluation} shows representations using the multi-lexsum-tiny model (full details are available in Appendix \ref{outputs}. In representations for topic 0 (Table \ref{topic_0}), we see there is a general agreement across models that the citing documents are discussing measurement. We can see that the topic representations appear to be split between measuring interpretability (multixscience, multi-lexsum-long) and those discussing the correlation between measures (ms2, multi-lexsum-long) or even potentially an additional topic of describing measures used (multi-lexsum-tiny). Conflicting summaries are not surprising given issues in MDS with regards to summarizing diverse and potentially conflicting documents \citep{deyoung_ms2_2021}. Table \ref{topic_0} shows a diversity of topic labels that might be appropriate under different scenarios of applying topic models. Labels like the ones in Table \ref{topic_evaluation} might be useful for labels that are easy and fast to read while longer summaries in multixscience and multi-lexsum-long might be useful for users who want to engage deeper.

Table \ref{ctm_training} describes the evaluation of all the CTM \citep{bianchi_pre-training_2021} models we trained by coherence (NPMI), diversity, and topic number. We trained the model on a dataset of 183 citation statements which are in-text passages from publications citing \citep{lau_machine_2014} retrieved from scite.ai \citep{nicholson_scite_2021}. This dataset was also used as the reference corpus for evaluating NPMI in this initial case study. We acknowledge that when building robust topic models a standard reference corpus should be used so results can be comparable in future works.
{"value": "-0.25", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "10"}} {"value": "0.97", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "10"}, "model": "CTM", "model settings": {"topic number": "10"}} {"value": "10", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"} {"value": "-0.27", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "20"}} {"value": "0.97", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "20"}, "model": "CTM", "model settings": {"topic number": "20"}} {"value": "20", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"} {"value": "-0.32", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "25"}} {"value": "0.96", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "25"}, "model": "CTM", "model settings": {"topic number": "25"}} {"value": "25", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"} {"value": "-0.32", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "15"}} {"value": "0.97", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "15"}, "model": "CTM", "model settings": {"topic number": "15"}} {"value": "15", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"} {"value": "-0.35", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "5"}} {"value": "1.00", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "5"}, "model": "CTM", "model settings": {"topic number": "5"}} {"value": "5", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"} {"value": "-0.38", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "3"}} {"value": "1.00", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "3"}, "model": "CTM", "model settings": {"topic number": "3"}} {"value": "3", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"} {"value": "-0.38", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": ["coherence (NPMI)", "NPMI"], "experimental settings": {"xx": "yy"}, "model": "CTM", "model settings": {"topic number": "50"}} {"value": "0.97", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": "topic modeling", "metric": "diversity", "experimental settings": {"topic number": "50"}, "model": "CTM", "model settings": {"topic number": "50"}} {"value": "50", "type": "Hyper-parameter/Architecture", "model": "CTM", "parameter/architecture name": "topic number", "dataset": "183 citation statements"}