\begin{table}[]
\begin{tabular}{lll}
\hline
Model & R-1 & BERTScore \\ \hline
multi-lexsum-long & 38 & 85 \\
multi-lexsum-tiny & 3 & 81 \\
ms2 & 3 & 81 \\
multixscience & 15 & 80 \\
topic lists & 1 & 76 \\ \hline
\end{tabular}
\caption{
\label{study_evaluation}
Rouge-1 (R-1) and BERTScore (F1) results for each models topic representations measured against.
}
\end{table}
\begin{abstract}
Topic models represent groups of documents as a list of words (the topic labels). This work asks whether an alternative approach to topic labeling can be developed that is closer to a natural language description of a topic than a word list. To this end, we present an approach to generating human-like topic labels using abstractive multi-document summarization (MDS). We investigate our approach with an exploratory case study. We model topics in citation sentences in order to understand what further research needs to be done to fully operationalize MDS for topic labeling. Our case study shows that in addition to more human-like topics there are additional advantages to evaluation by using clustering and summarization measures instead of topic model measures. However, we find that there are several developments needed before we can design a well-powered study to evaluate MDS for topic modeling fully. Namely, improving cluster cohesion, improving the factuality and faithfulness of MDS, and increasing the number of documents that might be supported by MDS. We present a number of ideas on how these can be tackled and conclude with some thoughts on how topic modeling can also be used to improve MDS in general.
\end{abstract}

Topic models have tended to represent a topic as a list of words. Traditional topic labels are supposed to be “a set of terms, when viewed together, enable human recognition of an identifiable category” \citep{hoyle_is_2021}. However, a set of terms do not align with our intuitive understandings of what a topic is: a common theme or concept explicated as a word, phrase, or natural language description \citep{shadrova_topic_2021}. In this paper, we present an exploratory case study using multi-document summaries (MDS) as labels for clusters of citations in order to understand current limitations and future work needed for using abstractive topic labels for human-like topics of scientific documents. To our knowledge, it is the first work that proposes to use MDS for topic labeling on top of topic clusters constructed with contextualized embeddings.

Some work has anticipated this challenge by developing topic representations with phrases \citep{popa_bart-tl_2021} and summaries \citep{basave_automatic_2014, gourru_united_2018, wan_automatic_2016}. But those works tend to be extractive, drawing the phrase or summary from a single document in the cluster\footnote{see \citet{alokaili_automatic_2020,popa_bart-tl_2021} for recent abstractive works.}. In the extractive setting, \textit{there may be no existing and fluent phrase or sentence that is capable of describing all documents in the cluster} or there may be multiple and even conflicting subtopics in the cluster that require a longer abstractive representation for producing a factual summary.

\subsection{Topic modeling as clustering and MDS}
In order to address the issues presented above, we propose using abstractive MDS as an approach to topic labeling. Topic modeling can be reframed as a set of two tasks: (1) finding meaningful clusters for documents \citep{sia_tired_2020, zhang_is_2022} and (2) performing MDS on those individual clusters to find meaningful topic labels. In this framework, LDA \cite{blei_latent_2003} uses document-word distributions to construct clusters and word lists drawn from those clusters as a form of MDS. Since we are looking at abstractive MDS that moves beyond word lists, we propose that the \textit{topic representation be a sentence or paragraph} but there is no reason why an abstractive MDS can't be trained to generate phrases or even word lists (see \citet{alokaili_automatic_2020}) since word lists may still be appropriate in some situations.

\subsection{Evaluation}
Topic model evaluation is challenging (see \citet{chang_reading_2009, hoyle_is_2021, doogan_topic_2021}). Traditional metrics like coherence (NPMI), perplexity, and diversity scores are studied in the context of topic word lists and validated with correlation to human ratings of the utility or coherence of those topic word lists. Since we suggest developing abstractive topic representations, we want a way to compare various forms of both abstractive and extractive topic representations presented by the model. Since we are treating representation as a summarization task and this task includes measures that work across extractive and abstractive settings, we suggest that we start with standard summarization metrics such as overlap metrics like Rogue (as used in \citet{cui_topic-guided_2021} or semantic metrics such as BERTScore \citep{zhang_is_2022} (as used in \citet{alokaili_automatic_2020}).

\section{Case study: how has a scientific document been cited?}
To evaluate our proposed method, we chose topic modeling over scientific documents as a setting. While several methods exist for determining citation intent function \citep{basuki_sdcf_2022, nicholson_scite_2021} and the relationship between two papers \citep{luu_explaining_2021}, there is very little work on topic models over citations (for some representative work on "citation summary" see \citet{elkiss_blind_2008, wang_generating_2021, zou_citation_2021}). Topic representations of citations are interesting for characterizing trends in how a paper has been cited or helping researchers identify relevant citations to read among potentially thousands of other citations. In this work, we treat topic labels as a "citation intent" label and use the proposed approach to understand the utility of MDS for topic modeling in this setting.

In order to identify meaningful groups of clusters we use contextualized topic models (CTM) \citep{bianchi_pre-training_2021} since this method uses contextualized word embeddings (we used SPECTER for constructing embeddings \citep{cohan-etal-2020-specter}). We selected CTM since we still get word lists as topic labels which we used for evaluation. In order to select the number of topics hyperparameter, we trained CTM several times steadily increasing the number of topics from 3 to 50 and selected the best model according to coherence (NPMI) resulting in a 10 topic model (see Appendix \ref{ctm} for more details) over 183 citation statements.

Table \ref{study_evaluation} shows the Rogue-1 (R-1) and BERTScore (average F1 across topics) for each of the models selected for generating topic representations using MDS as well as the topic lists generated by CTM. It is important to underscore that R-1 and BERTScore are not validated against human studies for topic representations and this is simply a small case study on what an approach might look like. In spite of this, our results paint an initial picture of how these methods perform, especially when compared to model outputs (see Appendix \ref{outputs} for samples). Topic word lists have the worst R-1 and BERTScore. The MDS models do a little bit better with multi-lexsum-long having the best overall score. multixscience also does well with regards to R-1. Since multixscience and multi-lexsum-long are long form summaries, it appears that R-1 is potentially biased towards longer summaries and may not be a good measure across representations, in particular it may be uninformative for evaluating the performance of topic lists. ms2 and multi-lexsum-tiny are smaller and have better BERTScore then multixscience indicating they might provide more semantically similar representations. We are also not sure whether BERTScore suffers from the same bias towards longer or more sentence-like inputs.
{"value": "38", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["Rouge-1", "R-1"], "experimental settings": {"xx": "yy"}, "model": "multi-lexsum-long", "model settings": {"xx": "yy"}} {"value": "85", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["BERTScore", "F1", "BERTScore (F1)"], "experimental settings": {"xx": "yy"}, "model": "multi-lexsum-long", "model settings": {"xx": "yy"}} {"value": "3", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["Rouge-1", "R-1"], "experimental settings": {"xx": "yy"}, "model": "multi-lexsum-tiny", "model settings": {"xx": "yy"}} {"value": "81", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["BERTScore", "F1", "BERTScore (F1)"], "experimental settings": {"xx": "yy"}, "model": "multi-lexsum-tiny", "model settings": {"xx": "yy"}} {"value": "3", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["Rouge-1", "R-1"], "experimental settings": {"xx": "yy"}, "model": "ms2", "model settings": {"xx": "yy"}} {"value": "81", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["BERTScore", "F1", "BERTScore (F1)"], "experimental settings": {"xx": "yy"}, "model": "ms2", "model settings": {"xx": "yy"}} {"value": "15", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["Rouge-1", "R-1"], "experimental settings": {"xx": "yy"}, "model": "multixscience", "model settings": {"xx": "yy"}} {"value": "80", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["BERTScore", "F1", "BERTScore (F1)"], "experimental settings": {"xx": "yy"}, "model": "multixscience", "model settings": {"xx": "yy"}} {"value": "1", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["Rouge-1", "R-1"], "experimental settings": {"xx": "yy"}, "model": "topic lists", "model settings": {"xx": "yy"}} {"value": "76", "type": "Result", "training data/set": "183 citation statements", "test data/set": "183 citation statements", "task": ["topic modeling", "topic representation"], "metric": ["BERTScore", "F1", "BERTScore (F1)"], "experimental settings": {"xx": "yy"}, "model": "topic lists", "model settings": {"xx": "yy"}}