# Towards Turing-Complete Quantum Computing Coming From Classical Assembler

Thomas Gabor, Marian Lingsch Rosenfeld, Claudia Linnhoff-Popien  LMU Munich
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Instead of producing quantum languages that are fit for current quantum
computers, we build a language from standard classical assembler and augment
it with quantum capabilities so that quantum algorithms become a subset of it.
This paves the way for the development of hybrid algorithms directly from
classical software, which is not feasible on today’s hardware but might
inspire future quantum programmers.

###### Index Terms:

quantum computing, quantum language, assembler

##  I Introduction

The capabilities of current quantum computers remain very limited, most
importantly with respect to the depth of the executed algorithm. As
astonishing as it is that we can still run notably important algorithms (maybe
even better than classical ones [1, 2]) on this kind of hardware, the
algorithms that can be fully implemented on a quantum computer lack several
characteristics that programmers are used to for classical algorithms, most
notably: recursion. The absence of arbitrary-depth recursion (or simply jumps)
prohibits pure quantum algorithm from emulating every algorithm that can be
run on a Turing machine.111We leave out the discussion that finite-memory
systems can never execute all algorithms a Turing machine can execute. For
today’s quantum computers, this is a more than sensible assumption as
decoherence times only allow for a couple of gates to be executed in sequence
before all useful information is lost. But what if we had the best quantum
computer we could wish for?

In contrast to most other approaches (most notably quantum assembly languages
[3]) we attempt to design an assembly language not for quantum computers with
today’s capabilities but for an imaginary quantum computer that is not limited
by today’s challenges in hardware design and maybe not even limited by all the
rules of quantum mechanics at all times. However, we argue that such a
language (even though conceivably it will not run on any real quantum computer
for a long time) can still be useful for two main reasons:

  * •

First, for now and at least for the immediate future a large amount of quantum
algorithms will be run on classical simulators. As these classical simulators
run on classical machines, it is reasonable to write algorithms that can
access these classical machines’ capabilities directly and thus forego any
quantum simulation when possible. This may produce algorithms that – while not
compatible with (current) quantum computers – may at least not be as
superfluously slow when run on a quantum simulator for testing purposes or
simply due to a lack of better quantum options.

  * •

Second, we need tools to discover new quantum algorithms that are more
accessible to programmers well versed in standard programming languages. As of
now, most setups of quantum software are built around a few algorithm
archetypes which are known or suspected to provide some form of quantum
advantage. As these few archetypes already hold up all the fascination
regarding quantum computers, the discovery of new archetypes should be a big
step towards more usage scenarios for quantum computers. Enabling programmers
to write an algorithm classically first and then translate it part by part to
quantum-suitable reformulations (while maintaining a runnable program) could
enable new creativity in algorithm design.

##  II Approach

In this work-in-progress paper we provide a very simple assembly language that
we augmented with a few quantum-specific instructions and capabilities. Table
I shows all instructions at our disposal. We provide basic mathematical and
logical operations as well as the “set” and “swap” instructions. We then
provide a set of instructions to manipulate program flow including the very
simple “setpc” and “jump” instructions and the more elaborate “ifte”
instruction, which provides the functionality of a standard if-then-else
expression in most programming languages. However, all of our registers can be
quantum registers, which means that they can contain a superposition instead
of a single scalar value. This also means that our program counter can be in
superposition, which causes the entire program state to be in superposition
and possibly execute different instructions in each of its superposed
branches.

To actually introduce said superpositions into the registers, Table I also
shows a few quantum-specific instructions. Most notably, we can apply a
Hadamard gate to a single qubit or a range of qubits. We also provide a
shortcut for the diffusion operator used within Grover’s algorithm as well as
for phase multiplication.

Obviously, this set of instruction is far from complete even for classical
algorithms but it allows us to formulate a few simple examples of how to use
our language.222See https://github.com/marian-lingsch/quantum-assembler.

Classic Instructions |
---|---
add d$i$ d$j$ d$k$ |  add the value of cell $i$ with the value of cell $j$ and write the result to cell $k$
mul d$i$ d$j$ d$k$ |  multiply the value of cell $i$ with the value of cell $j$ and write the result to cell $k$
div d$i$ d$j$ d$k$ |  divide the value of cell $i$ with the value of cell $j$ and write the result to cell $k$
sub d$i$ d$j$ d$k$ |  substract the value of cell $j$ from the value of cell $i$ and write the result to cell $k$
sqrt d$i$ d$j$ |  write the square root of the value of cell $i$ into cell $j$
mod d$i$ d$j$ d$k$ |  writes into cell $k$ the modulo of the value of cell $i$ with respect to the value of cell $j$
neg d$i$ |  negate the boolean value of the cell $i$
and d$i$ d$j$ d$k$ |  write into cell $k$ the boolean value resulting from v$i$ $\land$ v$j$, where v$l$ is the value of cell d$l$
or d$i$ d$j$ d$k$ |  write into cell $k$ the boolean value resulting from v$i$ $\lor$ v$j$, where v$l$ is the value of cell d$l$
set d$i$ $j$ |  set the value of cell $i$ to $j$
swap d$i$ d$j$ |  swap the values of cells $i$ and $j$
setpc d$i$ |  set the program counter to the value of cell $i$
jump $i$ |  set the program counter to the value of $i$
skip |  do nothing
stop |  remain in this instruction indefinitely
ifte d$i$ $j$ $k$ |  if cell $i$ is true, set the pc to the value of $j$ else set the pc to the value of $k$
Quantum Instructions |
havoc d$i$ $j$ $k$ |  apply Hadamard gates to the qubits in range from $j$ to $k$ of cell $i$
havocb $i$ |  apply a Hadamard gate to the data qubit $i$
diffusion |  apply Grover’s diffusion operator
phase $x$ $y$ |  multiply the phase of the state by $x+\mathrm{i}\mkern 1.0mu\cdot y$
TABLE I: Instruction set for assembly with quantum capabilities

##  III Examples

Listing 1 shows a simple example of what we can easily produce when we apply
superposition to the program counter. Lines 12–15 all multiply the value in d4
by 2 (see line 6). However, we do not always execute all of them. Instead, we
initialize the program counter to the start of these lines (see line 5) and
then add the equally distributed superposition of the values $\langle
0,1,2,3\rangle$ to that value. When all superposed branches of the program
have been executed, we end up with an equal superposition of values $\langle
1,2,4,8\rangle$ saved in register d4. We were able to define such a superposed
value using patterns familiar from imperative programming and the approach can
be generalized to more complicated superposed program flows.

Listing 2 shows how we could evolve quantum algorithms from classical ones. We
use a staple example of quantum computing, i.e., finding the prime factors of
a given number; but we start out with the most common classical approach to
that problem, i.e., the sieve of Eratosthenes [4]. If we assume that in the
classical world we would simply derive the prime factors of a number by trying
out all possible divisors, it appears evident that such a search for a divisor
might be accelerated by employing Grover’s algorithm [5]. Naturally, this is a
bit counterintuitive to anyone familiar with Shor’s algorithm [6], which we
will discuss in a bit. However, the approach still works: In Listing 2, we
assume that the number we want to split into factors is given as “NUMBER1”
(see line 4) and just to keep things simple we assume that said number’s bit
count is given as “NUMBER2”. Obviously, a slightly more powerful assembly
language could just derive that information with a single instruction. Line 8
then creates a superposition of all numbers of the same length as our input
“NUMBER1” and with line 9 we ensure that this superposition no longer contains
the values $0$ and $1$ so that we can try all remaining values as possible
divisors. In line 10 we again utilize the superposed value in d1 to compute
the remainder for all divisors. We then set up the parameters for Grover’s
search (lines 11–16) and subsequently perform the calculated amount of
iterations within Grover’s search, looking for a single $0$ remainder within
the superposed value of d2. Note that we implemented the instructions “phase”
and “diffusion” within the instruction set specifically to enable a
straightforward definition of algorithms like Listing 2 based on Grover’s
search. Specialized instructions like these not only increase readability but
might also allow future versions of the simulator to implement classical
shortcuts for certain behavior.

⬇

1; nondeterministic multiplication

; produces multiple powers of 2

3start:

havoc d0 0 2

5 set d1 subMul

set d2 2

7 set d4 1

add d0 d1 d3

9 setpc d3

11subMul:

mul d4 d2 d4

13 mul d4 d2 d4

mul d4 d2 d4

15 mul d4 d2 d4

stop

Listing 1: Nondeterministic multiplication

⬇

1start:

; set number for which is to

3 ; be factorized

set d0 NUMBER1

5 ; havoc is used in range format

; each int is represented by 8 bits

7 ; NUMBER2 equals log2(sqrt(NUMBER1))

havoc d1 0 NUMBER2

9 add d1 2 d1

mod d0 d1 d2

11 ; d3 is the amount of iterations

; of Grover’s algorithm

13 add d0 0 d3

sqrt d3 d3

15 div d3 4 d3

mul d3 3 d3

17

grover:

19 ifte d2 jumpTrue jumpFalse

21jumpFalse:

phase -1.0 0.0

23 jump subDiffusion

25jumpTrue:

skip

27 jump subDiffusion

29subDiffusion:

diffusion

31 sub d3 1 d3

ifte d3 subStop grover

33

subStop:

35 stop

Listing 2: Factoring via Grover

We compared the quasi-quantum program in Listing 2 to its purely classical
counterpart, for which we do not provide an additional listing to save space
(and because it looks entirely as expected). It is important to note that we
also wrote the classical algorithm for the sieve of Eratosthenes using the
same quantum-enabled assembly language and ran it using the same quantum
simulator [7]. Figure 1 shows the respective run times of both algorithms for
the whole range of inputs. We can see that the classical algorithm takes the
longest on actual prime numbers where it needs to try out every possible
divisor while the quantum algorithm needs to produce the superposition of all
possible divisors for every single input value regardless. However, it is
interesting to note that within our simulator, both approaches show a
remarkably similar performance. This justifies our approach to offer a “soft
transition” from a purely classical execution of a purely classical algorithm
to a classical simulation of an algorithm with quantum parts without
necessarily blow up the run time disproportionally.

Figure 1: Comparison between classical and quantum factoring algorithms
written in our assembly language.

##  IV Conclusion

The assembly language shown here is very limited but is intended to provide an
idea how to re-think hybrid programming coming from full-fledged classical
languages. We have analyzed how quantum program flow might look like on an
assembly level and how it may allow to construct complex quantum states in an
intuitive and imperative manner (Listing 1). We have also presented how very
versatile quantum approaches like Grover’s search can be integrated into well-
known classical algorithms in a straightforward way (Listing 2), even though
we sacrifice the ability to run the resulting algorithms on today’s quantum
computers immediately. In that case, we can also see where the approach still
falls short: While implementing Grover’s search here may provide some
benefits, quantum programmers know that Shor’s algorithm provides an even
better way to solve the factorization problem on a quantum computer. However,
that algorithm is not as easily derived from its classical counterpart – or at
least our language does not yet allow to do so. Future versions of the
language should provide more versatile and more powerful instruction sets in
order to provide a “soft transition” into quantum software for more quantum
algorithms.

In principle, there is no reason to keep this “transition language” at an
assembly level; in fact, for usability it might be especially important to
translate our concepts of quantum program flow to more high-level language
models (and perhaps offer automatic translation back to assembly level to
maintain compatibility). An ideal case might be to allow a programmer to use a
modern programming language and insert quantum-compatible expressions one by
one until an interpreter or compiler notices that a certain module is now
fully quantum-compatible and then offers the option to also run it on quantum
hardware (compare Tyagi et al. [8] for the case of reversible computing).
However, high-level abstractions for quantum computing that are both intuitive
and powerful are still very much sought after.

## References

  * [1] S. Bravyi, D. Gosset, and R. König, “Quantum advantage with shallow circuits,” _Science_ , vol. 362, no. 6412, pp. 308–311, 2018.
  * [2] F. Arute, K. Arya, R. Babbush, D. Bacon, J. C. Bardin, R. Barends, R. Biswas, S. Boixo, F. G. Brandao, D. A. Buell _et al._ , “Quantum supremacy using a programmable superconducting processor,” _Nature_ , vol. 574, no. 7779, pp. 505–510, 2019.
  * [3] A. W. Cross, A. Javadi-Abhari, T. Alexander, N. de Beaudrap, L. S. Bishop, S. Heidel, C. A. Ryan, J. Smolin, J. M. Gambetta, and B. R. Johnson, “OpenQASM 3: A broader and deeper quantum assembly language,” _arXiv preprint arXiv:2104.14722_ , 2021.
  * [4] C. Hoare, “Proof of a structured program: ‘The sieve of Eratosthenes’,” _The Computer Journal_ , vol. 15, no. 4, pp. 321–325, 1972.
  * [5] A. Mandviwalla, K. Ohshiro, and B. Ji, “Implementing Grover’s algorithm on the IBM quantum computers,” in _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 2018, pp. 2531–2537.
  * [6] B. P. Lanyon, T. J. Weinhold, N. K. Langford, M. Barbieri, D. F. James, A. Gilchrist, and A. G. White, “Experimental demonstration of a compiled version of Shor’s algorithm with quantum entanglement,” _Physical Review Letters_ , vol. 99, no. 25, p. 250505, 2007.
  * [7] T. Gabor, M. Lingsch Rosenfeld, and C. Linnhoff-Popien, “Simple quantum state encodings for hybrid programming of quantum simulators,” in _2nd Workshop on Quantum Software Architecture (QSA)_ , 2022.
  * [8] N. Tyagi, J. Lynch, and E. D. Demaine, “Toward an energy efficient language and compiler for (partially) reversible algorithms,” in _International Conference on Reversible Computation_. Springer, 2016, pp. 121–136.


11institutetext: Jožef Stefan Institute, Ljubljana, Slovenia 11email:
<EMAIL_ADDRESS>
# Retrieval-efficiency trade-off of Unsupervised
Keyword Extraction

Blaž Škrlj 11 Boshko Koloski 11 Senja Pollak 11

###### Abstract

Efficiently identifying keyphrases that represent a given document is a
challenging task. In the last years, plethora of keyword detection approaches
were proposed. These approaches can be based on statistical (frequency-based)
properties of e.g., tokens, specialized neural language models, or a graph-
based structure derived from a given document. The graph-based methods can be
computationally amongst the most efficient ones, while maintaining the
retrieval performance. One of the main properties, common to graph-based
methods, is their immediate conversion of token space into graphs, followed by
subsequent processing. In this paper, we explore a novel unsupervised approach
which merges parts of a document in sequential form, _prior to_ construction
of the token graph. Further, by leveraging personalized PageRank, which
considers frequencies of such sub-phrases alongside token lengths during node
ranking, we demonstrate state-of-the-art retrieval capabilities while being up
to two orders of magnitude faster than current state-of-the-art unsupervised
detectors such as YAKE and MultiPartiteRank. The proposed method’s scalability
was also demonstrated by computing keyphrases for a biomedical corpus
comprised of 14 million documents in less than a minute.

Note: The paper was accepted to DS2022.

###### Keywords:

Keyphrase detection Natural language processing Text mining

##  1 Introduction

With the increasing amounts of freely available text-based data sets, methods
for efficient keyphrase detection are becoming of high relevance [13]. These
methods, given a single or multiple documents, output a ranked list of short
phrases (or single tokens), which represents key aspects of the input text. In
the recent years, plethora of keyphrase extraction methods were presented;
broadly, they can be divided into unsupervised and supervised ones. This paper
focuses on unsupervised keyphrase extraction, i.e. the process where no
training set of document is needed to learn to estimate keyphrases – they are
estimated solely based on statistical/topological properties of a given
document. The unsupervised methods can be further divided to the ones which
construct a graph based on token co-occurrences and the ones which leverage
statistical properties of n-grams [26]. Recently, neural language model-based
keyphrase extraction was also proposed [12]. With the abundance of methods,
optimization of a single metric becomes less relevant – methods which maximize
e.g., F1@k are common. This paper aims to inform the reader that a realm of
highly relevant properties beyond simple retrieval performance can be
meaningful in practice, and should be the focus of any novel method proposed
(including the adaptation of an existing one presented in this paper). The
contributions of this paper are multifold:

Figure 1: Performance trade-off (time vs. performance) of keyphrase detection
methods averaged across fifteen data sets.

  1. 1.

We present RaKUn 2.0, a graph-based keyphrase extractor optimized for
retrieval-efficiency optimality when considering both retrieval capabilities
and performance.

  2. 2.

A polygon-based visualization suitable for studying and comparing multiple
criteria for multiple keyphrase detection algorithms.

  3. 3.

An extensive benchmark of RaKUn 2.0 against strong baselines (including e.g.,
the recently introduced KeyBERT).

  4. 4.

Friedman-Nemenyi-based analysis of average ranks of the algorithms (and their
similarity).

##  2 Selected related work

This section contains an overview of the existing keyphrase detection methods,
key underlying ideas and possible caveats of different paradigms. This paper
focuses exclusively on _unsupervised_ keyphrase extraction – the process of
transforming an input document $D$ in to a ranked collection of keyphrases,
i.e. $K=\\{(p,s)_{k}\\};s_{k+1}\leq s_{k}$, where $k$ represents the top $k$
hits (detected keyphrases), $p$ a given keyphrase and $s$ a given keyphrase’s
score. The first branch of approaches are based on text-to-graph
transformations, followed by subsequent processing of the obtained graphs.
Such methods are able to exploit multilevel structure of a document [6]
(MultiPartiteRank), hierarchical structure [29] (SingleRank). An example token
graph is shown in Figure 2.

Figure 2: An example token graph.

One of the first graph-based methods was TextRank [23], which demonstrated the
robustness of graph-based keyphrase detection (and was one of the first to do
so). More involved approaches, capable of incorporating topic-level
information were also proposed [7] (TopicalPageRank). One of the key issues
with graph-based representations is that of node denoising – the process of
identifying the relevant space of nodes which are commonly subject to ranking.
The graph-based methods are highly dependant on the graph construction
approach (based on co-occurrence, syntactic, semantic and similarity
information) and node ranking algorithm (e.g. degree, closeness, Page Rank,
selectivity, etc.) [3]. A detailed overview of graph-based methods for keyword
extraction and various node-ranking measures is provided in [3].

Alongside graph-based methods, statistical methods are also actively
developed. One of the most recent examples includes YAKE! [8], an approach
which considers large amounts of n-gram patterns and scores them so that they
represent relevant keyphrases. It operates by extracting statistical features
from single documents to select the most important keywords of a text.
Keyphrase detection was also considered as a task solvable by considering
neural language models [12]. An example of this family of models is
AttentionRank [10], which exploits the transformer-based neural language model
to extract relevant keywords. A more detailed overview of general keyword
detection methods is given in [17].

The discussed approaches seldom focus on metrics beyond retrieval capabilities
(e.g., Precision, Recall and F1). One of the purposes of this paper is a
comprehensive evaluation of the discussed algorithms with regards to multiple
criteria, including computation time and duplication rates (how frequent is a
token amongst the space of detected keyphrases).

##  3 Proposed algorithm

The proposed approach sources the core idea from the recent paper on meta
vertex-based keyphrase detection RaKUn [28]. The considered extension,
proposed in this paper is optimized specifically to push the boundary of the
retrieval-efficiency front between retrieval performance and retrieval time.
We begin with a general overview of the algorithm, followed by theoretical
analysis of its complexity (space and time). We refer to the proposed approach
as RaKUn 2.0. A high-level overview is shown as Algorithm 1.

Data: Input document $D$, merge factor $\tau$

tokens $\leftarrow$ tokenizeDocument($D$)

$\ddagger$ Tokenization.

tokens $\leftarrow$ mergeTokens($D$)

$\ddagger$ Merging.

$G\leftarrow$ documentGraph(tokens)

$\ddagger$ Weighted graph.

1 $f\leftarrow$ tokenFrequencies(tokens)

tokenRanks $\leftarrow$ personalizedPR($G$,$f$)

$\ddagger$ Ranking.

$K\leftarrow$ sort($N(G)$, tokenRanks)

$\ddagger$ Sorting.

return K;

Algorithm 1 RaKUn 2.0

The main steps include _tokenization_ , _token merging_ , _document graph
construction_ and _node ranking_. Instead of first constructing (larger)
graphs which are subject to node merging into meta vertices, RaKUn 2.0
conducts the merging step at the sequence level, making it more efficient.
This step was considered based on an observation that pre-merging tokens in
close proximity already offers sufficient results – by considering only tokens
close to one another, no specialized metric for string comparison (possibly
expensive) was needed, which substantially sped up the detection process. The
second idea which substantially sped up the process is related to _bi-gram
hashing_. It refers to constructing a mapping between each bi-gram and its
count in the document, enabling fast lookup of this information as follows.
for each subsequent token pair ($t_{i},t_{j}$) term counts are retrieved (they
are pre-computed during tokenization). We next compute a _merge threshold_
score as:

$\textrm{MScore}=\frac{|\\#t_{i}-\\#b_{ij}|+|\\#t_{j}-\\#b_{ij}|}{\\#t_{i}+\\#t_{j}}\;$

where $t_{i}$ and $t_{j}$ are two subsequent tokens, and $b_{ij}$ is the bi-
gram comprised of the two tokens. If MScore is lower than a user-specified
threshold (hyperparameter), the merged token is added as a new token to the
token space, and term counts of the two individual tokens are diminished by
MScore as $\\#t_{i}=\textrm{MScore}*\\#t_{i}$, i.e., multiplied with the
computed score. Values of MScore, lower than one, imply more emphasis of
multi-term keyphrases (individual terms are not as emphasized), and values
larger than one imply more individual token keyphrases. Hence, the MScore
serves as an intermediary step which _emphasizes_ specific tokens during the
ranking step.

The token graph $G$ is constructed from the modified list of tokens by
considering subsequent, lower-cased tokens as edges. The edge weights are
incremented every time a given bi-gram repeats – the transitions between
tokens which commonly co-occur are emphasized. The next step is _node
ranking_. Here, a real-valued score is assigned to each (pre-merged) token. We
consider personalized PageRank algorithm [25], where the personalization
vector is constructed based on term counts. This step results in real-valued
scores (between 0 and 1) for each token. The final set of scores is obtained
by computing an element-wise product between the PageRank scores and token
lengths. This step emphasizes longer keyphrases. We traverse the space of
scored tokens and remove case-level duplicates (e.g., ‘City’ and ‘city’).

The described algorithm for keyphrase detection was conceived with simplicity
in mind. This property also resonates with its computational complexity. Let
$T$ represent the number of tokens after the merge step (cardinality
difference is negligible with regards to the runtime). Both graph construction
and merging need one pass across the token sequence ($\mathcal{O}(|T|)$. The
computationally most expensive part is computation of personalized PageRank.
In theory, PageRank’s complexity is $\mathcal{O}(|T|+l)$, where $l$ is the
number of links in the constructed token graph. In practice, the obtained
graphs are very sparse – only selected bi-grams co-occur. The opposite case,
where dense, clique-like graphs would be produced would imply appearance of
tokens in highly diverse contexts, which is highly unlikely. The final step
requires sorting of tokens based their scores. This yields the final
complexity of $\mathcal{O}(|T|\log{|T|}\cdot l)$. Assuming very sparse graphs
(as observed during the experiments), the complexity remains linear with
regards to the number of tokens in the token set after the merge step.

##  4 Evaluation

We next discuss the evaluation procedures used to estimate the performance of
individual algorithms, followed by a discussion regarding their comparison. We
evaluate each algorithm with regards to three main aspects; retrieval
performance, keyword duplication rate and computation time. The retrieval
performance was measured as done in the previous work [8]. Precision@k is
defined as $\frac{|\textrm{Gold}\cap\textrm{k-predicted}|}{k}$. Recall@k is
defined as $\frac{|\textrm{Gold}\cap\textrm{k-predicted}|}{|\textrm{Gold}|}$.
Precision represents the number of keyphrases retrieved with regards to top
$k$ predicted ones, while recall represents the overall retrieval capability.
We also computed (macro) F1, which is the harmonic mean of precision and
recall, averaged across documents.

The second score is the _duplication rate_. We compute this score as follows;
for each detected keyphrase, we first split it to separate tokens (if multi-
token keyphrase is considered). For each part, we traverse the space of
detected tokens. If there is a match, we increment a duplicate counter,
otherwise, we increment the non_duplicate counter. The final score is computed
as $\frac{\\#duplicates}{\\#non\\_duplicates+1}$, and was observed to be in
the interval [0, 1]. The computation time was measured in seconds (for each
document).

Table 1: Summary of the considered data sets.

Dataset | #Docs | #KW | Mean KW tokens | Mean doc len
---|---|---|---|---
wiki20 [22] | 20 | 35.5 | 2.0 | 7728.0
fao30 [21] | 30 | 32.2 | 1.6 | 4710.3
theses100 [19] | 100 | 6.7 | 2.0 | 4813.9
citeulike180 [20] | 183 | 17.4 | 1.3 | 4517.9
Nguyen2007 [24] | 209 | 12.0 | 2.1 | 4425.6
SemEval2010 [15] | 243 | 15.6 | 2.2 | 7093.3
SemEval2017 [2] | 493 | 17.3 | 2.9 | 168.3
500N-KPCrowd-v1.1 [18] | 500 | 49.2 | 1.4 | 393.9
PubMed [1] | 500 | 14.2 | 1.9 | 3880.2
kdd [11] | 755 | 4.1 | 2.0 | 74.1
fao780 [21] | 779 | 8.0 | 1.6 | 4685.0
Schutz2008 [27] | 1231 | 45.3 | 1.5 | 2362.6
www [11] | 1330 | 4.8 | 1.9 | 82.0
Inspec [14] | 2000 | 14.1 | 2.2 | 112.5
Krapivin2009 [16] | 2304 | 5.3 | 2.1 | 7094.1

For visualization of retrieval-efficiency tradeoffs with regards to the
mentioned scores it makes sense to have uniform meaning of large and small
values. Hence, we introduce the following adapted scores which reflect this
idea. The retrieval capability already corresponds to e.g., F1 score, meaning
that higher values are preferred. We additionally normalize F1 scores to range
between 0 and 1 based on the worst-best performing algorithms (on average).
This way, an algorithm scored with 0 is the worst-performing one, while the
top performing is scored with 1 (see Figure 8). Similar adaptations were
considered for time performance (normalized inverse times) and duplication
rates (normalized inverse duplication rates). One of the main results of this
paper is a visualization which jointly considers all three aspects. The
considered collection of data sets is summarized in Table 1.

The considered baselines are discussed next. The graph-based baselines include
MultiPartiteRank [6], SingleRank [29], TextRank [23] and TopicalPageRank [7].
The statistical baseline considered was YAKE [8]. The language model-based
baseline is the recent KeyBERT [12]. For all approaches, we considered the
default hyperparameter configurations, as we were interested in out-of-the-box
performance. We computed, however, two variants of KeyBERT, one which emits
single tokens (KeyBERT-(1,1)) and one which permits two term tokens
(KeyBERT-(1,2)). Default configuration of KeyBERT variants performed worse
than term frequency-based extraction 111We considered unigrams. Inverse
document frequencies were not computed as they require the whole corpus,
making them not directly comparable to purely unsupervised methods., and
offered (1,1) adequate performance only when we set the ‘maxsum‘ and ‘mmr‘
flags to ‘true‘. The stopwords used were the same for all approaches (NLTK’s
default English stopwords [4]). Other algorithms’ implementations were based
on the PKE library [5].

##  5 Results

A summary of algorithm run times (relative to one another) is shown in Figure
3.

Figure 3: Pairwise time comparison of average algorithm run times
($\log_{10}(\frac{A}{B})$.

As expected, the simplest baseline (term frequency) is up to three orders of
magnitude faster than e.g., BERT-based model. The second approach that
performs substantially better, while remaining up to two orders of magnitude
faster is the proposed RaKUn 2.0. It is closely followed by SingleRank and
TopicalPageRank. The duplication levels are shown in Figure 4.

Figure 4: Duplication levels for different algorithms.

The duplication ablation indicates the highest duplication levels were
observed for YAKE, TopicalPageRank and TextRank. MultiPartiteRank and
SingleRank had notably lower duplication levels (KeyBERT-(1,1) as well the
term frequency (unigram) baseline. The proposed RaKUn 2.0 is at the lower end
of the approaches with regards to this score, albeit not being optimal.

We continue the discussion by presenting the retrieval performance. A
systematic investigation of algorithm performance is shown in Figure 5.

Figure 5: F1 score for different top $k$ keyphrases, averaged across all data
sets.

The results indicate that on average, MultiPartiteRank is the leading
algorithm in the low $k$ scenarios. RaKUn 2.0, however, performs very
similarly for up to ten keyphrases, which is one of the most common usecases
of such algorithms. A more detailed overview of the scores on the per-data set
level is given in Tables 1-5. The color codes represent top three performers
for each data set (gold=first, silver=second, bronze=third).

Table 2: F1@10 (gold=first, silver=second, bronze=third, per row)

Algorithm | KeyBERT-(1,1) | KeyBERT-(1,2) | MultiPartiteRank | RaKUn 2.0 | SingleRank | TextRank | TopicalPageRank | YAKE | TFreq
---|---|---|---|---|---|---|---|---|---
Dataset |  |  |  |  |  |  |  |  |
500N-KPCrowd-v1.1 | 0.012 | 0.012 | 0.171 | 0.138 | 0.164 | 0.057 | 0.094 | 0.127 | 0.106
Inspec | 0.0 | 0.0 | 0.22 | 0.143 | 0.207 | 0.126 | 0.24 | 0.195 | 0.041
Krapivin2009 | 0.051 | 0.057 | 0.109 | 0.097 | 0.094 | 0.007 | 0.02 | 0.118 | 0.011
Nguyen2007 | 0.099 | 0.058 | 0.168 | 0.141 | 0.152 | 0.025 | 0.053 | 0.188 | 0.035
PubMed | 0.1 | 0.021 | 0.087 | 0.083 | 0.072 | 0.002 | 0.004 | 0.087 | 0.036
Schutz2008 | 0.088 | 0.023 | 0.23 | 0.194 | 0.219 | 0.015 | 0.031 | 0.15 | 0.075
SemEval2010 | 0.071 | 0.053 | 0.152 | 0.139 | 0.133 | 0.01 | 0.023 | 0.155 | 0.023
SemEval2017 | 0.0 | 0.0 | 0.216 | 0.132 | 0.203 | 0.122 | 0.224 | 0.175 | 0.056
citeulike180 | 0.205 | 0.03 | 0.172 | 0.225 | 0.14 | 0.004 | 0.013 | 0.185 | 0.097
fao30 | 0.16 | 0.027 | 0.176 | 0.233 | 0.161 | 0.008 | 0.011 | 0.15 | 0.072
fao780 | 0.116 | 0.013 | 0.141 | 0.138 | 0.118 | 0.004 | 0.009 | 0.138 | 0.064
kdd | 0.0 | 0.001 | 0.107 | 0.144 | 0.094 | 0.058 | 0.109 | 0.144 | 0.056
theses100 | 0.099 | 0.017 | 0.149 | 0.103 | 0.128 | 0.004 | 0.006 | 0.093 | 0.042
wiki20 | 0.222 | 0.013 | 0.186 | 0.226 | 0.163 | 0.0 | 0.0 | 0.135 | 0.021
www | 0.0 | 0.001 | 0.11 | 0.113 | 0.099 | 0.065 | 0.109 | 0.129 | 0.062

Table 3: Precision@10 (gold=first, silver=second, bronze=third, per row)

Algorithm | KeyBERT-(1,1) | KeyBERT-(1,2) | MultiPartiteRank | RaKUn 2.0 | SingleRank | TextRank | TopicalPageRank | YAKE | TFreq
---|---|---|---|---|---|---|---|---|---
Dataset |  |  |  |  |  |  |  |  |
500N-KPCrowd-v1.1 | 0.046 | 0.037 | 0.38 | 0.323 | 0.36 | 0.129 | 0.173 | 0.262 | 0.192
Inspec | 0.0 | 0.0 | 0.174 | 0.112 | 0.165 | 0.101 | 0.189 | 0.152 | 0.032
Krapivin2009 | 0.037 | 0.04 | 0.079 | 0.069 | 0.068 | 0.005 | 0.014 | 0.084 | 0.008
Nguyen2007 | 0.09 | 0.05 | 0.151 | 0.124 | 0.138 | 0.022 | 0.048 | 0.166 | 0.032
PubMed | 0.065 | 0.013 | 0.057 | 0.055 | 0.047 | 0.001 | 0.003 | 0.057 | 0.023
Schutz2008 | 0.193 | 0.047 | 0.504 | 0.433 | 0.48 | 0.029 | 0.065 | 0.329 | 0.163
SemEval2010 | 0.075 | 0.055 | 0.159 | 0.146 | 0.14 | 0.011 | 0.024 | 0.162 | 0.023
SemEval2017 | 0.0 | 0.001 | 0.293 | 0.184 | 0.278 | 0.169 | 0.3 | 0.235 | 0.077
citeulike180 | 0.208 | 0.03 | 0.172 | 0.228 | 0.14 | 0.003 | 0.012 | 0.183 | 0.097
fao30 | 0.183 | 0.033 | 0.21 | 0.28 | 0.19 | 0.01 | 0.013 | 0.18 | 0.087
fao780 | 0.075 | 0.008 | 0.092 | 0.09 | 0.077 | 0.002 | 0.006 | 0.089 | 0.041
kdd | 0.0 | 0.001 | 0.064 | 0.087 | 0.056 | 0.036 | 0.065 | 0.085 | 0.034
theses100 | 0.064 | 0.011 | 0.098 | 0.068 | 0.084 | 0.002 | 0.004 | 0.06 | 0.027
wiki20 | 0.19 | 0.01 | 0.155 | 0.19 | 0.135 | 0.0 | 0.0 | 0.12 | 0.02
www | 0.0 | 0.001 | 0.066 | 0.068 | 0.06 | 0.04 | 0.065 | 0.076 | 0.037

Table 4: Recall@10 (gold=first, silver=second, bronze=third, per row)

Algorithm | KeyBERT-(1,1) | KeyBERT-(1,2) | MultiPartiteRank | RaKUn 2.0 | SingleRank | TextRank | TopicalPageRank | YAKE | TFreq
---|---|---|---|---|---|---|---|---|---
Dataset |  |  |  |  |  |  |  |  |
500N-KPCrowd-v1.1 | 0.007 | 0.007 | 0.144 | 0.119 | 0.139 | 0.041 | 0.087 | 0.129 | 0.113
Inspec | 0.0 | 0.0 | 0.356 | 0.233 | 0.331 | 0.194 | 0.388 | 0.326 | 0.07
Krapivin2009 | 0.097 | 0.119 | 0.212 | 0.187 | 0.182 | 0.014 | 0.041 | 0.236 | 0.021
Nguyen2007 | 0.135 | 0.087 | 0.23 | 0.216 | 0.205 | 0.036 | 0.078 | 0.279 | 0.05
PubMed | 0.27 | 0.065 | 0.223 | 0.209 | 0.181 | 0.009 | 0.013 | 0.239 | 0.096
Schutz2008 | 0.063 | 0.017 | 0.161 | 0.133 | 0.153 | 0.011 | 0.022 | 0.104 | 0.053
SemEval2010 | 0.071 | 0.054 | 0.152 | 0.139 | 0.133 | 0.01 | 0.024 | 0.156 | 0.023
SemEval2017 | 0.0 | 0.0 | 0.183 | 0.11 | 0.17 | 0.101 | 0.189 | 0.15 | 0.046
citeulike180 | 0.221 | 0.034 | 0.187 | 0.242 | 0.151 | 0.005 | 0.016 | 0.205 | 0.104
fao30 | 0.149 | 0.023 | 0.159 | 0.21 | 0.147 | 0.007 | 0.01 | 0.134 | 0.065
fao780 | 0.335 | 0.036 | 0.396 | 0.39 | 0.321 | 0.01 | 0.025 | 0.39 | 0.193
kdd | 0.0 | 0.003 | 0.384 | 0.514 | 0.346 | 0.188 | 0.398 | 0.562 | 0.194
theses100 | 0.266 | 0.045 | 0.389 | 0.262 | 0.326 | 0.019 | 0.02 | 0.254 | 0.116
wiki20 | 0.294 | 0.017 | 0.251 | 0.297 | 0.221 | 0.0 | 0.0 | 0.166 | 0.023
www | 0.0 | 0.004 | 0.393 | 0.412 | 0.352 | 0.221 | 0.394 | 0.502 | 0.237

Table 5: Retrieval time (s). (gold=first, silver=second, bronze=third, per
row)

Algorithm | KeyBERT-(1,1) | KeyBERT-(1,2) | MultiPartiteRank | RaKUn 2.0 | SingleRank | TextRank | TopicalPageRank | YAKE | TFreq
---|---|---|---|---|---|---|---|---|---
Dataset |  |  |  |  |  |  |  |  |
500N-KPCrowd-v1.1 | 0.422 | 0.763 | 0.477 | 0.009 | 0.454 | 0.417 | 1.552 | 0.699 | 0.0
Inspec | 0.202 | 0.337 | 0.399 | 0.006 | 0.4 | 0.394 | 1.554 | 0.74 | 0.0
Krapivin2009 | 1.561 | 11.282 | 6.144 | 0.08 | 4.332 | 1.111 | 2.852 | 1.423 | 0.007
Nguyen2007 | 1.333 | 6.318 | 3.528 | 0.05 | 2.45 | 0.87 | 2.505 | 1.304 | 0.005
PubMed | 1.237 | 4.865 | 2.823 | 0.046 | 1.945 | 0.766 | 2.312 | 1.249 | 0.004
Schutz2008 | 1.58 | 6.926 | 3.993 | 0.038 | 2.675 | 0.772 | 2.428 | 1.236 | 0.004
SemEval2010 | 1.675 | 12.479 | 6.135 | 0.076 | 4.213 | 1.117 | 2.707 | 1.378 | 0.008
SemEval2017 | 0.213 | 0.431 | 0.403 | 0.007 | 0.395 | 0.389 | 1.596 | 0.947 | 0.0
citeulike180 | 1.556 | 8.006 | 3.937 | 0.051 | 2.411 | 0.812 | 2.446 | 1.322 | 0.005
fao30 | 1.528 | 7.599 | 4.665 | 0.056 | 2.793 | 0.877 | 2.47 | 1.573 | 0.005
fao780 | 1.531 | 7.806 | 5.284 | 0.056 | 3.111 | 0.838 | 2.53 | 1.479 | 0.005
kdd | 0.153 | 0.268 | 0.394 | 0.006 | 0.394 | 0.383 | 1.371 | 0.549 | 0.0
theses100 | 1.52 | 7.069 | 4.05 | 0.053 | 2.603 | 0.811 | 2.293 | 1.644 | 0.004
wiki20 | 1.598 | 10.453 | 5.674 | 0.066 | 3.8 | 0.952 | 2.586 | 1.429 | 0.006
www | 0.152 | 0.269 | 0.39 | 0.006 | 0.396 | 0.396 | 1.283 | 0.552 | 0.0

We additionally conducted rank-based difference significance evaluation [9],
where the average algorithm ranks are compared across all data sets. If the
algorithms are linked with a red line, they perform very similarly ($p<0.05$).
The diagrams are shown as Figures 7 and 7.

Figure 6: Critical difference diagram - F1@15. RaKUn 2.0’s performance is
(statistically) comparable to the recent state-of-the-art approaches.

Figure 7: CD diagrams – time per document. Higher ranks indicate faster
compute time. RaKUn 2.0 is significantly faster when compared to other state-
of-the-art methods.

The tests indicate that the difference between the top-performing approaches
(MultiPartiteRank, YAKE and RaKUn 2.0) is insignificant. Similar observations
can be made based on tabular summaries. Overall, however, we can observe a
marginal dominance of RaKUn 2.0 w.r.t. precision. Similar retrieval
performance amplifies the purpose of this paper, which transcends the
retrieval-only evaluation and incorporates also other properties of either the
algorithms or the retrieved space.

In Figure 8, the selected approaches are compared across the three main
evaluation criteria – retrieval performance, duplication performance (inverse
of duplication rate) and time performance (inverse of normalized times across
all algorithms). Larger values are better for each criterion. It can be
observed that MultiPartiteRank outperforms the others at the front considering
duplication and retrieval performance, however, RaKUn 2.0 outperforms the
others when considering retrieval capabilities and computation time.

Figure 8: A visualization comparing best and worst-performing approaches with
regards to three different criteria relevant in practice. Note that the scores
are relative with regards to the considered methods’ performances.

###  5.1 Scaling to 14M documents

A direct way of testing the complexity bounds stated in the methods section
was to attempt and run RaKUn 2.0 directly on the collection of approximately
14 million biomedical articles – the MeDAL corpus
[30]222https://www.reddit.com/r/MachineLearning/comments/jx63fd/r_a_14m_articles_dataset_for_medical_nlp/.
The corpus was parsed into a list of documents and fed into the default
configuration of RaKUn 2.0. The computation took approximately forty seconds
(including text reading) on a virtual machine with 12 cores and 32GB of RAM.
The list of top ten keyphrases is shown as Table 6.

Table 6: 14M articles summarized as top ten keyphrases. Keyphrase | Score
---|---
presence | 0.02041868080426608
molecular weights | 0.01313742352650019
glutamine synthetase | 0.01081927396059080
growth hormone | 0.01081481738381907
arterial blood | 0.00973761662559790
investigated | 0.00926714499542069
rate constant | 0.00904369510973679
blood flow | 0.00899499866920862
molecular weight | 0.00865807865159297
sodium dodecyl | 0.00865611530561878

The top keyphrases correspond to rather general biological terms, which are
some of the main topics related to the considered documents. The results were
obtained by maintaining the merge_threshold hyperparameter set to one – single
term keyphrases can be obtained if this threshold is lowered. For example, if
set to 0.5, the top three keyphrases are ‘activity’, ‘concentration’ and
‘enzyme’.

##  6 Discussion and conclusions

In this paper we presented an approach to unsupervised keyphrase detection,
aimed specifically at pushing the limits of computation time and retrieval
performance. The main contributions of this paper are an algorithm for
keyphrase detection that performs substantially (significantly) faster than
current state-of-the-art methods, while maintaining the retrieval performance.
The algorithmic novelties introduced touch upon the transformation of token
sequences into graphs, and re-address the question of meta vertices by
constructing them at the sequence level, which is substantially faster.
Further, by exploiting personalized PageRank, global token information is
incorporated into keyphrase ranking alongside token lengths. By conducting an
extensive benchmark against established baselines, this paper presents an
evaluation which incorporates both retrieval capabilities, but further details
into computation time and duplication rates amongst the retrieved keyphrases.

Analysis of keyphrase detection algorithms with regards to multiple evaluation
criteria is becoming of higher relevance, as many low-latency applications
cannot afford expensive detection phase. To our knowledge, this paper is
similarly one of the first to evaluate the performance based on critical
difference diagrams, exactly assessing the significance of observed
differences (in time and retrieval performance).

Further work includes exploration of lower-level implementations of top-
performing approaches, alongside their parts that could be subject to
parallelism. A potentially interesting endeavor would also include background
knowledge (as graphs), possibly enabling detection of keywords beyond the ones
found in a given document, while remaining unsupervised.

##  7 Replicability

The RaKUN 2.0 algorithm is available as a simple-to-use Python library
available at https://github.com/SkBlaz/rakun2.

## Acknowledgements

The work was supported by the Slovenian Research Agency (ARRS) core research
programme Knowledge Technologies (P2-0103), and projects Computer-assisted
multilingual news discourse analysis with contextual embeddings (J6-2581) and
Quantitative and qualitative analysis of the unregulated corporate financial
reporting (J5-2554). The work was also supported by the Ministry of Culture of
Republic of Slovenia through project Development of Slovene in Digital
Environment (RSDO).

## References

  * [1] Aronson, A.R., Bodenreider, O., Chang, H.F., Humphrey, S.M., Mork, J.G., Nelson, S.J., Rindflesch, T.C., Wilbur, W.J.: The nlm indexing initiative. In: Proceedings of the AMIA Symposium. p. 17. American Medical Informatics Association (2000)
  * [2] Augenstein, I., Das, M., Riedel, S., Vikraman, L., McCallum, A.: SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). pp. 546–555. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/S17-2091, https://aclanthology.org/S17-2091
  * [3] Beliga, S., Meštrović, A., Martincic-Ipsic, S.: An overview of graph-based keyword extraction methods and approaches. Journal of Information and Organizational Sciences 39, 1–20 (07 2015)
  * [4] Bird, S., Klein, E., Loper, E.: Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.” (2009)
  * [5] Boudin, F.: pke: an open source python-based keyphrase extraction toolkit. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations. pp. 69–73. Osaka, Japan (December 2016), http://aclweb.org/anthology/C16-2015
  * [6] Boudin, F.: Unsupervised keyphrase extraction with multipartite graphs. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 667–672. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-2105, https://aclanthology.org/N18-2105
  * [7] Bougouin, A., Boudin, F., Daille, B.: TopicRank: Graph-based topic ranking for keyphrase extraction. In: Proceedings of the Sixth International Joint Conference on Natural Language Processing. pp. 543–551. Asian Federation of Natural Language Processing, Nagoya, Japan (2013), https://aclanthology.org/I13-1062
  * [8] Campos, R., Mangaravite, V., Pasquali, A., Jorge, A., Nunes, C., Jatowt, A.: Yake! keyword extraction from single documents using multiple local features. Information Sciences 509, 257–289 (2020). https://doi.org/https://doi.org/10.1016/j.ins.2019.09.013, https://www.sciencedirect.com/science/article/pii/S0020025519308588
  * [9] Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7(1), 1–30 (2006), http://jmlr.org/papers/v7/demsar06a.html
  * [10] Ding, H., Luo, X.: AttentionRank: Unsupervised keyphrase extraction using self and cross attentions. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 1919–1928. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.146, https://aclanthology.org/2021.emnlp-main.146
  * [11] Gollapalli, S.D., Caragea, C.: Extracting keyphrases from research papers using citation networks. In: Brodley, C.E., Stone, P. (eds.) Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, July 27 -31, 2014, Québec City, Québec, Canada. pp. 1629–1635. AAAI Press (2014), http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8662
  * [12] Grootendorst, M.: Keybert: Minimal keyword extraction with bert. (2020). https://doi.org/10.5281/zenodo.4461265, https://doi.org/10.5281/zenodo.4461265
  * [13] Hasan, K.S., Ng, V.: Automatic keyphrase extraction: A survey of the state of the art. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1262–1273. Association for Computational Linguistics, Baltimore, Maryland (2014). https://doi.org/10.3115/v1/P14-1119, https://aclanthology.org/P14-1119
  * [14] Hulth, A.: Improved automatic keyword extraction given more linguistic knowledge. In: Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing. pp. 216–223 (2003), https://aclanthology.org/W03-1028
  * [15] Kim, S.N., Medelyan, O., Kan, M.Y., Baldwin, T.: SemEval-2010 task 5 : Automatic keyphrase extraction from scientific articles. In: Proceedings of the 5th International Workshop on Semantic Evaluation. pp. 21–26. Association for Computational Linguistics, Uppsala, Sweden (2010), https://aclanthology.org/S10-1004
  * [16] Krapivin, M., Autaeu, A., Marchese, M.: Large dataset for keyphrases extraction (2009)
  * [17] Kumar, T., Mahrishi, M., Meena, G.: A comprehensive review of recent automatic speech summarization and keyword identification techniques. Artificial Intelligence in Industrial Applications pp. 111–126 (2022)
  * [18] Marujo, L., Viveiros, M., da Silva Neto, J.P.: Keyphrase cloud generation of broadcast news (2013)
  * [19] Medelyan, O.: Human-competitive automatic topic indexing. Ph.D. thesis, The University of Waikato (2009)
  * [20] Medelyan, O., Frank, E., Witten, I.H.: Human-competitive tagging using automatic keyphrase extraction. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. pp. 1318–1327. Association for Computational Linguistics, Singapore (2009), https://aclanthology.org/D09-1137
  * [21] Medelyan, O., Witten, I.H.: Domain-independent automatic keyphrase indexing with small training sets. ArXiv preprint abs/10.1002 (2010), https://arxiv.org/abs/10.1002
  * [22] Medelyan, O., Witten, I.H., Milne, D.: Topic indexing with wikipedia. In: Proceedings of the AAAI WikiAI workshop. vol. 1, pp. 19–24 (2008)
  * [23] Mihalcea, R., Tarau, P.: TextRank: Bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. pp. 404–411. Association for Computational Linguistics, Barcelona, Spain (2004), https://aclanthology.org/W04-3252
  * [24] Nguyen, T.D., Kan, M.Y.: Keyphrase extraction in scientific publications. In: Goh, D.H.L., Cao, T.H., Sølvberg, I.T., Rasmussen, E. (eds.) Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers. pp. 317–326. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)
  * [25] Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab (1999), http://ilpubs.stanford.edu:8090/422/, previous number = SIDL-WP-1999-0120
  * [26] Papagiannopoulou, E., Tsoumakas, G.: A review of keyphrase extraction. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10(2), e1339 (2020)
  * [27] Schutz, A.T., et al.: Keyphrase extraction from single documents in the open domain exploiting linguistic and statistical methods. M. App. Sc Thesis (2008)
  * [28] Škrlj, B., Repar, A., Pollak, S.: Rakun: Rank-based keyword extraction via unsupervised learning and meta vertex aggregation. In: Martín-Vide, C., Purver, M., Pollak, S. (eds.) Statistical Language and Speech Processing. pp. 311–323. Springer International Publishing, Cham (2019)
  * [29] Wan, X., Xiao, J.: CollabRank: Towards a collaborative approach to single-document keyphrase extraction. In: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008). pp. 969–976. Coling 2008 Organizing Committee, Manchester, UK (2008), https://aclanthology.org/C08-1122
  * [30] Wen, Z., Lu, X.H., Reddy, S.: MeDAL: Medical abbreviation disambiguation dataset for natural language understanding pretraining. In: Proceedings of the 3rd Clinical Natural Language Processing Workshop. pp. 130–135. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.clinicalnlp-1.15, https://aclanthology.org/2020.clinicalnlp-1.15