# Deep Keyphrase Completion

Yu Zhao1,, Jia Song1, Huali Feng1, Fuzhen Zhuang2,3, Qing Li1, Xiaojie Wang4,
Ji Liu5
1Fintech Innovation Center, Financial Intelligence and Financial Engineering
Key Laboratory,
Southwestern University of Finance and Economics, Chengdu, China
2Institute of Artificial Intelligence, Beihang University, Beijing 100191,
China
3Xiamen Data Intelligence Academy of ICT, CAS, China
4 School of Artifical Intelligence, Beijing University of Posts and
Telecommunications, China.
5AI platform and Seattle AI Lab, Kwai Inc  Corresponding author: Y. Zhao
(zhaoyu@swufe.edu.cn).

###### Abstract

Keyphrase provides accurate information of document content that is highly
compact, concise, full of meanings, and widely used for discourse
comprehension, organization, and text retrieval. Though previous studies have
made substantial efforts for automated keyphrase extraction and generation,
surprisingly, few studies have been made for keyphrase completion (KPC). KPC
aims to generate more keyphrases for document (e.g. scientific publication)
taking advantage of document content along with a very limited number of known
keyphrases, which can be applied to improve text indexing system, etc. In this
paper, we propose a novel KPC method with an encoder-decoder framework. We
name it deep keyphrase completion (DKPC) since it attempts to capture the deep
semantic meaning of the document content together with known keyphrases via a
deep learning framework. Specifically, the encoder and the decoder in DKPC
play different roles to make full use of the known keyphrases. The former
considers the keyphrase-guiding factors, which aggregates information of known
keyphrases into context. On the contrary, the latter considers the keyphrase-
inhibited factor to inhibit semantically repeated keyphrase generation.
Extensive experiments on benchmark datasets demonstrate the efficacy of our
proposed model.

##  1 Introduction

Figure 1: Example of keyphrase completion for text retrieval system.

Keyphrases are highly concise phrases that can provide compact and accurate
information of given document content. Since high-quality keyphrases can
facilitate the understanding, organizing, and accessing of document content,
they are wildly used for NLP tasks, such as document clustering Hammouda et
al. (2005); Chiu et al. (2020), text summarization Qazvinian et al. (2010);
Cano and Bojar (2019), and text retrieval Boudin et al. (2020). Due to the
broadly demand, many automatic keyphrases extraction and generation methods
are proposed over the years Meng et al. (2017).

In this paper, we concentrate on the problem of keyphrase completion (KPC),
which aims to predict a relative large fixed size of keyphrases for document
management. Consider the following situation where a publications retrieval
system is going to be established and improved, each academic paper is
demanded for N keyphrases to fill in while it only provides a very limited
number of known ones (i.e. much lower than N) and cannot meet the demand. In
this kind of cases, it needs and expects for KPC, i.e. to predict more
keyphrases for completion.

Most existing keyphrase prediction models, whether they be extracted-based
methods Mihalcea and Tarau (2004); Wan and Xiao (2008) or generated-enhanced
methods Meng et al. (2017), solely take advantage of the content of document.
In fact, as we mentioned before, a document like scientific publication
usually has already provided a few known keyphrases, with indicative topic
information for the paper. For example, in Figure 1 we show an example of
keyphrase completion for text retrieval system, in which the scientific
document have already existed two known keyphrases and be demanded at least
five ones to complete. A serious drawback of using existing models for KPC is
that they ignore the role of the known keyphrases and consequently fail to
consider the already summarized information about them.

In this paper, we propose a novel KPC method with an encoder-decoder
framework. We name it as Deep KeyPhrase Completion (DKPC) since it attempts to
capture the deep semantic meaning of the document content together with known
keyphrases via a deep learning framework.

Specifically, we believe that the known keyphrases have different roles in
keyphrase completion, including keyphrase-guiding factors and keyphrase-
inhibited factors. Firstly, we take the keyphrase-guiding factors into
consideration for the encoder, which aggregates information of known
keyphrases into context. Secondly, we take the keyphrase-inhibited factor into
consideration for the decoder, which inhibits semantically repeated keyphrase
generation. We conduct a comprehensive comparison on four benchmark datasets
against two unsupervised models and two supervised deep learning models as
baselines, and the results demonstrate the effectiveness of our proposed
model.

The contributions of this paper are threefold:

  * •

To the best of our knowledge, this is the first work that considers the
problem of keyphrase completion using the deep learning method.

  * •

We propose a novel KPC method for keyphrase completion using an encoder-
decoder framework.

  * •

We conduct a comprehensive comparison on four benchmark datasets against four
baselines, and the results demonstrate the effectiveness of our proposed
model.

##  2 Related Work

###  2.1 Keyphrase Extraction

Extractive models aim to extract present keyphrases from text documents, which
can be broadly categorized into two approaches: unsupervised and supervised
Kim et al. (2013); Hu and Wu (2006); Nguyen and Kan (2007); Augenstein et al.
(2017).

####  2.1.1 Unsupervised Approaches.

They are mostly based on statistical models like TF-IDF, clustering and graph-
based ranking. Generally, they directly treat keyphrase extraction as a
ranking task. For example, inspired by PageRank Brin and Page (1998), Mihalcea
and Tarau (2004) propose TextRank, which is the first attempt to utilize an
unsupervised graph-based approach to rank keyphrases on a word graph.
Following this framework, a substantial amount of enhanced unsupervised
keyphrase extraction models are proposed via exploring various word graph from
different perspectives. For example, SemanticRank Tsatsaronis et al. (2010)
take account of semantic relatedness on word graphs for extracting keyphrases.
CiteTextRank Gollapalli and Caragea (2014) and ExpandRank Wan and Xiao (2008)
use neighborhood knowledge, such as citation networks Gollapalli and Caragea
(2014) and topic-related documents Wan and Xiao (2008), to improve keyphrase
extraction. Moreover, MIKE Zhang et al. (2017) integrates multidimensional
heterogeneous information into a unified framework to improve keyphrase
extraction. Recently, YAKE! Campos et al. (2018) focuses on multi-lingual
keyword extraction from single documents. Key2vec Mahata et al. (2018)
leverages phrase embedding for keyword extraction in scientific articles.

####  2.1.2 Supervised approaches.

They typically treat keyphrase extraction problem as a binary classification
task (i.e., keyphrase,$\lnot$keyphrase) Hulth (2003), where a classification
model is trained on various features (i.e., tf.idf, candidate length, POS
tags, section information, frequency) of labelled keyphrases to determine
candidate phrase. Current supervised keyphrase extraction algorithms includes
Decision Trees Sterckx et al. (2016), Support Vector Machine (SVM) Lopez and
Romary (2010), Graph Attention Networks (GAT) Prasad and Kan (2019). For
example, Caragea et al. (2014) proposed to take citation network information
into consideration for keyphrase extraction. Prasad and Kan (2019)
incorporated word importance in graph attention networks for keyphrase
extraction. Recently, some methods treat keyphrase extraction as sequence
labeling task Gollapalli et al. (2017); Zhang et al. (2016); Chowdhury et al.
(2019); Al-Zaidy et al. (2019). For instance, Gollapalli et al. (2017)
extracted keyphrases from research papers utilizing token-based features
incorporating expert knowledge through sequence labeling. Sun et al. (2019)
proposed an end-to-end method to extract diverse keyphrase by combining
traditional-based ranking methods and RNN-based approaches. Al-Zaidy et al.
(2019) and propose a joint model by combining CRF and BiLSTM.

###  2.2 Keyphrase Generation

Different from keyphrase extraction, keyphrase generation models could
generate absent keyphrases by modeling such task as a sequence-to-sequence
(seq2seq) learning problem. Meng et al. (2017) first proposed CopyRNN, a
seq2seq framework with copy mechanism for keyphrase generation. Following this
framework, many enhanced variants of CopyRNN are proposed Ye and Wang (2018);
Chen et al. (2019); Cano and Bojar (2019); Chan et al. (2019); Yuan et al.
(2020); Chen et al. (2020); Liu et al. (2020). For example, Ye and Wang (2018)
proposed a semi-supervised methods by leveraging both labeled and unlabeled
data. Chen et al. (2019) proposed TG-Net taking title information into
consideration. Chan et al. (2019) utilized reinforcement learning with
adaptive rewards to generate more sufficient and accurate keyphrase. Chen et
al. (2020) designed a hierarchical decoding process and an exclusion mechanism
to avoid generating duplicated keyphrases. Moreover, Swaminathan et al.
(2020a, b) proposed a keyphrase generation approach using conditional
Generative Adversarial Networks. Ye et al. (2021) proposed a new training
paradigm ONE2SET without predefining an order to concatenate the keyphrases.
Ahmad et al. (2021) proposed a method for neural keyphrase generation with
layer-wise coverage attention.

##  3 Methodology

In this section, we introduce the details of our method. As we stated before,
the existing keyphrases play two kinds of roles in keyphrase completion, as
follows:

  * •

Guiding Factor Since keyphrases are a group of words that condense the core
information of a document, they always belong to the related topics. For
instance, the keyphrases of this paper, “keyphrase generation" and "attention
mechanism" are both related to Natural Language Processing. Motivated by this
observation, we can capture such relevance and let existing ground-truth
keyphrases to guide us complete more keyphrases. We call it the guiding
factor.

  * •

Inhibiting Factor It is reported that the keyphrases of more than 85%
documents in the largest keyphrase generation benchmark dataset have different
first word Chen et al. (2020). In the generation stage, if the first word of
the predicted keyphrase is the same as one of the existing keyphrases, it will
be eliminated. We call it inhibiting factor.

Therefore, motivated the basic ideas above, our proposed model contains two
modules: keyphrase-guided encoder and keyphrase-inhibited decoder. Before we
introduce our model, we first give a formal problem definition.

###  3.1 Problem Definition

The task of keyphrase generation is usually formulated as follows: given a
text dataset ${\mathcal{D}}{}=\\{{\bf x}^{i},{\bf y}^{i}\\}_{i=1}^{N}$ where
${\bf x}_{i}$ is $i$-th source text and ${\bf y}^{i}=\\{{\bf y}^{i,1},{\bf
y}^{i,2},...,{\bf y}^{i,M}\\}$ is a set of keyphrases of it. $N$ is the number
of documents, and $M$ is the number of keyphrases of the $i$-th document. Both
the ${\bf x}_{i}$ and the $j$-th keyphrase of it are sequences of words,
denoted as ${\bf x}^{i}=(^{i}_{1},x^{i}_{2},...,x^{i}_{L_{x^{i}}})$ and ${\bf
y}^{i,j}=(y_{1}^{i,j},y_{2}^{i,j},...,y_{L_{y^{i,j}}}^{i,j})$, where
$L_{x^{i}}$ and $L_{y^{i,j}}$ are the number of words of the i-th texts and
its j-th keyphrase respectively.

The goal of the model is to map from ${\bf x}$ to ${\bf y}$. Since our task is
complete more keyphrases based on document content and a limited number of
known keyphrases, we split the data into $({\bf x},{\bf k},{\bf y})$ where
$\bf x$ is source text, ${\bf k}$ is the given existing keyphrases of document
$x$ in advance, and ${\bf y}$ is the rest true keyphrases that we expect to
generate for document completion.

###  3.2 Keyphrase-Guided Encoder

In the keyphrase-guided encoder, we aim to utilize the known keyphrases to
guide more keyphrase generation. To this end, we take advantage of the
promising attention mechanism in the encoder, which can gather the information
of given keyphrases to the source text.

##### Source Text Representation.

For words in source text, we use a bi-directional GRU Cho et al. (2014) to
learn the contextual representation. The word embeddings are obtained from an
initial embedding lookup table. The encoder Bi-GRU reads the input sequence
$\bf x=(x_{1},x_{2},...,x_{L_{x}})$ forwardly and backwardly to converts them
into two sets of hidden representation by iterating with the following
equation along time $t$, respectively.

$\overrightarrow{{\bf h}^{x}_{t}}=f({\bf x}_{t},\overrightarrow{{\bf
h}^{x}_{t-1}})\ ,$ (1)

$\overleftarrow{{\bf h}^{x}_{t}}=f({\bf x}_{t},\overleftarrow{{\bf
h}^{x}_{t-1}})\ ,$ (2)

where $f$ is a non-linear function. $\overrightarrow{{\bf h}_{t}^{x}}$ is the
hidden state at time $t$ when input sequence is ordered from ${\bf x}_{1}$ to
${\bf x}_{L_{x}}$ and $\overleftarrow{{\bf h}_{t}^{x}}$ is the same when input
sequence is ordered from ${\bf x}_{L_{x}}$ to ${\bf x}_{1}$. $L_{x}$ is the
number of words of source text ${\bf x}$. The forward hidden state and
backward hidden state are concatenated as ${\bf
h}^{x}_{i}=[\overrightarrow{{\bf h}_{i}^{x}};\overleftarrow{{\bf h}_{i}^{x}}]$
to represent the $i$-th word of ${\bf x}$.

##### Keyphrase Representation.

Simple averaging is sufficient for keyphrase representation since keyphrase
typically consists of a small number of words (in most cases 1 or 2). On the
contrary, complicated models (like CNN or RNN) tend to overfit Xin et al.
(2018). Therefore, similar to Xin et al. (2018), if a given keyphrase contains
$n_{k}$ words, we use the average of embedding vectors of the $n_{k}$ words to
represent this keyphrase, as follows:

${\bf k}=\frac{{\bf w}_{1}+{\bf w}_{2}+...+{\bf w}_{n_{k}}}{n_{k}}\ ,$ (3)

where ${\bf k}$ is an aggregated semantic representation vector and $w_{i}$ is
the embedding vector of $i$-th word of this keyphrase from the embedding
lookup table. In the phrase of model training, these vectors will be
constantly updated to learn more appropriate semantic representations.

##### Attentional Hidden State.

To effectively exploit the information provided by given keyphrases, an
attention layer is used to calculate the relevance of given keyphrases and
source text, and then obtain an aggregated information vector. This formal
process is as follows:

$e_{ij}=({\bf h}_{i}^{x})^{\top}\bf{W_{1}}{{\bf k}_{j}}\ ,$ (4)

$\alpha_{i,j}=\frac{exp(e_{ij})}{\sum^{n_{K}}_{n=1}exp(e_{in})}\ ,$ (5)

${\bf c}_{i}=\sum^{n_{k}}_{j=1}\alpha_{i,j}{\bf k}_{j}\ ,$ (6)

where $\alpha_{ij}$ is the normalized attention score of the i-th word of
source text and the j-th given keyphrase. ${\bf c}$ is an aggregated
information vector which summarizes the information of all the given
keyphrases. It is calculated based on the similarity of two vectors. We use
the general mode as our alignment function Luong et al. (2015).

Finally, we concatenate the aggregated information vector of given keyphrases
${\bf c}$ and the source text representation ${\bf h}^{x}$ as encoder hidden
state ${\bf h}$ for the downstream decoding process.

${\bf h}=\ [{\bf c}\|{\bf h}^{x}\ ]$ (7)

Therefore, we can obtain $[{\bf h}_{1},{\bf h}_{2},...,{\bf h}_{L_{x}}]$
according to this formula at different time steps. With the proposed
keyphrase-guided encoder, the existing keyphrases can provide some useful
information related to the domain of the source text and then guide to
complete more ground-truth keyphrases.

###  3.3 Keyphrase-Inhibited Decoder

In the keyphrase-inhibited decoder, we aim to inhibit semantically repeated
keyphrase generation. To this end, we employ Bi-GRU with attention mechanism
and copy mechanism Gu et al. (2016) as the building blocks of the decoder. It
decompresses the source text into a context vector through an attention layer
and generates the target keyphrase word by word. The process of Decoder is
designed as follows:

${\bf s}_{t}={\text{GRU}}([{\bf e}_{t-1};\zeta_{t-1}],{\bf s}_{t-1})\ ,$ (8)

${\bf c}_{t}=attn({\bf s}_{t},[{\bf h}_{1},{\bf h}_{2},...,{\bf
h}_{L_{x}}],\bf{W_{2}})\ ,$ (9)

$\widetilde{{\bf h}_{t}}=tanh(\bf{W_{3}}[{\bf c}_{t};{\bf s}_{t}])\ ,$ (10)

where ${\bf e}_{t-1}$ is the embedding vector of $y_{t-1}$ and $\zeta_{t-1}$
is position representation of it by selective reading in copy mechanism. The
context vector ${\bf c}_{t}$ is the aggregated vector for ${\bf s}_{t}$ from
the encoder’s source text representation $[{\bf h}_{1},{\bf h}_{2},...,{\bf
h}_{L_{x}]}$ via attention mechanism.

Target hidden state ${\bf s}_{t}$ and the context vector ${\bf c}_{t}$ that
weighs the sum of the hidden state of the encoder are combined through a
simple concatenation layer to produce an attentional hidden sate. Later, the
attentional hidden state goes through a softmax layer and outputs the
probability distribution. Note that the probability of a word be produced is
composed of two parts: the probability of generating it and the probability of
coping it from source text. The process is as follows:

$\displaystyle p(y_{t}|y_{t-1},\mathbf{x},\mathbf{k})=$ $\displaystyle
p(y_{t},g|y_{t-1},\mathbf{x},\mathbf{k})+$ (11) $\displaystyle
p(y_{t},c|y_{t-1},\mathbf{x},\mathbf{k})\ ,$ (12)

where $p(y_{t},c|·)$ is the probability of $y_{t}$ being copied from source
text and $p(y_{t},g|·)$ is the probability of $y_{t}$ being generating from
the large vocab. Specifically, the calculation process is as follows:

$p(y_{t},g|·)=softmax(\mathbf{W_{4}}\widetilde{{\bf h}_{t}})\ ,$ (13)
$p(y_{t},c|·)=tanh(\mathbf{W_{5}}{\bf h}_{t}){\bf s}_{t}\ .$ (14)

Finally, normalize the sum of the two probabilities as the final probability
that $y_{t}$ is generated at the current step. If $y_{t}$ is the same as the
initial word of one of the given keyphrases, then it won’t be the output as
result.

###  3.4 Training

The widely used negative log likelihood loss is adopted to train our model:

${\mathcal{L}}{}=-\sum^{n}_{t=1}\log
p(y_{t}|\mathbf{y_{t-1}},\mathbf{x},\mathbf{k})\ ,$ (15)

where $\mathbf{y}_{t}$ is the t-th predicted word sequence. $n$ is the length
of target keyphrase $\mathbf{y}_{t}$, $y_{t}$ is the t-th word of which.

##  4 Experiments

###  4.1 Experimental Settings

Table 1: The performance of expanding keyphrases of various models on four benchmark datasets. Methods | Inspec | Krapivin | SemEval | KP20k
---|---|---|---|---
F1@5 | F1@10 | F1@5 | F1@10 | F1@5 | F1@10 | F1@5 | F1@10
TF-IDF | 0.107 | 0.151 | 0.068 | 0.067 | 0.061 | 0.073 | 0.108 | 0.101
TextRank | 0.172 | 0.190 | 0.076 | 0.084 | 0.028 | 0.039 | 0.065 | 0.065
RNN | 0.084 | 0.082 | 0.068 | 0.060 | 0.077 | 0.084 | 0.103 | 0.087
CopyRNN | 0.179 | 0.194 | 0.127 | 0.106 | 0.106 | 0.128 | 0.161 | 0.142
DKPC (Ours) | 0.212 | 0.218 | 0.161 | 0.138 | 0.120 | 0.141 | 0.181 | 0.154

####  4.1.1 Datasets

We utilize the public benchmark datasets for the experiments, includes: KP20k
Meng et al. (2017), Inspec Hulth (2003), Krapivin Krapivin et al. (2009),
SemEval 2010 Kim et al. (2010).

  * •

KP20k Meng et al. (2017) is the largest dataset on Keyphrase Generation. KP20k
contains 567,830 scientific articles, where 20,000 articles for training and
20,000 articles for testing.

  * •

Inspec Hulth (2003) is composed of 2000 abstracts of scientific articles
totally. The testing part which is composed of 500 abstracts is used to test
our model.

  * •

Krapivin Krapivin et al. (2009) is not split to testing part and training part
by its original authors. We thus follow the set in Meng et al. (2017) that
uses the first 400 papers in alphabetical order as the testing part in total
2,304 papers.

  * •

SemEval 2010 Mahata et al. (2018) consists of 288 full length ACM articles.
The testing part has 100 articles.

Note that our model is only trained once using the training of KP20k, and
evaluated on the testing part of all other datasets, which is a normal
strategy like in CopyRNN Meng et al. (2017), TG-Net Chen et al. (2019), and
ParaNet Zhao and Zhang (2019).

####  4.1.2 Baselines

To demonstrate the effectiveness of our proposed model for keyphrase
extraction, we compare it with the following baselines, including two
unsupervised graph-based methods and two state-of-the-art (SOTA) deep learning
models.

##### Unsupervised graph-based extraction methods:

  * •

TF-IDF: a widely used statistical weighting methods.

  * •

TextRank Mihalcea and Tarau (2004): a PageRank based unsupervised algorithm
for keyphrase extraction.

##### Deep learning-based generation methods:

  * •

RNN Meng et al. (2017): a basic model for sequence data, without copy
mechanism compared to CopyRNN.

  * •

CopyRNN Meng et al. (2017): a state-of-the-art model which is the first
applies the encoder-decoder framework to keyphrase generation.

####  4.1.3 Evaluation Protocol

We follow the common practice and evaluate the performance of our model in
previous work Meng et al. (2017). Similar to them, we utilize F-measure (F1)@N
as an evaluation metric based on the macro-averaged Precision, Recall. $N$
indicates the number we need to complete, which is set among {5, 10 or 50}.
The precision is computed by the number of keyphrases generated correctly over
the number of all generated keyphrases. The recall is calculated by the number
of keyphrases generated correctly over the number of keyphrases that are the
targets of the sample. For each method, we give the F-measure at top 5 and top
10 predictions on four real-world benchmark datasets.

####  4.1.4 Implementation Details

During data preprocessing, we tokenize, lowercase, and stemming the text.
After that, we use the symbol $<digit>$ to replace each digit. Since the task
is to use some pre-designated keyphrases and document content to complete more
useful keyphrases, we need to take out a few keyphrases before training. $M$
denotes the number of original targets of a data record and $N$ denotes the
number of pre-designated keyphrases we need to take out at random. We design
the processing rules as follows:

$N=\begin{cases}delete\ this\ sample,&if\ M=0\\\ 1,&if\ M\in\\{2,3\\}\\\
2,&if\ M\in\\{4,5\\}\\\ 3,&if\ M\in[6,20]\\\ 5,&if\ M\textgreater 20\\\
\end{cases}$ (16)

The vocabulary includes 50,000 words that appear most frequently in the
datasets. We set the embedding dimension to 200 and the hidden size to 100. We
use teacher forcing to help training model. One batch contains 128 data
samples. Our model is optimized by Adam Kingma and Ba (2014) with an initial
learning rate 0.001. The learning rate will update every 10000 batches
according to StepLR111A method for interval adjustment of learning rate in
PyTorch. . When F1@10 on the validation set doesn’t rise for 100 consecutive
steps, we stop training early. During testing phrase, we use beam search to
select effective keyphrases and beam size is set to 50.

Table 2: The performance of our proposed model using different number of known keyphrases. Methods | 10% | 20% | 30% | 40% | 50%
---|---|---|---|---|---
F1@5 | F1@10 | F1@5 | F1@10 | F1@5 | F1@10 | F1@5 | F1@10 | F1@5 | F1@10
Inspec | 0.182 | 0.231 | 0.212 | 0.238 | 0.206 | 0.228 | 0.190 | 0.202 | 0.183 | 0.187
Krapivin | 0.118 | 0.200 | 0.179 | 0.181 | 0.172 | 0.167 | 0.165 | 0.148 | 0.147 | 0.122
SemEval | 0.124 | 0.157 | 0.122 | 0.142 | 0.113 | 0.134 | 0.098 | 0.127 | 0.103 | 0.122
KP20k | 0.243 | 0.348 | 0.210 | 0.207 | 0.205 | 0.191 | 0.187 | 0.166 | 0.164 | 0.142
Table 3: The performance of CopyRNN and our proposed model in the same case. Methods | F1@5 | F1@10 | P@5 | P@10 | R@5 | R@10 | Similarity
---|---|---|---|---|---|---|---
CopyRNN | 0.400 | 0.500 | 0.600 | 0.500 | 0.300 | 0.50 | 0.387
DKPC | 0.533 | 0.800 | 0.800 | 0.800 | 0.400 | 0.800 | 0.473

###  4.2 Results and Analysis

Table 1 shows the performance of our method against four baselines, from which
we observe that our proposed model performs better than all baselines for
keyphrase completion in terms of both F1@5 and F1@10 metrics on four datasets.
The best scores are highlighted in bold. It confirms the capability of our
method in modeling the deep semantic meaning of document content and known
keyphrases for keyphrases completion.

##### Analysis.

(1) To make a fair comparison, we take out a few keyphrases in advance as
known keyphrases according to our designed rules mentioned before, which is
applied to all testing datasets. In this case, fewer predictable keyphrases
will aggravate the keyphrase prediction challenges for all methods, which
leads to lower scores222The absolute scores in Table 1 are thus lower than
their performance in original papers..

(2) As mentioned before, the key motivation behind our work is that we are
interested in the proposed model’s capability for completing keyphrases based
on document content and limited known keyphrases. It is worth noting that such
a completion task is a challenging problem. To the best of our knowledge, no
existing pertinent methods are proposed specifically to handle this new task.
The unsupervised keyphrase extraction models, i.e. TF-IDF and TextRank, and
the keyphrase generation models, RNN and CopyRNN, inevitably fail to take the
known keyphrases into consideration. We can observe from Table 1 that all
baseline perform worse than ours.

(3) In KPC task, the known keyphrases can play an important role. We believe
they are always highly semantically related to other unknown keyphrases that
needed to be completed. Our method can capture such relation and utilize known
semantic information to select the most conducive sentences to the keyphrases
completion from the source document with an attention mechanism. In addition,
the given keyphrases are also aggregated into the context representation which
more fully exploits the information already known. On the four datasets, our
method achieves a cumulative 11.3% improvement at F1@5 and F1@10.

###  4.3 Keyphrase Completion Analysis

Table 2 shows the performance of our proposed model using different numbers of
known keyphrases. We randomly select 10 percent, 20 percent, 30 percent, 40
percent, and 50 percent of target keyphrases as our known keyphrases in
testing data. In each case, we calculate the keyphrase completion performance
on all datasets. Note that when we extract known keyphrases, we round down the
number of original targets. Therefore, the samples with less than 10 original
targets will not be considered in the case when 10 percent keyphrases are
extracted. Similarly, samples with less than 2 original targets will not be
taken into account in the case when 50 percent keyphrases are extracted. All
the samples that do not meet the minimum keyphrase requirement are deleted. We
can observe from Table 2 that the more given keyphrases will not always lead
to better model performance. The usage of 10 percent or 20 percent keyphrases
for completion broadly performs best. In KPC task, the appropriate number of
known keyphrases is important for model performance. Using too many known
keyphrases is not always suitable for predicting the rest of the keyphrase.
Because it may generate semantic repeated keyphrases based on the existing
keyphrases, therefore, these keyphrases are not able to comprehensively
summarize the article.

Figure 2: A keyphrase completion example of our proposed DKPC compare to
previous method CopyRNN.

###  4.4 Case Study

Figure 2 and Table 3 show a case study of keyphrase completion by our method
against CopyRNN. we can observe from Figure 2 that our model not only
completes keyphrases accurately but also completes more targets with less
computational resources. Table 3 shows the performance of the model on
Precision, Recall and F1-measure, from which we observe that DKPC is
significantly better than CopyRNN. Specifically, we use Bert Devlin et al.
(2019) to obtain the embedding of each keyphrase, and then calculate
similarity by non-repeating pairs between keyphrases. The sum of similarity
divided by the number of keyphrase pairs is similarity. Due to the guidance of
known keyphrases, our proposed method can search for accurate keyphrases from
a smaller semantic space than CopyRNN’s. That’s why our model can find more
real keyphrases in less time. The keyphrases inhibition factor is also
helpful. For example, in contrast to CopyRNN, our model never includes the
known keyphrase “practical” as a target to complete, or any keyphrase
beginning with the first word of any known keyphrase.

##  5 Conclusion and Future Work

In this paper, we concentrate on the new keyphrase completion task. To the
best of our knowledge, this is the first work that considers the problem of
keyphrase completion. To solve this problem, we propose a novel KPC method
that takes an encoder-decoder framework. The encoder and the decoder in our
method make full use of the guidance factor and inhibition factor of known
keyphrases, respectively. The comprehensive experimental results demonstrate
the effectiveness of our proposed model in the task of keyphrase completion.
Some interesting future work may include: (1) Our model only explore post-
processing for inhibition of known keyphrases. One interesting direction is to
design some specific layers on the deep learning model structure to make the
inhibition of known keyphrases work better. (2) We use the most classical
model, RNN based encoder-decoder framework with attention mechanism and copy
mechanism. In the future, we can investigate the performance of other models
such as the pre-training model. Some pre-training models trained on the larger
dataset may be able to capture the semantic similarity between known
keyphrases and keyphrases needed completed better.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China
under Grant No. 61906159, U1836206, 61773361, 62072379, and in part supported
by the Fundamental Research Funds for the Central Universities under Grant No.
kjcx20210102.

## References

  * Ahmad et al. (2021) Wasi Uddin Ahmad, Xiao Bai, Soomin Lee, and Kai-Wei Chang. 2021.  Select, extract and generate: Neural keyphrase generation with layer-wise coverage attention.  In _Proceedings of ACL_ , pages 1389–1404.
  * Al-Zaidy et al. (2019) Rabah A. Al-Zaidy, Cornelia Caragea, and C. Lee Giles. 2019.  Bi-LSTM-CRF sequence labeling for keyphrase extraction from scholarly documents.  In _Proceedings of WWW_ , pages 2551–2557.
  * Augenstein et al. (2017) Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017.  Semeval 2017 task 10: Scienceie - extracting keyphrases and relations from scientific publications.  In _Proceedings of Workshop on Semantic Evaluation, ACL_ , pages 546–555.
  * Boudin et al. (2020) Florian Boudin, Ygor Gallina, and Akiko Aizawa. 2020.  Keyphrase generation for scientific document retrieval.  In _Proceedings of ACL_ , pages 1118–1126.
  * Brin and Page (1998) Sergey Brin and Lawrence Page. 1998.  The anatomy of a large-scale hypertextual web search engine.  _Computer Networks and ISDN Systems_ , 30:1–7.
  * Campos et al. (2018) Ricardo Campos, Vitor Mangaravite, Arian Pasquali, Alipio Mario Jorge, Celia Nunes, and Adam Jatowt. 2018.  YAKE! collection-independent automatic keyword extractor.  In _Proceedings of ECIR_ , pages 806–810.
  * Cano and Bojar (2019) Erion Cano and Ondrej Bojar. 2019.  Keyphrase generation: A text summarization struggle.  In _Proceedings of NAACL_ , pages 666–672.
  * Caragea et al. (2014) Cornelia Caragea, Florin Adrian Bulgarov, Andreea Godea, and Sujatha Das Gollapalli. 2014.  Citation enhanced keyphrase extraction from research papers: A supervised approach.  In _Proceedings of EMNLP_ , pages 1435–1446.
  * Chan et al. (2019) Houpong Chan, Wang Chen, Lu Wang, and Irwin King. 2019.  Neural keyphrase generation via reinforcement learning with adaptive rewards.  In _Proceedings of ACL_ , pages 2163–2174.
  * Chen et al. (2020) Wang Chen, Houpong Chan, Piji Li, and Irwin King. 2020.  Exclusive hierarchical decoding for deep keyphrase generation.  In _Proceedings of ACL_ , pages 1095–1105.
  * Chen et al. (2019) Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R. Lyu. 2019.  Title-guided encoding for keyphrase generation.  In _Proceedings of AAAI_ , pages 6268–6275.
  * Chiu et al. (2020) Billy Chiu, Sunil Kumar Sahu, Derek Thomas, Neha Sengupta, and Mohammady Mahdy. 2020\.  Autoencoding keyword correlation graph for document clustering.  In _Proceedings of ACL_ , pages 3974–3981.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Chowdhury et al. (2019) Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2019.  Keyphrase extraction from disaster-related tweets.  In _Proceedings of WWW_ , pages 1555–1566.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _arXiv preprint arXiv:1810.04805_.
  * Gollapalli and Caragea (2014) Sujatha Das Gollapalli and Cornelia Caragea. 2014.  Extracting keyphrases from research papers using citation networks.  In _Proceedings of AAAI_ , pages 1629–1635.
  * Gollapalli et al. (2017) Sujatha Das Gollapalli, Xiaoli Li, and Peng Yang. 2017.  Incorporating expert knowledge into keyphrase extraction.  In _Proceedings of AAAI_ , pages 3180–3187.
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016.  Incorporating copying mechanism in sequence-to-sequence learning.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1631–1640, Berlin, Germany. Association for Computational Linguistics.
  * Hammouda et al. (2005) Khaled M Hammouda, Diego N Matute, and Mohamed S Kamel. 2005.  Corephrase: Keyphrase extraction for document clustering.  In _International workshop on machine learning and data mining in pattern recognition_ , pages 265–274.
  * Hu and Wu (2006) Xinghua Hu and Bin Wu. 2006.  Automatic keyword extraction using linguistic features.  In _Proceedings of ICDM Workshops_ , pages 19–23.
  * Hulth (2003) Anette Hulth. 2003.  Improved automatic keyword extraction given more linguistic knowledge.  In _Proceedings of EMNLP_ , pages 216–223.
  * Kim et al. (2010) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010.  SemEval-2010 Task 5: Automatic keyphrase extraction from scientific articles.  In _Proceedings of Workshop on Semantic Evaluation, ACL_ , pages 21–26.
  * Kim et al. (2013) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2013.  Automatic keyphrase extraction from scientific articles.  _Language Resources and Evaluation_ , 47(3):723–742.
  * Kingma and Ba (2014) D. Kingma and J. Ba. 2014.  Adam: A method for stochastic optimization.  _Computer Science_.
  * Krapivin et al. (2009) M. Krapivin, A. Autaeu, and M. Marchese. 2009.  Large dataset for keyphrases extraction.
  * Liu et al. (2020) Rui Liu, Zheng Lin, and Weiping Wang. 2020.  Keyphrase prediction with pre-trained language model.  In _arXiv preprint arXiv:2004.10462_.
  * Lopez and Romary (2010) Patrice Lopez and Laurent Romary. 2010.  Humb: Automatic key term extraction from scientic articles in grobid.  In _Proceedings of Workshop on Semantic Evaluation, ACL_ , pages 248–251.
  * Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics.
  * Mahata et al. (2018) Debanjan Mahata, John Kuriakose, Rajiv Ratn Shah, and Roger Zimmermann. 2018.  Key2vec: Automatic ranked keyphrase extraction from scientific articles using phrase embeddings.  In _Proceedings of NAACL_ , pages 634–6392018.
  * Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017.  Deep keyphrase generation.  In _Proceedings of ACL_ , pages 582–592.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedding of EMNLP_ , pages 404–411.
  * Nguyen and Kan (2007) Thuy Dung Nguyen and Min-Yen Kan. 2007.  Keyphrase extraction in scientific publications.  In _Asian Digital Libraries_ , pages 317–326.
  * Prasad and Kan (2019) Animesh Prasad and Min-Yen Kan. 2019.  Glocal: Incorporating global information in local convolution for keyphrase extraction.  In _Proceedings of NAACL_ , pages 1837–1846.
  * Qazvinian et al. (2010) Vahed Qazvinian, Dragomir R. Radev, and Arzucan Ozgur. 2010.  Citation summarization through keyphrase extraction.  In _Proceedings of COLING_ , pages 895–903.
  * Sterckx et al. (2016) Lucas Sterckx, Cornelia Caragea, Thomas Demeester, and Chris Develder. 2016.  Supervised keyphrase extraction as positive unlabeled learning.  In _Proceedings of EMNLP_ , pages 1924–1929.
  * Sun et al. (2019) Zhiqing Sun, Jian Tang, Pan Du, Zhi-Hong Deng, and Jian-Yun Nie. 2019.  DivGraphPointer: A graph pointer network for extracting diverse keyphrases.  In _Proceedings of SIGIR_ , pages 755–764.
  * Swaminathan et al. (2020a) Avinash Swaminathan, Raj Kuwar Gupta, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, and Rajiv Ratn Shah. 2020a.  Keyphrase generation for scientific articles using gans.  In _Proceedings of AAAI_ , pages 13931–13932.
  * Swaminathan et al. (2020b) Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, and Amanda Stent. 2020b.  A preliminary exploration of GANs for keyphrase generation.  In _Proceedings of EMNLP_ , pages 8021–8030.
  * Tsatsaronis et al. (2010) George Tsatsaronis, Iraklis Varlamis, , and Kjetil Norvag. 2010.  SemanticRank: Ranking keywords and sentences using semantic graphs.  In _Proceedings of COLING_ , pages 1074–1082.
  * Wan and Xiao (2008) Xiaojun Wan and Jianguo Xiao. 2008.  Single document keyphrase extraction using neighborhood knowledge.  In _Proceedings of AAAI_ , pages 855–860.
  * Xin et al. (2018) Ji Xin, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2018.  Improving neural fine-grained entity typing with knowledge attention.  In _Proceedings of AAAI_ , pages 5997–6004.
  * Ye and Wang (2018) Hai Ye and Lu Wang. 2018.  Semi-supervised learning for neural keyphrase generation.  In _Proceedings of EMNLP_ , pages 4142–4153.
  * Ye et al. (2021) Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021.  One2set: Generating diverse keyphrases as a set.  In _Proceedings of ACL_ , pages 4598–4608.
  * Yuan et al. (2020) Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2020.  One size does not fit all: Generating and evaluating variable number of keyphrases.  In _Proceedings of ACL_ , pages 7961–7975.
  * Zhang et al. (2016) Qi Zhang, Yang Wang, Yeyun Gong, and Xuanjing Huang. 2016.  Keyphrase extraction using deep recurrent neural networks on twitter.  In _Proceedings of EMNLP_ , pages 836–845.
  * Zhang et al. (2017) Yuxiang Zhang, Yaocheng Chang, Xiaoqing Liu, Sujatha Das Gollapalli, Xiaoli Li, and Chunjing Xiao. 2017.  Mike: keyphrase extraction by integrating multidimensional information.  In _Proceedings of CIKM_ , pages 1349–1358.
  * Zhao and Zhang (2019) Jing Zhao and Yuxiang Zhang. 2019.  Incorporating linguistic constraints into keyphrase generation.  In _Proceedings of ACL_ , pages 5224–5233.