g^{T}_{t}(\boldsymbol{x}_{t}-\boldsymbol{y}_{t+1})\leq-\frac{\rho}{2}\left\lVert\boldsymbol{x}_{t}-\boldsymbol{y}_{t+1}\right\rVert^{2}+\eta\left\lVert\boldsymbol{x}_{t}-\boldsymbol{y}_{t+1}\right\rVert
L\leq\frac{\eta^{2}L^{2}}{2\rho}$

The above inequalities are obtained using the strong convexity of $\Phi$ and
the update rule, Cauchy-Schwarz inequality, and the inequality $ax-
bx^{2}\leq\max_{x}ax-bx^{2}=a^{2}/4b$ as in the last step in the proof of [12,
Theorem 4.2], respectively. We have

(49)
$\displaystyle\textstyle\left\lVert\boldsymbol{x}_{t+1}-\boldsymbol{x}_{t}\right\rVert$
$\displaystyle\textstyle\leq\sqrt{\frac{2}{\rho}D_{\Phi}(\boldsymbol{x}_{t},\boldsymbol{x}_{t+1})}\leq\sqrt{\frac{2}{\rho}D_{\Phi}(\boldsymbol{x}_{t},\boldsymbol{y}_{t+1})}\stackrel{{\scriptstyle\eqref{eq:bounding_bregman_step}}}{{\leq}}\sqrt{2\eta^{2}\frac{L^{2}}{2\rho^{2}}}\leq\frac{L\eta}{\rho}.$

The first inequality is obtained using the strong convexity of $\Phi$, and the
second using the generalized Pythagorean inequality[14, Lemma 11.3]. We
combine Eq. (49) and (47) to obtain
$\mathbb{E}\left[\mathrm{UC}_{\boldsymbol{r}_{t}}(\boldsymbol{z}_{t},\boldsymbol{z}_{t+1})\right]\leq\gamma_{1}\gamma_{2}\left\lVert\boldsymbol{x}_{t+1}-\boldsymbol{x}_{t}\right\rVert\leq\gamma_{1}\gamma_{2}\frac{L\eta}{\rho}$.
The total update cost is
$\sum^{T-1}_{t=1}\mathbb{E}\left[\mathrm{UC}_{\boldsymbol{r}_{t}}(\boldsymbol{z}_{t},\boldsymbol{z}_{t+1})\right]\leq\gamma_{1}\gamma_{2}\frac{L\eta}{\rho}T$.
When OMD has a fixed learning rate
$\eta=\Theta\left(\frac{1}{\sqrt{T}}\right)$, we obtain
$\sum^{T-1}_{t=1}\mathbb{E}\left[\mathrm{UC}_{\boldsymbol{r}_{t}}(\boldsymbol{z}_{t},\boldsymbol{z}_{t+1})\right]=\operatorname{\mathcal{O}}\left(\sqrt{T}\right)$.
The expected service cost is
$\mathbb{E}\left[\sum^{T}_{t=1}f_{t}(\boldsymbol{z}_{t})\right]=\sum^{T}_{t=1}f_{t}(\boldsymbol{x}_{t})=\operatorname{\mathcal{O}}\left(T\right)$;
the first equality is obtained from the linearity of the expectation operator
and the function $f_{\boldsymbol{r}_{t}}$, and the second equality is obtained
using the bound in Eq. (30) with $\eta=\Theta\left(\frac{1}{\sqrt{T}}\right)$.

∎

###  B.5. Proof of Theorem 4

Lemma 2 and Lemma 4 guarantee that Algorithm 3 used with an OMD algorithm
satisfies the hypothesis of Theorem 1 and, hence, provides sublinear extended
regret.

###### Lemma 0.

The random integral cache state $\boldsymbol{z}$ obtained by calling Algorithm
3 with fractional cache configuration input $\boldsymbol{x}\in\mathcal{X}$
satisfies $\boldsymbol{z}\in\mathcal{Z}$ and
$\mathbb{E}_{\xi}[\boldsymbol{z}]=\boldsymbol{x}$.

###### Proof.

We employ the shorthand notation $m_{i}=\sum^{i}_{j=1}x_{j}$ and
$m^{-}_{i}=\lfloor m_{i}\rfloor$. The choice of $\xi$ (see Fig. 10) defines
$k$ different thresholds $\xi,\xi+1,\dots,\xi+k-1$. For each threshold, we
select the first item, whose accumulated mass exceeds the threshold. As
$m_{N}=k$, we are guaranteed to exceeds all $k$ thresholds, and as $x_{i}\leq
1$, we are guaranteed to select one item for each threshold. Therefore
$\boldsymbol{z}$ belongs to $\mathcal{Z}$. From Algorithm 3 for any
$i\in\mathcal{N}$ we have
$\mathbb{P}(z_{i}=1)=\mathbb{P}(\mathcal{I}_{i}\setminus\mathcal{I}_{i-1}=\\{i\\})=\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\right)$.

figurec

Figure 9. The support and distribution of the random variable
$|\mathcal{I}_{i-1}|$ for $m_{i-1}\geq m_{i}^{-}$ (left) and
$m_{i-1}<m_{i}^{-}$ (right). The blue shaded area represents the probability
that $|\mathcal{I}_{i-1}|$ takes the value indicated below the corresponding
image. This figure can be viewed as a subset of the rows in Fig. 10.

figurec

Figure 10. The random integral cache configuration obtained by calling Online
Rounding given the fractional cache state $\boldsymbol{x}$ and $\xi$ (left).
When $\xi$ is kept fixed, the probability over the initial choice of $\xi$ the
random integral cache configuration rounded from a new fractional state
$\boldsymbol{x}^{\prime}=\boldsymbol{x}-\delta\boldsymbol{e}_{3}+\delta\boldsymbol{e}_{7}$
is different is illustrated by the dashed areas (right).

See Fig. 9 (left). If $m_{i-1}\geq m^{-}_{i}$, then
$|\mathcal{I}_{i-1}|\in\\{m^{-}_{i},m^{-}_{i}+1\\}$ and

$\displaystyle\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\right)$
$\displaystyle=\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\,\,\big{|}\,\,|\mathcal{I}_{i-1}|=m^{-}_{i}\right)\mathbb{P}(|\mathcal{I}_{i-1}|=m^{-}_{i})$
(50)
$\displaystyle+\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\,\,\big{|}\,\,|\mathcal{I}_{i-1}|=m^{-}_{i}+1\right)\mathbb{P}(|\mathcal{I}_{i-1}|=m^{-}_{i}+1)$
(51)
$\displaystyle=\frac{m_{i}-m_{i-1}}{(m^{-}_{i}+1-m_{i-1})}\cdot(m^{-}_{i}+1-m_{i-1})+0\cdot(m_{i-1}-m^{-}_{i})=x_{i}.$

See Fig. 9 (right). If $m_{i-1}<m^{-}_{i}$, then
$|\mathcal{I}_{i-1}|\in\\{m^{-}_{i}-1,m^{-}_{i}\\}$ and

$\displaystyle\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\right)$
$\displaystyle=\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\,\,\big{|}\,\,|\mathcal{I}_{i-1}|=m^{-}_{i}-1\right)\mathbb{P}(|\mathcal{I}_{i-1}|=m^{-}_{i}-1)$
(52)
$\displaystyle+\mathbb{P}\left(m_{i}\geq\xi+|\mathcal{I}_{i-1}|\,\,\big{|}\,\,|\mathcal{I}_{i-1}|=m^{-}_{i}\right)\mathbb{P}(|\mathcal{I}_{i-1}|=m^{-}_{i})$
(53)
$\displaystyle=1\cdot(m^{-}_{i}-m_{i-1})+\frac{m_{i}-m^{-}_{i}}{m_{i-1}-m^{-}_{i}+1}\cdot(m_{i-1}-m^{-}_{i}+1)=m_{i}-m_{i-1}=x_{i}.$

∎

###### Lemma 0.

Consider a fractional cache configuration $\boldsymbol{x}^{\prime}$ obtained
by an elementary mass movement of $\delta$ from $u\in\mathcal{N}$ to
$v\in\mathcal{N}$ for configuration $\boldsymbol{x}\in\mathcal{X}$, i.e.,
$\boldsymbol{x}^{\prime}=\boldsymbol{x}-\delta\bm{e}_{u}+\delta\bm{e}_{v}$.
Algorithm 3 outputs the random integral cache configurations $\boldsymbol{z}$,
and $\boldsymbol{z}^{\prime}$, given the input fractional cache states
$\boldsymbol{x}$ and $\boldsymbol{x}^{\prime}$, respectively. The random
integral configurations satisfy
$\mathbb{E}_{\xi}\left[\left\lVert\boldsymbol{z}^{\prime}-\boldsymbol{z}\right\rVert_{1,\boldsymbol{w}^{\prime}}\right]\leq
2kN\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}\delta$, where
$\left\lVert\boldsymbol{x}\right\rVert_{1,\boldsymbol{w}^{\prime}}\triangleq\sum_{i\in\mathcal{N}}\left|x_{i}\right|w^{\prime}_{i}$,
and note that
$\mathrm{UC}_{\boldsymbol{r}_{t}}\left(\boldsymbol{z}_{t},\boldsymbol{z}_{t+1}\right)\leq\left\lVert\boldsymbol{z}_{t}-\boldsymbol{z}_{t+1}\right\rVert_{1,\boldsymbol{w}^{\prime}}$.

###### Proof.

The probability that a random integral cache state $\boldsymbol{z}^{\prime}$
is changed w.r.t $\boldsymbol{z}$ can be upper bounded as
$\mathbb{P}(\boldsymbol{z}\neq\boldsymbol{z}^{\prime})\leq\sum_{i\in\mathcal{N}}\mathbb{P}(z_{i}\neq
z^{\prime}_{i})$. Consider w.l.g that $u<v$, then $\mathbb{P}(z_{i}\neq
z^{\prime}_{i})=0$ for $i\in\mathcal{N}\setminus\\{u,u+1,\dots,v\\}$, and
$\mathbb{P}(z_{i}\neq z^{\prime}_{i})=\delta$ for $i\in\\{u,u+1,\dots,v\\}$.
We obtain
$\mathbb{P}(\boldsymbol{z}\neq\boldsymbol{z}^{\prime})\leq\delta(v-u+1)$
(e.g., see Fig. 10). More generally, for $u\neq v$ we have
$\mathbb{P}(\boldsymbol{z}\neq\boldsymbol{z}^{\prime})\leq\delta(|v-u|+1)$. We
conclude that

$\displaystyle\mathbb{E}\left[{\left\lVert\boldsymbol{z}^{\prime}-\boldsymbol{z}\right\rVert_{1,\boldsymbol{w}^{\prime}}}\right]$
$\displaystyle\leq\max_{(\boldsymbol{z},\boldsymbol{z}^{\prime})\in\mathcal{Z}^{2}}\left\lVert\boldsymbol{z}^{\prime}-\boldsymbol{z}\right\rVert_{1,\boldsymbol{w}^{\prime}}\cdot\mathbb{P}(\boldsymbol{z}\neq\boldsymbol{z}^{\prime})\leq
2k\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}(|v-u|+1)\delta\leq
2kN\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}\delta.\qed$

figurec

Figure 11. Coupling induced by Online Rounding Algorithm 3 when $\xi$ is
fixed. The flow $f_{i,j}$ is the joint probability
$\mathbb{P}(\boldsymbol{z}_{t+1}=\zeta_{t+1,j},\boldsymbol{z}_{t}=\zeta_{t,i})$,
so that the next state is $\zeta_{t+1,j}$ and the previously selected state is
$\zeta_{t,i}$.

###### Lemma 0.

The expected movement cost of the random integral cache states generate by
Algorithm 3 is
$\mathbb{E}_{\xi}\left[\mathrm{UC}_{\boldsymbol{r}_{t}}\left(\boldsymbol{z}_{t},\boldsymbol{z}_{t+1}\right)\right]=\operatorname{\mathcal{O}}\left(\left\lVert\boldsymbol{x}_{t}-\boldsymbol{x}_{t+1}\right\rVert_{1}\right)$,
when $\xi$ is sampled once u.a.r. from the interval $[0,1]$ and then fixed for
$t\in[T]$.

###### Proof.

The general fractional movement caused by a policy $\mathcal{A}$ changes the
cache state from fractional state $\boldsymbol{x}_{t}\in\mathcal{X}$ to
$\boldsymbol{x}_{t+1}\in\mathcal{X}$, and we denote by
$\mathcal{J}=\left\\{i\in\mathcal{N}:x_{t+1,i}-x_{t,i}>0\right\\}$ the set of
components that have a fractional increase. We have
$\boldsymbol{x}_{t+1}=\boldsymbol{x}_{t}+\sum_{j\in\mathcal{J}}\phi_{j}e_{j}-\sum_{i\in\mathcal{N}\setminus\mathcal{J}}\phi_{i}e_{i}$.
where $\phi_{i},i\in\mathcal{N}$ is the absolute fractional change in
component $i$ of the cache. Remark that we have
$\left\lVert\boldsymbol{x}_{t+1}-\boldsymbol{x}_{t}\right\rVert_{1,\bm{w}^{\prime}}=\sum_{i\in\mathcal{N}}w^{\prime}_{i}\phi_{i}$.
From the capacity constraint we know that
$\sum_{i\in\mathcal{N}\setminus\mathcal{J}}\phi_{i}=\sum_{j\in\mathcal{J}}\phi_{j}$.
If we want to decompose this general fractional change to elementary
operations, then we need to find a flow
$\left[\delta_{i,j}\right]_{(i,j)\in(\mathcal{N}\setminus\mathcal{J})\times\mathcal{J}}$
that moves $\sum_{j\in\mathcal{J}}\phi_{j}$ mass from the components in
$\mathcal{N}\setminus\mathcal{J}$ to to those in $\mathcal{J}$. This requires
at most $N-1$ elementary operations. We define the map
$\nu:\mathcal{N}^{2}\to\mathbb{N}$ that provides an order on the sequence of
elementary operations. Let $\boldsymbol{z}^{\nu(i,j)}$ be the random cache
state that could have been sampled after the $\nu(i,j)$-th elementary
operation where
$\mathbb{E}\left[\boldsymbol{z}^{\nu(i,j)}\right]=\boldsymbol{x}^{\nu(i,j)}$,
and the total number of operations is denoted by $|\nu|\leq N-1$. Note that by
definition $\boldsymbol{z}^{|\nu|}=\boldsymbol{z}_{t+1}$, and we take
$\boldsymbol{z}^{0}=\boldsymbol{z}_{t}$. For each of these operations we pay
in expectation at most
$2kN\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}{\delta_{i,j}}$
update cost from Lemma 3. Then the total expected movement cost is:

$\displaystyle\textstyle\mathbb{E}_{\xi}\big{[}\mathrm{UC}_{\boldsymbol{r}_{t}}(\boldsymbol{z}_{t}$
$\displaystyle,\boldsymbol{z}_{t+1})\big{]}\leq\textstyle\mathbb{E}_{\xi}[\left\lVert\boldsymbol{z}_{t+1}-\boldsymbol{z}_{t}\right\rVert_{1,\boldsymbol{w}^{\prime}}]=\mathbb{E}_{\xi}\left[\left\lVert\sum^{|\nu|-1}_{l=0}\left(\boldsymbol{z}^{l+1}-\boldsymbol{z}^{l}\right)\right\rVert_{1,\boldsymbol{w}^{\prime}}\right]\leq\sum^{|\nu|-1}_{l=0}\mathbb{E}_{\xi}\left[\left\lVert\boldsymbol{z}^{l+1}-\boldsymbol{z}^{l}\right\rVert_{1,\boldsymbol{w}^{\prime}}\right]$
$\displaystyle\leq\textstyle\sum_{i\in\mathcal{N}\setminus\mathcal{J}}\sum_{j\in\mathcal{J}}2k\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}N\delta_{i,j}=2kN\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}\sum_{i\in\mathcal{N}\setminus\mathcal{J}}\phi_{i}\leq
kN\left\lVert\boldsymbol{w}^{\prime}\right\rVert_{\infty}\left\lVert\boldsymbol{x}_{t+1}-\boldsymbol{x}_{t}\right\rVert_{1}.$

The update cost is thus
$\operatorname{\mathcal{O}}\left(\left\lVert\boldsymbol{x}_{t+1}-\boldsymbol{x}_{t}\right\rVert_{1}\right)$
in expectation. ∎


# Rethinking Model Selection and Decoding for Keyphrase Generation
with Pre-trained Sequence-to-Sequence Models

Di Wu, Wasi Uddin Ahmad, Kai-Wei Chang
University of California, Los Angeles
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Keyphrase Generation (KPG) is a longstanding task in NLP with widespread
applications. The advent of sequence-to-sequence (seq2seq) pre-trained
language models (PLMs) has ushered in a transformative era for KPG, yielding
promising performance improvements. However, many design decisions remain
unexplored and are often made arbitrarily. This paper undertakes a systematic
analysis of the influence of model selection and decoding strategies on PLM-
based KPG. We begin by elucidating why seq2seq PLMs are apt for KPG, anchored
by an attention-driven hypothesis. We then establish that conventional wisdom
for selecting seq2seq PLMs lacks depth: (1) merely increasing model size or
performing task-specific adaptation is not parameter-efficient; (2) although
combining in-domain pre-training with task adaptation benefits KPG, it does
partially hinder generalization. Regarding decoding, we demonstrate that while
greedy search achieves strong F1 scores, it lags in recall compared with
sampling-based methods. Based on these insights, we propose DeSel, a
likelihood-based decode-select algorithm for seq2seq PLMs. DeSel improves
greedy search by an average of 4.7% semantic F1 across five datasets. Our
collective findings pave the way for deeper future investigations into PLM-
based KPG.

##  1 Introduction

Keyphrases encapsulate the core information of a document. Due to their high
information density, they have been found valuable in areas such as
information retrieval (Wu and Bolivar, 2008; Dave and Varma, 2010; Kim et al.,
2013; Boudin et al., 2020), document clustering (Hammouda et al., 2005),
summarization (Zhang et al., 2004), and text classification (Berend, 2011). A
keyphrase is termed a present keyphrase if it is explicitly found within the
document and an absent keyphrase otherwise. The task of identifying present
keyphrases is defined as keyphrase extraction (KPE), whereas keyphrase
generation (KPG) involves predicting both types of keyphrases.

Recently, pre-trained language models (PLMs) have been widely incorporated in
KPG (Chowdhury et al., 2022; Zhao et al., 2022) via sequence-to-sequence
(seq2seq) generation, with promising performance on zero-shot (Kulkarni et
al., 2022), multilingual (Gao et al., 2022), and low-resource (Wu et al.,
2022a) KPG. However, existing literature typically focuses on a specific
subset of important components in this pipeline, such as data construction and
loss design, while making arbitrary choices for the others (Zhao et al., 2022;
Ray Chowdhury et al., 2022; Wu et al., 2022a; Garg et al., 2022). As a result,
KPG systems are often compared under different assumptions and the effect of
the arbitrary design choices remains unclear. To bridge this gap, this paper
focuses on two crucial questions that have not been systematically explored:

  1. 1.

Which PLM leads to the best KPG performance when fine-tuned?

  2. 2.

What is the best decoding strategy?

In practice, sub-optimal choices for these factors could lead to optimizing an
unnecessarily large model or sub-optimal results decoded from a strong KPG
model. To answer these two questions, we conduct in-depth analyses on KPG with
(1) PLMs of diverse size and pre-training strategies and (2) a diverse set of
decoding strategies.

To begin with, we posit that seq2seq PLMs are inherently suitable to KPG
(section 3). By drawing the correlations with a strong graph-based KPE
algorithm, we show that these PLMs implicitly compute phrase centrality
(Boudin, 2013) in their decoder attention patterns. This knowledge is also
directly translatable to a strong ranking function for KPE. On the other hand,
encoder-only models fail to carry such centrality information.

Next, we search for the best seq2seq PLM for KPG fine-tuning (section 4).
While common strategies for other NLP tasks might advocate for (1) scaling up
the model size, (2) in-domain pre-training, or (3) task adaptation, do these
approaches hold the same merit for KPG? Our findings reveal that a singular
emphasis on scaling or task adaptation does not ensure efficient performance
improvement. In contrast, in-domain pre-training consistently bolsters
performance across both keyphrase types and can benefit from task adaptation.
A robustness analysis reveals that a proper model choice and data-oriented
training approaches are complementary: without the latter, stronger PLMs are
more vulnerable to perturbed input, with over 14% recall drop under named
variation substitutions and over 5% recall drop under input paraphrasing.

Decoding strategy is also an essential component in PLM-based KPG, but much
under-explored by current literature. In section 5, we thoroughly compare six
decoding strategies including greedy search, beam search, and sampling-based
methods. Results suggest that when only generating a single sequence
consisting of concatenated keyphrases, greedy search achieves a strong F1
score. However, aggregating the predictions from multiple sampled sequences
outperforms greedy search due to a much higher recall.

Based on these findings, we introduce DeSel, a likelihood-based selection
strategy that selects from sampled phrases to augment the greedy search
predictions. DeSel utilizes the probability of phrases from greedy search’s
predictions as the baseline to filter out noisy predictions from a set of
sampled keyphrase candidates. Experiments on five KPG datasets show that DeSel
consistently improves greedy decoding by 7.9% $F1@M$ for present keyphrases,
25% $F1@M$ for absent keyphrases, and 4.7% Semantic F1 for all keyphrases,
achieving state-of-the-art KPG performance, underscoring the importance of
carefully examining the design choice of KPG.

To summarize, our primary contributions are:

  1. 1.

An in-depth exploration of the intrinsic suitability of seq2seq PLMs for KPG.

  2. 2.

A comprehensive examination of effective strategies for model enhancement in
KPG, spotlighting the merits of specific combinations and their implications
for robustness.

  3. 3.

We establish the trade-off between accuracy and concept coverage for different
decoding algorithms. Then, we introduce a probability-based decode-select
mechanism DeSel that consistently improves over greedy search.

  4. 4.

Our research illuminates the profound impact of under-emphasized factors on
KPG performance. To facilitate future research on KPG, we release our code and
models at https://github.com/uclanlp/DeepKPG.

##  2 Preliminaries

###  2.1 Keyphrase Generation

##### Problem Definition

We represent an example for KPG as a tuple $(\mathcal{X},\mathcal{Y})$,
corresponding to the input document $\mathcal{X}=(x_{1},x_{2},...,x_{d})$ and
the set of human-written reference keyphrases
$\mathcal{Y}=\\{y_{1},y_{2},...,y_{n}\\}$. Following Meng et al. (2017),
$y_{i}$ is classified as a present keyphrase if it is a substring of
$\mathcal{X}$ or an absent keyphrase otherwise. The KPG task requires
predicting $\mathcal{Y}$ in any order, and the KPE task only requires
predicting present keyphrases (Turney, 2000).

##### Evaluation

We adopt lexical-based and semantic-based evaluation to evaluate a model’s
predictions $\mathcal{P}=\\{p_{1},p_{2},...,p_{m}\\}$ against $\mathcal{Y}$.
For lexical evaluation, we follow Chan et al. (2019) and use the $P@M$, $R@M$,
and $F1@M$ scores. $\mathcal{P}$ and $\mathcal{Y}$ are stemmed with the Porter
Stemmer (Porter, 1980) and the duplicates are removed before the score
calculation. For semantic evaluation, we follow Wu et al. (2023) and report
$SemP$, $SemR$, and $SemF1$. Note that lexical metrics are separately
calculated for present and absent keyphrases while the semantic metrics are
calculated with all the phrases. We repeat all the experiments with three
random seeds and report the averaged performance.

##### Benchmark

Meng et al. (2017) introduce KP20k, which contains 500k Computer Science
papers. Following their work, we train on KP20k and evaluate on the title and
abstracts from the KP20k test set as well as four out-of-distribution testing
datasets: Inspec (Hulth, 2003), Krapivin (Krapivin et al., 2009), NUS (Nguyen
and Kan, 2007), and SemEval (Kim et al., 2010). Table 5 summarizes the
statistics of all testing datasets.

##### Baselines

We consider two strong supervised encoder-decoder models from Ye et al.
(2021):

  1. 1.

CopyTrans: A Transformer (Vaswani et al., 2017) with copy mechanism (See et
al., 2017).

  2. 2.

The SetTrans model, which performs order-agnostic KPG. The model uses control
codes trained via a k-step target assignment algorithm to generate keyphrases
in parallel.

As the goal of this work is thoroughly studying PLM-based methods, we only
provide the results of the strongest baselines as a point of reference. We
also include other baselines in appendix section G. In our analysis, we also
use MultipartiteRank (MPRank, Boudin (2018)), a performant graph-based
unsupervised KPE algorithm. More details about MPRank are discussed in section
3.1.

###  2.2 Sequence-to-Sequence PLMs

In this work, we focus on fine-tuning Transformer-based sequence-to-sequence
PLMs BART (Lewis et al., 2020) and T5 (Raffel et al., 2020) for KPG with the
"One2Seq" formulation. Concretely, following Ye and Wang (2018) and Yuan et
al. (2020), we use a separator token ; to join all the target keyphrases as
the target sequence $\mathcal{Y}=(y_{1}$ ; $...$ ; $y_{n})$. The models are
trained with the cross-entropy loss for generating $\mathcal{Y}$ based on
$\mathcal{X}$. At test time, greedy decoding is used, followed by a post-
processing stage to segment the output sequence into individual phrases. We
provide implementation details and hyperparameters in appendix D.

##  3 Do PLMs inherently carry significant knowledge of keyphrases?

Existing studies have justified their use of seq2seq PLMs by drawing the close
relationship between the pre-training tasks of BART (denoising language
modeling) or T5 (unified text-to-text transfer) and the formulation of KPG
(Gao et al., 2022; Zhao et al., 2022; Wu et al., 2022a) or KPE (Kong et al.,
2023). However, there is a lack of an in-depth understanding of why seq2seq
PLMs should be chosen for keyphrase-related tasks. In this section, we reason
based on phrase centrality (Litvak and Last, 2008; Boudin, 2013) and show that
PLMs with autoregressive decoders, including seq2seq PLMs, carry attention
heads that approximately function as centrality assigners and naturally as
potent keyphrase rankers.

###  3.1 Centrality of Phrases

The concept of phrase centrality originated from graph-based KPE, where
keyphrase candidates are represented as nodes. Various graph centrality
measures are used to determine a phrase’s importance in the document. We use
MPRank in our analysis, which encodes closeness-based and eigenvector-based
centrality (Boudin, 2013). MPRank first uses rules to obtain $C$ noun phrase
candidates and then performs lexical clustering to group the candidates into
topic clusters. Next, each candidate is represented as a graph node and
connected with the candidates from other topic clusters. TextRank (Mihalcea
and Tarau, 2004) is used to obtain a centrality score $c_{i}$ for each of the
nodes $n_{i}$. We refer the readers to Boudin (2018) for further details.

###  3.2 Attention intensities in BART and T5 decoders encode phrase
centrality

Using MPRank as a lens, we first investigate the extent to which PLMs
implicitly represent centrality information. We use the paper titles and
abstracts from the KP20k test set as the probing set. Each probing instance is
fed into a PLM and the attention weights from the self-attention layers are
collected. For the $h^{th}$ attention head at layer $l$, we denote the
attention from token $i$ to token $j$ as $\alpha^{l,h}_{i\rightarrow j}$. For
the $j^{th}$ token in the noun phrase candidate $n_{i}$, the global attention
weight on it is

$a^{l,h}_{ij}=\sum_{k=1,...,L}\alpha^{l,h}_{k\rightarrow j},$ (1)

where $L$ is the length of the text after tokenization. Then, the attention
weight of $n_{i}$ is calculated as

$a^{l,h}_{i}=|n_{i}|\sum_{j}a^{l,h}_{ij},$ (2)

where $|n_{i}|$ denotes the number of tokens in $n_{i}$.

Model | Size | Head | $\rho$ | Head | $\tau$
---|---|---|---|---|---
Encoder-only PLMs |  |  |
BERT-base | 110M | 3-0 | 0.300 | 3-0 | 0.206
BERT-large | 340M | 0-5 | 0.351 | 0-6 | 0.246
Decoder-only PLMs |  |  |
gpt2 | 117M | 0-11 | 0.626 | 0-11 | 0.479
gpt2-medium | 345M | 1-6 | 0.630 | 1-6 | 0.480
gpt2-large | 774M | 0-13 | 0.627 | 0-13 | 0.478
gpt2-xl | 1.5B | 0-6 | 0.626 | 0-6 | 0.476
Seq2seq PLMs |  |  |
BART-base | 140M | 0-6 | 0.608 | 0-6 | 0.459
BART-large | 406M | 0-9 | 0.585 | 0-9 | 0.438
T5-small | 60M | 4-4 | 0.624 | 4-4 | 0.471
T5-base | 223M | 8-4 | 0.621 | 8-4 | 0.466
T5-large | 770M | 0-2 | 0.628 | 0-2 | 0.471
T5-3B | 3B | 0-8 | 0.648 | 0-8 | 0.494

Table 1: Correlation between keyphrase candidates’ attention weights and
centrality scores. $l$-$h$ denotes attention head $h$ in layer $l$. Both $h$
and $l$ start from index 0. We report the attention head that achieves the
best scores. The highest score is boldfaced.

We study four families of models: BART, T5, BERT, and GPT-2 (Radford et al.,
2019). For BART and T5, we use their decoder attentions. We correlate
$a^{l,h}_{i}$ with $c_{i}$ using Spearman correlation $\rho$ and Kendall’s Tau
$\tau$ and present the best correlation for each model in Table 1.
Surprisingly, BART and T5 decoders contain attention heads that encode phrase
centrality similarly as MPRank. The head with the best correlation generally
appears in the lower layers, indicating that centrality understanding may be
more related to low-level features. Also, the upper bound of correlation
strength grows with model size for T5 while does not grow for BART. Beyond
centrality assigners, these attention heads are also potent keyphrase
extractors: simply ranking the noun phrase candidates by $a^{l,h}_{i}$
achieves similar $F1@5$ for present keyphrases or $SemF1@5$ score as MPRank
(appendix B).

Evaluating other types of PLMs, we find that BERT’s attention heads only show
weak centrality knowledge, with only 0.246 best Kendall Tau with MPRank. On
the other hand, GPT-2 exhibits a similar pattern to the decoders from seq2seq
PLMs, indicating that the observed pattern is strongly associated with
autoregressive decoders. As centrality is generally correlated with global
importance, our result aligns with the observations that masked language
modeling tends to exploit local dependency while causal language modeling can
learn long-range dependencies (Clark et al., 2019; Vig and Belinkov, 2019).

In summary, through attention-based analyses, we reveal novel insights into
the underlying keyphrase knowledge of PLMs with autoregressive decoders. Such
knowledge can be employed explicitly (via ranking) or implicitly (via fine-
tuning and prompting) to facilitate KPG. In the rest of the paper, we focus on
rethinking two basic designs for KPG with seq2seq PLMs.

##  4 Influence of PLM Choice for KPG

Three crucial design options exist for using seq2seq PLMs for KPG: the choice
of PLM to fine-tune, the fine-tuning data and objective, and the decoding
strategy. Previous work focuses on fine-tuning objective and data construction
(Meng et al., 2021; Ray Chowdhury et al., 2022; Garg et al., 2023) while often
making the other two choices in an ad hoc way, making it difficult to compare
among approaches. This section dives into the first question by evaluating
three "conventional wisdoms":

  1. 1.

Using PLMs with more parameters (section 4.1).

  2. 2.

Using in-domain PLMs (section 4.2).

  3. 3.

Using task-adapted PLMs (section 4.3).

Figure 1: KP20k test performance of models of various sizes and pre-training
strategies. +ID = using in-domain SciBART. +TAPT = second-stage training on
OAGKX. The results are averaged over 3 random seeds.

###  4.1 The scaling law for keyphrase generation

Although the effect of model sizes has been explored for a range of tasks, it
is poorly understood in the KPG literature, where most recent works employ a
single PLM with 100M to 500M parameters (Kulkarni et al., 2022; Wu et al.,
2022b; Zhao et al., 2022). To establish a common ground, we measure the
performance of fine-tuning BART-base/large (purple line) and
T5-small/base/large/3B (green line) and report the results on KP20k in Figure
1.

Surprisingly, fine-tuning BART or T5 is extremely parameter-inefficient
compared to task-specific architectures trained from scratch111We note that
this claim is orthogonal to the observations that PLMs are data-efficient (Wu
et al., 2022a).. For instance, although T5’s performance consistently
increases with the model size, around 8x more parameters are required to
achieve the same Present $F1@M$ on KP20k as SetTrans and 30x more parameters
are required to have a better $SemF1$. Closer inspection shows that SetTrans
excels in recall via its parallel control codes and the set loss. In
comparison, limited by the learning formulation and decoding strategy, fine-
tuned seq2seq PLMs fall behind in their recall of important keyphrases. In
section 5, we will show that this problem can be alleviated with a simple
decode-then-select strategy.

##### BART vs. T5

BART and T5 display similar scaling for $F1@M$ and $SemF1$. However, compared
to T5, BART’s recall scores increase more readily than the precision scores.
At the same number of parameters, BART also performs better on absent
keyphrases. One possible reason is that BART’s text infilling objective is
more advantageous for learning the knowledge for constructing spans absent
from text (Wu et al., 2022a).

##### Which score is more sensitive to scaling?

Compared to recall, precision is more sensitive to model size. For example,
T5-small achieves 98% $SemR$ compared to the 50x larger T5-3B. In addition,
absent keyphrases scores are more sensitive. Overall, this suggests that small
models are able to extract relevant keyphrases, but learn to selectively omit
unimportant keyphrases and create more absent keyphrases as the model size
grows. Indeed, the average number of predicted keyphrases decreases from
T5-small (6.75), T5-base (5.74), and T5-large (5.66), to T5-3B (5.48), while
the number of absent keyphrases increases from T5-small (0.91), T5-base
(0.99), T5-large (1.01), to T5-3B (1.05).

###  4.2 Domain knowledge is crucial to accurate keyphrase generation

In-domain pre-training has been shown effective in a wide range of tasks
requiring extensive domain knowledge (Beltagy et al., 2019; Lee et al., 2019).
As keyphrases often contain domain-specific terminologies, we hypothesize that
the domain of a PLM greatly affects its keyphrase generation ability. To test
this hypothesis, we pre-train in-domain BART models SciBART-base and SciBART-
large from scratch using the paper titles and abstracts from the S2ORC dataset
(Lo et al., 2020). The processed dataset contains 171.7M documents or 15.4B
tokens in total. The models are pre-trained on text infilling for 250k steps
with batch size 2048, learning rate 3e-4, 10k warm-up steps, and polynomial
learning rate decay. We present data processing and model training details in
appendix C.

The results of fine-tuning SciBART are presented with "+ID" (for "In-Domain")
in Figure 1. As expected, SciBART significantly improves over BART for all
three F1 metrics, outperforming the much larger T5-3B. Notably, SciBART also
has better parameter efficiency compared to general domain models: scaling
from SciBART-base to SciBART-large provides a much larger growth in $SemF1$
compared to scaling up BART and T5.

###  4.3 Task-adaptive pre-training is more effective with in-domain models

Task-adaptive pre-training (TAPT) is another common technique for adding task-
specific supervision signals (Gururangan et al., 2020). In this section, we
analyze the effect on KPG performance of adding two types of TAPT stages to
seq2seq PLMs: keyphrase generation or instruction following.

##### Keyphrase Pre-training

We directly use KeyBART (Kulkarni et al., 2022) (denoted as "+TAPT" in Figure
1), which is trained using the OAGKX dataset (Çano and Bojar, 2020) on KPG
with keyphrases corrupted from the input. To investigate the effects of TAPT
on in-domain PLMs, we also fine-tune SciBART on OAGKX with batch size 256,
learning rate 3e-5, and 250k steps. We denote this model as "+ID+TAPT" in
Figure 1.

##### Instruction Pre-training

Recently, instruction tuning has been introduced to improve the generalization
ability of PLMs (Mishra et al., 2022; Ouyang et al., 2022). As KPG is relevant
to classic NLP tasks such as information extraction and summarization, we
hypothesize that training with instruction data also serves as TAPT for
KPG222In fact, some variants of the keyphrase extraction task are included in
popular instruction datasets such as NIv2 (Wang et al., 2022) and Alpaca
(Taori et al., 2023).. To confirm, we benchmark FLAN-T5 (Chung et al., 2022),
a family of T5 models fine-tuned on instruction following datasets (yellow
line in Figure 1).

##### TAPT struggles to improve absent KPG but is more effective with in-
domain models.

Figure 1 suggests that both TAPT strategies lead to a similar amount of
improvement in the present keyphrase F1@M and SemF1. Surprisingly, the
absolute gain is small and TAPT hardly improves absent keyphrase performance.
For KeyBART, although its pre-training data (OAGKX) has a similar percentage
of absent keyphrases as KP20K (32% vs. 37%), its objective (recovering present
keyphrases from corrupted input) might still be different from absent
keyphrase generation. For FLAN-T5, we find that the KPG-related tasks in its
pre-training tasks often contain very short input text, representing a
significant distribution mismatch with KP20k. However, when applied on the in-
domain SciBART, TAPT can greatly improve the performance on KP20k. Combined
with section 4.2, we conclude that in-domain pre-training is more important
for KPG and TAPT serves a complementary secondary role.

###  4.4 Analysis: are strong KPG models sensitive to input perturbations?

As in-domain and task-adapted PLMs already greatly benefit KPG, are data
augmentation techniques no longer necessary? In this section, we reveal that
these designs increase the model’s sensitivity to input perturbations, and
data augmentation is still desired for better generalization.

####  4.4.1 Method

We design two input perturbations on KP20k to check the behaviors of BART-
based KPG models.

##### Name variation substitution

We construct 8905 perturbed inputs by replacing present keyphrases with their
name variations linked by Chan et al. (2019). Ideally, a robust KPG model
would have a similar recall for the original phrases and the name variations
as they appear in the same context.

##### Paraphrasing

We leverage gpt-3.5-turbo to paraphrase 1000 documents into a scientific
writing style. A good KPG model is expected to retain similar recall for
phrases that present both before and after paraphrasing since the inputs
describe the same ideas. The detailed prompt and several examples are
presented in appendix E.

####  4.4.2 Results

Model | Before | After | $\Delta$
---|---|---|---
Name Variation Substitution |
BART-large | 0.476 | 0.410 | -0.066 (13.9%$\downarrow$)
SciBART-large | 0.531 | 0.420 | -0.131 (20.9%$\downarrow$)
KeyBART | 0.463 | 0.398 |  -0.065 (14.2%$\downarrow$)
SciBART-large+TAPT | 0.525 | 0.434 | -0.091 (17.4%$\downarrow$)
Paraphrasing |
BART-large | 0.463 | 0.453 |  -0.010 (2.3%$\downarrow$)
SciBART-large | 0.524 | 0.494 | -0.030 (5.8%$\downarrow$)
KeyBART | 0.466 | 0.407 | -0.059 (12.6%$\downarrow$)
SciBART-large+TAPT | 0.522 | 0.487 | -0.035 (6.8%$\downarrow$)

Table 2: Results for the probing experiments. "Before" and "After" indicate
the recall scores before and after perturbation. $\Delta$ denotes the score
change from "Before" to "After". The best score is boldfaced.

Table 2 presents the perturbation results. We observe that models often fail
to predict name variations when they appear in place of the original synonyms,
while successfully maintaining their predictions given paraphrased inputs.
This indicates that the models may overfit to the high-frequency keyphrases in
the training set and input augmentation methods such as Garg et al. (2023) are
necessary to correct this pattern.

In addition, domain-specific or task-adapted models exhibit a larger
performance drop compared to BART-large, suggesting a trade-off between
domain/task specificity and generalization. Pre-trained on large-scale
keyphrase data, KeyBART may rely more on syntax and position information in
the data and thus be less sensitive to synonym change. On the other hand, pre-
trained on a large-scale scientific corpus, SciBART is more robust than
KeyBART on different scientific writing styles beyond the ones available in
KP20k.

###  4.5 Discussion

We summarize the main conclusions derived from the empirical results presented
in this section:

  * •

Naively scaling up BART and T5 is parameter-inefficient on KP20k compared to
SetTrans.

  * •

Domain knowledge is crucial for KPG performance and improves parameter
efficiency.

  * •

Task-adaptive training with keyphrase or instruction tuning data only
significantly improves KPG with in-domain models.

  * •

In-domain pre-training and TAPT harm generalization in different ways and data
augmentation during fine-tuning is desired.

Figure 2: A comparison of six strategies for decoding from the SciBART+TAPT
model. Greedy search achieves strong performance while performing worse than
beam search and sampling with multiple samples.

##  5 Decoding Strategy for KPG

While it is well-known that decoding strategies can strongly affect text
generation quality (Fan et al., 2018; Holtzman et al., 2020), surprisingly
there has been little study about decoding PLM-based KPG models. Previous
studies often directly use greedy search and variants of beam search (Gao et
al., 2022; Zhao et al., 2022; Wu et al., 2022a), limiting the understanding of
PLMs fine-tuned for KPG. To bridge this knowledge gap, we first carefully
evaluate six decoding strategies on the strongest PLM-based KPG model. We then
propose a simple yet effective decode-select strategy to mitigate the observed
deficiencies of greedy search.

###  5.1 Multi-sequence decoding: the trade-off between coverage and quality

We focus on decoding the SciBART-large+TAPT model fine-tuned on KP20k, under
the budget varying from 1 to 20 samples. The following six decoding algorithms
are compared. For each algorithm, their hyperparameters are chosen based on
the KP20k validation set.

  1. 1.

Greedy search.

  2. 2.

Beam search. We set the beam size to the number of desired samples.

  3. 3.

Diverse beam search (Vijayakumar et al., 2018). We set the number of groups to
the number of desired samples and the weight of the dissimilarity term to
$\lambda_{g}=0.1$.

  4. 4.

Vanilla sampling. We further apply a temperature scaling $\tau=0.7$.

  5. 5.

Top-k sampling (Fan et al., 2018). We use temperature $\tau=0.7$ and $k=2$ as
we find a large $k$ harms the generation quality.

  6. 6.

Nucleus sampling (Holtzman et al., 2020). We set $p=0.95$ and temperature
$\tau=0.5$.

Figure 2 presents the semantic-based evaluation results as a function of
sample size. In the single sample setting, greedy search achieves a strong
$SemF1$, only slightly outperformed by diverse beam search. For other methods,
we observe a clear trade-off between their information coverage ($SemR$) and
the noise in the final output ($SemP$) as the number of samples grows.
Nevertheless, all these methods are able to outperform greedy search at a
certain sample size, indicating that single-sequence decoding is sub-optimal.

###  5.2 A simple decode-select strategy boosts the performance of greedy
decoding

Greedy search captures the correlations in human-written labels but suffers
from local decisions and path dependency: high-quality keyphrases can be
missed with improbable first tokens. However, naively outputting the union of
multiple sampled sequences brings excessive noise. To achieve the balance
between the two, we introduce DeSel, a simple and effective three-stage
decoding strategy:

  1. 1.

Decode one sequence $G$ via greedy search.

  2. 2.

Sample $n$ sequences $\\{S_{1},...,S_{n}\\}$ to collect a set of candidate
keyphrases $S$.

  3. 3.

Select high quality phrases $\\{s_{1},...,s_{m}\\}\subset S$ and output the
sequence $(G\ ;\ s_{1}\ ;\ ...\ ;\ s_{m})$.

Method | KP20k | Inspec | Krapivin | NUS | SemEval
---|---|---|---|---|---
$P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$
CopyTrans | 0.376 | 0.046 | 0.562 | 0.333 | 0.023 | 0.569 | 0.365 | 0.063 | 0.547 | 0.429 | 0.044 | 0.579 | 0.321 | 0.022 | 0.377
SetTrans | 0.391 | 0.058 | 0.585 | 0.328 | 0.030 | 0.573 | 0.375 | 0.072 | 0.560 | 0.446 | 0.055 | 0.597 | 0.342 | 0.029 | 0.396
CorrKG† | 0.404 | 0.071 | N/A | 0.365 | 0.045 | N/A | N/A | N/A | N/A | 0.449 | 0.079 | N/A | 0.359 | 0.044 | N/A
BART-large | 0.392 | 0.047 | 0.575 | 0.333 | 0.024 | 0.565 | 0.347 | 0.051 | 0.517 | 0.435 | 0.048 | 0.586 | 0.311 | 0.024 | 0.381
SciBART-large | 0.396 | 0.057 | 0.587 | 0.328 | 0.026 | 0.557 | 0.329 | 0.056 | 0.503 | 0.421 | 0.050 | 0.567 | 0.304 | 0.033 | 0.382
\+ TAPT | 0.426 | 0.063 | 0.597 | 0.330 | 0.030 | 0.569 | 0.347 | 0.064 | 0.519 | 0.442 | 0.055 | 0.585 | 0.333 | 0.031 | 0.386
\+ TAPT + DeSel | 0.431 | 0.076 | 0.612 | 0.402 | 0.036 | 0.611 | 0.352 | 0.086 | 0.546 | 0.449 | 0.068 | 0.610 | 0.341 | 0.040 | 0.402

Table 3: Testing results on all datasets. $P$ and $A$ stand for $F1@M$ for
present and absent keyphrases. $Sem$ stands for $SemF1$. The best performance
is boldfaced. †copied from Zhao et al. (2022). The best entries in each column
are statistically significantly higher than the second best (p < 0.05) via a
paired bootstrap test. Full results in Table 8.

For step 3, we estimate $\Pr(s_{i}|\mathcal{X})$ for every phrase $s_{i}$ in
the $n$ samples and $\Pr(g_{j}|\mathcal{X})$ for every phrase $g_{j}\in G$.
Then, we use $G$ as a baseline to select at most $m$ most probable $s_{i}$
that satisfies

$\Pr(s_{i}|\mathcal{X})\geq\frac{\alpha}{|G|}\sum_{g_{j}\in
G}\Pr(g_{j}|\mathcal{X}),$ (3)

where $\alpha$ is a hyperparameter controlling the tradeoff between precision
and recall. The probability estimation is obtained with either the original
model or a newly trained "one2one" model333Starting from KeyBART, the one2one
model can be efficiently trained. We provide more details in appendix F. that
learns to generate a single keyphrase based on $\mathcal{X}$. We use nucleus
sampling with $p$ = 0.95 and $\tau$ = 0.5 for step 2, and set $m$ = 10, $n$ =
10, and $\alpha$ = 0.78.

Table 3 presents the test results of important models in this paper. DeSel
consistently improves the performance over the base model by a large margin.
In Table 4, we compare against other selection strategies including random
selection, input overlap using a sentence transformer model, and FreqFS
proposed in Zhao et al. (2022). DeSel is the only method that consistently
outperforms both greedy search and nucleus sampling.

Method | $P$ | $A$ | $Sem$
---|---|---|---
Greedy search ($G$) | 0.426 | 0.063 | 0.597
Nucleus sampling ($S$) | 0.385 | 0.074 | 0.599
Random Selection | 0.385 | 0.072 | 0.596
Input Overlap | 0.402 | 0.064 | 0.611
FreqFS (Zhao et al., 2022) | 0.426 | 0.072 | 0.610
DeSel (self) | 0.426 | 0.070 | 0.608
DeSel (one2one) | 0.431 | 0.076 | 0.612

Table 4: A comparison across different decoding strategies. Methods below the
dotted line merge $G$ with $S$.

##### Discussion

Compared to the single-sequence decoding baselines, DeSel wins by bringing in
the diversity. Compared to the baseline ranking methods, DeSel wins by
capturing the correlations between labels (encoded in the greedy search
outputs) and using the likelihood-based criteria to filter out high-quality
phrases from the diverse candidates.

##### Efficiency

DeSel harms the inference latency as it generates multiple sequences. To
improve the efficiency, one optimization is reusing the encoder’s outputs for
all the decoding and scoring operations. We implemented this strategy and
benchmarked it with the BART (base) model. DeSel with n = 10 (1 greedy search
and 10 sampling sequences decoded) takes 3.8x time compared to greedy
decoding.

##  6 Related Work

##### Keyphrase Generation

Meng et al. (2017) propose the task of Deep Keyphrase Generation and a strong
baseline model CopyRNN. Later works improve the architecture by adding
correlation constraints (Chen et al., 2018) and linguistic constraints (Zhao
and Zhang, 2019), exploiting learning signals from titles (Ye and Wang, 2018;
Chen et al., 2019b), and hierarchical modeling the phrases and words (Chen et
al., 2020). Ye and Wang (2018) reformulate the problem as generating a
sequence of keyphrases, while Ye et al. (2021) further uses a set generation
formulation to remove the influence of difference target phrase ordering.
Other works include incorporating reinforcement learning (Chan et al., 2019;
Luo et al., 2021), GANs (Swaminathan et al., 2020), and unifying KPE with KPG
(Chen et al., 2019a; Ahmad et al., 2021). Meng et al. (2021) conduct an
empirical study on architecture, generalizability, phrase order, and decoding
strategies, with the main focus on models trained from scratch instead of
PLMs.

##### PLMs for KPG

More recently, Wu et al. (2021), Chowdhury et al. (2022), Wu et al. (2022a),
Gao et al. (2022), and Wu et al. (2022b) consider fine-tuning prefix-LMs or
seq2seq PLMs for KPG. Kulkarni et al. (2022) use KPG as a pre-training task to
learn strong BART-based representations. Zhao et al. (2022) adopt optimal
transport for loss design and propose frequency-based filtering for decoding
to improve BART-based KPG.

##  7 Conclusion

This paper systematically investigated model selection and decoding for
building KPG models with seq2seq PLMs. Our analyses suggested much more
nuanced patterns beyond the "conventional wisdom" assumed by the majority of
current literature. Our novel decoding strategy, DeSel, significantly improved
the performance of greedy search across multiple datasets. More broadly, this
study underscores the distinct nature of the KPG task. One should not blindly
transpose conclusions or assumptions from other text generation tasks.
Instead, they warrant careful re-evaluation and empirical validation. Our work
also opens up exciting directions for future work, with deep groundings in
keyphrase literature. For instance, making KPG models more robust,
interpreting a KPG model, and designing better decoding algorithms for KPG.

## Limitations

While our study sheds light on important aspects of keyphrase generation (KPG)
models, several limitations present opportunities for future research.

First, our analysis focuses on model selection and decoding and thus uses
default cross entropy loss and original training set without data
augmentations. Investigating how the discussed design choices with more recent
data augmentation (Ray Chowdhury et al., 2022; Garg et al., 2022) or training
strategies (Zhao et al., 2022) is an important future study. In addition, the
best approach to combine the conclusions reached in this paper on long input
KPG (Garg et al., 2022) or KPG models trained with reinforcement learning
(Chan et al., 2019; Luo et al., 2021) worth future study.

Second, while in-domain pre-training combined with task adaptation was found
to enhance KPG performance, we did not fully investigate the underlying
mechanisms leading to these improvements. Further research could explore the
interplay between these two aspects and uncover more granular insights into
how they improve KPG.

Finally, although we revealed a compromise between performance optimization
and model robustness, we did not delve into designing new methods for
improving the robustness of these models against perturbed inputs. Future
research could further explore techniques to mitigate this trade-off,
developing models that maintain high performance while being resistant to
input perturbations.

## Ethics Statement

S2ORC and OAGKX are released under the Creative Commons By 4.0 License. We
perform text cleaning and email/URL filtering on S2ORC to remove sensitive
information, and we keep OAGKX as-is. We use the keyphrase benchmarking
datasets distributed by the original authors. No additional preprocessing is
performed before fine-tuning except lower-casing and tokenization. We do not
re-distribute any of the datasets used in this work.

Potential risks of SciBART include accidental leakage of (1) sensitive
personal information and (2) inaccurate factual information. For (1), we
carefully preprocess the data in the preprocessing stage to remove personal
information, including emails and URLs. However, we had difficulties
desensitizing names and phone numbers in the text because they overlapped with
the informative content. For (2), since SciBART is pre-trained on scientific
papers, it may generate scientific-style statements that include inaccurate
information. We encourage the potential users of SciBART not to rely fully on
its outputs without verifying their correctness.

Pre-training SciBART and fine-tuning the large T5 models are computationally
heavy, and we estimate the total CO2 emission to be around 3000 kg using the
calculation application provided by Lacoste et al. (2019). We will release the
fine-tuned checkpoints and we document the hyperparameters in the appendix
section D to help the community reduce the energy spent optimizing PLMs for
KPG and other various NLP applications.

## Acknowledgments

The research is supported in part by Taboola, NSF CCF-2200274, and an Amazon
AWS credit award. We thank the Taboola team for the helpful discussion. We
also thank anonymous reviewers, Da Yin, Tanmay Parekh, and other members of
the UCLA-NLP group for their valuable feedback.

## References

  * Ahmad et al. (2021) Wasi Ahmad, Xiao Bai, Soomin Lee, and Kai-Wei Chang. 2021.  Select, extract and generate: Neural keyphrase generation with layer-wise coverage attention.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1389–1404, Online. Association for Computational Linguistics.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  SciBERT: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  * Berend (2011) Gábor Berend. 2011.  Opinion expression mining by exploiting keyphrase extraction.  In _Proceedings of 5th International Joint Conference on Natural Language Processing_ , pages 1162–1170, Chiang Mai, Thailand. Asian Federation of Natural Language Processing.
  * Boudin (2013) Florian Boudin. 2013.  A comparison of centrality measures for graph-based keyphrase extraction.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 834–838, Nagoya, Japan. Asian Federation of Natural Language Processing.
  * Boudin (2018) Florian Boudin. 2018.  Unsupervised keyphrase extraction with multipartite graphs.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 667–672, New Orleans, Louisiana. Association for Computational Linguistics.
  * Boudin et al. (2020) Florian Boudin, Ygor Gallina, and Akiko Aizawa. 2020.  Keyphrase generation for scientific document retrieval.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1118–1126, Online. Association for Computational Linguistics.
  * Çano and Bojar (2020) Erion Çano and Ondřej Bojar. 2020.  Two huge title and keyword generation corpora of research articles.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 6663–6671, Marseille, France. European Language Resources Association.
  * Chan et al. (2019) Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019.  Neural keyphrase generation via reinforcement learning with adaptive rewards.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2163–2174, Florence, Italy. Association for Computational Linguistics.
  * Chen et al. (2018) Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018.  Keyphrase generation with correlation constraints.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4057–4066, Brussels, Belgium. Association for Computational Linguistics.
  * Chen et al. (2019a) Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a.  An integrated approach for keyphrase generation via exploring the power of retrieval and extraction.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2846–2856, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Chen et al. (2020) Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020.  Exclusive hierarchical decoding for deep keyphrase generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1095–1105, Online. Association for Computational Linguistics.
  * Chen et al. (2019b) Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R. Lyu. 2019b.  Title-guided encoding for keyphrase generation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 6268–6275.
  * Chowdhury et al. (2022) Md Faisal Mahbub Chowdhury, Gaetano Rossiello, Michael Glass, Nandana Mihindukulasooriya, and Alfio Gliozzo. 2022.  Applying a generic sequence-to-sequence model for simple and effective keyphrase generation.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Clark et al. (2019) Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019.  What does BERT look at? an analysis of BERT’s attention.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 276–286, Florence, Italy. Association for Computational Linguistics.
  * Dave and Varma (2010) Kushal S. Dave and Vasudeva Varma. 2010.  Pattern based keyword extraction for contextual advertising.  In _Proceedings of the 19th ACM International Conference on Information and Knowledge Management_ , CIKM ’10, page 1885–1888, New York, NY, USA. Association for Computing Machinery.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898, Melbourne, Australia. Association for Computational Linguistics.
  * Gao et al. (2022) Yifan Gao, Qingyu Yin, Zheng Li, Rui Meng, Tong Zhao, Bing Yin, Irwin King, and Michael Lyu. 2022.  Retrieval-augmented multilingual keyphrase generation with retriever-generator iterative training.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1233–1246, Seattle, United States. Association for Computational Linguistics.
  * Garg et al. (2023) Krishna Garg, Jishnu Ray Chowdhury, and Cornelia Caragea. 2023.  Data augmentation for low-resource keyphrase generation.
  * Garg et al. (2022) Krishna Garg, Jishnu Ray Chowdhury, and Cornelia Caragea. 2022.  Keyphrase generation beyond the boundaries of title and abstract.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 5809–5821, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8342–8360, Online. Association for Computational Linguistics.
  * Hammouda et al. (2005) Khaled Hammouda, Diego Matute, and Mohamed S. Kamel. 2005.  Corephrase: Keyphrase extraction for document clustering.  In _International workshop on machine learning and data mining in pattern recognition_ , pages 265–274.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Hulth (2003) Anette Hulth. 2003.  Improved automatic keyword extraction given more linguistic knowledge.  In _Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing_ , EMNLP ’03, page 216–223, USA. Association for Computational Linguistics.
  * Kim et al. (2010) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010.  SemEval-2010 task 5 : Automatic keyphrase extraction from scientific articles.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pages 21–26, Uppsala, Sweden. Association for Computational Linguistics.
  * Kim et al. (2013) Youngsam Kim, Munhyong Kim, Andrew Cattle, Julia Otmakhova, Suzi Park, and Hyopil Shin. 2013.  Applying graph-based keyword extraction to document retrieval.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 864–868, Nagoya, Japan. Asian Federation of Natural Language Processing.
  * Kong et al. (2023) Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, and Xiaoyan Bai. 2023.  Promptrank: Unsupervised keyphrase extraction using prompt.
  * Krapivin et al. (2009) Mikalai Krapivin, Aliaksandr Autaeu, and Maurizio Marchese. 2009.  Large dataset for keyphrases extraction.  Technical report, University of Trento.
  * Kulkarni et al. (2022) Mayank Kulkarni, Debanjan Mahata, Ravneet Arora, and Rajarshi Bhowmik. 2022.  Learning rich representation of keyphrases from text.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 891–906, Seattle, United States. Association for Computational Linguistics.
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019\.  Quantifying the carbon emissions of machine learning.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _CoRR_ , abs/1901.08746.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Litvak and Last (2008) Marina Litvak and Mark Last. 2008.  Graph-based keyword extraction for single-document summarization.  In _Coling 2008: Proceedings of the workshop Multi-source Multilingual Information Extraction and Summarization_ , pages 17–24, Manchester, UK. Coling 2008 Organizing Committee.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020.  S2ORC: The semantic scholar open research corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4969–4983, Online. Association for Computational Linguistics.
  * Luo et al. (2021) Yichao Luo, Yige Xu, Jiacheng Ye, Xipeng Qiu, and Qi Zhang. 2021.  Keyphrase generation with fine-grained evaluation-guided reinforcement learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 497–507, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Meng et al. (2021) Rui Meng, Xingdi Yuan, Tong Wang, Sanqiang Zhao, Adam Trischler, and Daqing He. 2021\.  An empirical study on neural keyphrase generation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4985–5007, Online. Association for Computational Linguistics.
  * Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017.  Deep keyphrase generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 582–592, Vancouver, Canada. Association for Computational Linguistics.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  TextRank: Bringing order into text.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 404–411, Barcelona, Spain. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * Nguyen and Kan (2007) Thuy Dung Nguyen and Min-Yen Kan. 2007.  Keyphrase extraction in scientific publications.  In _Asian Digital Libraries. Looking Back 10 Years and Forging New Frontiers_ , pages 317–326, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Porter (1980) Martin F. Porter. 1980.  An algorithm for suffix stripping.  _Program_ , 40:211–218.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ray Chowdhury et al. (2022) Jishnu Ray Chowdhury, Seo Yeon Park, Tuhin Kundu, and Cornelia Caragea. 2022.  KPDROP: Improving absent keyphrase generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4853–4870, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Swaminathan et al. (2020) Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, and Amanda Stent. 2020.  A preliminary exploration of GANs for keyphrase generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8021–8030, Online. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html_ , 3(6):7.
  * Turney (2000) Peter D Turney. 2000.  Learning algorithms for keyphrase extraction.  _Information retrieval_ , 2:303–336.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Vig and Belinkov (2019) Jesse Vig and Yonatan Belinkov. 2019.  Analyzing the structure of attention in a transformer language model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 63–76, Florence, Italy. Association for Computational Linguistics.
  * Vijayakumar et al. (2018) Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018.  Diverse beam search for improved description of complex scenes.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wu et al. (2022a) Di Wu, Wasi Ahmad, Sunipa Dev, and Kai-Wei Chang. 2022a.  Representation learning for resource-constrained keyphrase generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 700–716, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wu et al. (2023) Di Wu, Da Yin, and Kai-Wei Chang. 2023.  Kpeval: Towards fine-grained semantic-based evaluation of keyphrase extraction and generation systems.  _arXiv preprint arXiv:2303.15422_.
  * Wu et al. (2021) Huanqin Wu, Wei Liu, Lei Li, Dan Nie, Tao Chen, Feng Zhang, and Di Wang. 2021.  UniKeyphrase: A unified extraction and generation framework for keyphrase prediction.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 825–835, Online. Association for Computational Linguistics.
  * Wu et al. (2022b) Huanqin Wu, Baijiaxin Ma, Wei Liu, Tao Chen, and Dan Nie. 2022b.  Fast and constrained absent keyphrase generation by prompt-based learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 11495–11503.
  * Wu and Bolivar (2008) Xiaoyuan Wu and Alvaro Bolivar. 2008.  Keyword extraction for contextual advertisement.  In _Proceedings of the 17th International Conference on World Wide Web_ , WWW ’08, page 1195–1196, New York, NY, USA. Association for Computing Machinery.
  * Ye and Wang (2018) Hai Ye and Lu Wang. 2018.  Semi-supervised learning for neural keyphrase generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4142–4153, Brussels, Belgium. Association for Computational Linguistics.
  * Ye et al. (2021) Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021.  One2Set: Generating diverse keyphrases as a set.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4598–4608, Online. Association for Computational Linguistics.
  * Yuan et al. (2020) Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2020.  One size does not fit all: Generating and evaluating variable number of keyphrases.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7961–7975, Online. Association for Computational Linguistics.
  * Zhang et al. (2004) Yongzheng Zhang, Nur Zincir-Heywood, and Evangelos Milios. 2004.  World wide web site summarization.  _Web Intelli. and Agent Sys._ , 2(1):39–53.
  * Zhao et al. (2022) Guangzhen Zhao, Guoshun Yin, Peng Yang, and Yu Yao. 2022.  Keyphrase generation via soft and hard semantic corrections.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7757–7768, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhao and Zhang (2019) Jing Zhao and Yuxiang Zhang. 2019.  Incorporating linguistic constraints into keyphrase generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5224–5233, Florence, Italy. Association for Computational Linguistics.

Supplementary Material: Appendices

##  Appendix A Test Set Statistics

Table 5 summarizes the statistics of all testing datasets we use. We use the
version distributed by Meng et al. (2017). In this distribution, SemEval’s
documents and keyphrases are already stemmed while the other datasets are not.

Dataset | #Examples | #KP |  $\%$AKP | |KP|
---|---|---|---|---
KP20k | 20000 | 5.3 | 37.1 | 2.0
Inspec | 500 | 9.8 | 26.4 | 2.5
Krapivin | 400 | 5.9 | 44.3 | 2.2
NUS | 211 | 11.7 | 45.6 | 2.2
SemEval | 100 | 14.7 | 57.4 | 2.4
Table 5: Test sets statistics. #KP, $\%$AKP, and |KP| refers to the average
number of keyphrases per document, the percentage of absent keyphrases, and
the average number of words that each keyphrase contains.

##  Appendix B Attention heads as keyphrase extractors

We present the KPE performance of ranking noun phrase candidate using their
attention intensities in Table 6. We observe that BART and T5 contain
attention heads that function well as keyphrase extractors, some of which even
surpass the KPE performance of the strong MPRank algorithm.

Model | Size | Head |  PKP $F1@5$ | $SemF1@5$
---|---|---|---|---
MPRank | - | - | 0.188 | 0.429
BART-base | 140M | 5-3 | 0.182 | 0.451
BART-large | 406M | 11-9 | 0.191 | 0.452
T5-small | 60M | 4-4 | 0.183 | 0.420
T5-base | 223M | 8-1 | 0.164 | 0.388
T5-large | 770M | 17-15 | 0.187 | 0.436
T5-3B | 3B | 21-27 | 0.183 | 0.426

Table 6: Keyphrase extraction performance of utilizing attention intensity as
ranking criteria. BART-large’s attention head 11-9 is even able to outperform
MPRank. PKP = present keyphrases. Figure 3: Domain distribution of the S2ORC
dataset.

##  Appendix C SciBART Pre-training Details

##### Corpus and Data Preprocessing

The S2ORC dataset contains over 100M papers from a variety of disciplines
(Figure 3). We train on all the titles and abstracts to increase the coverage
of different topics. After removing non-English444We use guess_language for
language detection. or title-only entries, we fix wrong Unicode characters,
remove emails and URLs, and convert the text to ASCII encoding555We use clean-
text for data cleaning.. The final dataset contains 171.7M documents or 15.4B
tokens in total. We reserve 10k documents for validation and 10k for testing
and use the rest as training data.

##### Vocabulary

Beltagy et al. (2019) suggest that using a domain-specific vocabulary is
crucial to downstream in-domain fine-tuning performance. Following their
observations, we build a cased BPE vocabulary in the scientific domain using
the SentencePiece666 https://github.com/google/sentencepiece library on the
cleaned training data. We set the vocabulary size to 30K.

##### Training

For the pre-training objective, we only use text infilling as introduced in
Lewis et al. (2020). We mask 30% of all tokens in each example, with the spans
randomly sampled from a Poisson distribution ($\lambda=3.5$). For 10% of the
spans selected to mask, we replace them with a random token instead of the
mask token. We set the maximum sequence length to 512. The model is pre-
trained for 250k steps with batch size 2048, learning rate 3e-4, 10k warm-up
steps, and polynomial learning rate decay. We use the Adam optimizer for pre-
training. Using 8 Nvidia A100 GPUs (40G each), the training took eight days
for SciBART-base and twelve days for SciBART-large.

##  Appendix D Implementation details

###  D.1 Keyphrase Generation

For keyphrase generation with BART and T5, we use Huggingface Transformers and
train for 15 epochs with early stopping. We use learning rate 6e-5, polynomial
decay, batcsh size 64, and the AdamW optimizer. To fine-tune SciBART-base and
SciBART-large, we use the Translation task provided by
fairseq777https://github.com/facebookresearch/fairseq and train for 10 epochs.
We use learning rate 3e-5, polynomial decay, and the AdamW optimizer.

We perform a careful hyperparameter search over the learning rate, learning
rate schedule, batch size, and warm-up steps. The corresponding search spaces
are {1e-5, 3e-5, 6e-5, 1e-4, 3e-4}, {linear, polynomial}, {16, 32, 64, 128},
and {500, 1000, 2000, 4000}. The best hyperparameters are presented in Table
7. For DeSel, we tune the hyperparameters on the validation set of KP20k. the
search space for $\alpha$ is {0.5, 0.73, 0.78, 0.83, 0.88, 0.93, 0.98}.

The fine-tuning experiments are run on a local GPU server with RTX 2080 Ti
(11G each) and A6000 GPUs (48G each). We use gradient accumulation to achieve
the desired batch sizes.

Model | dropout | wdecay | optimizer | bsz | #epoch | #warm-up | lr | lr schedule
---|---|---|---|---|---|---|---|---
Seq2seq PLMs |  |  |
T5-small | 0.1 | 0.01 | AdamW | 64 | 25 | 2000 | 6e-5 | polynomial
T5-base | 0.1 | 0.01 | AdamW | 64 | 15 | 2000 | 6e-5 | polynomial
T5-large | 0.1 | 0.01 | AdamW | 64 | 5 | 1000 | 6e-5 | polynomial
T5-3B | 0.1 | 0.01 | AdamW | 64 | 5 | 1000 | 6e-5 | polynomial
BART-base | 0.1 | 0.01 | AdamW | 64 | 15 | 2000 | 6e-5 | polynomial
BART-large | 0.1 | 0.01 | AdamW | 64 | 10 | 2000 | 6e-5 | polynomial
In-domain Seq2seq PLMs |  |  |
SciBART-base | 0.1 | 0.01 | AdamW | 32 | 10 | 2000 | 3e-5 | polynomial
SciBART-large | 0.1 | 0.01 | AdamW | 64 | 10 | 2000 | 3e-5 | polynomial
Task-adaptive Seq2seq Models |  |  |
FLAN-T5-small | 0.1 | 0.01 | AdamW | 64 | 25 | 2000 | 6e-5 | polynomial
FLAN-T5-base | 0.1 | 0.01 | AdamW | 64 | 15 | 2000 | 6e-5 | polynomial
FLAN-T5-large | 0.1 | 0.01 | AdamW | 64 | 5 | 1000 | 6e-5 | polynomial
FLAN-T5-XL | 0.1 | 0.01 | AdamW | 64 | 5 | 1000 | 6e-5 | polynomial
KeyBART | 0.1 | 0.01 | AdamW | 64 | 15 | 2000 | 3e-5 | polynomial
SciBART-base + OAGKX | 0.1 | 0.01 | AdamW | 64 | 5 | 1000 | 1e-5 | polynomial
SciBART-large + OAGKX | 0.1 | 0.01 | AdamW | 64 | 5 | 1000 | 1e-5 | polynomial

Table 7: Hyperparameters for fine-tuning PLMs for keyphrase generation on
KP20k. The hyperparameters are determined using the loss on the KP20k
validation dataset. "wdecay" = weight decay, "bsz" = batch size, "#warm-up" =
the number of warm-up steps, "lr" = learning rate, "lr schedule" = learning
rate decay schedule. We use early stopping for all the models and use the
model with the lowest validation loss as the final model.

###  D.2 Baselines

For CopyTrans and SetTrans, we rerun with the original implementations to
measure their performance. We use the earliest version of KeyBART available at
https://zenodo.org/record/5784384#.Y0eToNLMJcA. For MPRank, we use the
implementation from pke888https://github.com/boudinfl/pke.

##  Appendix E Prompting GPT-3.5 for paraphrasing

To generate the paraphrased titles and abstracts used in section 4.4, we use
the following prompt to query gpt-3.5-turbo:

⬇

Paraphrase the following title and abstract for a scientific paper. Your
paraphrase should perfectly preserve the information in the original document
and should be as formal as the original text. Do not change the spelling or
case of the named entities in the original sentence. If the original entity is
all lower-cased, do not upper-case the first letter of these names in your
paraphrase.

Title: {original_title}

Abstract: {original_abstract}

Your Paraphrase:

Title:

We observe that the large language model can follow the specified format when
generating the paraphrased titles and abstracts. During the post-processing,
we split the titles and the abstract from the response and directly use them
for model testing. Figure 4 shows two examples of original and paraphrased
text. The paraphrased text has high quality, preserves the scientific writing
style, and retains most of the present keyphrases.

Original title: hybrid analytical modeling of pending cache hits , data
prefetching , and mshrs .
---
Original abstract: this article proposes techniques to predict the performance
impact of pending cache hits , hardware prefetching , and miss status holding
register resources on superscalar microprocessors using hybrid analytical
models . the proposed models focus on timeliness of pending hits and
prefetches and account for a limited number of mshrs . they improve modeling
accuracy of pending hits by 3.9 x and when modeling data prefetching , a
limited number of mshrs , or both , these techniques result in average errors
of 9.5 % to 17.8 % . the impact of non uniform dram memory latency is shown to
be approximated well by using a moving average of memory access latency .
Paraphrased title: a hybrid analytical model for pending cache hits , data
prefetching , and mshrs .
Paraphrased abstract: this scientific paper presents a novel approach to
predicting the performance impact of pending cache hits , hardware prefetching
, and miss status holding register resources on superscalar microprocessors .
the proposed hybrid analytical models focus on the timeliness of pending hits
and prefetches , while also accounting for a limited number of mshrs . by
utilizing these models , the accuracy of pending hit modeling is improved by
3.9 times . when modeling data prefetching , a limited number of mshrs , or
both , these techniques result in average errors ranging from 9.5 % to 17.8 %
. additionally , the paper demonstrates that the impact of non - uniform dram
memory latency can be approximated well by using a moving average of memory
access latency .
Original title: a variant of parallel plane sweep algorithm for multicore
systems .
Original abstract: parallel algorithms used in very large scale integration
physical design bring significant challenges for their efficient and effective
design and implementation . the rectangle intersection problem is a subset of
the plane sweep problem , a topic of computational geometry and a component in
design rule checking , parasitic resistance capacitance extraction , and mask
processing flows . a variant of a plane sweep algorithm that is embarrassingly
parallel and therefore easily scalable on multicore machines and clusters ,
while exceeding the best known parallel plane sweep algorithms on real world
tests , is presented in this letter .
Paraphrased title: a modified parallel plane sweep algorithm for multicore
systems .
Paraphrased abstract: the design and implementation of parallel algorithms for
physical design in very large scale integration pose significant challenges .
the rectangle intersection problem , which is a component in design rule
checking , parasitic resistance capacitance extraction , and mask processing
flows , is a subset of the plane sweep problem in computational geometry .
this letter presents a variant of the plane sweep algorithm that is
embarrassingly parallel and can be easily scaled on multicore machines and
clusters . the proposed algorithm outperforms the best known parallel plane
sweep algorithms on real - world tests .
Figure 4: Examples of documents paraphrased by gpt-3.5-turbo. We color the
present keyphrases in blue.

##  Appendix F One2one model for DeSel

To better capture $\Pr(s_{i}|\mathcal{X})$ and $\Pr(g_{j}|\mathcal{X})$ for
DeSel without the interference from other keyphrases co-occuring in the label,
we propose to train an "one2one" model that learns to generate a single
keyphrase given an input document. Starting from the KeyBART model, we fine-
tune on the KP20k training set for 0.5 epoch with learning rate 5e-5, batch
size 32, and the AdamW optimizer. We use a linear learning rate decay with
1000 warmup steps.

##  Appendix G All model testing results

We present the testing scores as well as their standard deviation in Table 8.
In addition to the models discussed in the main text, we also provide the
performance of CatSeq (Yuan et al., 2020), CatSeqTG-2RF1 (Chan et al., 2019),
ExHiRD-h (Chen et al., 2020) for further reference.

Method | |M| | KP20k | Inspec | Krapivin | NUS | SemEval
---|---|---|---|---|---|---
| $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$ | $P$ | $A$ | $Sem$
Trained from-scratch architectures |  |  |  |  |  |  |  |  |  |  |
CatSeq | 21M | 0.367 | 0.032 | N/A | 0.262 | 0.008 | N/A | 0.354 | 0.036 | N/A | 0.397 | 0.028 | N/A | 0.283 | 0.028 | N/A
ExHiRD-h | 22M |  0.3740 |  0.0250 | N/A |  0.2913 |  0.0162 | N/A |  0.3084 |  0.0334 | N/A | N/A | N/A | N/A |  0.28218 |  0.0216 | N/A
CopyTrans | 98M |  0.3762 |  0.0464 |  0.5621 |  0.3335 |  0.0231 |  0.5694 |  0.3657 |  0.0634 |  0.5473 |  0.4299 |  0.0449 |  0.5792 |  0.3218 |  0.0224 |  0.3771
SetTrans | 98M |  0.3912 |  0.0581 |  0.5852 |  0.3281 |  0.0301 |  0.5731 |  0.37511 |  0.0723 |  0.5600 |  0.44622 |  0.05517 |  0.5971 |  0.34214 |  0.0292 |  0.3963
PLM-based methods |  |  |  |  |  |  |  |  |  |  |
CorrKG† | 140M | 0.404 | 0.071 | N/A | 0.365 | 0.045 | N/A | N/A | N/A | N/A | 0.449 | 0.079 | N/A | 0.359 | 0.044 | N/A
BART-base | 140M |  0.3883 |  0.0422 |  0.5712 |  0.3237 |  0.0172 |  0.5615 |  0.3366 |  0.0496 |  0.5146 |  0.4248 |  0.0429 |  0.5813 |  0.32121 |  0.0212 |  0.3722
BART-large | 406M |  0.3922 |  0.0472 |  0.5751 |  0.3339 |  0.0244 |  0.5656 |  0.3473 |  0.0512 |  0.5175 |  0.43511 |  0.0489 |  0.5867 |  0.31116 |  0.0243 |  0.3816
KeyBART | 406M |  0.3982 |  0.0471 |  0.5761 |  0.3255 |  0.0232 |  0.5614 |  0.36514 |  0.0646 |  0.5332 |  0.43010 |  0.0557 |  0.5822 |  0.2894 |  0.0225 |  0.3654
SciBART-base | 124M |  0.3962 |  0.0524 |  0.5765 |  0.3288 |  0.0284 |  0.5624 |  0.32911 |  0.0548 |  0.5028 |  0.42114 |  0.0532 |  0.5728 |  0.3048 |  0.0221 |  0.3764
\+ TAPT | 124M |  0.4152 |  0.0521 |  0.5901 |  0.3306 |  0.0274 |  0.5732 |  0.3379 |  0.0577 |  0.5147 |  0.4246 |  0.0482 |  0.5792 |  0.3299 |  0.0240 |  0.3885
SciBART-large | 386M |  0.3964 |  0.0573 |  0.5872 |  0.32813 |  0.0262 |  0.5574 |  0.32912 |  0.0563 |  0.5037 |  0.42112 |  0.0507 |  0.5679 |  0.30412 |  0.0338 |  0.3829
\+ TAPT | 386M |  0.4260 |  0.0631 |  0.5971 |  0.3304 |  0.0301 |  0.5684 |  0.3476 |  0.0647 |  0.52010 |  0.44211 |  0.0555 |  0.5855 |  0.33319 |  0.0312 |  0.3869
T5-small | 60M |  0.3460 |  0.0190 |  0.5430 |  0.3543 |  0.0262 |  0.5722 |  0.2982 |  0.0261 |  0.4951 |  0.3980 |  0.0204 |  0.5622 |  0.3157 |  0.0131 |  0.3771
T5-base | 223M |  0.3880 |  0.0340 |  0.5700 |  0.3394 |  0.0203 |  0.5742 |  0.3501 |  0.0433 |  0.5241 |  0.4404 |  0.0512 |  0.5891 |  0.32613 |  0.0204 |  0.3833
T5-large | 770M |  0.3930 |  0.0350 |  0.5730 |  0.3433 |  0.0215 |  0.5731 |  0.3594 |  0.0456 |  0.5363 |  0.4385 |  0.0425 |  0.5875 |  0.3219 |  0.0202 |  0.3913
T5-3B | 3B |  0.4191 |  0.0461 |  0.5870 |  0.3195 |  0.0212 |  0.5591 |  0.3536 |  0.0481 |  0.5261 |  0.46010 |  0.0508 |  0.6004 |  0.3377 |  0.0274 |  0.3913
FLAN-T5-small | 60M |  0.3620 |  0.0240 |  0.5520 |  0.3541 |  0.0182 |  0.5690 |  0.3133 |  0.0240 |  0.4981 |  0.3981 |  0.0304 |  0.5681 |  0.3212 |  0.0171 |  0.3851
FLAN-T5-base | 223M |  0.3931 |  0.0350 |  0.5740 |  0.3584 |  0.0203 |  0.5774 |  0.3403 |  0.0494 |  0.5141 |  0.4360 |  0.0292 |  0.5860 |  0.3334 |  0.0172 |  0.3870
FLAN-T5-large | 770M |  0.4042 |  0.0360 |  0.5790 |  0.3471 |  0.0184 |  0.5731 |  0.3476 |  0.0412 |  0.5264 |  0.4414 |  0.0466 |  0.5980 |  0.32718 |  0.0212 |  0.3906
FLAN-T5-XL | 3B |  0.4271 |  0.0492 |  0.5921 |  0.3132 |  0.0241 |  0.5562 |  0.3561 |  0.0473 |  0.5174 |  0.4554 |  0.0542 |  0.5952 |  0.3325 |  0.0251 |  0.3922

Table 8: Testing results on all five datasets. $P$ and $A$ stand for $F1@M$
for present keyphrases and absent keyphrases. $Sem$ stands for $SemF1$. The
reported results are averaged across three runs with different random seeds.
The standard deviation of each entry is presented in the subscript. For
example, 23.56 means an average of 23.5 with a standard deviation of 0.6. We
omit the subscript for methods with a single run. †copied from Zhao et al.
(2022).