# Joint Embeddings of Hierarchical Categories and Entities

Yuezhang Li Ronghuo Zheng Tian Tian Zhiting Hu Rahul Iyer Katia Sycara
Carnegie Mellon University
5000 Forbes Ave.
Pittsburgh, PA 15213, USA
<EMAIL_ADDRESS>
###### Abstract

Due to the lack of structured knowledge applied in learning distributed
representation of categories, existing work cannot incorporate category
hierarchies into entity information. We propose a framework that embeds
entities and categories into a semantic space by integrating structured
knowledge and taxonomy hierarchy from large knowledge bases. The framework
allows to compute meaningful semantic relatedness between entities and
categories. Compared with the previous state of the art, our framework can
handle both single-word concepts and multiple-word concepts with superior
performance in concept categorization and semantic relatedness.

##  1 Introduction

Hierarchies, most commonly represented as Tree or DAG structures, provide a
natural way to categorize and locate knowledge in large knowledge bases (KBs).
For example, WordNet, Freebase and Wikipedia use hierarchical taxonomy to
organize entities into category hierarchies. These hierarchical categories
could benefit applications such as concept categorization [Rothenhäusler and
Schütze, 2009], document categorization [Gopal and Yang, 2013], object
categorization [Verma et al., 2012], and link prediction in knowlegde graphs
[Lin et al., 2015]. In all of these applications, it is essential to have a
good representation of categories and entities as well as a good semantic
relatedness measure.

Existing work does not use structured knowledge of KBs to embed
representations of entities and categories into a semantic space. Current
entity embedding methods cannot provide the relatedness measure between
entities and categories, although they successfully learn entity
representations and relatedness measure between entities [Hu et al., 2015].
Knowledge graph embedding methods [Wang et al., 2014, Lin et al., 2015] give
embeddings of entities and relations but lack category representations and
relatedness measure. Though taxonomy embedding methods [Weinberger and
Chapelle, 2009, Hwang and Sigal, 2014] learn category embeddings, they
primarily target documents classification not entity representations.

Figure 1: The organization of research elements comprising this paper.

In this paper, we propose two models to simultaneously learn entity and
category vectors from large-scale knowledge bases (KBs). They are Category
Embedding model and Hierarchical Category Embedding model. The Category
Embedding model (CE model) extends the entity embedding method of [Hu et al.,
2015] by using category information with entities to learn entity and category
embeddings. The Hierarchical Category Embedding model (HCE model) extends CE
model to integrate the hierarchical structure of categories. It considers all
ancestor categories of one entity. The final learned entity and category
vectors can capture meaningful semantic relatedness between entities and
categories.

We train the category and entity vectors on Wikipedia, and then evaluate our
methods from two applications: concept categorization [Baroni and Lenci, 2010]
and semantic relatedness [Finkelstein et al., 2001].

The organization of the research elements that comprise this paper,
summarizing the above discussion, is shown in Figure 1.

The main contributions of our paper are summarized as follows. First, we
incorporate category information into entity embeddings with the proposed CE
model to get entity and category embeddings simultaneously. Second, we add
category hierarchies to CE model and develop HCE model to enhance the
embeddings’ quality. Third, we propose a concept categorization method based
on nearest neighbor classification that avoids the issues arising from
disparity in the granularity of categories that plague traditional clustering
methods. Fourth, our model can handle multiple-word concepts/entities111In
this paper, concepts and entities denote same thing. such as “hot dog” and
distinguish it from “dog”. Overall, our model outperforms state-of-the-art
methods on both concept categorization and semantic relatedness.

##  2 Hierarchical Category Embedding

In order to find representations for categories and entities that can capture
their semantic relatedness, we use existing hierarchical categories and
entities labeled with these categories, and explore two methods: 1) Category
Embedding model (CE Model): it replaces the entities in the context with their
directly labeled categories to build categories’ context; 2) Hierarchical
Category Embedding (HCE Model): it further incorporates all ancestor
categories of the context entities to utilize the hierarchical information.

###  2.1 Category Embedding (CE) Model

Our category embedding (CE) model is based on the Skip-gram word embedding
model[Mikolov et al., 2013]. The skip-gram model aims at generating word
representations that are good at predicting _context_ words surrounding a
_target_ word in a sliding window. Previous work [Hu et al., 2015] extends the
entity’s context to the whole article that describes the entity and acquires a
set of entity pairs $\mathbf{D}=\\{(e_{t},e_{c})\\}$, where $e_{t}$ denotes
the target entity and $e_{c}$ denotes the context entity.

Our CE model extends those approaches by incorporating category information.
In KBs such as Wikipedia, category hierarchies are usually given as DAG or
tree structures, and entities are categorized into one or more categories as
leaves. Thus, in KBs, each entity $e_{t}$ is labeled with one or more
categories $(c_{1},c_{2},...,c_{k}),k\geq 1$ and described by an article
containing other context entities (see Data in Figure 1).

To learn embeddings of entities and categories simultaneously, we adopt a
method that incorporates the labeled categories into the entities when
predicting the context entities, similar to TWE-1 model [Liu et al., 2015].
For example, if $e_{t}$ is the _target_ entity in the document, its labeled
categories $(c_{1},c_{2},...,c_{k})$ would be combined with the entity $e_{t}$
to predict the context entities like $e_{c1}$ and $e_{c2}$ (see CE Model in
Figure 1). For each target-context entity pair $(e_{t},e_{c})$, the
probability of $e_{c}$ being context of $e_{t}$ is defined as the following
softmax:

$P(e_{c}|e_{t})=\frac{\exp{(e_{t}\cdot
e_{c})}}{\sum_{e\in\mathbf{E}}\exp{(e_{t}\cdot e)}},$ (1)

where $\mathbf{E}$ denotes the set of all entity vectors, and $\exp{}$ is the
exponential function. For convenience, here we abuse the notation of $e_{t}$
and $e_{c}$ to denote a target entity vector and a context entity vector
respectively.

Similar to TWE-1 model, We learn the target and context vectors by maximizing
the average log probability:

$\displaystyle L=$
$\displaystyle\frac{1}{|\mathbf{D}|}\sum_{(e_{c},e_{t})\in\mathbf{D}}\Big{[}\log
P(e_{c}|e_{t})$ $\displaystyle+\sum_{c_{i}\in\mathbf{C}(e_{t})}\log
P(e_{c}|c_{i})\Big{]},$ (2)

where $\mathbf{D}$ is the set of all entity pairs and we abuse the notation of
$c_{i}$ to denote a category vector, and $\mathbf{C}(e_{t})$ denotes the
categories of entity $e_{t}$.

Figure 2: Category and entity embedding visualization of the DOTA-all data set
(see Section 4.1.1). We use t-SNE [Van der Maaten and Hinton, 2008] algorithms
to map vectors into a 2-dimensional space. Labels with the same color are
entities belonging to the same category. Labels surrounded by a box are
categories vectors.

###  2.2 Hierarchical Category Embedding(HCE) Model

From the design above, we can get the embeddings of all categories and
entities in KBs without capturing the semantics of hierarchical structure of
categories. In a category hierarchy, the categories at lower layers will cover
fewer but more specific concepts than categories at upper layers. To capture
this feature, we extend the CE model to further incorporate the ancestor
categories of the target entity when predicting the context entities (see HCE
Model in Figure 1). If a category is near an entity, its ancestor categories
would also be close to that entity. On the other hand, an increasing distance
of the category from the target entity would decrease the power of that
category in predicting context entities. Therefore, given the target entity
$e_{t}$ and the context entity $e_{c}$, the objective is to maximize the
following weighted average log probability:

$\displaystyle L=$
$\displaystyle\frac{1}{|\mathbf{D}|}\sum_{(e_{c},e_{t})\in\mathbf{D}}\Big{[}\log
P(e_{c}|e_{t})$ $\displaystyle+\sum_{c_{i}\in\mathbf{A}(e_{t})}w_{i}\log
P(e_{c}|c_{i})\Big{]},$ (3)

where $\mathbf{A}(e_{t})$ represents the set of ancestor categories of entity
$e_{t}$, and $w_{i}$ is the weight of each category in predicting the context
entity. To implement the intuition that a category is more relevant to its
closer ancestor, for example, “NBC Mystery Movies” is more relevant to
“Mystery Movies” than “Entertainment”, we set
$w_{i}\propto\frac{1}{l(c_{c},c_{i})}$ where $l(c_{c},c_{i})$ denotes the
average number of steps going down from category $c_{i}$ to category $c_{c}$,
and it is constrained with $\sum_{i}w_{i}=1$.

Figure 2 presents the results of our HCE model for DOTA-all data set (see
Section 4.1.1). The visualization shows that our embedding method is able to
clearly separate entities into distinct categories.

###  2.3 Learning

Learning CE and HCE models follows the optimization scheme of skip-gram model
[Mikolov et al., 2013]. We use negative sampling to reformulate the objective
function, which is then optimized through stochastic gradient descent (SGD).

Specifically, the likelihood of each context entity of a target entity is
defined with the softmax function in Eq. 1, which iterates over all entities.
Thus, it is computationally intractable. We apply the standard negative
sampling technique to transform the objective function in equation (2.2) to
equation (2.3) below and then optimize it through SGD:

$\displaystyle L=$
$\displaystyle\sum_{(e_{c},e_{t})\in\mathbf{D}}\Big{[}\log\sigma(e_{c}\cdot
e_{t})+\sum_{c_{i}\in\mathbf{A}(e_{t})}w_{i}\log\sigma(e_{c}\cdot
c_{i})\Big{]}$
$\displaystyle+\sum_{(e_{c}^{\prime},e_{t})\in\mathbf{D^{\prime}}}\Big{[}\log\sigma(-e_{c}^{\prime}\cdot
e_{t})$
$\displaystyle+\sum_{c_{i}\in\mathbf{A}(e_{t})}w_{i}\log\sigma(-e_{c}^{\prime}\cdot
c_{i})\Big{]},$ (4)

where $\mathbf{D^{\prime}}$ is the set of negative sample pairs and
$\sigma(x)=1/(1+\exp(-x))$ is the sigmoid function.

##  3 Applications

We test the quality of our category and entity embedding with two different
applications: concept categorization and semantic relatedness.

###  3.1 Concept Categorization

Concept222In this paper, concept and entity denote the same thing.
categorization, also known as _concept learning_ or _noun categorization_ , is
a process of assigning a concept to one candidate category, given a set of
concepts and candidate categories. Traditionally, concept categorization is
achieved by concept clustering due to the lack of category representations.
Since our model can generate representations of categories, we propose a new
method of using nearest neighbor (NN) classification to directly categorize
each concept to a certain category.

####  3.1.1 Concept Clustering

The concept clustering is defined as: given a set of single-word concepts like
_dog, cat, apple_ and the corresponding gold standard categorizations, apply a
word space model to project all the concepts to a semantic space and perform
clustering. The clustering results can be evaluated by comparing with the gold
standard categorizations.

Previous methods [Almuhareb and Poesio, 2004, Almuhareb and Poesio, 2005]
managed to learn the properties and attributes of a concept; for example, the
concept dog has attributes of (dog color) and (dog size) and the properties of
(dog brown) and (dog small). By representing these attributes and properties
in a high-dimensional space, one can cluster concepts based on common
attributes and properties.

####  3.1.2 Nearest Neighbor (NN) Classification

Although the previous methods described above can generate vectors carefully
designed for capturing relations and attributes, the number of vector
dimensions can be very large for some methods: from 10,000+ to 1,000,000+,
e.g., [Almuhareb and Poesio, 2005, Rothenhäusler and Schütze, 2009]. Due to
the large dimensionality, the applicable clustering methods are restricted to
the ones that can scale to such high dimensions333[Rothenhäusler and Schütze,
2009] use CLUTO [Karypis, 2002], a clustering toolkit optimized to cluster
large-scale data in reasonable time, as their standard measurements. . Other
methods such as word embedding [Mikolov et al., 2013] may need lower
dimensionality vectors but suffer from _granularity problems_. Therefore, we
propose an alternative method, namely nearest neighbor (NN) classification,
and evaluate comparative trade-offs (see Table 3).

Using NN classification, we categorize concepts by directly comparing concept
vectors with candidate category vectors. Precisely, given a set of concepts
$\mathbf{E}$ and a set of candidate categories $\mathbf{C}$, we convert all
concepts to concept vectors and all candidate categories to category vectors.
Then we use the equation
$c=\operatornamewithlimits{argmin}_{c_{i}\in\mathbf{C}}||c_{i}-e||$ to assign
the concept vector $e$ with category $c$. Note that in this paper, concept and
entity denote the same thing so concept vector is exactly the same as entity
vector.

####  3.1.3 Evaluation Metrics

Since purity works as a standard evaluation metric for clustering
[Rothenhäusler and Schütze, 2009], to compare our model with the concept
clustering, we also use purity to measure our model’s performance. Generally,
purity is defined as:

$\text{purity}(\mathbf{\Omega},\mathbf{G})=\frac{1}{n}\sum_{k}{\max_{j}{|\omega_{k}\cap
g_{j}}|},$ (5)

where $\mathbf{\Omega}$ denotes a clustering solution of $n$ clusters,
$\mathbf{G}$ is a set of gold standard classes, $\omega_{k}$ represents the
set of labels in a cluster and $g_{j}$ is the set of labels in a class. A
higher purity indicates better model performance.

###  3.2 Semantic Relatedness

We also evaluate the entity and category embeddings by semantic relatedness.
Semantic relatedness measure is a process of assigning one relatedness score
for one word pair. We use a set of standard semantic benchmarks. Those
benchmarks consist of word pairs that have manually rated scores 0-10 for
semantic relatedness. The model performance is assessed by calculating the
correlation between scores generated by the model and the average scores given
by human subjects.

We use the Spearman’s rank correlation coefficient [pirie1988spearman] of
human assigned scores and system assigned scores to evaluate our result. Note
that scores given by humans are not necessarily the gold standard because of
the differences among humans and the difficulty of giving a clear definition
of word similarity. However, a good system should agree with humans to some
extent and thus should have a relatively high correlation coefficient
(Although when the coefficient is high enough, a higher coefficient would not
necessarily reflect the better model). The Spearman correlation score is
defined by the Pearson correlation coefficient between the ranked variables.
For scores $X_{i}$, $Y_{i}$ that are in sample size $n$, we use $x_{i}$ and
$y_{i}$ to denote the rankings of scores $X_{i}$, $Y_{i}$. The Spearman
correlation coefficient is defined as:

$\rho=1-\frac{6\sum_{i=1}^{n}(x_{i}-y_{i})^{2}}{n(n^{2}-1)}.$ (6)

##  4 Experiments

In the experiments, we use the dataset collected from Wikipedia on Dec. 1,
2015444https://dumps.wikimedia.org/wikidatawiki/20151201/ as the training
data. We preprocess the category hierarchy by pruning administrative
categories and deleting bottom-up edges to construct a DAG. The final version
of data contains 5,373,165 entities and 793,856 categories organized as a DAG
with a maximum depth of 18. The root category is “main topic classifications”.
We train category and entity vectors in various dimensions of 100, 200, 250,
300, 400, 500, with batch size $B=500$ and negative sample size $k=10$.

With the training dataset defined above, we conduct experiments on two
applications: concept categorization and semantic relatedness.

###  4.1 Concept Categorization

In this section, we first introduce datasets applied in concept
categorization, and then show baselines followed by experimental results.

####  4.1.1 Datasets

There are two datasets used in this experiment. The first one is the Battig
test set introduced by [Baroni and Lenci, 2010], which includes 83 concepts
from 10 categories. The Battig test set only contains single-word concepts
without any multiple-word concepts (e.g., “table tennis”). Hence, using this
dataset restricts the power of concept categorization to single-word level. We
use this dataset because it has been used as a benchmark for most previous
approaches for concept categorization.

Due to the limitations of the Battig test set, we construct a new entity
categorization dataset DOTA (Dataset Of enTity cAtegorization) with 450
entities categorized into 15 categories (refer to Appendix A). All the
categories and entities are extracted from Wikipedia, so the resulting dataset
does not necessarily contains only single-word entities. Thus, the dataset can
be split into two parts, DOTA-single that contains 300 single-word entities
categorized into 15 categories and DOTA-mult that contains 150 multiple-word
entities categorized into the same 15 categories. We design the DOTA dataset
based on the following principles:

  * •

Coverage vs Granularity: Firstly, the dataset should cover at least one
category of Wikipedia’s main topics including “Culture”, “Geography”,
“Health”, “Mathematics”, “Nature”, “People”, “Philosophy”, “Religion”,
“Society” and “Technology”. Secondly, categories should be in different
granularity, from large categories (e.g.,“philosophy”) to small categories
(e.g., “dogs”). Large categories are ones that are located within 5 layers
away from the root, medium categories are 6-10 layers away from the root,
while small categories have distance of 11-18 to the root. Our dataset
consists of 1/3 large categories, 1/3 medium categories, and 1/3 small
categories.

  * •

Single-Words vs Multiple-Words: Previous concept categorization datasets only
contain single-words. However, some concepts are multiple-words and cannot be
simply represented by single-words. For example, the concept “hot dog” is very
different from the concept “dog”. Word-level embedding cannot solve this
problem without phrase recognition, while entity-level embedding can solve it
naturally. Therefore, we make each category of the dataset contain 10
multiple-word entities and 20 single-word entities.

####  4.1.2 Baselines

Word Embedding (WE) trained with neutral networks on large corpus provides a
way to map a given text to a semantic space. We compare our entity and
category embeddings with two word embeddings.

  * •

WEMikolov[Mikolov et al., 2013]: [Baroni et al., 2014] conducted thorough
experiments on word counts and predictive based methods on word
representation. Their experimental results show that Mikolov’s word embedding
achieves state-of-the-art results in concept categorization. We trained word
embeddings with Mikolov’s word2vec
toolkit555https://code.google.com/archive/p/word2vec/ on the same Wikipedia
corpus as ours (1.7 million tokens) and then applied the Skip-gram model with
negative sample size of 10 and window size of 5 to vector dimensionality of
100, 200, 250, 300, 400, 500 respectively. The best results of various
parameter settings are reported in Table 1 and Table 2.

  * •

WESenna [Collobert et al., 2011]: We downloaded this 50-dimension word
embedding666http://ronan.collobert.com/senna/ trained on Wikipedia over 2
months. We use this embedding as a baseline because it is also trained on
Wikipedia.

To evaluate the advantage of utilizing category hierarchy in training entity
and category embedding, we also compare our Hierarchical Category Embedding
(HCE) model with our Category Embedding (CE) model that has no hierarchical
information.

####  4.1.3 Results

In the experiments, we used scikit-learn [Pedregosa et al., 2011] to perform
clustering. We tested k-means and hierarchical clustering with different
distance metrics (euclidean, cosine) and linkage criterion (ward, complete,
average). We reported the best result across different clustering parameters
in each experiment.

Table. 1 shows the experimental results of the concept clustering method. It
is clear that hierarchical category embedding (HCE) model outperforms other
methods in all datasets. For single-word entity categorization in Battig and
DOTA-single, our HCE model gives a purity of 89% and 92% respectively; while
for multiple-word entity categorization in DOTA-mult and DOTA-all, the
corresponding purities are 91% and 89% (higher is better).

| Battig | DOTA-single | DOTA-mult | DOTA-all
---|---|---|---|---
WESenna | 0.74 | 0.61 | 0.43 | 0.45
WEMikolov | 0.86 | 0.83 | 0.73 | 0.78
CE (Ours) | 0.84 | 0.86 | 0.83 | 0.85
HCE (Ours) | 0.89 | 0.92 | 0.88 | 0.89

Table 1: Purity of concept clustering method with different embeddings on
Battig and DOTA datasets. The numbers given are the best scores achieved under
different settings.

The excellent performance of our HCE model on DOTA-mult and DOTA-all lies in
the fact that it can naturally produce multiple-word entities. Since Mikolov’s
word embeddings can also capture common phrases, it performs well on DOTA-mult
dataset with a purity of 73%. However, the Senna word embeddings only contain
single-words. To get the embeddings of multiple-word, we use the mean word
vectors to denote multiple-word embeddings. As the meaning of a multiple-word
is not simply the aggregation of the meaning of the words it contains, the
purity of using Senna drops dramatically from 61% on DOTA-single to 43% only
on DOTA-mult.

Table.2 shows the experimental results of the nearest neighbor (NN)
classification method. The results indicate the feasibility of using category
vectors to directly predict the concept categories without clustering
entities. By changing the concept categorization method from concept
clustering to nearest neighbor classification, our model still achieves a
purity of 87% on Battig and around 90% on DOTA.

| Battig | DOTA-single | DOTA-mult | DOTA-all
---|---|---|---|---
WESenna | 0.44 | 0.62 | 0.32 | 0.33
WEMikolov | 0.74 | 0.74 | 0.67 | 0.69
CE (Ours) | 0.79 | 0.89 | 0.85 | 0.88
HCE (Ours) | 0.87 | 0.93 | 0.91 | 0.91

Table 2: Purity of nearest neighbor (NN) classification method with different
embeddings on Battig and DOTA datasets. The numbers given are the best scores
achieved under different settings.

Table.3 presents the best prediction results produced by our HCE model with
two different methods, namely concept clustering and nearest neighbor (NN)
classification.

Category |  Misclassified entities using clustering |  Misclassified entities using NN classifier
---|---|---
beverages |  pear |  pear
sports |  - |  -
emotions |  - |  jumping
weather |  - |  racing, snowboarding, ship, airplane, rocket, wetland, koi, tetra, flatfish, whisky
landforms |  flood, sandstone |  perch
trees |  mead |  -
algebra |  trigonometry, circle, square, polyhedron, surface, sphere, cube, icosahedron, hemipolyhedron, digon, midpoint, centroid, octadecagon, curvature, curve, zonohedron, cevian, orthant, cuboctahedron, midsphere |  curve, curvature
geometry |  - |  multiplication
fish |  - |  -
dogs |  - |  cider, cod
music |  - |  -
politics |  idealism, ethics |  idealism, ethics
philosophy |  ideology |  ideology
linguistics |  - |  -
vehicles |  - |  cycling
Table 3: Best prediction results of HCE by concept clustering and nearest
neighbor classification on DOTA-single dataset.

As for the concept clustering approach, the general performance is very good
except for one extreme case that many entities of _geometry_ category are
misclassified into _algebra_ category. Refer to Figure 2, it is clear that
because _algebra_ category overlaps with _geometry_ category, the concept
clustering clusters them together. This phenomenon is caused by the difference
in granularity of categories.

As for the nearest neighbor (NN) classification approach, we found an
interesting phenomenon – several entities from other categories are
misclassified into _weather_ category. Referring to the work of [Radovanović
et al., 2010, marcobaroni2015hubness], this phenomenon is called _hubness_ ,
which depicts that some vectors (“hubs”) tend to appear in the top neighbor
lists of many test items in high-dimensional space. In our case, the category
vector “weather” tend to be a “hub” vector.

Based on the analysis of these two methods above, we can conclude that the NN
classifier can address granularity problem but suffers from the _hubness_
problem, while concept clustering can deal with _hubness_ but has the
granularity problem.

###  4.2 Semantic relatedness

We now introduce the datesets and baselines for measuring semantic relatedness
and show the experimental results.

####  4.2.1 Datasets

We use a set of standard datasets and preprocess them to fit our method. Our
method requires mapping the words to corresponding Wikipedia entities or
categories. For example, we map the word “cat” to the Wikipedia entity “cat”
and the word “equipment” to the Wikipedia category “equipment”. Without loss
of generality, we first match a word to a Wikipedia entity based on lexical
similarity. If there is no matched entity, we match it to a Wikipedia
category. However, it is difficult for this approah to map some words to a
specific Wikipedia entity or category because of two reasons:

  * •

Wikipedia is a knowledge base that organizes categories and concepts, but some
words like adjectives (e.g. smart/stupid) cannot be mapped to any Wikipedia
entity or category. Moreover, our entity based approach cannot capture word
pairs with lexical differences such as “swim/swimming”. We thus eliminate
these kinds of words.

  * •

Some words are ambiguous so they have multiple corresponding entities in
Wikipedia. For example, the word “doctor” can work as entity “Doctor(title)”
that means the holder of an accredited doctoral graduate degree, and it can
also be entity “Physician” that means a professional who practices medicine.
Therefore, we simply discard all these ambiguous words.

Using the filtered datasets does not affect the fairness of our comparison,
since we conduct all the other baselines on the same subsets.

WS: The WordSim353 dataset is introduced by [Finkelstein et al., 2001]. It
contains 353 pairs of words and their semantic relatedness scores assigned by
13 to 16 human subjects. The work of [Agirre et al., 2009] splits the WS-353
dataset into two separate subsets: similarity subset (WSS) and relatedness
subset (WSR). The former one contains tighter taxonomy relations (e.g.,
plane/car, student/professor) whereas the latter contains tighter topical
relations (e.g., Jerusalem/Israel, OPEC/country). After data preprocessing,
184 pairs of words remain in WS-353, 105 pairs in WSS-353 (202 pairs
originally), and 125 pairs in WSR-353 (251 pairs originally).

MEN: The work of [Bruni et al., 2014] constructed this dataset with 3000 word
pairs that have semantic relatedness scores obtained by crowd-sourcing. After
data preprocessing, there are 1508 pairs of words left.

RG: A classic dataset contains 65 word pairs introduced by [Rubenstein and
Goodenough, 1965]. After data preprocessing, 28 pairs of words are left.

####  4.2.2 Baselines

We compare our methods with some state-of-the-art methods below.

WN+NR: In [Radhakrishnan and Varma, 2013], word similarity measure is derived
from Wikipedia category names integrated with WordNet similarity measure by
performing regression using a Support Vector Machine. WN+NR1 and WN+NR2 are
two of the best models reported in their paper.

WEMikolov: As described in Section 4.1.2, we trained word embedding using
Mikolov’s word2vec toolkit777https://code.google.com/archive/p/word2vec/ on
the same Wikipedia corpus (1.7 million tokens) as ours to make them
comparable. We use Skip-gram model with negative sample size of 10 and window
size of 5 , with vector dimensionality of 100, 200, 250, 300, 400, 500. We
report the best results obtained from various parameter settings in Table 4.

WESenna [Collobert et al., 2011]: We downloaded this 50-dimension word
embedding888http://ronan.collobert.com/senna/ trained on Wikipedia over 2
months.

####  4.2.3 Results

Table 4 shows the experimental results of the semantic relatedness tasks. We
can see that our HCE model yields best results of 57%, 69%, and 83% on WS, MEN
and RG datasets. This performance suggests that entity and category embeddings
can be used as an indicator of semantic relatedness between words. For WSS
dataset, the result of our method is comparable with WN+NR1 method, which
integrates WordNet similarity measures with the normalized representation of
category names. We also found that Mikolov’s word embedding performs better
than our method on WSR dataset, but performs worse than our method on WSS
dataset. The reason may be that the WSR dataset concentrates on topical
related words rather than taxonomy related words, and our method can better
capture taxonomy relationship than topic relationship.

| WS | WSS | WSR | MEN | RG
---|---|---|---|---|---
WN+NR1 | 0.55 | 0.67 | 0.46 | 0.53 | 0.40
WN+NR2 | 0.43 | 0.53 | 0.32 | 0.47 | 0.49
WEMikolov | 0.55 | 0.64 | 0.59 | 0.66 | 0.75
WESenna | 0.47 | 0.59 | 0.38 | 0.57 | 0.47
CE (Ours) | 0.55 | 0.63 | 0.49 | 0.64 | 0.78
HCE (Ours) | 0.57 | 0.67 | 0.53 | 0.69 | 0.83

Table 4: Correlation with human judgements on semantic relatedness evaluation
datasets.

##  5 Conclusion

In this paper, we proposed a framework to learn entity and category embeddings
to capture semantic relatedness between entities and categories. This
framework can incorporate taxonomy hierarchy from large scale knowledge bases.
Experiments on both concept categorization and semantic relatedness show that
our approach outperforms state of the art approaches. In the future work, we
aim at applying our method to more applications such as hierarchical document
classification.

## References

  * [Agirre et al., 2009]  Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa.  2009\.  A study on similarity and relatedness using distributional and wordnet-based approaches.  In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 19–27. Association for Computational Linguistics.
  * [Almuhareb and Poesio, 2004]  Abdulrahman Almuhareb and Massimo Poesio.  2004\.  Attribute-based and value-based clustering: An evaluation.  In EMNLP, volume 4, pages 158–165.
  * [Almuhareb and Poesio, 2005]  Abdulrahman Almuhareb and Massimo Poesio.  2005\.  Concept learning and categorization from the web.  In Proc. of CogSci.
  * [Baroni and Lenci, 2010]  Marco Baroni and Alessandro Lenci.  2010\.  Distributional memory: A general framework for corpus-based semantics.  Computational Linguistics, 36(4):673–721.
  * [Baroni et al., 2014]  Marco Baroni, Georgiana Dinu, and Germán Kruszewski.  2014\.  Don’t count, predict! a systematic comparison of context-counting vs. context-predicting semantic vectors.  In ACL (1), pages 238–247.
  * [Bruni et al., 2014]  Elia Bruni, Nam-Khanh Tran, and Marco Baroni.  2014\.  Multimodal distributional semantics.  J. Artif. Intell. Res.(JAIR), 49(1-47).
  * [Collobert et al., 2011]  Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.  2011\.  Natural language processing (almost) from scratch.  The Journal of Machine Learning Research, 12:2493–2537.
  * [Finkelstein et al., 2001]  Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin.  2001\.  Placing search in context: The concept revisited.  In Proceedings of the 10th international conference on World Wide Web, pages 406–414. ACM.
  * [Gopal and Yang, 2013]  Siddharth Gopal and Yiming Yang.  2013\.  Recursive regularization for large-scale classification with hierarchical and graphical dependencies.  In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 257–265. ACM.
  * [Hu et al., 2015]  Zhiting Hu, Poyao Huang, Yuntian Deng, Yingkai Gao, and Eric P Xing.  2015\.  Entity hierarchy embedding.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP), volume 1, pages 1292–1300.
  * [Hwang and Sigal, 2014]  Sung Ju Hwang and Leonid Sigal.  2014\.  A unified semantic embedding: Relating taxonomies and attributes.  In Advances in Neural Information Processing Systems, pages 271–279.
  * [Karypis, 2002]  George Karypis.  2002\.  Cluto-a clustering toolkit.  Technical report, DTIC Document.
  * [Lin et al., 2015]  Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu.  2015\.  Learning entity and relation embeddings for knowledge graph completion.  In AAAI, pages 2181–2187.
  * [Liu et al., 2015]  Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  2015\.  Topical word embeddings.
  * [Mikolov et al., 2013]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * [Pedregosa et al., 2011]  Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al.  2011\.  Scikit-learn: Machine learning in python.  The Journal of Machine Learning Research, 12:2825–2830.
  * [Radhakrishnan and Varma, 2013]  Priya Radhakrishnan and Vasudeva Varma.  2013\.  Extracting semantic knowledge from wikipedia category names.  In Proceedings of the 2013 workshop on Automated knowledge base construction, pages 109–114. ACM.
  * [Radovanović et al., 2010]  Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović.  2010\.  Hubs in space: Popular nearest neighbors in high-dimensional data.  The Journal of Machine Learning Research, 11:2487–2531.
  * [Rothenhäusler and Schütze, 2009]  Klaus Rothenhäusler and Hinrich Schütze.  2009\.  Unsupervised classification with dependency based word spaces.  In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics, pages 17–24. Association for Computational Linguistics.
  * [Rubenstein and Goodenough, 1965]  Herbert Rubenstein and John B Goodenough.  1965\.  Contextual correlates of synonymy.  Communications of the ACM, 8(10):627–633.
  * [Van der Maaten and Hinton, 2008]  Laurens Van der Maaten and Geoffrey Hinton.  2008\.  Visualizing data using t-sne.  Journal of Machine Learning Research, 9(2579-2605):85.
  * [Verma et al., 2012]  Nakul Verma, Dhruv Mahajan, Sundararajan Sellamanickam, and Vinod Nair.  2012\.  Learning hierarchical similarity metrics.  In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2280–2287. IEEE.
  * [Wang et al., 2014]  Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen.  2014\.  Knowledge graph and text jointly embedding.  In EMNLP, pages 1591–1601. Citeseer.
  * [Weinberger and Chapelle, 2009]  Kilian Q Weinberger and Olivier Chapelle.  2009\.  Large margin taxonomy embedding for document categorization.  In Advances in Neural Information Processing Systems, pages 1737–1744.

##  Appendix A The DOTA dataset: 300 single-word entities and 150 multi-word
entities from 15 Wikipedia Categories

Category |  Entities
---|---
beverages |  juice, beer, milk, coffee, tea, cocktail, wine, liqueur, sake, vodka, mead, sherry, brandy, gin, rum, latte, whisky, cider, gose, rompope, orange juice, masala chai, green tea, black tea, herbal tea, coconut milk, corn syrup, soy milk, rose water, hyeonmi cha
sports |  bowling, football, aerobics, hockey, karate, korfball, handball, floorball, skiing, cycling, racing, softball, shooting, netball, snooker, powerlifting, jumping, wallball, volleyball, snowboarding, table tennis, floor hockey, olympic sports, wheelchair basketball, crab soccer, indoor soccer, table football, roller skating, vert skating, penny football
emotions |  love, anxiety, empathy, fear, envy, loneliness, shame, anger, annoyance, happiness, jealousy, apathy, resentment, frustration, belongingness, sympathy, pain, worry, hostility, sadness, broken heart, panic disorder, sexual desire, falling in love, emotional conflict, learned helplessness, chronic stress, anxiety sensitivity, mental breakdown, bike rage
weather |  cloud, wind, thunderstorm, fog, snow, wave, blizzard, sunlight, tide, virga, lightning, cyclone, whirlwind, sunset, dust, frost, flood, thunder, supercooling, fahrenheit, acid rain, rain and snow mixed, cumulus cloud, winter storm, blowing snow, geomagnetic storm, blood rain, fire whirl, pulse storm, dirty thunderstorm
landforms |  lake, waterfall, stream, river, wetland, marsh, valley, pond, sandstone, mountain, cave, swamp, ridge, plateau, cliff, grassland, glacier, hill, bay, island, glacial lake, drainage basin, river delta, stream bed, vernal pool, salt marsh, proglacial lake, mud volcano, pit crater, lava lake
trees |  wood, oak, pine, evergreen, willow, vine, shrub, birch, beech, maple, pear, fir, pinales, lauraceae, sorbus, buxus, acacia, rhamnaceae, fagales, sycamore, alhambra creek, alstonia boonei, atlantic hazelwood, bee tree, blood banana, datun sahib, druid oak, new year tree, heart pine, fan palm
algebra |  addition, multiplication, exponentiation, tetration, polynomial, calculus, permutation, subgroup, integer, monomial, bijection, homomorphism, determinant, sequence, permanent, homotopy, subset, factorization, associativity, commutativity, real number, abstract algebra, convex set, prime number, complex analysis, natural number, complex number, lie algebra, identity matrix, set theory
geometry |  trigonometry, circle, square, polyhedron, surface, sphere, cube, icosahedron, hemipolyhedron, digon, midpoint, centroid, octadecagon, curvature, curve, zonohedron, cevian, orthant, cuboctahedron, midsphere, regular polygon, uniform star polyhedron, isogonal figure, icosahedral symmetry, hexagonal bipyramid, snub polyhedron, homothetic center, geometric shape, bragg plane, affine plane
fish |  goldfish, gourami, koi, cobitidae, tetra, goby, danio, wrasse, acanthuridae, anchovy, carp, catfish, cod, eel, flatfish, perch, pollock, salmon, triggerfish, herring, cave catfish, coachwhip ray, dwarf cichlid, moray eel, coastal fish, scissortail rasbora, flagtail pipefish, armoured catfish, hawaiian flagtail, pelagic fish
dogs |  spaniel, foxhound, bloodhound, beagle, pekingese, weimaraner, collie, terrier, poodle, puppy, otterhound, labradoodle, puggle, eurasier, drever, brindle, schnoodle, bandog, leonberger, cockapoo, golden retriever, tibetan terrier, bull terrier, welsh springer spaniel, hunting dog, bearded collie, picardy spaniel, afghan hound, brittany dog, redbone coonhound
music |  jazz, blues, song, choir, opera, rhythm, lyrics, melody, harmony, concert, comedy, violin, drum, piano, drama, cello, composer, musician, drummer, pianist, hip hop, classical music, electronic music, folk music, dance music, musical instrument, disc jockey, popular music, sheet music, vocal music
politics |  democracy, law, government, liberalism, justice, policy, rights, utilitarianism, election, capitalism, ideology, egalitarianism, debate, regime, globalism, authoritarianism, monarchism, anarchism, communism, individualism, freedom of speech, political science, public policy, civil society, international law, social contract, election law, social justice, global justice, group conflict
philosophy |  ethics, logic, ontology, aristotle, plato, rationalism, platonism, relativism, existence, truth, positivism, metalogic, subjectivism, idealism, materialism, aesthetics, probabilism, monism, truth, existence, western philosophy, contemporary philosophy, cognitive science, logical truth, ancient philosophy, universal mind, visual space, impossible world, theoretical philosophy, internal measurement
linguistics |  syntax, grammar, semantics, lexicon, speech, phonetics, vocabulary, phoneme, lexicography, language, pragmatics, orthography, terminology, pronoun, noun, verb, pronunciation, lexicology, metalinguistics, paleolinguistics, language death, historical linguistics, dependency grammar, noun phrase, comparative linguistics, word formation, cognitive semantics, syntactic structures, auxiliary verb, computational semantics
vehicles |  truck, car, aircraft, minibus, motorcycle, microvan, bicycle, tractor, microcar, van, ship, helicopter, airplane, towing, velomobile, rocket, train, bus, gyrocar, cruiser, container ship, school bus, road train, tow truck, audi a6, garbage truck, hydrogen tank, light truck, compressed air car, police car


# BoschAI @ PLABA 2023: Leveraging Edit Operations in End-to-End Neural
Sentence Simplification

Valentin Knappich1,2 Simon Razniewski1 Annemarie Friedrich2

1Bosch Center for Artificial Intelligence, Renningen, Germany 2University of
Augsburg, Germany
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Automatic simplification can help laypeople to comprehend complex scientific
text. Language models are frequently applied to this task by translating from
complex to simple language. In this paper, we describe our system based on
Llama 2, which ranked first in the PLABA shared task addressing the
simplification of biomedical text. We find that the large portion of shared
tokens between input and output leads to weak training signals and
conservatively editing models. To mitigate these issues, we propose sentence-
level and token-level loss weights. They give higher weight to modified
tokens, indicated by edit distance and edit operations, respectively. We
conduct an empirical evaluation on the PLABA dataset and find that both
approaches lead to simplifications closer to those created by human annotators
($+1.8\%$ / $+3.5\%$ SARI), simpler language ($-1$ / $-1.1$ FKGL) and more
edits ($1.6x$ / $1.8x$ edit distance) compared to the same model fine-tuned
with standard cross entropy. We furthermore show that the hyperparameter
$\lambda$ in token-level loss weights can be used to control the edit distance
and the simplicity level (FKGL).

##  1 Introduction

Figure 1: Example sentence pair from the PLABA dataset. Red background
indicates source tokens that were removed during simplification, green
background indicates target tokens that were added, white background indicates
tokens that remained unchanged.

Scientific texts are increasingly available on the internet, and constitute a
giant collection of knowledge Ronzano and Saggion (2016). However, they are
also difficult to comprehend for laypeople, due to complex sentence structure
and terminology Gopen and Swan (1990). Automatic text simplification systems
alleviate this issue by transforming complex text into easier-to-understand
simplifications. Such simplifications ideally contain the same essential
information, but convey it in a simpler, more comprehensible fashion Ondov et
al. (2022).

Text simplification is especially important in scientific domains that are
directly relevant to the general public, like the biomedical sciences. To
facilitate work on simplification of biomedical text, the PLABA dataset was
introduced by Attal et al. (2023). It contains sentence-aligned
simplifications for the abstracts of 750 biomedical papers. This paper reports
our submission to the PLABA 2023 shared
task111https://bionlp.nlm.nih.gov/plaba2023/ organized by
NIST222https://www.nist.gov/, the results of which are presented in Table 1.

Submission | SARI | Simp. avg | Acc. avg
---|---|---|---
Bee_Man | 40.59 | 82.74 | 75.00
MasonNLP | 39.51 | 91.28 | 92.52
PLABA_base | 35.97 | 93.53 | 94.87
Ours | 44.65 | 88.86 | 94.87
Table 1: Shared Task Results on the blind test set. SARI is automatically
evaluated against 4 references. Simp. avg and Acc. avg are average human
evaluation scores across multiple criteria concerning simplicity and accuracy,
repsectively.

Various approaches have been applied to the task of text simplification,
including lexical, syntactic, statistical and neural approaches Ondov et al.
(2022). Statistical and neural approaches essentially treat the task as a
translation task, translating from complex to simple language. Figure 1 shows
an example sentence pair, where the edit operations performed by the human
annotator are indicated using red and green colors. In contrast to cross-
lingual translations, simplifications share a significant portion of the
tokens with the input. We argue that this leads to suboptimal training because
the gradients from most tokens point to the trivial solution of copying that
token. We indeed find that models fine-tuned on such data tend to edit much
less than human annotators. To incentivize the model to edit more, we lend the
concept of edit operations (keep, replace, insert, delete), which are also
used in edit-based simplification Dong et al. (2019); Kumar et al. (2020); Xu
and Carpuat (2021). However, rather than directly predicting the edit
operations, we leverage them to inform the training process and correct the
ill-posed objective created by the shared tokens. In particular, we use the
edit operations to derive loss weights that assign higher importance to edited
tokens (token-level loss weights) or to highly edited sentences (sentence-
level). Our core contributions are:

  1. 1.

We propose and evaluate sentence-level and token-level loss weights for text
simplification and find that both are effective in increasing the edit
distance of the generated outputs, while keeping the outputs faithful.

  2. 2.

We investigate the relationship between edit distance and simplification
similarity measured by SARI Xu et al. (2016), and find that both loss
weighting schemes also significantly improve SARI scores.

  3. 3.

We perform a qualitative analysis of generated outputs to provide further
insights beyond the quantitative results.

##  2 Related Work

In this section, we present the most relevant previous works from the fields
of biomedical text simplification, controllable simplification and weighted
loss functions.

### PLABA

Plain Language Adaptation of Biomedical Abstracts (PLABA, Attal et al., 2023)
is a dataset containing 750 biomedical abstracts and 921 corresponding
simplifications. The dataset was created by scraping 75 common medical
questions from a forum, and retrieving the abstracts of 10 relevant papers per
question from PubMed. Human annotators were instructed in the guidelines to
replace arcane words with common synonyms, split complex sentences and omit
sentences that are irrelevant to consumer understanding. All simplified
abstracts are sentence-aligned, i.e., for every source sentence, there are
zero, one, or multiple target sentences.

### Biomedical Text Simplification

Ondov et al. (2022) survey models, evaluation methods and datasets for
biomedical text simplification. They identify data availability as one key
challenge in the application of neural models to the task. Guo et al. (2022)
investigate the usage of language models to improve health literacy. They
scrape a dataset of biomedical abstracts and corresponding summarizations, and
train models to simplify and summarize the abstracts. Their models simplify
the language while additionally compressing the information. In contrast,
PLABA (Attal et al., 2023) focuses on the simplification aspect of the task,
where the average text length even increases slightly.

### Controllable Text Simplification

Controllable text simplification aims to give users of language models control
over the target complexity of generated outputs. To that end, some works
Martin et al. (2020); Li et al. (2022) employ learned control tokens that
condition the language model to generate text in the desired complexity.
Others use reinforcement learning Yanamoto et al. (2022) or lexically
constrained decoding Zetsu et al. (2022). In this work, we also build on edit
operations, but use them to improve the training process of autoregressive
language models rather than predicting them directly.

### Weighted Loss Functions

In weighted loss functions, losses from individual samples are weighted
differently. This is commonly used to counteract issues arising from
imbalanced training sets Henning et al. (2023). In the context of text
simplification, complexity-based loss weights have been used to incentivize
the model to generate simpler text Kriz et al. (2019). The authors derived a
degree of complexity for every word in the vocabulary by analyzing their
occurrences in texts of different complexity levels. These complexity levels
were then used as loss weights, such that simpler words have a higher weight.
In the empirical evaluation, this modified loss function led to more
insertions, but also a higher FKGL and longer sentences.

|  | Similarity |  | Simplicity |  | Edit distance
---|---|---|---|---|---|---
|  | SARI | BLEU | ROUGE-L |  | FKGL | Difficult Words | #words |
Baselines | Copy | $15.6$ | $27.0$ | $53.9$ |  | $13.8$ | $8.5$ | $26.2$ |  | $0.0$
Empty | $21.7$ | $0.7$ | $0.0$ |  | $-15.7$ | $0.0$ | $0.0$ |  | $154.7$
| Ground Truth | $81.5$ | $98.8$ | $98.1$ |  | $11.3$ | $7.3$ | $27.6$ |  | $89.7$
Zero-Shot | Llama-2-7B | $29.8\pm 0.07$ | $6.4\pm 0.10$ | $20.5\pm 0.03$ |  | $8.7\pm 0.06$ | $12.1\pm 0.17$ | $89.8\pm 0.13$ |  | $355.3\pm 0.40$
Llama-2-13B | $28.1\pm 0.41$ | $6.8\pm 0.07$ | $20.5\pm 0.12$ |  | $8.2\pm 0.19$ | $12.6\pm 0.11$ | $84.7\pm 0.52$ |  | $331.6\pm 2.53$
| ChatGPT | $33.0\pm 0.10$ | $7.0\pm 0.09$ | $31.3\pm 0.14$ |  | $10.5\pm 0.34$ | $6.6\pm 0.23$ | $31.8\pm 0.11$ |  | $118.3\pm 0.63$
Finetuning | Llama-2-7B | $38.0\pm 0.46$ | $26.9\pm 0.11$ | $53.9\pm 0.16$ |  | $12.3\pm 0.18$ | $7.5\pm 0.06$ | $25.1\pm 0.25$ |  | $28.1\pm 1.34$
Llama-2-7B w/ sw | $39.9\pm 0.70$ | $25.1\pm 0.35$ | $51.9\pm 0.34$ |  | $11.8\pm 0.14$ | $7.5\pm 0.12$ | $25.9\pm 0.41$ |  | $40.1\pm 1.72$
Llama-2-7B w/ tw | $41.8\pm 0.27$ | $23.7\pm 0.38$ | $51.0\pm 0.51$ |  | $11.0\pm 0.30$ | $6.9\pm 0.20$ | $25.7\pm 0.45$ |  | $53.3\pm 2.70$
Llama-2-13B | $39.0\pm 0.66$ | $26.8\pm 0.38$ | $53.9\pm 0.34$ |  | $11.8\pm 0.29$ | $7.3\pm 0.12$ | $24.8\pm 0.34$ |  | $33.1\pm 2.15$
Llama-2-13B w/ sw | $40.8\pm 0.36$ | $24.0\pm 0.44$ | $50.2\pm 1.06$ |  | $10.8\pm 0.66$ | $7.2\pm 0.26$ | $25.6\pm 0.73$ |  | $51.7\pm 4.52$
Llama-2-13B w/ tw | $42.5\pm 0.31$ | $22.5\pm 1.47$ | $50.2\pm 1.59$ |  | $10.7\pm 0.45$ | $6.8\pm 0.10$ | $25.8\pm 0.47$ |  | $60.0\pm 8.06$
Llama-2-13B w/ tw + rs | $43.9\pm 0.33$ | $22.3\pm 1.32$ | $50.7\pm 1.55$ |  | $11.8\pm 0.56$ | $7.8\pm 0.12$ | $29.5\pm 0.64$ |  | $62.3\pm 6.24$

Table 2: Experimental Results. Each metric is reported as $mean\pm std$ across
5 runs. sw = sentence-level loss weights. tw = token-level loss weights. rs =
repeated sampling.

##  3 Modelling

We base our system on Llama 2 7B and 13B Touvron et al. (2023) which are pre-
trained Transformer models Vaswani et al. (2017). We fine-tune them on the
PLABA dataset in a sequence-to-sequence fashion. For every position $i$ in the
target sequence, the cross entropy loss $l$ between the ground truth token
$y_{i}$ and the predicted token $\hat{y}_{i}$ is backpropagated.

$\displaystyle loss=\sum_{i}l(y_{i},\hat{y}_{i})$

As illustrated in Figure 1, more than half of the tokens from the input are
kept in the simplification. Therefore, more than half of the gradients point
to the trivial solution of copying the input token. We argue that this is a
suboptimal training setting. In preliminary experiments, we confirm this by
measuring the edit distance between original and simplified sentences, finding
that, on average, the model edits less than half as many characters compared
to the human annotator. We hypothesize that editing more would lead to better
simplifications, and propose using sentence-level loss weights and token-level
loss weights to incentivize the model to edit more. A higher loss weight
$w\in\mathbb{R}^{+}$ effectively scales the gradient of that sample by $w$.

### Sentence-level Loss Weights

We derive the sentence-level loss weights from the edit distance, i.e., a
sample with a large distance between the original sentence $x$ and the
simplified sentence $y$ is weighted higher and will therefore have a larger
impact on the model parameters. More formally, the edit distance
$d(\cdot,\cdot)$ is mapped to a loss weight by a function $w(\cdot)$.

$\displaystyle loss=w(d(x,y))\cdot\sum_{i}l(y_{i},\hat{y}_{i})$

We experiment with linear and quadratic variants of $w$, as depicted in Figure
2. For all variants, we define slopes and offsets such that the mean edit
distance in the dataset ($86.49$) is assigned a loss weight of $1$.

Figure 2: Histogram of edit distances in the dataset and linear and quadratic
loss weighting functions $w$. The mean of the distribution is $86.49$, where
all loss weighting functions assign a weight of $1$, indicated by the red
cross.

### Token-level Loss Weights

One drawback of sentence-level loss weights is that the losses for all tokens
from one sentence are equally re-weighted, including both edited tokens and
shared tokens. To get more fine-grained re-weighting of the losses, we further
propose token-level loss weights. To that end, we first compute the edit
operations required to transform the original sentence $x$ into its
simplification $y$ using
difflib333https://docs.python.org/3/library/difflib.html. We then increase the
loss weight of all target tokens that are part of an edit operation to a
hyperparameter $\lambda$.

$\displaystyle\lambda_{i}$ $\displaystyle=\begin{cases}\lambda&\text{if
}y_{i}\text{ is part of an edit operation}\\\ 1&\text{else}\end{cases}$
$\displaystyle loss$ $\displaystyle=\sum_{i}\lambda_{i}\cdot
l(y_{i},\hat{y}_{i})$

### Repeated Sampling

To further improve the performance in the shared task, we sample from our best
model 10 times with varying temperatures. We then prompt ChatGPT with the
generated simplifications and instruct it to select the best one in a few-shot
setting.

##  4 Results

In this section, we analyze the results of our empirical evaluation shown in
Table 2.

### Evaluation and Experimental Setup

In the shared task, the simplifications are automatically evaluated using the
SARI metric Xu et al. (2016), measuring the $F_{1}$ score of n-gram edit
operations. The shared task submissions are furthermore manually evaluated
based on sentence simplicity, term simplicity, term accuracy, fluency,
completeness, and faithfulness. In addition to the shared task evaluation on
the blind test set, we conduct a comprehensive automatic evaluation of all
models on the labelled test set. To measure the similarity to the ground
truth, we use SARI, BLEU Papineni et al. (2002) and ROUGE-L Lin (2004). We
furthermore use Flesch-Kincaid Grade Level (FKGL, Flesch, 1943), the number of
difficult words444https://github.com/textstat/textstat and the total number of
words to measure the simplicity of the generated outputs. Moreover, we report
the edit distance Levenshtein et al. (1966) to measure how many characters
were edited by the models.

For all model configurations, we train and generate five times in order to
approximate statistical significance using standard deviations. To save
compute, we perform all fine-tuning using LoRA Hu et al. (2021), i.e., only
low-rank decompositions of the full parameter matrices are trained. The
hyperparameters are reported in Appendix A.

### SARI and Edit Distance

Both sentence-level and token-level loss weights lead to much higher edit
distances compared to training without loss weights. In particular, the edit
distance increases by 81% with token-level and 56% with sentence-level loss
weights. We furthermore observe a very strong correlation ($0.99$) between
edit distance and SARI as depicted in Figure 3, confirming our hypothesis that
incentivizing the model to edit more will also lead to better simplifications.
Adding token-level loss weights in the 7B model is much more effective ($+3.9$
SARI) than scaling up to 13B without loss weights ($+1.1$ SARI) while being
significantly cheaper during training and inference.

###  $\lambda$ as Control Parameter

Motivated by the finding that incentivizing the model to edit more also leads
to higher SARI scores, we experiment with the hyperparameter $\lambda$ which
determines the weight of edited tokens during training. The results are
depicted in Figure 4. Increasing $\lambda$ monotonically leads to a higher
edit distance and lower FKGL. That is, $\lambda$ can be effectively used as a
control mechanism for the edit distance and FKGL. Contrary to the edit
distance, SARI does not monotonically increase with $\lambda$ but peaks at
$\lambda=2.5$. Interestingly, this peak occurs far before the model’s edit
distance has matched that of the human annotators ($60.0$ vs $89.7$). There
are two possible reasons for this: either the model starts performing
incorrect edits, or the edits are correct but not present in the single
reference simplification.

Figure 3: Scatter plot of edit distances and SARI scores for 7B and 13B models
fine-tuned without loss weights, with sentence-level loss weights, and with
token-level loss weights. The Pearson correlation is $0.99$.

### Repeated Sampling

Repeated sampling further improves the SARI score by almost $1.5$ percentage
points. On the other hand, the simplifications are more complex compared to
sampling just a single time, indicated by higher FKGL, more difficult words
and longer sentences. I.e., ChatGPT seems to be biased towards the more
complex simplifications. We leave experimentation with alternative critics to
future work.

Author | Text
---|---
Original |  The mechanisms and kinetics of the immunosuppressive effects of alternate-day prednisone were investigated in a group of patients with a variety of inflammatory diseases receiving a range of alternate-day prednisone doses from 5 to 120 mg.
Human |  The process and reactions of the immune system from taking alternate- or every-other-day prednisone (a type of steroid drug that works by lowering the activity of the immune system) are investigated in a group of patients with different inflammatory diseases (diseases in which the immune system attacks the body’s own tissues).
Model |  We looked at how the immune system was affected by alternate-day prednisone. We looked at this in a group of patients with different inflammatory diseases. The patients were taking different doses of alternate-day prednisone.
Original |  The first trimester screening programme offers a noninvasive option for the early detection of aneuploidy pregnancies.
Human |  The first trimester screening program offers a noninvasive (nonsurgical) option for the early detection of aneuploidy pregnancies. Aneuploidy refers to the conditions of having an abnormal number of chromosomes.
Model |  The first trimester screening program offers a noninvasive option for the early detection of aneuploidy pregnancies.

Table 3: Examples of original sentence, human-written simplification and
model-written simplification for qualitative analysis. The model is trained
with token-level loss weights and $\lambda=2.5$.

### Zero-Shot

Fine-tuning outperforms zero-shot prediction by a large margin across models
in our experiments. We find that Llama-2 is unable to generate only the
simplified sentence, but also includes conversational phrases like “Sure, here
you go,” as well as additional explanations. We experimented with different
instructions to avoid these pitfalls, but found that they are not effective
enough to consistently control the target format. We leave experimentation
with few-shot approaches to future work.

Figure 4: SARI score and edit distance for different values of $\lambda$ in
token-level loss weights. Increasing $\lambda$ monotonically leads to higher
edit distances and lower FKGL, whereas SARI peaks at $\lambda=2.5$.

### BLEU and ROUGE

In our experiments, BLEU and ROUGE scores seem to be negatively correlated
with SARI. Not performing any edits (copy baseline) actually achieves the
highest BLEU and ROUGE scores. This could be because many of the tokens are
not edited by the human annotators, while there are many possible variants of
good simplification operations. That is, editing a token will likely decrease
n-gram precision even if the edit is good. We see similar trends in prior work
on simplification Attal et al. (2023); Li et al. (2023). This underlines the
importance of simplification-specific evaluation metrics that take the
original sentence into account, as well as using multiple reference
simplifications.

### Human Evaluation

The human evaluation was performed by the shared task organizers. Annotators
were asked to rate samples on multiple criteria, where each criterion could be
rated as -1, 0 or 1. The final score per criterion was then calculated by
summing all values and mapping the interval to a scale from 0 to 100. We
submitted the predictions from Llama 2 13B trained with token-level loss
weights and $\lambda=2.5$, i.e., the model with the best SARI score in our
evaluation. Overall, we were first in the accuracy category including the
completeness and faithfulness criteria with a mean score of 94.87. The model
therefore generated simplifications that matched the input very well in terms
of information content. In the simplicity category including sentence
simplicity, term simplicity, term accuracy and fluency, we ranked second with
a mean score of 88.86. Among these criteria, all ratings were above 90, except
for term simplicity with a value of 77.25. That means that the model did not
always explain or replace complex terms.

##  5 Discussion

### Low FKGL with Few Edits

As shown in Table 2, the fine-tuned models typically achieve a lower FKGL
score, despite editing much fewer characters on average. One might conclude
that the model is able to simplify the sentence more efficiently than the
human. We manually analyze cases where the model has much lower FKGL and edit
distance and find that this is rather due to different simplification styles.
An example is depicted in the first row of Table 3. The human annotator added
long term explanations in parentheses, whereas the model split up the sentence
into 3 shorter sentences. On the other hand, the model did not explain or
simplify the term “prednisone.” Generally, we observe that the model, compared
to the human annotator, focuses more on grammatical simplification than term
simplification. This insight is also reflected in the human evaluation, where
our model achieved much higher sentence simplicity than term simplicity.

### Controllability

When varying the $\lambda$ hyperparameter for the token-level loss weights, we
found that $\lambda$ strongly correlates with edit distance and FKGL. We
conclude that the method has potential applications for controllable
generation. However, unlike many other control mechanisms, such as control
tokens Martin et al. (2020), $\lambda$ has to be set during training. Control
at inference time would therefore require multiple training runs.
Nevertheless, $\lambda$ can be used to tune the model towards a target
simplicity level during training.

### Generalization to Other Domains

The proposed loss weights have shown significant improvements on the PLABA
dataset. Yet, the method should be evaluated on additional datasets to gain
more broadly applicable insights. We argue that most sentence pairs in
simplification datasets share a large portion of the tokens. Therefore,
improvements through loss weights are plausible across domains.

### Validity of Automated Metrics

We used several automated metrics in our empirical evaluation. BLEU and ROUGE
do not seem to be informative, as copying the input achieves very competitive
scores. That is, shared tokens pose challenges not only during training but
also during evaluation. SARI has been shown to correlate with human judgement,
but we find patterns where SARI produces unintuitive results. SARI often does
not reflect that terms have been properly substituted or explained. For
instance, the model-generated output in line 2 of Table 3 achieves a very high
SARI score of $72.4$ despite not explaining “noninvasive” and “aneuploidy.”
This underlines the importance of task-specific metrics that align with the
human quality criteria. For instance, term simplicity could be indicated by
the number of difficult terms that are either substituted or explained in
parentheses, measured by semantic similarity.

##  6 Conclusion

In this work, we first identified the issue that tokens that are not edited by
the human annotator have gradients pointing to the trivial solution of copying
the input. We indeed found that models trained with equally weighted gradients
tend to edit very conservatively. We hypothesized that incentivizing the model
to edit more will also lead to better simplifications, as measured by SARI. We
proposed token-level and sentence-level loss weights based on the edit
operations and the edit distance, respectively. Both led to significantly
increased edit distance and decreased FKGL, while also improving SARI. Due to
the more fine-grained adjustment, token-level loss weights performed best and
led to an increase of $3.9$ percentage points in SARI compared to regular
training. We additionally used repeated sampling with ChatGPT as critic and
ranked first in the automatic evaluation with a gap of $2.5$ percentage points
in SARI to second place.

## References

  * Attal et al. (2023) Kush Attal, Brian Ondov, and Dina Demner-Fushman. 2023.  A dataset for plain language adaptation of biomedical abstracts.  _Scientific Data_ , 10(1):8.
  * Dong et al. (2019) Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019.  EditNTS: An Neural Programmer-Interpreter Model for Sentence Simplification through Explicit Editing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3393–3402, Florence, Italy. Association for Computational Linguistics.
  * Flesch (1943) Rudolf Flesch. 1943.  Marks of readable style; a study in adult education.  _Teachers College Contributions to Education_.
  * Gopen and Swan (1990) George D. Gopen and J. A. Swan. 1990.  The Science of Scientific Writing.  _American Scientist_.
  * Guo et al. (2022) Yue Guo, Wei Qiu, Yizhong Wang, and Trevor Cohen. 2022.  Automated Lay Language Summarization of Biomedical Scientific Reviews.
  * Henning et al. (2023) Sophie Henning, William Beluch, Alexander Fraser, and Annemarie Friedrich. 2023\.  A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 523–540, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  LoRA: Low-Rank Adaptation of Large Language Models.
  * Kriz et al. (2019) Reno Kriz, João Sedoc, Marianna Apidianaki, Carolina Zheng, Gaurav Kumar, Eleni Miltsakaki, and Chris Callison-Burch. 2019.  Complexity-Weighted Loss and Diverse Reranking for Sentence Simplification.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3137–3147, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Kumar et al. (2020) Dhruv Kumar, Lili Mou, Lukasz Golab, and Olga Vechtomova. 2020.  Iterative Edit-Based Unsupervised Sentence Simplification.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7918–7928, Online. Association for Computational Linguistics.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet Physics Doklady_ , volume 10, pages 707–710. Soviet Union.
  * Li et al. (2023) Zihao Li, Samuel Belkadi, Nicolo Micheletti, Lifeng Han, Matthew Shardlow, and Goran Nenadic. 2023.  Large Language Models and Control Mechanisms Improve Text Readability of Biomedical Abstracts.
  * Li et al. (2022) Zihao Li, Matthew Shardlow, and Saeed Hassan. 2022.  An Investigation into the Effect of Control Tokens on Text Simplification.  In _Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)_ , pages 154–165, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Martin et al. (2020) Louis Martin, Benoît Sagot, and Antoine Bordes. 2020.  Controllable Sentence Simplification.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4689–4698, Marseille, France. European Language Resources Association.
  * Ondov et al. (2022) Brian Ondov, Kush Attal, and Dina Demner-Fushman. 2022.  A survey of automated methods for biomedical text simplification.  _Journal of the American Medical Informatics Association_ , 29(11):1976–1988.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Ronzano and Saggion (2016) Francesco Ronzano and Horacio Saggion. 2016.  Knowledge Extraction and Modeling from Scientific Publications.  In _Semantics, Analytics, Visualization. Enhancing Scholarly Data_ , Lecture Notes in Computer Science, pages 11–25, Cham. Springer International Publishing.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016.  Optimizing statistical machine translation for text simplification.  _Transactions of the Association for Computational Linguistics_ , 4:401–415.
  * Xu and Carpuat (2021) Weijia Xu and Marine Carpuat. 2021.  EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints.  _Transactions of the Association for Computational Linguistics_ , 9:311–328.
  * Yanamoto et al. (2022) Daiki Yanamoto, Tomoki Ikawa, Tomoyuki Kajiwara, Takashi Ninomiya, Satoru Uchida, and Yuki Arase. 2022.  Controllable Text Simplification with Deep Reinforcement Learning.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 398–404, Online only. Association for Computational Linguistics.
  * Zetsu et al. (2022) Tatsuya Zetsu, Tomoyuki Kajiwara, and Yuki Arase. 2022.  Lexically Constrained Decoding with Edit Operation Prediction for Controllable Text Simplification.  In _Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022)_ , pages 147–153, Abu Dhabi, United Arab Emirates (Virtual). Association for Computational Linguistics.

##  Appendix A Hyperparameters

Category | Parameter | Value
---|---|---
Training | Learning Rate | $5e-5$
Batch Size | 32
Epochs | 3
Precision | BF16
LoRa Rank | 8
LoRa $\alpha$ | 32
Generation | Temperature | 0.6
Top-P | 0.7
Max New Tokens | 128