[1, 2]Marko Pranjić 1]Jožef Stefan Institute, Jamova 39, Ljubljana, Slovenia
2]Jožef Stefan International Postgraduate School, Jamova 39, Ljubljana,
Slovenia

# Semantic change detection for Slovene language: a novel dataset and an
approach based on optimal transport

<EMAIL_ADDRESS>Kaja Dobrovoljc  Senja Pollak  Matej Martinc  [  [

###### Abstract

In this paper, we focus on the detection of semantic changes in Slovene, a
less resourced Slavic language with two million speakers. Detecting and
tracking semantic changes provides insights into the evolution of the language
caused by changes in society and culture. Recently, several systems have been
proposed to aid in this study, but all depend on manually annotated gold
standard datasets for evaluation. In this paper, we present the first Slovene
dataset for evaluating semantic change detection systems, which contains
aggregated semantic change scores for 104 target words obtained from more than
3000 manually annotated sentence pairs. We evaluate several existing semantic
change detection methods on this dataset and also propose a novel approach
based on optimal transport that improves on the existing state-of-the-art
systems with an error reduction rate of 22.8%.

###### keywords:

semantic change, diachronic shift, optimal transport, Slovene dataset

##  1 Introduction

Language is a dynamic system that reflects the cultural or technological
development of society [1]. This means that the meaning of a word is
constantly changing and evolving through use in social interactions and
changes in cultural practices. The change is usually gradual and long-term,
and direct shifts from one meaning to another are rare. Instead, the change in
meaning of a word usually goes through several polysemous stages. This makes
the identification and understanding of these changes notoriously difficult.
Nonetheless, work on this topic is important for linguistic research and
social analysis, as changes in language reflect changes in society and can be
used as a proxy for the detection of cultural and linguistic trends [2]. The
detection of these changes can also be used to improve many natural language
processing (NLP) tasks. For example, it could improve the temporal sensitivity
of current contextual language models, which are currently static and cannot
adapt to changes over time [3].

The field of lexical semantic change detection is a very active area of
research. While the first systems for automatic change detection were
developed more than a decade ago, research on this topic gained momentum with
the idea of using embeddings for the construction of temporal representations
[4]. This led to the development of several state-of-the-art semantic change
detection systems and the need for manually labeled gold standard datasets of
high quality for training and evaluating these systems. To address the problem
of the lack of gold standard and the lack of standardization in terms of
evaluation tasks and datasets, several shared tasks on this topic have
recently been organized [5, 6, 7, 8].

These shared tasks mostly covered lexical semantic change detection in high-
resourced languages with many speakers, such as English, Russian, German,
Italian, and Spanish. Although some less-resourced languages, e.g. Latin [5],
were also considered, for most less-resourced languages still lack evaluation
datasets that could be used for the evaluation of lexical semantic change
detection systems. In the absence of gold standards available to the
community, this hinders progress and leads to a widening of the already large
gap in NLP research between low-resourced and high-resourced languages.

To address this problem, in this paper we present a gold standard semantic
change detection dataset for Slovene, a language with about two million
speakers and with significantly fewer resources than high-resourced languages,
such as English [9]. The dataset covers two time periods with a 20-year gap in
between, extracted from the Slovene reference corpus Gigafida 2.0 [10].

The first period contains texts from 1990 to 1997 and the second period
contains texts from 2018. In total, the dataset consists of 105 target words
with manually annotated aggregated semantic change scores, 3150 sentence pairs
with manually annotated semantic change scores in which these words occur, as
well as fiction, textbook and news texts from the two periods. The dataset is
publicly available. Furthermore, we propose a novel unsupervised semantic
change detection system that uses an optimal transport for semantic change
detection and shows promising performance.

More specifically, the main contributions of this work are as follows:

  * •

A construction of the first manually labeled gold standard semantic change
detection dataset for Slovene, a low-resourced language with about two million
speakers.

  * •

A novel unsupervised semantic change detection system based on optimal
transport that outperforms three state-of-the-art embedding-based semantic
change detection systems on the new dataset.

  * •

The first comparative evaluation of multiple semantic change detection systems
in Slovene.

  * •

A thorough analysis of the constructed dataset, the annotation effort and the
resulting annotations.

The paper is structured as follows – in Section 2 we discuss related work on
semantic change detection. In Section 3 we describe the construction of the
dataset, while in Section 4 we describe our newly proposed approach for
automatic semantic change detection. The experimental setup is discussed in
Section 5 and the results of the evaluation experiments are presented in
Section 6. Finally, in Section 7 we draw conclusions from this work and
provide suggestions for further work.

##  2 Related work

The study of semantic change has occupied scholars long before the advent of
modern linguistics in the late 19th and early 20th centuries, which ushered in
a methodological shift in the study of language change [11]. Automatic
detection of temporal semantic change has recently become increasingly
important, as it is useful not only in linguistics and lexicography, but also
in various text stream monitoring tasks, such as event detection [12],
viewpoint analysis [13, 14], news stream analysis [15], or the monitoring of
discourse changes during crisis events [16]. Several studies and shared tasks
have recently been published on this topic [5, 6, 7, 8] to promote the
development of new systems and compare different approaches.

Nevertheless, the evaluation of this task remains a challenge as it relies on
manually annotated gold standard corpora covering multiple word usages, which
are still scarce. The first gold standard dataset for the evaluation of
semantic change systems that we are aware of, is the dataset created in [17].
It contains English words from two time periods, the 1960s and 1990s. The
annotators determined the degree of semantic change score for each word based
on their intuition, i.e. without looking at the context. This procedure was
later considered problematic, as an annotator might forget or not know a
particular meaning of the word [15].

For this reason, the procedure most commonly used in the creation of recent
corpora for assessing semantic change is to (1) annotate pairs of sentences in
which the target words have either the same or a different usage (i.e. the
procedure first used in the creation of word sense disambiguation datasets
[18]) and (2) aggregate these pairwise annotations into semantic shift scores.
The most common method for this is the COMPARE metric, which is used, for
example, in [19, 20, 21] and also in this research. While the COMPARE metric
is a simple average of pairwise relatedness scores, [5] relies on a more
sophisticated method of clustering a diachronic word usage graph to obtain
word sense distributions in different time periods.

When it comes to number of time periods and the number of words in the
manually annotated semantic shift datasets, most of them contain only two time
periods [17, 5, 19, 21, 7, 8] and about 100 target words or less, which is due
to the time-consuming manual annotations. As far as we know, the Russian
RuShiftEval dataset is the only dataset that contains three distinct time
periods, namely pre-Soviet, Soviet and post-Soviet. In terms of the number of
words, RuShiftEval is also the largest with 111 target words. It is closely
followed by the English dataset proposed in [17], the Russian dataset created
in [21], and the Spanish dataset from [8], all consisting of about 100 words.
In contrast, the datasets created for four languages (English, German, Latin,
Swedish) as part of SemEval-2020 Task 1: Unsupervised Lexical Semantic Change
Detection [5] are smaller and contain between 30 and 50 words each. The
smallest are an Italian dataset from EVALITA 2020 task: Diachronic lexical
semantics in Italian (DIACR-Ita)[7], and German DURel dataset [19], which
contain only 23 and 22 words respectively. An overview of comparable datasets
with available languages and number of annotated words is provided in Table 1.

Table 1: Size comparison with comparable semantic change detection datasets with available languages and number of words in the dataset. Dataset | Language | Number of words
---|---|---
SemEval2020: Task1[5] | ENG | 37
GER | 48
LAT | 40
SWE | 31
DURel[19] | GER | 22
DIACR-Ita[7] | ITA | 23
Google Books Ngram[17] | ENG | 100
RuSemShift[21] | RUS | 100
LSCDiscovery[8] | SPA | 100
RuShiftEval[6] | RUS | 111111RuShiftEval dataset contains annotations from three time periods, in effect annotating three times more word comparisons.
Slovene SCD[22] (ours) | SLO | 104222Out of 105 words selected for annotation, one word was removed due to unrelated words being matched in the initial word selection step.

The first systems for automatically detecting semantic change were developed
more than a decade ago. They relied on frequency-based methods [23, 24], which
are rarely used today, as the invention of word embedding representations [25]
led to the development of more efficient methods for this task. A detailed
overview of these slightly older frequency and embedding-based methods can be
found in [4], [26] and [27].

Current work on semantic change detection employs two distinct unsupervised
methodologies. They are based on the construction of temporal representations
using either static or contextual embeddings. The methodology employing static
embeddings is based on training a static embedding model for each temporal
segment of the corpus and then aligning these models to make them comparable.
This can be achieved by using incremental updating [28], where an embedding
model is trained from scratch on the first time slice of the corpus and then
updated at each subsequent time slice. Another option is to use vector space
alignment [29]. Here, the embedding models are trained independently for each
time slice and at the end an alignment is performed by optimizing the
geometric transformation. Another approach, that relies on static embedding
models, is based on the comparison of target word neighbors (i.e. words with
embedding representations that are very similar to the target word
representations) in different time slices [30, 31, 32]. In [32], for example,
they use static embeddings to obtain period-specific representations. In each
time period, a word is represented by its top nearest neighbors according to
the cosine distance and the semantic change is measured as the size of the
intersection between the lists of nearest neighbors of two periods.

All methods using static embedding models suffer from limitations in
sensitivity and interpretability due to the fact that each word has only one
representation within a time slice111One exception is the system from [33],
which analyzes the evolution of sense groups using a Bayesian model.. These
limitations can be mitigated by using contextual embeddings such as BERT [34],
where a different embedding representation is generated for each context in
which the word occurs, and enables modeling of the word polysemy.

These approaches tend to use an aggregation step in which contextual token
embeddings are combined into meaningful temporal representations. Different
methods are used in different studies, ranging from simple averaging [35], to
pairwise comparison of temporal vectors [36], and to clustering of token
embeddings where clusters are used as approximations of the set of meanings of
a given word [37, 38, 15]. The most popular method for comparing the derived
cluster distributions from different time periods and thus deriving a semantic
shift score, is the Jensen-Shannon divergence (JSD) [39], which is used in
studies of [37] and [38]. In [15], they propose to replace JSD with
Wasserstein distance (WD) [40], which was motivated by the observation that
WD, in addition to comparing cluster distributions, also takes into account
the position of clusters in semantic space, leading to better performance.

Another interesting approach, also using contextual embeddings, was proposed
in [3]. To make a BERT-like model sensitive to time-specific word usage, they
propose to concatenate a special time token to each text sequence during the
fine-tuning of the masked language model, thus directly incorporating temporal
information into the training process. The language model fine-tuned in this
way is able to predict the time of each sentence and detect semantic change by
looking at the distribution of the predicted times for each temporal segment
(e.g. a uniform distribution indicates no semantic change, while a non-uniform
distribution implies a change). Beside fine-tuning on the temporal token
classification, in [41] the performance of the semantic change detection with
regards to fine-tuning on other NLP tasks was evaluated. It has been shown
that fine-tuning on any single NLP task like Grammatical Error Correction,
Part-of-Speech tagging or Natural Language Inference the performance of the
model on the semantic change detection is decreased. But this trend reverses
if the word representation is taken to be an average of contextual
representations independantly fine-tuned on several tasks. Even though the
fine-tuning step itself can be replaced with Transformer Adapters[42], the
downside of combining different combinations is that it is unclear if a
particular fine-tuning task will lead to better performance in combination
with other tasks or will, in fact, decrease the overall performance.

Further research in the direction of the word representations led to a recent
result from [43] where improvement to semantic change performance was achieved
by representing a word with a set of $k$ most likely replacements from a
masked language model. Such replacements are aggregated for each target word
across the whole time slice to a single distribution of top-$k$ replacements
and JSD metric is used to calculate the difference between such time-slice
replacements distributions. Advantage of this approach is that dominant
semantic meaning for the time slice naturally emerge as most likely
replacements, thus providing a level of explainability to semantic change
detection methods. A downside of the approach is that performance improvement
is not consistent across datasets. Although there is no explicit clustering
step in this approach, the resulting list of replacements does represent a
cluster of most likely words.

Related work described above is more concerned with word representation and
several different research directions are described. The metric used to
compare between such representations is usually reused from previous work and
mainly revolves around measuring JSD of clusters or distributions. If multiple
evaluation metrics are used, like in [15, 41], the results are inconsistent
and better performance on one metric doesn’t imply improvement on the second
one. The optimal transport problem [44] has proven to be a valuable tool in
many natural language processing applications, such as training of text
generation models [45], the matching of interpretable text features [46] or
the optimal choice of model vocabulary [47]. It is often used to derive some
kind of metric on the data. In Word Movers Distance [48], optimal transport is
used to model the distance between text documents represented as a set of
trained word vector representations. The MoverScore [49], a scoring metric for
text generation with contextualized embeddings, uses optimal transport to find
an optimal global alignment of tokens. [50] uses optimal transport to provide
interpretable semantic text similarity. In the context of semantic change
detection, [15] applies optimal transport to detect the semantic change, but
unlike our work, their approach focuses on the measuring distances between
word cluster centroids, while we completely avoid the clustering step, give
equal weight to each usage example, and allow for finer-grained differences.

For Slovene, the automatic semantic change detection methods have not yet been
systematically evaluated, as no evaluation data was available. However, there
has been some related work dealing with computer-aided analysis of language
change, mainly in the field of computer-mediated communication. In [51] a
comparative collocation extraction based on statistical measures was used to
identify collocates typical of Slovene computer-mediated communication, while
in [52] embeddings to extract semantic change candidates from Slovene Twitter
data was used. Similarly, in [53] the Slovene reference corpus of computer-
mediated communication was leveraged to identify changes in vocabulary and
meaning through collocation analysis.

##  3 Dataset construction

###  3.1 Corpus Selection

In order to obtain a temporal corpus for the evaluation of the semantic change
detection models, we extracted texts from one of the largest Slovene corpora,
Gigafida 2.0[10], the reference corpus of written standard Slovene. It was
selected due to its size (more than 1 billion words) and the fact that it
contains texts from 1990 to 2018. The corpus includes fiction, textbooks and
news. Newspapers make up almost half of the corpus, online texts about a
quarter and magazines contribute about a sixth of the corpus in terms of word
count.

From this corpus, we extracted two different temporal sections with as large a
temporal gap as possible between them, which ensures an identifiable and
significant semantic change between several words. In order to obtain temporal
chunks of sufficient size, we decided that each chunk should cover at least
one year. The first chunk we chose therefore consists of texts from 2018, the
last year covered by the Gigafida 2.0 corpus and containing about 80 million
words. For the second chunk, we initially considered using texts from 1990,
the oldest year period covered by the Gigafida 2.0 corpus, to ensure the
greatest possible distance between the temporal chunks. However, since the
Gigafida 2.0 corpus did not contain enough texts from 1990 to create a
temporal chunk of sufficient size for evaluating (and training) semantic
change detection models, we decided to extend the time period to eight years.
The final second chunk therefore comprised texts from 1990 to 1997, totaling
about 70 million words. The detailed statistics on the datasets can be found
in Table 2.

Table 2: Dataset statistics at the level of different time periods (1990–1997 and 2018) and sources. Note that only sources with five or more documents have been included in the table. Therefore, the number of documents and the number of words do not match with the numbers given in the rows “All”. Sources from 1990 to 1997 | Num. docs | Num. words
---|---|---
Dolenjski list | 1776 | 9,540,845
Novi tednik | 1599 | 1,048,747
Tehniška založba Slovenije | 1159 | 2,256,914
Državni zbor Republike Slovenije | 630 | 1,387,402
Urbar | 392 | 336,617
Dnevnik | 374 | 17,449,311
Mladina | 339 | 13,263,063
neznani založnik | 155 | 1,117,711
DZS | 112 | 5,313,083
Gorenjski glas | 84 | 4,093,257
Zgodovinsko društvo za Južno Primorsko | 63 | 637,040
Društvo izdajateljev časnika 2000 | 40 | 757,399
Krka zdravilišča | 27 | 180,655
Radio-Tednik | 23 | 4,769,337
Infomediji | 21 | 2,004,767
Študentska organizacija Univerze, Študentska založba | 14 | 590,129
Cistercijanska opatija Stična | 13 | 1,081,784
Kmetijska založba | 12 | 280823
Zavod RS za šolstvo | 8 | 359655
Zveza geografskih društev Slovenije | 8 | 107,697
Sidarta | 7 | 317,158
Delo | 6 | 70,966
Klub študentov MF | 6 | 3,717
Desk | 6 | 840,479
Karantanija | 6 | 244,399
Društvo 2000 | 5 | 107,584
All | 6939 | 69,794,466
Sources from 2018 | Num. docs | Num. words
sta.si | 260 | 24,263,826
rtvslo.si | 231 | 24,289,885
siol.net | 102 | 10,561,982
delo.si | 87 | 7,228,310
svet24.si | 86 | 8,185,470
dnevnik.si | 53 | 5,147,473
24ur.com | 21 | 1631,404
slovenskenovice.si | 19 | 1,162,930
Litera | 5 | 273,111
All | 870 | 83,111,440

###  3.2 Word List Creation

When creating the target word dataset, we had to ensure that the following
conditions were met:

  * •

The dataset contains words whose usage has changed between two time periods.

  * •

The dataset contains words whose usage remains constant in both time periods.

  * •

Since the usage of words is usually gradual, i.e. it hardly ever changes
directly from one usage to another, but typically goes through several
polysemous phases, the dataset should contain several polysemous words that
are in different phases of usage change. This would allow us to measure the
degree of change between two different time periods, rather than just making a
binary decision about whether the word has changed or not.

  * •

The change/consistency in the usage of a particular selected target word
should be reflected in the temporal dataset we create.

To fulfill the above conditions, we proceeded in much the same way as in [21,
5, 6] by first selecting changed words and then supplementing the list of
target words with filler words, i.e. random words that have a similar
frequency distribution in both time periods of the dataset. To find changed
words, we first searched for related work on this topic and were able to find
a list of words with identified and labeled semantic changes created as part
of a study in [53]. Unfortunately, we found that this list mainly contains
neologisms (e.g. “miška” (eng. “mouse”), which is on the list due to the
invention of the computer device) and informal slang words (e.g., “nor” (eng.
“crazy”),which is on the list due to its informal use as a synonym for “very
good”) that were not used in the constructed dataset in a meaning specified by
the list or were very rare. For this reason, we decided to fill the list semi-
automatically.

More specifically, we used three automatic change detection systems to compute
the semantic change score for all words in the dataset that occur at least 30
times in each temporal segment (i.e., slightly less than 50,000 words). The
systems used were the clustering method proposed in [15], the Nearest
Neighbors method proposed in [32], and the SGNS+OP+CD method proposed in [54],
which refers to a semantic change detection method using static word
embeddings – we apply the Skip-Gram with Negative Sampling (SGNS) model
independently to two periods, align the embeddings using Orthogonal Procrustes
(OP) and use Cosine Distance (CD) to compute the semantic change. Each of
these models provided a list of words ordered by usage change, which allowed
us to compute an average rank across the three systems for each word in the
vocabulary. Two native speakers manually checked the final list of words
ordered by the average rank. After deduplication and filtering (e.g., we
manually removed several corpus artefacts), they manually selected 49 words.
The manual selection criteria followed two main principles, i.e., semantic and
evolutionary diversity.

To appease semantic diversity, the manual selection tried to maximize the
number of topics covered by the main senses of the selected words. For this
reason, several highly ranked words belonging to the over-represented topics
were not chosen. An example of this would be the word “diagonalen (eng.
diagonal)”, which was ranked as the 12th most changed word according to the
average rank criteria. The manual inspection showed that it is very often used
in the football context of the “diagonalna podaja (eng. diagonal pass)”. Since
the word “globinski (eng. deep)”, which was ranked as the fifth most changed
word, was used frequently in the exact same football context, and made it to
the final list of changed words, the word “diagonalen” was discarded.

To appease the evolutionary diversity criteria, we opted to not just select
the 49 most changed words according to the average rank criteria, but instead
obtain words in very different phases of usage change, which would evaluate
the tested models’ ability to not just derive a binary (i.e.
changed/unchanged) prediction, but rather to distinguish between different
stages and types of semantic changes. To achieve this, we manually checked all
1,000 best ranked words according to the average rank criteria. The
assumptions were, that, on the one hand, this set of 1,000 most changed words
contains words that were exposed to enough semantic change to be measurable,
and on the other hand, that this threshold is large enough that there is
enough diversity when it comes to the stage and type of evolution of these
words. In order to try to maximize the evolutionary diversity, 49 words with
diverse average ranking were picked from this set of 1000 words, i.e., the
final selection contained 8 out of 10 most changed words, 17 out of 50 most
changed words, 26 out of 100 most changed words, 40 out of 500 most changed,
and 49 out of 1000 most changed words words.

The manual inspection by the two native speakers also revealed several
interesting words that ranked below the 1000 threshold according to the
average rank, but were nevertheless ranked very high by one of the systems. An
example of this type of word is “evro (euro)”, with an average rank of
5,245.7. This word was ranked as 8th by the clustering system proposed in
[15], as 14,849th by the Nearest Neighbors method proposed in [32], and as
880th by the SGNS+OP+CD method proposed in [54]. We figured that inclusion of
words with such a disparity in the ranking would additionally increase the
difficulty of the final test set and also allow for a more fine-grain
comparison of different types of models, by revealing what kind of shifts they
can or cannot detect. Therefore, additionally, we included seven words with
very similar disparity in the ranking, namely evro, kontaktirati, kotacija,
poceniti, priključek, zapreka, zmerno.

After that, 49 filler words, which were selected for each target word
unambiguously identified by all systems as changed (i.e., words with an
average rank less than 1,000), were sampled so that they match the part of
speech and frequency distribution across both time periods, same as in [6]. By
doing this, we ensured that part of speech and frequency information cannot be
used to distinguish the target words from fillers. The final list given to the
annotators contained 105 words.

###  3.3 Annotation

For each target word in the list, we extracted 30 usage examples (sentences)
from the dataset from the years 1990–1997 and 30 usage examples from the year
2018. The sentences from both time periods were randomly matched (i.e. each
pair contains a random sentence from 1990–1997 and a random sentence from
2018, both containing the same target word), resulting in 3150 sentence pairs.
These pairs were annotated by three human annotators using the DURel framework
[19], on a scale of semantic relatedness from 1 to 4:

  * •

1 means that the usages in the sentences are not related to each other

  * •

2 means that the uses in the sentences are distantly related

  * •

3 means that the uses in the sentences are closely related to each other

  * •

4 means that the usages are identical (i.e. they have the same meaning)

The corresponding annotation guidelines consisted of a short document in which
the four categories were explained in more detail and illustrated with some
prototypical examples. In particular, annotators were instructed to assign the
label 1 - unrelated meaning - to cases where the two meanings of a word are
completely unrelated, e.g. burka as a dramatic composition (‘a farce’) on the
one hand and burka as a garment (‘a burqa’) on the other, whereby the
equivalence in form is purely coincidental (homonymy). Label 2 - distantly
related meaning - was assigned to sentence pairs with two different but
diachronically related meanings of a word, e.g. dopisnica as postcard on the
one hand and dopisnica as correspondent (reporter on site) on the other, both
of which are derived from the concept of correspondence (communication through
exchanging letters). In contrast, label 3 - closely related meaning - was
assigned to pairs of phrases with very similar but not identical meanings,
such as the adjective globinski ‘deep’, which denotes something that goes far
into the depths (e.g. a deep sea) on the one hand, or something with a strong
effect (e.g. a deep cleaning) on the other, the difference in meaning being
much more subtle and context-dependent compared to sentence pairs labeled with
1 and 2. Finally, annotators were instructed to use the label 4 – identical
meaning - for sentence pairs where there is no significant difference in the
meaning and syntactic context of a word, such as when using the noun razbitina
‘wreck’ to describe a severely damaged vehicle, regardless of the vehicle type
(e.g. car/train wreck).

After the preliminary round of annotations, the guidelines were expanded to
provide additional clarifications regarding three specific groups of words
that appear in the original list: named entities, such as Zenit (a soccer
club), names of team members, such as telički ‘(Dallas) Mavericks’, and
adjectives with homographic lemmas, i.e. lemmas that have the same spelling
but a different pronunciation, such as testen, which can refer to either a
test (with testnega as the genitive form) or a dough (with testenega as the
genitive form). While the annotators were instructed to discard the first
group by using the 0-label described below, the names of team members, which
are usually lowercase in Slovene, were left in the dataset as typical examples
of metaphorical derivations of meaning. Likewise, pairs of homographic
adjectives were retained as examples of (loose) homonymy, especially since
some are also diachronically related, such as vezen ‘linking’ as in vezni člen
‘linking element’ and vezen ‘embroidered’ as in vezeni prt ‘embroidered
tablecloth’, both of which refer to the term vez ‘a bond’.

The additional label 0 - not applicable - was included to mark instances that,
in the opinion of the annotators, could not be labeled using the above scale
(e.g. due to lack of context, ambiguity, preprocessing noise). However, this
label was only used sparingly (3.2% of all decisions), mainly examples of the
word zenit, where the mentions of the soccer club Zenit were incorrectly
matched to examples of the common noun zenit ‘zenith’ in the preprocessing
phase. Therefore, this word was removed from the final dataset, while other
examples labeled 0 were discarded in the subsequent analyzes and calculations
described later in this section.

In absolute terms, the three annotators matched on 1939 out of 3120 decisions
(62%) with a Krippendorff’s alpha of 0.721, somewhat higher than the inter-
annotator agreement reported in related annotation campaigns focusing on this
particular type of semantic disambiguation [19, 20, 5, 21, 8]. Despite the
relatively subjective nature of this annotation task, our results show
considerable pairwise agreement between annotators, with the pairwise weighted
Kappa for values 1 to 4 ranging from 0.63 to 0.70. As expected given the
notoriously difficult task of delineating word meaning[55], annotators most
frequently disagreed on whether there was a slight difference in meaning
(label 3) or no difference at all (label 4) and whether the two identified
meanings were closely related (label 3) or distantly related (label 2),
accounting for 55% and 21% of all disagreements between annotators,
respectively. A sample of annotated dataset is provided in the Appendix A.

###  3.4 Analysis and Evaluation

Finally, the annotated dataset was used to calculate the degree of semantic
change for each word between two time periods using the COMPARE metric [54],
which is a simple average of the relatedness scores for all sentence pairs and
all three annotators. Essentially, higher scores are given to words with
relatively stable meanings (e.g. the full score of 4 for words such as
dokumentarec ‘documentary’, metafora ‘metaphor’ and odstavljen ‘removed’),
while words with changing or competing meanings receive lower scores (e.g. 1.2
for burka ‘burqa/farce’, 2.1 for portal ‘doorway/website’, 2.2 for replika
‘replica/reply’).

As expected, the resulting list shows a moderate correlation with the three
baseline systems used to identify the target words (with Spearman’s Rho
between 0.47 and about 0.53 – see results in Section 6 for details), but some
discrepancies in the ranking of certain words can also be observed, such as
the noun izkrcanje ‘disembarkation’, which was identified as a meaning-
changing polysemous word by all three systems used in the creation of the
original list, but received a score of 4.0 in the final gold standard list,
meaning that for all 30 sentence pairs examined, all 3 annotators agreed that
there was no difference in meaning. Future work is needed to investigate
whether this is due to the limitations of the original word list selection on
the one hand, or sentence selection and matching on the other. Nonetheless,
the number of such discrepancies remains relatively small, especially when
zooming in on the word list with the lowest score and the shift in meaning.

In particular, these confirm earlier observations in Slovene lexical semantics
[53, 52] that recent semantic changes have mostly occurred in the context of
new technologies (portal ‘website’, čarovnik ‘wizzard’, ikona ‘icon’,
zakrožiti ‘to go viral’) and daily events, such as trending sports events
(vrag ‘devil’, teliček ‘maverick’, plezalka ‘climber’). However, a more
systematic analysis would be needed to determine whether the observed
differences in word usage are primarily due to new meanings emerging or
becoming more prominent in the language in general, or whether they are also
due to a bias in the data, such as the different types of texts
produced/present in the two time periods studied.

##  4 Semantic change detection through optimal transport

In this section, we present a novel approach for detecting a semantic change
based on optimal transport. First, we give a brief overview of the optimal
transport method. Next, we describe related work on semantic change detection
and highlight their differences with the proposed method. Finally, we present
the new method and the steps required to quantify the semantic change of a
word.

###  4.1 Optimal transport problem

Solving an optimal transport problem allows us to find the best mapping
between two groups of objects; or the best match between elements of a
complete bipartite graph such that the source is completely mapped to the
target. In this context, "best" implies a loss function that is minimized. In
the case of the optimal transport problem, this loss is a linear sum of the
losses incurred by each individual mapping between pairs across two sets. Due
to the linear nature of the underlying loss function, the entire procedure
allows for a high degree of interpretability. There is an exact, globally
optimal solution to the transportation problem. This solution is a transport
plan that assigns the source to the destination in such a way that the
transported amount weighted by the transport costs is minimized.

We can formally define the transport plan that moves the quantity $a_{ij}$
between the i-th part of the source ($S_{i}$) and the j-th part of the
destination ($D_{j}$) with the penalty $c_{ij}$ from the cost matrix (C).

$OT(S,D,C)=\min\limits_{A}{\sum_{i=1}^{m}\sum_{j=1}^{n}a_{i,j}c_{i,j}}$ (1)

To convert this into a linear program, we constrain the solution so that the
total quantity available from S is equal to the quantity transported to D.
This type of optimal transport problem, where the total quantity at the source
is equal to the total quantity at the destination, is also called a balanced
transport problem.

A common use case for an optimal transport problem is the calculation of a
distance between two histograms or distributions. This is also known as
Wasserstein Distance (WD) or Earth Movers Distance (EMD). This approach was
taken in [15], where the distribution of word semantics is determined by
clustering. All occurrences of a word are divided into a small number of
semantic clusters and the relative size of these clusters represents the
semantic probability distribution of a word as a histogram. The change in this
distribution from one time slice to another is an indicator of semantic change
that can be easily tracked. A transport problem for such a case is defined as
the cost of moving a semantic probability distribution as it appears in one
time slice to the distribution of the next time slice.

The central part of modeling the problem as a transport problem is the
determination of a cost matrix. The cost matrix represents the unit cost of
transport or a distance between each point of the source and the destination.
In our application of optimal transport to semantic change detection, both the
source and the destination are a set of numerical vectors obtained from the
deep language model. A common approach to compare such vectors is to compute a
cosine similarity [56, 38, 15] between them, defined as

$sim(\vec{p},\vec{q})=cos(\vec{p},\vec{q})=\frac{\vec{p}\cdot\vec{q}}{||\vec{p}||\,||\vec{q}||}$
(2)

The similarity is not applicable as a cost function, but we can convert it
into a cosine distance with

$dist(\vec{p},\vec{q})=1-sim(\vec{p},\vec{q})$ (3)

###  4.2 Detecting the semantic change

Existing approaches for detecting semantic change [38, 15, 37] use clustering
to divide occurrences of a single word into different semantic meanings. The
text is processed by a deep learning language model such as BERT and a hidden
layer vector is used as a word representation. The underlying model represents
each word with one or more subword tokens and a pooling scheme is used to
convert a set of subword representations into the representation of a whole
word. When using the subword tokenization scheme, the usual approach for
encoding the whole word is to take an average of all the subword tokens of
that word [15, 35]. Since language models such as BERT have a number of hidden
layers, the word representation for semantic tasks is usually created from the
combination of several last hidden layers [14]. The rationale for the choice
of final layers is validated by exploratory tasks, which show that the initial
layers are less suitable for semantic tasks [57, 58].

We question the assumption that the reconstruction of semantic clusters is
required for good semantic change detection results. In contrast to existing
approaches [15, 38, 37], we avoid explicit clustering of word representations.
It has already been established that clustering of token representations does
not lead to meaning-specific clusters [38, 59]. Moreover, as shown in [38],
the task of detecting semantic change is very sensitive to the number of
clusters, and the usual methods for optimization using the Silhouette score do
not correlate with better performance.

Second, we question the choice of using the embedding from the final hidden
layer semantic shift detection. Related work showing that the decreased
performance of initial layers on semantic tasks [57, 58] also points to the
decreased performance of the last layer on the same tasks. Futhermore, direct
measurement of semantic features across layers done in [60] imply that
semantic information depends both on the model size as well as architecture.

####  4.2.1 Method

We present an algorithm based on optimal transportation to evaluate a semantic
change that is compatible with the COMPARE metric. For a BERT-like language
model with $k$ layers and for a token $i$, we denote
$H_{i}=(h_{i,0},h_{i,1},...,h_{i,k})$ as a set of hidden layer representations
of the token $i$. For a target word that starts with the $j$th input token and
spans $n$ tokens, we generate a representation vector $\vec{r}$ as the average
of the hidden layer vectors corresponding to the target tokens. Based on the
model size and architecture (described in Section 5) we follow the findings
from [60] that show a sub-optimal performance of the final layer and select
second-to-last layer embeddings as word representations:

$\vec{r}=\sum_{i=j}^{j+n}\frac{\vec{h}_{i,k-1}}{n}$ (4)

Using the Equation (4), we encode a set of words in both time slices of the
dataset and obtain a set of vectors $S={s_{0},s_{1},...}$ and
$D={d_{0},d_{1},...}$. We construct a cost matrix $C$ for the transport
problem by applying Equation (3) to each pair of representations in a
Cartesian product of $S$ and $D$. The resulting matrix is used as the cost
matrix. For a source and the destination of the transport problem, we use
uniform values with $1/|S|$ for the source and $1/|D|$ for the destination.
With this setup, the solution of the optimal transport is normalized by the
number of words in both the source and the destination.

##  5 Experimental setup

To analyze our approach, we evaluate it using the newly created dataset
presented in Section 3 and compare it with several other methods. We chose the
monolingual Slovene SloBERTa model with 12 layers [61] as the deep learning
language model for generating the word representations. The usual approach to
evaluate semantic change detection is to fine-tune the language model on the
target data to adapt the model to the target distribution [37, 36, 15, 35].
Since the target data is already part of the corpus used to pre-train the
language model (and since the fine-tuning of the baseline models did not yield
positive results — see Section 6 for details), we skip this step. We infer all
sentences from the evaluation data and average the hidden layer
representations of subword tokens according to Equation (4). For each sentence
containing the target word, we pair each sentence from 1990–1997 with each
sentence from 2018 and compute the cosine distance between the representations
of the target word for each sentence pair. The resulting cosine distances are
used as the cost matrix for the optimal transport problem. Finally, we assign
the source and destination of the transport as described in Section 4.2.1 and
solve222We use the publicly available Python Optimal Transport (POT)
library[62]. for the optimal solution. The resulting optimal transport costs
are used as a measure for the semantic change of the target word.

Our proposed optimal transport-based approach is compared with the cluster-
based method proposed in [15], the SGNS+OP+CD method [54], which does not use
contextual word embeddings, and the Nearest Neighbors methods proposed in
[32], which also use static embeddings. These methods are described in Section
2.

For the cluster-based baseline method for detecting usage change in the target
words described in Section 3, we follow the related work and fine-tune the
SloBERTa model [61] for ten epochs and generate contextual embeddings for the
texts from both periods in the dataset. We also generate contextual embeddings
with a non-fine-tuned model to determine the impact of fine-tuning on the
performance of the model. We use the k-means algorithm with $k=5$ for
clustering (as in [15] and [37]) and, as in [15], test two different measures
of semantic change: the Jensen-Shannon divergence (JSD) and the Wasserstein
distance (WD). In the two baseline methods that use static embeddings, both
static embeddings are trained independently on two periods, and are aligned
using Orthogonal Procrustes (OP). Cosine Distance (CD) is used to compute the
semantic change in the SGNS+OP+CD method. In the nearest neighbor method, the
semantic change for a particular word is measured as the size of the
intersection between the nearest neighbor lists of two periods.

We evaluate all systems using Spearman’s rank correlation with the gold
standard rankings obtained using the COMPARE metric described in Section 3.

##  6 Results

Table 3: Spearman’s Rank Correlation between system output rankings and ground
truth rankings on the new Slovene semantic change detection dataset.

Approach | Spearman’s Rank Correlation
---|---
Static embedding methods
SGNS+OP+CD | 0.477
Nearest Neighbors | 0.527
Clustering-based methods
k-means 5 JSD not fine-tuned | 0.458
k-means 5 JSD fine-tuned | 0.477
k-means 5 WD not fine-tuned | 0.523
k-means 5 WD fine-tuned | 0.504
Optimal transport based (ours)
Optimal transport score | 0.635

We present the results of the evaluation on the dataset described in Section 3
in Table 3. Of the baseline methods, the Nearest Neighbors method has the best
performance. It is closely followed by the method k-means 5 WD not fine-tuned,
which shows a comparable performance. The worst performing methods are
SGNS+OP+CD and k-means 5 JSD fine-tuned, with a Spearman’s rank correlation of
0.477. The cluster-based methods show that it is clearly advantageous to use
WD instead of JSD for the semantic change measure. On the other hand, fine-
tuning the model does not lead to an improvement in performance in most cases.
In three out of four cases, fine-tuning the model even leads to a
deterioration of the model’s performance. Overall, the results of the baseline
methods are consistent with the results reported in the related work [14].

Figure 1: Performance of our approach at different layers and layer
combinations. Results are shown from the embedding layer (Layer0) through all
hidden layers (Layer1 to Layer12). Results are also shown with the popular
strategies from related work, namely the average pooling of the last five
(AvgPool8-12) and the last four layers (AvgPool9-12), as well as a comparison
with the approach without the final layer (AvgPool8-11 and AvgPool9-11).

The newly proposed method based on optimal transport, which is described in
Section 4, achieves the highest score on this dataset with an error reduction
rate of 22.8% compared to the best baseline approach. To further analyze the
approach and understand its behavior, we repeat the experiment and vary the
hidden layers used to encode the semantic information of the target word. The
results are shown in Figure 1. We find that the performance is strong across a
range of layers used. In fact, using embeddings from any layer above Layer3 to
encode the semantic meaning of the word improves on the best result achieved
by the baseline models. Although the performance of the final layer
representation is significantly lower than most others, this is not reflected
in the performance of the representation that uses an average pooling of
multiple layers.

To better understand the influence of the final hidden layer on the semantic
vector, we take a closer look at the magnitudes of these vectors. The Figure 2
shows a distribution of the $L^{2}$ layer norms for the target words in the
dataset across the hidden layers. We show that the magnitude of the vector
representation changes as it passes through the network layers. The second-to-
last layer has a 39% higher average norm than the final layer. Since the most
common way to combine these vectors is to average their hidden layer
representations (see e.g. [15, 14]), the overall contribution of the final
layer representation, when averaged with other representations having a larger
norm, is significantly reduced.

Figure 2: Analysis of the magnitudes of hidden state representations across
layers. For each hidden layer, we collect target word representations and plot
their norms.

##  7 Conclusion and future work

In this research, we have created a new dataset for training and evaluating
semantic change detection systems for Slovene, a low-resource Slavic language
with 2 million speakers. We use this dataset to evaluate several unsupervised
semantic change detection models, including a novel model based on optimal
transport that significantly outperforms baseline models.

In the absence of previous work on semantic change in Slovene, which did not
allow us to obtain a list of manually identified Slovene words with observed
semantic change in a given time period, we decided to find these words semi-
automatically (similar to [21, 5, 6]), i.e. by manually selecting the words
from a list of candidate words proposed by three systems for automatic change
detection. For this reason, it is quite possible that the final target list is
missing some words that are subject to a significant semantic change and were
overlooked by all systems. While this does not affect the function of the
constructed dataset as an evaluation dataset for semantic change detection
systems, it does impose some limitation on the use of the dataset as a
historical linguistic resource. For this reason, we plan to expand the target
list in the future to include words identified by newly developed semantic
change detection models as well as manually identified words to make it more
complete.

During creation of a list of semantically changed words, some words were
flagged as semantically changed by all three systems and later manually
annotated as not having any semantic change. Such words with a large
discrepancy between automated procedure and manual verification are of special
interest as they have the ability to highlight the edge-cases of automated
systems and provide insights toward their improvement. On the other hand, this
can also be just a limitation of the original word list, an artefact of random
sampling of sentence pairs. We leave a thorough analysis of a rationale behind
this effect for future work.

In the SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection
shared task [5], a large discrepancy was found between the results of the
different systems participating in the shared task for different languages. In
particular, they pointed out the poorer performance of systems based on
contextual embedding models compared to systems based on static embeddings. On
the other hand, we observe a more consistent performance of the different
baseline systems on the new dataset and no major differences in the
performance of systems based on static and contextual embeddings. As mentioned
in [15], the discrepancy in performance between the two types of models could
be related to the fact that the sentences in all SemEval evaluation corpora
are mixed, which means that transformer models cannot use the usual sequence
of 512 tokens as context, but are limited to the number of tokens in the
sentence, which negatively affects their performance. The more competitive
performance of the contextual embedding-based models on the proposed Slovene
dataset can therefore be explained by the fact that our evaluation corpus does
not contain mixed, non-preprocessed texts on which pre-trained transformer
models can use an entire 512-token context.

It has been shown [57, 58, 60] that the choice of hidden layer for optimal
performance on semantic tasks depends on the model family and the overall
depth of the model, and in all cases the final layer was not the optimal
choice. Our experiments provide additional evidence for this conclusion and
performance of individual layers, as presentend in Figure 1, match results
published in [60]. The semantic change detection performance is significantly
lower when the final hidden layer (Layer12) is used to encode the semantic
meaning of the word, compared to the previous layers. This observation calls
into question the effectiveness of the usual approach at using the final layer
embedding by itself, or as a part of the averaged representation, to obtain a
semantic representation in the unsupervised semantic tasks [34]. Nonetheless,
our layer averaging experiments also show that including the final layer in
the semantic representation does not significantly influence the performance
of our approach. Analyis suggest that this is due to the fact that the final
layer of the RoBERTa model has a significantly smaller vector norm than the
previous layers, which indicate that it contributes less to the final averaged
representation. Future work should take a closer look at this phenomena and
determine if the mismatch of the final layer norm is a general trend among
BERT-like models, a specifics of the RoBERTa architecture or an artefact of
the SloBERTa training procedure.

Although solving a single Optimal transport problem is fast in the absolute
sense, and we have shown it is a good fit as an evaluation measure of semantic
change for a limited set of words, it is not sufficiently fast to scale to the
whole-corpus analysis. Significantly speeding up the approach is needed before
our approach can be adapted for a whole dataset analysis. One way to proceed
is to use a Sinkhorn-Knopp matrix scaling algorithm as proposed in [63] that
has a potential to provide a several orders of magnitude faster computation.
We leave this extension for the future work.

Data Availability The dataset is available via CLARIN.SI333Semantic change
detection dataset for Slovenian 1.0, available from:
http://hdl.handle.net/11356/1651 and the code via GitHub444The code used in
experiments, available here: https://github.com/sharpsy/slovene-OT-semchange.

Acknowledgements The authors acknowledge the financial support from the
Slovenian Research and Innovation Agency for research core funding for the
programmes Knowledge Technologies (No. P2-0103) and the Language Resources and
Technologies for Slovene (No. P6-0411) as well as projects Embeddings-based
techniques for Media Monitoring Applications (No. L2-50070), Computer-assisted
multilingual news discourse analysis with contextual embeddings (No. J6-2581)
and Hate speech in contemporary conceptualizations of nationalism, racism,
gender and migration (No. J5-3102). We also acknowledge the project
Development of Slovene in a Digital Environment co-financed by the Republic of
Slovenia and the European Union under the European Regional Development Fund -
The project is being carried out under the Operational Programme for the
Implementation of the EU Cohesion Policy in the period 2014 – 2020.

## Statements and Declarations

  * •

Funding: Slovenian Research and Innovation Agency grants: No. P2-0103, No.
P6-0411, J6-2581, J5-3102.

  * •

Conflict of interest/Competing interests: The authors declare no conflict of
interest.

  * •

Data and Code availability: publicly available, MIT licence

  * •

Author contribution: MP: design and implementation, manuscript writing, KD:
data annotation desing and guidance, analysis, manuscript writing, SP: method
design, manuscript writing, MM: design and implementation, manuscript writing,
evaluation design.

##  Appendix A Sample of annotated data

Table 4: A sample of the Slovene dataset for semantic change detection with a target word, year of the sentence and scores from three annotators. Below each sentence, an English translation is provided with a target word marked with an underline. Higher annotation scores correspond to a closer semantic match (less change) between two sentences. Word | Year |  Sentence | Scores
---|---|---|---
globinski | 1997 |  Pri plazovih sprijetega snega so to pogosto stične ploskve med starim-in-novim snegom ter skrajno labilne šibke plasti zasneženega površinskega ali globinskega sreža
(eng.) In the case of avalanches of stuck snow, these are often contact surfaces between old and new snow and extremely labile weak layers of snow-covered surface or deep snow. | 2 3 2
2018 |  Veliko je izdelkov , s katerimi lahko nadomestimo draga globinska čistila za obraz .
(eng.) There are many products that can be used to replace expensive deep
facial cleansers.
burka | 1997 |  Burka je reč , ki jo lahko najlaže uprizorimo , in sicer iz dveh razlogov .
(eng.) Burlesque is the easiest thing to stage, for two reasons. | 1 1 1
2018 |  V primeru potrditve referenduma , bo Gallen drugi švicarski kanton , v katerem bodo prepovedali burke in nikabe , pred dvema letoma so to naredili v Ticinu .
(eng.) If the referendum is approved, Gallen will be the second Swiss canton
in which burkas and niqabs will be banned, two years ago they did so in
Ticino.
glinast | 1997 |  V soboto , 20. aprila , je bilo na olimpijskem strelišču v Ormožu 2. kolo v tretji državni ligi v streljanju na glinaste golobe disciplina trap .
(eng.) On Saturday, April 20, at the Olympic shooting range in Ormož, the 2nd round of the third national league in clay pigeon shooting, trap discipline, took place. | 4 4 4
2018 |  Prekmurka je zadela 114 glinastih golobov , do preboja v veliki finale šestih najboljših strelk na svetu pa sta jo ločila dva zadetka .
(eng.) The girl from Prekmurje scored 114 clay pigeons, only two hits away
from reaching the grand final among the six best shooters in the world.
ogaben | 1997 |  Na atletskem stadionu je postavala nepregledna množica primerno ogabnih brkatih ljubiteljev distorzije , ki pa kakšnega posebnega zanimanja za Metallico niso kazali .
(eng.) A huge crowd of pretty disgusting mustachioed distortion fans lined up at the athletic stadium, but they didn’t show any particular interest in Metallica. | 4 3 4
2018 |  Po objavljenih posnetkih so na vrata luksuznih hotelov potrkali policisti in turistični inšpektorji , ki so hotelsko upravo kaznovali in pozvali , naj opustijo sramotno in ogabno prakso .
(eng.) According to the published footage, policemen and tourist inspectors
knocked on the doors of luxury hotels, fined the hotel management and called
for abandonment of the shameful and disgusting practice.
gazela | 1997 |  V zahvalo za meso , ki sem jim ga prepustil , sta dva od njih rada šla z menoj ter nesla glavo in stegno gazele .
(eng.) As a thank you for the meat that I left to them, two of them gladly went with me and carried the head and thigh of a gazelle. | 2 2 2
2018 |  Z veseljem je ugotovil , da nobeno od podjetij , ki se je do danes okitilo z nazivom gazela , ni skrenilo s poti , ampak so-uspela .
(eng.) He was happy to note that none of the companies that have so far been
awarded the Gazelle prize have not gone astray, but have succeeded.

## References

  *   * Aitchison [2001] Aitchison, J.: The ever-whirling wheel: The inevitability of change. Language change: Progress or decay? Cambridge Approaches to Linguistics. Cambridge University Press, Cambridge (2001)
  * Gillani and Levy [2019] Gillani, N., Levy, R.: Simple dynamic word embeddings for mapping perceptions in the public sphere. In: Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, pp. 94–99 (2019)
  * Rosin et al. [2022] Rosin, G.D., Guy, I., Radinsky, K.: Time masking for temporal language models. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 833–841 (2022)
  * Tahmasebi et al. [2018] Tahmasebi, N., Borin, L., Jatowt, A.: Survey of computational approaches to diachronic conceptual change. arXiv preprint arXiv:1811.06278 (2018)
  * Schlechtweg et al. [2020] Schlechtweg, D., McGillivray, B., Hengchen, S., Dubossarsky, H., Tahmasebi, N.: SemEval-2020 task 1: Unsupervised lexical semantic change detection. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 1–23. International Committee for Computational Linguistics, Barcelona (online) (2020). https://www.aclweb.org/anthology/2020.semeval-1.1
  * Kutuzov et al. [2021] Kutuzov, A., Pivovarova, L., et al.: RuShiftEval: a shared task on semantic shift detection for russian. In: Computational Linguistics and Intellectual Technologies: Papers from the Annual Conference Dialogue. (2021). Redkollegija sbornika
  * Basile et al. [2020] Basile, P., Caputo, A., Caselli, T., Cassotti, P., Varvara, R.: Diacr-ita @ evalita2020: Overview of the evalita2020 diachronic lexical semantics (diacr-ita) task. EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020 (2020)
  * Zamora-Reina et al. [2022] Zamora-Reina, F.D., Bravo-Marquez, F., Schlechtweg, D.: LSCDiscovery: A shared task on semantic change discovery and detection in Spanish. In: Tahmasebi, N., Montariol, S., Kutuzov, A., Hengchen, S., Dubossarsky, H., Borin, L. (eds.) Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pp. 149–164. Association for Computational Linguistics, Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.lchange-1.16 . https://aclanthology.org/2022.lchange-1.16
  * Marra et al. [2022] Marra, E., Lösch, A., Racioppa, S., Mazo, H., Giagkou, M., et al.: Sustainable Language Data Sharing to Support Language Equality in Multilingual Europe - Why Language Data Matters: ELRC White Paper, 3rd edn. ELRC Consortium, Germany (2022)
  * Krek et al. [2020] Krek, S., Holdt, Š.A., Erjavec, T., Čibej, J., Repar, A., Gantar, P., Ljubešić, N., Kosem, I., Dobrovoljc, K.: Gigafida 2.0: the reference corpus of written standard slovene. In: Proceedings of the 12th Language Resources and Evaluation Conference, pp. 3340–3345 (2020)
  * Tahmasebi et al. [2021] Tahmasebi, N., Borin, L., Jatowt, A., Xu, Y., Hengchen, S. (eds.): Computational Approaches to Semantic Change. Language Variation, vol. 6. Language Science Press, Berlin (2021). https://doi.org/10.5281/zenodo.5040241
  * Kutuzov et al. [2017] Kutuzov, A., Velldal, E., Øvrelid, L.: Tracing armed conflicts with diachronic word embedding models. In: Proceedings of the Events and Stories in the News Workshop, pp. 31–36. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/W17-2705 . https://www.aclweb.org/anthology/W17-2705
  * Azarbonyad et al. [2017] Azarbonyad, H., Dehghani, M., Beelen, K., Arkut, A., Marx, M., Kamps, J.: Words are malleable: Computing semantic shifts in political and media discourse. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1509–1518 (2017)
  * Martinc et al. [2021] Martinc, M., Perger, N., Pelicon, A., Ulčar, M., Vezovnik, A., Pollak, S.: EMBEDDIA hackathon report: Automatic sentiment and viewpoint analysis of slovenian news corpus on the topic of lgbtiq+. In: Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, pp. 121–126 (2021)
  * Montariol et al. [2021] Montariol, S., Martinc, M., Pivovarova, L., et al.: Scalable and interpretable semantic change detection. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies (2021). The Association for Computational Linguistics
  * Stewart et al. [2017] Stewart, I., Arendt, D., Bell, E., Volkova, S.: Measuring, predicting and visualizing short-term change in word representation and usage in VKontakte social network. In: Eleventh International AAAI Conference on Web and Social Media (2017)
  * Gulordava and Baroni [2011] Gulordava, K., Baroni, M.: A distributional similarity approach to the detection of semantic change in the Google Books ngram corpus. In: Pado, S., Peirsman, Y. (eds.) Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pp. 67–71. Association for Computational Linguistics, Edinburgh, UK (2011)
  * Erk et al. [2013] Erk, K., McCarthy, D., Gaylord, N.: Measuring word meaning in context. Computational Linguistics 39(3), 511–554 (2013)
  * Schlechtweg et al. [2018] Schlechtweg, D., Walde, S., Eckmann, S.: Diachronic usage relatedness (DURel): A framework for the annotation of lexical semantic change. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 169–174. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-2027 . https://www.aclweb.org/anthology/N18-2027
  * Kutuzov and Pivovarova [2021] Kutuzov, A., Pivovarova, L.: Three-part diachronic semantic change dataset for Russian. In: Proceedings of the 2nd International Workshop on Computational Approaches to Historical Language Change 2021, pp. 7–13. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.lchange-1.2 . https://aclanthology.org/2021.lchange-1.2
  * Rodina and Kutuzov [2020] Rodina, J., Kutuzov, A.: RuSemShift: a dataset of historical lexical semantic change in Russian. In: Proceedings of the 28th International Conference on Computational Linguistics, pp. 1037–1047. International Committee on Computational Linguistics, Barcelona, Spain (Online) (2020). https://doi.org/10.18653/v1/2020.coling-main.90 . https://aclanthology.org/2020.coling-main.90
  * Martinc et al. [2022] Martinc, M., Dobrovoljc, K., Pollak, S.: Semantic change detection datasets for Slovenian 1.0. Slovenian language resource repository CLARIN.SI (2022). http://hdl.handle.net/11356/1651
  * Juola [2003] Juola, P.: The time course of language change. Computers and the Humanities 37(1), 77–96 (2003)
  * Hilpert and Gries [2008] Hilpert, M., Gries, S.T.: Assessing frequency changes in multistage diachronic corpora: Applications for historical corpus linguistics and the study of language acquisition. Literary and Linguistic Computing 24(4), 385–401 (2008)
  * Mikolov et al. [2013] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems 26 (2013)
  * Kutuzov et al. [2018] Kutuzov, A., Øvrelid, L., Szymanski, T., Velldal, E.: Diachronic word embeddings and semantic shifts: a survey. In: Bender, E.M., Derczynski, L., Isabelle, P. (eds.) Proceedings of the 27th International Conference on Computational Linguistics, pp. 1384–1397. Association for Computational Linguistics, Santa Fe, New Mexico, USA (2018). https://aclanthology.org/C18-1117
  * Tang [2018] Tang, X.: A state-of-the-art of semantic change computation. Natural Language Engineering 24(5), 649–676 (2018)
  * Kim et al. [2014] Kim, Y., Chiu, Y.-I., Hanaki, K., Hegde, D., Petrov, S.: Temporal analysis of language through neural language models. In: Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pp. 61–65 (2014). https://doi.org/10.3115/v1/W14-2517 . http://aclweb.org/anthology/W14-2517
  * Hamilton et al. [2016a] Hamilton, W.L., Leskovec, J., Jurafsky, D.: Diachronic word embeddings reveal statistical laws of semantic change. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1489–1501 (2016). https://doi.org/10.18653/v1/P16-1141 . http://aclweb.org/anthology/P16-1141
  * Hamilton et al. [2016b] Hamilton, W.L., Leskovec, J., Jurafsky, D.: Cultural shift or linguistic drift? comparing two computational measures of semantic change. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2116–2121. Association for Computational Linguistics, Austin, Texas (2016). https://doi.org/10.18653/v1/D16-1229 . https://www.aclweb.org/anthology/D16-1229
  * Yin et al. [2018] Yin, Z., Sachidananda, V., Prabhakar, B.: The global anchor method for quantifying linguistic shifts and domain adaptation. In: Advances in Neural Information Processing Systems, pp. 9412–9423 (2018)
  * Gonen et al. [2020] Gonen, H., Jawahar, G., Seddah, D., Goldberg, Y.: Simple, interpretable and stable method for detecting words with usage change across corpora. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 538–555. Association for Computational Linguistics, Online (2020). https://www.aclweb.org/anthology/2020.acl-main.51
  * Frermann and Lapata [2016] Frermann, L., Lapata, M.: A Bayesian model of diachronic meaning change. Transactions of the Association for Computational Linguistics 4, 31–45 (2016) https://doi.org/10.1162/tacl_a_00081
  * Devlin et al. [2019] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1423 . https://www.aclweb.org/anthology/N19-1423
  * Martinc et al. [2020] Martinc, M., Kralj Novak, P., Pollak, S.: Leveraging contextual embeddings for detecting diachronic semantic shift. In: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4811–4819. European Language Resources Association, Marseille, France (2020). https://aclanthology.org/2020.lrec-1.592
  * Kutuzov and Giulianelli [2020] Kutuzov, A., Giulianelli, M.: UiO-UvA at SemEval-2020 task 1: Contextualised embeddings for lexical semantic change detection. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 126–134. International Committee for Computational Linguistics, Barcelona (online) (2020). https://www.aclweb.org/anthology/2020.semeval-1.14
  * Giulianelli et al. [2020] Giulianelli, M., Del Tredici, M., Fernández, R.: Analysing lexical semantic change with contextualised word representations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3960–3973. Association for Computational Linguistics, Online (2020). https://www.aclweb.org/anthology/2020.acl-main.365
  * Martinc et al. [2020] Martinc, M., Montariol, S., Zosa, E., Pivovarova, L.: Capturing evolution in word usage: Just add more clusters? In: Companion Proceedings of the Web Conference 2020. WWW ’20, pp. 343–349. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3366424.3382186 . https://doi.org/10.1145/3366424.3382186
  * Lin [1991] Lin, J.: Divergence measures based on the shannon entropy. IEEE Transactions on Information theory 37(1), 145–151 (1991)
  * Solomon [2018] Solomon, J.: Optimal transport on discrete domains (2018)
  * Zhou et al. [2023] Zhou, W., Tahmasebi, N., Dubossarsky, H.: The finer they get: Combining fine-tuned models for better semantic change detection. In: Alumäe, T., Fishel, M. (eds.) Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 518–528. University of Tartu Library, Tórshavn, Faroe Islands (2023). https://aclanthology.org/2023.nodalida-1.52
  * Pfeiffer et al. [2020] Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I.: Adapterhub: A framework for adapting transformers. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pp. 46–54. Association for Computational Linguistics, Online (2020). https://www.aclweb.org/anthology/2020.emnlp-demos.7
  * Card [2023] Card, D.: Substitution-based semantic change detection using contextual embeddings. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 590–602. Association for Computational Linguistics, Toronto, Canada (2023). https://doi.org/10.18653/v1/2023.acl-short.52 . https://aclanthology.org/2023.acl-short.52
  * Monge [1781] Monge, G.: Mémoire sur la Théorie des Déblais Et des remblais. De l’Imprimerie Royale, Paris, France (1781)
  * Li et al. [2020] Li, J., Li, C., Wang, G., Fu, H., Lin, Y., Chen, L., Zhang, Y., Tao, C., Zhang, R., Wang, W., Shen, D., Yang, Q., Carin, L.: Improving text generation with student-forcing optimal transport. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9144–9156. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.emnlp-main.735 . https://aclanthology.org/2020.emnlp-main.735
  * Swanson et al. [2020] Swanson, K., Yu, L., Lei, T.: Rationalizing text matching: Learning sparse alignments via optimal transport. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5609–5626. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.496 . https://aclanthology.org/2020.acl-main.496
  * Xu et al. [2021] Xu, J., Zhou, H., Gan, C., Zheng, Z., Li, L.: Vocabulary learning via optimal transport for neural machine translation. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7361–7373. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.acl-long.571 . https://aclanthology.org/2021.acl-long.571
  * Kusner et al. [2015] Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From word embeddings to document distances. In: Bach, F., Blei, D. (eds.) Proceedings of the 32nd International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 37, pp. 957–966. PMLR, Lille, France (2015). https://proceedings.mlr.press/v37/kusnerb15.html
  * Zhao et al. [2019] Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C.M., Eger, S.: MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 563–578. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1053 . https://aclanthology.org/D19-1053
  * Lee et al. [2022] Lee, S., Lee, D., Jang, S., Yu, H.: Toward interpretable semantic textual similarity via optimal transport-based contrastive sentence learning. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5969–5979. Association for Computational Linguistics, Dublin, Ireland (2022). https://doi.org/10.18653/v1/2022.acl-long.412 . https://aclanthology.org/2022.acl-long.412
  * Pollak et al. [2019] Pollak, S., Gantar, P., Arhar Holdt, Š.: What’s New on the Internetz? Extraction and Lexical Categorisation of Collocations in Computer-Mediated Slovene. International Journal of Lexicography 32(2), 184–206 (2019) https://doi.org/10.1093/ijl/ecy026
  * Fišer and Ljubešić [2019] Fišer, D., Ljubešić, N.: Distributional modelling for semantic shift detection. International journal of lexicography 32(2), 163–183 (2019)
  * Gantar et al. [2018] Gantar, P., Holdt, Š.A., Pollak, S.: Leksikalne novosti v besedilih računalniško posredovane komunikacije. Slavisticna Revija 66(4), 459–472 (2018)
  * Schlechtweg et al. [2019] Schlechtweg, D., Hätty, A., Del Tredici, M., Walde, S.: A wind of change: Detecting and evaluating lexical semantic change across times and domains. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 732–746. Association for Computational Linguistics, Florence, Italy (2019). https://www.aclweb.org/anthology/P19-1072
  * Hanks [2013] Hanks, P.: Lexical Analysis: Norms and Exploitations. The MIT Press, Cambridge, MA, USA (2013)
  * Reimers and Gurevych [2019] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using siamese BERT-networks. ArXiv abs/1908.10084 (2019)
  * Zhang et al. [2020] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: Evaluating text generation with BERT. ArXiv abs/1904.09675 (2020)
  * Liu et al. [2019] Liu, N.F., Gardner, M., Belinkov, Y., Peters, M.E., Smith, N.A.: Linguistic knowledge and transferability of contextual representations. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1073–1094. Association for Computational Linguistics, Minneapolis, Minnesota (2019)
  * Coenen et al. [2019] Coenen, A., Reif, E., Yuan, A., Kim, B., Pearce, A., Viégas, F., Wattenberg, M.: Visualizing and measuring the geometry of BERT. Curran Associates Inc., Red Hook, NY, USA (2019)
  * Turton et al. [2021] Turton, J., Smith, R.E., Vinson, D.: Deriving contextualised semantic features from BERT (and other transformer model) embeddings. In: Rogers, A., Calixto, I., Vulić, I., Saphra, N., Kassner, N., Camburu, O.-M., Bansal, T., Shwartz, V. (eds.) Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pp. 248–262. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.repl4nlp-1.26 . https://aclanthology.org/2021.repl4nlp-1.26
  * Ulčar and Robnik-Šikonja [2021] Ulčar, M., Robnik-Šikonja, M.: SloBERTa: Slovene monolingual large pretrained masked language model (2021)
  * Flamary et al. [2021] Flamary, R., Courty, N., Gramfort, A., Alaya, M.Z., Boisbunon, A., Chambon, S., Chapel, L., Corenflos, A., Fatras, K., Fournier, N., Gautheron, L., Gayraud, N.T.H., Janati, H., Rakotomamonjy, A., Redko, I., Rolet, A., Schutz, A., Seguy, V., Sutherland, D.J., Tavenard, R., Tong, A., Vayer, T.: Pot: Python optimal transport. Journal of Machine Learning Research 22(78), 1–8 (2021)
  * Cuturi [2013] Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transport. In: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2. NIPS’13, pp. 2292–2300. Curran Associates Inc., Red Hook, NY, USA (2013)