**footnotetext: Authors are in alphabetical order. Equal
contributions.aafootnotetext: Corresponding author.11institutetext:  1Computer
Science and Engineering Department, National University of Science and
Technology POLITEHNICA Bucharest, Romania
2Research Institute for Artificial Intelligence “Mihai Drăgănescu”, Romanian
Academy
11email<EMAIL_ADDRESS>
# HistNERo: Historical Named Entity Recognition for the Romanian Language

Andrei-Marius Avram1 Andreea Iuga1,* George-Vlad Manolache1,* Vlad-Cristian
Matei1,* Răzvan-Gabriel Micliuş1,* Vlad-Andrei Muntean1,* Manuel-Petru
Sorlescu1,* Dragos-Andrei Serban1,* Adrian-Dinu Urse1,* Vasile Păis2 and
Dumitru-Clementin Cercel1,a

###### Abstract

This work introduces HistNERo, the first Romanian corpus for Named Entity
Recognition (NER) in historical newspapers. The dataset contains 323k tokens
of text, covering more than half of the 19th century (i.e., 1817) until the
late part of the 20th century (i.e., 1990). Eight native Romanian speakers
annotated the dataset with five named entities. The samples belong to one of
the following four historical regions of Romania, namely Bessarabia, Moldavia,
Transylvania, and Wallachia. We employed this proposed dataset to perform
several experiments for NER using Romanian pre-trained language models. Our
results show that the best model achieved a strict F1-score of 55.69%. Also,
by reducing the discrepancies between regions through a novel domain adaption
technique, we improved the performance on this corpus to a strict F1-score of
66.80%, representing an absolute gain of more than 10%.

###### Keywords:

Named Entity Recognition Historical Newspapers Romanian Language Novel Dataset
Transformer.

##  1 Introduction

A key area of artificial intelligence (AI) is natural language processing
(NLP), concerned with how computers and human language interact. Its
fundamental goal is to empower machines to comprehend, interpret, and reply to
human languages in a valuable and meaningful manner [2]. This field has made
significant contributions to the growth of AI by developing algorithms and
models that facilitate machine understanding of text and speech, enabling
applications such as automatic translation [26], sentiment analysis [11], and
chatbots [18].

Named Entity Recognition (NER) is a crucial part of nearly all NLP systems. It
is responsible for identifying and categorizing entities, which include
persons, organizations, dates, and organizations [33]. In the context of
historical documents, NER helps historians and researchers sift through vast
amounts of textual data, automatically identifying and categorizing key
entities. This speeds up the process of comprehending historical content and
helps to extract insightful information and connections that could have been
difficult or time-consuming to obtain through manual analysis alone [14].

As a testament to the growing importance of NER in historical research, we
introduce a novel corpus - the Historical Named Entity Recognition for the
Romanian language (HistNERo). This dataset addresses the unique linguistic
nuances of the old Romanian language. It encompasses a diverse range of
historical documents spanning various periods (i.e., from 1817 to 1990) from
four regions where Romanian was historically spoken: Bessarabia, Moldavia,
Transylvania, and Moldavia. HistNERo was meticulously annotated with five
named entities (i.e., PERSON, ORGANIZATION, LOCATION, PRODUCT, and DATE), as
proposed by the first edition of HIPE (Identifying Historical People, Places
and other Entities) [13], employing a group of eight annotators, two for each
region. The dataset comprises 10,026 sentences that contain 323,865 tokens and
9,601 annotated entities. The dataset was gathered from the ROmanian
DIachronic Corpus with Annotations (RODICA) corpus [7].

Building upon the creation of the HistNERo dataset, we conducted extensive
experiments to evaluate its effectiveness in conjunction with several Romanian
large language models available in the literature. Our results outlined that
the best-performing model is RoBERT-base [21], which obtained a strict
F1-score of 55.69% on this dataset. Additionally, since the historical
documents were collected from different historical regions of Romania, each
one with its linguistical characteristics, we experimented with a novel domain
adaptation technique called loss reversal, which reverses the sign of the
domain discriminator loss instead of the gradient before entering the feature
extractor, as proposed by Ganian et al. [15]. As the results suggest,
employing this technique allows the large language models to better
distinguish between the four regions. Thus, the performance of our best-
performing model, RoBERT-base, was raised to 63.85% strict F1-score and even
surpassed by RoBERT-large, which obtained a strict F1-score of 66.80%.

To summarize, our contribution is threefold:

  * •

We make available under an open
license111https://huggingface.co/datasets/avramandrei/histnero a historical
corpus of 323k tokens annotated with five named entities by a group of
annotators.

  * •

We analyze the performance of various Romanian pre-trained language models on
this dataset.

  * •

We propose a new technique (called loss reversal) to improve the performance
of most of the tested models by forcing them to learn embeddings that are more
easily separable in the latent space.

##  2 Related Work

###  2.1 Named Entities in Historical Documents

Recent research on NER in historical texts highlighted the challenges posed by
noisy and heterogeneous inputs [12]. As part of the Europeana Newspapers
project, Neudecker [27] detailed the creation of 100-page datasets for NER in
historical newspapers for the languages Dutch, French, and German. Hubková et
al. [17] presented a NER corpus of Czech historical newspapers. Acknowledging
the problems associated with historical text, the authors propose an entity
class named ambiguous for text spans; however, the annotator is unsure of the
correct type. Historical newspaper texts produced in French, German, Finnish,
and Swedish between 1850 and 1950 are available in the NewsEye corpus [16],
which has been annotated for NER.

Additionally, to enable the performance comparison of NER on historical texts
and to improve the robustness of current NER systems on non-standard inputs,
the first edition of the HIPE competition (named HIPE-2020) [13] was proposed
in 2020. The participants had to forecast the same five entities as in
HistNERo, and it was aimed at historical newspapers in French, German, and
English. However, in addition to our task, the participants had to predict
finer-grained entities, as well as link the entities to a unique referent in a
knowledge database (i.e., Wikipedia or the NIL node if the referent for the
mention was not located in the database). The La Rochelle University-
affiliated L3i team generally obtained the highest scores, which approached
the task in a multi-task setting using a hierarchical transformer-based model
[46].

The last HIPE competition (named HIPE-2022) [14] was held in 2022, with the
primary goal of providing new perspectives on the transferability of NER. The
competition presented participants with the difficulties of multilingualism,
as well as customizing their systems to domain-specific entities and various
annotation sets. More specifically, HIPE-2022 focused on the same three tasks
as HIPE-2020. In addition to working with historical newspapers, the HIPE-2022
dataset also included classical commentaries; the former were gathered from
five languages (English, German, French, Finnish, and Swedish), while the
latter were collected from three languages (English, German, and French).
However, compared to HIPE-2020, there were no clear winners on this dataset,
with a different team ranking first on diverse subtasks.

###  2.2 Romanian Named Entity Recognition

Considering the Romanian language, ongoing activities aim to make historical
documents available for research. In this context, the goal of the DeLORo
project [6] is to develop technology that can translate ancient printed and
uncial Cyrillic-Romanian documents into Latin characters.

However, most NER research for the Romanian language is currently focused on
contemporary language. RoNEC [9] is a corpus covering standard NE classes and
expressions (i.e., person, location, organization, and date) and additional
NEs, such as money, quantity, and work of art. LegalNERo [40] is a corpus with
NEs in the Romanian legal domain, constructed primarily on legislation,
extracted from the larger MARCELL-RO corpus [45, 44]. In addition to common
entity classes, it provides legal references. These were further refined by
Costea et al. [5]. NER systems in the legal domain were trained by [37, 35].
SiMoNERo [23, 3] is a NER extension to the MoNERo corpus [24]. SiMoNERo
contains NEs specific to the biomedical domain, such as anatomical parts,
chemicals, disorders, and procedures. [25] proposed improving the recognition
of biomedical NEs using a biologically inspired lateral inhibition approach.
MicroBloggingNERo [38, 39] is a corpus constructed based on social media text.
It combines SiMoNERo and LegalNERo annotation schemes and adapts them to the
social media context. A NER system based on MicroBloggingNERo was introduced
by [29].

##  3 HistNERo Corpus

###  3.1 RELATE Platform

Dataset annotation was realized in the RELATE
platform222https://relate.racai.ro. RELATE [30] is a modern platform
incorporating a large number of tools [36] for processing the Romanian
language. It was previously used for automatic annotation of large corpora,
such as the MARCELL legislative corpus [45, 44] and the CURLICAT corpus [47],
as well as for creating Romanian language named entity corpora, such as
MicroBloggingNERo and LegalNERo. The platform offers modules for corpus
management, automatic annotations, and manual NE annotation. The manual NE
annotation is accomplished at the span level employing the integrated BRAT
component [42]. From the corpus management module, the annotator opens a
document in the BRAT component and selects a text span with the mouse to which
an NE class will be associated. The RELATE platform keeps track of the last
annotated document, making it easier for the annotator to resume work or
access the following document in his allocated batch.

###  3.2 Annotation Process

The annotation of the HistNERo dataset was performed by following the
guidelines333https://zenodo.org/records/3604227 of the HIPE-2020 competition,
using a group of eight annotators, two for each region. When several entities
were in a sequence, we annotated only the longest one. Also, we did not
annotate nested entities. The annotation process took about four months and
involved a process composed of four steps as follows:

  1. 1.

In the first step of the process, the annotators of each region annotated the
first two documents, resulting in an average Cohen’s Kappa (CK) inter-
annotator agreement (IAA) of 76.4% and an average F1-score of 66.7%, as
depicted in Table 1. Also, as can be observed in Table 2, the highest IAA was
on the DATE entity, which usually is easier to identify and correctly
annotate, even on historical documents. On the other hand, the PRODUCT entity
is more challenging to determine in the old text due to the archaisms and
regionalisms usually used to denote this category of entities. As a result, it
obtained the lowest IAA score, i.e., a CK of 15.8% and an F1-score of 7.3%.

  2. 2.

In the second step of the process, the annotators of each region discussed how
they annotated the entities in the first two documents and then individually
re-annotated the documents. This significantly boosted the IAA, resulting in
an average 92.2% CK and an average 86.3% F1-score. Particularly, this
discussion boosted the performance on the PRODUCT entity to a CK of 50.1% and
an F1-score of 38.6%.

  3. 3.

In the third step of the process, the rest of the documents in HistNERo were
annotated, resulting in an average CK of 79.3% and an average F1-score of
72.5%.

  4. 4.

In the fourth step of the process, the two annotators of each region agreed on
the correct way of annotating the remaining entities in disagreement in
HistNERo.

Table 1: For each of the four regions, the inter-annotator agreement on the first two documents before the first annotation discussion (i.e., Step 1), after the first annotation discussion (i.e., Step 2), and after all the documents have been annotated (i.e., Step 3). Region | Step 1 | Step 2 | Step 3
---|---|---|---
CK |  F1 |  CK |  F1 |  CK |  F1
Bessarabia |  72.4 |  63.9 |  92.5 |  89.56 |  80.9 |  75.4
Moldavia |  86.6 |  87.7 |  94.9 |  90.0 |  77.3 |  71.7
Transylvania |  67.8 |  52.6 |  89.9 |  81.2 |  77.8 |  69.8
Wallachia |  78.9 |  62.5 |  91.3 |  84.5 |  81.2 |  73.0
Table 2: For each of the five annotated named entities, the inter-annotator agreement on the first two documents before the first annotation discussion (i.e., Step 1), after the first annotation discussion (i.e., Step 2), and after all the documents have been annotated (i.e., Step 3). Named Entity | Step 1 | Step 2 | Step 3
---|---|---|---
CK |  F1 |  CK |  F1 |  CK |  F1
PERSON |  71.2 |  65.4 |  95.6 |  90.4 |  82.1 |  77.3
ORGANISATION |  60.7 |  39.6 |  84.5 |  73.1 |  75.7 |  63.2
LOCATION |  75.2 |  83.5 |  96.0 |  90.9 |  80.8 |  83.0
PRODUCT |  15.8 |  7.3 |  50.1 |  38.6 |  44.0 |  27.6
DATE |  95.0 |  82.7 |  98.8 |  87.4 |  91.3 |  84.9

###  3.3 Data Preprocessing

The output of the annotation processes consisted of files in the BRAT format
that cannot be directly fed into a model; thus, they required preprocessing.
The manner we approached this challenge was to tokenize the sentences in
HistNERo using spaCy444https://spacy.io and then automatically align the
annotated entities based on their start and end character indexes. Moreover,
we also annotate the resulting labels with the inside-outside-beginning format
[41], ensuring an unambiguous representation of the entities.

###  3.4 HistNERo Statistics

The resulting corpus contains 10,026 sentences and 323,865 tokens, annotated
with 9,601 entities and 17,015 entity tokens, as depicted in Table 3. Most of
the annotated entities belong to the LOCATION with 3,032 instances, closely
followed by the PERSON entity with 2,914 instances, and the least amount of
annotated entities belong to the PRODUCT entity with 424 instances. This
distribution illustrates how most of the collected historical documents have
focused more on people and places than on the goods those societies produced
at the time.

Table 3: Named entity statistics on HistNERo.  Named Entity | Region |  Total # Tokens |  Total # Entities |  # Tokens per Entity
---|---|---|---|---
PERSON | Bessarabia |  1,573 |  732 |  2.14
Moldavia |  1,270 |  689 |  1.84
Transylvania |  1,803 |  788 |  2.41
Wallachia |  1,204 |  705 |  1.70
Total |  5,950 |  2,914 |  2.04
ORGANISATION | Bessarabia |  1,268 |  481 |  2.63
Moldavia |  253 |  145 |  1.74
Transylvania |  1,048 |  509 |  2.05
Wallachia |  1,768 |  1,283 |  1.37
Total |  4,337 |  2,418 |  1.73
LOCATION | Bessarabia |  1,276 |  959 |  1.36
Moldavia |  449 |  383 |  1.17
Transylvania |  1,056 |  733 |  1.44
Wallachia |  1,073 |  957 |  1.12
Total |  3,854 |  3,032 |  1.27
PRODUCT | Bessarabia |  66 |  30 |  2.20
Moldavia |  19 |  7 |  2.71
Transylvania |  339 |  72 |  4.70
Wallachia |  636 |  315 |  2.29
Total |  1,060 |  424 |  2.50
DATE | Bessarabia |  456 |  186 |  2.45
Moldavia |  319 |  151 |  2.11
Transylvania |  574 |  250 |  2.29
Wallachia |  465 |  226 |  2.05
Total |  1,814 |  813 |  2.23
Total | - |  17,015 |  9,601 |  1.77

###  3.5 Dataset Comparison

Compared to other NER datasets available in the Romanian language (see Table
4), HistNERO ranks second in terms of the number of annotated sentences and
the number of total tokens, with 10,026 sentences and 323,865 tokens, being
surpassed only by the second version of the Romanian Named Entity Corpus
(RONECv2)555https://github.com/dumitrescustefan/ronec, which has 12,330
sentences and 553,163 tokens. However, HistNERo includes the fewest annotated
entities (i.e., 9,601). This number is influenced by both the frequency of
entities in historical documents (HistNERo contains less than one entity per
sentence) and by how many entities were used to annotate the corpus (e.g., 16
entities are used in RONECv1 and 15 in RONECv2).

Table 4: HistNERo statistics (training, validation, test, and total) compared to other NER datasets in the Romanian language. Dataset |  # Sents. |  # Tokens |  # Entities |  # Entities Tokens |  # Tokens per Entity
---|---|---|---|---|---
RONECv1 [9] |  5,127 |  180,467 |  26,377 |  50,760 |  2.13
RONECv2 |  12,330 |  553,163 |  80,283 |  50,760 |  2.13
LegalNERo [31] |  8,284 |  264,335 |  13,614 |  54,456 |  4.00
MBNERo [32] |  7,800 |  207,076 |  11,099 |  16,745 |  1.50
SiMoNERo [22] |  5,418 |  163,707 |  15,493 |  25,563 |  1.65
HistNERo-Train |  8,020 |  251,651 |  6,686 |  11,967 |  1.79
HistNERo-Valid |  1,003 |  36,082 |  1,465 |  2,505 |  1.71
HistNERo-Test |  1,003 |  36,132 |  1,485 |  2,543 |  1.71
HistNERo-Total |  10,026 |  323,865 |  9,601 |  17,015 |  1.77

###  3.6 TF-IDF-based Data Analysis

We further perform a Term Frequency-Inverse Document Frequency (TF-IDF)
analysis on the documents of each region in HistNERo. The statistical
technique known as TF-IDF consists of two parts: (1) TF measures the frequency
with which a word appears in a document, and (2) IDF shows the term’s
commonality among all documents [43]. Thus, the TF-IDF score is the product of
TF and IDF, with higher TF-IDF scores indicating words that are uncommon
overall in the collection and frequent in a particular document.

We depict the results of this analysis in Table 5 from which we can observe
that several common words like "year(s)" or "government" obtain a high TF-IDF
score in more than one region. Also, we can notice that the documents
available for the Bessarabia region may contain a more nationalistic tone
since the region’s name or words like "country" are often used and are given
higher importance by the TF-IDF algorithm.

Table 5: TF-IDF analysis of the top five most common words found in each region of HistNERo. Region |  Romanian Words |  Translation |  TF-IDF Score
---|---|---|---
Bessarabia |  basarabia |  Bessarabia | 1.699
ani |  years | 1.138
tării |  country’s | 1.086
basarabiei |  Bessarabia’s | 1.081
moldovei |  Moldavia’s | 1.061
Moldavia |  lei |  lions | 0.603
guvernul |  government | 0.476
ziarul |  newspaper | 0.476
ani |  year | 0.413
aviatie |  aviation | 0.407
Transylvania |  voru |  want | 0.807
politică |  politics | 0.687
anulu |  year | 0.671
romanu |  Romanian | 0.663
suntu |  are | 0.660
Wallachia |  făcut |  done | 0.801
românia |  Romania | 0.697
berlin |  Berlin | 0.659
guvernul |  government | 0.633
tara |  country | 0.609

##  4 Method

###  4.1 Baseline Models

Regarding the architectures selected for training, transformer-based model
fine-tuning has emerged as the de facto method for resolving NLP tasks in
recent years [4]. In our methodology, we employ several such models trained
specifically on Romanian text.

Therefore, we assess four families of pre-trained language models for Romanian
on HistNERo as follows: the cased and uncased versions of BERT-ro (i.e., BERT-
ro-c and BERT-ro-uc) [8], the RoBERT versions (i.e., RoBERT-small, RoBERT-
base, and RoBERT-large) [21], the cased and uncased versions of the distilled
Romanian BERT (i.e., DistRoBERT-c and DistRoBERT-uc) [1], and the three
versions of the Romanian GPT (i.e., RoGPT-base, RoGPT-medium, and RoGPT-large)
[28].

###  4.2 Domain Adaptation

Additionally, since the documents in HistNERo were collected from different
historical regions of Romania, each with its linguistic features, we also
employ a novel domain adaptation method that helps our foundational models
learn region-independent features, boosting performance on the NER task.

We name this domain adaptation technique as loss reversal, and it is inspired
by the gradient reversal method in the domain adaptation technique [15]. More
precisely, given a feature extractor $F$ whose role is to extract features
from the input data, a label classifier $C$ whose role is to classify the
features into the classes $y$, and a domain discriminator $D$ whose role is to
classify the given features into the domains $d$, we reverse the gradient
coming from the domain discriminator before backpropagating into the feature
extractor. The following equations explain the gradient reversal [15]:

$\theta_{C}=\theta_{C}-\alpha\frac{\partial L_{y}}{\partial\theta_{C}}$ (1)

$\theta_{D}=\theta_{D}-\alpha\frac{\partial L_{d}}{\partial\theta_{D}}$ (2)

$\theta_{F}=\theta_{F}-\alpha\biggl{(}\frac{\partial
L_{y}}{\partial\theta_{F}}-\lambda\frac{\partial
L_{d}}{\partial\theta_{F}}\biggr{)}$ (3)

where $L_{y}$ is the label classifier’s loss, $\theta_{C}$ represents the
label classifier’s parameters, and $y$ are the labels. The domain
discriminator’s parameters are $\theta_{D}$, and the loss it incurs when
predicting the domain labels $d$ is $L_{d}$. The feature extractor’s
parameters are $\theta_{F}$. Moreover, the learning rate is $\alpha$, and the
hyperparameter $\lambda$ is used to scale the reversed gradients.

Figure 1: The loss reversal algorithm applied to the third token of a
tokenized sentence. We note that FF stands for a feed-forward layer.

To predict the NER label of each token using the domain adaption technique
proposed in this paper, called loss reversal, we just switch the sign of the
domain discriminator loss, as depicted in Figure 1. In addition, the following
equation provides a clear description of the loss reversal employed in this
work:

$\mathcal{L}^{i}=\mathcal{L}_{y}^{i}-\lambda\mathcal{L}_{r}^{i}$ (4)

where $\mathcal{L}^{i}$ is the total loss of our system for the token $i$,
$\mathcal{L}_{y}^{i}$ is the NER loss for the token $i$, $\mathcal{L}_{r}^{i}$
is the region loss for the token $i$, and $\lambda$ is a hyperparameter that
controls the strength of the adversarial loss.

We believe that the loss reversal technique is much easier to implement than
the gradient reversal in almost any framework, and, at least on the NER task
applied to historical documents, we obtain better results, as it will be
emphasized in Section 6. However, more experiments on other tasks are required
to confirm the general higher performance of our method, but such experiments
would be beyond the scope of this paper.

##  5 Implementation Details

We train the models for 15 epochs, using the Adam optimizer [19] with a
learning rate of $2e-5$, a weight decay of $0.01$, and a batch size of 32 on
an NVIDIA A100 GPU. Furthermore, we stabilize the training process by clipping
[34] the gradient of the backpropagation algorithm to 2. We also employ the
gradient checkpoint method to reduce memory consumption. To control the
strength of the loss reversal, we set the hyperparameter $\lambda$ to 0.1,
which we empirically observed to produce the best results in general.
Furthermore, we split the HistNERo dataset into the train, validation, and
test subsets, maintaining an 80%, 10%, and 10% ratio. We made this dataset
splitting publicly available to enhance reproducibility and the ease of
comparison with our results.

##  6 Results

The results of all the evaluated language models on HistNERo are depicted in
Table 6 considering the four regions and the total performance. Also, Table 7
shows the performance of each of the five individual named entities. We
measure the accuracy and the strict F1-score when we evaluate based on the
regions and only the strict F1-score when we evaluate based on the named
entities.

Table 6: The results of our evaluated language models on the four regions found in HistNERo: Bessarabia (Bess.), Moldavia (Mold.), Transylvania (Trans.), and Wallachia (Wall.), with the gradient reversal (Grad. Rev.) and loss reversal (Loss Rev.) domain adaptation algorithms. Model | Bess. | Mold. | Trans. | Wall. | Total
---|---|---|---|---|---
Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1
BERT-ro-c [8] |  97.60 |  66.93 |  99.30 |  60.46 |  94.69 |  39.77 |  97.07 |  53.11 |  97.18 |  55.03
w/ Grad. Rev. |  97.92 |  74.60 |  97.98 |  52.27 |  95.74 |  51.08 |  97.67 |  65.04 |  97.35 |  62.51
w/ Loss Rev. |  97.95 |  75.69 |  97.83 |  50.05 |  96.50 |  61.53 |  97.80 |  66.92 |  97.55 |  66.34
BERT-ro-uc [8] |  97.42 |  63.24 |  99.10 |  40.09 |  94.22 |  31.95 |  96.87 |  52.96 |  96.93 |  50.57
w/ Grad. Rev. |  97.06 |  66.91 |  97.13 |  30.58 |  96.41 |  49.35 |  97.27 |  60.71 |  96.77 |  56.61
w/ Loss Rev. |  97.68 |  70.10 |  97.50 |  34.66 |  95.77 |  54.62 |  97.07 |  53.11 |  97.18 |  55.03
RoBERT-small [21] |  97.57 |  66.93 |  98.68 |  48.14 |  93.96 |  41.62 |  97.02 |  50.37 |  96.87 |  53.03
w/ Grad. Rev. |  97.03 |  66.81 |  97.90 |  3.73 |  95.29 |  42.10 |  97.39 |  55.06 |  96.92 |  53.80
w/ Loss Rev. |  97.60 |  66.93 |  99.30 |  60.46 |  94.69 |  39.77 |  97.07 |  53.11 |  97.18 |  55.03
RoBERT-base [21] |  97.47 |  67.21 |  99.33 |  61.90 |  94.82 |  35.29 |  97.71 |  57.25 |  97.21 |  55.69
w/ Grad. Rev. |  97.87 |  78.74 |  97.98 |  42.50 |  96.62 |  57.00 |  97.69 |  65.04 |  97.56 |  64.97
w/ Loss Rev. |  97.87 |  75.50 |  98.09 |  49.38 |  96.26 |  54.37 |  97.77 |  65.11 |  97.51 |  63.85
RoBERT-large [21] |  97.42 |  64.16 |  99.13 |  50.01 |  95.28 |  39.49 |  97.05 |  51.88 |  97.22 |  53.25
w/ Grad. Rev. |  97.49 |  69.69 |  97.28 |  41.56 |  95.35 |  48.94 |  97.63 |  63.11 |  97.01 |  58.83
w/ Loss Rev. |  98.00 |  76.67 |  97.83 |  45.45 |  96.68 |  59.72 |  97.90 |  69.96 |  97.64 |  66.80
DistRoBERT-c [1] |  96.28 |  55.97 |  97.79 |  44.18 |  95.80 |  53.11 |  97.37 |  52.30 |  96.81 |  52.86
w/ Grad. Rev. |  94.98 |  46.54 |  97.57 |  48.57 |  95.32 |  43.90 |  96.36 |  38.23 |  96.00 |  42.73
w/ Loss Rev. |  96.82 |  62.15 |  97.17 |  37.83 |  95.50 |  57.33 |  97.08 |  49.21 |  96.67 |  53.32
DistRoBERT-uc [1] |  97.25 |  69.84 |  97.87 |  45.56 |  94.80 |  46.61 |  97.42 |  57.95 |  96.86 |  57.14
w/ Grad. Rev. |  97.22 |  66.14 |  97.24 |  37.36 |  94.29 |  45.90 |  97.29 |  51.36 |  96.58 |  52.76
w/ Loss Rev. |  97.33 |  69.67 |  97.68 |  44.15 |  94.56 |  46.53 |  97.10 |  52.34 |  96.69 |  54.99
RoGPT2-base [28] |  96.76 |  62.81 |  97.35 |  35.41 |  95.32 |  40.00 |  97.63 |  54.61 |  96.83 |  51.15
w/ Grad. Rev. |  96.33 |  55.36 |  97.39 |  38.63 |  94.98 |  38.84 |  97.27 |  50.38 |  96.54 |  47.66
w/ Loss Rev. |  96.52 |  54.67 |  97.65 |  46.75 |  94.98 |  36.66 |  96.82 |  49.43 |  96.48 |  47.51
RoGPT2-med [28] |  97.17 |  63.00 |  97.17 |  30.76 |  95.20 |  40.90 |  97.46 |  54.76 |  96.82 |  50.98
w/ Grad. Rev. |  96.82 |  62.45 |  96.98 |  38.53 |  95.02 |  40.31 |  97.12 |  52.77 |  96.54 |  50.69
w/ Loss Rev. |  96.28 |  57.24 |  97.39 |  46.31 |  95.17 |  38.59 |  97.18 |  49.62 |  96.53 |  48.78
RoGPT2-large [28] |  97.09 |  66.66 |  97.32 |  36.17 |  94.89 |  40.32 |  97.63 |  56.71 |  96.81 |  51.01
w/ Grad. Rev. |  96.55 |  58.62 |  97.35 |  41.58 |  94.92 |  40.57 |  97.11 |  51.18 |  96.49 |  49.57
w/ Loss Rev. |  97.39 |  61.12 |  97.52 |  37.17 |  94.99 |  41.32 |  97.83 |  57.71 |  97.21 |  51.83
Table 7: The strict F1-scores of our evaluated language models on the five named entities (i.e., PERS → PERSON, ORG → ORGANISATION, LOC → LOCATION, PROD → PRODUCT, and DATE) found in HistNERo, with the gradient reversal (Grad. Rev.) and loss reversal (Loss Rev.) domain adaptation algorithms. Model |  PERS |  ORG |  LOC |  PROD |  DATE
---|---|---|---|---|---
BERT-ro-c [8] |  54.76 |  53.68 |  65.83 |  21.05 |  40.57
w/ Grad. Rev. |  62.29 |  55.14 |  69.70 |  40.13 |  75.40
w/ Loss Rev. |  66.66 |  61.12 |  71.42 |  41.17 |  78.12
BERT-ro-uc [8] |  40.96 |  52.63 |  66.10 |  10.81 |  35.29
w/ Grad. Rev. |  55.31 |  51.13 |  61.53 |  32.25 |  72.72
w/ Loss Rev. |  54.76 |  53.68 |  65.83 |  21.05 |  40.57
RoBERT-small [21] |  47.05 |  45.74 |  69.44 |  24.00 |  41.93
w/ Grad. Rev. |  51.79 |  45.83 |  64.62 |  26.66 |  65.62
w/ Loss Rev. |  54.76 |  53.68 |  65.83 |  21.05 |  40.57
RoBERT-base [21] |  58.22 |  56.15 |  66.93 |  10.85 |  32.23
w/ Grad. Rev. |  68.31 |  53.08 |  70.99 |  46.15 |  81.25
w/ Loss Rev. |  64.46 |  56.30 |  69.29 |  44.44 |  81.96
RoBERT-large [21] |  55.71 |  47.31 |  63.56 |  10.81 |  41.26
w/ Grad. Rev. |  58.48 |  49.79 |  69.78 |  33.35 |  73.01
w/ Loss Rev. |  67.43 |  60.60 |  74.67 |  32.43 |  76.19
DistRoBERT-c [1] |  52.41 |  44.94 |  61.66 |  30.30 |  65.67
w/ Grad. Rev. |  38.09 |  30.43 |  54.48 |  12.15 |  52.16
w/ Loss Rev. |  58.13 |  39.63 |  66.13 |  22.05 |  73.08
DistRoBERT-uc [1] |  58.92 |  47.30 |  64.73 |  27.50 |  73.33
w/ Grad. Rev. |  49.63 |  56.78 |  66.47 |  16.39 |  63.76
w/ Loss Rev. |  50.57 |  47.74 |  66.93 |  17.76 |  70.96
RoGPT2-base [28] |  44.36 |  47.35 |  64.82 |  27.58 |  52.45
w/ Grad. Rev. |  37.53 |  44.25 |  62.50 |  16.21 |  59.74
w/ Loss Rev. |  41.73 |  41.76 |  62.55 |  13.33 |  51.42
RoGPT2-med [28] |  48.05 |  39.65 |  65.58 |  16.66 |  64.61
w/ Grad. Rev. |  46.10 |  42.90 |  67.20 |  27.90 |  55.96
w/ Loss Rev. |  44.03 |  41.89 |  63.77 |  10.05 |  50.13
RoGPT2-large [28] |  47.26 |  40.81 |  66.14 |  29.41 |  58.46
w/ Grad. Rev. |  46.70 |  51.37 |  64.75 |  20.00 |  56.75
w/ Loss Rev. |  48.27 |  43.14 |  66.30 |  12.5 |  57.14

The highest overall accuracy was obtained by RoBERT-large with 97.22%, and the
highest overall strict F1-score was obtained by DistilRoBERT-uc with 57.14%.
When it comes to each individual region, the highest strict F1-score was
obtained by DistilRoBERT-uc for the Bessarabia region with 69.84%, by RoBERT-
base for the Moldavia region with 61.90%, by DistilRoBERT-c for the
Transylvania region with 53.11%, and by DistilRoBERT-uc for the Wallachia
region with 57.95%. On the other hand, when considering the performance on
each individual category of named entities, the highest strict F1-score was
obtained by DistRoBErT-uc for the PERSON entity with 58.92%, by RoBERT-base
for the ORGANIZATION entity with 56.15%, by RoBERT-small for the LOCATION
entity with 69.44%, by DistilRoBERT-c for the PRODUCT entity with 30.30%, and
by DistRoBERT-uc for the DATE entity with 73.33%.

By incorporating the two domain adaptation techniques we experimented with in
this work, the gradient and loss reversal, we significantly boosted the
performance of our language models. Specifically, we improved the performance
of 7 out of 10 models, with an average relative improvement of 1.40% for the
gradient reversal and an average relative improvement of 6.36% for the loss
reversal. Furthermore, we achieved a new highest overall accuracy of 97.64%
and a new highest overall strict F1-score of 66.80% with RoBERT-large and the
loss reversal algorithm. When considering each region, a significant boost can
be observed in the Transylvania region, which obtained a relative strict
F1-score score improvement of 16.34% with the gradient reversal algorithm and
a relative strict F1-score improvement of 20.63% with the loss reversal
algorithm. On the other hand, the lowest relative strict F1-scores
improvements were observed for the Moldavia region with -12.63% when the
gradient reversal was employed and 2.32% when the loss reversal was used.

By looking at the relative strict F1-scores of each named entity, we can
observe that the highest average improvements were obtained on the DATE named
entity with 72.79% for the gradient reversal algorithm and 47.61% for the loss
reversal algorithm. We believe that this is because this entity is the easiest
to standardize between regions compared to other more region-specific named
entities such as PERSON or ORGANIZATION, and it benefits much more from domain
adversarial training whose aim in this work is to create region independent
features. On the other hand, the lowest strict F1-scores improvements were
obtained on the PRODUCT named entity with -0.43% for the gradient reversal
algorithm and 2.85% for the loss reversal algorithm. We believe that low
improvement is due to the rarity of this named entity in the dataset, which
affects the overall performance and possible refinements. Other strategies for
class imbalance should be used to significantly improve the performance on the
PRODUCT named entity.

###  6.1 Inter-Regional Evaluation

Figure 2: Inter-regional strict F1-scores of the RoBERT-large model.

We also tested the inter-regional performance of RoBERT-large (i.e., the best-
performing model in Table 6) by training it on the subset determined by each
region and then evaluating the strict F1-score of the model on each of the
four areas. The results are depicted in Figure 2. We can observe that the
highest strict F1-scores were obtained on the main diagonal, which is a
regional evaluation of the model (i.e., trained and evaluated on the same
region), with the Bessarabia region leading the benchmark and obtaining a
strict F1-score of 69.23%. On the other hand, the highest inter-regional
strict F1-score was obtained when RoBERT-large was trained on the Transylvania
region and evaluated on the Wallachia region, achieving a strict F1-score of
49.37%, at less than 1% difference compared to the intra-regional result,
showing the possible similarities between the two geographical areas.
Similarly, we can also observe that the results of the inter-regional
evaluation between the Bessarabia and Moldavia regions are also well
correlated, demonstrating once more the historical closer ties that existed
between the two areas compared to other parts of Romania.

###  6.2 T-SNE Visualizations

Figure 3: Clustering of the HistNERo test set embeddings produced by the BERT-
large model using t-SNE with and without loss reversal.

We further analyze the embeddings produced by BERT-large on the HistNERo test
set to understand the effects of the loss reversal algorithm. Thus, we average
the embeddings produced by BERT-large on each token and visualize the results
using t-SNE [20]. The results are depicted in Figure 3. We can notice that
without the loss reversal, there is no clear separation between the
embeddings, making the model less robust when challenged with linguistic
variations from the Romanian regions. On the other hand, when the loss
reversal algorithm is employed, we can easily observe a clear separation in
the embedding space between the Bessarabia, Transylvania, and Wallachia
regions. However, BERT-large struggles to create a separate cluster for the
Moldavia region. We believe that this is because Moldavia and Bessarabia were
part of the same country for much of their history, and there are few
linguistic differences between them. Thus, the model struggled to create
another cluster in the embedding space for the Moldavia region.

##  7 Conclusion and Future Work

Analyzing historical documents is essential in research, providing vital
insights for a nuanced understanding of the past. Thus, we introduced HistNERo
in this work, a valuable resource that will advance the state of the art in
historical NER, facilitating a more accurate and nuanced analysis of Romanian
historical documents and contributing to a deeper understanding of the
country’s historical evolution. Additionally, since the HistNERo corpus
contains documents collected from various historical regions of Romania, we
developed a novel domain adaptation that adds the region prediction loss to
the overall optimization target but with the reversed sign. Our experimental
results outlined that, by incorporating this simple technique, the performance
of our models relatively improved by an average of 6.36%, the best model with
the loss reversal achieved a strict F1-score of 66.80%, an absolute
improvement of more than 10% compared to the best model that did not employ
any domain adaptation technique. Finally, we intend to incorporate both the
HistNERo dataset and the results obtained by the evaluated models in the LiRo
benchmark [10].

## Acknowledgements

This work was supported by a grant from the National Program for Research of
the National Association of Technical Universities - GNAC ARUT 2023.

## References

  * [1] Avram, A.M., Catrina, D., Cercel, D.C., Dascalu, M., Rebedea, T., Păis, V., Tufiş, D.: Distilling the knowledge of romanian berts using multiple teachers. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference. pp. 374–384 (2022)
  * [2] Avram, A.M., Mititelu, V.B., Pais, V., Cercel, D.C., Trausan-Matu, S.: Multilingual multiword expression identification using lateral inhibition and domain adaptation. Mathematics 11(11), 2548 (2023)
  * [3] Barbu Mititelu, V., Mitrofan, M.: The romanian medical treebank - simonero. In: Barbu Mititelu, V., Irimia, E., Tufis, D., Dan, C. (eds.) Proceedings of the 15th International Conference Linguistic Resources and Tools for Natural Language Processing. pp. 7–16. online (December 2020)
  * [4] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021)
  * [5] Costea, D., Moldovan, A., Păis, V.: Fine-grained legal references in the legalnero corpus. In: Proceedings of the 17th Edition of the International Conference on Linguistic Resources and Tools for Natural Language Processing (CONSILR 2022). Chisinău, Moldova (2022), https://profs.info.uaic.ro/~consilr/2022/program/
  * [6] Cristea, D., Pădurariu, C., Rebeja, P., Scutelnicu, A., Onofrei, M.: Data structure and acquisition in deloro - a technology for deciphering old-cyrillic-romanian document. In: Proceedings of the 16th International Conference Linguistic Resources and Tools for Natural Language Processing. pp. 59–74 (2021)
  * [7] Daniela, G.: Recovering old romanian lemmata. In: Conference proceedings of» eLearning and Software for Education «(eLSE). vol. 13, pp. 19–24. Carol I National Defence University Publishing House (2017)
  * [8] Dumitrescu, S., Avram, A.M., Pyysalo, S.: The birth of romanian bert. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 4324–4328 (2020)
  * [9] Dumitrescu, Ş.D., Avram, A.M.: Introducing ronec-the romanian named entity corpus. In: Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 4436–4443 (2020)
  * [10] Dumitrescu, S.D., Rebeja, P., Lorincz, B., Gaman, M., Avram, A., Ilie, M., Pruteanu, A., Stan, A., Rosia, L., Iacobescu, C., et al.: Liro: Benchmark and leaderboard for romanian language tasks. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2021)
  * [11] Echim, S.V., Smădu, R.A., Avram, A.M., Cercel, D.C., Pop, F.: Adversarial capsule networks for romanian satire detection and sentiment analysis. In: International Conference on Applications of Natural Language to Information Systems. pp. 428–442. Springer (2023)
  * [12] Ehrmann, M., Hamdi, A., Pontes, E.L., Romanello, M., Doucet, A.: Named entity recognition and classification in historical documents: A survey. ACM Comput. Surv. 56(2) (sep 2023). https://doi.org/10.1145/3604931
  * [13] Ehrmann, M., Romanello, M., Flückiger, A., Clematide, S.: Overview of clef hipe 2020: Named entity recognition and linking on historical newspapers. In: Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11. pp. 288–310. Springer (2020)
  * [14] Ehrmann, M., Romanello, M., Najem-Meyer, S., Doucet, A., Clematide, S.: Overview of hipe-2022: named entity recognition and linking in multilingual historical documents. In: International Conference of the Cross-Language Evaluation Forum for European Languages. pp. 423–446. Springer (2022)
  * [15] Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International conference on machine learning. pp. 1180–1189. PMLR (2015)
  * [16] Hamdi, A., Linhares Pontes, E., Boros, E., Nguyen, T.T.H., Hackl, G., Moreno, J.G., Doucet, A.: A multilingual dataset for named entity recognition, entity linking and stance detection in historical newspapers. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 2328–2334. SIGIR ’21, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3404835.3463255, https://doi.org/10.1145/3404835.3463255
  * [17] Hubková, H., Kral, P., Pettersson, E.: Czech historical named entity corpus v 1.0. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 4458–4465. European Language Resources Association, Marseille, France (May 2020), https://aclanthology.org/2020.lrec-1.549
  * [18] Kim, C.Y., Lee, C.P., Mutlu, B.: Understanding large-language model (llm)-powered human-robot interaction. In: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction. pp. 371–380 (2024)
  * [19] Kingma, D., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR). San Diega, CA, USA (2015)
  * [20] van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research 9, 2579–2605 (2008)
  * [21] Masala, M., Ruseti, S., Dascalu, M.: Robert–a romanian bert model. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 6626–6637 (2020)
  * [22] Mititelu, V.B., Mitrofan, M.: The romanian medical treebank-simonero. In: Proceedings of the 15th International Conference on Linguistic Resources and Natural Language Processing Tools. p. 7 (2020)
  * [23] Mitrofan, M.: Extragere de cunostinte din texte în limba română si date structurate cu aplicatii în domeniul medical. Ph.D. thesis, Romanian Academy (2019)
  * [24] Mitrofan, M., Mititelu, V.B., Mitrofan, G.: MoNERo: a biomedical gold standard corpus for the Romanian language. In: Proceedings of the 18th BioNLP Workshop and Shared Task. pp. 71–79 (2019)
  * [25] Mitrofan, M., Păis, V.: Improving romanian bioner using a biologically inspired system. In: Proceedings of the 21st Workshop on Biomedical Language Processing. pp. 316–322. Association for Computational Linguistics, Dublin, Ireland (2022), https://aclanthology.org/2022.bionlp-1.30
  * [26] Mohammadshahi, A., Nikoulina, V., Bérard, A., Brun, C., Henderson, J., Besacier, L.: Small-100: Introducing shallow multilingual machine translation model for low-resource languages. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pp. 8348–8359 (2022)
  * [27] Neudecker, C.: An open corpus for named entity recognition in historic newspapers. In: Calzolari, N., Choukri, K., Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). pp. 4348–4352. European Language Resources Association (ELRA), Portorož, Slovenia (May 2016), https://aclanthology.org/L16-1689
  * [28] Niculescu, M.A., Ruseti, S., Dascalu, M.: Rogpt2: Romanian gpt2 for text generation. In: 2021 IEEE 33rd International Conference on Tools with Artificial Intelligence (ICTAI). pp. 1154–1161. IEEE (2021)
  * [29] Pais, V., Barbu Mititelu, V., Irimia, E., Mitrofan, M., Gasan, C.L., Micu, R.: Romanian micro-blogging named entity recognition including health-related entities. In: Gonzalez-Hernandez, G., Weissenbacher, D. (eds.) Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task. pp. 190–196. Association for Computational Linguistics, Gyeongju, Republic of Korea (Oct 2022), https://aclanthology.org/2022.smm4h-1.49
  * [30] Păis, V., Ion, R., Tufis, D.: A processing platform relating data and tools for Romanian language. In: Rehm, G., Bontcheva, K., Choukri, K., Hajič, J., Piperidis, S., Vasiļjevs, A. (eds.) Proceedings of the 1st International Workshop on Language Technology Platforms. pp. 81–88. European Language Resources Association, Marseille, France (May 2020), https://aclanthology.org/2020.iwltp-1.13
  * [31] Păis, V., Mitrofan, M., Gasan, C.L., Ianov, A., Coneschi, V.S., Onut, A., et al.: Legalnero: A linked corpus for named entity recognition in the romanian legal domain. Semantic Web pp. 1–14
  * [32] Păis, V., Mitrofan, M., Mititelu, V.B., Irimia, E., Micu, R., Gasan, C.L.: Challenges in creating a representative corpus of romanian micro-blogging text. In: Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-10). pp. 1–7 (2022)
  * [33] Pakhale, K.: Comprehensive overview of named entity recognition: Models, domain-specific applications and challenges. arXiv preprint arXiv:2309.14084 (2023)
  * [34] Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. In: International conference on machine learning. pp. 1310–1318. PMLR (2013)
  * [35] Păiş, V., Mitrofan, M., Gasan, C.L., Coneschi, V., Ianov, A.: Named entity recognition in the Romanian legal domain. In: Proceedings of the Natural Legal Language Processing Workshop 2021. pp. 9–18. Association for Computational Linguistics, Punta Cana, Dominican Republic (Nov 2021), https://aclanthology.org/2021.nllp-1.2
  * [36] Păis, V.: Multiple annotation pipelines inside the relate platform. In: The 15th International Conference on Linguistic Resources and Tools for Natural Language Processing. pp. 65–75 (2020)
  * [37] Păis, V., Mitrofan, M.: Towards a named entity recognition system in the Romanian legal domain using a linked open data corpus. In: Workshop on Deep Learning and Neural Approaches for Linguistic Data. pp. 16–17. Skopje, North Macedonia (Sep 2021)
  * [38] Păis, V., Mitrofan, M., Barbu-Mititelu, V., Irimia, E., Gasan, C.L., Micu, R., Marin, L., Dicusar, M., Florea, B., Badila, A.: Romanian micro-blogging named entity recognition (MicroBloggingNERo) (Jul 2022). https://doi.org/10.5281/zenodo.6905235, https://doi.org/10.5281/zenodo.6905235
  * [39] Păis, V., Mitrofan, M., Barbu Mititelu, V., Irimia, E., Micu, R., Gasan, C.L.: Challenges in creating a representative corpus of romanian micro-blogging text. In: Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-10). pp. 1–7. European Language Resources Association, Marseille, France (2022), https://aclanthology.org/2022.cmlc10-1.1
  * [40] Păis, V., Mitrofan, M., Gasan, C.L., Ianov, A., Ghită, C., Coneschi, V.S., Onut, A.: Romanian Named Entity Recognition in the Legal domain (LegalNERo) (May 2021). https://doi.org/10.5281/zenodo.4772094
  * [41] Ramshaw, L.A., Marcus, M.P.: Text chunking using transformation-based learning. In: Natural language processing using very large corpora, pp. 157–176. Springer (1999)
  * [42] Stenetorp, P., Pyysalo, S., Topić, G., Ohta, T., Ananiadou, S., Tsujii, J.: brat: a web-based tool for NLP-assisted text annotation. In: Segond, F. (ed.) Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. pp. 102–107. Association for Computational Linguistics, Avignon, France (Apr 2012), https://aclanthology.org/E12-2021
  * [43] Thapa, S., Rauniyar, K., Shiwakoti, S., Poudel, S., Naseem, U., Nasim, M.: Nehate: Large-scale annotated data shedding light on hate speech in nepali local election discourse. In: ECAI 2023, pp. 2346–2353. IOS Press (2023)
  * [44] Tufis, D., Mitrofan, M., Păis, V., Ion, R., Coman, A.: Collection and annotation of the Romanian legal corpus. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 2773–2777. European Language Resources Association, Marseille, France (2020), https://www.aclweb.org/anthology/2020.lrec-1.337
  * [45] Váradi, T., Koeva, S., Yamalov, M., Tadić, M., Sass, B., Nitoń, B., Ogrodniczuk, M., Pęzik, P., Barbu Mititelu, V., Ion, R., Irimia, E., Mitrofan, M., Păis, V., Tufis, D., Garabik, R., Krek, S., Repar, A., Rihtar, M., Brank, J.: The MARCELL legislative corpus. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 3761–3768. European Language Resources Association, Marseille, France (2020), https://www.aclweb.org/anthology/2020.lrec-1.464
  * [46] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [47] Váradi, T., Nyéki, B., Koeva, S., Tadić, M., Štefanec, V., Ogrodniczuk, M., Nitoń, B., Pęzik, P., Barbu Mititelu, V., Irimia, E., Mitrofan, M., Păis, V., Tufis, D., Garabík, R., Krek, S., Repar, A.: Introducing the curlicat corpora: Seven-language domain specific annotated corpora from curated sources. In: Proceedings of the Language Resources and Evaluation Conference. pp. 100–108. European Language Resources Association, Marseille, France (June 2022), https://aclanthology.org/2022.lrec-1.11