130 Università del Piemonte Orientale, Vercelli, Italy
131 Università di Brescia, Brescia, Italy
132 Variable Energy Cyclotron Centre, Homi Bhabha National Institute, Kolkata,
India
133 Warsaw University of Technology, Warsaw, Poland
134 Wayne State University, Detroit, Michigan, United States
135 Westfälische Wilhelms-Universität Münster, Institut für Kernphysik,
Münster, Germany
136 Wigner Research Centre for Physics, Budapest, Hungary
137 Yale University, New Haven, Connecticut, United States
138 Yonsei University, Seoul, Republic of Korea
139 Zentrum für Technologie und Transfer (ZTT), Worms, Germany
140 Affiliated with an institute covered by a cooperation agreement with CERN
141 Affiliated with an international laboratory covered by a cooperation
agreement with CERN.


# Enhancing deep neural networks with morphological information

Matej Klemen, Luka Krsnik, Marko Robnik-Šikonja
University of Ljubljana, Faculty of Computer and Information Science
Večna pot 113, Ljubljana, Slovenia
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Currently, deep learning approaches are superior in natural language
processing due to their ability to extract informative features and patterns
from languages. Two most successful neural architectures are LSTM and
transformers, the latter mostly used in the form of large pretrained language
models such as BERT. While cross-lingual approaches are on the rise, a vast
majority of current natural language processing techniques is designed and
applied to English, and less-resourced languages are lagging behind. In
morphologically rich languages, plenty of information is conveyed through
changes in morphology, e.g., through different prefixes and suffixes modifying
stems of words. The existing neural approaches do not explicitly use the
information on word morphology. We analyze the effect of adding morphological
features to LSTM and BERT models. As a testbed, we use three tasks available
in many less-resourced languages: named entity recognition (NER), dependency
parsing (DP), and comment filtering (CF). We construct sensible baselines
involving LSTM and BERT models, which we adjust by adding additional input in
the form of part of speech (POS) tags and universal features. We compare the
obtained models across subsets of eight languages. Our results suggest that
adding morphological features has mixed effects depending on the quality of
features and the task. The features improve the performance of LSTM-based
models on the NER and DP tasks, while they do not benefit the performance on
the CF task. For BERT-based models, the added morphological features only
improve the performance on DP when they are of high quality (i.e. manually
checked), while they do not show any practical improvement when they are
predicted. As in NER and CF datasets manually checked features are not
available, we only experiment with the predicted morphological features and
find that they do not cause any practical improvement in performance.

_K_ eywords Deep learning $\cdot$ Natural language processing $\cdot$
Morphologically rich languages

##  1 Introduction

The use of deep learning for processing natural language is becoming a
standard, with excellent results in a diverse range of tasks. Two state-of-
the-art architectures for text-related modeling are long short-term memory
(LSTM) networks (Hochreiter and Schmidhuber, 1997) and transformers (Vaswani
et al., 2017). LSTMs are recurrent neural networks that process the text
sequentially, meaning that they process text one token at a time, building up
its internal representation in hidden states of the network. Due to the
recurrent nature of LSTM, which degrades the efficiency of parallel
processing, as well as demonstrated improvements in performance, models based
on the transformer architecture are gradually replacing LSTMs across many
tasks. Transformers can process the text in parallel, using self-attention and
positional embeddings to model the sequential nature of the text.

A common trend in using transformers is to first pre-train them on large
monolingual corpora with abstract, general-purpose objective, and then fine-
tune them for a specific task, such as text classification. For example, the
BERT (Bidirectional Encoder Representations from Transformers) architecture
Devlin et al. (2019) uses transformers and is pretrained with masked language
modelling and order of sentences prediction tasks to build a general language
understanding model. During the fine-tuning for a specific downstream task,
additional layers are added to the BERT model, and the model is trained on
specific data to capture the specific knowledge required to perform the task.

Most of the research in the natural language processing (NLP) area focuses on
English, ignoring the fact that English is specific in terms of the low amount
of information expressed through morphology (English is so-called analytical
language). In our work, we focus on adapting modern deep neural networks,
namely LSTMs and BERT, for several morphologically rich languages, by
explicitly including the morphological information. The languages we analyze
contain rich information about grammatical relations in the morphology of
words instead of in particles or relative positions of words (as is the case
in English). For comparison, we also evaluate our models on English. Although
previous research has shown that the state of the art methods such as BERT
already captures some information contained in the morphology (Pires et al.,
2019; Edmiston, 2020), our experiments involve several languages with rich
morphology where neural networks could benefit from explicit morphological
features.

Specifically, we present methods which combine BERT with separately encoded
morphological properties: universal part of speech tags (uPOS tags) and
universal features (grammatical gender, tense, conjugation, declination, etc).
We evaluate them on three downstream tasks: named-entity recognition (NER),
dependency parsing (DP), and comment filtering (CF). We perform similar
experiments on LSTM networks and compare the results for both architectures.
Besides English, we analyze eight more languages: Croatian, Estonian, Finnish,
Latvian, Lithuanian, Russian, Slovene and Swedish. The choice of these
languages reflects a mix of different language groups (Balto-Slavic, Germanic,
and Uralic), for which we were able to obtain sufficient resources (datasets,
embeddings, and corpora), due to their coverage in the EU EMBEDDIA
project111http://embeddia.eu.

Our experiments show that the addition of morphological features has mixed
effects depending on the task. Across the tasks where the added morphological
features improve the performance, we show that (1.) they benefit the LSTM-
based models even if the features are noisy and (2.) they benefit the BERT-
based models only when the features are of high quality (i.e. human checked),
suggesting that BERT models already capture the morphology of the language;
however, there is a room for improvement either in designing pre-training
objectives that can capture these properties or when high-quality features are
available.

The remainder of this paper is structured as follows. In Section 2, we present
different attempts to use morphological information in machine learning, in
particular neural networks, as well as an overview of recent work in the three
evaluation tasks. In Section 3, we describe the used datasets and their
properties. In Section 4, we present the baseline models and models with
additional morphological information, whose performance we discuss in Section
5. Finally, we summarize our work and present directions for further research
in Section 6.

##  2 Related work

We shortly review the related work on the use of morphological information
within neural networks. We split the works based on three evaluation tasks:
NER, DP, and CF.

###  2.1 Morphological features in NER

Recent advances in NER are mostly based on deep neural networks. One of the
first usages of deep neural networks in NER is the work by Collobert and
Weston (2008), who propose an architecture that is jointly trained on six
different tasks, including NER, and show that this transfer learning approach
generalizes better due to learning a joint representation for different tasks.
A standard approach to NER is to use word (Huang et al., 2015) or character
(Kuru et al., 2016) embeddings of the input, followed by one or more neural
layers, mostly recurrent ones, such as LSTMs.

While sequence modelling on the character-level already has a potential to
encode morphological information, several authors show that the performance of
neural networks on the NER task can be improved by adding explicit information
about morphological properties of the text. Güngör et al. (2017) show that
embedding morphosyntactic properties improves NER performance on the Turkish
language, while Simeonova et al. (2019) use part of speech embeddings to
achieve improved performance on the NER task in the Bulgarian language. Our
work is similar to these approaches in the sense that we add different
morphological features to the neural networks. However, we experiment on a
larger scale, using more languages from different language groups, different
evaluation tasks, and with modern neural architectures (BERT), taking the LSTM
networks as the baseline.

###  2.2 Morphological features in dependency parsing

Similarly to NER, recent progress in DP is dominated by neural approaches,
which can be roughly divided into two categories, transition-based (Yamada and
Matsumoto, 2003; Nivre, 2003) and graph-based approaches (McDonald et al.,
2005). Some works do not fall into either category, e.g., DP is treated as a
sequence-to-sequence task by Li et al. (2018). The two categories differ in
how dependency trees are produced from the output of prediction models. In the
transition-based approach, a model is trained to predict a sequence of parsing
actions which produce a valid dependency tree, while in the graph-based
approach, a model is used to score candidate dependency trees via the sum of
scores of their substructures (e.g., arcs). One of the earlier successful
approaches to neural DP was presented by Chen and Manning (2014), who replaced
the commonly used sparse features with dense embeddings of words, part of
speech tags and arc labels, in combination with the transition-based parser.
This approach improved both the accuracy and parse speed. Pei et al. (2015)
introduced a similar approach to graph-based parsers. Later approaches improve
upon the earlier methods by automatically extracting more information that
guides the parsing. E.g., researchers use LSTM networks to inject context into
local embeddings (Kiperwasser and Goldberg, 2016), apply contextual word
embeddings (Kulmizev et al., 2019), or train graph neural networks (Ji et al.,
2019).

The use of morphological features in DP is quite common. For example, Chen and
Manning (2014) note that POS tag embeddings strongly contribute to the
performance of their system in English and Chinese. Morphological features are
commonly used with morphologically rich languages, e.g., Arabic (Marton et
al., 2010), German (Seeker and Kuhn, 2011), Lithuanian (Kapočiūtė-Dzikienė et
al., 2013), or Persian language (Khallash et al., 2013), with the consensus
that they improve the accuracy of parsing. Concerning adding morphological
properties to BERT, Kondratyuk and Straka (2019) train the multilingual BERT
model for multiple tasks, including POS and morphological tagging, and achieve
improved results across multiple languages. This approach uses transfer
learning to implicitly injects the morphological information into BERT models,
while our approach does that explicitly, through morphological features on the
input.

###  2.3 Morphological features in comment filtering

The literature for the task of NER covers multiple related tasks, such as hate
speech, offensive speech, political trolling, detecting commercialism, etc.
Recent approaches involve variants of deep neural networks, though standard
machine learning approaches are still popular as shown in the survey of
Fortuna and Nunes (2018). These approaches typically use features such as
character n-grams, word n-grams, and sentiment of the sequence. Two examples
are the works of Malmasi and Zampieri (2017), who classify hate speech in
English tweets, and Van Hee et al. (2015), who classify different levels of
cyberbullying in Dutch posts on ask.fm. Scheffler et al. (2018) combine word
embeddings with the above-mentioned features to classify German tweets. They
observe that combining both n-gram features and word embeddings brings only a
small improvement over only using only one of them. The effectiveness of using
features describing syntactic dependencies for toxic comments classification
on English Wikipedia comments is shown by Shtovba et al. (2019).

Neural architectures used include convolutional neural networks
(Georgakopoulos et al., 2018) and LSTM networks (Gao and Huang, 2017; Miok et
al., 2019), typically improving the performance over standard machine learning
approaches. The NER topic has also been the focus of shared tasks on
identification and categorization of offensive language (Zampieri et al.,
2019) and multilingual offensive language identification (Zampieri et al.,
2020). The reports of these tasks show the prevalence and general success of
large pretrained contextual models such as BERT, though, surprisingly, the
best performing model for the subtask B of SemEval-2019 Task 6 was rule-based
(Han et al., 2019).

###  2.4 Linguistic knowledge combined with neural networks

Large pretrained models such as BERT show superior performance across many
tasks. Due to a lack of theoretical understanding of this success, many
authors study how and to what extent BERT models can capture various
information, including different linguistic properties. An overview of recent
studies in this area sometimes referred to as BERTology, is compiled by Rogers
et al. (2020). Two common approaches to study BERT are i) add additional
properties to BERT models and observe the difference in performance on
downstream tasks, ii) a technique called probing (Conneau et al., 2018), where
the BERT model is trained (fine-tuned) to predict a studied property. For
example, Jawahar et al. (2019) investigate what type of information is learned
in different layers of the BERT English model and find that it captures
surface features in lower layers, syntactic features in middle layers, and
semantic features in higher layers. Similarly, Lin et al. (2019) find that
BERT encodes positional information about tokens in lower layers and then
builds increasingly abstract hierarchical features in higher layers. Tenney et
al. (2019) use probing to quantify where different types of linguistic
properties are stored inside BERT’s architecture and suggest that BERT
implicitly learns the steps performed in classical (non-end-to-end) natural
language processing pipeline. However, Elazar et al. (2020) point out possible
flaws in the probing technique, suggesting amnesic probing as an alternative.
They arrive at slightly different conclusions about BERT layer importance; for
example, they show that POS information affects the predictive performance
much more in upper layers.

Specifically for morphological properties, Zhou et al. (2020a) study whether
the inclusion of POS information is still necessary for a neural dependency
parser. Somewhat similarly to the previously mentioned transfer learning
approach of Kondratyuk and Straka (2019), they add POS information to the DP
task in English and Chinese by training a model with multi-task learning
objective; they conclude that POS tags are still beneficial in a neural
dependency parser, which our experiments on a larger set of languages confirm.
In contrast to our work, which studies the impact of morphological features in
BERT on the performance in downstream tasks, Edmiston (2020) studies the
morphological content present in BERT. The author uses probing, i.e. he trains
a linear classifier to predict morphological features based on features of a
certain hidden layer. Based on the achieved high performance, he argues that
BERT models capture much morphological information and partition their
embedding space into linearly-separable regions, correlated with morphological
properties.

##  3 Data

In this section, we describe the datasets used in our experiments, separately
for each of the three tasks: NER, DP, and NER.

###  3.1 Named entity recognition

In the NER experiments, we use datasets in eight languages: Croatian, English,
Estonian, Finnish, Latvian, Russian, Slovene and Swedish. We omit the
Lithuanian language from this experiment, as models for obtaining POS tags and
universal features were not available in the used tools at the time of
performing the experiments. The number of sentences and tags present in the
datasets is shown in Table 1. The label sets used in datasets for different
languages vary, meaning that some contain more fine-grained labels than
others. To make results across different languages consistent, we trim labels
in all datasets to the four common ones: location (LOC), organization (ORG),
person (PER), and “no entity” (OTHR).

Table 1: The collected datasets for NER task and their properties: the number of sentences and tagged words. Language | Dataset | Sentences | Tags
---|---|---|---
Croatian | hr500k | $24794$ | $28902$
English | CoNLL-2003 NER | $20744$ | $43979$
Estonian | Estonian NER corpus | $14287$ | $20965$
Finnish | FiNER data | $14484$ | $16833$
Latvian | LV Tagger train data | $9903$ | $11599$
Russian | factRuEval-2016 | $4907$ | $9666$
Slovene222The Slovene ssj500k originally contains more sentences, but only $9489$ are annotated with named entities. | ssj500k | $9489$ | $9440$
Swedish | Swedish NER | $9369$ | $7292$

###  3.2 Dependency parsing

To test morphological neural networks on the DP task, we used datasets in nine
languages (Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian,
Slovene and Swedish). The datasets are obtained from the Universal
Dependencies (Nivre et al., 2020). The number of sentences and tokens is shown
in Table 2.

Table 2: Dependency parsing datasets and their properties: the treebank, number of sentences, number of tokens, and information about the size of the split.  Language | Treebank | Tokens | Sentences | Train | Validation | Test
---|---|---|---|---|---|---
Croatian | SET | $199409$ | $9010$ | $6914$ | $960$ | $1136$
English | EWT | $254855$ | $16622$ | $12543$ | $2002$ | $2077$
Estonian | EDT | $438171$ | $30972$ | $24633$ | $3125$ | $3214$
Finnish | TDT | $202697$ | $15135$ | $12216$ | $1364$ | $1555$
Latvian | LVTB | $220536$ | $13643$ | $10156$ | $1664$ | $1823$
Lithuanian | ALKSNIS | $70051$ | $3642$ | $2341$ | $617$ | $684$
Russian | GSD | $98000$ | $5030$ | $3850$ | $579$ | $601$
Slovene | SSJ | $140670$ | $8000$ | $6478$ | $734$ | $788$
Swedish | Talbanken | $96858$ | $6026$ | $4303$ | $504$ | $1219$

###  3.3 Comment filtering

While there exist comparable datasets across different languages for the NER
and DP task, for the CF task, no such standard datasets exist. For that
reason, in our experiments on CF, we used two languages with adequate
datasets: English and Croatian.

For English experiments, we use a subset of toxic comments from Wikipedia’s
talk page edits333https://www.kaggle.com/c/jigsaw-toxic-comment-
classification-challenge/data. The comments are annotated with six possible
labels: toxic, severe toxic, obscene language, threats, insults, and identity
hate (making a total of six binary target variables). We extracted comments
from three categories: toxic, severe toxic, threats, and identity hate; a
total of $21,541$ instances. We randomly chose the same amount of comments
that do not fall in any of the mentioned categories, obtaining the final
dataset of $43,082$ instances.

For Croatian language experiments, we form a dataset from user comments
published in the Croatian news site 24sata. The comments used are either
unproblematic or labelled with one of the eight rules they break. Below we
briefly describe the types of problematic content present in the database.

  1. 1.

Advertising, content unrelated to the topic, spam, copyright infringement,
citation of abusive comments or any other comments that are not allowed on the
portal.

  2. 2.

Direct threats to other users, journalists, admins or subjects of articles,
which may also result in criminal prosecution.

  3. 3.

Verbal abuse, derogation and verbal attack based on national, racial, sexual
or religious affiliation, hate speech and incitement.

  4. 4.

Collecting and publishing personal information, uploading, distributing or
publishing pornographic, obscene, immoral or illegal content and using a
vulgar or offensive nickname that contains the name and surname of others.

  5. 5.

Publishing false information for deception or slander, and “trolling” -
deliberately provoking other commentators.

  6. 6.

Use of bad language, unless they are used as a stylistic expression, or are
not addressed directly to someone.

  7. 7.

Writing in other languages besides the Croatian, in other scripts besides
Latin or writing with all caps.

  8. 8.

Verbally abusing other users and their comments, article authors, and direct
or indirect article subjects, calling the admins out or arguing with them in
any way.

Unfortunately, this data is proprietary and is not publicly available. We
extracted all the $17,868$ comments from the third category (the hate speech
label) and approximately the same number of comments that do not break any
rules. Our final dataset contains $35,635$ instances.

##  4 Neural networks with morphological features

In this section, we describe the architectures of neural networks used in our
experiments. Their common property is that we enhance standard word embeddings
based inputs with embeddings of morphological features. We work with recent
successful neural network architectures, LSTMs and transformers, i.e. BERT
models. A detailed description of architectures is available in the following
subsections, separately for each of the three evaluation tasks. For each task
and architecture, we describe the baseline architecture and the enhanced one.

###  4.1 Named entity recognition models

In the NER task, we use two baseline neural networks (LSTM and BERT), and the
same two models with additional morphological information: POS tag embeddings
and universal feature embeddings. The baseline models and their enhancements
are displayed in Figure 1.

The first baseline model (left-hand side of Figure 1) is a unidirectional
(left-to-right) LSTM model, which takes as an input a sequence of tokens,
embedded using $300$-dimensional fastText embeddings (Bojanowski et al., 2017)
which are particularly suitable for morphologically rich languages as they
work with subword inputs444The precomputed embeddings are available at
https://fasttext.cc/docs/en/crawl-vectors.html.. For each input token, its
LSTM hidden state is extracted and passed through the linear layer to compute
its NER score (probability for each of the four NER labels).

The second baseline model (right-hand side of Figure 1) is the multilingual
BERT model (the base cased variant). In our experiments, we follow the
sequence tagging approach suggested by the authors of BERT (Devlin et al.,
2019). Here, the input sequence is prepended with a special token [CLS] and
passed through the BERT model. The output of the last BERT hidden layer is
passed through the linear layer to obtain the predictions for NER labels.

Both baseline models (LSTM and BERT) are enhanced with the same morphological
information: POS tag embeddings and universal feature embeddings for each of
the input tokens. We embed the POS tags using $15$-dimensional embeddings. For
each of the $23$ universal features used (we omitted the Typo feature, as the
version of the POS tagger we used did not annotate this feature), we
constructed $15$-dimensional embeddings. We computed the POS tags and
morphological features using the Stanza (Qi et al., 2020) system in the
universal dependencies mode. In the enhanced architectures, we included
another linear layer before the final linear classification layer to model
possible interactions.

Figure 1: The baseline LSTM-based (left) and BERT-based (right) models for the
NER task along with our modifications with morphological information. The
dotted border of POS vectors, morphological features (feats), and the linear
layer marks that their use is optional and varies across experiments. The
$\odot$ symbol between layers represents the concatenation operation. The
$w_{i}$ symbol stands for token $i$; in case of LSTM, tokens enter the model
sequentially and we show the unrolled network, while BERT processes all token
simultaneously.

###  4.2 Dependency parsing models

As the baseline model in the DP task, we use the deep biaffine graph-based
dependency parser (Dozat and Manning, 2016). The enhancements with the
morphological information are at the input level. The baseline model and its
enhancements are shown in Figure 2.

The baseline parser combines a multi-layer bidirectional LSTM network with a
biaffine attention mechanism to jointly optimize prediction of arcs and arc
labels. We leave the majority of baseline architectural hyperparameters at
values described in the original paper ($3$-layer bidirectional LSTM with
$100$-dimensional input word embeddings, and the hidden state size of $400$).

In our experiments, we concatenate the non-contextual word embeddings with
various types of additional information. The first additional input is
contextual word embeddings, which we obtain either by using the hidden states
of an additional single-layer unidirectional LSTM or by using a learned linear
combination of all hidden states of BERT. Although the LSTM layers are already
present in the baseline parser, we include an additional layer at the input
level to keep the experimental settings similar across our three evaluation
tasks. The second additional input is universal POS embeddings (UPOS), and the
third one are universal feature embeddings (feats). These embeddings are
concatenated separately for each token of the sentences. The size of the
additional LSTM layer, POS tag embeddings and universal feature embedding are
treated as tunable hyperparameters. As the baseline input embeddings, we use
pre-trained $100$-dimensional fastText embeddings, which we obtain by reducing
the dimensionality of publicly available $300$-dimensional vectors with
fastText’s built-in dimensionality reduction tool.

In DP experiments, we use POS tags and morphological features of two origins.
The first source is human annotations provided in the used datasets. The
second source of morphological information is predictions of Stanza models (Qi
et al., 2020). These two origins are used in the assessment of the quality of
morphological information; namely, we check if manual human annotations
provide any benefit compared to automatically determined POS tags and
features.

Figure 2: The deep biaffine graph-based dependency parser along with our
enhancements at the input level. The dotted border of input embedding vectors,
POS vectors, and morphological features (feats) is optional and varies across
experiments. The $\odot$ symbol between layers represents the concatenation
operation. The $w_{i}$ symbol stands for token $i$; tokens enter the LSTM
model sequentially, and we show the unrolled network.

###  4.3 Comment filtering models

In the NER evaluation task, we add additional morphological information to
standard LSTM and BERT models. The baseline models and enhanced models for
this task are similar to those in the NER evaluation, though we operate at the
sequence level here as opposed to the token level in the NER task. The
architecture of models is shown in Figure 3. As baselines, we take a single
layer unidirectional LSTM network (the top part of Figure 3), and the
multilingual base uncased BERT model (the bottom part of Figure 3). The
difference in the used BERT dialect (_uncased_ as opposed to the _cased_ in
the NER task) is due to better performance detected in preliminary experiments
on a separate Croatian validation set.

In the LSTM baseline model, the words of the input sequence are embedded using
pre-trained $300$-dimensional fastText embeddings. As the representation of
the whole sequence, we take the output of the last hidden state, which then
passes through the linear layer to obtain the prediction scores. In the BERT
baseline model, we take the sequence classification approach suggested by the
authors of BERT. The input sequence is prepended with the special [CLS] token
and passed through BERT. The sequence representation corresponds to the output
of the last BERT hidden layer for the [CLS] token, which is passed through a
linear layer to obtain the prediction scores.

As in other tasks, we augment the baseline models with POS tag and universal
feature embeddings. We obtain the embeddings for each token separately from
UDPipe models (Straka and Straková, 2017). The reason for not using Stanza in
this set of experiments is that we started experimenting before its release.
Additional testing with Stanza did not show a difference in performance, so we
stayed with the UDPipe models for this set of experiments. We combine the
obtained embeddings using three different pooling mechanisms: mean, weighted
combination, or LSTM pooling. Given the POS tag or universal feature
embeddings, the mean pooling outputs the mean of all token embeddings; the
weighted pooling outputs the weighted combination of token embeddings, and the
LSTM pooling outputs the last hidden state obtained by passing the sequence of
embeddings through the LSTM network. Both the embedding sizes and the type of
pooling are treated as tunable hyperparameters. The coefficients of the
weighted combination are learned by projecting the sequence embeddings into a
sequence of independent dimension values which are normalized with the
softmax. This compresses the embedding sequence, establishes its fixed length,
and allows for different morphological properties to have a different impact
in the sequence representation. This approach tests the contextual encoding of
morphological properties. For example, adjectives might be assigned a higher
weight than other POS tags due to their higher emotional contents that often
indicates insults.

Figure 3:  The baseline LSTM (top) and BERT (bottom) models for the NER task
along with our modifications with morphological information. The dotted border
of UPOS vectors and morphological features (feats) marks that their use is
optional and varies across experiments. The $\odot$ symbol between layers
represents the concatenation operation. The $w_{i}$ symbol stands for token
$i$; in case of LSTM, tokens enter the model sequentially and we show the
unrolled network, while BERT processes all token simultaneously.

##  5 Evaluation

In this section, we first present the evaluation scenario for the three
evaluation tasks, followed by results, presented separately for each of the
tasks. We end the section with the ablation study, performed on the DP task.
In the ablation study, we investigate the effect of additional morphological
features in three situations where we tweak one aspect of training procedure
at a time: (1) we increase the maximum training time of our models by
additional $5$ epochs, (2) we replace the human-annotated features with the
ones automatically predicted by machine learning models, and (3) we replace
the embeddings from general multilingual BERT with those from more specific
multilingual and monolingual BERT models.

###  5.1 Experimental settings

The experimental settings differ between the three evaluation tasks, so we
describe them separately for each task, starting with the NER task, and
followed by the DP and NER tasks.

####  5.1.1 Experimental settings for NER

For NER, we train each BERT model for $10$ epochs and each LSTM model for $50$
epochs. These parameters were determined during preliminary testing on the
Slovene dataset. The selected numbers of epochs are chosen to balance the
performance and training times of the models. All NER models are evaluated
using $10$-fold cross-validation.

We evaluate the models with the $F_{1}$ score, which is a harmonic mean of
precision and recall measures. This measure is typically used in NER in a way
that precision and recall are calculated separately for each of the three
entity classes (location, organization, and person, but not ”no entity”). For
each metric, we compute the weighted average over the class scores using the
frequencies of class values in the datasets as the weights. Ignoring the ”no
entity” (OTHR) label is a standard approach in the NER evaluation and
disregards words that are not annotated with any of the named entity tags,
i.e. the evaluation focuses on the named entities.

####  5.1.2 Experimental settings for dependency parsing

We train each DP model for a maximum of $10$ epochs, using an early stopping
tolerance of $5$ epochs in the process. All models are evaluated using
predefined splits into training, validation and testing set, determined by the
respective treebank authors and maintainers (see Table 2). The final models,
evaluated on the test set, are selected based on the maximum mean of unlabeled
and labelled attachment scores (UAS and LAS) on the validation set. The UAS
and LAS are standard accuracy metrics in DP. The UAS score is defined as the
proportion of tokens that are assigned the correct syntactic head, while the
LAS score is the proportion of tokens that are assigned the correct syntactic
head as well as the dependency label (Jurafsky and Martin, 2009).

####  5.1.3 Experimental settings for comment filtering

In our NER experiments, we train BERT models for a maximum of $10$ epochs and
LSTM models for a maximum of $50$ epochs, using early stopping with the
tolerance of 5 epochs.

We evaluate the models using the classification accuracy metric. The choice
reflects the fact that the distributions of class labels are approximately
balanced in our datasets. We use fixed training, validation and test sets,
obtained by randomly dividing the datasets in the ratio $60\%$:$20\%$:$20\%$.
The initial results in the Croatian hate speech experiments showed large
variance; therefore, we repeated each experiment five times. For this task and
Croatian language, we report the obtained mean classification accuracy and its
standard deviation.

###  5.2 Experimental results

In this section, we compare the results of baseline models with their
enhancements using additional morphological information. We split the
presentation into three parts, according to the evaluation task: NER, DP, and
CF.

####  5.2.1 Results for the NER task

For the NER evaluation task, we present results of the baseline NER models and
models enhanced with the POS tags and morphological features, as introduced in
Section 4.1. Table 3 shows the results for LSTM and BERT models. We compute
the statistical significance of the differences between the baseline LSTM
models and their best-performing counterpart with morphological additions. We
use Wilcoxon signed-rank test (Wilcoxon et al., 1970) and underline the
statistically significant differences at $p=0.01$ level. We do not perform the
significance of differences tests for BERT as the differences for those models
are marginal.

As expected, the baseline models involving BERT outperform their LSTM
counterparts across all languages by a large margin. When adding POS tags or
universal features to LSTM-based models, we observe a noticeable increase in
performance over the baselines for seven languages. For three of them
(Croatian, Russian, and Swedish), the increase in $F_{1}$ score is under one
per cent. For three of them (Estonian, Finnish, and Slovene), the differences
in performance are statistically significant, and increases range from $1.4\%$
(Finnish) to $4.6\%$ (Slovene). The only language for which there is no
improvement with the LSTM-based models is English.

In BERT-based models, the additional morphological features do not seem to
make a practical difference. For all languages except Slovene, the increase in
$F_{1}$ values over the baseline is under $0.5\%$. However, since preliminary
tests in this task were performed on Slovene, the larger observed increase
might be due to manual overfitting.

Table 3: $F_{1}$ scores for different models on the NER task in different languages. The upper part of the table shows LSTM models and the lower part shows BERT models. The best scores for each language are marked with the bold typeface separately for LSTM and BERT models. Best scores for which the difference to the baseline is statistically significant are underlined. Model | CRO | ENG | EST | FIN | LAT | RUS | SLO | SWE
---|---|---|---|---|---|---|---|---
fT + LSTM | 0.700 | 0.890 | 0.769 | 0.814 | 0.573 | 0.752 | 0.651 | 0.785
fT + LSTM + UPOS | 0.708 | 0.889 | 0.792 | 0.824 | 0.590 | 0.754 | 0.692 | 0.787
fT + LSTM + UPOS + feats | 0.706 | 0.883 | 0.781 | 0.828 | 0.588 | 0.759 | 0.697 | 0.785
fT + LSTM + feats | 0.697 | 0.885 | 0.777 | 0.826 | 0.581 | 0.760 | 0.669 | 0.793
BERT | 0.874 | 0.948 | 0.875 | 0.926 | 0.766 | 0.868 | 0.848 | 0.885
BERT + UPOS | 0.875 | 0.948 | 0.874 | 0.927 | 0.773 | 0.870 | 0.857 | 0.886
BERT + UPOS + feats | 0.875 | 0.949 | 0.879 | 0.927 | 0.763 | 0.869 | 0.853 | 0.883
BERT + feats | 0.874 | 0.947 | 0.874 | 0.928 | 0.769 | 0.869 | 0.851 | 0.878

####  5.2.2 Results for the dependency parsing task

For the DP evaluation task on different languages (described in Section 4.2),
we present UAS and LAS accuracies in Tables 4 and 5, respectively. We first
show the scores of the baseline models (multi-layer bidirectional LSTM network
with biaffine attention) followed by these models enhanced with additional
inputs: LSTM or BERT contextual embeddings, POS tags and morphological
features, introduced in Section 4.2. The results in the upper part of Section
4.2 represent enhancements incorporating LSTM embeddings, and in the lower
part, we show the enhancements including BERT embeddings. We also
statistically test the differences in UAS and LAS scores between the best
performing enhanced variants and their baselines without morphological
additions. As the splits are fixed in the DP tasks, we use Z-test for the
equality of two proportions (Kanji, 2006) at $p=0.01$ level. The null
hypothesis is that the scores of compared models are equal. For languages
where the null hypothesis can be rejected, we underline the respective best
result.

Similarly to the other two evaluation tasks, the models involving BERT
embeddings outperform the baselines involving LSTM embeddings on all languages
by a large margin. Enhancements with LSTM embeddings improve over parsers
without contextual embeddings. Models with added POS tags or universal
features noticeably improve over baselines with only LSTM embeddings in both
UAS and LAS score for all languages. The increase ranges between $1.2\%$
(Russian) and $6.8\%$ (Lithuanian) for UAS and between $2.4\%$ (Russian) and
$10.42\%$ (Lithuanian) for LAS. All compared differences between the LSTM
baselines and the best enhanced variants are statistically significant at
$p=0.01$.

Contrary to the results observed on the other two tasks, the addition of
morphological features to the baseline models with BERT embeddings improves
the performance scores for all languages. For some languages, the increase
seems to be practically insignificant (under one per cent). However, for six
languages out of nine, the increase in UAS is over one per cent and
statistically significant; the same is true for eight languages out of nine
for the LAS score. For these languages, the improvement ranges from $1.1\%$
(Finnish) to $1.8\%$ (Lithuanian) for UAS, and from $1.1\%$ (Croatian) to
$4.4\%$ (Lithuanian) for LAS. The only language for which both UAS and LAS
scores are practically equal to the baselines with BERT embeddings is Russian.

Table 4: UAS scores for different models on the DP task in different languages. The upper part of the table shows baseline parser with enhancements using LSTM-based contextual embeddings; the lower part shows baseline parser enhanced with variants of BERT embeddings. The best scores for each language are marked with the bold typeface separately for LSTM and BERT embeddings extensions. Statistically significant differences in best scores at $p=0.01$ level are underlined. Model | CRO | ENG | EST | FIN | LAT | LIT | RUS | SLO | SWE
---|---|---|---|---|---|---|---|---|---
baseline | $84.83$ | $83.54$ | $79.35$ | $82.45$ | $81.51$ | $70.51$ | $83.04$ | $86.66$ | $80.68$
+LSTM | $86.48$ | $84.87$ | $81.92$ | $84.13$ | $83.63$ | $71.82$ | $85.02$ | $86.92$ | $84.03$
+LSTM+UPOS | $87.81$ | $\underline{\textbf{87.26}}$ | $85.30$ | $86.82$ | $86.73$ | $75.83$ | $85.75$ | $91.99$ | $86.78$
+LSTM+UPOS+f. | $\underline{\textbf{87.82}}$ | $87.21$ | $\underline{\textbf{86.12}}$ | $\underline{\textbf{87.64}}$ | $\underline{\textbf{87.97}}$ | $\underline{\textbf{78.62}}$ | $86.19$ | $\underline{\textbf{92.42}}$ | $\underline{\textbf{87.19}}$
+LSTM+feats | $87.52$ | $85.97$ | $84.31$ | $85.23$ | $86.09$ | $78.10$ | $\underline{\textbf{86.26}}$ | $90.67$ | $84.90$
+mBERT | $91.72$ | $91.43$ | $88.02$ | $89.99$ | $87.87$ | $79.79$ | $90.32$ | $93.66$ | $90.16$
+mBERT+UPOS | 92.17 | 91.64 | $89.26$ | $90.53$ | $89.11$ | $80.48$ | $90.53$ | $94.70$ | $91.21$
+mBERT+UPOS+f. | $91.97$ | $91.56$ | $\underline{\textbf{89.51}}$ | $\underline{\textbf{91.09}}$ | $\underline{\textbf{89.63}}$ | $\underline{\textbf{81.64}}$ | 90.65 | $\underline{\textbf{94.95}}$ | $\underline{\textbf{91.46}}$
+mBERT+feats | $91.72$ | $91.35$ | $88.32$ | $90.33$ | $88.92$ | $81.61$ | $90.64$ | $94.42$ | $90.64$
Table 5: LAS scores for different models on the DP task in different languages. The upper part of the table shows baseline parser with enhancements using LSTM-based contextual embeddings; the lower part shows baseline parser with enhancements using BERT embeddings. The best scores for each language are marked with the bold typeface separately for LSTM and BERT embedding extensions. Statistically significant differences in best scores at $p=0.01$ level are underlined. Model | CRO | ENG | EST | FIN | LAT | LIT | RUS | SLO | SWE
---|---|---|---|---|---|---|---|---|---
baseline | $77.55$ | $79.28$ | $73.17$ | $77.09$ | $75.77$ | $62.23$ | $77.22$ | $82.12$ | $75.06$
+LSTM | $79.59$ | $80.69$ | $76.24$ | $78.60$ | $78.08$ | $63.65$ | $79.24$ | $83.03$ | $78.97$
+LSTM+UPOS | $81.95$ | $\underline{\textbf{84.68}}$ | $81.36$ | $83.06$ | $82.75$ | $70.45$ | $80.88$ | $88.75$ | $82.65$
+LSTM+UPOS+f. | $\underline{\textbf{82.84}}$ | $84.48$ | $\underline{\textbf{83.46}}$ | $\underline{\textbf{83.97}}$ | $\underline{\textbf{84.27}}$ | $\underline{\textbf{74.07}}$ | $\underline{\textbf{81.68}}$ | $\underline{\textbf{90.32}}$ | $\underline{\textbf{83.17}}$
+LSTM+feats | $82.00$ | $82.96$ | $81.00$ | $81.16$ | $81.81$ | $72.40$ | $81.64$ | $88.38$ | $80.31$
+mBERT | $86.37$ | $88.09$ | $84.30$ | $86.07$ | $83.02$ | $72.62$ | $85.99$ | $91.67$ | $86.67$
+mBERT+UPOS | $\underline{\textbf{87.63}}$ | $\underline{\textbf{89.35}}$ | $86.72$ | $87.32$ | $85.43$ | $74.99$ | $86.24$ | $92.90$ | $87.80$
+mBERT+UPOS+f. | $87.44$ | $89.13$ | $\underline{\textbf{87.19}}$ | $\underline{\textbf{87.99}}$ | $\underline{\textbf{86.16}}$ | $\underline{\textbf{77.04}}$ | 86.56 | $\underline{\textbf{93.40}}$ | $\underline{\textbf{88.38}}$
+mBERT+feats | $86.81$ | $88.81$ | $85.72$ | $87.07$ | $85.19$ | $76.33$ | $86.37$ | $92.74$ | $87.25$

####  5.2.3 Results for the comment filtering task

Table 6 shows the NER results for LSTM and BERT baselines and their
enhancements, described in Section 4.3. All BERT-based models outperform the
LSTM-based models by a large margin for both languages, Croatian and English.

Adding POS tags and universal features to neural architectures of either type
does not increase the classification accuracy and may even decrease it in some
cases. As the accuracy does not increase with the best set of hyperparameters,
we do not further discuss the effect of different pooling types on the
performance, though we note that none of the pooling approaches consistently
performed best.

Table 6: Classification accuracies for baseline and enhanced models on the NER task in different languages. The upper part of the table shows LSTM models, and the lower part shows BERT models. The best scores for each language are marked with the bold typeface separately for LSTM and BERT models. We report the mean and standard deviation over five random splits of the data. The differences between the models of the same type and language are marginal.  | CRO | ENG
---|---|---
Model | Hate speech | Toxic comments
fastText + LSTM | $\mathbf{77.69\pm 0.47\%}$ | $88.14\pm 0.12\%$
fastText + LSTM + uPOS | $76.63\pm 0.77\%$ | $88.16\pm 0.15\%$
fastText + LSTM + uPOS + feats | $75.64\pm 1.21\%$ | $86.90\pm 1.20\%$
fastText + LSTM + feats | $76.52\pm 0.99\%$ | $\mathbf{88.25\pm 0.27\%}$
mBERT | $86.20\pm 0.20\%$ | $92.23\pm 0.12\%$
mBERT + uPOS | $85.05\pm 0.49\%$ | $\mathbf{92.33\pm 0.11\%}$
mBERT + uPOS + feats | $\mathbf{86.34\pm 0.28\%}$ | $92.29\pm 0.20\%$
mBERT + feats | $86.22\pm 0.35\%$ | $92.20\pm 0.13\%$

###  5.3 Ablation study

To further analyze different aspects of the proposed morphological
enhancements, we conducted several ablation studies on the DP task where the
datasets and evaluation settings allow many experiments. Similarly as in
section 5.2.2, we statistically evaluate differences in performance between
the baseline model and the best enhancement using the Z-test for the equality
of two proportions. In cases where the null hypothesis can be rejected at
$p=0.01$ level, we underline the respective compared score. We test the impact
of additional training time, quality of morphological information, and
different variants of BERT models.

####  5.3.1 Additional training time

To test if the observed differences in performance are due to random variation
in the training of models which could be reduced with longer training time, we
increase the maximum training time from $10$ to $15$ epochs. From the studied
languages, we arbitrarily choose a subset of $6$ languages on which we
experiment. The results are shown in Table 7.

Table 7: UAS/LAS scores achieved by models that are trained for up to $5$ additional epochs (a maximum training time of $15$ epochs instead of $10$). The results for $10$ epochs are presented in Tables 4 and 5. Model | CRO | ENG | EST | FIN | SLO | RUS
---|---|---|---|---|---|---
baseline+LSTM | $86.22/79.49$ | $85.40/81.47$ | $82.28/76.72$ | $85.30/80.28$ | $88.31/84.72$ | $85.38/79.79$
baseline+LSTM+UPOS | $\underline{\textbf{88.43}}/82.72$ | $87.30/84.74$ | $85.35/81.52$ | $87.27/83.77$ | $92.11/89.19$ | $86.46/81.35$
baseline+LSTM+UPOS+f. | $88.26/\underline{\textbf{83.26}}$ | $\underline{\textbf{87.92}}/\underline{\textbf{85.26}}$ | $\underline{\textbf{86.40}}/\textbf{83.65}$ | $\underline{\textbf{87.83}}/\underline{\textbf{84.22}}$ | $\underline{\textbf{92.64}}/\underline{\textbf{90.64}}$ | $86.68/82.18$
baseline+LSTM+feats | $88.08/82.74$ | $86.61/83.58$ | $84.50/81.24$ | $85.71/81.95$ | $90.73/88.60$ | $\underline{\textbf{86.75}}/\underline{\textbf{82.21}}$
baseline+mBERT | $92.06/87.03$ | $91.10/87.77$ | $88.06/84.60$ | $90.58/87.06$ | $93.97/92.11$ | $90.26/86.15$
baseline+mBERT+UPOS | $92.03/87.55$ | $\underline{\textbf{91.77}}/\underline{\textbf{89.56}}$ | $89.35/86.60$ | $91.16/87.98$ | $94.54/92.80$ | $91.05/87.04$
baseline+mBERT+UPOS+f. | $\textbf{92.09}/\textbf{87.71}$ | $91.74/89.49$ | $\underline{\textbf{89.49}}/\underline{\textbf{87.26}}$ | $\underline{\textbf{91.32}}/\underline{\textbf{88.42}}$ | $\underline{\textbf{95.16}}/\underline{\textbf{93.78}}$ | $\textbf{91.11}/\textbf{87.23}$
baseline+mBERT+feats | $91.93/86.97$ | $91.33/88.89$ | $88.37/85.78$ | $90.85/87.85$ | $94.49/93.08$ | $90.79/86.64$

We can observe that longer training times slightly increases the scores for
all model variants, though their relative order stays the same. The models
with added morphological features still achieve better results, so the
performance increases do not seem to be the effect of random fluctuations in
training due to the different amount of training steps.

####  5.3.2 Quality of morphological information

In the second ablation study, we evaluate the impact of the quality of
morphological information. We replace the high quality (human-annotated) POS
tags and morphological features used in Section 4.2 with those predicted by
machine learning models. In this way, we test a realistic setting where the
morphological information is at least to a certain degree noisy. We obtain POS
tags and morphological features from Stanza models prepared for the involved
languages (Qi et al., 2020). To avoid overly optimistic results, we make sure
to use models that are not trained on the same datasets used in our DP
experiments. This is possible for a subset of five languages. Table 8 shows
the results of DP models, trained with predicted morphological features. The
first two lines of the table show the quality of used features, expressed as
the proportion of tokens, for which POS tags and all morphological features
are correctly predicted, i.e. we report their accuracy.

The general trend is that using predicted features results in much smaller
(best case) performance increases, though some languages still see significant
increases. For LSTM models, the increases range from $0.42\%$ (English) to
$1.60\%$ (Slovene) for UAS, and from $0.67\%$ (English) to $2.41\%$ (Finnish)
for LAS. For BERT models, the increases are marginal and range from $0.00\%$
(English) to $0.49\%$ (Finnish) for UAS, and from $-0.10\%$ (i.e. decrease,
Slovene) to $0.66\%$ (Finnish) for LAS. These results are consistent with the
results of our NER experiments in section 5.2.1, where we have no access to
human-annotated features and find that noisy features only help LSTM-based
models.

As noisy features might require different training, we also tried to increase
the maximal number of training steps to $15$ epochs and repeated the tuning of
hyperparameters for one language (Estonian). None of these changes improved
the scores by a practically relevant margin.

These results indicate that adding predicted morphological features to models
with BERT embeddings might not be practically useful since their quality needs
to be very high. However, since human annotated morphological features improve
the performance on the DP task, this suggests that there is a room for
improvement in BERT pre-training. It seems that pre-training tasks of BERT
(masked language modelling and next sentence prediction) do not fully capture
the morphological information present in the language.

Table 8: UAS/LAS scores achieved by models that are trained with predicted (noisy) instead of human-annotated morphological features. In the first two lines, we report the proportion of tokens for which UPOS tags and UPOS features are correctly predicted. | ENG | EST | FIN | SLO | RUS
---|---|---|---|---|---
UPOS accuracy (%): | $92.45$ | $91.15$ | $87.59$ | $80.45$ | $89.32$
feats accuracy (%): | $93.92$ | $88.86$ | $86.20$ | $78.74$ | $85.22$
baseline+LSTM | $84.87/80.69$ | $81.92/76.24$ | $84.13/78.60$ | $86.92/83.03$ | $85.02/79.24$
baseline+LSTM+UPOS | $85.18/\textbf{81.36}$ | $\textbf{82.44}/77.15$ | $85.33/80.61$ | $88.14/84.06$ | $84.90/79.53$
baseline+LSTM+UPOS+f. | $85.04/81.02$ | $82.24/\underline{\textbf{77.39}}$ | $\underline{\textbf{85.56}}/\underline{\textbf{81.01}}$ | $\underline{\textbf{88.52}}/\underline{\textbf{85.00}}$ | $\textbf{85.82}/\underline{\textbf{80.89}}$
baseline+LSTM+feats | $\textbf{85.29}/81.22$ | $81.42/76.19$ | $84.89/80.43$ | $88.12/84.12$ | $84.74/79.56$
baseline+mBERT | $\textbf{91.43}/88.09$ | $88.02/84.30$ | $89.99/86.07$ | $93.66/\textbf{91.67}$ | $90.32/85.99$
baseline+mBERT+UPOS | $\textbf{91.43}/\textbf{88.33}$ | $\textbf{88.20}/\textbf{84.69}$ | $90.05/86.33$ | $93.38/91.42$ | $\textbf{90.59}/\textbf{86.17}$
baseline+mBERT+UPOS+f. | $91.08/88.05$ | $88.06/84.39$ | $\textbf{90.48}/\textbf{86.73}$ | $\textbf{93.73}/91.57$ | $90.34/85.74$
baseline+mBERT+feats | $90.90/87.74$ | $87.88/84.42$ | $89.77/85.91$ | $93.37/91.23$ | $89.65/85.27$

####  5.3.3 Specific BERT models

In the third ablation study, we replace the embeddings obtained from the
multilingual uncased BERT model with those obtained from more specific
multilingual BERT models and monolingual BERT models. In experiments involving
multilingual BERT models, we use Croatian/Slovene/English CroSloEngual BERT
and Finnish/Estonian/English FinEst BERT (Ulčar and Šikonja, 2020). In
experiments with monolingual BERT models, we use English bert-base-uncased
(Devlin et al., 2019), Finnish bert-base-finnish-cased-v1 (Virtanen et al.,
2019), and Russian RuBert (Kuratov and Arkhipov, 2019). We only perform the
experiments for a subset of studied languages for which we were able to find
more specific BERT models. The aim is to check if the additional morphological
features improve the performance of more language-specific BERT models. These
are trained on a lower number of languages, and larger amounts of texts in the
involved languages compared to the original multilingual BERT model Devlin et
al. (2019). Due to this language-specific training, we expect these BERT
models to capture the nuances of the languages better, thus possibly
benefiting less from the additional morphological features. The results are
shown in Table 9 for trilingual BERT models and in Table 10 for monolingual
BERT models.

In most cases, the specific multilingual BERT models without additional
features do as well as or better than the best performing original
multilingual BERT model with additional features. The only worse LAS scores
are achieved on English and Estonian, indicating that trilingual BERT models
are better adapted to the task than the original multilingual models. The
addition of morphological features increases the UAS and LAS even further. For
UAS, the improvements are generally smaller than before (see Figure 2), with
the only improvement larger than $1\%$ being on Estonian. For LAS, the
improvements are larger compared to UAS and range from $0.54\%$ (Slovene) to
$2.23\%$ (Estonian); only one language sees an improvement of less than $1\%$.
These results indicate that the additional morphological features still
contain valuable information for the DP task, which the BERT models do not
capture. We hypothesise that slightly lower performance of FinEst BERT
(without additions) on Estonian as well as slightly larger increase when
adding morphological features might be due to variability in the training
process.

For monolingual models (the most language-specific), the results are mixed,
but the general trend is that the additional features help little or not at
all. The English monolingual model without and with morphological additions
performs comparably to its multilingual (CroSloEngual) counterpart, and we do
not see any additional increase in the performance. The Russian monolingual
model achieves comparable UAS and LAS to the best performing original
multilingual BERT model with additional features. With the addition of
morphological features, the scores increase, but only marginally (under
$0.5\%$). The Finnish monolingual model outperforms the best performing
multilingual (FinEst) counterpart. The addition of morphological features to
the monolingual Finnish model increases the scores further: UAS by $0.33\%$
and LAS by $0.68\%$, but these increases are not statistically significant.

Table 9: UAS/LAS scores achieved by trilingual BERT models, each trained on a smaller set of languages (3) than the original multilingual BERT model (104). BERT variant | CroSloEngual BERT | FinEst BERT
---|---|---
Model/Language | CRO | ENG | SLO | FIN | EST
baseline+BERT | $92.63/87.84$ | $91.69/88.37$ | $95.48/93.98$ | $92.61/89.35$ | $89.84/86.51$
baseline+BERT+UPOS | $\underline{\textbf{93.29}}/\underline{\textbf{89.01}}$ | $\textbf{92.05}/\underline{\textbf{89.69}}$ | $95.72/94.27$ | $\underline{\textbf{93.39}}/\underline{\textbf{90.66}}$ | $\underline{\textbf{91.06}}/88.67$
baseline+BERT+UPOS+f. | $92.95/88.33$ | $91.99/89.65$ | $\textbf{95.82}/\textbf{94.52}$ | $93.23/90.61$ | $91.01/\underline{\textbf{88.74}}$
baseline+BERT+feats | $92.87/87.98$ | $91.27/88.58$ | $95.33/94.05$ | $93.12/90.54$ | $89.94/87.44$
Table 10: UAS/LAS achieved by monolingual BERT models. BERT variant | bert-base-uncased | bert-base-finnish-cased-v1 | RuBert
---|---|---|---
Model | ENG | FIN | RUS
baseline + BERT | $91.82/88.74$ | $94.20/91.51$ | $90.83/86.39$
baseline + BERT + UPOS | $\textbf{91.92}/89.60$ | $\textbf{94.53}/\textbf{92.19}$ | $\textbf{91.24}/\textbf{87.31}$
baseline + BERT + UPOS + f. | $91.81/\underline{\textbf{89.62}}$ | $94.08/91.73$ | $91.04/87.04$
baseline + BERT + feats | $91.23/88.76$ | $94.09/91.80$ | $90.88/86.61$

##  6 Conclusion

We analysed adding explicit morphological information in the form of
embeddings for POS tags and morphological features to two currently dominant
neural network architectures used in NLP: LSTM networks and transformer-based
BERT models. We compared models enhanced with morphological information with
baselines on three tasks (NER, DP, and NER). To obtain general conclusions, we
used subsets of eight morphologically-rich languages from different language
families.

The results indicate that adding morphological information to NER prediction
models is not beneficial, but it improves the performance in the NER and DP
tasks. For the DP task, the improvement depends on the quality of the
morphological features. The additional morphological features consistently
benefited LSTM-based models for NER and DP, both when they were of high
quality and predicted (noisy). For BERT-based models, the _predicted_ features
do not make any practical difference for the NER and DP task but improve the
performance in the DP task when they are of _high quality_. Testing different
variants of BERT shows that language specialised variants improve the
performance on the DP task and the additional morphological information is
beneficial, though less and less as we shift from multilingual towards
monolingual models.

The comparison of different BERT variants indicates that BERT models do not
completely capture the language morphology. Since the release of BERT, several
new pre-training objectives have been proposed, such as syntactic and semantic
phrase masking (Zhou et al., 2020b) and span masking (Joshi et al., 2020). In
further work, it makes sense to apply these models to the DP task in order to
test how well they capture the morphology. Further, the effect of
morphological features could be analysed on additional tasks and languages,
since the explicit morphological information does not seem to benefit them
equally.

### Acknowledgements

This paper is supported by European Union’s Horizon 2020 Programme project
EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European
News Media, grant no. 825153). The research was supported by the Slovene
Research Agency through research core funding no. P6-0411. The Titan X Pascal
used for a part of this research was donated by the NVIDIA Corporation.

## References

  * Bojanowski et al. [2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146, 2017.
  * Chen and Manning [2014] Danqi Chen and Christopher Manning.  A fast and accurate dependency parser using neural networks.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 740–750, Doha, Qatar, October 2014\. Association for Computational Linguistics.
  * Collobert and Weston [2008] Ronan Collobert and Jason Weston.  A unified architecture for natural language processing: Deep neural networks with multitask learning.  In _Proceedings of the 25th International Conference on Machine Learning_ , ICML ’08, page 160–167, New York, NY, USA, 2008. Association for Computing Machinery.  ISBN 9781605582054.
  * Conneau et al. [2018] Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni.  What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2126–2136, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * Dozat and Manning [2016] Timothy Dozat and Christopher D. Manning.  Deep biaffine attention for neural dependency parsing, 2016.
  * Edmiston [2020] Daniel Edmiston.  A systematic analysis of morphological content in BERT models for multiple languages.  arXiv:2004.03032, 2020.
  * Elazar et al. [2020] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg.  When Bert forgets how to POS: Amnesic probing of linguistic properties and MLM predictions, 2020.
  * Fortuna and Nunes [2018] Paula Fortuna and Sérgio Nunes.  A survey on automatic detection of hate speech in text.  _ACM Comput. Surv._ , 51(4), July 2018.  ISSN 0360-0300.
  * Gao and Huang [2017] Lei Gao and Ruihong Huang.  Detecting online hate speech using context aware models.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 260–266, Varna, Bulgaria, September 2017. INCOMA Ltd.
  * Georgakopoulos et al. [2018] Spiros V Georgakopoulos, Sotiris K Tasoulis, Aristidis G Vrahatis, and Vassilis P Plagianakos.  Convolutional neural networks for toxic comment classification.  In _Proceedings of the 10th Hellenic Conference on Artificial Intelligence_ , pages 1–6, 2018.
  * Güngör et al. [2017] Onur Güngör, Eray Yıldız, Suzan Uskudarli, and Tunga Gungor.  Morphological embeddings for named entity recognition in morphologically rich languages.  _arXiv preprint arXiv:1706.00506_ , 2017.
  * Han et al. [2019] Jiahui Han, Shengtan Wu, and Xinyu Liu.  jhan014 at SemEval-2019 task 6: Identifying and categorizing offensive language in social media.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 652–656, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780, November 1997.  ISSN 0899-7667.
  * Huang et al. [2015] Zhiheng Huang, Wei Xu, and Kai Yu.  Bidirectional LSTM-CRF models for sequence tagging.  _ArXiv_ , abs/1508.01991, 2015.
  * Jawahar et al. [2019] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah.  What does BERT learn about the structure of language?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3651–3657, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Ji et al. [2019] Tao Ji, Yuanbin Wu, and Man Lan.  Graph-based dependency parsing with graph neural networks.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2475–2485, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Joshi et al. [2020] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77, 2020.
  * Jurafsky and Martin [2009] Daniel Jurafsky and James H. Martin.  _Speech and Language Processing (2nd Edition)_.  Prentice-Hall, Inc., USA, 2009.  ISBN 0131873210.
  * Kanji [2006] Gopal K Kanji.  _100 statistical tests_.  Sage, 2006.
  * Kapočiūtė-Dzikienė et al. [2013] Jurgita Kapočiūtė-Dzikienė, Joakim Nivre, and Algis Krupavičius.  Lithuanian dependency parsing with rich morphological features.  In _Proceedings of the fourth workshop on statistical parsing of morphologically-rich languages_ , pages 12–21, 2013.
  * Khallash et al. [2013] Mojtaba Khallash, Ali Hadian, and Behrouz Minaei-Bidgoli.  An empirical study on the effect of morphological and lexical features in Persian dependency parsing.  In _Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 97–107, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
  * Kiperwasser and Goldberg [2016] Eliyahu Kiperwasser and Yoav Goldberg.  Simple and accurate dependency parsing using bidirectional LSTM feature representations.  _Transactions of the Association for Computational Linguistics_ , 4:313–327, 2016.
  * Kondratyuk and Straka [2019] Dan Kondratyuk and Milan Straka.  75 languages, 1 model: Parsing universal dependencies universally.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2779–2795, 2019.
  * Kulmizev et al. [2019] Artur Kulmizev, Miryam de Lhoneux, Johannes Gontrum, Elena Fano, and Joakim Nivre.  Deep contextualized word embeddings in transition-based and graph-based dependency parsing - a tale of two parsers revisited.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2755–2768, 2019.
  * Kuratov and Arkhipov [2019] Yuri Kuratov and Mikhail Arkhipov.  Adaptation of deep bidirectional multilingual transformers for Russian language, 2019.
  * Kuru et al. [2016] Onur Kuru, Ozan Arkan Can, and Deniz Yuret.  CharNER: Character-level named entity recognition.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 911–921, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee.
  * Li et al. [2018] Zuchao Li, Jiaxun Cai, Shexia He, and Hai Zhao.  Seq2seq dependency parsing.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3203–3214, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.
  * Lin et al. [2019] Yongjie Lin, Yi Chern Tan, and Robert Frank.  Open Sesame: Getting inside BERT’s linguistic knowledge, 2019.
  * Malmasi and Zampieri [2017] Shervin Malmasi and Marcos Zampieri.  Detecting hate speech in social media.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing_ , pages 467–472, 2017.
  * Marton et al. [2010] Yuval Marton, Nizar Habash, and Owen Rambow.  Improving Arabic dependency parsing with lexical and inflectional morphological features.  In _Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages_ , pages 13–21, Los Angeles, CA, USA, June 2010. Association for Computational Linguistics.
  * McDonald et al. [2005] Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajič.  Non-projective dependency parsing using spanning tree algorithms.  In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_ , pages 523–530, Vancouver, British Columbia, Canada, October 2005. Association for Computational Linguistics.
  * Miok et al. [2019] Kristian Miok, Dong Nguyen-Doan, Blaž Škrlj, Daniela Zaharie, and Marko Robnik-Šikonja.  Prediction uncertainty estimation for hate speech classification.  In _International Conference on Statistical Language and Speech Processing_ , pages 286–298. Springer, 2019.
  * Nivre [2003] Joakim Nivre.  An efficient algorithm for projective dependency parsing.  In _Proceedings of the Eighth International Conference on Parsing Technologies_ , pages 149–160, Nancy, France, April 2003.
  * Nivre et al. [2020] Joakim Nivre, Mitchell Abrams, Željko Agić, et al.  Universal Dependencies 2.6, 2020.  URL http://hdl.handle.net/11234/1-2988.  LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Pei et al. [2015] Wenzhe Pei, Tao Ge, and Baobao Chang.  An effective neural network model for graph-based dependency parsing.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 313–322, Beijing, China, July 2015. Association for Computational Linguistics.
  * Pires et al. [2019] Telmo Pires, Eva Schlinger, and Dan Garrette.  How multilingual is multilingual BERT?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4996–5001, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Qi et al. [2020] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning.  Stanza: A Python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2020.
  * Rogers et al. [2020] Anna Rogers, Olga Kovaleva, and Anna Rumshisky.  A primer in BERTology: What we know about how BERT works, 2020.
  * Scheffler et al. [2018] Tatjana Scheffler, Erik Haegert, Santichai Pornavalai, and Mino Lee Sasse.  Feature explorations for hate speech classification.  In _14th Conference on Natural Language Processing KONVENS 2018_ , volume 6, page 8, 2018.
  * Seeker and Kuhn [2011] Wolfgang Seeker and Jonas Kuhn.  On the role of explicit morphological feature representation in syntactic dependency parsing for German.  In _Proceedings of the 12th International Conference on Parsing Technologies_ , pages 58–62, Dublin, Ireland, October 2011. Association for Computational Linguistics.
  * Shtovba et al. [2019] Serhiy Shtovba, Olena Shtovba, and Mykola Petrychko.  Detection of social network toxic comments with usage of syntactic dependencies in the sentences.  In _Proceedings of the Second International Workshop on Computer Modeling and Intelligent Systems_ , pages 313–323, 2019.
  * Simeonova et al. [2019] Lilia Simeonova, Kiril Simov, Petya Osenova, and Preslav Nakov.  A morpho-syntactically informed LSTM-CRF model for named entity recognition, 2019.
  * Straka and Straková [2017] Milan Straka and Jana Straková.  Tokenizing, POS tagging, lemmatizing and parsing UD 2.0 with UDPipe.  In _Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 88–99, Vancouver, Canada, August 2017. Association for Computational Linguistics.
  * Tenney et al. [2019] Ian Tenney, Dipanjan Das, and Ellie Pavlick.  BERT rediscovers the classical NLP pipeline.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics.
  * Ulčar and Šikonja [2020] Matej Ulčar and Marko Robnik Šikonja.  FinEst BERT and CroSloEngual BERT: less is more in multilingual models, 2020.
  * Van Hee et al. [2015] Cynthia Van Hee, Els Lefever, Ben Verhoeven, Julie Mennes, Bart Desmet, Guy De Pauw, Walter Daelemans, and Véronique Hoste.  Detection and Fine-Grained Classification of Cyberbullying Events.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing_ , pages 672–680, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , page 6000–6010, 2017.
  * Virtanen et al. [2019] Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo.  Multilingual is not enough: BERT for Finnish, 2019.
  * Wilcoxon et al. [1970] Frank Wilcoxon, SK Katti, and Roberta A Wilcox.  Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test.  _Selected tables in mathematical statistics_ , 1:171–259, 1970.
  * Yamada and Matsumoto [2003] Hiroyasu Yamada and Yuji Matsumoto.  Statistical dependency analysis with support vector machines.  In _Proceedings of the Eighth International Conference on Parsing Technologies_ , pages 195–206, Nancy, France, April 2003.
  * Zampieri et al. [2019] Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar.  SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval).  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 75–86, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
  * Zampieri et al. [2020] Marcos Zampieri, Preslav Nakov, Sara Rosenthal, Pepa Atanasova, Georgi Karadzhov, Hamdy Mubarak, Leon Derczynski, Zeses Pitenis, and Çağrı Çöltekin.  SemEval-2020 task 12: Multilingual offensive language identification in social media (OffensEval 2020), 2020.
  * Zhou et al. [2020a] Houquan Zhou, Yu Zhang, Zhenghua Li, and Min Zhang.  Is POS tagging necessary or even helpful for neural dependency parsing?, 2020a.
  * Zhou et al. [2020b] Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang.  LIMIT-BERT: Linguistic informed multi-task BERT, 2020b.