# XL-WiC: A Multilingual Benchmark
for Evaluating Semantic Contextualization

Alessandro Raganato⋆♡ Tommaso Pasini⋆♢
Jose Camacho-Collados♣ Mohammad Taher Pilehvar♠
♡ Department of Digital Humanities, University of Helsinki, Finland
♢ SapienzaNLP Group, Computer Science Department, Sapienza University of Rome,
Italy
♣ School of Computer Science and Informatics, Cardiff University, United
Kingdom
♠ Tehran Institute for Advanced Studies, Iran
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The ability to correctly model distinct meanings of a word is crucial for the
effectiveness of semantic representation techniques. However, most existing
evaluation benchmarks for assessing this criterion are tied to sense
inventories (usually WordNet), restricting their usage to a small subset of
knowledge-based representation techniques. The Word-in-Context dataset (WiC)
addresses the dependence on sense inventories by reformulating the standard
disambiguation task as a binary classification problem; but, it is limited to
the English language. We put forward a large multilingual benchmark, XL-WiC,
featuring gold standards in 12 new languages from varied language families and
with different degrees of resource availability, opening room for evaluation
scenarios such as zero-shot cross-lingual transfer. We perform a series of
experiments to determine the reliability of the datasets and to set
performance baselines for several recent contextualized multilingual models.
Experimental results show that even when no tagged instances are available for
a target language, models trained solely on the English data can attain
competitive performance in the task of distinguishing different meanings of a
word, even for distant languages. XL-WiC is available at
https://pilehvar.github.io/xlwic/. ††Authors marked with a star (⋆)
contributed equally.

##  1 Introduction

One of the desirable properties of contextualized models, such as BERT Devlin
et al. (2019) and its derivatives, lies in their ability to associate dynamic
representations to words, i.e., embeddings that can change depending on the
context. This provides the basis for the model to distinguish different
meanings (senses) of words without the need to resort to an explicit sense
disambiguation step. The conventional evaluation framework for this property
has been Word Sense Disambiguation (Navigli, 2009, WSD). However, evaluation
benchmarks for WSD are usually tied to external sense inventories (often
WordNet Fellbaum (1998)), making it extremely difficult to evaluate systems
that do not explicitly model sense distinctions in the inventory, effectively
restricting the benchmark to inventory-based sense representation techniques
and WSD systems. This prevents a direct evaluation of lexical semantic
capacity for a wide range of inventory-free models, such as the dominating
language model-based contextualized representations.

Pilehvar and Camacho-Collados (2019) addressed this dependence on sense
inventories by reformulating the WSD task as a simple binary classification
problem: given a target word $w$ in two different contexts, $c_{1}$ and
$c_{2}$, the task is to identify if the same meaning (sense) of $w$ was
intended in both $c_{1}$ and $c_{2}$, or not. The task was framed as a
dataset, called Word-in-Context (WiC), which is also a part of the widely-used
SuperGLUE benchmark Wang et al. (2019). Despite allowing a significantly wider
range of models for direct WSD evaluations, WiC is limited to the English
language only, preventing the evaluation of models in other languages and in
cross-lingual settings.

In this paper, we present a new evaluation benchmark, called XL-WiC , that
extends the WiC dataset to 12 new languages from different families and with
different degrees of resource availability: Bulgarian (BG), Chinese (ZH),
Croatian (HR), Danish (DA), Dutch (NL), Estonian (ET), Farsi (FA), French
(FR), German (DE), Italian (IT), Japanese (JA) and Korean (KO). With over 80K
instances, our benchmark can serve as a reliable evaluation framework for
contextualized models in a wide range of heterogeneous languages. XL-WiC can
also serve as a suitable testbed for cross-lingual experimentation in settings
such as zero-shot or few-shot transfer across languages. As an additional
contribution, we tested several pretrained multilingual models on XL-WiC,
showing that they are generally effective in transferring sense distinction
knowledge from English to other languages in the zero-shot setting. However,
with more training data at hand for target languages, monolingual approaches
gain ground, outperforming their multilingual counterparts by a large margin.

##  2 Related Work

XL-WiC is a benchmark for inventory-independent evaluation of WSD models
(Section 2.1), while the multilingual nature of the dataset makes it an
interesting resource for experimenting with cross-lingual transfer (Section
2.2).

###  2.1 Word Sense Disambiguation

The ability to identify the intended sense of a polysemous word in a given
context is one of the fundamental problems in lexical semantics. It is usually
addressed with two different kinds of approaches relying on either sense-
annotated corpora Bevilacqua and Navigli (2020); Scarlini et al. (2020);
Blevins and Zettlemoyer (2020) or knowledge bases Moro et al. (2014); Agirre
et al. (2014); Scozzafava et al. (2020). Both are usually evaluated on
dedicated benchmarks, including at least five WSD tasks in Senseval and
SemEval series, from 2001 Edmonds and Cotton (2001) to 2015 Moro and Navigli
(2015a) that are included in the Raganato et al. (2017)’s test suite. All
these tasks are framed as classification problems, where disambiguation of a
word is defined as selecting one of the predefined senses of the word listed
by a sense inventory. This brings about different limitations such as
restricting senses only to those defined by the inventory, or forcing the WSD
system to explicitly model sense distinctions at the granularity level defined
by the inventory.

Stanford Contextual Word Similarity Huang et al. (2012) is one of the first
datasets that focuses on ambiguity but outside the boundaries of sense
inventories, and as a similarity measurement between two words in their
contexts. Pilehvar and Camacho-Collados (2019) highlighted some of the
limitations of the dataset that prevent a reliable evaluation, and proposed
the Word-in-Context (WiC) dataset. WiC is the closest dataset to ours, which
provides around 10K instances (1400 instances for 1184 unique target nouns and
verbs in the test set), but for the English language only.

###  2.2 Cross-lingual NLP

A prerequisite for research on a language is the availability of relevant
evaluation benchmarks. Given its importance, construction of multilingual
datasets has always been considered as a key contribution in NLP research and
numerous benchmarks exist for a wide range of tasks, such as semantic parsing
Hershcovich et al. (2019), word similarity Camacho-Collados et al. (2017);
Barzegar et al. (2018), sentence similarity Cer et al. (2017), or WSD Navigli
et al. (2013); Moro and Navigli (2015b). A more recent example is XTREME Hu et
al. (2020), a benchmark that covers around 40 languages in nine syntactic and
semantic tasks.

On the other hand, pre-trained language models have recently proven very
effective in transferring knowledge in cross-lingual NLP tasks Devlin et al.
(2019); Conneau et al. (2020). This has further magnified the requirement for
rigorous multilingual benchmarks that can be used as basis for this direction
of research Artetxe et al. (2020b).

Lang. |  Target Word |  Sentence 1 |  Sentence 2 | Label
---|---|---|---|---
EN | Beat | We beat the competition. | Agassi beat Becker in the tennis championship. | True
DA | Tro | Jeg tror på det, min mor fortalte. | Maria troede ikke sine egne øjne. | True
ET | Ruum | Ühel hetkel olin väljaspool aega ja ruumi. | Ümberringi oli lõputu tühi ruum. | True
FR | Causticité | Sa causticité lui a fait bien des ennemis. | La causticité des acides. | False
KO |  mj 틀림 |  mj 틀림이 있는지 없는지 세어 보시오. |  mj 그 아이 하는 짓에 틀림이 있다면 모두 이 어미 죄이지요. | False
ZH |  mj 發 |  mj 建築師希望發大火燒掉城市的三分之一。 |  mj 如果南美洲氣壓偏低，則印度可能發乾旱 | True
FA |  |  |  | False

Table 1: Sample instances from XL-WiC for different languages.

##  3 XL-WiC: The Benchmark

In this section, we describe the procedure we followed to construct the XL-WiC
benchmark. Our framework is based on the original WiC dataset, which we extend
to multiple languages.

###  3.1 English WiC

Each instance of the original WiC dataset Pilehvar and Camacho-Collados (2019)
is composed of a target word (e.g., justify) and two sentences where the
target word occurs (e.g., “Justify the margins” and “The end justifies the
means”). The task is a binary classification: to decide whether the same sense
of the target word (justify) was intended in the two contexts or not. The
dataset was built using example sentences from resources such as Wiktionary,
WordNet Miller (1995) and VerbNet Schuler et al. (2009).

###  3.2 XL-WiC

We followed Pilehvar and Camacho-Collados (2019) and constructed XL-WiC based
on example usages of words in sense inventories. Example usages are curated in
a way to be self contained and clearly distinguishable across different senses
of a word; hence, they provide a reliable basis for the binary classification
task. Specifically, for a word $w$ and for all its senses
$\\{s_{1}^{w},...,s_{n}^{w}\\}$, we extract from the inventory all the example
usages. We then pair those examples that correspond to the same sense
$s_{i}^{w}$ to form a positive instance (True label) while examples from
different senses (i.e., $s_{i}^{w}$ and $s_{j}^{w}$ where $i\neq j$) are
paired as a negative instance (False label).

We leveraged two main sense inventories for this extension: Multilingual
WordNet (Section 3.2.1) and Wiktionary (Section 3.2.2).

Split | Stat | WiC | Multilingual WordNet |  | Wiktionary
---|---|---|---|---|---
EN | BG | DA | ET | FA | HR | JA | KO | NL | ZH |  | DE | FR | IT
Train | Instances | $5428$ | – | – | – | – | – | – | – | – | – |  | $48\,042$ | $39\,428$ | $1144$
Unique Words | $1265$ | – | – | – | – | – | – | – | – | – |  | $23\,213$ | $20\,221$ | $721$
Avg. Context Len | 16.8 | – | – | – | – | – | – | – | – | – |  | 32.7 | 32.3 | 23.2
Dev | Instances | $638$ | $998$ | $852$ | $98$ | $200$ | $104$ | $208$ | $404$ | $250$ | $3046$ |  | $8870$ | $8588$ | $198$
Unique Words | $599$ | $354$ | $542$ | $63$ | $174$ | $82$ | $137$ | $183$ | $150$ | $867$ |  | $4383$ | $3517$ | $136$
Avg. Context Len | 17.1 | 8.4 | 32.6 | 19.8 | 24.8 | 17.9 | 20.8 | 5.7 | 20.1 | 45.7 |  | 32.5 | 34.3 | 23.2
Test | Instances | $1400$ | $1220$ | $3406$ | $390$ | $800$ | $408$ | $824$ | $1014$ | $1004$ | $5538$ |  | $24\,268$ | $22\,232$ | $592$
Unique Words | $1184$ | $567$ | $2088$ | $276$ | $533$ | $305$ | $476$ | $475$ | $600$ | $1888$ |  | $11\,734$ | $3517$ | $394$
Avg. Context Len | 17.2 | 8.5 | 32.6 | 19.4 | 23.5 | 18.1 | 20.6 | 6.0 | 19.9 | 46.0 |  | 32.9 | 36.4 | 23.4

Table 2: Statistics for WordNet and Wiktionary datasets for different
languages.

####  3.2.1 Multilingual WordNet

WordNet Miller (1995) is the de facto sense inventory for English WSD. The
resource was originally built as an English lexical database in 1995, but
since then there have been many efforts to extend it to other languages Bond
and Paik (2012). We took advantage of these extensions to construct XL-WiC. In
particular, we processed the WordNet versions of Bulgarian Simov and Osenova
(2010), Chinese Huang et al. (2010), Croatian Raffaelli et al. (2008), Danish
Pedersen et al. (2009), Dutch Postma et al. (2016), Estonian Vider and Orav
(2002), Japanese Isahara et al. (2008), Korean Yoon et al. (2009) and Farsi
Shamsfard et al. (2010).111We tried other WordNet versions such as Albanian
Ruci (2008), Basque Pociello et al. (2008), Catalan Benítez et al. (1998),
Galician Guinovart (2011), Hungarian Miháltz et al. (2008), Italian Pianta et
al. (2002), Slovenian Fišer et al. (2012) and Spanish Atserias et al. (2004);
but, they did not contain enough examples.

##### Farsi: Semi-automatic extraction.

FarsNet v3.0 Shamsfard et al. (2010) comprises 30K synsets with over 100K word
entries. Many of these synsets are mapped to the English database; however,
each synset provides just one example usage for a target word. This prevents
us from applying the automatic extraction of positive examples. Therefore, we
utilized a semi-automatic procedure for the construction of the Farsi set. To
this end, for each word, we extracted all example usages from FarsNet, and
asked an annotator to group them into positive and negative pairs. The
emphasis was to make a challenging dataset with sense distinctions that are
easily interpretable by humans. This can also be viewed as a case study to
understand the real gap between human and machine performance in settings
where manual curation of instances is feasible.

##### Filtering.

WordNet is often considered to be a fine-grained resource, especially for
verbs Duffield et al. (2007). In some cases, the exact meaning of a word can
be hard to assess, even for humans. For example, WordNet lists 29 distinct
meanings for the noun line, two of which correspond to the horizontally and
the vertically organized line formations. Therefore, to cope with this issue,
we followed Pilehvar and Camacho-Collados (2019) and filtered out all pairs
whose target senses were connected by an edge (including sister-sense
relations) in WordNet’s semantic network or if they belonged to the same
supersense, i.e., one of the 44 lexicographer
files222wordnet.princeton.edu/documentation/lexnames5wn in WordNet which
cluster concepts into semantic categories, e.g., Animal, Cognition, Food, etc.
mj For example, the Japanese instance “成長中の企業は大胆な指導者いなければならない” (“Growing
companies must have bold leaders”), “彼は安定した大きな企業に投資するだけだ” (“He just invested
in big stable companies”) for the target word “企業” (“company”) is discarded as
its corresponding synsets, i.e., “An organization created for business
ventures” and “An institution created to conduct business”, are grouped under
the same supersense in WordNet, i.e., Group.

Finally, all datasets are split into development333Development sets are
intended to be used for different purposes, such as training or validation, as
we will show in our experiments in Section 5.1. and test. At the end of this
step, we ensure that both test and development sets have the same number of
positive and negative instances. An excerpt of examples included in some of
our datasets are shown in Table 1.

####  3.2.2 Wiktionary

Wiktionary is one of the richest free collaborative lexical databases,
available for dozens of languages. In this online resource, each word is
provided with definitions for its various potential meanings, some of which
are paired with example usages. However, each language has a specific format,
and therefore the compilation of these examples requires a careful language-
specific parsing. We extracted examples for three European languages for which
we did not have WordNet-based data, namely French, German, and
Italian.444French WordNet, WoNeF Pradet et al. (2014), is built automatically;
German WordNet, GermaNet Hamp and Feldweg (1997), has a very restrictive
license; and the Italian WordNet Pianta et al. (2002) provides too few
examples. Once these examples were compiled, the process to build the final
dataset was analogous to that for the WordNet-based datasets (see Section
3.2.1), except for the filtering step, which was not feasible as Wiktionary
entries are not connected through paradigmatic relations as in WordNet.

For the case of Wiktionary, the number of examples was considerably higher;
therefore, we also compiled language-specific training sets, which enabled a
comparison between cross-lingual and monolingual models (see Section 5.2). All
Wiktionary datasets are split into balanced training, development and test
splits, in each of which there are equal number of positive and negative
instances.

###  3.3 Statistics

Table 2 shows the statistics of all datasets, including the total number of
instances, unique words, and the context length average.555We used the
multilingual Stanford NLP toolkit, Stanza Qi et al. (2020), with the available
pre-trained neural models. Wiktionary-based datasets are substantially larger
than the WordNet-based ones, and also provide training sets. The Chinese
datasets feature longer contexts on average and contain the largest number of
development and testing instances among WordNet-based datasets. Korean, on the
other hand, is the one with the shortest contexts, which is expected given its
agglutinative nature. As for the training corpora, German and French datasets
contain almost ten times the number of instances in the English training set.
This allows us to perform a large-scale comparison between cross-lingual and
monolingual settings (see Section 5.2) as well as a few-shot analysis (Section
6.2).

###  3.4 Validation and human performance

To verify the reliability of the datasets, we carried out manual evaluation
for those languages for which we had access to annotators. To this end, we
presented a set of 100 randomly sampled instances from each dataset to the
corresponding annotator in the target language.666For Farsi, three checker
annotators were involved in the validation, each annotating the 100-instance
subset. In this case, we report the average accuracy. Annotators were all
native speakers of the target language with high-level education. They were
provided with a minimal guideline: a brief explanation of their task and a few
tagged examples. We did not provide any lexical resource (or any other
detailed instructions) to the annotators with the emphasis to make a
challenging dataset with sense distinctions that are easily interpretable to
the layman. Given an instance, i.e., a pair of sentences containing the same
target word, their task consisted of tagging it with a True or False label,
depending on the the intended meanings of the word in the two contexts.

Table 3 reports human performance for eight datasets in XL-WiC. All accuracy
figures are around 80%, i.e., in the same ballpark as the original WiC English
dataset, which attests the reliability of underlying resources and the
construction procedure. The only exception is for Farsi, for which the checker
annotators agree with the gold labels in 97% of the instances (by average).
This corroborates our emphasis on the annotation procedure for this manually-
created dataset to have sense distinctions that are easily interpretable by
humans. As for Wiktionary, the human agreements are lower than those for the
WordNet counterparts.777Even if not part of XL-WiC, we also compiled and
validated a small Italian WordNet dataset to compare it with its Wiktionary
counterpart. Table 8 in Appendix includes an additional table comparing the
nature of these two datasets (including zero-shot cross-lingual transfer
results). This was partly expected given that the semantic network-based
filtering step (see Section 3.2.1) was not feasible for the case of Wiktionary
datasets due to the nature of the underlying resource.

WiC |  | WordNet |  | Wiktionary
---|---|---|---|---
EN |  | DA | FA | IT | JA | KO | ZH |  | DE | IT
80.0∗ |  | 87.0 | 97.0 | 82.0 | 75.0 | 76.0 | 85.0 |  | 74.0 | 78.0

Table 3: Human performance (in terms of accuracy) for different languages in
XL-WiC. ∗From the original English WiC dataset.

##  4 Experimental Setup

For our experiments, we implemented a simple, yet effective, baseline based on
a Transformer-based text encoder Vaswani et al. (2017) and a logistic
regression classifier, following Wang et al. (2019). The model takes as input
the two contexts and first tokenizes them, splitting the input words into sub-
tokens. The encoded representations of the target words are concatenated and
fed to the logistic classifier. For those cases where the target word was
split by the tokenizer into multiple sub-tokens, we followed Devlin et al.
(2019) and considered the representation of its first sub-token.

As regards the text encoder, we carried out the experiments with three
different multilingual models, i.e., the multilingual version of BERT Devlin
et al. (2019) (mBERT) and the base and large versions of XLM-RoBERTa Conneau
et al. (2020) (XLMR-base and XLMR-large, respectively). In the monolingual
setting, we used the following language-specific models: BERT-de 888
huggingface.co/dbmdz/bert-base-german-cased, CamemBERT-large Martin et al.
(2020)999huggingface.co/camembert/camembert-large, BERT-it
101010huggingface.co/dbmdz/bert-base-italian-xxl-cased, and
ParsBERT111111github.com/hooshvare/parsbert Farahani et al. (2020),
respectively, for German, French, Italian, and Farsi. As for all the other
languages covered by the WordNet datasets, i.e., Bulgarian, Chinese, Croatian,
Danish, Dutch, Estonian, Japanese and Korean, we used the pre-trained models
made available by TurkuNLP.121212github.com/TurkuNLP/wikibert We refer to each
language-specific model as L-BERT.

In all experiments we trained the baselines to minimize the binary cross-
entropy loss between their prediction and the gold label with the Adam Kingma
and Ba (2015) optimizer. Training is carried out for 10 epochs with the
learning rate fixed to $1e^{-5}$ and weight decay set to $0$. As for tuning,
results are reported for the best training checkpoint (among the 10 epochs)
according to the performance on the development set.

Model | BG | DA | ET | FA | HR | JA | KO | NL | ZH
---|---|---|---|---|---|---|---|---|---
Zero-shot cross-lingual setting Train: EN – Dev: EN
mBERT | 58.28 | 64.86 | 62.56 | 71.50 | 63.97 | 62.26 | 59.76 | 63.84 | 69.36
XLMR-base | 60.73 | 64.79 | 62.82 | 69.88 | 62.01 | 60.44 | 66.96 | 65.73 | 65.78
XLMR-large | 66.48 | 71.11 | 68.71 | 75.25 | 72.30 | 63.83 | 69.63 | 72.81 | 73.15
Test instances translated to English Train: EN – Dev: EN
mBERT | 63.52 | 62.71 | 68.46 | – | 60.54 | 63.95 | – | 66.53 | –
XLMR-base | 60.98 | 60.24 | 62.82 | – | 60.78 | 61.77 | – | 64.64 | –
XLMR-large | 64.43 | 66.64 | 63.84 | – | 69.85 | 64.44 | – | 72.11 | –
L-BERT | 64.02 | 65.38 | 64.62 | – | 69.61 | 65.90 | – | 68.43 | –
Train and Dev instances translated from English to target language Train: T-EN
– Dev: T-EN
mBERT | 56.97 | 60.25 | 59.48 | – | 66.91 | 58.13 | – | 60.06 | –
XLMR-base | 56.07 | 52.85 | 57.18 | – | 64.22 | 56.19 | – | 60.56 | –
XLMR-large | 62.13 | 63.39 | 64.87 | – | 66.18 | 59.47 | – | 66.73 | –
L-BERT | 54.26 | 60.57 | 59.49 | – | 61.52 | 58.98 | – | 60.46 | –

Table 4: Results on the WordNet test sets when using only English training
data in WiC, either in zero-shot cross-lingual setting (top block) or
translation-based settings (the lower two blocks). T-EN is a target language
dataset, automatically constructed by translating English instances in WiC.

###  4.1 Evaluation settings

We evaluated the baselines with different configuration setups, depending on
the data used for training and tuning.

##### Cross-Lingual Zero-shot.

This setting aims at assessing the capabilities of multilingual models in
transferring knowledge captured in the English language to other languages. As
training set, we used the English training set of WiC. As for tuning,
depending on the setting, we either used the English development set of WiC or
language-specific development sets of XL-WiC (Section 3.2.1). We report
results on all WordNet and Wiktionary datasets of XL-WiC, i.e., Bulgarian,
Chinese, Croatian, Danish, Dutch, Estonian, Farsi, French, German, Italian,
Japanese and Korean.

##### Multilingual Fine-Tuning.

In this setting, models are first trained on the WiC’s English training set,
and then further fine-tuned on the development sets of the target languages in
XL-WiC. Depending on the training set used, we report results for two
configurations: (i) EN+Target Language, combining with WiC’s training data and
the language-specific WordNet development sets for each language, and (ii)
EN+All Languages, combining that with all WordNet development sets for all
languages in XL-WiC, merged as one dataset.

##### Monolingual.

In this setting, we trained each model on the corresponding training set of
the target language only. For the case of WordNet datasets (where no training
sets are available), we used the development sets for training. In this case
we split each development set into two subsets with $9$:$1$ ratio (for
training and development). As for the Wiktionary datasets, we used the
corresponding training and development sets for each language (Section 3.2.2).

##### Translation.

In this last setting we make use of existing neural machine translation (NMT)
models to translate either the training or the test set, essentially reducing
the cross-lingual problem to a monolingual one. In particular, we used the
general-domain translation models from the Opus-MT
project131313github.com/Helsinki-NLP/Opus-MT Tiedemann and Thottingal (2020)
available for the following language pairs: English–Bulgarian,
English–Croatian, English–Danish, English–Dutch, English–Estonian, and
English–Japanese. The models are trained on all OPUS parallel corpora
collection Tiedemann (2012), using the state-of-the-art 6-layer Transformer-
based architecture Vaswani et al. (2017).141414More details about the NMT
models and their translation quality are given in the Appendix (Table 13). In
this configuration, as the original target word may be lost during automatic
translation, we view the task as context (sentence) similarity as proposed by
Pilehvar and Camacho-Collados (2019).151515The Appendix (Table 12) includes
another translation baseline that uses dictionary alignments to identify the
target word as comparison, but performed worse overall. Therefore, for each
model, the context vector is given by the start sentence symbol. We note that
while training custom optimized NMT models for each target language may result
in better overall performance, this is beyond the scope of this work.

##### Evaluation Metrics.

Since all datasets are balanced, we only report accuracy, i.e., the ratio of
correctly predicted instances (true positives or true negatives) to the total
number of instances.

Model | EN | BG | DA | ET | FA | HR | JA | KO | NL | ZH
---|---|---|---|---|---|---|---|---|---|---
Train: EN – Dev: Target Language
mBERT | – | 59.18 | 64.59 | 63.08 | 70.38 | 64.95 | 59.95 | 63.31 | 64.04 | 70.48
XLMR-base | – | 60.74 | 64.80 | 60.77 | 67.75 | 62.50 | 57.65 | 66.96 | 61.85 | 65.78
XLMR-large | – | 66.48 | 71.10 | 68.72 | 73.63 | 72.30 | 60.92 | 69.63 | 69.62 | 73.15
Train: EN+Target Language – Dev: EN
mBERT | – | 71.72 | 62.62 | 63.08 | 69.38 | 72.30 | 60.92 | 70.91 | 62.95 | 76.72
XLMR-base | – | 64.51 | 64.45 | 60.00 | 65.38 | 71.57 | 58.37 | 65.68 | 64.54 | 73.46
XLMR-large | – | 75.41 | 70.52 | 68.97 | 73.75 | 69.61 | 63.11 | 73.47 | 74.50 | 77.52
Train: EN+All Languages – Dev: EN
mBERT | – | 73.03 | 65.09 | 62.31 | 73.63 | 72.30 | 65.53 | 71.01 | 67.73 | 76.53
XLMR-base | – | 67.30 | 67.62 | 59.49 | 64.50 | 66.18 | 57.77 | 67.06 | 66.33 | 71.02
XLMR-large | – | 78.44 | 71.49 | 72.05 | 78.25 | 76.96 | 66.38 | 76.53 | 77.49 | 78.95
Train: Target Language – Dev: Target Language
mBERT | 66.71 | 82.30 | 62.13 | 58.21 | 63.75 | 77.45 | 61.04 | 70.71 | 64.84 | 76.09
XLMR-base | 64.36 | 79.75 | 64.00 | 64.36 | 66.25 | 79.17 | 58.86 | 70.61 | 66.33 | 78.11
XLMR-large | 70.14 | 82.05 | 66.53 | 59.23 | 68.00 | 76.72 | 55.22 | 73.08 | 69.42 | 81.83
L-BERT | 69.60 | 81.23 | 62.60 | 58.46 | 76.63 | 76.47 | 56.07 | 58.68 | 68.73 | 77.36

Table 5: Results on the WordNet test sets when using language-specific data,
either for training or for tuning.

##  5 Results

In this section, we report the results for the configurations discussed in the
previous section on the XL-WiC benchmark. We organize the experiments into two
parts, based on the test dataset: WordNet (Section 5.1) and Wiktionary
(Section 5.2).

###  5.1 WordNet datasets

##### Using English data only.

Table 4 shows results on the XL-WiC WordNet test sets, when only WiC’s English
data was used for training and tuning purposes. Across the board, XLMR-large
consistently achieves the best results, while mBERT and XLMR-base attain
scores in the same ballpark. Indeed, the massive pretraining and the number of
parameters of XLMR-large play key roles behind this lead in performance. As
regards the translation-based settings (lower two blocks), the performance
generally falls slightly behind the zero-shot cross-lingual counterpart. This
shows that the usage of good quality English data and multilingual models
provide a stronger training signal than noisy automatically-translated data.
This somehow contrasts with the observations made on other cross-lingual tasks
in XTREME Hu et al. (2020), especially in question answering datasets Artetxe
et al. (2020a); Lewis et al. (2020); Clark et al. (2020), where translating
data was generally better. This difference could perhaps be reduced with
larger monolingual models or accurate alignment, but this would further
increase the complexity, and extracting these alignments from NMT models is
not trivial Koehn and Knowles (2017); Ghader and Monz (2017); Li et al.
(2019).

##### Utilizing language-specific data.

Table 5 shows results for settings where target language-specific data was
used for training or tuning. Comparing the results in the top block (where
target language data was used for tuning) with the middle two blocks (where
target language data was instead used for training) reveals that it is more
effective to leverage the target language data for training, rather than using
it for tuning only. Overall, it is clear that adding multilingual data during
training drastically improves the results in all languages. In this case,
training (fine-tuning) is performed on a larger dataset which, despite having
examples from different languages, provides a stronger signal to the models,
enabling them to better generalize across languages. On the contrary, when
only using target language data for training and tuning (last block in the
table), results drop for most languages. This highlights the fact that having
additional training data is beneficial, reinforcing the utility of
multilingual models and cross-lingual transfer.

| Model | DE | FR | IT
---|---|---|---|---
Z-Shot | mBERT | 58.27 | 56.00 | 58.61
XLMR-base | 58.30 | 56.13 | 55.91
XLMR-large | 65.83 | 62.50 | 64.86
Mono | mBERT | 81.58 | 73.67 | 71.96
XLMR-base | 80.84 | 73.06 | 68.58
XLMR-large | 84.03 | 76.16 | 72.30
L-BERT | 82.90 | 78.14 | 72.64

Table 6: Results on the Wiktionary test sets in different training settings:
zero-shot (Z-Shot) and monolingual training (Mono). L-BERT stands for
language-specific models, i.e., BERT-de, CamemBERT-large and BERT-it for
German, French and Italian, respectively.

###  5.2 Wiktionary Datasets

In Table 6 we show results for the Wiktionary datasets. Differently from the
results reported for the WordNet datasets (Table 4), models are less effective
in the zero-shot setting, performing from $10$ to almost $20$ points lower
than their counterparts trained on data in the target language. This can be
attributed to the size of the available training data. Indeed, while in the
WordNet datasets we only have a very small amount of data at our disposal for
training (see statistics in Table 2), Wiktionary training sets are much
larger, hence providing enough data to the models to better generalize. Once
again, XLMR-large proves to be the best model in the zero-shot setting and a
competitive alternative to the language-specific models (L-BERT row) in the
monolingual setting, performing $1.1$ points higher in German and $2$ and
$0.3$ lower in French and Italian, respectively.

##  6 Analysis

In this section, we delve into the performance of the models on XL-WiC and
analyze relevant aspects about their behaviour.

###  6.1 Seen and Unseen Words

For this analysis, we aim at measuring the difference in performance when a
given target word was seen (as a target word) at training time or not. To this
end, we evaluate our baselines when trained on the German, French, and Italian
Wiktionary training sets and tested on two different subsets of the larger
language-specific Wiktionary test sets: In-Vocabulary (IV), containing only
the examples whose target word was seen at training time; and Out-of-
Vocabulary (OOV), containing only the examples whose target word was not seen
during training. We report the results in Table 7.

| Model | DE | FR | IT
---|---|---|---|---
IV | mBERT | 81.86 | 72.92 | 73.15
XLMR-base | 81.17 | 71.92 | 70.69
XLMR-large | 84.24 | 75.61 | 75.12
L-BERT | 83.23 | 77.62 | 73.89
OOV | mBERT | 70.08 | 71.24 | 68.54
XLMR-base | 71.31 | 71.14 | 62.36
XLMR-large | 72.54 | 73.93 | 65.17
L-BERT | 76.64 | 78.00 | 69.10

Table 7: Results on the in-vocabulary (IV) and out-of-vocabulary (OOV)
Wiktionary test sets. L-BERT stands for each language-specific model, i.e.,
BERT-base-de, camemBERT and BERT-base-xxl-it for German, French and Italian,
respectively.  Figure 1:  The impact of training set size (# of instances) on
performance, for the Wiktionary datasets.

In general, multilingual models are less reliable when classifying unseen
instances, lagging between 1 and $12$ points behind in performance, depending
on the language and on the model considered. This can be attributed to the
fact that their vocabulary is shared among several languages, and therefore
may have less knowledge stored about particular words that do not occur often.
The performance drop of language-specific models (L-BERT) is less pronounced,
with the French architecture (CamemBERT-large) attaining even higher
performance ($0.4$ points more) on the OOV set.

###  6.2 Few-shot Monolingual

As an additional experiment, we investigate the impact of training size on
performance. To this end, we leveraged the Wiktionary datasets for German,
French and Italian, which allow us to use varying-sized training sets, and
created $7$ training sets with 10, 25, 50, 100, 250, 500, 1000
instances.161616Bigger datasets are supersets of smaller ones.

The results of this experiment are displayed in Figure 1. When providing only
10 examples, most of the models perform similarly or even worse than random,
i.e., $50\%$ accuracy. In this setting, language-specific models (L-BERT)
attain better results than their multilingual counterparts, showing better
generalization capabilities when fewer examples are provided. This also goes
in line with what we found in the previous experiment on seen and unseen
words. With less than $5\%$ of the training data (1000 instances in French and
German and 50 instances in Italian), all models attain roughly $85\%$ of their
performance with full training data, comparable to results reported for the
zero-shot setting (Table 6).

##  7 Conclusions

In this paper we have introduced XL-WiC, a large benchmark for evaluating
context-sensitive models. XL-WiC comprises datasets for a heterogeneous set of
13 languages, including the original English data in WiC Pilehvar and Camacho-
Collados (2019), providing an evaluation framework not only for contextualized
models in those languages, but also for experimentation in a cross-lingual
transfer setting. Our evaluations show that, even though current language
models are effective performers in the zero-shot cross-lingual setting (where
no instances in the target language are provided), there is still room for
improvement, especially for far languages such as Japanese or Korean.

As for future work, we plan to investigate using languages other than English
for training (e.g., our larger French and German training sets) in our cross-
lingual transfer experiments, since English may not always be the optimal
source language Anastasopoulos and Neubig (2020). Finally, while in our
comparative analysis we have focused on a quantitative evaluation for all
languages, an additional error analysis per language would be beneficial in
revealing the weaknesses and limitations of cross-lingual models.

## Acknowledgments

We would like to thank Qincheng Zhang (Chinese), Janine Siewert (Danish),
Houman Mehrafarin, Hossein Mohebbi, and Ali Modarresi (Farsi), Ángela Collados
Aís (German), Asahi Ushio (Japanese), and Yunseo Joung (Korean) for their help
with the manual evaluation of the datasets.

Alessandro gratefully acknowledges the support of the FoTran project, funded
by the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (grant agreement No 771113), and the CSC –
IT Center for Science, Finland, for computational resources.

Tommaso gratefully acknowledges the support of the ERC Consolidator Grant
MOUSSE No. 726487 under the European Union’s Horizon 2020 research and
innovation programme.

## References

  * Agirre et al. (2014) Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. 2014.  Random walks for knowledge-based word sense disambiguation.  _Computational Linguistics_ , 40(1):57–84.
  * Anastasopoulos and Neubig (2020) Antonios Anastasopoulos and Graham Neubig. 2020.  Should all cross-lingual embeddings speak English? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8658–8679, Online. Association for Computational Linguistics.
  * Artetxe et al. (2020a) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a.  On the cross-lingual transferability of monolingual representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4623–4637, Online. Association for Computational Linguistics.
  * Artetxe et al. (2020b) Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020b.  A call for more rigor in unsupervised cross-lingual learning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7375–7388, Online. Association for Computational Linguistics.
  * Atserias et al. (2004) Jordi Atserias, Luís Villarejo, and German Rigau. 2004.  Spanish WordNet 1.6: Porting the Spanish Wordnet across Princeton versions.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal. European Language Resources Association (ELRA).
  * Barzegar et al. (2018) Siamak Barzegar, Brian Davis, Manel Zarrouk, Siegfried Handschuh, and Andre Freitas. 2018.  SemR-11: A multi-lingual gold-standard for semantic similarity and relatedness for eleven languages.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)_ , Miyazaki, Japan. European Languages Resources Association (ELRA).
  * Benítez et al. (1998) Laura Benítez, Sergi Cervell, Gerard Escudero, Mònica López, German Rigau, and Mariona Taulé. 1998.  Methods and tools for building the catalan wordnet.  _Proceedings of ELRA Workshop on Language Resources for European Minority Languages_.
  * Bevilacqua and Navigli (2020) Michele Bevilacqua and Roberto Navigli. 2020.  Breaking through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information.  In _Proc. of ACL_ , pages 2854–2864.
  * Blevins and Zettlemoyer (2020) Terra Blevins and Luke Zettlemoyer. 2020.  Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders.  In _Proc. of ACL_ , pages 1006–1017.
  * Bond and Paik (2012) Francis Bond and Kyonghee Paik. 2012.  A survey of wordnets and their licenses.  In _GWC 2012 6th International Global Wordnet Conference_ , volume 8, pages 64–71.
  * Camacho-Collados et al. (2017) Jose Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017.  SemEval-2017 task 2: Multilingual and cross-lingual semantic word similarity.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 15–26, Vancouver, Canada. Association for Computational Linguistics.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Christodouloupoulos and Steedman (2015) Christos Christodouloupoulos and Mark Steedman. 2015.  A massively parallel corpus: the bible in 100 languages.  _Language resources and evaluation_ , 49(2):375–395.
  * Clark et al. (2020) Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactionsof the Association of Computational Linguistics_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Duffield et al. (2007) Cecily Jill Duffield, Jena D Hwang, Susan Windisch Brown, Dmitriy Dligach, Sarah Vieweg, Jenny Davis, and Martha Palmer. 2007.  Criteria for the manual grouping of verb senses.  In _Proceedings of the Linguistic Annotation Workshop_ , pages 49–52.
  * Edmonds and Cotton (2001) Philip Edmonds and Scott Cotton. 2001.  SENSEVAL-2: Overview.  In _Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems_ , pages 1–5, Toulouse, France. Association for Computational Linguistics.
  * Farahani et al. (2020) Mehrdad Farahani, Mohammad Gharachorloo, Marzieh Farahani, and Mohammad Manthouri. 2020.  Parsbert: Transformer-based model for persian language understanding.
  * Fellbaum (1998) Christiane Fellbaum, editor. 1998.  _WordNet: An Electronic Database_.  MIT Press, Cambridge, MA.
  * Fišer et al. (2012) Darja Fišer, Jernej Novak, and Tomaž Erjavec. 2012.  slownet 3.0: development, extension and cleaning.  In _Proceedings of 6th International Global Wordnet Conference (GWC 2012)_ , pages 113–117.
  * Ghader and Monz (2017) Hamidreza Ghader and Christof Monz. 2017.  What does attention in neural machine translation pay attention to? In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 30–39, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Guinovart (2011) Xavier Gómez Guinovart. 2011.  Galnet: Wordnet 3.0 do galego.  _Linguamática_ , 3(1):61–67.
  * Hamp and Feldweg (1997) Birgit Hamp and Helmut Feldweg. 1997.  Germanet-a lexical-semantic net for german.  In _Automatic information extraction and building of lexical semantic resources for NLP applications_.
  * Hershcovich et al. (2019) Daniel Hershcovich, Zohar Aizenbud, Leshem Choshen, Elior Sulem, Ari Rappoport, and Omri Abend. 2019.  SemEval-2019 task 1: Cross-lingual semantic parsing with UCCA.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 1–10, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  _Proceedings of the 37th International Conference on Machine Learning (ICML)_.
  * Huang et al. (2010) Chu-Ren Huang, Shu-Kai Hsieh, Jia-Fei Hong, Yun-Zhu Chen, I-Li Su, Yong-Xiang Chen, and Sheng-Wei Huang. 2010.  Chinese Wordnet: Design, Implementation and Application of an Infrastructure for Cross-Lingual Knowledge Processing.  _Journal of Chinese Information Processing_ , 24(2):14.
  * Huang et al. (2012) Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012.  Improving word representations via global context and multiple word prototypes.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 873–882, Jeju Island, Korea. Association for Computational Linguistics.
  * Isahara et al. (2008) Hitoshi Isahara, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama, and Kyoko Kanzaki. 2008.  Development of the Japanese WordNet.  In _Sixth International conference on Language Resources and Evaluation_.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Lewis et al. (2020) Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020\.  MLQA: Evaluating cross-lingual extractive question answering.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7315–7330, Online. Association for Computational Linguistics.
  * Li et al. (2019) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019.  On the word alignment from neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1293–1303, Florence, Italy. Association for Computational Linguistics.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Miháltz et al. (2008) Márton Miháltz, Csaba Hatvani, Judit Kuti, György Szarvas, János Csirik, Gábor Prószéky, and Tamás Váradi. 2008.  Methods and Results of the Hungarian WordNet Project.  In _Proceedings of The Fourth Global WordNet Conference_ , pages 311–321.
  * Miller (1995) George A Miller. 1995.  WordNet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41.
  * Moro and Navigli (2015a) Andrea Moro and Roberto Navigli. 2015a.  SemEval-2015 task 13: Multilingual all-words sense disambiguation and entity linking.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 288–297, Denver, Colorado. Association for Computational Linguistics.
  * Moro and Navigli (2015b) Andrea Moro and Roberto Navigli. 2015b.  Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking.  _Proceedings of SemEval-2015_.
  * Moro et al. (2014) Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014.  Entity Linking meets Word Sense Disambiguation: a Unified Approach.  _Transactions of the Association for Computational Linguistics (TACL)_ , 2:231–244.
  * Navigli (2009) Roberto Navigli. 2009.  Word Sense Disambiguation: A survey.  _ACM Computing Surveys_ , 41(2):1–69.
  * Navigli et al. (2013) Roberto Navigli, David Jurgens, and Daniele Vannella. 2013.  SemEval-2013 task 12: Multilingual word sense disambiguation.  In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013)_ , pages 222–231, Atlanta, Georgia, USA. Association for Computational Linguistics.
  * Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012.  BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network.  _Artificial Intelligence_ , 193:217–250.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Pedersen et al. (2009) Bolette S. Pedersen, Sanni Nimb, Jørg Asmussen, Nicolai Hartvig Sørensen, Lars Trap-Jensen, and Henrik Lorentzen. 2009.  DanNet: the challenge of compiling a wordnet for Danish by reusing a monolingual dictionary.  _Language Resources and Evaluation_ , 43:269–299.
  * Pianta et al. (2002) Emanuele Pianta, Luisa Bentivogli, and Christian Girardi. 2002.  MultiWordNet: Developing an Aligned Multilingual Database.  In _Proceedings of the 1st International Conference on Global WordNet_ , pages 293–302.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the Word-in-Context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota.
  * Pociello et al. (2008) Eli Pociello, Antton Gurrutxaga, Eneko Agirre, Izaskun Aldezabal, and German Rigau. 2008.  WNTERM: Enriching the MCR with a terminological dictionary.  In _LREC 2008_.
  * Postma et al. (2016) Marten Postma, Emiel van Miltenburg, Roxane Segers, Anneleen Schoen, and Piek Vossen. 2016.  Open Dutch WordNet.  In _Proceedings of the Eight Global Wordnet Conference_.
  * Pradet et al. (2014) Quentin Pradet, Gaël de Chalendar, and Jeanne Desormeaux Baguenier. 2014.  Wonef, an improved, expanded and evaluated automatic french translation of wordnet.  In _Proceedings of the Seventh Global Wordnet Conference (GWC2014)_ , pages 32–39.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A Python Natural Language Processing Toolkit for Many Human Languages.  In _In Association for Computational Linguistics (ACL) System Demonstrations_.
  * Raffaelli et al. (2008) Ida Raffaelli, Marko Tadić, Božo Bekavac, and Željko Agić. 2008\.  Building croatian wordnet.  In _Fourth global wordnet conference (gwc 2008)_.
  * Raganato et al. (2017) Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017.  Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison.  In _Proc. of EACL_ , pages 99–110.
  * Ruci (2008) Ervin Ruci. 2008.  On the current state of Albanet and related applications.  Technical report, University of Vlora.  (http://fjalnet.com/technicalreportalbanet.pdf).
  * Scarlini et al. (2020) Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020.  With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation.  In _Proc. of EMNLP_.
  * Schuler et al. (2009) Karin Kipper Schuler, Anna Korhonen, and Susan Brown. 2009.  VerbNet overview, extensions, mappings and applications.  In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Tutorial Abstracts_ , NAACL-Tutorials ’09, page 13–14, USA. Association for Computational Linguistics.
  * Scozzafava et al. (2020) Federico Scozzafava, Marco Maru, Fabrizio Brignone, Giovanni Torrisi, and Roberto Navigli. 2020.  Personalized pagerank with syntagmatic information for multilingual word sense disambiguation.  In _Proc. of ACL_ , pages 37–46.
  * Shamsfard et al. (2010) Mehrnoush Shamsfard, Akbar Hesabi, Hakimeh Fadaei, Niloofar Mansoory, Payam Noor, Ali Reza Gholi Famian, Somayeh Bagherbeigi, Elham Fekri, and Maliheh Monshizadeh. 2010.  Semi automatic development of FarsNet; the Persian WordNet.  In _Proceedings of 5th global WordNet conference_.
  * Simov and Osenova (2010) Kiril Simov and Petya Osenova. 2010.  Constructing of an Ontology-based Lexicon for Bulgarian.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , Valletta, Malta. European Language Resources Association (ELRA).
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT — Building open translation services for the World.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (EAMT)_ , Lisbon, Portugal.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Vider and Orav (2002) Kadri Vider and Heili Orav. 2002.  Estonian WordNet and Lexicography.  In _Proceedings of the Eleventh International Symposium on Lexicography_ , pages 549–555.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  In _Proceedings of NeurIPS_.
  * Yoon et al. (2009) Ae-Sun Yoon, Soon-Hee Hwang, Eun-Ryoung Lee, and Hyuk-Chul Kwon. 2009.  Construction of Korean WordNet.  _Journal of KIISE: Software and Applications_ , 36(1):92–108.

## Appendix

##  Appendix A Comparison between Italian WordNet and Wiktionary datasets

In Table 8 we provide a small comparison between the Italian WordNet and
Wiktionary datasets, which is the only language that overlaps. While this
comparison is quite limited, it provides a few hints on the qualitative
differences between Wiktionary and WordNet datasets.

|  | WordNet | Wiktionary
---|---|---|---
Stats | Instances | 260 | 592
Unique Words | 105 | 394
Avg. Ctx length | 14.53 | 23.39
| Human acc. | 82.0 | 78.0
Z-shot | XLM-R base | 66.15 | 55.91
XLM-R large | 80.00 | 64.86
mBERT | 70.00 | 58.61
Table 8: Statistics and comparison between the Italian WordNet and the Italian
Wiktionary WiC datasets. Zero-shot results are computed by using the original
English WiC Pilehvar and Camacho-Collados (2019) for training and development.

##  Appendix B Models’ Parameters

In Table 9 we report the parameters of the models used in our experiments.

Model | Trainable Parameters
---|---
mBERT | 110M
BERT-large | 335M
XLMR-base | 270M
XLMR-large | 550M
Table 9: Number of parameters for our comparison systems.

##  Appendix C Additional experimental results

##### WordNet datasets.

Table 10 includes details on the variability of the results, in particular the
average results from three runs, including the standard deviation, for the
zero-shot cross-lingual setting - this is the setting producing a higher
variability in the results.

##### Wiktionary Datasets.

In Table 11 we show the development and test results in the monolingual
settings of the multilingual language models trained, tuned and tested on the
XL-WiC language-specific datasets from Wiktionary.

##### Translation setting + Dictionary alignment.

We include a setting where, after translating the English training set to each
target language, we also retrieve the corresponding translation of the English
target word through a multilingual dictionary. We use BabelNet Navigli and
Ponzetto (2012) as multilingual dictionary for all languages, discarding the
sentences where the translated target word could not be found. Table 12 shows
the results.

##  Appendix D Translation models

Translation models are trained using the MarianNMT framework Junczys-Dowmunt
et al. (2018) on a filtered version of all OPUS parallel corpora collection
using a language identifier (CLD2). As hyper-parameters, each model is based
on the base version of the Transformer architecture Vaswani et al. (2017). All
models and training details are available at https://github.com/Helsinki-
NLP/Opus-MT. To give an idea of the translation quality, Table 13 reports the
BLEU scores Papineni et al. (2002) for each model. We report the performance,
as described within the Opus-MT project, on the latest available test sets
from the series of WMT news translation shared tasks, or on 5K sentences taken
from either the Tatoeba corpus Tiedemann (2012), or the Bible corpus
Christodouloupoulos and Steedman (2015):

  * •

Bulgarian (BG): Tatoeba, model checkpoint EN$\leftrightarrow$BG
opus-2019-12-18

  * •

Danish (DA): Tatoeba, model checkpoint EN$\leftrightarrow$DA opus-2019-12-18

  * •

Estonian (ET): newstest2018, model checkpoint EN$\leftrightarrow$ET
opus-2019-12-18

  * •

Croatian (HR): Tatoeba, model checkpoint EN$\to$HR opus-2019-12-04, HR$\to$EN
opus-2019-12-05

  * •

Japanese (JA): bible-uedin, model checkpoint EN$\to$JA opus-2020-01-08,
JA$\to$EN opus-2019-12-18

  * •

Dutch (NL): Tatoeba, model checkpoint EN$\to$NL opus-2019-12-04, NL$\to$EN
opus-2019-12-05

Language | mBERT | XLMR-base | XLMR-large | Size
---|---|---|---|---
BG | $58.72\pm 1.60$ | $58.42\pm 3.74$ | $\textbf{64.70}\pm 2.05$ | $1220$
DA | $63.26\pm 2.25$ | $62.29\pm 2.34$ | $\textbf{68.91}\pm 2.45$ | $3406$
ET | $59.40\pm 2.74$ | $62.74\pm 2.43$ | $\textbf{68.29}\pm 2.97$ | $390$
FA | $67.71\pm 3.36$ | $64.67\pm 5.07$ | $\textbf{73.58}\pm 1.49$ | $800$
HR | $65.93\pm 2.25$ | $63.07\pm 2.06$ | $\textbf{68.63}\pm 3.20$ | $408$
JA | $62.58\pm 0.37$ | $59.63\pm 3.11$ | $\textbf{63.23}\pm 1.72$ | $824$
KO | $61.21\pm 2.34$ | $64.04\pm 3.93$ | $\textbf{69.40}\pm 3.61$ | $1014$
NL | $63.55\pm 1.37$ | $64.41\pm 1.40$ | $\textbf{72.14}\pm 1.88$ | $1004$
ZH | $68.85\pm 0.50$ | $61.69\pm 3.78$ | $\textbf{70.68}\pm 2.94$ | $5538$
Table 10: Zero-shot results on mBERT, XLMR-base and XLMR-large on the WordNet-based datasets when using the English WiC training and development sets.  | mBERT | XLMR-base | XLMR-large | Size
---|---|---|---|---
Language | Dev | Test | Dev | Test | Dev | Test | Train | Dev | Test
DE | 79.73 | 81.58 | 78.93 | 80.84 | 83.03 | 84.03 | $48\,042$ | $8870$ | $24\,268$
FR | 71.62 | 73.67 | 71.43 | 73.06 | 75.00 | 76.16 | $39\,428$ | $8588$ | $22\,232$
IT | 73.23 | 71.96 | 75.25 | 68.58 | 74.24 | 72.30 | $1144$ | $198$ | $592$
Table 11: Results on mBERT, XLMR-base and XLMR-large on the Wiktionary-based datasets when using the language-specific training and development data. Model | EN | BG | DA | ET | FA | HR | JA | KO | NL | ZH
---|---|---|---|---|---|---|---|---|---|---
All instances translated to Target Language + Dictionary Alignment (Train:
T-EN - Dev: T-EN)
mBERT | - | 60.66 | 60.16 | 61.79 | - | 68.87 | 52.79 | - | 57.57 | -
XLMR-base | - | 57.30 | 57.34 | 51.79 | - | 59.80 | 51.70 | - | 60.26 | -
XLMR-large | - | 63.36 | 66.27 | 61.54 | - | 66.42 | 53.88 | - | 69.42 | -
L-BERT | - | 56.31 | 58.07 | 56.67 | - | 59.31 | 53.40 | - | 58.47 | -
Table 12: Results on the WordNet test sets when using automatically-translated data with a multilingual dictionary-alignment technique. Opus-MT | BG | DA | ET | HR | JA | NL
---|---|---|---|---|---|---
EN$\to$XX | 50.0 | 60.4 | 23.3 | 48.3 | 42.1 | 57.1
XX$\to$EN | 59.4 | 63.6 | 30.3 | 58.7 | 41.7 | 60.9
Table 13: BLEU score of the translation models.


# First-order continuation method for steady-state variably saturated
groundwater flow modeling

Denis Anuprienko  Marchuk Institute of Numerical Mathematics, Russian Academy
of Sciences  Nuclear Safety Institute, Russian Academy of Sciences

###### Abstract

Recently, the nonlinearity continuation method has been used to numerically
solve boundary value problems for steady-state Richards equation. The method
can be considered as a predictor-corrector procedure with the simplest form
which has been applied to date having a trivial, zeroth-order predictor. In
this article, effect of a more sophisticated predictor technique is examined.
Numerical experiments are performed with finite volume and mimetic finite
difference discretizations on various problems, including real-life examples.

###### keywords:

Groundwater flow , Unsaturated conditions , Vadose zone , Richards equation ,
Continuation , Predictor-corrector , Finite volume , Mimetic finite difference

###### MSC:

(2010) 65H20 , 65N22

††journal: arxiv.org

##  1 Introduction

Hydrogeological modeling is widely used for problems such as water resources
management and safety assessment of hazardous objects and requires numerical
modeling of groundwater flow. The objects of interest are often located near
the surface, in partially saturated zone. In these cases, groundwater flow is
described by Richards equation [1, 2]. Nonlinearity of the Richards equation
limits applications of analytical methods and creates significant difficulties
in numerical solution process, which are mainly attributed to arising
nonlinear systems. Efficient solution techniques for both transient and
steady-state cases are subject of extensive research [5, 12].

This paper focuses on improvement of a special solution approach for boundary
value problems for the steady-state Richards equation, namely, the
nonlinearity continuation method [13]. In this procedure, the governing
Richards equation in original problem is parametrized in a way that its
”degree of nonlinearity” can be controlled. Varying the parameter, one can get
a sequence of problems from a simple linear problem to the original one. Each
problem in the sequence is then solved with an iterative solver (Newton,
Picard or their combinations [14]) with the solution of previous problem given
as initial guess. Following [15], this procedure belongs to the class of
predictor-corrector continuation methods with a trivial, zeroth-order
predictor step and the corrector step being application of an iterative
nonlinear solver.

This paper contributes to the nonlinearity continuation method for the steady-
state Richards equation in the following ways.

  1. 1.

The nonlinearity continuation method is formulated in predictor-corrector
terms, the first-order predictor is applied and its effect on performance is
examined;

  2. 2.

Discretization framework, which was previously limited to finite volume
schemes, is extended to mimetic finite difference method in order to test the
nonlinearity continuation procedure on structurally different nonlinear
systems.

The paper is organized as follows. The second section contains mathematical
description of the groundwater flow model, the third section describes
discretization and resulting nonlinear systems, the fourth section formulates
nonlinearity continuation in predictor-corrector terms, the fifth section
presents results of numerical experiments, and the conclusion provides some
discussion.

##  2 Groundwater flow problem in variably saturated conditions

###  2.1 Governing partial differential equations

From the mass balance equation for water

$\nabla\cdot\mathbf{q}=Q$ (1)

and the Darcy law for water flux

$\mathbf{q}=-K_{r}(\theta)\mathbb{K}\nabla\left(\psi+z\right)$ (2)

the steady-state Richards equation [1, 2]

$-\nabla\cdot\left(K_{r}(\theta)\mathbb{K}\nabla\left(\psi+z\right)\right)=Q$
(3)

is derived. Here the following variables are used:

  * 1.

$\mathbf{q}$ – water flux;

  * 2.

$\psi$ – capillary pressure head;

  * 3.

$\theta(\psi)$ – volumetric water content in medium;

  * 4.

$\mathbb{K}$ – hydraulic conductivity tensor, a 3$\times$3 s.p.d. matrix; for
isotropic medium this tensor is scalar and can be characterized by one number
$K$;

  * 5.

$K_{r}(\theta)$ – relative permeability for water in medium;

  * 6.

$Q$ – specific sink and source terms.

It is perhaps more convenient to work with the steady-state Richards equation
(3) in terms of hydraulic head $h=\psi+z$ (here $z$ is the vertical
coordinate). Indeed, the relative permeability can be rewritten as a function
of hydraulic head $h$ in the following way:
$K_{r}(\theta)=K_{r}(\theta(\psi))=K_{r}(\theta(h-z))$ and for simplicity
denoted as $K_{r}(h)$. Then, one can formulate steady-state Richards equation
in the following form:

$-\nabla\cdot\left(K_{r}(h)\mathbb{K}\nabla h\right)=Q.$ (4)

The same applies to the flux expression (2), and it becomes

$\mathbf{q}=-K_{r}(h)\mathbb{K}\nabla h.$ (5)

To completely define the boundary value problem, boundary conditions and
additional constitutive relationships between water content $\theta$, water
pressure head $\psi$ and relative permeability $K_{r}$ are required.

###  2.2 Boundary conditions

The following boundary conditions are used for the steady-state Richards
equation:

  * 1.

Dirichlet: specified hydraulic head $h$;

  * 2.

Neumann: specified normal flux $\mathbf{q}\cdot\mathbf{n}$.

###  2.3 Van Genuchten – Mualem constitutive relationships

This model is widely used in vadose zone applications and is capable of
predicting capillary effects in soil. It is based on nonlinear functions for
$\theta(\psi)$ and $K_{r}(\theta)$ proposed by van Genuchten [16] and Mualem
[17]:

$\theta(\psi)=\theta_{r}+\frac{\theta_{s}-\theta_{r}}{(1+|\alpha\psi|^{n})^{m}},$
(6)

$K_{r}(\theta)=S_{e}^{1/2}\cdot\left(1-\left(1-S_{e}^{1/m}\right)^{m}\right)^{2},$
(7)

where $\theta_{s}$ is the water content at full saturation, $\theta_{r}$ is
the residual water content,
$S_{e}=(\theta-\theta_{r})/(\theta_{s}-\theta_{r})$ is the effective
saturation and $n>1$, $m=1-1/n$ and $\alpha$ are parameters of the model.

##  3 Discretization techniques

Complex geometries and layered structure of geological domains, as well as
presence of small objects like wells, imply use of unstructured grids in
groundwater modeling applications. Two types of grids are considered in this
paper: triangular prismatic grids with occasional occurrence of other
polyhedra and hexahedral grids with octree refinement and possibility to cut
cells for better representation of the boundaries [18]. In practical
applications, these meshes often contain flattened cells that may be some
arbitrary polyhedra.

Therefore, the discretization scheme should be able to deal with cells of
quite general shape. Other desirable properties include mass conservation on
discrete level, absence of non-physical solutions and low computational
complexity. Discretizations considered in this paper include finite volume
(FV) and mimetic finite difference (MFD) methods. These are techniques for
discretization of diffusion-type operators and have been employed in various
applications such as simulation of groundwater flow or multiphase flow in
porous media. While finite volume schemes have been already employed in
earlier work [13, 14], the mimetic scheme is considered for the first time
within the nonlinearity continuation method and is included in order to test
the solution strategy for discretization other than finite volume.

###  3.1 Finite volume schemes

The finite volume method (see, for example, [19]) is a common choice for
subsurface applications due to its local conservativity and ability to handle
meshes with polyhedral cells. After integrating equation (4) over a cell and
using the divergence theorem, one gets a sum of normal fluxes over the cell
faces. Approximation of these fluxes is the key point of a finite volume
scheme. The simplest linear two-point flux approximation (TPFA) remains the
most popular choice for its simplicity. However, TPFA does not give
approximation for non-$\mathbb{K}$-orthogonal grids, which are very common in
practical applications. Other FV schemes include linear multi-point flux
approximations [20, 21, 22] and schemes with nonlinear two- and multi-point
flux approximations that are monotone or satisfy discrete maximum principle
for diffusion equation [23, 24, 25, 26, 27, 28].

Of all finite volume schemes described, the following schemes are considered
in this work:

  * 1.

linear TPFA;

  * 2.

MPFA-O scheme [20];

  * 3.

nonlinear two-point flux approximation (NTPFA-B) [28];

  * 4.

nonlinear multi-point flux approximation (NMPFA-B) [28].

The relative permeability $K_{r}(h)$ for a face can be either calculated as a
half-sum of values from the two neighboring cells (central approximation) or
taken from the cell with greater water head value (upwind approximation). In
the author’s experience, central approximation leads to better convergence of
nonlinear solvers since the structure of arising system does not change during
iterations, but results in saturation oscillations. Upwind approximation gives
more physically reasonable saturation distributions, but can result in severe
convergence problems of nonlinear solvers.

Discretization leads to a system of nonlinear equations which takes the form

$F(h)\equiv A(h)h-b(h)=0,$ (8)

where $A(h)$ is a matrix with solution-dependent coefficients.

###  3.2 Mimetic finite difference scheme

The mimetic finite difference method is a way to construct discretization
schemes with discrete analogues of differential operators that mimic
properties of their continuous counterparts. An extensive overview of MFD
history may be found in [29], while theoretical foundations, derivation of
mimetic schemes and their applications are presented in [30]. Recently,
mimetic schemes have been applied in subsurface flows simulation, including
groundwater flow described by Richards equation [31, 32] and multiphase flows
[33].

Mimetic approach benefits from Richards equation considered as two first-order
PDEs (1) and (5). These equations represent diffusion problem in mixed
formwith nonlinear solution-dependent coefficient. The scheme used in this
work chooses the divergence as the primary operator and uses finite-volume-
like approximation of it in cells, while the approximation of the gradient
operator is built in such a way that it is negatively adjoint to the discrete
divergence in appropriate discrete spaces with their scalar products. Details
on the discrete gradient construction can be found in [34]. The presence of
solution-dependent relative permeability in (5) makes the construction of the
scheme more complicated, since some approximations of nonlinear coefficient
may lead to incorrect solutions for transient equations [35]. Following [31],
upwind approximation of relative permeability was used in present work.

The resulting mimetic scheme uses cell-centered hydraulic head unknowns (one
value per cell) and face-centered flux unknowns (one value per face,
representing the normal flux). The respective nonlinear system, therefore, has
significantly larger size compared to cell-centered finite volume schemes.
That complexity is the price to pay for a number of attractive properties,
among which are

  * 1.

ability to work with a wide range of cell types, including degenerate and non-
convex cells;

  * 2.

simultaneous calculation of fluxes;

  * 3.

easy incorporation of boundary conditions.

Another important property of the scheme is its similarity to finite volume
schemes, which results from the same assignation of mesh unknowns and the same
approximation of divergence operator. This ensures local conservativity and
allows for easy incorporation into existing finite volume framework, including
some constitutive relationships and procedures designed specifically for
finite volume paradigm [9].

##  4 Nonlinearity continuation method in predictor-corrector terms and first
order predictor

###  4.1 General ideas

Both finite volume and mimetic schemes result in a system of nonlinear
equations of the form

$F(x)=0,$ (9)

where $x$ is the respective vector of discrete unknowns (for finite volume
schemes it consists of cell-based head unknowns, while for the mimetic scheme
it also contains face-based fluxes). An obvious way to solve the system is to
apply standard iterative solvers such as Newton and Picard methods. However,
these fail often. Newton method needs a good initial guess, finding which for
a complicated flow structure is a hard task which becomes harder in case of an
advanced discretization. Picard needs contraction properties of discrete
operator and often suffers from slow convergence or lack thereof. Additional
difficulties result from upwinding of the relative permeability, since it can
change function $F(x)$ during iterations. Relaxation and line search methods
[4, 5, 36], as well as using Picard iterations to improve the initial guess
[4, 14], may improve the robustness, but still are not effective enough.

A more powerful approach is the nonlinearity continuation method [13]. The
relative permeability function $K_{r}(h)$, which makes the equation nonlinear,
is parametrized with a continuation parameter $q\in[0;1]$ such that the
resulting function, $\mathcal{K}(h,q)$, has the following properties:

  * 1.

$\mathcal{K}(h,0)\equiv 1$;

  * 2.

$\mathcal{K}(h,1)\equiv K_{r}(h)$.

From these properties it follows that with $q=1$ one gets the original steady-
state Richards equation, and with $q=0$ one gets a linear equation
representing the full saturation case.

Several options to choose function $\mathcal{K}(h,q)$ exist, for example:

  * 1.

linear function $\mathcal{K}_{lin}(h,q)=1+q\cdot(K_{r}(h)-1)$;

  * 2.

power function $\mathcal{K}_{pow}(h,q)=(K_{r}(h))^{q}$.

In this paper, only power fucntion $\mathcal{K}_{pow}$ is considered.

Nonlinear systems arising from discretization of the problems for parametrized
Richards equation with parameter $q$ are denoted as

$F(x_{q},q)=0,$ (10)

where the subscript in $x_{q}$ emphasizes that solution corresponds to a
certain value of the continuation parameter.

The nonlinearity continuation method in its previously studied form [13, 14]
is a stepping procedure. It starts with $q=0$ and gradually increases it to
$q=1$. This way a sequence of problems with different $q$ is obtained. Each
problem in the sequence is solved with an iterative solver: Newton, Picard or
a mix of the two, all combined with a line search technique. The solver takes
solution of the previous problem as initial guess. For the first problem, with
$q=0$, there is no such initial guess, and a constant hydraulic head
distribution is chosen as initial guess; however, since this problem with
$q=0$ is linear (it represents the fully saturated case), iterative solvers
converge easier (for linear discretization schemes they converge in 1
iteration). The number of problems in the sequence or, in other words, number
of steps from $q=0$ to $q=1$, is not known a priori. These steps are chosen
within the solution process.

###  4.2 Predictor-corrector formulation

The nonlinearity continuation procedure described above lies in the class of
predictor-corrector methods [15], where the corrector step is application of
an iterative solver and the predictor step is setting initial guess for the
said solver. Outline of the method with emphasized predictor and corrector
steps is presented in algorithm 1. In its form studied in [13, 14], the
nonlinearity continuation method has a trivial, or zeroth-order predictor: it
simply sets solution of the previous problem as initial guess for the next
problem.

$q=0$;

$x=\text{const}$;

Find $x$: $F(x,0)=0$ with an iterative solver;

$\Delta q_{last}=1$;

while  _$q <1$_ do

$\Delta q=\min(1-q~{},2\cdot\Delta q_{last})$;

while  _$\Delta q >10^{-4}$_ do

1\. Predictor step:

set initial guess for $x_{q+\Delta q}$;

2\. Corrector step:

Find $x_{q+\Delta q}$: $F(x_{q+\Delta q},q+\Delta q)=0$ with an iterative
solver;

if  _solved_ then

$x=x_{q+\Delta q}$;

$q=q+\Delta q$;

$\Delta q_{last}=\Delta q$;

break;

else

$\Delta q=\Delta q/2$;

end if

end while

if  _didn’t find $\Delta q$_ then

failed;

stop;

end if

end while

Algorithm 1 Nonlinearity continuation method in predictor-corrector terms

###  4.3 First-order predictor

In algorithm 1 the predictor step is setting initial guess for $x_{q+\Delta
q}$. Previously, a trivial, zeroth-order predictor was used, which has the
form

$x_{q+\Delta q}^{0}=x_{q}.$ (11)

A more sophisticated predictor may be used. In this so-called first-order
predictor a special vector $\frac{\partial x}{\partial q}$ is calculated,
which describes sensitivity of the solution $x$ to the continuation parameter
$q$. After a problem with $q=q^{*}$ is solved, this vector is calculated from
the linear system

$J(x_{q^{*}},q^{*})\frac{\partial x}{\partial q}=-\frac{\partial F}{\partial
q}\left(x_{q^{*}},q^{*}\right)\approx-\frac{F\left(x_{q^{*}},q^{*}+\delta\right)-F\left(x_{q^{*}},q^{*}\right)}{\delta},$
(12)

where $J(x_{q^{*}},q^{*})$ is the Jacobian matrix calculated for the last
obtained solution, and the right-hand side is approximated with a finite-
difference expression. The value of of $\delta$ is often chosen to be around
square root of machine epsilon [37] for rather simple problems, and in this
work $\delta=10^{-7}$ is chosen.

###  4.4 Details on the corrector

Different nonlinear solvers may be used as correctors. Previous work [14]
studied Newton and Picard solvers as well as their combination. Experiments
showed that the use of mixed solver may help to complete the nonlinearity
continuation procedure in 1 step for simpler constitutive model from [9], but
requires setting parameters, such as initial number of Picard iterations and
initial relaxation coefficient. For the VGM model, the mixed solver gave no
clear advantage. This, and the fact that the first-order predictor improves
initial guess (which is more crucial for the Newton method) are the reasons to
use pure Newton method in this paper. Additionally, line search in accordance
with Armijo rule is used [36, 14], but it is not applied for the first 5
iterations. Convergence is controlled by the tolerances $\varepsilon_{rel}$
and $\varepsilon_{abs}$, and not more than $maxit$ iterations are allowed.

##  5 Numerical experiments

The purpose of numerical experiments is to study effect of first-order
predictor on the number of continuation steps and nonlinear iterations,
compared to the zeroth-order predictor.

###  5.1 Implementation details

All methods described in this paper are implemented in C++ as a part of the
GeRa (Geomigration of Radionuclides) software [38, 39]. GeRa was originally
designed for safety assessment of radioactive waste repositories and nowadays
is a general-purpose hydrogeological modelling tool. Computational core of
GeRa is based on INMOST [40, 41], a platform which provides tools for
numerical modeling on general unstructured grids with built-in MPI
parallelization.

Of the discretization schemes used, only TPFA and MPFA-O are available in
GeRa. NTPFA-B and NMPFA-B schemes are included for research purposes from an
external package [28]. The MFD scheme has been implemented by the author
recently and has not been used extensively yet.

Automatic differentiation module of INMOST is used to obtain the Jacobian
matrix, and the corresponding linear systems are solved with Inner_MPTILUC,
which is an internal INMOST solver based on Bi-CGSTAB preconditioned by second
order Crout-ILU decomposition [42] with inverse-based condition estimation
[43] and maximum product transversal reordering [44].

###  5.2 Capillary barrier test

The first problem, the capillary barrier test, describes infiltration in a
two-layered domain. It was first presented in [45], and later used for
verification of modeling software, including TOUGH2 [46], FEFLOW [47] and GeRa
[48]. The problem is essentially two-dimensional and features domain
(presented at figure 1) which is approximately 100 m long and 6 m high. The
domain is composed of two inclined layers (angled at 5%), each 0.5 m thick:
the upper layer is sand, the lower layer is gravel. The media parameters are
given in the table 1. On the lower and right boundaries zero hydraulic head
condition is imposed, on the top boundary the infiltration rate of 0.0048
m/day is prescribed.

An important feature of the capillary barrier is the high contrast in media
properties. In fully saturated conditions gravel is much more permeable than
sand. However, in partially saturated state it is less permeable, which leads
to occurrence of the capillary barrier effect. The complicated structure of
the total conductivity curve contributed to the choice of power continuation
function $\mathcal{K}_{pow}$ in the present work.

Figure 1: Computational domain and boundary conditions for the capillary barrier test (stretched 5 times vertically) |  $K$, m/day |  $\alpha$, 1/m | $n$ | $\theta_{r}$ | $\theta_{s}$
---|---|---|---|---|---
Sand | 18.144 | 3.9 | 5.74 | 0.154 | 0.39
Gravel | 8640 | 490 | 2.19 | 0.011 | 0.42
Table 1: Media paramaters for the capillary barrier test

Numerical experiments were conducted on a 3200-cell hexahedral grid and the
following Newton parameters: $\varepsilon_{rel}=\varepsilon_{abs}=10^{-5}$,
$maxit=25$. Upwind approximation of $K_{r}$ was used. Computed hydraulic head
and water saturation distrbutions (which vary very slightly for different
discretizations) are depicted in figure 2, while solution characteristics are
presented in table 2. For the first three schemes the first-order predictor
was able to decrease total computation time by up to 40% by reducing number of
continuation steps and Newton iterations. For the NMPFA-B scheme, the first-
order predictor decreased the number of steps and iterations too, but the
total time did not decrease since the number of function evaluations in the
line search increased. For the MFD scheme, the first-order predictor gave no
advantage. It should be noted that solution with the MFD scheme was the
fastest, although it is the most expensive scheme on per-iteration basis, and
the reason for this is the lowest number of iterations.

|  $T_{comp}$, s | # of successful (failed) steps | # of Newton iterations
---|---|---|---
TPFA, 0 | 89.7 | 33 (35) | 829
1 | 51.9 | 27 (28) | 463
MPFA-O, 0 | 375.0 | 31 (33) | 667
1 | 281.5 | 24 (25) | 432
NTPFA-B, 0 | 48.7 | 35 (37) | 409
1 | 29.3 | 26 (27) | 265
NMPFA-B, 0 | 63.0 | 32 (34) | 493
1 | 63.2 | 25 (27) | 413
MFD, 0 | 35.8 | 13 (14) | 117
1 | 57.0 | 23 (24) | 198
Table 2: Comparison of zero- and first-order predictors for the capillary
barrier test, power continuation function $\mathcal{K}_{pow}$

Figure 2: Hydraulic head and water saturation distributions for the capillary
barrier test (stretched 5 times vertically)

###  5.3 Realistic case

The second problem involves groundwater flow in a vertical cross-section of a
real-life waste repository located near ground surface. The domain (see fig.
3) is approximately 90 m long and 12 m high and is composed of 6 isotropic
materials with their properties listed in table 3. Rainfall recharge rate of
0.0001 m/day is prescribed on the top boundary, while on other boundaries
hydraulic head vale of 0.5 m is imposed.

Several features contribute to difficulty of solving this problem:

  * 1.

Hydraulic conductivity $K$ ranges from 0.048 to 100 m/day with jumps up to 3
orders of magnitude at some interfaces;

  * 2.

Relative permeability curves (7) are non-smooth for some materials;

  * 3.

Dry zones as well as multiple fully saturated ones are present;

  * 4.

Central approximation of $K_{r}$ results in significant unphysical
oscillations of the water saturation, and upwind approximation should be used
instead, which leads to convergence problems;

  * 5.

As often done in practice, computational mesh was coarsened in larger
subdomains to reduce number of cells. This resulted in considerable
non-$\mathbb{K}$-orthogonality and problems with TPFA were expected.

The mesh consisted of 2800 cells. Solution was performed with the following
Newton parameters: $\varepsilon_{abs}=\varepsilon_{rel}=10^{-4}$, $maxit=20$.
The problem turned out to be troublesome with only two schemes finishing: TPFA
and MFD. Other schemes failed due to inability to find admissible $\Delta q$.
Solution process characteristics are presented in table 4 and show that the
first-order predictor reduced total computational time by more than 50% for
TPFA and by almost 40% for MFD. In MFD case, the total number of continuation
steps increased while the total number of iterations decreased. This may be
attributed to the fact that inner Newton solver converges faster when first-
order prediction is good and diverges faster when it is not.

Calculated water saturation distributions are presented in figure 4. As
expected, TPFA produced some numerical artifacts, namely, ”checkerboarding” in
saturation (see figure 5) which likely results from inconsistent flux
approximation. MFD, on the other hand, produced reasonable results.

Figure 3: Computational domain and boundary conditions for the realistic case |  $K$, m/day | $\alpha$ | $n$ | $\theta_{r}$ | $\theta_{s}$
---|---|---|---|---|---
Clay | 0.048 | 0.8 | 1.09 | 0.068 | 0.38
Soil | 0.1 | 5.9 | 3 | 0.1 | 0.35
Gravel | 100 | 6 | 3 | 0.04 | 0.3
Clay loam | 0.2496 | 3.6 | 1.56 | 0.078 | 0.43
Sand | 7.128 | 14.5 | 2.68 | 0.045 | 0.43
Repository | 0.1 | 14.5 | 3 | 0.045 | 0.4
Table 3: Media properties for the realistic case |  $T_{comp}$, s | # of successful (failed) steps | # of Newton iterations
---|---|---|---
TPFA, 0 | 47.2 | 17 (36) | 383
1 | 20.1 | 11 (23) | 186
MFD, 0 | 335.5 | 18 (39) | 510
1 | 185.5 | 19 (41) | 312
Table 4: Comparison of zero- and first-order predictor for the realistic test

Figure 4: Water saturation distributions computed for the realistic case with
TPFA (top) and MFD (bottom) Figure 5: ”Checkerboarding” in water saturation
observed for TPFA in a part of the domain

##  6 Conclusion

The nonlinearity continuation method, which has been recently applied for
boundary value problems for the steady-state Richards equation, can be
considered as a predictor-corrector procedure. In its previously studied form,
the method had a trivial zeroth-order predictor. In this paper, a more
sophisticated first-order predictor was considered. The cost of such predictor
is solution of one linear system per continuation steps.

The first-order predictor was compared to the zeroth-order one on two test
cases with finite volume and mimetic finite difference discretizations. Both
test cases featured highly nonlinear constitutive relationships. In most
cases, the first-order predictor reduced computational time by decreasing
number of continuation steps and Newton iterations.

A part of the work was testing the mimetic finite difference discretization
scheme in terms of nonlinear solvers performance. Surprisingly, the scheme was
the fastest to produce solution for the capillary barrier test. This fact is
attributed to small number of continuation steps. For the realistic test case,
it was the only scheme besides TPFA which completed solution. For the
capillary barrier test the first-order predictor gave no advantage, while for
the realistic test case it reduced computational time significantly.

## Acknowledgements

The reported study was partially funded by Russian Foundation for Basic
Research (RFBR), project number 20-31-90126.

## References

  * [1] L. A. Richards, Capillary conduction of liquids through porous mediums, Physics 1 (5) (1931) 318–333.
  * [2] J. Bear, A. H.-D. Cheng, Modeling groundwater flow and contaminant transport, Vol. 23, Springer Science & Business Media, 2010.
  * [3] M. A. Celia, E. T. Bouloutas, R. L. Zarba, A general mass-conservative numerical solution for the unsaturated flow equation, Water Resources Research 26 (7) (1990) 1483–1496.
  * [4] C. Paniconi, M. Putti, A comparison of Picard and Newton iteration in the numerical solution of multidimensional variably saturated flow problems, Water Resources Research 30 (12) (1994) 3357–3374.
  * [5] M. W. Farthing, C. E. Kees, T. S. Coffey, C. T. Kelley, C. T. Miller, Efficient steady-state solution techniques for variably saturated groundwater flow, Advances in Water Resources 26 (8) (2003) 833–849.
  * [6] C. T. Miller, G. A. Williams, C. T. Kelley, M. D. Tocci, Robust solution of richards’ equation for nonuniform porous media, Water Resources Research 34 (10) (1998) 2599–2610.
  * [7] J. E. Jones, C. S. Woodward, Newton–Krylov-multigrid solvers for large-scale, highly heterogeneous, variably saturated flow problems, Advances in Water Resources 24 (7) (2001) 763–774.
  * [8] H.-J. Diersch, P. Perrochet, On the primary variable switching technique for simulating unsaturated–saturated flows, Advances in Water Resources 23 (3) (1999) 271–301.
  * [9] D. V. Anuprienko, I. V. Kapyrin, Modeling groundwater flow in unconfined conditions: numerical model and solvers’ efficiency, Lobachevskii Journal of Mathematics 39 (7) (2018) 867–873.
  * [10] W. Jäger, J. Kačur, Solution of porous medium type systems by linear approximation schemes, Numerische Mathematik 60 (1) (1991) 407–427.
  * [11] I. S. Pop, F. Radu, P. Knabner, Mixed finite elements for the Richards’ equation: linearization procedure, Journal of Computational and Applied Mathematics 168 (1-2) (2004) 365–373.
  * [12] M. W. Farthing, F. L. Ogden, Numerical solution of Richards’ equation: a review of advances and challenges, Soil Science Society of America Journal 81 (6) (2017) 1257–1269.
  * [13] D. Anuprienko, I. Kapyrin, Nonlinearity continuation method for steady-state groundwater flow modeling in variably saturated conditions, Journal of Computational and Applied Mathematics (2021) 113502.
  * [14] D. Anuprienko, Comparison of nonlinear solvers within continuation method for steady-state variably saturated groundwater flow modeling, Russian Journal of Numerical Analysis and Mathematical Modeling 36 (4) (2021) 183–195.
  * [15] E. L. Allgower, K. Georg, Introduction to numerical continuation methods, SIAM, 2003\.
  * [16] M. T. Van Genuchten, A closed-form equation for predicting the hydraulic conductivity of unsaturated soils 1, Soil Science Society of America Journal 44 (5) (1980) 892–898.
  * [17] Y. Mualem, A new model for predicting the hydraulic conductivity of unsaturated porous media, Water Resources Research 12 (3) (1976) 513–522.
  * [18] A. V. Plenkin, A. Y. Chernyshenko, V. N. Chugunov, I. V. Kapyrin, Adaptive unstructured mesh generation methods for hydrogeological problems, Numerical Methods and Programming 16 (2015) 518–533.
  * [19] R. Eymard, T. Gallouët, R. Herbin, Finite volume methods, Handbook of numerical analysis 7 (2000) 713–1018.
  * [20] I. Aavatsmark, T. Barkve, O. Bøe, T. Mannseth, Discretization on unstructured grids for inhomogeneous, anisotropic media. Part I: Derivation of the methods, SIAM Journal on Scientific Computing 19 (5) (1998) 1700–1716.
  * [21] L. Agelas, D. Di Pietro, I. Kapyrin, R. Masson, The mpfa g scheme for heterogeneous anisotropic diffusion problems on general meshes, in: ECMOR XI-11th European Conference on the Mathematics of Oil Recovery, European Association of Geoscientists & Engineers, 2008, pp. cp–62.
  * [22] L. Agélas, D. A. Di Pietro, J. Droniou, The g method for heterogeneous anisotropic diffusion on general meshes, ESAIM: Mathematical Modelling and Numerical Analysis 44 (4) (2010) 597–625.
  * [23] C. Le Potier, Schéma volumes finis monotone pour des opérateurs de diffusion fortement anisotropes sur des maillages de triangles non structurés, Comptes Rendus Mathematique 341 (12) (2005) 787–792.
  * [24] I. Kapyrin, A family of monotone methods for the numerical solution of three-dimensional diffusion problems on unstructured tetrahedral meshes, in: Doklady Mathematics, Vol. 76, Nauka/Interperiodica, 2007, pp. 734–738.
  * [25] A. Danilov, Y. V. Vassilevski, A monotone nonlinear finite volume method for diffusion equations on conformal polyhedral meshes.
  * [26] O. Misiats, K. Lipnikov, Second-order accurate monotone finite volume scheme for richards’ equation, Journal of Computational Physics 239 (2013) 123–137.
  * [27] K. Nikitin, K. Novikov, Y. Vassilevski, Nonlinear finite volume method with discrete maximum principle for the two-phase flow model, Lobachevskii Journal of Mathematics 37 (5) (2016) 570–581.
  * [28] K. M. Terekhov, B. T. Mallison, H. A. Tchelepi, Cell-centered nonlinear finite-volume methods for the heterogeneous anisotropic diffusion problem, Journal of Computational Physics 330 (2017) 245–267.
  * [29] K. Lipnikov, G. Manzini, M. Shashkov, Mimetic finite difference method, Journal of Computational Physics 257 (2014) 1163–1227.
  * [30] L. B. da Veiga, K. Lipnikov, G. Manzini, The mimetic finite difference method for elliptic problems, Vol. 11, Springer, 2014.
  * [31] K. Lipnikov, D. Moulton, D. Svyatskiy, New preconditioning strategy for Jacobian-free solvers for variably saturated flows with Richards’ equation, Advances in Water Resources 94 (2016) 11–22.
  * [32] E. T. Coon, J. D. Moulton, E. Kikinzon, M. Berndt, G. Manzini, R. Garimella, K. Lipnikov, S. L. Painter, Coupling surface flow and subsurface flow in complex soil structures using mimetic finite differences, Advances in Water Resources 144 (2020) 103701.
  * [33] A. S. Abushaikha, K. M. Terekhov, A fully implicit mimetic finite difference scheme for general purpose subsurface reservoir simulation with full tensor permeability, Journal of Computational Physics 406 (2020) 109194.
  * [34] K. Lipnikov, G. Manzini, Discretization of mixed formulations of elliptic problems on polyhedral meshes, in: Building Bridges: Connections and Challenges in Modern Approaches to Numerical Partial Differential Equations, Springer, 2016, pp. 311–342.
  * [35] K. Lipnikov, G. Manzini, J. D. Moulton, M. Shashkov, The mimetic finite difference method for elliptic and parabolic problems with a staggered discretization of diffusion coefficient, Journal of Computational Physics 305 (2016) 111–126.
  * [36] L. Armijo, Minimization of functions having Lipschitz continuous first partial derivatives, Pacific Journal of Mathematics 16 (1) (1966) 1–3.
  * [37] D. A. Knoll, D. E. Keyes, Jacobian-free newton–krylov methods: a survey of approaches and applications, Journal of Computational Physics 193 (2) (2004) 357–397.
  * [38] I. Kapyrin, V. Ivanov, G. Kopytov, S. Utkin, Integral code GeRa for RAW disposal safety validation, Gornyi Zhurnal (10) (2015) 44–50.
  * [39] GeRa software website, http://en.ibrae.ac.ru/code/gera/, accessed: 2021-04-14.
  * [40] Y. Vassilevski, K. Terekhov, K. Nikitin, I. Kapyrin, Parallel Finite Volume Computation on General Meshes, Springer Nature, 2020.
  * [41] INMOST, a toolkit for distributed mathematical modeling, http://www.inmost.org/, accessed: 2021-04-14.
  * [42] I. Konshin, I. Kaporin, K. Nikitin, Y. Vassilevski, Parallel linear systems solution for multiphase flow problems in the INMOST framework, in: Russian Supercomputing Days, 2015, pp. 96–103.
  * [43] N. Li, Y. Saad, E. Chow, Crout versions of ILU for general sparse matrices, SIAM Journal on Scientific Computing 25 (2) (2003) 716–728.
  * [44] I. S. Duff, J. Koster, The design and use of algorithms for permuting large entries to the diagonal of sparse matrices, SIAM Journal on Matrix Analysis and Applications 20 (4) (1999) 889–901.
  * [45] C. M. Oldenburg, K. Pruess, On numerical modeling of capillary barriers, Water Resources Research 29 (4) (1993) 1045–1056.
  * [46] S. W. Webb, Generalization of Ross’ tilted capillary barrier diversion formula for different two-phase characteristic curves, Water Resources Research 33 (8) (1997) 1855–1859.
  * [47] H.-J. G. Diersch, FEFLOW: finite element modeling of flow, mass and heat transport in porous and fractured media, Springer Science & Business Media, 2013\.
  * [48] I. Kapyrin, V. Suskin, A. Rastorguev, K. Nikitin, Verification of unsaturated flow and transport in vadose zone models using the computational code gera, Vopr. At. Nauki Tekh., Ser.: Mat. Model. Fiz. Protsess (1) (2017) 60–75.