AP_AJT | 2 (0.00%) | 1 (0.00%) | 1 (0.00%) | 4 (0.00%)
X_CMP | 2 (0.00%) | 1 (0.00%) | 0 (0.00%) | 4 (0.00%)
X_CNJ | 2 (0.00%) | 1 (0.00%) | 0 (0.00%) | 3 (0.00%)
X_MOD | 2 (0.00%) | 0 (0.00%) | 0 (0.00%) | 2 (0.00%)
AP_CMP | 1 (0.00%) | 0 (0.00%) | 0 (0.00%) | 2 (0.00%)
R | 1 (0.00%) | 0 (0.00%) | 0 (0.00%) | 1 (0.00%)
VNP_CNJ | 1 (0.00%) | 0 (0.00%) | 0 (0.00%) | 1 (0.00%)
AP_SBJ | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 1 (0.00%)
NP_SVJ | 0 (0.00%) | 0 (0.00%) | 0 (0.00%) | 0 (0.00%)
Total | 114,491 | 22,496 | 25,033 | 162,020

####  3.6.2 Evaluation Metrics

The evaluation metrics for KLUE-DP are 1) unlabeled attachment score (UAS) and
2) labeled attachment score (LAS), which are popular metrics for DP. Given the
goal of DP is to predict head indices (HEAD) and dependency relation classes
(DEPREL), UAS only counts HEAD prediction while LAS counts both HEAD and
DEPREL. Specifically, UAS calculates macro F1 score on HEAD prediction, while
LAS calculates macro F1 score on DEPREL whose HEAD prediction is correct. Both
scores give the same importance to all classes. For LAS, since DEPREL
distribution is highly skewed, we combine the predictions on the labels with a
cumulative frequency of 1% from the bottom into a single label (OTHERS) and
then calculate F1 score. The less-appeared labels are referred in Table 14.

####  3.6.3 Related Work

The Penn Treebank [84] is a constituency parsed dataset created from 1989 to
1996. It has a size of about 3 million words, including IBM computer manuals,
nursing notes, Wall Street Journal articles, phone conversations. A total of
48 POS tags and 18 syntax tags are used by combining meta tags such as
symbols. The Penn Treebank was the best-known parsing data set before
dependency parsing became prevalent. Later studies convert the Penn Treebank
to dependency parsing [86, 20].

A representative DP corpus is the Universal Dependencies(UD)
dataset.404040https://universaldependencies.org UD is de facto standard of DP
data, aiming for a unified treebank annotation in various languages. Google
Universal POS [106] composed of 12 tags, was developed and a corpus was built
that applied it to 25 different languages. Also de Marneffe and Manning [28]
studied guidelines for dependency parsing markers used in Stanford parsers
[65]. The Universal Dependency Treebank Project in 2013 attempted to combine
the two studies above to have a consistent annotation system for multiple
languages. UD started by modifying and supplementing this. UD first started in
2015 and Nivre et al. [97] and 10 corpora in a total of 10 languages were
released through the website. As of 2021, UD (UD 2.7v) offers 104 languages
and 183 corpora.

The corpus constructed with UD scheme is made according to a certain structure
called CoNLL-U format. Since UD aims to study general linguistic universality
by using the same tag and annotation system in different languages, a unified
format for integrating and managing each corpus is needed. The CoNLL-U format
is a modified version of this CoNLL format so that it can well represent
Universal dependency parsing. CoNLL-U format consists of 10 columns, each
column display a word index (ID), word form (FORM), lemma of word form
(LEMMA), universal part-of-speech tag (UPOS), language-specific part- It
represents of-speech tag (XPOS), list of morphological features (FEATS), head
of the current word (HEAD), dependency relation (DEPREL), enhanced dependency
graph (DEPS), any other annotation (MISC). In this format, each word stands on
a line along with different associated features (word form, lemma, POS tag,
etc.) and we adopt this format in our final dataset.

In Korean, DP corpus is divided into those that follow the UD scheme and those
that do not. Among the former are The Google Korean Universal Dependency
Treebank (GKT), The KAIST Korean Universal Dependency Treebank (KTB), and The
Penn Korean Universal Dependency Treebank (PKT). These three datasets are
converted from The Google Korean Treebank [86], The Kaist Treebank [128], and
The Penn Korean treebank [47] according to the UD scheme, respectively [20].
These were first automatically converted according to the head-finding rule
and then heuristically modified. These are composed of 6k, 27k, and 5k
sentences, respectively, and include the genres of blog, newswire, literature,
academic, and manuscript. Among them, PKT was revised by changing the analysis
unit and several rules to further reveal the characteristics of Korean [99].

Corpora that do not follow the UD scheme include the TTA DP Corpus built by
Electronics and Telecommunications Research Institute (ETRI) and the Modu
Corpus [98] built by the National Institute of the Korean Language (NIKL).
Both corpora follow the CoNLL format, and use their own tagset, which was
developed from the 21st century Sejong Plan corpus. The syntactic analysis
corpus of the 21st century Sejong Plan is constructed according to the
constituency grammar, following a scheme similar to the Penn Treebank. Unlike
the dependency grammar, which grasps only the dominant relationship between
two words, the constituency grammar identifies the relationship between words
hierarchically. However, since Korean has relatively free word order,
dependency parsing is more suitable than phrase-structure parsing. Studies on
converting the 21st century Sejong Plan corpus into DP format have been
conducted, and since then, corpora that use the 21st century Sejong Plan’s
tagset but follow dependency parsing have been constructed. The size of TTA DP
corpus is about 27k, and the Modu corpus about 2000k. Unlike UD, which
emphasizes general linguistic characteristics, better represents the
characteristics of Korean as an individual language, and serves as a national
standard for Korean DP tagging. Also, there are already corpus annotated
according to the TTA scheme, so we consider compatibility between them and our
benchmark. For this reason, we constructed the KLUE-DP using the TTA tagset.

####  3.6.4 Conclusion

We build a Korean DP benchmark KLUE-DP consisting of formal news and informal
user-generated web data. KLUE-DP is helpful for developing a DP model that can
be used in multiple domains. POS tagging is performed together to improve DP
performance, and the tagset and guideline for DP and POS tagging are applied
by revising the existing TTA dataset. This guideline is customized to reflect
the characteristics of Korean (agglutinative, free word order, etc.), and it
also tackles omission of predicates in web data or errors in spacing. We hope
that our benchmarks will help in the development of Korean DP models and other
natural language processing.

###  3.7 Machine Reading Comprehension (MRC)

Machine reading comprehension (MRC) is a task designed to evaluate models’
abilities to read a given text passage and then answer a question about the
passage, that is, its ability of comprehension.

Most of existing, widely-used MRC benchmarks are largely in English [21, 56,
60, 112, 113, 145, 150]. Those resources are widely used in evaluating pre-
trained language models since it is one of the most intuitive methods for
measuring text comprehension. SQuAD 1.1 [112] and SQuAD 2.0 [113] are popular
evaluation tasks along with GLUE [30, 82, 22, 71]. BoolQ [21], ReCoRD [150],
and MultiRC [60] are selected as a member of SuperGLUE for rigorous evaluation
of language models. Recently, open-domain QA task which can be viewed as an
MRC task without a given text passage [69, 56, 145, 38], is included in a
knowledge-intensive NLP task benchmark [105].

Motivated by those datasets, MRC has become an essential task in NLU
benchmarks for various languages such as Indonesian [137], Chinese [142], and
Russian [125]. In Korean, however, an appropriate MRC benchmark is not
available because existing Korean MRC datasets are either less challenging,
limited in access, or simply machine-translated from an English dataset [79,
1, 74]. We therefore include MRC in KLUE and create a new challenging Korean
MRC benchmark (KLUE-MRC) with the following contributions:

  * •

Providing multiple question types: In order to evaluate different aspects of
MRC capability of models, we provide three question types: paraphrase, multi-
sentence reasoning, and unanswerable. We collect questions by following strict
guidelines with specific sets of rules for each type.

  * •

Preventing reasoning shortcuts: We prevent MRC models from exploiting
reasoning shortcuts with simple word-matching by enforcing lexical and
syntactic variations when workers generate questions. Also, we aim to generate
questions which can be answered by considering the full query sentence.

  * •

Multiple passage domains accessible to everyone : We include news domain
passages as well as Wikipedia. To guarantee CC BY-SA license of KLUE-MRC, we
made signed contracts with corresponding news providers.

We formulate MRC as a task of predicting the answer span of the question from
the given text passage. The input is a concatenated sequence of the question
and the passage separated with a delimiter. The output is the start and end
positions of the predicted answer span within the passage.

We evaluate models with two metrics: 1) exact match (EM) and 2) character-
level ROUGE-W. Note that character-level ROUGE-W is different from the
character-level F1 score used in the previous Korean MRC datasets. If the
question is unanswerable within the given passage, the model should predict
the empty answer string. The motivation of our metrics are described in
Section 3.7.2.

####  3.7.1 Dataset Construction

##### Source Corpora

First, we collect passages from Korean WIKIPEDIA and news articles provided by
The Korea Economy Daily and ACROFAN. WIKIPEDIA articles are one of the most
commonly used resources for creating MRC datasets. We additionally include
news articles reporting contemporary social issues to enhance diversity of
passages. They are provided by The Korea Economy Daily and ACROFAN. As news
articles are generally copyrighted work, we sign a contract with the news
providers to use and redistribute the articles under CC BY-SA license only for
building a dataset for machine learning purposes. We believe multi-domain
corpus can help MRC models enhance their generalizability.

We preprocess the corpus to collect passages. For WIKIPEDIA articles, we
remove duplicates in other existing Korean MRC benchmarks (e.g., KorQuAD) for
precise evaluation of models. Then, we split each article by its sections to
obtain passages. For the news articles, we filter out political articles and
articles belonging to categories which have less than 100 articles. We finally
gather all preprocessed passages whose length is longer than 512 and shorter
than 2048 in characters.

##### Annotation Protocol

We annotate questions and answers by giving passages to crowdworkers. We
provide a detailed tutorial session to introduce our guidelines. 60 out of 80
workers are selected after a pilot test of creating 15 question-answer pairs
with a given passage. The selected workers generate questions and label
corresponding answers spans (for Types 1 and 2) or fake answers spans (for
Type 3). We use Tagtog annotation toolkit 414141https://www.tagtog.net/ for
the annotation. We assign three inspectors for each question type to validate
the generated questions and answers following common and type-specific
guidelines. If the generated question-answer pair fails to pass the
inspection, the worker refines it based on the feedback given by the
inspector.

##### 1\. Common Guidelines

We first build common annotation guidelines for crowdworkers. The workers are
required to follow the instructions during corresponding question generation
and answer span annotation for all three question types as below.

A question should:

  * •

Be natural as a web search query: Trying to generate challenging questions can
lead to the use of unnatural expressions. We guide workers to make natural
questions as if they are for web search queries. We care about the
generalizability of questions to extend the task to open-domain QA tasks in
future work.

  * •

Avoid omission: Pronoun-dropping is prevalent in Korean, hence it tends to
omit the subject or object. The omission can result in ambiguity for finding
answers. We explicitly guide workers to keep all grammatical components in
generated questions.

  * •

Not copy a phrase in the passage: Questions might have similar meaning to some
phrases in the passage but should not contain exactly the same phrase. This
mitigates high word-overlap between questions and the passage as reported in
previous work [122].

  * •

Not refer to external knowledge: Questions need to be fully understood without
any external knowledge. We do not allow workers using their background
knowledge or any world knowledge to generate a question. Questions should be
derived solely based on the given passage.

  * •

Be meaningful in every part for finding the answer: Answers should not be
found with only a small fraction of the question. We encourage workers to
generate questions that require understanding of the whole question text to
find an answer. We do not allow the use of expressions that appear only once
in the passage because models can easily infer answers without understanding
the whole question.

An answer should:

  * •

Be a unique entity within a passage: To clarify what to ask, only a single
answer should be inferred from the question. When answers can be represented
in various lexical forms, workers should mark all answer spans (e.g.
Television, TV).

  * •

Not be the main topic or title: We aim to prevent a known artifact which the
most frequently appeared words within a given passage are likely to be the
answer [66].

##### 2\. Type-Specific Guidelines

Here we elaborate question type-specific guidelines. These guidelines are
additionally presented to workers along with the common guideline above.

2.1. Question Paraphrasing (Type 1) Type 1 examples focus on paraphrasing the
passage sentences when generating questions to reduce word overlap between
them. The paraphrasing enables us to validate whether the model can correctly
understand the semantics of the paraphrased question and infer the answer
[122].

Table 15: Paraphrase question type example for Korean and its English
translation

Type Korean English (Translated) Passage 브르타뉴 공국은 939년 트랑라포레 전투에서 기원을 했으며,
브르타뉴와 노르망디 간에 경계인 쿠에농강에 세워졌다. Duchy of Brittany originates in the Battle of
Trans-la-Forêt of year 939, and was established on and around the Couesnon
River, the boundary of Britanny and Normandy. Good Question 브르타뉴와 노르망디를 구분짓는
것은? What distinguishes Britanny and Normandy? Bad Question 브르타뉴와 노르망디 간의 경계는?
What is the boundary of Britanny and Normandy? Answer 쿠에농 강 Cuesnon River

In our annotation guide for paraphrased questions, we further prevent workers
from generating them by simply shifting the order of subsequent phrases or
changing the functional particles. Specifically, workers create questions by
following principles:

  * •

Either syntactic or lexical variation should be applied to text snippets in
the passage.

    * –

For syntactic variation, reconstructing the structure of the original sentence
is preferred. Minimal changes such as swapping order between the nearby
phrases is not allowed.

    * –

For lexical variation, changing verbs or modifiers is required, and variation
of noun phrases is recommended.

  * •

More than half of the words in the question should not overlap with the
corresponding part (sentence) of the passage.

Then we comprehensively check whether the question has low word overlap with
the passage. As shown in Table 15, where Cuesnon River is the answer, the
question paraphrases the given evidence sentence (“the boundary of Britanny
and Normandy”) by providing a new sentence structure. The structure is changed
from a wh-question with copula to a question with an intransitive verb. Also,
a new term “distinguishes” is introduced which replaces the phrase “is the
boundary of”.

##### 2.2. Multiple-Sentence Reasoning (Type 2)

Type 2 examples focus on making questions requiring multiple-sentence
reasoning. Multiple-sentence reasoning requires models to derive answers from
the questions by reasoning over at least two sentences in the passage. We
focus on evaluating whether an MRC model can infer the answer span by
comprehensively aggregating the information spread across the passage. We
carefully design annotation guidelines for multiple-sentence reasoning
examples. Since Min et al. [90] report multiple-sentence reasoning questions
can easily fall back to single-sentence reasoning, we aim to avoid such cases
by guiding workers to follow the steps below:

  * •

(Step 1.) Find at least two statements in the given passage that share some
properties.

  * •

(Step 2.) Select one as the answer among the entities regarding the shared
properties.

  * •

(Step 3.) Generate a question with the selected entity.

Table 16: Multiple sentence reasoning question type example for Korean and its
English translation

Type Korean English (Translated) Passage 트라야누스 황제 시대(98년-117년)에 로마 제국은 북으로는
스코틀랜드에서 남으로는 아프리카 수단까지, 서로는 포르투갈의 대서양 연안에서 동으로는 카프카스 지방까지 최대 판도를 이룩했다. 오늘날
면적으로 환산하면 현재 미국 면적의 2/3에 달하고 인구도 현 미국의 절반에 약간 안되는 정도로 추산된다. 서기 5세기 경 서로마 제국은
멸망 후 게르만족의 여러 독립 국가로 갈라져 프랑크 왕국, 신성 로마 제국 등 로마의 후계자를 자처하는 여타 서유럽의 정치 세력들이
나타난다. Under Trajan(AD 98-AD 117), Roman Empire reached its territorial peak.
In terms of the area today, it is estimated to represent two-thirds of the
current U.S. area and its population is only slightly less than half that of
the current U.S. Plagued by internal instability and attacked by various
migrating peoples, the western part of the empire broke up into independent
barbarian kingdoms in the 5th century. Statement A 오늘날 면적으로 환산하면 현재 미국 면적의
2/3에 달하고 인구도 현 미국의 절반에 약간 안되는 정도로 추산된다. In terms of the area today, it is
estimated to represent two-thirds of the current U.S. area Statement B 트라야누스
황제 시대(98년-117년)에 로마 제국은 북으로는 스코틀랜드에서 남으로는 아프리카 수단까지, 서로는 포르투갈의 대서양 연안에서 동으로는
카프카스 지방까지 최대 판도를 이룩했다. Under Trajan(AD 98-AD 117), Roman Empire reached its
territorial peak. Shared Properties 로마제국, 트라야누스 황제시대(98년-117년) Roman Empire,
Trajan(AD 98-AD 117) Question 로마 제국의 면적이 현재 미국 면적의 2/3에 다다르던 시기는 언제인가? When
was the area of the Roman Empire two-thirds of that of the current U.S.?
Answer 트라야누스 황제 시대(98년-117년) / 트라야누스 황제 시대 / 98년-117년 Trajan(AD 98-AD 117) /
Trajan / AD 98-AD 117

The example in Table 16 follows the steps described above. Shared properties
between the two statements (A: “In terms of the area today, it is estimated to
represent two-thirds of the current U.S. area”, B: “Under Trajan(AD 98-AD
117), Roman Empire reached its territorial peak.”) are “Roman Empire” and
“(Under) Trajan(AD 98 -AD 117)”. Assume we pick “Trajan(AD 98-AD 117)” as the
answer and generate a question “When was the area of the Roman Empire two-
thirds of that of the current U.S.?”. Neither of the two statements is enough
to single out the answer alone. When given the question and statement A alone,
it is not possible to resolve an answer between “Trajan(AD 98-AD 117)” and
“the 5th century” which both represent time period in the passage. Since
statement B does not include specific information on the territory size, it is
also not sufficient to narrow down the answer by itself. Therefore, statement
A and B both needs to be aggregated to correctly answer the question.

##### 2.3. Unanswerable Questions (Type 3)

Type 3 examples are questions unable to be answered within the given passage.
We name these as ‘unanswerable’ questions. In the real world, a question is
often unanswerable if a passage is not available. If a model is built upon the
premise that an answer always exists within the passage, it would not
effectively handle such cases as SQuAD 2.0 [113] pointed out. We therefore add
unanswerable questions in our benchmark to incentivize models to identify
whether a question is answerable or not.

In the annotation guideline, we present the following principles to generate
desirable unanswerable questions. The unanswerable questions should:

  * •

Be relevant to the passage: An entity appearing in the given passage should be
included in the question. The entity makes the question relevant to the
passage. Otherwise the question would be too easy to be determined as
unanswerable.

  * •

Have fake answers within the passage: A fake answer is plausible but incorrect
regarding the corresponding question. The fake answer should exist in the
passage as a distractor.

  * •

Not have correct answers within the passage: Despite the existence of fake
answers, the correct answer for the generated question must not exist in the
passage.

In our setting, a model is likely to fail to identify the question’s
unanswerability based on the word overlap between the question and the
passage. Our question includes entities from the passage which increases the
overlap. In that case, an MRC model would choose plausible fake answers that
exist in the passage.

Table 17 shows an example of the Type 3 problem. The question includes
“bandwidth” which also exists in the given passage. The question is asking for
the bandwidth, so “1 to 5 Gigabit Ethernet” would be a plausible but incorrect
answer. There is no cue about the correct answer to “bandwidth-hungry server”
from the question in the passage. The model should predict the empty string
using the out of the context span range, such as using the [CLS] token from
BERT.

Table 17: Unanswerable question type example for Korean and its English
translation

Type Korean English (Translated) Passage 아직도 대역폭이 많이 필요하지 않은 서버는 1-5기가비트 이더넷을
사용한다. Servers that still do not require much bandwidth use 1 to 5 Gigabit
Ethernet. Question 대역폭이 많이 필요한 서버는 어느 대역을 사용하는가? Which bandwidth does the
bandwidth-hungry server use? Fake Answers 1-5기가비트 이더넷 1 to 5 Gigabit Ethernet

##### Annotation Process

We employ workers recruited by SelectStar,424242https://selectstar.ai/ a
Korean crowdsourcing platform, to make each type of question and answers for a
given passage. For type 1, 28 workers annotate examples and other 3 workers
inspect all of them. For types 2 and 3, 19 and 13 workers annotate examples
and other 3 and 2 workers validate them, respectively. If the generated
question is rejected by the inspectors, it is regenerated based on the
feedback.

To meet the ethical standards, examples which contain sexual, violent, hate,
bias or any other ethically inappropriate contents are eliminated. We manually
re-check all examples at the end of the annotation process. Through the
filtering process, we remove 173 examples in total.

##### Final Dataset

KLUE-MRC consists of 12,207 paraphrasing-based questions, 7,895 multi-sentence
reasoning questions, and 9,211 unanswerable questions. In total, 29,313
examples are made with 22,343 documents and 23,717 passages. As we prioritize
creating a challenging dataset for evaluating MRC models, we give more
examples to dev and test splits. We set the train/dev/test split ratios as
6:2:2 for robust evaluation, resulting in 17,554 training, 5,841 development
and 5,918 test examples. For each split, we randomly sample the examples from
each question type independently to balance the number of question types.
Tables 18 and 19 show the detailed statistics of KLUE-MRC.

Table 18: Number of examples per each dataset split and question types. | Paraphrase | Multi-sentence | Unanswerable | Total
---|---|---|---|---
| (41.65%) | (26.93%) | (31.42%) | (100.0%)
Train (60%) | 7,308 | 4,729 | 5,517 | 17,554
Dev (20%) | 2,437 | 1,571 | 1,833 | 5,841
Test (20%) | 2,462 | 1,595 | 1,861 | 5,918
Total (100%) | 12,207 | 7,895 | 9,211 | 29,313
Table 19: Statistics of KLUE-MRC. | |Train| | |Dev| | |Test| | Total
---|---|---|---|---
# Documents | 12,174 | 5,075 | 5,094 | 22,343
# Passages | 13,072 | 5,310 | 5,335 | 23,717
# Questions | 17,554 | 5,841 | 5,918 | 29,313
Avg Length of Passage | 1,004.62 | 1,014.64 | 1,010.13 | 1,008.10
Avg Length of Question | 29.00 | 29.05 | 29.01 | 29.01
Avg Length of Answer | 6.03 | 6.03 | 5.82 | 5.99

####  3.7.2 Evaluation Metrics

The evaluation metrics for KLUE-MRC are 1) exact match (EM) and 2) character-
level ROUGE-W (ROUGE) [80], which can be viewed as longest common consecutive
subsequence (LCCS)-based F1 score. EM is the most commonly used metric for MRC
tasks, which measures the equality of ground truth and predicted answer
string. If there are multiple gold labels, a model can earn score when at
least one prediction is matched. In contrast, ROUGE gives a partial score
although a model fails to predict exactly matched answer. Due to the
characteristics of Korean, an answer span can be located inside a single word,
hence subword-level span should be considered. ROUGE calculates F1 score of
the length ratio of LCCS to a prediction and the length ratio of LCCS to the
ground truth string. In the case when multiple ground truth answer spans have
the same meaning but different lexical variations (e.g. TV, Television), we
use the maximum ROUGE score among the combinations of answers and the
prediction. We do not adopt character-level F1 score (char F1), which is used
in all the previous Korean MRC datasets, since it measures character overlap
regardless of the order. When a model predicts “한국의 위인들 (great people in
Korea)” and an answer is “국한된 범위 (limited scope)”, a metric should give a low
score. ROUGE scores 15.38, whereas char F1 gives 54.55 due to the overlap of
“한”, “국”, and “위”.

####  3.7.3 Analysis

We investigate KLUE-MRC by comparing with other Korean MRC datasets. KorQuAD
1.0 [79] and 2.0 [63] are commonly used in Korean MRC research, released with
training and evaluation sets separately. However, KorQuAD 2.0 shows a quite
different composition in contents compared to ours and KorQUAD 1.0, in
specific, HTML tags and tables. Therefore, we conduct comparison only with
KorQuAD 1.0 dataset. KorQuAD 1.0 dev set was leveraged because its test set is
not available.

##### Question Difficulty of KLUE-MRC Evaluation Set

As we aim to make KLUE-MRC challenging, it is necessary to check the
difficulty of test sets. We compare the performance on the evaluation set of
ours and KorQuAD 1.0 with the model trained with both datasets in Table 20. We
fine-tune the model434343We fine-tune pretrained RoBERTa-base model with
following hyperparameters: epochs 5, batch size 16, learning rate 3e-5, lr
warmup ratio 0.0. with a collection of the train set from both datasets and
test the model on each evaluation set.

KorQuAD 1.0 train examples (60,407) are almost four times larger than KLUE-MRC
(17,554), and it may have resulted in the higher performance for KorQuAD 1.0
dev set. We additionally conduct fine-tuning with a collection of the same
amount of both train datasets for fair comparison. We adjust KorQuAD 1.0 train
set to fit the size of KLUE-MRC train set by random sampling. Table 20 shows
consistently lower scores of KLUE-MRC compared to KorQuAD 1.0. Thus, our
dataset is more challenging regardless of the size of train set.

Table 20: Difficulty comparison between KLUE-MRC test and KorQuAD 1.0 dev set. | KLUE + KorQuAD (Full:Full) | KLUE + KorQuAD (1:1)
---|---|---
Evaluation Dataset | EM | ROUGE | EM | ROUGE
KorQuAD 1.0 (Dev) | 86.59 | 94.19 | 85.00 | 93.07
KLUE-MRC (Test) | 70.42 | 75.42 | 69.75 | 75.20

##### Lexical Overlap

As high lexical overlap between question and passage can cause shortcut
reasoning, reducing lexical overlap is important to build a challenging
dataset. To investigate the effects of the proposed strict guidelines and
challenging question types, we calculate the lexical overlap of ours and
KorQuAD 1.0. The lexical overlap ratio is calculated by dividing the number of
common components between question and passage into the number of components
in the question. We exclude functional particles such as postposition(조사,
josa) and ending components (어미, eomi) when computing the overlap ratio via an
open-sourced Korean POS tagger.444444Twitter tagger of KoNLPy [101]. We
observe our lexical overlap ratio is almost 10%p lower than that of KorQuAD
dataset (70%). For each questions types, Types 1 and 3 show similar ratio in
range from 55% to 59%. Type 2 exhibits 68% overlap ratio.

##### Human Evaluation

We evaluate human performance on our KLUE-MRC to measure its difficulty
concerning human reading comprehension capabilities. We randomly sample 1,000
examples from our test set and hire three workers to solve them. We select the
score of the top scoring worker as human performance. Table 21 reports the
comparison between human performance and base model.

Table 21: Comparison of evaluation scores between model prediction and human answer | Paraphrase | Multi-sentence | Unanswerable | Total
---|---|---|---|---
| EM | ROUGE | EM | ROUGE | EM | ROUGE | EM | ROUGE
$\text{KLUE-RoBERTa}_{\text{BASE}}$ | 67.74 | 75.73 | 65.07 | 73.13 | 72.48 | 72.48 | 68.51 | 74.01
Human | 84.18 | 88.33 | 87.72 | 90.91 | 86.53 | 86.53 | 85.90 | 88.48

####  3.7.4 Related Work

In recent years, significant progress has been achieved in English MRC
research with challenging datasets of various question types, including but
not limited to: paraphrase, multi-sentence, and unanswerable.

Paraphrased questions have low word overlap between the question and reading
passage, which prevents MRC models from exploiting simple word-matching.
Trischler et al. [130] create a NewsQA dataset by generating questions from
news headlines and summarizing them via crowdsourcing. They reduce the word
overlap by annotating answers on the main articles which are not given during
question generation. Saha et al. [120] leverage pairs of plot summaries for
the same movies from Wikipedia and IMDb.454545https://www.imdb.com/ They
generate questions from the shorter plots and annotate answers on the longer
ones to obtain naturally paraphrased questions. Sen and Saffari [122] report
that datasets with low question-passage overlap will enhance the
generalizability of MRC models.

Multi-sentence questions require reasoning over multiple sentences. As a
result, they are more difficult compared to single-sentence questions. Joshi
et al. [56] introduce TriviaQA, a dataset of questions from trivia websites.
Since they gather evidence passages from various sources (e.g., Wikipedia and
the Web), multiple sentences are naturally required for answering the given
question. Khashabi et al. [60] explicitly generate multi-sentence questions on
various texts through crowdsourcing and release the MultiRC dataset. The
SuperGLUE [132] benchmark adopts the MultiRC dataset as one of its tasks.

Several MRC datasets have incorporated unanswerable questions [143, 130, 96,
113, 69]. Rajpurkar et al. [113] report performance drop in MRC models when
unanswerable questions are included in the dataset.

Compared to MRC research on English, Korean MRC research stands on a small
number of existing datasets. The primary benchmark for Korean MRC has been
KorQuAD [79, 63], which adopts the same data collection process as SQuAD 1.0
[112]. However, the model performance on KorQuAD has already exceeded human
performance in a short period, leaving little headroom for further research.
Moreover, unlike SQuAD, KorQuAD is under CC BY-ND license and does not allow
derivative works (e.g., adding unanswerable questions). AI Hub MRC dataset [1]
is based on newspapers and includes unanswerable questions. However, its
access is strictly limited to native Korean researchers, prohibiting
collaboration even with international researchers residing in Korea. K-QuAD
[74] leverage Google Translate464646https://translate.google.co.kr/ to
translate SQuAD 1.0 [112] into Korean. Since the K-QuAD dataset does not get
updated over time, its quality depends on the machine translator’s performance
at the time of release. Our KLUE-MRC is different from the existing Korean MRC
benchmarks in terms of accessibility-enhance license and more challenging
difficulty.

####  3.7.5 Conclusion

We create a new challenging Korean MRC benchmark named (KLUE-MRC). In order to
evaluate different aspects of MRC capabilities, KLUE-MRC includes multi-domain
passages and three types of questions: paraphrase, multi-sentence reasoning,
and unanswerable. KLUE-MRC shows improvements in question type diversity,
difficulty, and lexical overlap compared to existing Korean MRC datasets.

###  3.8 Dialogue State Tracking (DST)

Building a human-computer conversation system has been increasingly attracting
attention, and a task-oriented dialogue system is one type of the dialogue
systems [14]. A core module of task-oriented dialogue systems, namely,
Dialogue State Tracking (DST) is about predicting the dialogue states from a
given dialogue context. As illustrated in Table 22, dialogue states are sets
of slot and value pairs that are relevant categories (e.g. hotel type) and
their possible values (e.g. guest house, hotel, motel), respectively.

Several recent works have considered task-oriented dialogue (TOD) as an
important problem of natural language understanding. For instance, DecaNLP
[85] includes a DST, which is a key component of TOD, into one of their
benchmark tasks, while DialoGLUE [88] releases the first task-oriented
dialogue benchmark containing various sub-tasks including DST. In light of
such, we include DST as a part of the KLUE benchmark.

The task of dialogue state tracking is to predict slot and value pairs after
each user utterance, and the potential pairs are predefined by a task schema
and knowledge base (KB), tied to the choice of a scenario. For evaluation, we
use joint goal accuracy (JGA) and slot micro F1 score. The JGA checks if all
of the predicted slot-value pairs are exactly matched with the ground-truths
for every turn, while the slot micro F1 computes f1 score for each slot-value
pair independently.474747We adopt the evaluation script of
https://github.com/jasonwu0731/trade-dst

Table 22: An example of dialogue state tracking in our WoS. Note that all
dialogue states are cumulative in the actual dataset and that we only track
states in the user turns.

Utterances (English Translations) | Dialogue States
---|---
User: 안녕하세요. (Hello.) | -
|  Sys: 네. 안녕하세요. 무엇을 도와드릴까요? (Hello. How can I help?)
---
User: 서울 중앙에 위치한 호텔을 찾고 있습니다. 외국인 친구도 함께
갈 예정이라서 원활하게 인터넷을 사용할 수 있는 곳이었으면 좋겠어요.
(I’m looking for a hotel at the city center. I’m going with a foreign friend,
so easy access to the Internet should be available.)
| Hotel-area: center
---
Hotel-type: hotel
Hotel-internet: yes
|  Sys: 네 확인해보겠습니다. 혹시 추가로 필요하신 사항이 있으실까요?
---
(Sure, let me check. Do you need anything else?)
User: 음… 예약 인원은 총 8명이고요. 아, 가격대는 크게 상관 없습니다.
(Hmm.. I want to reserve for 8 people. Ah, the price range doesn’t matter.)
|  Hotel-area: center
---
Hotel-type: hotel
Hotel-internet: yes
Hotel-book people: 8
Hotel-price range: dontcare
|  Sys: 네, 확인 감사합니다. 숙박을 원하시는 요일과 기간 같이 확인 부탁드립니다.
---
(Great, thanks for confirming. Please let us know when and how long
you want to stay.)
User: 아, 중요한 걸 깜빡했네요. 일요일에 2일간 예약하고 싶습니다.
(Right, I forgot an important thing. I would like to book for two days
from Sunday.)
|  Hotel-area: center
---
Hotel-type: hotel
Hotel-internet: yes
Hotel-book people: 8
Hotel-price range: dontcare
Hotel-book day: Sunday
Hotel-book stay: 2

####  3.8.1 Dataset Construction

Our dataset construction protocol is a modified version of the Wizard-of-Oz
framework (WOZ) [59], which is a widely used paradigm for building dialogue
datasets. The WOZ setting is a particular type of human-to-human dialogue
collection, which employs two people that either takes a user and a system
role. However, it is arguably time-consuming, complex, and expensive [11]. To
overcome the limitations, we adopt ‘Self-dialog’ scheme which requests a
single worker to play both user and system roles [11]. In addition, we
introduce a new design choice to obtain a more precise dialogue dataset.

##### Overview

We construct WoS by following five steps: 1) define a task schema, 2) create
knowledge base (KB), 3) design an annotation system, 4) collect and annotate a
dataset, and 5) finalize the dataset. Different from the other datasets in our
benchmarks, WoS is not following an ordinary protocol: collecting raw corpus
followed by annotating labels. Rather, we collect dialogues (raw corpus) and
their corresponding dialogue states (labels) at the same time.

Table 23: Task schema for all five domains in Wizard of Seoul (WoS) which shows names of the domain and their slots. Star⋆, asterisk∗, cross†, doubly-crosses‡ denote required, boolean type, booking-related, and requestable after booking slot, respectively. Domains | Informable Slots | Requestable Slots
---|---|---
Hotel |  name, type⋆, area⋆, price range⋆, book day†, book stay†, book people†, walkability∗, parking∗, internet∗, breakfast∗, smoking∗, fitness∗, swimming pool∗, spa∗ |  rating, nearby station, minutes walk from station, address, phone number, business hour, reference number‡
Restaurant |  name, type⋆, area⋆, price range⋆, book day†, book time†, book people†, alcohol∗, walkability∗, parking∗, internet∗, smoking∗, outdoor table∗ |  rating, nearby station, minutes walk from station, address, phone number, business hour, last order time, representative menu, reference number‡
Attraction |  name, type⋆, area⋆, walkability∗, parking∗, heritage∗, educational∗, scenic∗, cultural∗ |  rating, nearby station, minutes walk from station, address, phone number, business hour, entrance fee
Taxi |  leave at⋆, departure⋆, arrive by, destination⋆, type |  phone number, cost, duration
Metro |  leave at, departure⋆, destination⋆ |  departure line, destination line, arrive by, cost, duration, transfer, optimal path

##### 1\. Defining Task Schema

We first define a task schema which expresses the scenario of task-oriented
dialogue. Our task schema consists of slots across five domains (hotel,
restaurant, attraction, taxi, and metro) as shown in Table 23. Typically,
slots are categorized into informable slots and requestable slots. The
informable slots cover properties which can constrain a user goal484848A user
goal is what the worker playing user should follow as shown in Table 24. such
as “price range”, “area”, and “booking day”. The requestable slots provide
additional information that a user may ask, but not necessarily need to be
specified as a user goal constraint. A typical example of a requestable slot
is “phone number”, which a user may ask for, but would not work to narrow down
the probable candidates of the goal [50, 51].

Based on this schema, we include additional attributes to the informable and
requestable slots to provide an easy-to-operate annotation system and easy-to-
follow guidelines. A slot could have one or more attributes among whether it
is 1) boolean type, 2) required or not (Required), 3) related to booking
(Booking-related), and 4) only available after booking is confirmed
(Requestable after booking. e.g., reference number). The boolean type slots
can have either yes or no as their values, such as “Parking (availability)”
and “(has) swimming pool”. Such boolean type values do not appear in the
dialogue context explicitly. In other words, they have abstractive properties.
A model which understands abstractive properties is desirable, so we have much
more boolean type slots than MultiWOZ [10]; WoS has 20 boolean slots across
the domains while MultiWOZ includes only 2. Meanwhile, the required slots have
to be specified with values in order to fill out a user intent. This helps us
to simulate an actual service scenario in which an agent is not allowed to
take the next steps without specifying required values [114].

Table 24: An example of goal instruction. Unlike MultiWOZ, we present all instructions from the beginning to prevent ordering bias as in CoCo [77]. The booking-related slots, “restaurant-book time (22:41)” and “restaurant-book day (Wednesday)” appear before a confirmation of KB entity. Korean |  English (Translated)
---|---
당신은 오늘 22:41에 서울 중앙에서 식사할 계획을 가지고 있습니다. 아참 오늘은 수요일 입니다. 그런 곳을 찾았다면 먼저 대표 메뉴를 확인하세요. 그리고 나선 1명으로 예약 거세요. 예약 이후엔 영업 시간을 문의하시구요. 그리고 나선 식당 근처에서 잘 곳을 찾아야 합니다. 그 곳은 반드시 흡연이 불가해야 합니다. 찾았다면 같은 요일에 예약하세요. 같은 인원으로 4일간 머물러야 합니다. 예약에 성공했다면 예약 번호를 묻고, 흡연 가능 유무를 더블 체크하세요. 그런 다음 마지막으로 택시를 하나 부르세요. 식당에서 숙소로 향해야 합니다. 찾았다면 소요 시간을 문의하세요. |  You have a plan to eat in the center of Seoul at 22:41 today. Oh, today is Wednesday. If you find such a place, first check the representative menu. Then, make a booking for 1 person. After booking, inquire about the business hour. Then, you have to find a hotel to sleep in near the restaurant. The restaurant must be non-smoking. If you find it, book on the same day. You must stay for 4 days as the same number of people. If the booking is done, ask for the reference number and double-check the smoking allowed. Then, finally, call a taxi. You have to go to the hotel from the restaurant. If you call the taxi, inquire about the duration.

##### 2\. Creating Knowledge Base

We construct a knowledge base (KB) based on the task schema of each domain to
obtain a set of predefined realization candidates of a user’s goal. For the
hotel and restaurant domains, we manually create virtual instances, whereas
for the attraction and metro domains, we leverage real names (e.g. Gangnam
Station or Namsan Tower) collected from the web. On the other hand, for the
taxi domain, we do not define instances in advance, but dynamically generate
the instances during the dialogue collection as in MultiWOZ [10]. With ethical
considerations in mind, any personally identifiable information (PII, e.g.,
phone number, address) is replaced with randomly generated instances using
faker.494949https://faker.readthedocs.io Table 25 shows the KB statistics for
each domain.

Table 25: Statistics of Knowledge Base in WoS. Domain | # Instances | # Slots
---|---|---
Hotel | 101 | 19
Restaurant | 56 | 20
Attraction | 100 | 17
Taxi | - | 8
Metro | 3,306 | 10

##### 3\. Designing the Annotation System

In this section, we describe the annotation platform we used for collecting
data from both the User-side and System-side.

##### 3.1. User Side

We provide a goal instruction for a user side role. The instruction includes
descriptions of a user’s specific goal with corresponding slot values in
natural language. It also contains the user’s context including persona for
variety of dialogues. A user is asked to generate utterances following the
instruction. An example is shown Table 24.

To devise a multi-domain dialogue scenario where slots are shared across
multiple domains, we include domain transition in the instruction [10, 154].
For example, in case of a user aiming to book a hotel, the user might seek
information about transportation (taxi, metro, etc.) to get there. In this
dialogue, initial domain is changed to another (hotel to taxi/metro). It is
more challenging compared with single domain in terms of dialogue state
tracking. Because user could express their goal implicitly where values should
be inferred by co-referencing other values of preceding domains.

The goal instructions are realized by the templates containing placeholders
for goal constraining slots and their values. We design diverse templates for
each domain, in order to cover various scenarios of the dialogues. Like
MultiWOZ, the goal templates include a series of subgoals with corresponding
slots. We carefully design the sentences to promote lexical entailment or co-
referencing during conversation, which can be naturally observed in the user
context or during the domain transition. When filling a template to complete
instructions, we randomly assign the instances from KB built upon the domain-
specific task schema to the given placeholders. The values of the instruction
should be specifically mentioned by a user during a conversation. Each
trackable slot either has valid values, None or Dontcare.505050Dontcare means
a user has no preference and None means a user is yet to specify a valid value
for given slot [50, 51].

To properly evaluate a model’s generalization ability, we further add
counterfactual goals and introduce unseen KB instances during the process
[77]. To add new goal instruction based on the current slot distribution, we
keep monitoring the slot value frequency and co-occurrence over slots during
the construction. Specifically, we add new goal instructions which cover
infrequent slot values or rarely co-occurring combination among slots. For
example, when “(hotel-parking, no)” is infrequent pair in the as-is
distribution, we promote it to appear in dialogues by designing goal
instruction including it as constraint. Moreover, we differentiate KB
instances between particular subset of dataset (train and dev/test set) to
simulate realistic scenario regarding unseen slot values in the test time.

Figure 7: Graphical web interface for system side worker.

##### 3.2. System Side

The role of a system side worker (wizard) is to 1) annotate dialogue states of
user utterances while 2) generate responses by accessing to the KB if
necessary, for every turn. First, the wizard is asked to fill in appropriate
slot values inferred from the current dialogue context. If the word uttered by
the user is not clear to directly map to a specific slot value, the wizard
should clarify the meaning of the word first and then fill the slot when the
word has the same meaning as the value. The annotation of a dialogue state is
an explicit action of understanding a user request to fully focus on providing
the required information. Then, the wizard generates response either to
request or to convey information. If the values of required slots are absent,
the system side worker is allowed to ask for the missing values to the user.
Otherwise, the system provides the user with the adequate information. We
enable the system to query the external knowledge base, if needed. When there
are more than three search results, the system worker could request more
details or recommend one among them.

To support the wizard to perform such complicate work effectively and
efficiently, we provide a graphical web interface with a newly introduced
feature: dropdown components (Figure 7). Dropdown interface enables the system
side worker can choose a value from a list of pre-populated candidates. We
present most probable value candidates based on the goal instruction and
domain-specific knowledge base, since a dropdown might become worthless when
too many options are presented to workers. This procedure naturally prevents
several type of annotation errors, such as multi-annotations, mis-annotations,
typos, and value canonicalization reported in MutiWOZ 2.1 [37].

##### 4\. Dataset Construction

We adapt ‘Self-dialog’ scheme inspired by taskmaster-1 [11] to efficiently
collect diverse dialogue dataset while reducing the cost and time. Self-dialog
is effective to collect various dialogue data. By having both roles, a worker
freely controls a flow of dialogue such as the order of slot occurrence in
user utterances and recommendation of system responses. This also leads
workers to speak their own styles naturally such that different personas are
included. However, we found annotation errors in pilot phase such as early-
markup (system pre-fills the values before receiving the values from the user)
and delayed-markup (the system fills the values behindhand the proper turn)
errors. We further improve the scheme by utilizing explicit turn-switching
between user and system roles with providing error correction interface.

To elaborate, we train and select trustworthy workers to participate in the
main collection process. Prior to the main phase, we conduct several pilot
studies with crowdworkers to avoid aforementioned early-markups and delayed-
markup errors, and to generate more realistic dialogues including some
miscommunications. We also implement an explicit turn-switching between the
two roles, in order to immerse themselves in a user/system role, which
mitigates overly reduced miscommunications as well. Throughout the pilot, we
finally employ 15 selected workers who can effectively handle such issues.

##### Final Dataset

Table 26 shows statistics of our dataset. WoS contains overall 10,000
dialogues with 146,692 turns across 5 domains. The evaluation (dev/test) set
specifically includes dialogues with counterfactual goals and unseen KB
instances against train set. The dev/test set contains 294 and 361
counterfactual goal-based dialogues, respectively. All the splits contain
sufficient number of dialogues with domain transition.

Table 26: Statistics of Wizard-of-Seoul (WoS). | |Train| | |Dev| | |Test| | Total
---|---|---|---|---
# Dialogues | 8,000 | 1,000 | 1,000 | 10,000
# Single Domain Dialogues | 1,806 | 263 | 226 | 2,295
# Multi Domain Dialogues | 6,194 | 737 | 774 | 7,705
# Counterfactual Dialogues | 0 | 294 | 361 | 655
# Total Turns | 117,584 | 14,448 | 14,660 | 146,692
# Total Tokens | 899,450 | 114,169 | 114,914 | 1,128,533
Avg Turns per Dialogue | 14.70 | 14.45 | 14.66 | 14.67
Avg Tokens per Turn | 7.65 | 7.90 | 7.84 | 7.69

####  3.8.2 Evaluation Metrics

For evaluation metrics for WoS is 1) joint goal accuracy (JGA) and 2) slot
micro F1 score. JGA measures the proportion of exactly matched dialogue state
which consists of a set of slot-value pairs with the ground truth dialogue
state among the total number of dialogue turns. Slot micro F1 score is an
average of micro F1 scores in each turn. For each turn, micro F1 score is
defined as the harmonic mean of precision and recall in terms of predicted
slot-value pairs and ground-truth pairs. Note that the slot micro F1 score
ignores when value of the ground truth is “None”.

####  3.8.3 Analysis

When splitting the train and dev/test set based on the counterfactual goals
and unseen KB instances, like Li et al. [77], we observe performance drop as
shown in Table 27. This demonstrates that the counterfactual goal makes WoS
more challenging.

Table 27: Comparison regarding data split strategy. Random is splitting train, dev and test set randomly. The CF-goal indicates they are included in dev and test set with unseen KB instances. Domain Split | Joint Goal Accuracy
---|---
Random | 57.53
CF-goal | 47.38

####  3.8.4 Related Work

Wizard-of-Oz (WOZ) [59] is a popular scheme in dialogue collection. In fact,
conventional WOZ setting allows to collect various type of dialogues by
employing role-playing of two humans. Each human should choose a role between
the two: user and system. As taking a role, dialogues are collected by turn-
taking generation of utterances with background information provided in
advance. In the case of building a task-oriented dialogue, goal is given to a
user while knowledge base is allowed to be accessed to system. The system can
use its knowledge base when responding to user’s request.

Many dialogue datasets closely follow WOZ settings [135, 35, 36], however, it
costs a lot of time and money because two crowdworkers must be matched at the
same time and successfully play each role, which prevents collecting dialogues
at scale. We refer this limitation to ‘worker coexistence constraints’. To
overcome the limitation, MultiWOZ [10] slightly change this conventional WOZ
to asynchronously collect dialogues turn-by-turn from crowdworkers, which
allows different workers to play the same user or system in a single dialogue.
This approach costs less but error-prone because every worker must adapt to
dialogue already progressed so that their response might be incoherent to the
previous context [37]. Recently, CrossWOZ and RiSAWOZ thus pair only selected
trustworthy workers to collect dialogues in synchronous manner as suggested in
the WOZ settings to keep the annotation quality despite the high construction
cost [154, 109].

Byrne et al. [11] also argue that conventional WOZ settings are time
consuming, complex and expensive, requiring considerable technical
implementation as well as administrative procedures to train and manage both
agents and crowdsourced workers, accordingly suggesting Self-dialog as an
alternative. Self-dialog is a collection scheme in which workers write the
entire dialogue playing both user and system roles. To demonstrate their idea,
Byrne et al. [11] build a large-scale dialogue datase, Taskmaster-1, which is
built upon the Self-dialog which stands on the WOZ schemes: 1) two people
playing user and system roles (conventional WOZ setting) and 2) one person
playing both roles (Self-dialog). As a result, Self-dialog can remedy the cost
of ‘worker coexistence constraint’ effectively with avoiding incoherent
dialogue generation caused by asynchronous dialogue collection. However, there
is a tendency that little miscommunication occurs in the dialogue which
compared to the real world conversations because the same person produces
utterances in both roles, which might lead to a gap from reality.

Some researchers further tries to employ only machines to create such
dialogues to maximize cost-efficiency. It builds the dialogues on top of a
simulator which is able to create utterances by turn automatically from
elaborately designed rules and given task schema. The simulator first
generates psuedo-dialogue, then adopts crowdsourcing to paraphrase them to the
natural utterances [124, 114]. It requires much less human effort, however,
heavily relies on the simulator.

Meanwhile, previous works exists addressing robust evaluation of DST models.
According to CoCo [77], state-of-the-art DST models are not robust to
realistic scenarios since they scarcely appear in the train data. As the name
CoCo (controllable counterfactuals) suggests, it generates infrequent but
realistic dialogues based on the predefined slot-value pairs. They show that
even a state-of-the-art DST model’s performance drops significantly when they
are evaluated on such dialogues including counterfactual goals. It means the
current TOD benchmarks should be improved in terms of robustness to unseen but
realistic scenarios.

As for Korean, there is a task-oriented dialog dataset is provided by National
Information Society Agency (NIA). It covers about 10 domains related to the
civil complaints and consists of more than 500k dialogues. The utterances are
divided into four types: 1) a main question that a user asks, 2) a sub
question that a system could ask for clarification, 3) a user answer, and 4) a
system answer. Additionally, user intents are annotated and entities are
extracted from each utterance. We find that this dataset does not follow any
of the aforementioned settings; there is no dialogue state represented as
slot-value pairs, only regarding single turn judgement. It also lacks
information about task schema and redistribution is restricted which does not
satisfy our accessibility principle, which motivates us to newly create a DST
benchmark.

####  3.8.5 Conclusion

We introduce Wizard-of-Seoul (WoS), the first large-scale Korean multi-domain
task-oriented dialogue dataset that simulates conversations between Seoul
tourists and travel agents. We adapt ‘Self-dialog’ for efficiently scaling up
of dialogue collection scheme. In addition, consideration on annotation
interfaces (drop-down menu and turn-switching) mitigates erroneous cases and
diverse goal instructions including counterfactual ones promote each
conversation to be more natural and challenging. We hope that WoS sparks
various future dialogue research in Korean and also offers valuable insights
to pushing forward end-to-end dialogue modeling.

##  4 Pretrained Language Models

In order to facilitate further research using KLUE, we provide strong
baselines for all the benchmark tasks within it. As a part of this effort, we
pretrain and release large-scale language models for Korean, which we hope
would reduce the burden of retraining large-scale language models from
individual researchers. More specifically, We pretrain language models (PLM),
including BERT [30] and RoBERTa [82], from scratch.

###  4.1 Language Models

We pretrain multiple Korean language models while varying training
configuration. This enables us to explore effective settings for pretraining
Korean models and further establish simple yet effective baseline models for
KLUE. We train KLUE-BERT and KLUE-RoBERTa. We vary the choice of a pretraining
corpus, preprocessing procedure, tokenization strategy, and other training
configurations.

Table 28: Statistics of the pretraining corpus.

MODU CC-100-Kor NAMUWIKI NEWSCRAWL PETITION Total # Sentences 167M 103M 14M
183M 5.2M 473M # Words 1,892,814,395 1,593,887,022 265,203,602 2,716,968,038
50,631,183 6,519,504,240 size (GB) 18.27 15.46 2.52 25.87 0.53 62.65

##### Pretraining Corpora

We gather the following five publicly available Korean corpora from diverse
sources to cover a broad set of topics and many different styles. We combine
these corpora to build the final pretraining corpus of size approximately
62GB. See Table 28 for overall statistics:

  * •

MODU : Modu515151 A transliteration of a Korean word ‘모두’ which means
‘Everyone’.  Corpus [98] is a collection of Korean corpora distributed by
National Institute of Korean Languages.525252https://corpus.korean.go.kr/ It
includes both formal articles (news and books) and colloquial text
(dialogues).

  * •

CC-100-Kor : CC-100535353 http://data.statmt.org/cc-100/ is the large-scale
multilingual web crawled corpora by using CC-Net [136]. This is used for
training XLM-R [26]. We use the Korean portion from this corpora.

  * •

NAMUWIKI : NAMUWIKI is a Korean web-based encyclopedia, similar to Wikipedia,
but known to be less formal. Specifically, we download the dump created on
March 2nd, 2020.545454 http://dump.thewiki.kr

  * •

NEWSCRAWL : NEWSCRAWL consists of 12,800,000 news articles published from 2011
to 2020, collected from a news aggregation platform.

  * •

PETITION : Petition is a collection of public petitions posted to the Blue
House asking for administrative actions on social issues. We use the articles
in the Blue House National Petition555555
https://www1.president.go.kr/petitions published from August 2017 to March
2019.565656 https://ko-nlp.github.io/Korpora/en-
docs/corpuslist/korean_petitions.html

##### Preprocessing

We filter noisy text and non-Korean text using the same methods from Section
2.3. Each document in the corpus is split into sentences using C++
implementation (v1.3.1.) of rule-based Korean Sentence Splitter (KSS).575757
https://github.com/likejazz/korean-sentence-splitter For CC-100-Kor and
NEWSCRAWL, we keep sentences of length greater than equal to 200 characters,
as a heuristics to keep well-formed sentences. We then remove sentences
included in our benchmark task datasets, using BM25 as a sentence similarity
metric [118].

##### Ethical Considerations

Because we collect and use as much publicly available data as possible for
pretraining, these corpora often contain undesirable social biases.
Furthermore, we noticed earlier quite a bit of PII in these corpora, although
they were all publicly available. Both of these are problematic. Social biases
in the corpus may result in a language model that learns such biases. PII in
the corpus may be memorized by a language model and can subsequently be
retrieved by adversarial attacks [12].

We do not filter out socially biased contents nor hate speech for three
reasons. First, manual inspection is infeasible for this large-scale
pretraining corpora. Second, it is a challenging problem on its own to
automatically detect socially biased contents or hate space, as both of these
highly depend on the context in which they appear [62]. Lastly, being blind to
such harmful contents prevents the future use of a language model for
detecting and correcting these harmful contents, such as using it as an anti-
expert [81]. We expect future research on the pretrained language models we
release to focus on how to detect and correct biases encoded in these models
and on how to debias them, as has been recently demonstrated by Cheng et al.
[15].

In contrast, we pseudonymize PII in our corpora as much as possible. We detect
16 personal data types using regular expressions based on the guideline from
the Korea Internet and Security Agency (KISA).585858
https://www.kisa.or.kr/public/laws/laws2_View.jsp?cPage=1&mode=view&p_No=282&b_No=282&d_No=3
It is relatively easy to pseudonymize PII while keeping linguistic patterns,
since the selected PII has standardized pattern. We then replace the original
information, using either the faker library595959
https://github.com/joke2k/faker or random generation based on the pattern. As
a result, we pseudonymize 1.2% of the pretraining corpora. Details are
illustrated in Table 29.

Table 29: Our pseudonymization methods and examples. The examples are from
faker library documentation or the public.

Private Information Pseudonymization Pseudonymised Example Telephone Number
Faker 055-604-8764 Social Security Number Faker 600408-2764759 Foreign
Registration Number Faker 110527-1815659 Email Address Faker
<EMAIL_ADDRESS>IP Address Faker 166.186.169.69 MAC Address Faker
c5:d7:14:84:f8:cf Mention(@) Faker @gildong Address Random Number Generation
경상북도 성남시 서초대64가 Bank Account Number Random Number Generation 110-245-124678
Passport Number Random Generation M123A4567 Driver’s License Random Number
Generation 11-17-174133-01 Business Registration Number Random Number
Generation 123-45-67890 Health Insurance Information Random Number Generation
1-2345678901 Credit or Debit Card Number Random Number Generation
1234-5678-9012-3456 Vehicle Registration Place Random Generation 55구 1601
Homepage URL Random Generation www.example.com

##### Tokenization

We design and use a new tokenization method, morpheme-based subword
tokenization. When building a vocabulary, we pre-tokenize a raw text into
morphemes using a morphological analyzer, and then we apply byte pair encoding
(BPE) [123] to get the final vocabulary. For morpheme segmentation, we use
Mecab-ko,606060 https://bitbucket.org/eunjeon/mecab-ko MeCab [68] adapted for
Korean, and for BPE segmentation, we use the wordpiece tokenizer from
Huggingface Tokenizers library.616161
https://github.com/huggingface/tokenizers We specify the vocabulary size to
32k. After building the vocabulary, we only use the BPE model during
inference, which allows us to tokenize a word sequence by reflecting morphemes
without a morphological analyzer. This improves both usability and speed.
Examples are presented in Table 30.

The motivation behind this method is that Korean is an agglutinative language,
which is to say, a word is a constitution of morphemes - stems and affixes.
The morphemes tend to remain unchange on different unions, and the boundary is
generally clear. Although BPE has been widely used across many languages due
to its effectiveness, it struggles to identify morphemes correctly as
demonstrated in Table 30.

Table 30: An input text “조경현은 인공지능 분야의 저명한 연구자이다. (Kyunghyun Cho is a
prominent AI researcher.)” is segmented with various tokenization strategies.
We denote slash (/) as a token separator. The mBERT tokenizer [30] splits the
input text into nearly in characters resulting in a longer sequence than
monolingual tokenizers. BPE tokenizer generates tokens spanning multiple
morphemes (##현은, ##명한). Morpheme-based subword tokenizer, on the other hand,
better splits text into morphemes (##은, ##의).

Tokenization Tokenized Sequence Raw Text 조경현은 인공지능 분야의 저명한 연구자이다. BPE
(Multilingual) 조 / ##경 / ##현 / ##은 / 인 / ##공 / ##지 / ##능 / 분 / ##야 / ##의 / 저 /
##명한 / 연구 / ##자 / ##이다 / . BPE 조경 / ##현은 / 인공지능 / 분야의 / 저 / ##명한 / 연구 / ##자이 /
##다 / . Morpheme 조경현 / 은 / 인공지능 / 분야 / 의 / 저명 / 한 / 연구자 / 이 / 다 / . Morpheme-
based Subword 조경 / ##현 / ##은 / 인공지능 / 분야 / ##의 / 저명 / ##한 / 연구자 / ##이다 / .

##### Training Configurations

We choose BERT [30] and RoBERTa [82] architectures for our language models.
Table 31 describes the implementation details. All models take sequences of at
most 512 tokens long each and are pretrained with a static or dynamic masking
strategy following the original training procedure. When masking tokens, we
use whole word masking (WWM) which masks all of the the tokens that form a
single word. BERT also performs next sentence prediction (NSP). Other
hyperparameters not specified in Table 31 nor in the pretraining procedure
details are same as the original configurations from [30, 82]. Due to the
resource constraints, we could increase batch size only up to 2,048, unlike
Liu et al. [82] who use the batch size of 8k. We decrease the learning rate
accordingly. We fix the learning rate to $10^{-4}$ for both BERT and RoBERTa.

Table 31: Implementation details of KLUE-BERT and KLUE-RoBERTa. WWM refers to
the whole word masking strategy.

Model # Parameter Masking Training Steps Batch Size Learning Rate Device
$\text{KLUE-BERT}_{\text{BASE}}$ 110M Static, WWM 1M 256 $10^{-4}$ TPU v3-8
$\text{KLUE-RoBERTa}_{\text{SMALL}}$ 68M Dynamic, WWM 1M 2048 $10^{-4}$
8$\times$ V100 GPUs $\text{KLUE-RoBERTa}_{\text{BASE}}$ 110M Dynamic, WWM 1M
2048 $10^{-4}$ 8$\times$ V100 GPUs $\text{KLUE-RoBERTa}_{\text{LARGE}}$ 337M
Dynamic, WWM 500k 2048 $10^{-4}$ 8$\times$ V100 GPUs

###  4.2 Existing Language Models

In addition to our own language models, we evaluate the following two existing
multilingual language models and two Korean monolingual language models on our
benchmark:

  * •

mBERT [30] : A multilingual BERT introduced and released by Devlin et al.
[30]. It is trained with the MLM and NSP objectives on a multilingual corpus
covering 104 languages including Korean.

  * •

XLM-R [26] : A RoBERTa [82] trained on a large multilingual corpus by using
the MLM objective.

  * •

KR-BERT [75] : An open-sourced character-level Korean language model based on
BERT. We use the KR-BERT character WordPiece which uses a vocabulary of 16,424
unique tokens.

  * •

KoELECTRA [102] : An open-source Korean language model trained with the MLM
and replaced token detection objectives, as was done by Clark et al. [22]. For
training corpora, Park [102] uses own crawled news data and MODU corpus [98].

##  5 Fine-tuning Language Models

###  5.1 Task-Specific Architectures

8 main tasks of KLUE benchmark can be categorized into 4 types based on the
fine-tuning strategy. Topic classification (TC) and relation extraction (RE)
are single sentence classification tasks. Sentence textual similarity (STS)
and natural language inference (NLI) are sentence pair classification /
regression tasks. Dialogue state tracking (DST) is a multiple-sentence slot-
value prediction task. Finally, named entity recognition (NER), dependency
parsing (DP) and machine reading comprehension (MRC) are sequence tagging
tasks.

We use $x_{i}$ to refer to the $i$-th token of input and
$h_{i}\in\mathbb{R}^{H}$ to its corresponding final hidden state from a
pretrained language models (PLM), where $H$ is the hidden dimensionality.
Following the conventional fine-tuning setup [30], we use [CLS] as the first
input token $x_{0}$ and [SEP] as a delimiter token to separate inputs (e.g.,
two sentences in STS and NLI, a passage and a question in the case of MRC, and
dialogue turns for DST.)

####  5.1.1 Single Sentence Classification

In the single sentence classification task, such as TC and RE, a classifier
classifies a single sentence into a set of predefined labels. Following the
convention, the last hidden state of [CLS] token $h_{0}$ is linearly mapped to
the number of labels ($K$) with $W\in\mathbb{R}^{K\times H}$ and the entire
model is trained to minimize the cross entropy loss.

YNAT is a single sentence classification task where $K$ is 7 for predefined
topic labels, and does not require any special treatment of each input. KLUE-
RE on the other hand requires a special procedure to indicate entities within
the input sentence. We use <subj>, </subj>, <obj>, and </obj> to mark the
beginnings and the ends of subject and object entities, respectively,
following Baldini Soares et al. [5]. We expand the embedding matrix to add
these four extra tokens.

####  5.1.2 Sentence Pair Classification / Regression

In the sentence pair classification / regression task, a model is asked to
determine the relationship between two sentences. A pair of input sentences
are concatenated with a special separator token, often [SEP], in-between.

In KLUE-STS, each sentence pair is annotated with a real-valued similarity
$[0,5]$. The model is thus trained to map from the final hidden state of [CLS]
to a real number, by minimizing the mean squared error (MSE). In the case of
KLUE-NLI, each sentence pair, consisting of a premise and hypothesis, is
coupled with one of three classes. The model thus maps the hidden state of
[CLS] token to a three-dimensional real-valued vectors and is trained to
minimize the cross-entropy loss.

####  5.1.3 Multiple-Sentence Slot-Value Prediction

WoS is a slot-value prediction task for a given dialogue context, where the
prediction should be considered across multiple turns instead of a single
utterance. We employ an encoder-decoder model following the architecture of
TRADE [140], which consists of an utterance encoder, a state generator, and a
slot gate classifier (Figure 8). In our implementation, we change the
utterance encoder from GRU [17] to PLM to get better representations. Thus,
the state generator takes the final hidden state of [CLS] token $h_{0}$ as the
first decoder hidden state. We also modify the slot gate classifier to predict
additional two slot gate labels (yes, no), since WoS contains relatively more
Boolean type slots than MultiWOZ [10]. We jointly minimize the cross-entropy
loss of the state generator and slot gate classifier.

Figure 8: A baseline architecture for WoS based on TRADE [140].

####  5.1.4 Sequence Tagging

Figure 9: An input and label example of KLUE-NER. We realign original
character-level label sequence of Figure 3 for tokens from our Morpheme-based
subword tokenization.

KLUE-NER is a token-level tagging task, where each character is assigned a
label. This requires a care in using tokenization, as the labels from the
characters within each subword token must be aggregated, and the predicted
label of each subword token must be properly distributed across the characters
within it. See Figure 9 for an example. We linearly map each of the final
hidden states from the encoder $h\in\mathbb{R}^{|x|\times H}$ into a
12-dimensional real-valued vectors, corresponding to the 12 named-entity
categories. We then minimize the cross-entropy loss summed over all the
tokens.

KLUE-MRC is a span prediction task in which a model tags the beginning and end
tokens of the answer span within a passage, given a question. The input to the
model is the concatenation of a tokenized passage and an associated question
(separated by [SEP]). The final hidden state of each token in the passage is
linearly projected to a 2-dimensional real-valued vector. The dimensions in
this vector correspond to the logits of two binary classifiers, the start and
end token classifiers. The special token [CLS] is considered as both the
correct start and end tokens, when given question is unanswerable. We minimize
the cross-entropy loss to train the model.

We frame KLUE-DP as a sequence tagging problem. Each token within an input
sentence is tagged twice, once with its head token and the other with the type
of the arc connecting the head and the current token. Our baseline
architecture follows the model proposed by Fernández-González and Gómez-
Rodríguez [39], except word representation and attention mechanisms. Similarly
to KLUE-NER, we must be careful in handling subword tokens, as the annotation
is done at the word level. In our approach, we use a pretrained language model
(to be fine-tuned) to extract subword representations and concatenate the
first and last subword token representations of each word, to form word vector
representations. Each of these word representations is optionally concatenated
with the part-of-speech embedding. For the attention layers, we use biaffine
attention [33] to predict HEAD, and bilinear attention [64] to predict arc
type (DEPREL) for each word. Just like KLUE-NER and KLUE-MRC, we minimize the
cross-entropy loss to fine-tune the entire model. See Figure 10 for a
graphical illustration of the model architecture.

Figure 10: An overview of KLUE-DP baseline model architecture, which we take
advantage of Fernández-González and Gómez-Rodríguez [39].

###  5.2 Fine-Tuning Configurations

For all the experiments, we use Huggingface Transformers [139] and PyTorch-
Lightning.626262 https://github.com/PyTorchLightning/pytorch-lightning We use
AdamW optimizer [83] with the learning rate selected from $\\{10^{-5},2\times
10^{-5},3\times 10^{-5},5\times 10^{-5}\\}$, the warm-up ratio from
$\\{0.,0.1,0.2,0.6\\}$ and the weight decay coefficient from $\\{0.0,0.01\\}$.
We choose the batch size from $\\{8,16,32\\}$ and the number of epochs from
$\\{3,4,5,10\\}$. We use the maximum sequence length of 512 for KLUE-MRC and
WoS, and 128 for all the other tasks. We report the score obtained from the
best hyperparameter configuration based on the dev set performance.

###  5.3 Evaluation Results

Table 32: Evaluation results of our pretrained LMs and other baselines on KLUE
benchmark. The F1 refers to a macro-F1 score. The F1E and F1C of KLUE-NER
indicates entity-level and character-level macro-F1 score, respectively. The
F1mic of KLUE-RE is micro-averaged F1 score ignoring the no_relation. The F1S
of WoS is an average of slot-value pair level micro-F1 scores. The RP of KLUE-
STS denotes Pearson correlation. Bold shows the best performance across the
models, and underline indicates the best performance among BASE models.

YNAT KLUE-STS KLUE-NLI KLUE-NER KLUE-RE KLUE-DP KLUE-MRC WoS Model F1 RP F1
ACC F1E F1C F1mic AUC UAS LAS EM ROUGE JGA F1S $\text{mBERT}_{\text{BASE}}$
81.55 84.66 76.00 73.20 75.14 87.51 57.88 53.82 90.30 86.66 44.66 55.92 35.46
88.63 $\text{XLM-R}_{\text{BASE}}$ 83.52 89.16 82.01 77.33 80.73 91.37 57.46
54.98 89.20 87.69 27.48 53.93 39.82 89.61 $\text{XLM-R}_{\text{LARGE}}$ 86.06
92.97 85.86 85.93 81.81 92.49 58.39 61.15 92.71 88.70 35.99 66.77 41.20 89.80
$\text{KR-BERT}_{\text{BASE}}$ 84.58 88.61 81.07 77.17 75.37 90.42 62.74 60.94
89.92 87.48 48.28 58.54 45.33 90.70 $\text{KoELECTRA}_{\text{BASE}}$ 84.59
92.46 84.84 85.63 86.82 92.79 62.85 58.94 92.90 87.77 59.82 66.05 41.58 89.60
$\text{KLUE-BERT}_{\text{BASE}}$ 85.49 90.85 82.84 81.63 84.77 91.28 66.44
66.17 92.14 87.77 62.32 68.51 48.99 91.86 $\text{KLUE-RoBERTa}_{\text{SMALL}}$
84.30 90.50 83.92 79.12 84.99 91.10 60.85 58.76 89.32 87.74 57.79 63.78 45.65
91.22 $\text{KLUE-RoBERTa}_{\text{BASE}}$ 85.12 92.41 84.60 84.97 85.13 91.52
66.66 67.74 90.31 88.30 68.52 74.02 47.48 91.55 $\text{KLUE-
RoBERTa}_{\text{LARGE}}$ 86.42 93.37 85.89 89.43 85.79 91.77 69.59 72.39 93.32
88.72 76.78 81.43 50.49 92.11

In this section, we present the evaluation results including our KLUE-PLMs and
existing PLMs on the KLUE benchmark, in Table 32.636363 See Appendix A for the
corresponding table however computed on the development set.  Different from
other NLU benchmarks, we do not average the scores over tasks, since simple
averaging of scores of different scales and interpretations could be highly
misleading. Rather, we describe and discuss the result of each task
separately. Within the Korean BASE models, $\text{KLUE-BERT}_{\text{BASE}}$
performs best for YNAT and WoS, $\text{KLUE-RoBERTa}_{\text{BASE}}$ for KLUE-
RE and KLUE-MRC, and $\text{KoELECTRA}_{\text{BASE}}$ for KLUE-STS and KLUE-
NLI.

We make two major observations. First, we see that $\text{KLUE-
RoBERTa}_{\text{LARGE}}$, which is the largest model among the baseline PLM’s
we tested, outperforms all the other models across all the tasks except for
KLUE-NER. This observation agrees well with the recent trend which has
demonstrated the correlation between the model size and task performance [58,
9]. This indicates that KLUE will be useful for the future investigation into
how much gain we can expect by simply increasing the model size further. The
second observation is that the monolingual models, which were specifically
designed for and trained with a more carefully curated corpus in the target
language (Korean), generally outperform the multilingual counterparts,
especially when we compare models of similar sizes. We make this observation
again across all the tasks, except for KLUE-NER, where the
$\text{XLM-R}_{\text{LARGE}}$ performs similarly to the best performer,
$\text{KoELECTRA}_{\text{BASE}}$, in terms of character-level F1 score. This
observation re-iterates the importance of investing effort in understanding a
target language and customizing data, models and learning algorithms for the
target language.

###  5.4 Analysis of Models

There were two major decisions we made in preparing the pretraining corpus and
preprocessing data. They were 1) whether to pseudonymize PIIs and 2) the
tokenzation strategy. In this section, we analyze the impact of our choices,
using $\text{KLUE-RoBERTa}_{\text{BASE}}$ by training it on the MODU corpus
only.

##### Corpus Pseudonymization

It can be expected that noise introduced in the process of pseudonymization
may have detrimental effect on the downstream task performance. Our finding,
presented in Table 33, however shows that there is some drop in a subset of
the tasks, but such drop is quite minimal. This suggests that the minimal
level of pseudonymization, just like what we have done, is already a good way
to balance the task performance and the risk of leaking private information.

Table 33: Evaluation results according to whether the corpus pseudonymization
is conducted in the preprocessing step.

YNAT KLUE-STS KLUE-NLI KLUE-NER KLUE-RE KLUE-DP KLUE-MRC WoS Pretraining
Corpus F1 RP F1 ACC F1E F1C F1mic AUC UAS LAS EM ROUGE JGA F1S Original 83.40
92.06 84.70 81.60 84.84 91.03 65.25 64.79 92.17 88.34 62.13 67.46 47.14 91.60
Pseudonymized 83.39 91.11 82.85 78.50 84.99 91.22 62.79 62.96 92.02 88.02
62.88 67.58 46.21 91.23

Table 34: Comparison our tokenization strategy with other baselines.

YNAT KLUE-STS KLUE-NLI KLUE-NER KLUE-RE KLUE-DP KLUE-MRC WoS Tokenization F1
RP F1 ACC F1E F1C F1mic AUC UAS LAS EM ROUGE JGA F1S BPE 83.40 91.91 85.19
82.07 68.75 89.47 64.39 65.04 89.89 89.47 51.12 65.79 21.38 77.68 Morpheme-
based Subword 83.40 92.06 84.70 81.60 84.84 91.03 65.25 64.79 92.17 88.34
62.13 67.46 47.14 91.60

##### Tokenization Strategy

We contrast our tokenization scheme, morpheme-based subword tokenization,
against the standard byte pair encoding (BPE). First, we investigate the
difference in how words are segmented into subword tokens. Following Rust et
al. [119], we consider subword fertility, proportion of continued words, and
UNK ratio. On the subword fertility, which measures the average number of
subwords produced per word, the proposed tokenization scheme ends up slightly
higher than BPE does. However, when we look at the proportion of continued
words, which measures the number of words that were split into at least two
subwords, we observe the opposite trend. This implies that our algorithm
maintains the original words as much as it can, and only when it is necessary,
it splits each word into potentially more subword pieces. The efficacy of the
proposed scheme over BPE is evident from the UNK ratio, as it produces fewer
UNK tokens compared to BPE when the vocabulary size was controlled to be 32k
for both methods. See Table 35.

Table 35: Overview of tokenization metrics. We build each vocabs using MODU
corpus and compare them on WIKIPEDIA corpus.

Tokenization # Vocabs Fertility $\downarrow$ % Continued Word $\uparrow$ UNK
Ratio $\downarrow$ BPE 32k 2.073 0.578 0.011 Morpheme-based Subword 32k 2.468
0.765 0.009

We find these qualitative differences between two schemes lead to significant
differences in the task performance in the cases of KLUE-NER, KLUE-MRC and
WoS. These tasks often involve tagging, detection and even generation at the
morpheme level, and we suspect that morphologically consistent tokenization
facilitates better prediction overall. On the other hand, the difference in
the tokenization strategy does not manifest itself in the performance of
classification or word-level tagging, likely as a corresponding NLU system can
more readily overcome inconsistencies in subword segmentation when merging
subword token representations into that of a larger unit. Overall, we
recommend future researchers use the proposed tokenization strategy as a
default option.

##  6 Ethical Considerations

In building KLUE and accompanying baseline models, we have incorporated
various mechanisms to avoid any harmful and negative consequences from
releasing both data and models. These mechanisms are described in detail
wherever they were introduced and used, but in this section, we summarize
these mechanisms, considerations and our principles behind them.

###  6.1 Copyright and Accessibility

Most NLP datasets are built upon the existing text sources. This raises a
question on the terms of using such datasets, especially when the underlying
source datasets are not well-specified nor carefully investigated. In order to
avoid any such doubt on the terms of using KLUE and to accelerate NLP research
in Korean, we fully adhere to the copyright act of Korea, which went effective
on Dec. 8, 2020.646464
https://www.law.go.kr/%EB%B2%95%EB%A0%B9/%EC%A0%80%EC%9E%91%EA%B6%8C%EB%B2%95
and include only text for which we know we can release under a license that
permits both redistribution and re-mix without any restriction on the use.

##### Source Corpora

Our goal is to secure and maximize the continued availability and usefulness
of the benchmark. In other words, we must guarantee the possibility of derive
new work and redistribute it freely, which comes together with CC BY-SA. To
release KLUE under CC BY-SA, we have built a source corpus set by including
only text that is either 1) not protected by copyright or 2) under CC0, CC BY,
CC BY-SA or KOGL Type 1 license. In the case of news articles, which are
copyrighted, we have signed contracts with the providers, Korean Economics
Daily (KED) news media and Acrofan, that allow us to make KLUE-MRC and release
them under CC BY-SA.

##### Task-Specific (Annotated) Datasets

We subsample and annotate the source corpus for each KLUE benchmark task. We
release each under CC BY-SA. This allows users of KLUE benchamrk to copy,
redistribute, remix, transform and build upon it for both commercial and non-
commercial purposes, as long as derivatives are distributed under the same
license (CC BY-SA). We expect this to greatly facilitate future NLP research
and development.

##### Pretraining Corpora and Language Models

As was discussed earlier 4.1, we cannot guarantee that our pretraining corpus,
built using MODU, CC-100-Kor and NEWSCRAWL, does not contain any copyrighted
work, although these are all created from publicly available text.
Unfortunately without these corpora, it is not possible to find a sufficiently
large resource to train large-scale language models for Korean. We thus use
them for pretraining but do not publicly release the pretraining corpora in
order to avoid any issues in the future, which is in contrast to KLUE.
Instead, we openly release pretrained language models to facilitate future
research. As the parameters of a language model does not [express] human
thoughts and emotions, they do not meet the requirement of being copyrighted.

###  6.2 Toxic Content

Although large-scale, accessible benchmark datasets advance machine learning
and its applications to adjacent fields, such as natural language processing,
toxic and unwanted contents within these datasets may be amplified via large-
scale models we train on them. We have been aware of this issue from the
beginning of the project, and here we describe how we have addressed these
toxic contents in KLUE.

##### Task-Specific Datasets

For each task-specific dataset, we apply three stages to minimize the
introduction of toxic contents. First, we automatically detect hate speech and
gender-biased sentences using toxicity classifiers and remove those even
before sending these sentences for annotation (see Section 2.3). Second, we
explicitly and clearly instruct annotators to mark any instance that exhibits
social biases and/or is toxic (see Section 3), after providing them with clear
definitions of bias and hate speech. Finally, we manually examine these marked
sentences and exclude them from the final dataset. This three-stage process
may not catch all possible such instances, and we plan to use an online
forum656565https://github.com/KLUE-benchmark/KLUE/issues to receive feedback
and complaints from users of KLUE.

##### Pretrained Language Models

We use our pretraining corpora (see Section 4.1) as they are, for three
reasons. First, manual inspection is simply not tractable due to the sheer
scale. Second, it is challenging to build an automated tool to detect hate
speech and biased sentences. This issue is made even more severe for Korean,
because there is only one known hate speech dataset of limited size [92].
Lastly, we envision the future in which these pretrained language models are
used to build better tools for automatically detecting various toxic contents
as well as undesirable social biases. In order for such pretrained models to
be aware of these issues, they must have been trained with such toxic contents
as well.

###  6.3 Personally Identifiable Information

It has recently been discovered by that a large-scale, pretrained language
model memorizes a large amount of personally identifiable information (PII)
and that an algorithm can be designed to retrieved those private information.
We thus design two different approaches for pseudonymizing task-specific
datasets and pretraining corpora, respectively.

##### Task-Specific Datasets

In the case of task-specific datasets, we rely on manual inspection during
annotation to detect PII. We discard any sentences that was reported to
contain PII after manual inspection. In the case of DST, which relies on
simulated dialogues, we pseudonymize the database entries rather than actual
text, using the faker library.666666 https://github.com/joke2k/faker

##### Pretraining Corpora

There is a trade-off between removing PII and the performance of a pretrained
language model, as we will demonstrate later in this paper. We thus
pseudonymize 16 PII types that are detectable purely by regular expressions.
See Section 4.1

##  7 Related Work

##### General-Purpose NLU Benchmarks

General Language Understanding Evaluation (GLUE) [133] benchmark, a collection
of evaluation dataset for English, was the first general-purpose evaluation
benchmarks for NLU. It is general-purpose in that it is not limited to a
single task. It consists of 11 downstream tasks, including tasks that measures
the capability of capturing semantic textual similarity (QQP, MRPC, STS) [3,
13], measures the capability of inference (MNLI, QNLI, RTE, WNLI) [8, 138] and
that evaluates the capability of classifying a single sentence into a
predefined set of categories (CoLA, SST) [134, 126]. GLUE exclusively focused
on English, and its variants in different languages have been built and
released over the past couple of years, including CLUE in Chinese [142], a
French version [72], an Indonesian version [137], a version for Indic
languages [57], Russian SuperGLUE [125], and Persian GLUE [61]. In all these
cases, substantial efforts were carried out to follow the philosophy of the
original GLUE, covering a broad spectrum of domains and tasks, while
incorporating language-specific characteristics. On the other hand, there have
been efforts to build a multilingual version of such benchmark, largely
relying on automated methods, such as XGLUE [78] and XTREME [54]. Korean as a
language has been included in subsets of these latter benchmarks, but there
has not been a serious attempt at building a general-purpose language
understanding evaluation suite for Korean, until this paper.

##### Absence of a Standard NLU Benchmark in Korean

Until this paper, a number of task-specific benchmarks in Korean have been
proposed and released. For example, NSMC is used for sentiment classification,
PAWS-X [144] for paraphrase detection, KorNLI and KorSTS [45] for NLI and STS,
KorQuAD 1 and 2 [79, 63] for MRC, and BEEP! [92] for hate speech detection. At
this point, one may wonder whether it would have been easier and more
convenient to simply aggregate these datasets to build KLUE. After all, this
has been a popular strategy for constructing monolingual [133] as well as
multilingual [78, 54] benchmarks. Unfortunately this approach comes with two
major issues that we directly address in this paper.

First, the existing datasets are constructed individually without considering
other datasets and their properties. In other words, the aggregate of these
individual datasets is unlikely to cover a broad spectrum of domains and
writing styles, unlike KLUE for which we carefully curate the source corpora
as well as subsets for downstream tasks to have broad coverage of domains and
styles. This goes beyond domains and styles, but also the coverage of
linguistic phenomena under evaluation. Most of the existing benchmarks, listed
above, focus on semantics rather than syntax, and it is difficult to find any
widely-available benchmark that captures pragmatics. We address this issue by
carefully selecting a set of downstream tasks.

Second, these existing datasets are not always publicly available,676767 Some
of these are publicly available in Korea but not internationally.  and some
are distributed with a highly restrictive license that prohibits
redistribution nor the transformation of the original. These are often the
ones published and released by government-affiliated institutes. In some
cases, it is necessary to obtain a special permission to access datasets,
which is often not easily accessible by non-Korean researchers. We address all
these issues with KLUE by releaseing the entire benchmark data under CC BY-SA,
both by careful curation of source corpora and by direct agreements with
publishers.

##### Pretrained Language Models (PLMs)

The recent trend of large-scale pretrained language models was sparked by the
success of earlier models, such as ELMo [104], GPT-2 [110] and BERT [30], on
the GLUE and other similar NLU benchmarks. This earlier success has led to a
series of advances in large-scale language models, including XLNet [146],
ALBERT [71], RoBERTa [82], ELECTRA [22], and Deberta [49], again largely
driven by the availability of standardized benchmarks. This advance in
language models, not only in terms of the model size but also in learning
algorithms, in turn also sparked the interest in building and improving
existing language understanding benchmarks. Some of the recently released,
challenging benchmarks include SuperGLUE [132] and KILT [105]. The
availability of such a standard language understanding benchmark, such as KLUE
from this paper, is expected to start such a virtuous cycle for Korean
language understanding.

##### Pretrained Language Models for Korean

Inspired by the development in other languages and multilingual models, PLM’s
for the Korean language have been trained and released by multiple research
groups and individuals. SKT released KoBERT,686868
https://github.com/SKTBrain/KoBERT followed by KorBERT696969
https://aiopen.etri.re.kr/service_dataset.php from ETRI, HanBERT707070
https://github.com/tbai2019/HanBert-54k-N from TwoBlock AI, KR-BERT [75] from
Seoul National University. There are a few pretrained models released by
individual researchers, such as KoELECTRA [102] and KcBERT [73].

Unfortunately, it is unclear how we should compare this stream of pretrained
language models in Korean, due to the lack of a standarded benchmark in
Korean. Subsets of these models have been compared based on subsets of a few
downtream NLP tasks in Korean above, but because these are not standardized,
it is not easy to draw solid conclusions from these limited experiments. We
expect the proposed KLUE benchmark will serve as a standard way to track the
progress of research in language models for Korean.

##  8 Discussion

##### Open Access

We distribute KLUE under CC BY-SA. The license allows everyone to freely copy
and redistribute our benchmarks in any medium or format. In addition, one can
improve our benchmark to build more challenging datasets after performance
saturation. To function as a NLU benchmark, open access is a must. If the
original author does not allow derivative development of the benchmark, other
researchers cannot improve it, for example by removing toxic content, or
building a more challenging dataset to accelerate research for technical
improvements. If commercial use is not allowed, researchers working at for-
profit organizations would not be able to benefit from nor to (easily)
contribute to the benchmark. Redistribution is another crucial factor because
it significantly limits research if, for example, sharing the datasets with
another researcher is prohibited. Another existing practice that limits
research is transferring the responsibility of copyright infringement of
related conflicts to researchers. To set a good precedent for open access of
data, we allow using our datasets for 1) any purpose, 2) derivative work, and
3) redistribution, as long as the existing copyrights in our benchmark
datasets are respected. We also open our pretrained Korean language models and
the implementation of pretraining and fine-tuning pipelines. This enhances
reproducibility of our work, and allows anyone to fix and improve our data and
models. We hope to contribute to the Korean NLP research community as well the
wider NLP community.

##### Facilitating Korean NLP Research

We developed KLUE with the aim of facilitating Korean NLP research, in
response to the recent active development efforts of large Korean language
models. The entire NLP community has seen BERT [30] and its variants
outperforming the previous NLU models for GLUE [133] and SuperGLUE [132], as
well as the more recent GPT3 [9] with outstanding performance without fine-
tuning (and with in-context learning) in natural language understanding and
generation. Motivated by these models, many Korean researchers at various
institutions rushed to pretrain large-scale Transformer-based Korean language
models. Consequently, a number of nearly identical pretrained language models
have been released to open-source communities. However, we could not
systematically understand the behaviors and characteristics of these models
because of the lack of well-designed general-purpose benchmarks like GLUE for
Korean. KLUE will allow us to conduct controlled experiments to understand how
and why various Korean LMs perform on certain tasks and thus obtain detailed
insights into those models. Furthermore, since KLUE includes many
representative NLU tasks that are also conducted in other languages, KLUE will
function as a fundamental resource to NLP researchers who aim to conduct
multilingual research with Korean and other languages.

##### Measuring Overall Performance of NLU models

We do not average all scores gained from each task in KLUE. The performance of
all tasks are measured by different evaluation metrics. This is because we
carefully choose the metric for each task with considering its own
characteristics. Their granularity differs by tasks, for example, KLUE-MRC and
KLUE-NER employ character-level metrics because an entity can exist within a
word in Korean whereas KLUE-STS and KLUE-NLI use sentence-level metrics.
Furthermore, we use various metrics across tasks, such as F1 score, accuracy,
area under the curve, UAS, LAS, ROUGE-W, joint goal accuracy, and Pearson’s
correlation. In this situation, simply computing the average of all tasks as
in GLUE [133] results in misleading overall performance measure. The average
will lose its interpretability as well as giving higher weights to a certain
task in unintended ways. Accordingly, an alternative way to estimating a
model’s NLU capability is necessary. Recently, analyzing correctness of a
model’s prediction by using Item Response Theory (IRT) framework to estimate
such capability is proposed [70], however, we find that it is not clear how it
should be applied precisely in our benchmark. As of now, we thus decide to
evaluate a model for each task separately without any summarization of overall
performance measure. This is our limitation, and we leave this problem for the
future.

##### Rapid Saturation of KLUE

We expect fast saturation of KLUE based on our observations of for instance
GLUE [94]. However, we do not artificially make our benchmark challenging by
e.g. filtering out easy examples for models. Because the main purpose of KLUE
is to properly evaluate models in terms of various aspects of NLU, we avoid
focusing on enlarging headrooms for improvement over our baseline pretrained
language models. We expect our license policy would positively affect the
advancement of our benchmark after saturation by collectively developing more
challenging tasks with other researchers, such as building the first open-
domain question answering for Korean.

##### Analysis of Korean Language Models

We observe various patterns when comparing performances of the baseline models
on each tasks, however, most of them are understudied to precisely explain the
phenomena. With more thorough investigations, we hope to enhance understanding
of the complex interaction of a model, corpus, linguistic properties of Korean
and training mechanisms in future work.

##  9 Conclusion

We present KLUE, a suite of Korean NLU benchmarks that includes diverse tasks.
We open KLUE to everyone, and we also provide Korean language models trained
to outperform multilingual models and other existing open-sourced Korean
language models. We set high standards from the outset, as we built the
benchmark and trained the models from scratch. We designed the benchmark
datasets and trained the annotators rigorously to consider potential ethical
issues including private information and hate speech. We documented in detail
all of the benchmark construction and testing processes. We also discussed
broader impacts and limitations of KLUE and our models. Despite the
limitations, KLUE and the accompanying language models will facilitate future
Korean NLP research by setting a valuable precedent describing how datasets
and language models should be created and spread to a wider community.

## Acknowledgments and Disclosure of Funding

Upstage sponsored annotation cost and built the leaderboard. NAVER CLOVA
provided data annotation cost and GPU cloud computing infrastructure (NSML).
We also thank to Google’s TensorFlow Research Cloud (TFRC) and Kakao
Enterprise’s BrainCloud. The three computing resources were used to pretrain
and fine-tune language models. Scatter Lab, SelectStar, Riiid!, DeepNatural
and KAIST sponsored data annotation cost. In addition, we appreciate The Korea
Economy Daily and Acrofan for supporting their news articles for MRC datasets.

The authors thank Cheoneum Park for discussions about task selection and DP
task, Jinhyuk Lee and Minjoon Seo for discussions on MRC task, Sujeong Kim and
DongYeon Kim for considerable efforts to manage annotators in MRC dataset, and
Sangah Park for careful consideration of data construction in DP, NER, and RE.
We appreciate Junyeop Lee, Geonhee Lee, Jiho Lee, Daehyun Nam, and Yongjin Cho
for the leaderboard and evaluation system implementation.

This study is reviewed and approved by the KAIST Institutional Review Board
(#KH2020-173).

## References

  * Agency [2018] National Information Society Agency.  MRC AI Dataset.  https://aihub.or.kr/aidata/86, 2018.
  * Agirre et al. [2015] Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe.  SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 252–263, Denver, Colorado, June 2015. Association for Computational Linguistics.  doi: 10.18653/v1/S15-2045.  URL https://www.aclweb.org/anthology/S15-2045.
  * Agirre et al. [2016] Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 497–511, San Diego, California, June 2016\. Association for Computational Linguistics.  doi: 10.18653/v1/S16-1081.  URL https://www.aclweb.org/anthology/S16-1081.
  * Allwood [2000] Jens Allwood.  An activity based approach to pragmatics.  In Harry Bunt and William Black, editors, _Abduction, belief and context in dialogue: Studies in computational pragmatics_ , chapter 2, pages 47–80. John Benjamins, Amsterdam, Netherlands, 2000.
  * Baldini Soares et al. [2019] Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski.  Matching the blanks: Distributional similarity for relation learning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2895–2905, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1279.  URL https://www.aclweb.org/anthology/P19-1279.
  * Bender and Friedman [2018] Emily M. Bender and Batya Friedman.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604, 2018.  doi: 10.1162/tacl_a_00041.  URL https://www.aclweb.org/anthology/Q18-1041.
  * Bowman and Dahl [2021] Samuel R Bowman and George E Dahl.  What will it take to fix benchmarking in natural language understanding?  _arXiv preprint arXiv:2104.02145_ , 2021.
  * Bowman et al. [2015] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal, September 2015\. Association for Computational Linguistics.  doi: 10.18653/v1/D15-1075.  URL https://www.aclweb.org/anthology/D15-1075.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Budzianowski et al. [2018] Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1547.  URL https://www.aclweb.org/anthology/D18-1547.
  * Byrne et al. [2019] Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Ben Goodrich, Daniel Duckworth, Semih Yavuz, Amit Dubey, Kyu-Young Kim, and Andy Cedilnik.  Taskmaster-1: Toward a realistic and diverse dialog dataset.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4516–4525, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1459.  URL https://www.aclweb.org/anthology/D19-1459.
  * Carlini et al. [2020] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  _arXiv preprint arXiv:2012.07805_ , 2020.
  * Cer et al. [2017] Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada, August 2017. Association for Computational Linguistics.  doi: 10.18653/v1/S17-2001.  URL https://www.aclweb.org/anthology/S17-2001.
  * Chen et al. [2017] Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang.  A survey on dialogue systems: Recent advances and new frontiers.  _SIGKDD Explor. Newsl._ , 19(2):25–35, November 2017.  ISSN 1931-0145.  doi: 10.1145/3166054.3166058.  URL https://doi.org/10.1145/3166054.3166058.
  * Cheng et al. [2021] Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin.  FairFil: Contrastive neural debiasing method for pretrained text encoders.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=N6JECD-PI5w.
  * Chinchor [1998] Nancy A. Chinchor.  Overview of MUC-7.  In _Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 - May 1, 1998_ , 1998.  URL https://www.aclweb.org/anthology/M98-1001.
  * Cho et al. [2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar, October 2014. Association for Computational Linguistics.  doi: 10.3115/v1/D14-1179.  URL https://www.aclweb.org/anthology/D14-1179.
  * Cho et al. [2020a] Won Ik Cho, Jong In Kim, Young Ki Moon, and Nam Soo Kim.  Discourse component to sentence (DC2S): An efficient human-aided construction of paraphrase and sentence similarity dataset.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 6819–6826, Marseille, France, May 2020a. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://www.aclweb.org/anthology/2020.lrec-1.842.
  * Cho et al. [2020b] Won Ik Cho, Sangwhan Moon, and Youngsook Song.  Open Korean corpora: A practical report.  In _Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS)_ , pages 85–93, Online, November 2020b. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.nlposs-1.12.
  * Chun et al. [2018] Jayeol Chun, Na-Rae Han, Jena D. Hwang, and Jinho D. Choi.  Building Universal Dependency treebanks in Korean.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).  URL https://www.aclweb.org/anthology/L18-1347.
  * Clark et al. [2019] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1300.  URL https://www.aclweb.org/anthology/N19-1300.
  * Clark et al. [2020] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/pdf?id=r1xMH1BtvB.
  * Commission [2009] Korea Copyright Commission.  Newspapers and copyright, 2009.
  * Conneau et al. [2017] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark, September 2017\. Association for Computational Linguistics.  doi: 10.18653/v1/D17-1070.  URL https://www.aclweb.org/anthology/D17-1070.
  * Conneau et al. [2018] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1269.  URL https://www.aclweb.org/anthology/D18-1269.
  * Conneau et al. [2020] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://www.aclweb.org/anthology/2020.acl-main.747.
  * Dagan et al. [2006] Ido Dagan, Oren Glickman, and Bernardo Magnini.  The PASCAL recognising textual entailment challenge.  In Joaquin Quiñonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d’Alché Buc, editors, _Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment_ , pages 177–190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.  ISBN 978-3-540-33428-6.
  * de Marneffe and Manning [2008] Marie-Catherine de Marneffe and Christopher D. Manning.  The Stanford typed dependencies representation.  In _Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation_ , pages 1–8, Manchester, UK, August 2008. Coling 2008 Organizing Committee.  URL https://www.aclweb.org/anthology/W08-1301.
  * de Marneffe et al. [2006] Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning.  Generating typed dependency parses from phrase structure parses.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)_ , Genoa, Italy, May 2006. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2006/pdf/440_pdf.pdf.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://www.aclweb.org/anthology/N19-1423.
  * Doddington et al. [2004] George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel.  The automatic content extraction (ACE) program – tasks, data, and evaluation.  In _Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04)_ , Lisbon, Portugal, May 2004. European Language Resources Association (ELRA).  URL http://www.lrec-conf.org/proceedings/lrec2004/pdf/5.pdf.
  * Dolan and Brockett [2005] William B. Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005.  URL https://www.aclweb.org/anthology/I05-5002.
  * Dozat and Manning [2017] Timothy Dozat and Christopher D. Manning.  Deep biaffine attention for neural dependency parsing.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=Hk95PK9le.
  * Eberhard and Simons [2021] David M. Eberhard and Charles D. Simons, Gary F. Fenning.  _Ethnologue: Languages of the World_.  SIL International, Dallas, Texas, 24 edition, 2021.  URL http://www.ethnologue.com.
  * El Asri et al. [2017] Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman.  Frames: a corpus for adding memory to goal-oriented dialogue systems.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 207–219, Saarbrücken, Germany, August 2017\. Association for Computational Linguistics.  doi: 10.18653/v1/W17-5526.  URL https://www.aclweb.org/anthology/W17-5526.
  * Eric et al. [2017] Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning.  Key-value retrieval networks for task-oriented dialogue.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 37–49, Saarbrücken, Germany, August 2017. Association for Computational Linguistics.  doi: 10.18653/v1/W17-5506.  URL https://www.aclweb.org/anthology/W17-5506.
  * Eric et al. [2020] Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur.  MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 422–428, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://www.aclweb.org/anthology/2020.lrec-1.53.
  * Fan et al. [2019] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.  ELI5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3558–3567, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1346.  URL https://www.aclweb.org/anthology/P19-1346.
  * Fernández-González and Gómez-Rodríguez [2019] Daniel Fernández-González and Carlos Gómez-Rodríguez.  Left-to-right dependency parsing with pointer networks.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 710–716, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1076.  URL https://www.aclweb.org/anthology/N19-1076.
  * Finkel et al. [2005] Jenny Rose Finkel, Trond Grenager, and Christopher Manning.  Incorporating non-local information into information extraction systems by Gibbs sampling.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 363–370, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.  doi: 10.3115/1219840.1219885.  URL https://www.aclweb.org/anthology/P05-1045.
  * Gebru et al. [2018] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  _arXiv preprint arXiv:1803.09010_ , 2018.
  * Glockner et al. [2018] Max Glockner, Vered Shwartz, and Yoav Goldberg.  Breaking NLI systems with sentences that require simple lexical inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 650–655, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-2103.  URL https://www.aclweb.org/anthology/P18-2103.
  * Grishman and Sundheim [1996] Ralph Grishman and Beth Sundheim.  Message Understanding Conference- 6: A brief history.  In _COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics_ , 1996.  URL https://www.aclweb.org/anthology/C96-1079.
  * Gururangan et al. [2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2017.  URL https://www.aclweb.org/anthology/N18-2017.
  * Ham et al. [2020] Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh.  KorNLI and KorSTS: New benchmark datasets for Korean natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 422–430, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.39.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.39.
  * Han et al. [2020] Ji Yoon Han, Tae Hwan Oh, Lee Jin, and Hansaem Kim.  Annotation issues in Universal Dependencies for Korean and Japanese.  In _Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020)_ , pages 99–108, Barcelona, Spain (Online), December 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.udw-1.12.
  * Han et al. [2006] Na-Rae Han, Shijong Ryu, Sook-Hee Chae, Seung-yun Yang, Seunghun Lee, and Martha Palmer.  Korean treebank annotations version 2.0.  _Linguistic Data Consortium (LDC), Philadelphia_ , 2006.
  * Han et al. [2018] Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun.  FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4803–4809, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1514.  URL https://www.aclweb.org/anthology/D18-1514.
  * He et al. [2021] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.  DeBERTa: Decoding-enhanced BERT with disentangled attention.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=XPZIaotutsD.
  * Henderson et al. [2014a] Matthew Henderson, Blaise Thomson, and Jason D. Williams.  The second dialog state tracking challenge.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , pages 263–272, Philadelphia, PA, U.S.A., June 2014a. Association for Computational Linguistics.  doi: 10.3115/v1/W14-4337.  URL https://www.aclweb.org/anthology/W14-4337.
  * Henderson et al. [2014b] Matthew Henderson, Blaise Thomson, and Jason D Williams.  The third dialog state tracking challenge.  In _2014 IEEE Spoken Language Technology Workshop (SLT)_ , pages 324–329. IEEE, 2014b.
  * Hendrickx et al. [2010] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz.  SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ , pages 33–38, Uppsala, Sweden, July 2010. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/S10-1006.
  * Hu et al. [2020a] Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss.  OCNLI: Original Chinese Natural Language Inference.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3512–3526, Online, November 2020a. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.314.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.314.
  * Hu et al. [2020b] Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR, 13–18 Jul 2020b.  URL http://proceedings.mlr.press/v119/hu20b.html.
  * Jabbari et al. [2020] Ali Jabbari, Olivier Sauvage, Hamada Zeine, and Hamza Chergui.  A French corpus and annotation schema for named entity recognition and relation extraction of financial news.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2293–2299, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://www.aclweb.org/anthology/2020.lrec-1.279.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-1147.  URL https://www.aclweb.org/anthology/P17-1147.
  * Kakwani et al. [2020] Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar.  IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4948–4961, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.445.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.445.
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Kelley [1984] J. F. Kelley.  An iterative design methodology for user-friendly natural language office information applications.  _ACM Trans. Inf. Syst._ , 2(1):26–41, January 1984.  ISSN 1046-8188.  doi: 10.1145/357417.357420.  URL https://doi.org/10.1145/357417.357420.
  * Khashabi et al. [2018] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-1023.  URL https://www.aclweb.org/anthology/N18-1023.
  * Khashabi et al. [2020] Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, et al.  ParsiNLU: a suite of language understanding challenges for persian.  _arXiv preprint arXiv:2012.06154_ , 2020.
  * Kiela et al. [2020] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 2611–2624. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/1b84c4cee2b8b3d823b30e2d604b1878-Paper.pdf.
  * Kim et al. [2020] Youngmin Kim, Seungyoung Lim, Hyunjeong Lee, Soyoon Park, and Myungji Kim.  KorQuAD 2.0: Korean QA dataset for web document machine comprehension.  _Journal of KIISE_ , 47:577–586, 2020.  ISSN 2383-630X.  URL https://scienceon.kisti.re.kr/srch/selectPORSrchArticle.do?cn=NART99691770&dbt=NART.
  * Kiperwasser and Goldberg [2016] Eliyahu Kiperwasser and Yoav Goldberg.  Simple and accurate dependency parsing using bidirectional LSTM feature representations.  _Transactions of the Association for Computational Linguistics_ , 4:313–327, 2016.  doi: 10.1162/tacl_a_00101.  URL https://www.aclweb.org/anthology/Q16-1023.
  * Klein and Manning [2003] Dan Klein and Christopher D. Manning.  Accurate unlexicalized parsing.  In _Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics_ , pages 423–430, Sapporo, Japan, July 2003. Association for Computational Linguistics.  doi: 10.3115/1075096.1075150.  URL https://www.aclweb.org/anthology/P03-1054.
  * Ko et al. [2020] Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang.  Look at the first sentence: Position bias in question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1109–1121, Online, November 2020\. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.84.  URL https://www.aclweb.org/anthology/2020.emnlp-main.84.
  * Krippendorff [2011] K. Krippendorff.  Computing Krippendorff’s alpha-reliability.  2011\.
  * Kudo [2006] Taku Kudo.  MeCab: Yet another part-of-speech and morphological analyzer, 2006.  URL https://taku910.github.io/mecab/.
  * Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466, March 2019.  doi: 10.1162/tacl_a_00276.  URL https://www.aclweb.org/anthology/Q19-1026.
  * Lalor and Yu [2020] John P. Lalor and Hong Yu.  Dynamic data selection for curriculum learning via ability estimation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 545–555, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.48.  URL https://www.aclweb.org/anthology/2020.findings-emnlp.48.
  * Lan et al. [2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=H1eA7AEtvS.