# Automatically generating question-answer pairs for assessing basic reading
comprehension in Swedish

Dmytro Kalpakchi
Division of Speech, Music and Hearing
KTH Royal Institute of Technology
Stockholm, Sweden
<EMAIL_ADDRESS>
&Johan Boye
Division of Speech, Music and Hearing
KTH Royal Institute of Technology
Stockholm, Sweden
<EMAIL_ADDRESS>

###### Abstract

This paper presents an evaluation of the quality of automatically generated
reading comprehension questions from Swedish text, using the Quinductor
method. This method is a light-weight, data-driven but non-neural method for
automatic question generation (QG). The evaluation shows that Quinductor is a
viable QG method that can provide a strong baseline for neural-network-based
QG methods.

##  1 Introduction

In this article, we aim to establish a strong non-neural, but still data-
driven, baseline for the automatic generation of reading comprehension
questions (RC-QG) in Swedish. It is well-known that reading comprehension is a
complex and multi-layered process, ranging from the simple decoding of
individual words to advanced analytical tasks concerning the quality, veracity
and purpose of whole texts (Alderson, 2000; Shaw and Weir, 2007). In this
work, we aim at only generating RC questions on the _the information-locating
level_ , i.e. questions that facilitate the assessment of readers’ ability to
scan, locate and retrieve relevant information from a single text. This is the
most basic level of RC, as identified by the PISA 2018 report (OECD, 2019,
p.34), but still a crucial reading ability in need of assessment.

Our approach is data-driven, meaning that no handcrafted rules or linguistic
expertise is required. Adapting to new kinds of questions is done via adding
new text-question pairs to the training material. Furthermore, the generated
questions and their respective answers (the QA-pairs) will quite faithfully
reuse the wordings of the text, but can also use words and phrases that are
not explicitly present in the text. Our system is completely open-source with
source code available on GitHub
(https://github.com/dkalpakchi/swe_quinductor).

##  2 Related work

To the best of our knowledge, the literature on reading comprehension question
generation (RC-QG) particularly for Swedish is very limited. Wilhelmsson
(2011, 2012) presented a system for generating questions using manually
specified grammatical transformations (syntactic fronting of unbounded
constituents and substitution of suitable question elements with question
words). By design, their system is able to generate QA-pairs only using
formulations that appear word-by-word in the text. The generated questions
were limited to two categories, the first of which encompasses questions
starting with “vem” (eng. “who/whom”), “vad” (eng. “what”), “vilken” (eng.
“which”) that would concern nominal constituents. The second category concerns
questions to some adverbials. The author also presented a preliminary
evaluation of the questions generated for ten random Wikipedia articles, but
unfortunately neither specified the exact articles nor released the source
code of his system, making a direct comparison impossible.

Lately, neural-network-based text generation methods have become very popular
due their impressive results. Indeed, these methods have been applied to RC-QG
as well, mostly for English (see e.g. Liao et al. (2020); Dong et al. (2019)).
Our goal here is not to compete with these approaches, but rather to present a
more light-weight but still strong baseline to which neural methods can be
compared, but which is also interesting and useful in its own right.

##  3 Data

We have used the _SweQUAD-MC_ dataset (Kalpakchi and Boye, 2021a) consisting
of texts and multiple-choice reading comprehension questions (MCQs) for the
given texts. It was created by three paid linguistics students. They were
instructed to formulate unambiguous questions, such that (1) the answer to the
question appears verbatim in the text, (2) the question cannot be answered
without reading the text (i.e., the text is necessary), and (3) the answer
does not require extra knowledge not present in the text (i.e., the text is
sufficient). These characteristics make the dataset suitable for assessing RC
on the information-locating level.

The dataset is relatively small with the training set consisting of 962 MCQs,
the development (dev) set – of 126 MCQs and the test set – of 102 MCQs. The
distribution of the first two words111which is a decent proxy for question
words in questions in the training set is shown in Figure 1.

Figure 1: The distribution of the first two words in questions of the training
set of SweQUAD-MC

##  4 Method

We used Quinductor Kalpakchi and Boye (2021c), which is a mostly deterministic
data-driven method applicable to any language having a dependency parser based
on Universal Dependencies (Nivre et al., 2020). In particular, it is
applicable to Swedish. Quinductor is only capable of inducing QA-pairs based
on single declarative sentences, which is suitable for assessing RC on the
information-locating level. The method is data-driven, requiring a corpus of
texts with the associated QA-pairs as training material, as well as some
additional files, detailed in Appendix A.

Figure 2: The dependency tree for the sentence “John graduated in 2010” Figure
3: The dependency tree for the sentence “Stocks crased during previous summer
months”

Quinductor works in two stages: first inducing the templates and then using
them to generate questions from unseen sentences. In short, the first stage
involves learning from data to express each given QA-pair in terms of the
dependency structures of the source sentence, which is a basis for both the
question and the answer. To exemplify, consider the sentence “John graduated
in 2010” (with its dependency tree in Figure 2) and the question “When did
John graduate?” with the answer being in “2010”. Quinductor will then learn to
induce the template (4) for the question and the template (4) for the answer.

When did |  [r.nsubj#1] |  [r.lemma] ?
---|---|---
|  (John) |  (graduate)

<r.obl#4>
---
(in 2010)

At test time, Quinductor applies an overgenerate-and-rank strategy, and
attempts to apply all of the induced templates to each given sentence.
Clearly, the more sentences with similar dependency structures will be present
in unseen data, the more successful the method will be. Also, the more
template expressions with angled brackets (matching a whole phrase, as in (4)
above) are present in a template, the higher the generalization chance is. To
exemplify, if we get a new sentence “Stocks crashed during previous summer
months” (with its dependency tree in Figure 3), our previously induced
template will be able to fire and produce the QA-pair “When did stocks crash?”
– “during previous summer months”, although the trees are clearly not
identical. For further details on the method and its generalization
capabilities we refer to the original article.

Statistics | Value
---|---
Number of induced templates | 248
Support per template |
Mean $\pm$ STD | $1.04\pm 0.23$
Median (Min - Max) | 1 (1 - 3)
Table 1:  Descriptive statistics of the templates produced using the training
set of the SweQUAD-MC dataset. Support per template is the number of sentences
from the training set that yield the same template.

In this work we have induced templates based on the training set of SweQUAD-MC
(see more implementation details in Appendix A). As can be seen in Table 1,
this resulted in 248 templates with most of them being unique, i.e. induced
from (being supported by) only one sentence from the training set. Some
templates had higher support with the maximum of 3 sentences per template.

##  5 Evaluation and discussion

The proportion of source sentences (the ones where the correct answer is
found) from the dev. and test sets of SweQUAD-MC, for which Quinductor could
generate something is reported in Table 2.

| dev | test
---|---|---
# of questions in the set | 126 | 102
# of generated questions | 207 | 213
(1) SS with $\geq$ 1 applicable template | 64 | 44
(2) SS with $\geq$ 1 generated question after basic filtering | 49 | 36
(3) SS with $\geq$ 1 generated question after mean filtering | 29 | 24
(3) as % of the respective set | 23% | 23.5%
Generated questions per SS |  |
Mean | 3.2 | 4.8
Standard deviation | 4.1 | 7.2
Median | 2 | 3
Minimum | 0 | 0
Maximum | 23 | 46
Table 2:  Descriptive statistics of the questions induced on the dev. and test
sets of the SweQUAD-MC using the templates, mentioned in Table 1. “SS” stands
for “source sentence(s)”, i.e., the sentence in which the correct answer is
found. “$\geq 1$ applicable template” means that at least 1 question was
induced from the given SS.

For evaluation, we took all 29 QA-pairs generated for the development set and
all 24 QA-pairs generated for the test set of SweQUAD-MC. These 53 QA-pairs
were combined with 53 original QA-pairs corresponding to the same source
sentences from the respective corpora (later referred to as gold QA-pairs).
The resulting 106 QA-pairs and the corresponding source sentences formed 106
evaluation triples, and were presented to 2 human judges (one native Swedish
speaker and one non-native, but with a high proficiency) in a random order
(different for each judge). Following Kalpakchi and Boye (2021c), we required
judges to evaluate each triple using a questionnaire consisting of 9 criteria.
Each criterion required a judgement on a 4-point Likert-type scale sfrom 1
(“Disagree”) to 4 (“Agree”). The evaluation itself was conducted online on an
in-house instance of Textinator Surveys Kalpakchi and Boye (2022). The
evaluation guidelines are reported in Appendix B.

Five criteria concerned questions and were formulated as statements asking
whether a question:

  1. C1

is grammatically correct $(\uparrow)$

  2. C2

makes sense $(\uparrow)$

  3. C3

would be clearer if more information were provided $(\downarrow)$

  4. C4

would be clearer if less information were provided $(\downarrow)$

  5. C5

is relevant to the given sentence $(\uparrow)$

The remaining 4 criteria concerned the answer and asked whether the suggested
answer:

  1. C6

correctly answers the question $(\uparrow)$

  2. C7

would be clearer if phrased differently $(\downarrow)$

  3. C8

would be clearer if more information were provided $(\downarrow)$

  4. C9

would be clearer if less information were provided $(\downarrow)$

$\uparrow$ ($\downarrow$) indicates that the higher (lower) the judgements on
the Likert scale, the better.

Criterion |  | dev | test
---|---|---|---
gold | gen | gold | gen
C1 $\uparrow$ | $\kappa$ | 0.86 | 0.54 | 0.83 | 0.33
$\gamma$ | 0.92 | 0.78 | 0.91 | 0.55
C2 $\uparrow$ | $\kappa$ | 0.59 | 0.49 | 0.88 | 0.39
$\gamma$ | 0.83 | 0.79 | NA/4 | 0.75
Table 3: Inter-annotator agreement for criteria C1 and C2 on the development
and test sets of SweQUAD-MC. $\gamma=\text{NA/X}$ means that at least one
annotator gave the same score X to all evaluation triples, so it is impossible
to count concordant and discordant pairs.

Following Kalpakchi and Boye (2021c), we have measured IAA using Randolph’s
$\kappa$ Randolph (2005) and Goodman-Kruskall’s $\gamma$ (Goodman and Kruskal,
1979). The former, $\kappa$, ranging between $-1$ and $1$, accounts for
agreement in absolute rankings, i.e. being boosted if the annotators gave
exactly same score to an evaluation triple. The value of 0 indicates the level
of agreement that could be expected by chance, the positive (negative) values
indicate agreement better (worse) than chance. The latter, $\gamma$, also
ranging between $-1$ and $1$, accounts for agreement in relative ordering,
i.e. being boosted if the annotators ordered a pair of two evaluation triples
in the same way, no matter the actual scores. $\gamma=0$ indicates no
agreement, $\gamma=1$ denotes a complete agreement and $\gamma=-1$ hints at a
perfect disagreement.

Without a doubt, the two basic criteria, which must necessarily have high
judgements in order to even consider looking at all other criteria, are C1 and
C2. Table 3 shows that the evaluators had high agreement on both criteria,
especially when it comes to relative ordering of triples ($\gamma>0.5$).

As can be seen in Figure 4, nearly all _gold_ questions of the dev and test
sets were judged highly on C1 (median $\geq 3$). At the same time, 17 ($\sim
58\%$) and 10 ($\sim 42\%$) of the _generated_ questions of the dev and test
sets respectively were rated highly on C1, resulting in a total of $50.9\%$
between the sets.

Figure 4 also reveals that surprisingly 4 _gold_ questions in total between
the dev and test sets were judged as not making sense (median $<3$). At the
same time only 11 ($\sim 38\%$) and 8 ($\sim 33\%$) of the _generated_
questions of the dev and test sets respectively were rated highly on C2
(median $\geq 3$), resulting in a total of $35.8\%$ between the sets. We also
observed that only one generated question was judged highly on C2, but lower
on C1.

(a) Development set

(b) Test set

Figure 4: Barplots of the number of evaluation triples with rated highly
(median $\geq$ 3) on C1 (grammatical), C2 (makes sense), and both C1 and C2.
The horizontal red lines indicate a total number of evaluation triples that
were considered (equal for gold and generated data)

We analyzed further only 19 _generated_ QA-pairs between the sets that were
judged highly on both C1 and C2. The analysis was carried out in terms of
criteria C5 and C6, the agreement on both of which was reasonably high (see
Appendix F). Out of these 19, 17 (11 on the dev set and 6 on the test set),
were also rated highly (median $\geq 3$) on C5 and 12 ($63.2\%$), 7 on the dev
set and 5 on the test set, were rated highly on C6. Crucially, there were no
QA-pairs with the suggested answer being judged as correct (high median rating
on C6), but with the question being deemed as irrelevant to the given sentence
(low median judgement on C5). This means that out of all generated QA-pairs,
12 ($22\%$) were judged as completely valid, 7 ($24\%$) on the dev set and 5
($21\%$) on the test set, making Quinductor a resonably strong RC-QG baseline
for SweQUAD-MC.

Some examples of successful generation are presented in Appendix C. Further
analysis (in particular of errors) and related generation examples are
presented in Appendix D. Although automatic evaluation metrics provide very
limited insights, we also report them in Appendix E, following Kalpakchi and
Boye (2021c).

## Acknowledgments

This work was supported by Vinnova (grant 2019-02997), and Digital Futures
(project SWE-QUEST).

## References

  * Alderson (2000) Charles J Alderson. 2000.  _Assessing reading_.  Cambridge University Press.
  * Denkowski and Lavie (2014) Michael Denkowski and Alon Lavie. 2014.  Meteor universal: Language specific translation evaluation for any target language.  In _Proceedings of the ninth workshop on statistical machine translation_ , pages 376–380.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  _Advances in Neural Information Processing Systems_ , 32.
  * Goodman and Kruskal (1979) Leo A Goodman and William H Kruskal. 1979.  Measures of association for cross classifications.  _Measures of association for cross classifications_ , pages 2–34.
  * Kalpakchi and Boye (2021a) Dmytro Kalpakchi and Johan Boye. 2021a.  BERT-based distractor generation for Swedish reading comprehension questions using a small-scale dataset.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 387–403, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * Kalpakchi and Boye (2021b) Dmytro Kalpakchi and Johan Boye. 2021b.  Minor changes make a difference: a case study on the consistency of UD-based dependency parsers.  In _Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021)_ , pages 96–108, Sofia, Bulgaria. Association for Computational Linguistics.
  * Kalpakchi and Boye (2021c) Dmytro Kalpakchi and Johan Boye. 2021c.  Quinductor: a multilingual data-driven method for generating reading-comprehension questions using universal dependencies.  _arXiv preprint arXiv:2103.10121_.
  * Kalpakchi and Boye (2022) Dmytro Kalpakchi and Johan Boye. 2022.  Textinator: an internationalized tool for annotation and human evaluation in natural language processing and generation.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 856–866, Marseille, France. European Language Resources Association.
  * Liao et al. (2020) Yi Liao, Xin Jiang, and Qun Liu. 2020.  Probabilistically masked language model capable of autoregressive generation in arbitrary word order.  _arXiv preprint arXiv:2004.11579_.
  * Nivre et al. (2020) Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020.  Universal dependencies v2: An evergrowing multilingual treebank collection.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 4034–4043.
  * OECD (2019) OECD. 2019.  _PISA 2018 Assessment and Analytical Framework_.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * Randolph (2005) Justus J Randolph. 2005.  Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss’ fixed-marginal multirater kappa.  _Online submission_.
  * Sharma et al. (2017) Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017.  Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation.  _CoRR_ , abs/1706.09799.
  * Shaw and Weir (2007) Stuart D Shaw and Cyril J Weir. 2007.  _Examining writing: Research and practice in assessing second language writing_ , volume 26.  Cambridge University Press.
  * Wilhelmsson (2011) Kenneth Wilhelmsson. 2011.  Automatic question generation from swedish documents as a tool for information extraction.  In _Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011)_ , pages 323–326.
  * Wilhelmsson (2012) Kenneth Wilhelmsson. 2012.  Automatic question generation for swedish: The current state.  In _Proceedings of the SLTC 2012 workshop on NLP for CALL; Lund; 25th October; 2012_ , 080, pages 71–79. Linköping University Electronic Press.

##  Appendix A Implementation details

We have used dependency parsing models from the package called
Stanza222https://github.com/stanfordnlp/stanza Qi et al. (2020), version 1.4.2
and used quinductor333https://github.com/dkalpakchi/quinductor package
Kalpakchi and Boye (2021c), version 0.2.2. This is important, because it is
likely that the induced templates will differ if different models are used.
The supplementary files necessary for either generating templates or during
ranking the generated questions were obtained as follows:

  * •

the IDFs were calculated based on the SweQUAD-MC training set (necessary for
inducing templates);

  * •

the morphological n-gram model was calculated based on training, dev and test
sets of the UD’s Talbanken
treebank444https://universaldependencies.org/treebanks/sv_talbanken/index.html
(necessary for ranking);

  * •

the question-word model was calculated based on the SweQUAD-MC training set
(necessary for ranking).

Note that the question-word model was induced using one of the earlier
versions of Stanza, which is why the distributions differ slightly, compared
to the 1.4.2. All these suplementary files are released in the GitHub
repository, associated with this paper.

##  Appendix B Human evaluation guidelines

Here we report the exact wording of the guidelines presented to the human
evaluators.

Tack att du deltar i vår utvärdering av läsförståelsefrågor! Du kommer att se
ett antal meningar (en i taget) tillsammans med fråga och det rätta svaret
(QA-par). Du kommer också att se ett antal påstående för varje QA-par. Din
uppgift är att bestämma i vilken utsträckning du håller med varje påstående.
Om frågan är obegriplig ska du välja ”1” för alla påståenden relaterade till
det föreslagna svaret. Vänligen ignorera alla möjliga formatteringsfel, t.ex.
skiljetecken (.,!?:;) eller versaler som saknas.

##  Appendix C Generation examples

> Sentence 1: inom hälso- och sjukvården arbetar dietisten med
> nutritionsbehandling och kostrådgivning, både med enskilda patienter och i
> grupp.
> Question 1: var arbetar dietisten med nutritionsbehandling och
> kostrådgivning?
> Suggested answer 1: inom hälso- och sjukvården

> Sentence 2: om du är borta längre än ett år eller om du planerar att bosätta
> dig i ett annat land kan migrationsverket återkalla ditt uppehållstillstånd.
> Question 2: vad kan migrationsverket göra om du är borta längre än ett år
> eller om du planerar att bosätta dig i ett annat land?
> Suggested answer 2: återkalla uppehållstillstånd

> Sentence 3: hovslagaren behandlar även skador i hoven till exempel
> hovsprickor eller hovbölder.
> Question 3: vilka behandlar hovslagaren?
> Suggested answer 3: skador i hoven

> Sentence 4: pulverfärg appliceras ofta på metallkomponenter.
> Question 4: vad appliceras ofta på metallkomponenter?
> Suggested answer 4: pulverfärg

> Sentence 5: miljö- och hälsoskyddsinspektörer samarbetar till exempel med
> andra tjänstemän inom kommuner, företag och myndigheter.
> Question 5: vad gör miljö- och hälsoskyddsinspektörer?
> Suggested answer 5: samarbetar till med andra tjänstemän inom kommuner ,
> företag och myndigheter

##  Appendix D Error analysis

The QA-pairs that are rated low on C2 (question makes sense) represent the
worst kind of errors, which are impossible to correct automatically (like,
say, some grammatical errors). Hence, let’s examine closer some examples of
evaluation triples with generated QA-pairs containing questions that were
rated low (median $<3$) on C2. Please ignore formatting errors (like absense
of punctuation or capital letters).

The questions from the following triples got a median score of 1 on C2:

> Sentence 1: arbetsuppgifterna varierar beroende på var man är anställd men
> den främsta uppgiften är att se till så att hyresgästerna känner sig trygga
> och nöjda i sin arbets- och boendemiljö.
> Question 1: vad gör arbetsuppgifterna beroende på var man är anställd men
> den främsta uppgiften är att se till så att hyresgästerna känner sig trygga
> och nöjda i sin arbets- och boendemiljö?
> Suggested answer 1: varierar

> Sentence 2: du kan då behöva vänta på ditt nya beslut utanför sverige.
> Question 2: vad kan du göra på ditt nya beslut utanför sverige?
> Suggested answer 2: vänta

The questions from the following triples got a median score of 1.5 on C2:

> Sentence 3: den största delen består av barrskog, men i söder breder lummig
> lövskog ut sig.
> Question 3: vad består den största delen från?
> Suggested answer 3: barrskog

> Sentence 4: en stuntman ersätter filmskådespelare i riskfyllda situationer.
> Question 4: var ersätter stuntman?
> Suggested answer 4: i riskfyllda situationer

The question from the following triple got a median score of 2 on C2:

> Sentence 5: till exempel tillbringar kirurgen en stor del av sin tid i
> operationssalen, medan en psykiatriker arbetar mycket med samtal med sina
> patienter.
> Question 5: vem tillbringar till en stor del av sin tid i operationssalen
> medan en psykiatriker arbetar mycket med samtal med sina patienter?
> Suggested answer 5: kirurgen

The questions from the following triples got a median score of 2.5 on C2:

> Sentence 6: en sjukskriven medarbetare kostar lågt räknat 2 740 kronor per
> dag*.
> Question 6: vad gör en sjukskriven medarbetare?
> Suggested answer 6: kostar kronor

> Sentence 7: man bör ha god fysik då arbetet kan slita på nacke och axlar.
> Question 7: vad bör man göra då arbetet kan slita på nacke och axlar?
> Suggested answer 7: ha fysik

Question 1 was generated using the template vad gör [r.nsubj#1] <r.advcl#2>?.
The problematic one in the case of sentence 1 turned out to be the last
template expression <r.advcl#2>, which takes the whole subtree, whose root can
be found by following the arc labeled advcl from the root r of the dependency
tree. In this case the subtree was too large and thus the generated question
contains a lot of unnecessary information. This is one typical type of errors
that could be referred to as _overgeneralization_ , i.e. the case when the
induced templates become too general. However, a balance between letting
Quinductor to induce too generic or too specific templates is difficult to
strike, so such kinds of errors are inevitable. Question 5 suffers from the
same problem.

Question 2 is based on the template vad [r.aux#1] [r.nsubj#2] göra <r.obl#3>?,
which in turn was generated from the sentence “vad behöver
ambulanssjuksköterskan göra vid större olyckor med många skadade?”. The
structure is clearly very similar, but the preposition of the oblique nominal
happened to be different, so the template expression <r.obl#3> did not
generalize correctly. In fact questions 3 suffers from similar problems.

Question 4 represents the case when the question word from the template is
wrong. All question words are guaranteed to be recorded verbatim as strings
and not deduced from the dependency tree of the sentence, so such errors are
also inevitable in the current version of Quinductor.

QA-pairs 6 and 7 represent more successful applications of templates for the
questions, although the questions are still not perfectly intelligble. The
major problem with these QA-pairs are the answers, which in fact suffer an
inverse problem compared to that of question 1. For instance, the template for
the suggested answer for question 7 is [r] [r.obj#4]. The problem with this
template is that it just picks up only specific nodes of the dependency tree
and is prone to errors if the object, referred by r.obj#4 will turn out to
have some dependents, which will also be relevant to include in the answer.
This problem could be referred to as _undergeneralization_. As previously
mentioned, the balance between over- and undergeneralization is hard to
strike, especially given that Quinductor is a data-driven method and all
templates do depend on the data at hand.

We noted that sometimes these errors arise, because of inconsistency of
dependency parsers themselves, which is not a new observation. For instance,
Kalpakchi and Boye (2021b) showed that even changes as minor as replacing one
4-digit numeral by another can cause surprisingly large inconsistencies in the
resulting trees using the state-of-the-art dependency parsers, in particular
for Swedish.

##  Appendix E Automatic evaluation metrics

Following Kalpakchi and Boye (2021c), we have calculated BLEU-N, ROUGE-L and
CIDEr using nlg-eval Sharma et al. (2017) and METEOR using METEOR-1.5
Denkowski and Lavie (2014) and specifying the language to be Swedish.

Metric | dev | test
---|---|---
BLEU-1 | 0.39 | 0.22
BLEU-2 | 0.29 | 0.15
BLEU-3 | 0.22 | 0.09
BLEU-4 | 0.18 | 0.06
METEOR | 0.33 | 0.21
ROUGE-L | 0.38 | 0.27
CIDEr | 0.76 | 0.84
Table 4:  Automatic evaluation on the development and test sets of SweQUAD-MC
only for generated questions ranked first.

##  Appendix F Detailed IAA analysis

Inter-annotator agreement (IAA) for criteria C3 - C9 is presented in Table 5.
Let’s start by analyzing the agreement on the development set. We observed
that agreement on C5 - C8 is quite strong for both gold and generated QA-
pairs. On the other hand, $\gamma$ is strongly negative for C3 (the question
would be learer if more information were provided), indicating that the
annotators relative ranking are almost exactly opposite. At the same time
$\kappa$ on C3 is quite weak as well, especially on the generated QA-pairs.
For the generated questions both $\kappa=0.08$, so $\gamma=-0.7$ indicate
almost no agreement in absolute scores and a strong disagrement in relative
orderings.

Criterion |  | dev | test
---|---|---|---
gold | gen | gold | gen
C3 $\downarrow$ | $\kappa$ | 0.45 | 0.08 | 0.83 | 0.17
$\gamma$ | -1.0 | -0.7 | NA/1 | -0.09
C4 $\downarrow$ | $\kappa$ | 0.95 | 0.91 | 1.0 | 0.83
$\gamma$ | NA/1 | -1.0 | NA/1 | 1.0
C5 $\uparrow$ | $\kappa$ | 0.77 | 0.72 | 0.67 | 0.44
$\gamma$ | 0.82 | 0.93 | NA/4 | 0.81
C6 $\uparrow$ | $\kappa$ | 0.72 | 0.63 | 0.67 | 0.56
$\gamma$ | 0.79 | 0.88 | -1.0 | 0.78
C7 $\downarrow$ | $\kappa$ | 0.63 | 0.49 | 0.94 | 0.83
$\gamma$ | 0.62 | 1.0 | NA/1 | 1.0
C8 $\downarrow$ | $\kappa$ | 0.54 | 0.63 | 0.94 | 0.83
$\gamma$ | 0.4 | 0.73 | NA/1 | 0.93
C9 $\downarrow$ | $\kappa$ | 1.0 | 1.0 | 1.0 | 1.0
$\gamma$ | NA/1 | NA/1 | NA/1 | NA/1
Table 5: Inter-annotator agreement for criteria C3 - C9 on the development and
test sets of SweQUAD-MC. $\gamma=\text{NA/X}$ means that at least one
annotator gave the same score X to all evaluation triples, so it is impossible
to count concordant and discordant pairs.

The scores for the gold questions on C3 are much more peculiar, namely
$\gamma=-1.0$, while $\kappa=0.45$, indicating complete opposite relative
ordering between the annotators, while also having a moderate agreement in
absolute ranks. If we look closer at the data, it turns out that annotator A
gave almost all gold triples the score of 1 on C3, except one triple (let’s
call it $T_{x}$) that got the score of 3. Hence, this is the only triple that
can be used to establish relative ordering for the annotator A. Now that very
same $T_{x}$ got the score of 1 on C3 from the annotator B. This means that
$T_{x}$ will always be scored higher than all other triples for the annotator
A, whereas it will be scored lower than other triples for the annotator B.
This, in turn, means that all relative orderings are opposite between the
annotators, resulting in $\gamma=-1$. This example is a good illustrataion of
why only $\kappa$ or $\gamma$ is not enough for assessing the IAA, but indeed
both are needed and should be interpreted with caution. A similar situation is
observed for the generated questions from the development set on the criterion
C4, and for the gold questions from the test set on the criterion C6.


# ’Tis but Thy Name: Semantic Question Answering Evaluation
with 11M Names for 1M Entities

Albert Huang
The Nueva School
<EMAIL_ADDRESS>

###### Abstract

Classic lexical-matching-based QA metrics are slowly being phased out because
they punish succinct or informative outputs just because those answers were
not provided as ground truth. Recently proposed neural metrics can evaluate
semantic similarity but were trained on small textual similarity datasets
grafted from foreign domains. We introduce the Wiki Entity Similarity (WES)
dataset, an 11M example, domain targeted, semantic entity similarity dataset
that is generated from link texts in Wikipedia. WES is tailored to QA
evaluation: the examples are entities and phrases and grouped into semantic
clusters to simulate multiple ground-truth labels. Human annotators
consistently agree with WES labels, and a basic cross encoder metric is better
than four classic metrics at predicting human judgments of correctness.

##  1 Introduction

Information retrieval tools have already revolutionized how we interact with
knowledge: what would have taken a half-hour library trip can be learned in
minutes through the internet. Question answering (QA) models can take this one
step further because they can respond to queries with summative insights
instead of just listing relevant documents. Because they dictate the
optimization goal of QA research, QA evaluation metrics are a key in directing
the future of QA model development.

Classic question-answering evaluation metrics only consider the token overlap
between a model’s output and human-annotated ground-truth answers. However,
these evaluation metrics fail to consider the plurality of possible correct
answers for every question. For instance, a “when” question can be answered
using either a year or an event name (i.e. “During Super Bowl XXXI. ”)
Furthermore, correct answers can have wildly different lexical signatures if
they contain different words, phrasing, or level of detail. Existing QA
metrics also cannot evaluate against multiple ground-truth annotations
simultaneously, leading to underutilization of dataset resources. Therefore,
metrics that rely on the tokens in QA model outputs unfairly punish non-
conforming models, including those that provide creatively succinct or
informative answers. These metrics force models to imitate the threadbare,
minimal ground-truth answers that crowd workers tend to write, exacerbating
the problem. Current QA metrics are limiting the usefulness of QA models. With
the rise of generative QA, in which models have more freedom to use different
words or phrasing than extractive QA, the need for semantic answer evaluation
has only grown.

Previous semantic techniques used latent word or phrase embeddings to
calculate the similarity between answers. More recently, end-to-end neural
text similarity models have been applied to QA evaluation. However, they are
trained on small datasets grafted from other tasks. In particular, QA models
often answer questions about specific entities or phrases; there are currently
no entity or short phrase semantic similarity datasets.

Figure 1: Graphical comparison of the most similar existing QA metrics. Figure
2: The WES generation pipeline. (article title, link text) pairs are extracted
from the wikitext, then negative examples are generated by pairing article
titles with link texts from other article titles. Groups of link text that are
associated with the same article are all synonyms. These "synonym clusters"
can be used to train evaluation metrics that consider multiple ground-truth
answers simultaneously.

In this paper, we introduce the Wiki Entity Similarity dataset: a large, task-
specific semantic similarity dataset to train question-answering metrics. WES
has over 11 million high-quality examples of synonymous and non-synonymous
entities and phrases. In addition, WES is systematically mined from a
democratic data corpus (Wikipedia), reducing bias from individual authors or
annotators. Furthermore, WES generates synonym clusters of multiple phrases,
which can be broken down pairwise into the classic semantic similarity
formulation or grouped to simulate multiple ground-truth answers. Despite
being auto-generated, human annotators consistently agree with WES labels. We
show that a basic cross-encoder trained on WES is better at predicting human
answers of correctness than classic metrics.

For the remainder of this paper, “outputs” are produced by QA models and
“answers” are the ground-truth annotations that the model is trained on. In
section 2, we take a closer look at current QA evaluation metrics and analyze
their weaknesses. In section 3, we describe the generation of the novel WES
dataset. In section 4, we characterize WES and present a baseline model
trained on it. Finally, we conclude in section 5 with directions for future
work.

##  2 Related Work

Although there are numerous desirable qualities for question answering models,
we focus on evaluating the factual or conversational correctness of answers.
Other metrics exist for testing information content, answer coverage, and
other desirable qualities; however, this work is orthogonal to such metrics
and should be used in conjunction with them. This section details existing
metrics for evaluating the correctness of a model-generated output based on
human-generated ground-truth answers.

Exact Match (EM) is a standard binary metric for evaluating extractive QA.
Before comparison, outputs and answers are often normalized by removing
punctuation and stop words and converting to lowercase, as in (Rajpurkar et
al., 2016). This metric relies on exact lexical matching: outputs that contain
synonyms or informational prepositional phrases are considered incorrect by
this metric.

F1 is a standard floating-point metric that measures the token overlap between
the output and answer. It punishes longer, more informative outputs because
such answers are less likely to exactly match the annotated ground truth.
Finally, although F1 is more forgiving than EM, it fails to differentiate
lexical and semantic dissimilarities. As a result, there is no threshold at
which the F1 score correlates well with correctness.

EM and F1 are the most prevalent question answering metrics today, serving as
the leaderboard metrics for many of the most popular datasets (Rajpurkar et
al., 2016, Kwiatkowski et al., 2019, Yang et al., 2018, Choi et al., 2018).
However, numerous works have found that lexical metrics can diverge from human
judgements of model performance (Min et al., 2021, Chen et al., 2019, Risch et
al., 2021).

BLEU, ROUGE-N, ROUGE-L, METEOR: (Papineni et al., 2002), (Lin and Hovy, 2003),
(Lin, 2004), (Banerjee and Lavie, 2005) are common Natural Language Generation
metrics based on measuring the n-gram overlap between output and answer. For
instance, ROUGE-L measures the longest common subsequence and METEOR weights
tokens by importance before applying an F1 analog. Like F1, these metrics
punish longer, more detailed answers and unduly punish optional but
informative prepositional phrases in model outputs.

In summary, each of the above metrics relies on lexical matching to perform
similarity checks. As a result, these metrics unfairly punish longer, more
informative outputs that have unnecessary but useful prepositional phrases,
and are particularly sensitive to the exact wording of the ground-truth
annotation. Furthermore, the above metrics fail to model clause order and
logical dependency in model outputs, which advantages illogical outputs that
happen to contain words from the ground-truth answer. In recent years, newer
metrics have been proposed that use contextual or semantic embeddings to
compare sequences.

BERTScore (Zhang et al., 2020) compares BERT contextual embeddings instead of
raw tokens, which may more accurately capture underlying meanings and long-
term dependencies. However, BERTScore still unfairly punishes optional but
useful prepositional phrases as it is fundamentally a token-wise comparison.
Finally, (Chen et al., 2019) has shown METEOR to correlate better with human
judgment than BERTScore.

Semantic Answer Similarity (SAS) (Risch et al., 2021) takes an end-to-end
approach to evaluating generative QA. They compare bi-encoder, cross-encoder,
and BERTScore approaches to QA evaluation, and find that a cross-encoder
trained on answer similarity correlates most strongly with human judgment.
However, SAS is trained on the Semantic Textual Similarity Benchmark (Cer et
al., 2017), which has only 7k foreign-domain training examples. The creation
of such semantic similarity datasets is tedious and susceptible to bias, and
the small dataset size means models are prone to overfitting. We aim to
improve on this cross-encoder approach by pretraining on a new, large, domain-
specific, auto-generated dataset.

In summary, neural metrics are an emerging standard for more accurate
question-answering evaluation. We aim to improve neural QA evaluation by
training a new metric on a novel, large, QA-specific dataset that encodes
entity similarity rather than general sentence similarity.

##  3 Wiki Entity Similarity Dataset

The WES dataset is generated by pairing link text and target article text for
links in Wikipedia. We generate our dataset in two stages: filtered link
collection and negative example generation.

###  3.1 Link Filtering

Following (Karpukhin et al., 2020), we mine links from the English Wikipedia
XML dump from Dec. 20, 2018. The XML format encodes all links in the form
‘[target article title|displayed link text]‘, where the link will be displayed
on the source page. We search directly for links that match the following
criteria:
Article title contains no hashtag as hashtags denote links to subsections of
Wikipedia articles.
Article title contains no colon as colons denote pages under internal
categories, such as User, User talk, Category, etc. (Wikipedia contributors,
2022)
Article title contains no parentheses as most (90% of 500 randomly sampled)
links to these articles simply remove the parenthetical instead of containing
a synonym of the article title.

See Table 1 for examples of the motivation behind filtering out links with the
above characters.

Rule | Sample Title | Sample Link Text
---|---|---
`#` | `VAX#Name` | `VAX trademark`
`:` | `WP:TWINKLE` | `TW`
`(` | `Styx (Band)` | `Styx`
Table 1: Motivation for excluding links to article titles that contain certain
characters.

After character-level filtering, we use additional heuristics to improve the
quality of the link corpus.
Deduplication: we remove duplicate (article title, link text) pairs to improve
dataset balance. Link texts with the same content but different capitalization
are considered different to preserve the semantic meaning of capitalization in
proper nouns.
Incoming link threshold: we enforce that each link in the corpus must link to
an article that has a minimum of parameter n incoming links (citations) to
filter out rarely used stub articles. Higher values of n ensure higher quality
articles and a larger, more representative sample of synonyms for each article
title.
Dictionary word linktext: we remove links that have a dictionary word for the
link text but a named entity or phrase for the target article title as these
dictionary words are rarely fully qualified synonyms to the linked article
title. Words are considered to be dictionary words if their WordNet (Fellbaum,
2000) lemmatization is contained in the Python `nltk`’s (Bird et al., 2009)
`words` corpus.

We experiment with link threshold values $k=5$, $k=10$, and $k=20$, obtaining
link corpora of 5,787,081, 4,407,409, and 3,195,545 distinct synonym pairs,
respectively.

###  3.2 Negative Example Generation

After harvesting synonyms from the Wikipedia links, we generate negative (non-
synonymous) title-text pairs by matching article titles with link texts from
other articles. The negative examples for an article title with $k$ link
synonyms are generated by sampling $k$ other articles from the link corpus,
then sampling one associated link text from each article to be paired with the
article title (see how a negative example in the pink synonym cluster is
created by copying a blue link text from a different article in Fig. 2). This
results in each article title being associated with the same number of non-
synonyms as distinct synonyms, preserving dataset balance. Choosing non-
synonyms from different articles ensures a representative covering of the
sample space and precludes duplicate negative examples. After negative example
generation, the largest version of WES (with incoming link threshold $k=5$)
has 11M+ examples, and the highest quality version (with $k=20$) contains 6.4M
examples.

Future iterations of the WES dataset may be made more difficult through
adversarial generation. Such methods condition negative example generation on
the article title to create negative examples which are more similar to but
not synonymous with the paired phrase. Possible future generation techniques
include:
F1 ranking: negative examples generated from the most lexically similar
alternatives will create textually similar but semantically different
examples, increasing difficulty for token-based models.
Substring selection: negative examples generated from short substrings of the
article title may be synonymous in context, but will not be specific enough to
globally qualify a concept. Such examples force models to watch for
appropriate specificity.
Word co-occurrence: negative examples generated from non-synonymous words with
high co-occurrence will create pairs of grammatically feasible phrases,
increasing difficulty for part-of-speech-based models.
Article co-occurrence: the titles of often co-current but semantically
distinct articles can be treated as very difficult non-synonymous examples,
teaching models to differentiate between parts of speech and improving
sensitivity.

##  4 Analysis

We use human evaluation to check the quality of the collected WES dataset. For
each task, we randomly sample 25 positive and 25 negative training pairs from
the link threshold $k=20$ dataset. Two annotators rate the synonymy of the
pairs following the ranking scheme used in (Risch et al., 2021). We treat
annotated scores of 2 or 3 as “synonymous” and 1 as “non-synonymous,” and find
that only 2% of labels are incorrect. Dataset label accuracies and inter-
annotator agreement from different stages in the filtering process are listed
in Table 1.

Dataset (link threshold $k=20$) | Acc | $r$
---|---|---
Full Dataset | 98% | 0.932
Without Dictionary Filtering | 91% | 0.908
Table 2: Average accuracy and Pearson’s $r$ correlation between annotators in
the full dataset and ablations.

##  5 Conclusion

We introduce WES, an 11M example semantic entity similarity dataset for
training question answering evaluation models. WES is generated by treating
Wikipedia link texts and target article titles as synonyms then filtering for
quality. WES is targeted to question answering evaluation, independent of
human annotators, and consistent with human judgment. We hope that future
question-answering datasets will implement semantic evaluation metrics in
their leaderboards to encourage the development of more free-form models. In
future works, link-mining similarity datasets like WES can be made more
challenging by generating negative examples adversarially as described at the
end of section 3, more consistent by unioning semantic clusters according to
Wikipedia’s internal redirect pages, and more comprehensive by leveraging
link-to-link pairwise synonymy within semantic clusters.

## Acknowledgements

We would like to sincerely thank Di Jin for his guidance during the literature
review and method design process, Lufan Wang and Houjun Liu for their feedback
on data collection methods and help proofreading this paper, and Yan Liu,
Qiong Huang for their help with dataset analysis and compute. We would also
like to thank (Si et al., 2021) for reminding us to have some fun in our
literature.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  " O’Reilly Media, Inc.".
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2019) Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Evaluating question answering evaluation.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 119–124, Hong Kong, China. Association for Computational Linguistics.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  Quac: Question answering in context.  _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Fellbaum (2000) Christiane D. Fellbaum. 2000.  Wordnet : an electronic lexical database.  _Language_ , 76:706.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin and Hovy (2003) Chin-Yew Lin and Eduard Hovy. 2003.  Automatic evaluation of summaries using n-gram co-occurrence statistics.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 150–157.
  * Min et al. (2021) Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen tau Yih. 2021.  Neurips 2020 efficientqa competition: Systems, analyses and lessons learned.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
  * Risch et al. (2021) Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021.  Semantic answer similarity for evaluating question answering models.  _Proceedings of the 3rd Workshop on Machine Reading for Question Answering_.
  * Si et al. (2021) Chenglei Si, Chen Zhao, and Jordan Boyd-Graber. 2021.  What’s in a name? answer equivalence for open-domain question answering.  _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Wikipedia contributors (2022) Wikipedia contributors. 2022.  Wikipedia:manual of style.  [Online; accessed 21-February-2022].
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.