# Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer
Explanation

Yuyan Chen1, Yanghua Xiao${}^{1,2\,\textrm{{\char 0\relax}}}$, Bang Liu3,4 1
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan
University, Shanghai, China  2 Fudan-Aishu Cognitive Intelligence Joint
Research Center, Shanghai, China  3 RALI & Mila, Université de Montréal,
Montréal, Québec, Canada  4 Canada CIFAR AI Chair<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Interpreting the predictions of existing Question Answering (QA) models is
critical to many real-world intelligent applications, such as QA systems for
healthcare, education, and finance. However, existing QA models lack
interpretability and provide no feedback or explanation for end-users to help
them understand why a specific prediction is the answer to a question. In this
research, we argue that the evidences of an answer is critical to enhancing
the interpretability of QA models. Unlike previous research that simply
extracts several sentence(s) in the context as evidence, we are the first to
explicitly define the concept of evidence as the supporting facts in a context
which are informative, concise, and readable. Besides, we provide effective
strategies to quantitatively measure the informativeness, conciseness and
readability of evidence. Furthermore, we propose Grow-and-Clip Evidence
Distillation (GCED) algorithm to extract evidences from the contexts by trade-
off informativeness, conciseness, and readability. We conduct extensive
experiments on the SQuAD and TriviaQA datasets with several baseline models to
evaluate the effect of GCED on interpreting answers to questions. Human
evaluation are also carried out to check the quality of distilled evidences.
Experimental results show that automatic distilled evidences have human-like
informativeness, conciseness and readability, which can enhance the
interpretability of the answers to questions.

###### Index Terms:

Explainable Question Answering, Evidence Distillation, Grow-and-Clip,
Informative-yet-Concise Evidence

##  I Introduction

Question Answering (QA) is an important task in Natural Language Processing
(NLP) and plays a vital role in many real-world applications, such as search
engines [37, 45], chatbots [12, 36, 11], and so on. However, most of the
current QA systems focus only on extracting or generating the correct answers,
but lacking reasonable evidences to support the answers simultaneously. This
hurts real users’ confidence on the answers and limits the utility of QA
systems in many real-world applications that require high interpretability,
such as evidence-based medicine [39, 33], children education [21, 49] and
online consulting. For example, evidences in medical QA [39, 33] offer key
clues of patients’ symptoms for clinical doctors, assisting them to make
correct diagnoses. But treatment without evidences will hardly be trusted or
accepted by patients. Therefore, evidences are important for providing humans
suggestive references and important clue information in the real world.

How to define and extract high-quality evidences is a non-trivial problem. A
direct idea is taking the documents where the answer comes from as the
evidence. But in most cases, a document is too long to be a satisfying
evidence. Hence, recent research tends to select a few sentences from factual
source text such as Wikipedia [14, 51] or Freebase [10, 28, 54] as the
evidence. For a sample QA-pair shown in Fig. 1, $S_{1}$ and $S_{2}$ represent
the evidence for a given QA pair generated by Min et al. [35]. The QA-related
part in this evidence has only 16 words, far shorter than the length of the
whole sentence-level evidence (37 words). Therefore, although sentence-level
evidences are informative which have a better coverage and contain rich
information to support answers, they are still too coarse-grained with
irrelevant parts and may introduce noises. And noisy information in the
evidence will inevitably distract end-users from the most QA-related part of
the evidence, increasing the difficulty to understand the explanation and
hurting the user-friendliness of the QA system.

Figure 1: An example of sentence-level evidence for QA. The evidence is
composed of two sentences with 37 words. The QA-related part is colored in red
with only 16 words, and other parts are considered as noises with 21 words,
even longer than the QA-related part.

Based on the above analysis, we argue that _both informativeness and
conciseness are essential for QA evidences._ However, it is a great challenge
to trade-off between informativeness and conciseness of the evidences, as they
are contradictory to each other. A too informative evidence tends to involve
redundant information, while a too concise evidence might miss some relevant
information. Furthermore, the difficulty is deteriorated by simultaneously
guaranteeing readability of the evidences, which is necessary for ensuring
user-friendliness of QA systems. Most previous relevant efforts, such as
Schuff et al. [43], only focus on informativeness of evidences without
considering the conciseness of the evidences. Although human experts are
further employed to write informative-yet-concise evidences in QA systems
[40], it inevitably incurs unaffordable human cost.

To automatically trade-off between informativeness and conciseness, and
guarantee readability of an evidence, we propose a novel task of _evidence
distillation_ , which aims at extracting informative-yet-concise evidences
from the given contexts to explain answers. To solve this task, we design a
systematical quantitative framework that evaluates the goodness of evidences
in terms of _informativeness, conciseness_ , and _readability_. Our core idea
is that a good evidence shall be supportive to the answer (informativeness),
contains few non-essential information to answer the question (conciseness),
and easy to understand (readability). Based on the framework, we further
propose a novel Grow-and-Clip Evidence Distillation (GCED) algorithm to
distill the optimal evidence while trickily balancing between the
informativeness and conciseness.

Different from previous solutions, GCED finds the optimal evidence at token-
level, which is more fine-gained than that at sentence-level and allows more
flexibility to generate more informative-yet-concise evidences. It is also
noteworthy to mention that our GCED algorithm has three advantages compared to
previous research: i) our evidence distillation does not need human
annotation; ii) each step is traceable, which enhances the interpretability of
generated evidences for QA pairs; and iii) the algorithm is applicable to use
either given documents or structured knowledge as the QA source repository,
and domain independent.

We carry out human evaluations to systematically assess the quality of our
generated evidences from two widely acknowledged QA datasets: SQuAD [42, 41]
and TriviaQA [27]. The results verify that our solution is able to find the
evidences with high informativeness, conciseness and readability, which can
explain answers very well. Moreover, if the QA systems have ground-truth
answers in an ideal setting or some applications (such as searching engine),
we can use the ground-truth answers to distill evidences. These evidences are
a form of concise and readable contexts which can also explain/support the
ground-truth answers.

To summarize, our contributions in this paper are threefold:

  * •

We establish a quantitative framework to evaluate the goodness
(informativeness, conciseness, and readability) of evidences for QA.

  * •

We propose a novel Grow-and-Clip Evidence Distillation (GCED) algorithm to
distill optimal evidences, with carefully designed heuristic to keep balance
between informativeness, conciseness and readability of the distilled
evidences.

  * •

We systematically evaluate the quality of our distilled evidences based on
ground-truth answers and predicted answers with human evaluation. Experimental
results demonstrate that the evidences are of good quality (informativeness,
conciseness, and readability) in both two situations.

##  II Problem formulation

In this section, we first formulate our problem and introduce the preliminary
concepts used in this paper. Then we present the quantitative evaluation
framework for QA evidences.

###  II-A Evidence Distillation

The evidence distillation task in this research can be summarized as the
following: given a tuple set $S=\\{(q_{i},a_{i},c_{i},e_{i})\\}_{i=1}^{N}$
where $N$ refers to the total number of QA pairs, evidence distillation aims
to take the natural language question $q_{i}$, the predicted answer $a_{i}$ as
well as the given contexts $c_{i}$ as the input, and outputs a good evidence
$e_{i}$ which helps to explain why $a_{i}$ is the answer to the question
$q_{i}$. Compared to the full context $c_{i}$, the distilled evidence $e_{i}$
is more concise but preserves essential information for explaining the answer
$a_{i}$.

A critical research problem in our task is how to evaluate the goodness of an
evidence. We argue that a good evidence is expected to have the following
three characteristics:

  * •

i) _Informativeness_. A good evidence is informative so that the input answer
can be inferred from the evidence. For example, the first candidate evidence
shown in Fig. 2 contains more relevant information to infer the answer of this
question, such as "duke" and "Battle of Hastings". It is thus more
informative. On the contrary, the second candidate evidence only contains
limited information such as "William the Conqueror" and "led", so we cannot
infer the answer from this evidence. Therefore, it’s not informative.

  * •

ii) _Conciseness_. A good evidence is expected to be concise which is as short
as possible but at least longer than the answer. For example, the second
candidate evidence shown in Fig. 2 only has 4 words, just one more word than
the answer which have 3 words. Therefore, it’s considered concise. On the
contrary, the third candidate evidence is quite redundant with 25 words, much
longer than the answer, which is not concise.

  * •

iii) _Readability_. A good evidence is readable and and (almost) grammatically
correct, so that it is understandable by humans. For example, the third
candidate evidence shown in Fig. 2 has clear logic and doesn’t have grammar
mistakes. The meaning is easy to understand, and thus is considered readable.
On the contrary, the first candidate evidence has grammar mistakes, such as
missing a preposition between "behalf" and "duke", and missing a verb between
"William the Conqueror" and "Battle of Hastings". Therefore, it’s difficult to
understand with bad readability.

Compared with the three evidences, the fourth candidate evidence shown in Fig.
2 satisfies all the three characteristics. It contains enough information to
answer the question without redundant noises. It also has better readability
without grammar mistakes, which is easy to understand for humans. Therefore,
the fourth candidate evidence is considered as a good evidence.

Figure 2: An example of four candidate evidences for the given QA pair. The
first three are informative or readable or concise. The fourth satisfies all
of the three characteristics (informative and readable and concise), so it’s
considered as a good evidence.

###  II-B Metrics of a Good Evidence

Next, we quantify the three above-mentioned characteristics in order to
measure the goodness of an evidence.

####  II-B1 Informativeness

The first important characteristic of an evidence is informativeness. We
resort to a QA model, such as a large-scale pre-trained language model
(denoted as $PLM$), to evaluate the informativeness of an evidence.

######  Principle 1.

An evidence is informative if a given answer can be predicted from this
evidence by a QA model.

The evidence is more informative if the input answer is predicted more
accurately from the evidence by the QA model. $PLM$ is a statistical model
that has already been trained by mountains of data, and has been widely
acknowledged as the foundation of variant NLP tasks, such as reading
comprehension, dialog, etc. In this paper, we use large-RoBERTa [9] as $PLM$,
which can be easily replaced with other pre-trained language models (such as
BERT [22], etc) without significant influence on the results. The detailed
process is as follows:

  * •

Step 1 (_QA model training_): In this step, we train a QA model based on
$PLM$. We first separate the question $q_{i}$ and the context $c_{i}$ by a
special separator [SEP], then divide the context $c_{i}$ into several segments
with a sliding window to keep the most informative context segment. Next, we
use "padding" to make the length of the input sequence the same. After that,
based on the encoded question and context, we train a QA model aiming at
minimizing the training loss.

  * •

Step 2 (_Answer prediction_): In this step, we input the question $q_{i}$ and
the evidence $e_{i}$ into the QA model trained in the above step to predict a
new answer $\hat{a_{i}}$. Specifically, answer prediction is based on the
maximum probability calculated by a Softmax function, and tokens of the
highest two probabilities are taken as the start and the end of $\hat{a_{i}}$.

  * •

Step 3 (_Evidence informativeness evaluation_): In this step, we use the
predicted answer and the input answer to evaluate the informativeness of the
evidence. Inspired by Schuff et al.[43], we use the overlap between the
predicted answer $\hat{a_{i}}$ and the input answer $a_{i}$ to evaluate the
informativeness of the evidence $e_{i}$. Specifically, we use F1 score, which
is widely used to evaluate the accuracy of answer prediction in machine
reading comprehension [42], as the informativeness score $I(e_{i})$ of the
evidence $e_{i}$. The procedure is shown in Eq. 1:

$\displaystyle\small\begin{aligned}
&Pre(e_{i})=\frac{N_{c}(\hat{a_{i}},a_{i})}{L(\hat{a_{i}})},Rec(e_{i})=\frac{N_{c}(\hat{a_{i}},a_{i})}{L(a_{i})}\\\
&I(e_{i})=\frac{2\times Pre(e_{i})\times
Rec(e_{i})}{Pre(e_{i})+Rec(e_{i})}\end{aligned}$ (1)

where $N_{C}$ means the number of common tokens in two sequences, and $L$
means the length of a sequence.

####  II-B2 Conciseness

The second important characteristic of an evidence is conciseness. We directly
use the reciprocal of the length of the evidence to measure its conciseness.

######  Principle 2.

An evidence is concise if it is as short as possible but at least longer than
the answer.

We introduce a conciseness score defined in Eq. 2 to quantify the conciseness
of an evidence. The longer of the evidence, the lower conciseness score of it.
If the length of the evidence is no longer than the length of the answer, we
discard this evidence.

$C(e_{i})=\left\\{\begin{array}[]{rcl}\frac{1}{L(e_{i})}&&{L(e_{i})>L(a_{i})}\\\
-\infty&&{L(e_{i})\leq L(a_{i})}\end{array}\right.$ (2)

####  II-B3 Readability

The third important characteristic of an evidence is readability.
Specifically, we use the reciprocal of the perplexity as the readability
score. The perplexity of an evidence is based on the QA model trained in the
informativeness evaluation. The higher the readability score is, the better
readability of the evidence. The calculating process of the readability score
is shown in Eq. 3:

$\displaystyle PPL(e_{i})$
$\displaystyle=P({(e_{i})}_{1},{(e_{i})}_{2},\cdots,{(e_{i})}_{L})^{\frac{1}{L}}$
(3)
$\displaystyle=\sqrt[L]{\prod_{j=1}^{L}\frac{1}{p({(e_{i})}_{j}|{(e_{i})}_{1},{(e_{i})}_{2},\cdots,{(e_{i})}_{j-1})}}$
$\displaystyle\small R(e_{i})=\frac{1}{PPL(e_{i})}$ (4)

where $P(e_{i})$ and $PPL(e_{i})$ are the probability of generating the
evidence $e_{i}$ by the QA model and the perplexity of the evidence $e_{i}$,
respectively.

####  II-B4 Hybrid score

For the sake of balancing informativeness, conciseness and readability, we set
three weight factors $\alpha$, $\beta$ and $\gamma$ to derive a Hybrid score
$H(e_{i})$ for the evidence $e_{i}$:

$\displaystyle\small H(e_{i})=\alpha I(e_{i})+\beta R(e_{i})+\gamma C(e_{i})$
(5)

where $\alpha>0$, $\beta>0$ and $\gamma>0$, $\alpha+\beta+\gamma=1$. The
detailed value of $\alpha$, $\beta$ and $\gamma$ will be determined by
experiments. $H(e_{i})=0$ means $e_{i}$ is a worst evidence while $H(e_{i})=1$
means $e_{i}$ is a best evidence.

##  III Methods

In this section, we introduce the Grow-and-Clip Evidence Distillation (GCED)
framework and elaborate its major modules.

###  III-A Algorithm Framework

The overall framework of GCED is displayed in Fig. 3. It consists of five core
modules. Answer-oriented Sentences Extractor extracts sentences that are
semantically related to a QA pair from the context. The sentences are referred
as answer-oriented sentence(s). Question-relevant Words Selector selects
question-relevant clue words in the answer-oriented sentences. Weighted
Syntactic Parsing Tree Constructor adopts Lexicalized Probabilistic Context-
Free Grammars (L-PCFGs) and multi-head attention to construct a weighted
syntactic parsing tree for the answer-oriented sentences. Evidence Forest
Constructor adopts dependency parser to discover evidence segments from the
weighted syntactic parsing tree. Optimal Evidence Distiller adopts Grow-and-
Clip strategy based on the hybrid scores and attention weights to distill a
good evidence.

Figure 3: Framework of proposed GCED with five modules: the Answer-oriented
Sentences Extractor, the Query-based Words Selector, the Weighted Syntactic
Parsing Tree Constructor, the Evidence Forest Constructor and the Optimal
Evidence Distiller.

###  III-B Answer-oriented Sentences Extractor (ASE)

Answer-oriented sentence(s) are the minimum sentence subset that has enough
information to predict a given answer. The minimum sentence subset may contain
one or several sentences. The input of this module is the QA pair and one or
more sentences in the context, and the output is the answer-oriented
sentence(s). We use a QA model to select sentences which are able to predict
(that is most semantically related one) given answer to obtain the answer-
oriented sentence(s). Specifically, we first feed the sentences in the context
one at a time into the QA model to make answer prediction. After that, the QA
model stops accepting sentences when it predicts the input answer for the
first time. Sentences so far are regarded as the minimum sentence subset that
covers enough information, and are expected to predict the input answer. These
sentences are called the answer-oriented sentence(s). If the QA model fails to
predict the input answer after processing all sentences, we calculate the
overlap between the input answer $a_{i}$ and each predicted answer
$\tilde{a_{i}}$ by Eq. 1. After that, the sentence subset with the maximum
overlap is taken as the answer-oriented sentence(s). The detailed process is
shown in Fig. 4.

Figure 4: Framework of ASE. It represents the answers predicted by different
sentence subsets. Assume there are $N$ sentences in a context, we select
sentence 2 and sentence $N$ from it based on a QA model.

###  III-C Question-relevant Words Selector (QWS)

This module select the question-relevant clue words from the answer-oriented
sentence(s). The words in the answer-oriented sentence(s) that are
semantically relevant with the significant words in the question are regarded
as question-relevant clue words. Specifically, we first remove insignificant
words in the question. Insignificant words include all question terms (such as
who, where), auxiliary verbs (such as do, did), functional words (such as
conj, art, prep, pron) and punctuations such as !?,.(). Next, for any
remaining word in the question, if the word and its synonyms, antonyms,
sibling terms sharing the same hypernym (by lookup from WordNet) appear in the
answer-oriented sentence(s), these words are regarded as the question-relevant
clue words [23].

We use the example in Fig. 5) to elaborate the process. The sample question is
"Which NFL team represented the AFC at Super Bowl 50?". From the significant
words (such as ”NFL”), we find question-relevant clue words "Football", "AFC",
"Broncos", "NFC", "Super", and "Bowl" in the answer-oriented sentence(s).

Figure 5: A sample of selecting question-relevant clue words by QWS. The first
line in $Q$ is a question, and the second line is remaining words after
removing insignificant words. $S_{1}$ is the answer-oriented sentence(s).
Words colored in red represent related-word-pairs in $Q$ and in $S_{1}$.
Numbers below words in $Q$ represent their index, and numbers below words in
$S_{1}$ represent the corresponding words in $Q$.

###  III-D Weighted Syntactic Parsing Tree Constructor (WSPTC)

This module establishes a relationship among all words in the answer-oriented
sentence(s) with a tree structure. We first use Lexicalized Probabilistic
Context-Free Grammars (L-PCFGs) $Gr=(M,\sum,R,S)$ [20, 13] to generate a
syntactic parsing tree. Each node in the tree has an index, which represents
the position of this word in the answer-oriented sentence(s). Then we
associate an attention weight to each edge in the syntactic parsing tree. We
use the attention derived from the first-layer encoder of $PLM$ as the
attention weights. Specifically, we first derive embeddings of each word in
the answer-oriented sentence(s) from $PLM$, denoted as $\vec{x^{i}}$. Then, we
adopt linear transformation to obtain representation of queries $Q$, keys $K$
and values $V$ shown in Eq. 6. After that, we make scaled dot-product
attention for $heads=16$ times and calculate self-attention score of each
attention head for each vector pair. We make scale by $d_{k}=64$ and normalize
with softmax for $Q$ and $K$ shown in Eq. 7. Finally, we concatenate outputs
of all attention heads to obtain the attention weights $W^{Attn}_{i}$ shown in
Eq. 8. $\bm{W_{t}^{Q}}$, $\bm{W_{t}^{K}}$, $\bm{W_{t}^{V}}$, $\bm{W_{o}}$ are
all trainable parameters.

After that, we annotate the attention weights for all edges of the syntactic
parsing tree to build a weighted syntactic parsing tree shown in Fig. 6 (a).
Higher weights between two nodes represent more attention from a node to its
child node in the answer-oriented sentence(s).

$\displaystyle\small
Q=\bm{W_{q}}\vec{x^{i}},K=\bm{W_{k}}\vec{x^{i}},V=\bm{W_{v}}\vec{x^{i}}$ (6)
$\displaystyle\small\begin{aligned}
&Q_{t}^{i}=Q\bm{W_{t}^{Q}},K_{t}^{i}=K\bm{W_{t}^{K}},V_{t}^{i}=V\bm{W_{t}^{V}},1\leq
t\leq 16\\\
&head_{t}^{i}=Attention(Q_{t}^{i},K_{t}^{i},V_{t}^{i})=softmax(\frac{{Q_{t}^{i}K_{t}^{i}}^{T}}{\sqrt{d_{k}}})V_{t}^{i}\end{aligned}$
(7) $\displaystyle\small\begin{aligned}
\bm{W^{Attn}_{i}}&=MultiHead(Q^{i},K^{i},V^{i})\\\
&=Concat(head_{1}^{i},\cdots,head_{T}^{i})\bm{W_{o}}\end{aligned}$ (8)

###  III-E Evidence Forest Constructor (EFC)

In this part, we use question-relevant clue words, the input answer and the
weighted syntactic parsing tree ($\mathcal{T}$) to construct the evidence
forest ($\mathcal{T}^{E}$). We find the question-relevant clue words and
answer words from $\mathcal{T}$, and the subtrees induced from these words as
well as their parents in $\mathcal{T}$ constitute $\mathcal{T}^{E}$. For
example, in Fig. 6 (b), nodes 3, 5, 7 are question-relevant clue words, nodes
13 and 15 are answer words. There parents as well as these nodes constitute
two evidence trees (tree with nodes 2, 3, and tree with nodes 5, 6, 7) and one
answer tree (tree with nodes 13, 14 and 15). These trees compose of the final
evidence forest.

###  III-F Optimal Evidence Distiller (OEC)

Finally, we distill the optimal evidence from the evidence forest. The
evidence forest is still not necessarily to be informative or readable. OEC is
designed to address this issue, that is keeping a balance between
informativeness. conciseness and readability of the evidences. The input of
this module is $\mathcal{T}$ and the evidence forest $\mathcal{F}^{E}$, and
the output is the optimal distilled evidence which is the final output
evidence. OEC adopts a Grow-and-Clip strategy to distill the optimal evidence
based on hybrid scores and attention weights. The Grow-and-Clip strategy
searches for a shortest path in $\mathcal{T}$ to connect evidence trees and
the answer trees in the evidence forest (introduced in Sec. III-E). It
consists of two fundamental search operations: Sequential Grow Searching (SGS)
and Sequential Clip Searching (SCS), as illustrated in Fig. 6(c), (d). The
algorithm of Grow-and-Clip search strategy is illustrated in Algorithm 1.

Figure 6: Subfigure (a) to (d) represent the process of the Grow-and-Clip
strategy. We use the weighted syntactic parsing tree and the evidence forest
to make SGS and SCS so as to distill the optimal evidence. Subfigure (e)
displays a detailed process of distilling the optimal evidence by GCED.

####  III-F1 Sequential Growth Searching (SGS)

We use SGS to connect all trees in the evidence forest ($\mathcal{F}^{E}$) to
obtain an informative and readable evidence (Fig. 6 (c)). SGS achieves this
goal by a step-wise growth of a carefully selected optimal tree in
$\mathcal{F}^{E}$. In each iterative step, a parent node that has the maximal
weight with the roots in the current evidence forest is selected to expand. In
this way, we ensure the final evidence evidence is informative and readable.
The detailed procedure of SGS is as follows and illustrated in Alg. 1 and
Example 1.

  * •

Step 1 (Select an optimal tree): We first select the tree whose root has a
maximum weight $W^{attn}$ with its parent node from the evidence forest
$\mathcal{F}^{E}$ (shown in line 3 of Grow Step in Alg. 1). This tree is
denoted as $\mathcal{T}_{opt}$. $W^{attn}$ is calculated in Sec. III-D. A
higher attention weight means more dependence between the root node and its
parent node.

  * •

Step 2 (Update the evidence forest): Next, we merge $\mathcal{T}_{opt}$ with
its parent node and sibling subtrees to replace $\mathcal{T}_{opt}$ (shown in
line 4 of Grow Step in Alg. 1). Note that this step will update the evidence
forest. This step ensures the readability as much as possible since the new
$\mathcal{T}_{opt}$ only expands to nodes that are closely-related to the old
$\mathcal{T}_{opt}$ (i.e., parent node and sibling nodes of the old root).

  * •

Step 3 (Construct an unclipped evidence tree): We perform Step 1 and 2
repeatedly on the newly updated $\mathcal{F}^{E}$ until all trees
$\mathcal{F}^{E}$ are connected. Then an unclipped evidence tree
$\mathcal{T}^{E}$ is constructed (shown in line 7 of Grow Step in Alg. 1).
This unclipped evidence tree is an informative and readable evidence after
ranking its nodes by indexes.

######  Example 1 (SGS).

In this example (Fig. 6 (e)), there are two evidence trees ($T_{1}$ and
$T_{2}$), and one answer tree ($T_{3}$). The question-relevant clue words are
node 3, 6, 14, 17, 29, 30 (colored in Tiffany blue) and the input answer words
are node 9,10 (colored in Sakura pink).

  * •

We first select an optimal tree whose root has a maximum attention weight with
its parent node. There are three roots nodes in the evidence forest:
"4-Conference", "15-Conference" and "31-title". Their attention weights with
their corresponding parent nodes are 0.2767, 0.2037 and 0.3230, respectively.
Therefore, $T_{3}$ rooted at node 31 is selected as $\mathcal{T}_{opt}$ (shown
in Grow Step line 3 in Alg. 1).

  * •

Then, we select the parent of node 31 to expand. That is "26-earn". As a
result $\mathcal{T}_{opt}$ is updated to be the subtree rooted at node 26. The
new $\mathcal{T}_{opt}$ as well as $T_{1},T_{2}$ is new evidence forest (shown
in Grow Step line 4 in Alg. 1).

  * •

Next, we repeat the above-mentioned steps until all trees in the evidence
forest are connected. Finally, we construct a unclipped evidence tree whose
root is "11-defeated" (shown in Grow Step line 7 in Alg. 1).

####  III-F2 Sequential Clip Searching (SCS)

We use SCS to remove redundant subtrees in the unclipped evidence tree to
obtain a concise but still readable evidence (Fig. 6 (d)). SCS uses the hybrid
scores (shown in Eq. 5) to select the subtrees of the highest priority to
prune, keeping the _conciseness_ and _readability_ of the final evidence. The
process of SCS strategy is as follows and the algorithm is shown in Alg. 1
Clip Step:

  * •

Step 1 (Select a candidate subtree to clip): We first select a candidate
subtrees $S$ in the unclipped evidence tree $\mathcal{T}^{E}$. Any subtree $S$
of $\mathcal{T}^{E}$ that does not appear in the evidence forest
$\mathcal{F}^{E}$ is a valid candidate. This ensures that any question-
relevant clue words and answer words will not be clipped, guaranteeing the
informativeness of the final evidence (shown in line 3 of Clip Step in Alg.
1).

  * •

Step 2 (Clip the worst candidate subtree): If the selected candidate substree
whose removal achieves the maximum hybrid score ($H_{max}$) of
$\mathcal{T}^{E}$, it is pruned. Note that the entire subtree including all
the children nodes and descendent nodes are deleted to guarantee the
readability of the evidence. If there are more than one candidate subtrees
achieving the highest hybrid score, the one whose root has a minimum weight
($W^{Attn}_{min}$) with its parent, will be selected to be pruned. This step
is illustrated in line 4-7 of Clip Step in Alg. 1.

  * •

Step 3 (Construct a clipped evidence tree): Step 2 is repeated until $M$
times, which is a hyperparameter which is tuned by experiments. When the
repeating procedure ends, a clipped evidence tree $\mathcal{T}^{E}$ is
constructed.

######  Example 2 (SCS).

Continue Example 1. We drive a clipped evidence tree $\mathcal{T}^{E}$ from
the result of SGS. This clipped evidence tree is an informative-yet-concise
and human-readable evidence after ranking its nodes by indexes.

  * •

There are four candidate clipped subtrees $\mathcal{S}$ (subtrees rooted at
node 32, 20, 22, 25, respectively). The hybrid scores of $\mathcal{T}^{E}$
after removing each subtrees are 0.6439, 0.6572, 0.7006 and 0.6145,
respectively. Only the subtree rooted at 22 achieves the maximum score.

  * •

Next, we clip the subtree $\mathcal{S}_{wor}$ rooted at 22. All the descendant
nodes including nodes 23, 24 are removed from $\mathcal{T}^{E}$. After that,
we repeat the above steps to clip other subtrees in the same way (colored in
light gray). In this example, the clip time $M$ is set to 1 according to
experiments.

After finishing Grow-and-Clip, we rearrange nodes in the final clipped
evidence tree $\mathcal{T}^{E}$ in terms of the indexes of nodes to obtain the
optimal evidence: "Football Conference (AFC) champion Denver Broncos defeated
Football Conference (NFC) champion Carolina Panthers to earn Super Bowl
title.", which is an informative-yet-concise and human-readable evidence.

Input : Evidence Forest
$\mathcal{F}^{E}=\\{\mathcal{T}_{1},\cdots,\mathcal{T}_{N}\\}$,
Weighted Syntactic Parsing Tree $\mathcal{T}$,
Times of clip $M$.

Output : Evidence Tree $\mathcal{T}^{E}$.

Grow Step:

1: $N\leftarrow|\mathcal{F}^{E}|$

// $N$ is the number of trees in $\mathcal{F}^{E}$

2: while $N>1$ do

3: $\mathcal{T}_{opt}\leftarrow
argmax_{\mathcal{T}_{i}\in\mathcal{F}^{E}}W^{attn}_{r}(\mathcal{F}^{E},\mathcal{T})$

// $\mathcal{T}_{opt}$ is the tree whose root has a maximum weight
$W^{attn}_{r}$ with its parent in $\mathcal{T}$

4: $\mathcal{T}_{opt}\leftarrow\text{ merge }\mathcal{T}_{opt}\text{ with
parent node and sibling subtrees}$

5: $N\leftarrow|\mathcal{F}^{E}|$

// re-calculate the number of trees in $\mathcal{F}^{E}$

6: end while

7: $\mathcal{T}^{E}\leftarrow\mathcal{T}_{opt}$

// now we get the unclipped evidence tree

Clip Step:

1: $i\leftarrow M$

// how many times we can clip

2: while $i>0$ do

3: $\mathcal{S}\leftarrow subtrees(\mathcal{T}^{E})-trees(\mathcal{F}^{E})$

// $\mathcal{S}$ doesn’t include any trees in $\mathcal{F}^{E}$

4: $\mathcal{S}_{wor}\leftarrow
argmax_{\mathcal{S}_{i}\in\mathcal{S}}(H(\mathcal{S}_{i},\mathcal{T}))$

// $\mathcal{S}_{wor}$ is the subtree of maximum hybrid score $H$

5: $\mathcal{T}^{E}\leftarrow\mathcal{T}^{E}-\mathcal{S}_{wor}$

// clip the redundant evidence tree

6: $i\leftarrow i-1$

7: end while

8: Output $\mathcal{T}^{E}$

// now we get the clipped evidence tree

Algorithm 1 The Grow-and-Clip strategy

##  IV Experiments

In this section, we carry out experiments on two reading comprehension
datasets SQuAD and TriviaQA. We adopt nine fine-tuned QA models on SQuAD and
another nine fine-tuned QA models on TriviaQA to predict answers. Then we take
these predicted answers and also the ground-truth answers as the input answers
to distill evidences, respectively. Next, we propose a human evaluation
protocol to access informativeness, conciseness and readability of the
distilled evidences on these datasets. Moreover, we use the distilled
evidences as the contexts to investigate whether an off-the-shelf QA model
could benefit from our distilled evidence. In addition, we carry out ablation
study to study the effect of each component of our GCED algorithm. We also
showcase the effectiveness of distilled evidence through case studies and
conduct error analysis to explore the limits of our approach.

###  IV-A Experimental Setup

####  IV-A1 Human evaluation

We conduct human evaluation to assess informativeness, readability and
conciseness of the distilled evidences. Specifically, we design a scoresheet
with five scales (1-5 scales, 5 is the best) to evaluate the three
characteristics, as shown in Tab. I.

TABLE I: A human evaluation scoresheet for the distilled evidences based on informativeness, conciseness and readability. Criteria | (Scores) Contents
---|---
Informativeness |  | (5) Extremely related to the QA pair, and the input answer can be completely inferred from the evidence.
---
(4) Generally related to the QA pair, and the input answer can be partly
inferred from the evidence.
(3) Generally related to the QA pairs, but the input answer can’t be inferred
from the evidence.
(2) Only some details between the evidence and the QA pair are identical, and
the input answer can’t be inferred from the evidence.
(1) The evidence is irrelevant with the QA pairs.
Conciseness |  | (5) Extremely concise.
---
(4) Generally concise. (1-1.5 longer than the expected evidence).
(3) Containing some redundant information. (1.5-2 longer than the expected
evidence)
(2) Containing too much redundant information. (2-3 longer than the expected
evidence)
(1) The evidence is the whole document. ($>$3 longer than the expected
evidence)
Readability |  | (5) Extremely fluent and logical.
---
(4) Can be understood with a few grammar mistakes (1-2).
(3) Can be understood by some extent, but with many grammar mistakes ($>$2).
(2) Can not be understood, but some segments are fluent.
(1) Not readable.

The procedure of human evaluation is as follows: We first enroll 9 graduate
student volunteers (not including authors) as human raters to rate the
distilled evidences according to the criterion specified in Tab. I. They are
kindly to offer their help without being compensated in any form. We divide
them into three groups, and raters in each group will be distributed with the
same evidences. We randomly select 3,000 QA pairs per QA model per dataset for
them to evaluate. Next, we calculate Inter-rater agreement of Krippendorff’s
Alpha (IRA) shown in Tab. II to evaluate the rating quality. Some
controversial evidences which have low agreements ($<$0.7) are discarded.
Finally we average human evaluation of each group to get the final quality
score of the distilled evidences. We set the weight factor of informativeness,
conciseness and readability as the same.

TABLE II: Inter-rater agreement of Krippendorff’s Alpha in human evaluation. Criteria |  Group 1 |  Group 2 |  Group 3
---|---|---|---
Informativeness |  0.77 |  0.81 |  0.76
Conciseness |  0.83 |  0.80 |  0.75
Readability |  0.82 |  0.77 |  0.81
Hybrid Score |  0.81 |  0.79 |  0.78

####  IV-A2 QA models performance

We carry out experiments to observe the effect of distilled evidences for
improving the performance of QA models. we carry out experiments on GeForce
RTX 3090 GPU. For some QA models with too many parameters, we use TPU on
Google Colab. We use Stanfordcorenlp and nltk to do syntactic parsing, and use
Pytorch to train QA models. We set the maximum length of the input sequence as
384 with 64 for the question, 30 for the answer, 100 for the evidence, and 128
for the sliding windows in the context. Moreover, we use Adam optimizer with
default parameters, initialize the learning rate and batch size to 5e-5 and 8,
respectively, for 3 epochs.

###  IV-B Baselines and Metrics

####  IV-B1 Baselines

We select nine fine-tuned QA models on SQuAD, including BERT [22], RoBERTa
[9], SpanBERT [26], ALBERT [29], XLNet [53], ELECTRA [19], LUKE [50], T5 [5]
and DeBERTa [4]. And select another nine fine-tuned QA models on TriviaQA,
including BERT+BM25 [22, 8], GraphRetriever [6], Longformer [7], BIGBIRD [3],
RAG [30], PA+PDR [15, 16] and Hard-EM [34], to predict answers in two
situations:

  * •

The input of QA models are the questions and the contexts, and the output is
the answers predicted by QA models. These predicted answers are used to
distill predicted-answer-based evidences.

  * •

The input of QA models is the questions and the distilled evidences based on
given answers, and the output is the answers predicted by the QA models. These
predicted answers are used to evaluate the performance of QA models.

####  IV-B2 Metrics

The metrics includes human evaluation and machine evaluation shown as follows:

  * •

Human evaluation. We adopt human evaluation, as introduced in Sec. IV-A1, to
assess the quality (informativeness, conciseness, readability) of the
distilled evidences.

  * •

Machine evaluation. We use EM (Exact Match) and F1 (F1 score) to evaluate the
performance of QA models after using the distilled evidences as the contexts.
EM and F1 are same as the metrics introduced in [42, 41], which measures the
percentage of predictions that match any one of the ground-truth answers
exactly and the average overlap between the predicted answer and the ground-
truth answer, respectively.

###  IV-C Datasets

We adopt two popular reading comprehension datasets, SQuAD and TriviaQA, for
the evaluation. The statistics of datasets are shown in Tab. III. Each dataset
is divided into training set (for QA models training) and development set (for
QA models evaluation).

SQuAD [42, 41] have two versions: 1.1 and 2.0, which are both derived from
Wikipedia articles. SQuAD-1.1 consists of 107,785 question-answer pairs
composed from 536 articles by crowdworkers. SQuAD-2.0 combines the existing
questions in SQuAD-1.1 with 53,775 new unanswerable questions about the same
paragraphs. The answer to every question in SQuAD-1.1 and 2.0 is a segment of
text or a span from the corresponding paragraph.

TriviaQA [27] contains over 650K question-answer-evidence triples. After
removing the triples missing the correct answer, there remains 95K useful
question-answer pairs. The dataset is constructed from Web search results and
Wikipedia articles. Hence, we have two versions of datasets: TriviaQA-Web and
TriviaQA-Wiki.

TABLE III: Statistics of two datasets, SQuAD and TriviaQA.  Dataset |  Train |  Dev
---|---|---
SQuAD-1.1 |  87599 |  10570
SQuAD-2.0 |  130319 |  6078
TriviaQA-Wikipedia |  110647 |  14229
TriviaQA-Web |  100000 |  68621

###  IV-D Main results

Next, we present the main results of our experimental study.

####  IV-D1 Results of human evaluation

The results of human evaluation for distilled evidences are shown in Tab. IV
(on SQuAD) and Tab. V (on TriviaQA). We make human evaluation for both
predicted-answer-based evidences (row 1-9 in Tab. IV and Tab. V) and ground-
truth-answer-based evidences (the last row in Tab. IV and Tab. V).

The human evaluation reveals that for both predicted-answer-based evidences
and ground-truth-based evidences, the quality scores (informativeness,
conciseness, readability as well as the hybrid scores) of distilled evidences
are consistently larger than 0.75 across all baseline QA models and datasets.
It suggests that the quality of automatically distilled evidences is
satisfying, which is independent on different QA models. We also find that on
average 78.5% and 87.2% words have been reduced in the distilled evidences on
SQuAD and TriviaQA datasets, respectively. This further justifies the
conciseness of the evidences. We highlight that there is no significant
difference between human evaluation for predicted-answer-based evidences and
ground-truth-based evidences (the p-value is $>0.5$). The reason is that the
distilled evidences aim at explaining/supporting the input answers whatever
the input answers are predicted by QA models or labeled as ground-truth
answers by crowdworkers. The correctness of the answers do not affect the
quality of the distilled evidences, which is friendly for humans to understand
the source of the input answers.

TABLE IV: Human evaluation for predicted-answer-based evidences and ground-truth-answer-based evidences on SQuAD-1.1 and SQuAD-2.0. The predicted-answer-based evidences are distilled from the answers predicted by the QA models listed in the first column.  Datasets | SQuAD-1.1 | SQuAD-2.0
---|---|---
Source | I | C | R | H | I | C | R | H
BERT-large | 0.87 | 0.82 | 0.83 | 0.84 | 0.88 | 0.81 | 0.87 | 0.85
RoBERTa-500K | 0.86 | 0.89 | 0.84 | 0.86 | 0.85 | 0.90 | 0.88 | 0.88
SpanBERT | 0.88 | 0.86 | 0.87 | 0.87 | 0.85 | 0.82 | 0.84 | 0.84
ALBERT | 0.88 | 0.83 | 0.86 | 0.86 | 0.89 | 0.84 | 0.83 | 0.86
XLNet-large | 0.90 | 0.87 | 0.88 | 0.88 | 0.90 | 0.89 | 0.89 | 0.89
ELECTRA-1.75M | 0.91 | 0.87 | 0.85 | 0.88 | 0.91 | 0.85 | 0.88 | 0.88
LUKE | 0.86 | 0.83 | 0.85 | 0.85 | 0.90 | 0.86 | 0.89 | 0.88
T5 | 0.89 | 0.87 | 0.86 | 0.87 | 0.92 | 0.89 | 0.88 | 0.90
DeBERTa-large | 0.90 | 0.86 | 0.82 | 0.86 | 0.89 | 0.88 | 0.90 | 0.89
Ground-truth | 0.90 | 0.89 | 0.88 | 0.89 | 0.91 | 0.90 | 0.89 | 0.90
TABLE V:  Human evaluation for predicted-answer-based evidences and ground-truth-based evidences on TriviaQA datasets. The predicted-answer-based evidences are distilled based on the answers predicted by the following QA models.  Datasets | TriviaQA-Web | TriviaQA-Wiki
---|---|---
Source | I | C | R | H | I | C | R | H
BERT+BM25 | 0.82 | 0.81 | 0.80 | 0.81 | 0.83 | 0.82 | 0.81 | 0.82
GraphRetriever | 0.81 | 0.82 | 0.77 | 0.80 | 0.77 | 0.81 | 0.76 | 0.78
RoBERTa-base | 0.83 | 0.85 | 0.80 | 0.83 | 0.82 | 0.78 | 0.80 | 0.80
Longformer-base | 0.81 | 0.77 | 0.79 | 0.79 | 0.79 | 0.76 | 0.76 | 0.77
Bigbird-itc | 0.77 | 0.80 | 0.77 | 0.78 | 0.83 | 0.76 | 0.78 | 0.79
ELECTRA-base | 0.85 | 0.86 | 0.83 | 0.84 | 0.81 | 0.82 | 0.80 | 0.81
RAG-Sequence | 0.83 | 0.78 | 0.79 | 0.80 | 0.80 | 0.84 | 0.82 | 0.82
PA+PDR | 0.80 | 0.84 | 0.82 | 0.82 | 0.80 | 0.82 | 0.80 | 0.80
Hard-EM | 0.85 | 0.86 | 0.81 | 0.83 | 0.83 | 0.80 | 0.81 | 0.81
Ground-truth | 0.85 | 0.86 | 0.84 | 0.85 | 0.83 | 0.84 | 0.82 | 0.83

####  IV-D2 Performance gain of QA models augmented by ground-truth-answer-
based evidences

In an ideal setting or some applications (such as searching engine), we have
ground-truth answers, which are ideal sources to distill evidences. These
evidences could also be seen as concise-yet-informative contexts for a QA
model to find the correct answers. If the evidence is more concise than the
original context, and it contains essential information to answer a question,
then using the evidence as the input of a QA model (instead of the raw
context) shall improve its performance. Hence, we propose an experiment to
test the performance gain of QA modes which are augmented by evidence
distilled by our solution from the ground-truth-answers.

The results are shown in Tab. VI (SQuAD) and Tab. VII (TriviaQA). Results of
baselines are from their published papers except those with asterisks in the
upper right corner, which are missing in the corresponding papers. We retrain
these baselines with asterisks based on the same settings on our own machine
and report their results. We find that it performs consistently better for all
QA models when the contexts are replaced with the distilled evidences on all
tested datasets, which supports our conjecture.

####  IV-D3 Performance degradation of QA models augmented by predicted-
answer-based evidences

In a more realistic setting, we have no ground-truth answers, where we need to
predict the answer by a vanilla QA model. However, the evidence distilled from
a wrong predicted answer might be noisy for the question answering. Using such
evidences as the QA contexts is likely to produce wrong answers. Hence, QA
models augmented by predicted-answer-based evidences are expected to degrade
in their performance. Hopefully, the performance degradation is minor or
acceptable if our evidence distillation solution is effective to finding
effective information from the answer. This inspires our interest to test the
performance degradation of QA models which are augmented by the evidences
distilled by our solution from the predicted answers.

We randomly substitute $\delta$ percentage of ground-truth answers with
predicted answers with $\delta$ ranging in 0.2, 0.5, 0.8, 1, respectively, to
distill evidences. It means some distilled evidences are based on the ground-
truth answers and others are based on the predicted answers. We then
substitute the contexts with these evidences to retrain QA models. The results
are shown in Fig. 7. It’s obvious to find that the performance of QA models
fed with the predicted-answer-based evidences as the contexts indeed degrade
compared with that of the vanilla QA models, which confirms our conjecture.
However, for many QA models, even all the evidences come from predicted
answer, only 2-3% performance drop can be observed on SQuAD datasets. This
result suggests our solution has minor side effects but additionally generates
answering evidence, which are demanded in many real applications. More
performance degradation can be observed on TriviaQA dataset. Because this
dataset is more open and in general is more challenging, and most State-Of-
The-Art QA models have less than 80% performance (EM & F1).

This could be an interesting research topic in the future work. Whatever the
answer is correct or not, the corresponding evidence provides an informative-
yet-concise summarization of the context that explains how this answer was
predicted. In this way, a user can quickly know the information source of this
answer, leading to an explainable and reliable QA system. For example, given a
question "Where was Albert Einstein born?", if a QA system gives the answer
"Berlin" and the supporting evidence "Albert Einstein moved to Berlin 50 years
old.", the user will find this QA system is not reliable and he will not trust
this answer.

Figure 7: The performance degradation of QA models augmented by predicted-answer-based evidences (a: SQuAD-1.1, b: SQuAD-2.0, c: TriviaQA-Web and d: TriviaQA-Wiki). gt: use ground-truth answers to distill evidences; pred: use predicted answers to distill evidences; predx: substitute x% ground-truth answers with predicted answers to distill evidences. TABLE VI: Comparisons of nine baselines and their evidence-augmented variants on SQuAD-1.1 and SQuAD-2.0 datasets. The evidences are distilled from the ground-truth answers.  Datasets | SQuAD-1.1 | SQuAD-2.0
---|---|---
Models |  EM |  F1 |  EM |  F1
BERT-large |  84.1 |  90.9 |  79.0 |  81.8
+GCED |  88.1 |  92.3 |  85.0 |  90.9
RoBERTa-500K |  88.9 |  94.6 |  86.5 |  89.4
+GCED |  91.5 |  95.8 |  88.7 |  92.3
SpanBERT |  88.8 |  94.6 |  85.7 |  88.7
+GCED |  91.2 |  96.1 |  89.2 |  92.9
ALBERT |  89.3 |  94.8 |  87.4 |  90.2
+GCED |  92.0 |  96.1 |  90.6 |  93.1
XLNet-large |  89.7 |  95.1 |  87.9 |  90.6
+GCED |  92.8 |  96.2 |  90.5 |  93.5
ELECTRA-1.75M |  89.7 |  94.9 |  88.0 |  90.6
+GCED |  93.0 |  95.9 |  91.6 |  93.9
LUKE |  89.8 |  95 |  87.9* |  90.5*
+GCED |  92.8 |  96.7 |  91.4 |  93.4
T5 |  90.1 |  95.6 |  88.2* |  90.8*
+GCED |  93.7 |  97.0 |  91.8 |  94.0
DeBERTa-large |  90.1 |  95.5 |  88.0 |  90.7
+GCED |  93.1 |  97.1 |  91.0 |  93.0
+GCED (average) |  $\uparrow$3.5% |  $\uparrow$1.5% |  $\uparrow$4.1% |  $\uparrow$4.2%
TABLE VII: Comparisons of nine baselines and their evidence-augmented variants on TriviaQA datasets. The evidences are distilled from the ground-truth answers. Datasets | TriviaQA-Web | TriviaQA-Wiki
---|---|---
Models |  EM |  F1 |  EM |  F1
BERT+BM25 |  47.2 |  56.1* |  46.4* |  54.7*
+GCED |  63.8 |  70.5 |  62.1 |  69.0
GraphRetriever |  55.8 |  64.3* |  54.9* |  63.4*
+GCED |  69.3 |  75.5 |  68.2 |  73.9
RoBERTa-base |  69.7* |  76.8* |  67.6* |  74.3
+GCED |  80.4 |  84.8 |  78.4 |  82.1
Longformer-base |  74.6* |  78.6* |  72.0* |  75.2
+GCED |  82.1 |  86.4 |  79.8 |  83.0
Bigbird-itc |  77.6* |  81.8* |  75.7* |  79.5
+GCED |  85.1 |  90.4 |  84.3 |  89.2
ELECTRA-base |  68.9* |  75.6* |  65.4 |  73.8*
+GCED |  79.4 |  84.6 |  76.8 |  81.7
RAG-Sequence |  58.9* |  62.7* |  55.8 |  61.5*
+GCED |  71.4 |  74.8 |  68.9 |  73.5
PA+PDR |  62.3* |  69.0* |  60.1 |  66.7*
+GCED |  73.0 |  80.1 |  72.5 |  78.9
Hard-EM |  68.5* |  75.8* |  66.9 |  75.3*
+GCED |  80.1 |  83.2 |  78.4 |  83.8
+GCED (average) |  $\uparrow$18.2% |  $\uparrow$14.6% |  $\uparrow$19.3% |  $\uparrow$15.0%

###  IV-E Ablation Study

In this part, we conduct ablation study to further analyze each component of
GCED. Specifically, we use BERT which is augmented by the ground-truth-based
evidences on SQuAD-2.0 to make analysis. The results are shown in Tab. VIII.

####  IV-E1 Ablation Study on human evaluation

First, when we remove QWS and the informativeness score (w/o I), respectively,
we observe that the human informativeness scores (I) decrease. It indicates
that question-relevant clue words in QWS and $PLM$ in building informativeness
scores both extract useful information which can explain/support the input
answers. Next, we can find that when we remove ASE (w/o ASE), the clip step
(w/o Clip) from the Grow-and-Clip strategy, and the conciseness score (w/o C),
respectively, the human conciseness scores (C) decrease. It indicates that
each of these three modules can filter out much redundant information which
maintains the conciseness of the distilled evidences. After that, when we
remove grow step (w/o Grow) from the Grow-and-Clip strategy and the
readability score (w/o C) from the hybrid score, respectively, the human
readability scores (C) decrease. It indicates that both of the grow step and
the readability scores can guarantee the readability of the distilled
evidences. Therefore, human evaluation of each component in the proposed GCED
demonstrates that each component and the organic combination of all components
in GCED both have significant effects for distilling informative-yet-concise
and human-readable evidences.

####  IV-E2 Ablation Study on QA models which are augmented by ground-truth-
based evidences

Next, we make ablation Study on QA models which are augmented by ground-truth-
based evidences. We can find that ASE affects most on QA model’s performance.
When it’s removed (w/o ASE), the performance decreases most. This results
demonstrate that the answer-oriented sentence(s) contain suggestive
information in predicting correct answers if we use ground-truth-answer-based
evidences as the contexts to train QA models. Next, we remove QWS (w/o QWS),
and the performance also decreases a lot. The results demonstrate that
question-relevant clue words have a significant semantic relationship with the
ground-truth answers. After that, we remove Grow step (w/o Grow) and Clip step
(w/o Clip) in the Grow-and-Clip strategy. The degrading performance
demonstrates that both of Grow step and Clip step have critical effects in
predicting correct answers. Furthermore, we assess the individual effect of
the informativeness scores (w/o I), the conciseness scores (w/o C), and the
readability scores (w/o R). The results demonstrate all three criteria have
positive effect in predicting correct answers if these evidences are distilled
based on the ground-truth answers.

In brief, the proposed GCED is a powerful algorithm in distilling informative-
yet-concise and human-readable evidences, which can explain/support the input
answers. And if the evidences are distilled based on the ground-truth answers,
these evidences are an informative-yet-concise form of the contexts, which can
improve the performance of QA models besides explaining/supporting the input
answers.

TABLE VIII: The effect of each component of GCED. The evidences are distilled from the ground-truth answers with BERT on SQuAD-2.0. Sources | I | C | R | H | EM | F1
---|---|---|---|---|---|---
w/o ASE | 0.85 | 0.65 | 0.80 | 0.77 | 72.0 | 78.2
w/o QWS | 0.67 | 0.79 | 0.77 | 0.74 | 70.2 | 76.5
w/o Grow | 0.82 | 0.80 | 0.67 | 0.76 | 75.2 | 80.6
w/o Clip | 0.81 | 0.70 | 0.81 | 0.77 | 80.5 | 86.3
w/o I | 0.73 | 0.78 | 0.80 | 0.77 | 80.2 | 87.0
w/o C | 0.80 | 0.72 | 0.76 | 0.76 | 79.3 | 86.9
w/o R | 0.81 | 0.83 | 0.75 | 0.80 | 82.1 | 88.4
BERT+GCED | 0.86 | 0.83 | 0.82 | 0.84 | 85.0 | 90.9

###  IV-F Case Study

In this part, we display a good evidence distilled by the proposed GCED, as
shown in Fig. 8. Given a QA pair and the context which contains four
sentences, $S_{1}$ and $S_{2}$ are selected as the answer-oriented sentences
by ASE at first. Second, we select "Beyoncé", "performed", and "competitions"
in the answer-oriented sentences as the question-relevant clue words by QWS.
Third, we adopt $S_{1}$ and $S_{2}$ to construct a weighted syntactic parsing
tree by WSPTC with calculating attention weights for all edges. Fourth, we
obtain an evidence forest including an answer tree and two evidence trees. The
answer tree contains three nodes "singing", "and", "dancing". One evidence
tree contains five nodes "Beyoncé", "Giselle", "Knowles", "-", and "Carter".
The other evidence tree contains seven nodes "performed", "in", "various",
"singing", "and", "dancing", and "competitions". Fifth, we adopt OEC with the
Grow-and-Clip strategy to grow three nodes including "as", "a", and "child" to
achieve informativeness, and clip one node including "various" to achieve
conciseness. Due to space limitation, we haven’t display the nodes which have
been grown first and then clipped, such as "born", "and", "raised", "in",
"Houston", ",", "Texas", etc. These nodes mainly aim at guarantee readability
during the process of Grow-and-Clip. Finally, we rank the remaining nodes by
their indexes to distill the final optimal evidence, that is "Beyoncé Giselle
Knowles-Carter performed in singing and dancing competitions as a child".

We are satisfied to find that this evidence is informative (contain useful
information), concise (the length is short enough) and readable (without
grammar mistakes or fuzzy logic), demonstrating the proposed GCED can catch
important information to explain/support the input answer, filtering out
noises and is user-friendly.

Figure 8: A good evidence distilled by GCED. Given a QA pair, $S_{1}$ and
$S_{2}$ are selected as the answer-oriented sentences by ASE. Question-
relevant clue words (the highlighted words) are used to construct the evidence
forest (bold and colored in dark blue). We use Grow-and-Clip strategy to grow
words (bold and colored in brown) and clip words (colored in grey), and
connect them to the optimal evidence.

###  IV-G Error analysis

However, there are some unsatisfying evidences distilled by GCED. For example,
given a question "In the Bible, who was the mother of Solomon?", the the
answer predicted by a QA model is "bathsheba". The evidence distilled by GCED
is "Solomon had brothers through Bathsheba, Nathan, Shammua, and Shobab,
brothers through as many mothers.". However, it’s difficult to understand by
humans due to its bad readability . Because GCED doesn’t have knowledge to
know the relationship among child, David, and wife, so it can’t distill a more
human-readable informative-yet-concise evidence "Solomon was the child to
David and his wife Bathsheba.". Moreover, some contexts are comparatively
long, containing many sentences. The structure of most sentences are
complicated with nested clauses and hidden subjects, making it difficult to
extract more suggestive information and filter out redundant noises to distill
informative-yet-concise evidences.

Therefore, we will optimize our proposed GCED through adding world knowledge
and commonsense and improving the understanding on too complicated sentences
in the future work.

##  V Related work

Evidences extraction based on sentence-level for answer explanation. A
succession of research has been conducted to improve the interpretability of
QA. For the extractive and abstractive QA, such as SQuAD [42, 41] and TriviaQA
[27], most answers and their supporting evidences can be directly extracted
from the given contexts based on semantic and syntactic knowledge. The process
of evidences extraction can be summarized as extracting only a few sentences
in the documents to explain/support the answer. Previous studies mainly
extract evidences on sentence level. For example, Min et al. [35] extract the
minimal context based on a sentence selector and a QA model, giving the QA
model a reduced set of sentences with high selection scores in order to
explain/support answers. Yin et al. [55] study interdependent sentence pair
representations with three attention schemes, integrating mutual influence
between sentences into CNNs to explain/support answers on WikiQA [52]. Choi et
al. [17] combine a coarse, fast model for selecting relevant sentences and a
more expensive RNN for explaining/supporting answers and use this model to
maintain or even improve the performance of state-of-the-art QA models
simultaneously. Lin et al. [32] utilize a sentence-level selective attention
to aggregate the information of all sentences to extract relational facts.
They employ convolutional neural networks to embed the semantics of sentences.
Lin et al. [31] employs a paragraph selector to filter out those noisy
paragraphs and a paragraph reader to extract the correct answer from those
denoised paragraphs. Shen et al. [44] propose a distantly supervised open-
domain question answering (DS-QA) system which uses information retrieval
technique to obtain relevant text from Wikipedia as supporting facts for
answers. Wang et al. [48] use reinforcement learning to train target paragraph
selection and answer extraction jointly. They propose a re-ranking-based
framework to make use of the evidence from multiple passages in open-domain
QA, and perform evidence aggregation in existing open-domain QA datasets.
Atanasova et al. [1] generate veracity explanations on available claim
context, and show that veracity prediction can be modeled jointly with
veracity prediction and improves the performance of the veracity system.
However, sentence-level evidences still have redundant noises. To guarantee
continuous information, several whole sentences are selected out as supporting
evidences instead of emphasizing clue words or phrases. Different from them,
our research aims at token-level instead of sentence-level or paragraph-level
to distill informative-yet-concise and human-readable supporting evidences.
Human evaluation indicates that our distilled evidences are of high-quality
which are informative to explain/support the input answers, concise without
redundant noises and achieve better human readability.

Evidences extraction by neural networks for answer explanation. Besides, there
are a series of research make answer-explanations by neural networks on
complex datasets. For example, Schuff et al. [43] propose a hierarchical model
and a new regularization term to strengthen the answer-explanation coupling on
multi-hop HOTPOTQA. Thayaparan et al. [38] introduce a novel approach for
answering and explaining multiple-choice science questions by reasoning on
grounding and abstract inference chains. Wang et al. [47] extract evidence
sentences for multiple-choice MRC based on inference and utilization of prior
knowledge by distant supervision and deep probabilistic logic framework.
Jansen et al. [24] implement a fine-grained characterization of the knowledge
and inference requirements on science exam QA. Tran et al. [46] adopt an
explainable, evidence-based memory network architecture, which learns to
summarize the dataset and extract supporting evidences to make its decision on
TrecQA and WikiQA. However, due to the end-to-end attribute of neural
networks, the supporting evidences generated from neural networks are not
traceable and user-friendly for explaining/supporting given answers. Different
from their works, our solution GCED is built upon a pipelined principle based
on three characteristics heuristically, improving the controllability and
traceablility for the distilled evidences. Moreover, Some research such as
Atanasova et al. [2] aims to develop a comprehensive list of properties to
evaluate exsting explainability techniques (simplification, gradient-based,
etc). Therefore, we could evaluate our GCED on this methods in the future.

##  VI Conclusion and Future work

The interpretability of question answering is a great challenge in NLP, which
is expected to find supporting facts for given QA pairs so as to provide end-
users a better understanding for why a specific prediction is regarded as the
answer to that question. To automate the process of distilling high-quality
evidences to explain/support given answers, we propose an Informative-yet-
Concise Evidence Distillation algorithm (GCED) for answer explanation on QA.
Empirical experiments elaborate the distilled evidences are informative to
explain/support the input answers, concise without redundant noises and have
human-like readability. There is also some space to improve our algorithm. We
plan to verify the proposed GCED on more datasets and models, improving its
ability on understanding world knowledge/commonsense and speeding up the
process of evidence distillation in the future.

##  VII Acknowledgements

This work is supported by National Key Research and Development Project
(No.2020AAA0109302), Shanghai Science and Technology Innovation Action Plan
(No.19511120400) and Shanghai Municipal Science and Technology Major Project
(No.2021SHZDZX0103).

## References

  * [1] Atanasova Pepa, Simonsen Jakob Grue, Lioma Christina, and Augenstein Isabelle. 2020. Generating Fact Checking Explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.7352–7364. 10.18653/v1/2020.acl-main.656.
  * [2] Atanasova Pepa, Simonsen Jakob Grue, Lioma Christina, and Augenstein Isabelle. 2020. A Diagnostic Study of Explainability Techniques for Text Classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).3256–3274.10.18653/v1/2020.emnlp-main.263.
  * [3] Big Bird: Transformers for Longer Sequences.
  * [4] DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION.
  * [5] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
  * [6] Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering.
  * [7] Longformer: The Long-Document Transformer.
  * [8] The probabilistic relevance framework: Bm25 and beyond.
  * [9] RoBERTa: A Robustly Optimized BERT Pretraining Approach.
  * [10] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, 1533–1544. https://www.aclweb.org/anthology/D13-1160
  * [11] Ben Ahmed Mohamed Bghiel Afrae and Boudhir Anouar Abdelhakim. 2021. Smart Sustainable Cities: A Chatbot Based on Question Answering System Passing by a Grammatical Correction for Serving Citizens.
  * [12] José Gilvan R. Maia Caio Viktor S. Avila, Wellington Franco and Vania M. P. Vidal.2020. CONQUEST: A Framework for Building Template-Based IQA Chatbots for Enterprise Knowledge Graphs.
  * [13] Eugene Charniak and Mark Johnson. 2005. Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05). Association for Computational Linguistics, Ann Arbor, Michigan, 173–180. https://doi.org/10.3115/1219840.1219862
  * [14] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1870–1879. https://doi.org/10.18653/v1/P17-1171
  * [15] Hao Cheng, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2020. Probabilistic Assumptions Matter: Improved Models for Distantly-Supervised Document-Level Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5657–5667. https://doi.org/10.18653/v1/2020.acl-main.501
  * [16] Hao Cheng, Xiaodong Liu, Lis Pereira, Yaoliang Yu, and Jianfeng Gao. 2021. Posterior Differential Regularization with f-divergence for Improving Model Robustness. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 1078–1089. https://www.aclweb.org/anthology/202 naacl-main.85
  * [17] Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-Fine Question Answering for Long Documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 209–220. https://doi.org/10.18653/v1/P17-1020
  * [18] Noam Chomsky. 1965. Aspects of the Theory of Syntax. In MIT Press.
  * [19] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=r1xMH1BtvB
  * [20] Michael Collins. 2003. Head-Driven Statistical Models for Natural Language Parsing. Computational Linguistics 29, 4 (2003), 589–637. https://doi.org/10.1162/ 089120103322753356
  * [21] Philip Davies. 2002. What is Evidence-based Education?. In British Journal of Educational Studies.
  * [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota,4171–4186. https://doi.org/10.18653/v1/N19-1423
  * [23] Christiane Fellbaum. 2010. WordNet. In Theory and Applications of Ontology: Computer Applications. 231–243.
  * [24] Peter Jansen, Niranjan Balasubramanian, Mihai Surdeanu, and Peter Clark. 2016\. What’s in an Explanation? Characterizing Knowledge and Inference Requirements for Elementary Science Exams. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, 2956–2965. https: //www.aclweb.org/anthology/C16-1278
  * [25] Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 2021–2031. https://doi.org/10.18653/v1/D17-1215
  * [26] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics 8 (2020), 64–77.
  * [27] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large-scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, 1601–1611. https://doi.org/10.18653/v1/P17-1147
  * [28] Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling Semantic Parsers with On-the-Fly Ontology Matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, 1545–1556. https://www.aclweb.org/anthology/D13-1161
  * [29] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=H1eA7AEtvS
  * [30] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/ 6b493230205f780e1bc26945df7481e5-Abstract.html
  * [31] Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising Distantly Supervised Open-Domain Question Answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 1736–1745. https://doi.org/10.18653/v1/P18-1161
  * [32] Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural Relation Extraction with Selective Attention over Instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 2124–2133. https://doi.org/10.18653/v1/P16-1200
  * [33] David L.Sackett. 1997. Evidence-based medicine. In Seminars in Perinatology.3–5.
  * [34] Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. A Discrete Hard EM Approach for Weakly Supervised Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9t International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China,2851–2864. https://doi.org/10.18653/v1/D19-1284
  * [35] Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and Robust Question Answering from Minimal Context over Documents. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 1725–1735. https://doi.org/10.18653/v1/P18-1160
  * [36] Su-Jeong Song Min-Su Jeong and Yun-Gyung Cheong. 2020. Developing a Korean QA Chatbot for a Medical Domain Using a Multi-step Retrieval Approach.
  * [37] Rajarshee Mitra, Manish Gupta, and Sandipan Dandapat. 2020. Transformer Models for Recommending Related Questions in Web Search. In CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, Mathieu d’Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux (Eds.). ACM, 2153–2156. https://doi.org/10.1145/3340531.3412067
  * [38] Marco Valentino Mokanarangan Thayaparan and André Freitas. 2020. ExplanationLP: Abductive Reasoning for Explainable Science Question Answering.
  * [39] Graham Buckley R. M. Harden, Janet Grant and I. R. Hart. 2009. BEME Guide No.1: Best Evidence Medical Education. In Medical Teacher.
  * [40] Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain Yourself! Leveraging Language Models for Commonsense Reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Florence, Italy, 4932–4942. https://doi.org/10.18653/v1/P19-1487
  * [41] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Melbourne, Australia, 784–789. https: //doi.org/10.18653/v1/P18-2124
  * [42] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 2383–2392.https://doi.org/10.18653/v1/D16-1264
  * [43] Hendrik Schuff, Heike Adel, and Ngoc Thang Vu. 2020. F1 is Not Enough! Models and Evaluation Towards User-Centered Explainable Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7076– 7095\. https://doi.org/10.18653/v1/2020.emnlp-main.575
  * [44] Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. ReasoNet: Learning to Stop Reading in Machine Comprehension. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017. ACM, 1047–1055. https://doi.org/10.1145/3097983.3098177
  * [45] Linjun Shou, Shining Bo, Feixiang Cheng, Ming Gong, Jian Pei, and Daxin Jiang. 2020. Mining Implicit Relevance Feedback from User Behavior for Web Question Answering. In KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (Eds.). ACM, 2931–2941. https://dl.acm.org/doi/10.1145/3394486.3403343
  * [46] Quan Hung Tran, Nhan Dam, Tuan Lai, Franck Dernoncourt, Trung Le, Nham Le, and Dinh Phung. 2020. Explain by Evidence: An Explainable Memorybased Neural Network for Question Answering. In Proceedings of the 28th International Conference on Computational Linguistics. International Committee on Computational Linguistics, Barcelona, Spain (Online), 5205–5210. https://doi.org/10.18653/v1/2020.coling-main.456
  * [47] Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. 2019. Evidence Sentence Extraction for Machine Reading Comprehension. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). Association for Computational Linguistics, Hong Kong, China, 696–707. https://doi.org/10.18653/v1/K19-1065
  * [48] Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018. Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=rJl3yM-Ab
  * [49] John R. Weisz and Alan E. Kazdin. 2010. Evidence-Based Psychotherapies for Children and Adolescents. New York.
  * [50] Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto.2020. LUKE: Deep Contextualized Entity Representations with Entity-aware Selfattention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 6442–6454. https://doi.org/10.18653/v1/2020.emnlp-main.523
  * [51] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019. End-to-End Open-Domain Question Answering with BERTserini. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations). Association for Computational Linguistics, Minneapolis, Minnesota, 72–77. https://doi.org/10.18653/v1/N19-4013
  * [52] Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A Challenge Dataset for Open-Domain Question Answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, 2013–2018. https://doi.org/10.18653/v1/D15-1237
  * [53] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 5754–5764.
  * [54] Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, 1321–1331. https://doi.org/10.3115/v1/P15-1128
  * [55] Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. 2016. ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs. Transactions of the Association for Computational Linguistics 4 (2016), 259–272.