# Do Answers to Boolean Questions Need Explanations? Yes

Sara Rosenthal, Mihaela Bornea,
Avirup Sil, Radu Florian Scott McCarley
IBM Research AI
{sjrosenthal, mabornea, avi, raduf, jsmc} @us.ibm.com

###### Abstract

Existing datasets that contain boolean questions, such as BoolQ and TyDi QA ,
provide the user with a YES/NO response to the question. However, a one word
response is not sufficient for an explainable system. We promote
explainability by releasing a new set of annotations marking the evidence in
existing TyDi QA and BoolQ datasets. We show that our annotations can be used
to train a model that extracts improved evidence spans compared to models that
rely on existing resources. We confirm our findings with a user study which
shows that our extracted evidence spans enhance the user experience. We also
provide further insight into the challenges of answering boolean questions,
such as passages containing conflicting YES and NO answers, and varying
degrees of relevance of the predicted evidence.

##  1 Introduction

Answering questions that prompt a “YES/NO” response is a necessary capability
of a machine reading comprehension (MRC) Kwiatkowski et al. (2019); Clark et
al. (2019, 2020) system, but a one word response is not sufficient for an
explainable system. For example, the question “Can you buy Cadburys creme eggs
all year round?" represents a complex information need. A simple response of
“NO” leaves an information-hungry user of an MRC system wondering when Cadbury
creme eggs can be bought.

A successful user experience for boolean questions involves three parts: 1)
The user must receive a simple YES/NO answer. This has been the focus of
boolean datasets and leaderboards. 2) The user must receive supporting
evidence for the answer. Most boolean questions datasets provide the full
paragraph. 3) The supporting evidence must provide a positive user experience.
A good evidence span quickly guides the user to a concise explanation of the
answer. (See examples in Figure 1.) While the full paragraph provides some
explainability, the user is forced to filter out irrelevant information. In
addition, providing the degree of relevance of the evidence (highly relevant
evidence will be information-rich while weakly relevant evidence may require
additional inference from the user) and indicating multiple correct evidence
spans can also aid in providing a positive user experience. Figure 2.1 shows
an example with both strong and weak degrees of relevance. Figure 2.2 shows an
example with conflicting answers.

1\. Can you buy Cadbury’s creme eggs all year round? NO Creme eggs are
available annually between 1 January and Easter Day.NO In the UK in the 1980s,
Cadbury made Creme Eggs available year-round but sales dropped and they
returned to seasonal availability." 2\. Has anyone pitched back-to-back no
hitters? YES John Samuel Vander Meer (November 2, 1914 – October 6, 1997) was
an American professional baseball player. He played in Major League Baseball
as a pitcher, most notably for the Cincinnati Reds. Vander Meer is best known
for being the only pitcher in Major League Baseball history to throw two
consecutive no-hitters.YES After the impressive start to his major league
career, he experienced problems controlling the accuracy of his pitching, and
his later career was marked by inconsistent performances.

Figure 1: Examples of boolean questions and their answers that desire an
explanation as show in the provided passage. Supporting evidence for each
answer is highlighted in bold in the passage. YES, and NO

Existing MRC datasets do not support all aspects of this user experience.
Natural Questions Kwiatkowski et al. (2019), TyDi QA Clark et al. (2020), and
MS Marco Nguyen et al. (2016) contain from 1-10% boolean questions. Boolean
questions have also been explored individually in the BoolQ dataset Clark et
al. (2019). However, these datasets make several design assumptions during
annotation that simplify the task thereby reducing human understanding of
boolean questions: 1) Concise evidence is not provided to explain the YES/NO
response, 2) The response does not incorporate the relative relevance of the
answer, and 3) There is only one correct answer. Because of these limitations,
MRC systems Zhang et al. (2021); Wang et al. (2020c, b) largely ignore boolean
questions.

1\. Did Avicenna write any books? YES Avicenna was a Persian polymath who is
regarded as one of the most significant physicians, astronomers, thinkers and
writers of the Islamic Golden Age. He has been described as the father of
early modern medicine. Of the 450 works he is known to have writtenWEAK_YES,
around 240 have survived, including 150 on philosophy and 40 on medicine. His
most famous works are The Book of Healing, a philosophical and scientific
encyclopedia, and The Canon of Medicine, a medical encyclopediaYES which
became a standard medical text at many medieval universities[15] and remained
in use as late as 1650… 2\. Was there a year 0? YES/NO
Year zero does not exist in the anno Domini systemNO usually used to number
years in the Gregorian calendar and in its predecessor, the Julian calendar.
In this system, the year 1 BC is followed by AD 1. However, there is a year
zero in astronomical year numbering and in ISO 8601:2004 as well as in all
Buddhist and Hindu calendarsYES. Historical, astronomical and ISO year
numbering systems Historians. The Anno Domini era was introduced in 525 by
Scythian monk Dionysius Exiguus, who used it to identify the years on his
Easter table…

Figure 2: Examples illustrating the design assumptions that have been ignored
in popular datasets containing boolean questions. All the evidence for each
answer is highlighted in the passage. The superscripts indicate the evidence
response type: YES, WEAK YES, and NO.

In this paper, we explore improving user understanding through more
explainable answers to boolean questions in popular datasets. Our
contributions are:

  1. 1.

Comprehensive Annotations: We present comprehensive annotations on over 3,000
boolean questions from existing TyDi QA and BoolQ datasets that add multiple
supporting evidence spans to the YES/NO response as well as relevance of the
supporting span. The new annotations highlight the lack of complexity found in
these existing datasets. We will be releasing our annotations for research
purposes.

  2. 2.

Evidence Extraction: Our main focus is on extracting evidence spans of text to
be presented to the user to improve confidence in the system; We show that
incorporating supporting evidence for boolean questions in MRC models achieves
a 6 point F1 improvement in span extraction. To the best of our knowledge, we
are the first to explore evidence span extraction for boolean questions. In
contrast, prior work only uses evidence during training to improve yes/no
classification.

  3. 3.

User Experience: Our user study shows the positive impact that supporting
evidence extracted by the MRC model provides to users enhancing their
experience. Receiving concise evidence in addition to the YES/NO answer and
paragraph increases the confidence of the user in the answer.

The rest of the paper is outlined as follows: We first discuss related work in
Section 2. In Section 3 we talk about the data followed by our annotation
process in Section 4. We discuss our experiments and user study in Section 5
and 6. Sections 7.1, 7.2, and 7.3 analyze unanswerable questions, conflicting
answers, and weakly relevant answers.

##  2 Related Work

In addition to BoolQ Clark et al. (2019), datasets focusing on boolean
questions exist in other languages such as Russian Glushkova et al. (2020) and
Chinese Wang et al. (2020a). We only focus on English evaluation. StrategyQA
Geva et al. (2021) is a boolean multi-hop question answering dataset requiring
inference over non-boolean questions. Numerous works have explored various
approaches for improving results on BoolQ but have not explored the role of
supporting evidence Dzendzik et al. (2020); Micheli et al. (2021); Ross et al.
(2021); Yin et al. (2020).

Some papers use the spans of text internally to classify the YES/NO response
of the question, but do not learn to predict evidence spans or analyze how
useful evidence is to an end user. For example, the ReCo dataset Wang et al.
(2020a) includes evidence annotations for opinionated boolean questions in
Chinese. Niu et al. Niu et al. (2020) use a self-training approach to create
synthetic evidence for boolean questions using the gold YES/NO answer and
sentences in the passage.

Besides TyDi, other MRC datasets have included a mix of boolean and non-
boolean questions. The Natural questions dataset has only around 1% boolean
questions.Kwiatkowski et al. (2019) MS MARCO Bajaj et al. (2018) is a reading
comprehension dataset that was created from the Bing search engine query logs
with a substantial number of the boolean questions, but it does not provide
evidence spans for explanation.

The passages from BoolQ were annotated by Thorne et al. (2021) to obtain
supporting evidence for use in a claim verification task similar to FEVER
Thorne et al. (2018a, b). It is unclear whether this evidence is useful for
question answering because they transform the questions into claims for the
claim verification task. Boolean questions have also been used in other tasks
such as conversational question answering Choi et al. (2018); Reddy et al.
(2019).

##  3 Data

Dataset | YES | NO | NA | Total
---|---|---|---|---
TyDi QA Train | 404 | 98 | 60 | 562
TyDi QA Dev | 52 | 25 | 1 | 78
BoolQ Dev | 1300 | 763 | 315 | 2378
Table 1: Number of boolean instances in the TyDi English and BoolQ datasets
annotated for evidence. A breakdown of Yes, No, and Unanswerable (NA)
questions is provided for each dataset.

We use two datasets that contain boolean questions in our analysis and
experiments: TyDi QA Clark et al. (2020) and BoolQ Clark et al. (2019). We
chose these datasets due to their proportion of boolean questions and non-
restrictive licences. In the following subsections we describe how we use and
expand the datasets. For a detailed description of how the boolean questions
were created we direct interested readers to Kwiatkowski et al. (2019); Clark
et al. (2019, 2020). The sizes of the portions of the datasets that we use are
shown in Table 1. Both datasets provide a train and development set that are
available to download. The test sets are kept blind for leaderboard
submissions, thus our results are reported on the boolean portion of the
English dev sets.

###  3.1 TYDI QA

TyDi QA is a multilingual MRC dataset containing questions in multiple
languages. 10% of the questions are boolean. The TyDi QA boolean question
annotations include the paragraph the answer is found in and a label: YES or
NO. A boolean question can also be unanswerable. Each question in the dataset
is paired with a supporting Wikipedia document, including the offsets of all
its passages. The TyDi QA dataset provides a paragraph-length answer, called
the passage answer, that supports the answer to both boolean and factoid
questions. The dataset also provides a short answer for non-boolean questions.
It is important to note that this dataset is extremely skewed: 80% of
answerable boolean questions are YES. In this paper we explore the English
boolean questions which have an answer according to the TyDi QA annotations.

We provide our annotators with the passage answer and the preceding and
following paragraphs from the Wikipedia document. We annotated the full TyDi
QA English dataset for evidence consisting of 640 instances in total.

###  3.2 BoolQ-X

BoolQ is a dataset consisting of over 18k questions that are all boolean and
answerable. BoolQ provides the question, a boolean answer as YES or NO, and a
paragraph-length passage (around 100 words on average.) In contrast to TyDi QA
which is a document level MRC task, BoolQ is only a passage level MRC task. In
order to make the BoolQ data more useful in conjunction with the TyDi QA data,
we locate (by title) the source Wikipedia article for each BoolQ passage in
the 2018-12-20 Wikipedia dump. (Note that the original BoolQ passages had been
retrieved from the "live" Wikipedia.) This mismatch in Wikipedia versions
caused a loss of some questions in the original dataset from 3.2k to 2.4k dev
instances.

In order to obtain more useful context for annotation, we split each article
into 100 word chunks, and provide the annotators with a 500 word context
passage centered on the original BoolQ passage. We refer to our modification
of the BoolQ dataset as BoolQ-X in order to clearly distinguish that our
results are on BoolQ-X and not the original BoolQ dataset.

##  4 Evidence Annotation

| TyDi QA | BoolQ-X
---|---|---
Majority of annotators agree with Original Label | 98 (77.2%) | 210 (85.0%)
One or more annotators had conflicting YES and NO sentences by the same annotator | 24 (18.9%) | 16 (10.5%)
Two or more annotators had conflicting YES and NO sentences by the same annotator | 14 (11.0%) | 12 (4.9%)
Conflicting YES and NO sentences by two or more different annotators | 34 (26.8%) | 40 (16.2%)
Question was marked unanswerable by majority of annotators | 17 (13.4%) | 48 (19.4%)
Majority of annotators flipped YES <-> NO compared to original TyDi QA annotation | 7 (5.5%) | 16 (6.5%)
Total Annotations | 127 | 247
Table 2: Analysis of TyDi QA and BoolQ-X questions that were annotated by all
annotators.

We used an in-house annotation tool to annotate the evidence that explains the
YES/NO answer to the TyDi QA and BoolQ-X questions. The annotator is provided
with the question and passages. The passage displayed to the annotator were
divided into sentences. (See Figure 3 in the Appendix for a snapshot of the
annotation tool.) Although the original datasets provide us with a YES/NO
answer, we do not show the label to the annotators and ask them to provide us
with this information themselves. Given the question and passage, our
annotation task was as follows:

  1. 1.

Is there an answer? If not, it is "Unanswerable" (See Section 7.1 for further
discussion).

  2. 2.

If there is an answer, highlight 1-3 spans for YES and NO. Label each span:
YES, NO, WEAK_YES, or WEAK_NO. A span is a full sentence or a clause(s) in the
sentence.

  3. 3.

If the answer is WEAK, provide a reason:

     * •

EXTRA_INFO: The span contains extra words that don’t contribute to the
evidence.

     * •

MISSING_COREF: The first mention appears in the title or earlier in the
passage.

     * •

PARTIAL_ANSWER: The evidence does not satisfy the question completely.

     * •

QUESTION_AMBIGUITY: Ambiguity in the question such as poor vocabulary,
misspellings, and subjective meaning of a word

     * •

ANSWER_AMBIGUITY: Ambiguity in the answer span such as poor vocabulary,
misspellings, and subjective meaning of a word

     * •

OTHER: Any explanation not yet covered.

The data was annotated by 10 full time annotators. We had two pilot rounds to
ensure understanding of the task. All 10 annotators annotated the TyDi QA
dataset and 5 of the annotators annotated the BoolQ-X dataset. We assigned an
overlap of 10% of the TyDi QA training, the full TyDi QA dev set, and 10% of
the BoolQ-X dev set to be annotated by all the annotators to compute inter-
annotator agreement.

We compute inter-annotator agreement for the evidence on the sentence level
using F1. We consider the annotators to agree if highlighted spans are in the
same sentence and have the same answer label. For the purpose of computing
YES/NO sentence level agreement we ignored the WEAK labels by merging WEAK_YES
and YES and WEAK_NO and NO. Finally, we compute agreement at the document
level using accuracy. We consider the annotators to agree if they labeled one
or more of the sentences in the document as evidence with the same label.

Document level agreement to the original dataset: The majority of annotators
agreed with the original TyDi QA and BoolQ YES/NO labels indicating high
agreement as shown in Table 2.

Sentence and Document level inter-annotator agreement: We compute agreement
between each annotator and the annotator who was found to have the best
agreement with the consensus in a pilot study. At the sentence level, the
agreement was 55.3% F1 in TyDi QA and 49.8% in BoolQ-X. At the document level,
the annotators agree on average 68.9% in TyDi QA and 65.8% in BoolQ-X for the
full task. This indicates that there is moderate agreement for the full
dataset but the annotators do vary on which sentences they highlight.

While the majority of annotators agreed with the original TyDi QA and BoolQ
labels, we did find some inconsistencies. Table 2 shows the occurrence of
unanswerable questions, conflicting YES and NO labeled sentences for a single
question by a single annotator, conflicting YES and NO sentences by different
annotators, and the majority answer flipped from YES to NO or vice versa. We
discuss these inconsistencies further in Section 7.1 and 7.2.

###  4.1 Data Release and Stats

We prepare our annotated TyDi QA and BoolQ-X evidence spans for evaluation in
a document level111BoolQ is a passage level task. See Appendix for how we
prepared the BoolQ-X dataset for document level MRC. MRC system. During
training we keep one evidence span for each question. During evaluation, we
keep up to three evidence spans for each question. While we allowed the
annotators to choose a portion of a sentence as the span, we only consider
full sentences as evidence spans to avoid spans with overlapping offsets
across annotators. If there are more than three sentences, (this can occur on
occasion for the questions annotated by all annotators), we take the top three
based on frequency and give preference to strong answers over weakly relevant
answers. In the case of a tie we take the sentences earlier in the passage
first. In the YES/NO experiments we consider the original TyDi QA and BoolQ
labels to be correct even though on occasion our annotators disagreed with
them. We do not include questions that our annotators considered unanswerable.
Statistics about the evidence spans and YES/NO/Unanswerable labels can be
found in Table 3 and Table 1. Our data release will include the data in TyDi
QA format for easy replication of experiments.

Avg # Evidence | TyDi QA | BoolQ-X
---|---|---
sentences per document All | 2.3 | 1.5
sentences per document Overlap | 3.6 | 2.9
sentence length | 170 | 234
span length | 112 | 132
Table 3: Data Statistics for evidence annotations. Overlap refers to the data
annotated by all annotators.

##  5 Experiment Setup

In this section we describe our models and we evaluate the performance of
evidence span extraction and YES/NO answer prediction. We also demonstrate
that incorporating supporting evidence in the MRC training improves the
evidence span extraction performance.

Our approach is based on the XLM-Roberta language model Conneau et al. (2020),
as implemented in Wolf et al. (2020). We use separate models for evidence span
extraction and for answer classification (YES/NO). The evidence span
extraction is similar to the short answer extraction of Alberti et al. (2019).

In our experiments we used the TyDi QA training set to train all systems and
we evaluated on the annotated evidence spans in three experiments: TyDi QA
dev, TyDi QA cross-validation (CV), and BoolQ-X. The TyDi QA dev set only
contains 77 answerable English boolean questions. To compensate for the small
size of the TyDi QA dev dataset, we also perform 6-fold cross-validation using
the 562 training examples and we average the result over all 6 folds. The
BoolQ-X experiments are zero-shot: the models are trained on the TyDi QA train
set and evaluated on all the 2,063 answerable BoolQ-X annotated evidence
spans.

###  5.1 Evidence Span Extraction

We train an MRC system with all the examples in the TyDi QA training set,
including non-boolean questions. During training, the MRC objective is to
extract a span of text. This span will be the evidence when the question is
boolean and a short answer otherwise. We investigate different strategies for
incorporating evidence for boolean questions during MRC training.

##### Baseline (BASE):

A standard MRC system using out-of-the-box training data consisting of both
boolean and short answer questions. None of the boolean questions have
evidence spans during training and the YES/NO labels are ignored as in prior
work Zhang et al. (2021); Wang et al. (2020c, b).

##### Boolean Passage Evidence Span (BPES):

An MRC system baseline where gold passage spans are used to approximate
evidence spans for boolean questions during training. Both TyDi QA and BoolQ
provide the passage where the answer is found and we use these annotations to
compensate for the lack of short answer evidence spans in the BASE system.

##### Boolean Evidence Annotated Spans (BESA):

An MRC system where we use our sentence-level annotated evidence spans for the
boolean questions during training. Since our evidence-span annotations are
only on the English boolean training data, the BESA model is initialized from
the BPES model during fine-tuning.

Results: The evidence span extraction results are shown in Table 4 using the
F1 score computed with the official TyDi QA evaluation script. Other than TyDi
QA dev, where the dataset is too small to find significant trends, the
evidence span F1 increases with the quality of evidence span provided during
training. The baseline BPES system, which uses a rough approximation of
evidence spans with the gold passage, improves the performance by roughly 2 F1
points on both TyDi QA CV and BoolQ-X. The best performance is obtained using
the BESA system (our evidence span annotations) with an increase of over 6 F1
points on both TyDi QA CV and BoolQ-X.

| TyDi QA | TyDi QA | BoolQ-X
---|---|---|---
| CV | Dev | Dev
BASE | 29.3 | 34.6 | 25.7
BPES | 31.2 | 32.8 | 27.6
BESA | 35.5 | 35.7 | 31.9
Table 4: Evidence Span Extraction Results (F1) for TyDi QA cross validation
(CV) and TyDi QA and BoolQ development sets

###  5.2 Answer Classifier Experiments

After the evidence span is extracted, we use the answer classifier to produce
a YES/NO answer for the boolean questions at the document level. We use the
gold passages for all boolean questions in the TyDi QA dataset to train the
answer classifier and we show the YES/NO F1 score in Table 5. The answer
classifier training data contains examples from all TyDi QA languages while
the evaluation is on the English subset of TyDi QA , corresponding to our
annotations. Each example in the evaluation subset has a single YES/NO answer.

We compare the results with the majority baseline of always answering YES.
This is a strong baseline because both datasets are skewed (see Table 1.)

We tested the answer classifier using the passage containing the extracted
evidence span (BESA) and also the gold passage annotations from both TyDi QA
and BoolQ-X. Using the gold passage as context for the answer classifier
assumes the extracted evidence spans are always correct, and this is an upper
bound for our results.

Results: We notice that the evidence span classifier is close to the upper
bound on TyDi QA dataset, and there is little room for improvement, likely due
to the skewed distribution of YES/NO answers. On the BoolQ-X dataset there is
a larger difference (8 F1 points) when applying the classifier on system
passages compared to gold passages. Our system passage results outperform the
strong majority baseline for both datasets. This indicates that better
performance of evidence span extraction will also improve the YES/NO answer
prediction.

We also experimented with using only the predicted evidence span as input to
the answer classifier and obtained weaker results. We conjecture that the full
paragraph containing the evidence span provides additional context to the
system such as co-reference resolution.

| TyDi QA | TyDi QA | BoolQ-X
---|---|---|---
| CV | Dev | Dev
Majority | 80.8 | 77.1 | 63.0
Gold Passage (UB) | 85.5 | 85.7 | 78.9
Sys Passage | 82.0 | 87.0 | 74.3
Evidence Span | 81.2 | 85.0 | 70.1
Table 5:  YES/NO Results (F1) for TyDi QA cross validation (CV) and TyDi QA
and BoolQ development sets. The evidence span classifier is applied on: 1.
System Passage from BESA system containing the evidence span; 2. Gold Passage.
3. Extracted evidence span from BESA system;

##  6 Boolean Evidence Spans - User Study

Our experiments in the previous sections evaluated the performance of our
systems against the gold answers and showed that the BESA model, which
includes evidence spans at training time, performs best. However, it is
difficult to interpret how useful an end user would find the evidence. In this
section, we explore whether users find the highlighted evidence to be
informative for each of the models via a user study. This helps determine
which model provides the most informative explanation as evidence to answer
the question.

We perform a user study on the full answerable TyDi QA dev set of 77 questions
and an equal portion of the BoolQ-X dev set. An example of the user study
interface is provided in the Appendix Figure 4. Our findings from the user
study are shown in Table 6.

| TyDi QA | BoolQ
---|---|---
| BASE | BPES | BESA | BASE | BPES | BESA
Relevant | 45% | 42% | 55% | 32% | 42% | 58%
Irrelevant | 29% | 37% | 27% | 39% | 27% | 16%
Weakly Relevant | 26% | 21% | 17% | 29% | 31% | 26%
Relevant/Weak and in Gold | 38% | 36% | 37% | 26% | 34% | 38%
Relevant/Weak but not in Gold | 33% | 27% | 35% | 35% | 39% | 46%
Mean Reciprocal Rank | 0.79 | 0.77 | 0.88 | 0.71 | 0.77 | 0.87
Poor Highlighting | 8% | 14% | 3% | 32% | 20% | 5%
Table 6: User Study analysis of the evidence spans produced by each system
averaged across the three participants for the 77 TyDi QA and BoolQ-X
questions from the development sets.

The user study was completed by three of our annotators, each analyzing all 77
questions from each dataset. We provided these users with an interface that
contained the output from each of our three models: BASE, BPES, and BESA. The
output was randomized and anonymized (the users only saw PRED A, B, C instead
of the model name) ensuring that the users did not know which system the
output originated from. The study explored two aspects of the models which the
users had to answer: 1) For each model, is the highlighted evidence span
relevant, weakly relevant, or irrelevant for answering the question? 2) Rank
the passages based on preference of which highlighted span is best. In other
words, which highlight would you (the user) want to see for the given
question? Asking these two distinct questions regarding relevance and rank is
important because even if all the answers are irrelevant, some evidence spans
may be more appropriate than others based on their pertinence to the
question.222The highlighting of inappropriate spans can lead to embarrassing
answers. While in most cases the top rank is correlated with relevance, in our
study the rank is designed to compare the highlighted spans of the three
models to each other.

We first examine the relevance of the highlighted evidence spans and find that
all users picked BESA as having the most relevant evidence spans and the least
irrelevant evidnece spans as shown in Table 6 for both datasets. Weakly
relevant evidence spans were more common in BoolQ-X, which is zero-shot.
Further, in both datasets there are many evidence spans that are relevant but
were not found in the gold evidence span annotations. These are evidence spans
found by the system (which saw the full document) while the annotators only
received the candidate passage and its surrounding paragraphs. We find that
for all the models the number of relevant evidence spans not in gold is
comparable to or larger than the number of relevant evidence spans in gold for
both datasets. This indicates that our models predict relevant evidence spans
considerably more often than reflected by our evaluation shown in Table 4.

Next we examine the ranking of the three models. The rank given by the users
was from 1-3 where the best evidence span is given a rank of 1 and the worse
span is given a rank of 3. Multiple evidence spans could have the same rank in
cases of a tie. If there is a tie the count would continue by skipping the
next position (e.g. 1 1 3 instead of 1 1 2). We discouraged ties when the
highlights were not identical, although there are some cases where the users
considered two different evidence spans to be equally informative and they
were assigned the same rank. We used Mean Reciprocal Rank (MRR) to compare the
ranking of the three models. In all cases the annotators preferred BESA over
the other models. The MRR scores reflect our results shown in Table 4: BPES is
preferred for BoolQ-X and BASE is preferred for TyDi QA . The average MRR is
shown in Table 6.

We also asked the annotators to specifically record any predictions that had
poor highlighting span boundaries. Examples of poor highlighting span
boundaries include spans in which only one or two words were highlighted, and
cases where highlighting broke off mid-word or phrase. Some examples of good
and poor highlighting are available in Appendix B. Table 6 shows that poor
highlighting span boundaries was most infrequent for the BESA model for both
datasets. In general, highlighting was poorer for BoolQ-X. This is mostly
likely due to the zero-shot transfer from TyDi QA.

We also provided the users with the question and answer paragraph with and
without highlighting and asked them in general whether they prefer to have the
paragraph with or without the highlighting. All the annotators said they
preferred the highlighting. This indicates that adding highlighting to current
systems that provide a YES/NO answer with a full paragraph would be helpful to
the user.

##  7 Analysis

###  7.1 Unanswerable Questions

During the annotation task our annotators found some questions to be
unanswerable even though all of our annotations were performed on question-
answer pairs that had an answer of YES or NO in the original datasets. 9% of
TyDi QA and 13% of BoolQ-X questions were marked as unanswerable as shown in
Table 1. Table 2 shows the number of questions that were marked unanswerable
when all annotators saw the data.

We found that questions tend to be unanswerable for two main reasons. First,
sometimes the question is not answered in the page. While this could indicate
that the answer is NO, it is not clear. For example, consider the question “Do
US Nintendo 64 games work in Australia?" from the BoolQ dataset. While the
passage provided was about Nintendo 64 games, Australia was not mentioned
within the supporting document, so this question cannot be answered. Second,
sometimes the the inappropriate context was provided for the question. For
example, consider the question “Is Major General Douglas dead?” from the TyDi
QA dataset. This question is provided with the incorrect document about
“Douglas Haig, 1st Earl Haig” instead of the document about “Major General
Alexander Douglas Campbell”.

###  7.2 Conflicting Answers

Prior work on boolean questions makes the assumption that the answer is either
YES or NO. However, it is possible to have a question where there are
conflicting answers to a boolean question that are both correct. Consider the
following example from the BoolQ dataset:

Question: Do you need to say check in chess?

Title: Check (Chess)

Passage: A player must get out of check, if possible, by interposing a piece
between the threatening piece, or block the check with another piece. A king
cannot itself directly check the opposing king, since this would place the
first king in check as well. A move of the king could expose the opposing king
to a discovered check by another piece, however. In informal games, it is
customary to announce "check" when making a move that puts the opponent’ s
king in checkYES. In formal competitions, however, check is rarely
announcedNO…

The above question has both YES and NO answers in the passage, depending upon
the setting (informal game vs. formal competition) in which the chess game is
played. This example illustrates that conflicting answers often occur when a
question is under-specified. The answer may depend on certain demographics, a
time period, or other characteristics relevant to the question topic. Under-
specified questions are very common. For example, a person may ask the
question "Can I get the COVID vaccine?" which can be YES or NO depending on
when they are asking, their age, occupation, location, or other health
factors. Another example of conflicting answers in a single passage is shown
in Figure 2.2.

The frequency of conflicting answers in TyDi QA and BoolQ-X from the same
annotator and from multiple annotators is shown in Table 2. Conflicting
answers occur in both datasets, with more frequency in TyDi QA . The need for
more information to answer the question can be addressed using dialog with the
user, an interesting area to explore in future work.

###  7.3 Weakly Relevant Answers

|  TyDi QA Weak | BoolQ-X Weak
---|---|---
Partial Answer | 113 | 193
Missing Coref | 118 | 309
Answer Ambiguity | 184 | 328
Other | 15 | 3
Extra Info | 28 | 50
Question Ambiguity | 36 | 39
# Weak Sentences | 515 (37.6%) | 922 (27.7%)
# Weak Documents | 69 (23.9%) | 306 (12.4%)
Table 7: Analysis of reasons for weakly relevant sentences in the full TyDi QA
and BoolQ-X annotations. A sentence is considered weak if one or more
annotators marked it as weak. A document is considered weak if all the
annotations in the document are weak.

In some cases there is evidence in the passage that is relevant to the answer
but does not completely answer the question (See example in Figure 2.1). We
consider such evidence to be weakly relevant. During annotation, we allowed
the annotators to indicate that the evidence they selected was weak by
indicating that it was a WEAK_YES or a WEAK_NO. When a weak label was selected
we also requested that they provide a reason that it was weak as described in
Section 4333Note we did not strictly enforce the annotators to provide a
reason for weak annotations, thus the number of weak sentences is more than
the sum of the different reasons.. Table 7 shows the occurrence of weakly
relevant sentence annotations and the reasons in the TyDi QA and BoolQ-X
datasets. The count indicates that one or more annotators marked a weak
annotation in a sentence, and the reason. Different annotators could say a
sentence had weak evidence but provide different reasons for their annotation.
Weakly relevant evidence is more common in TyDi QA than BoolQ. The most common
reason evidence is weak is due to answer ambiguity such as in the following
example:

Question: Can the Queen dissolve parliament?

Title: Dissolution of the Parliament of the United Kingdom

Passage: The last dissolution of Parliament was on 3 May 2017, to make way for
the general election to be held on 8 June 2017. [7] It dissolved after a two-
thirds majority vote by the House of Commons, as required by the Fixed-term
Parliaments Act. 30 March 2015 was the first time Parliament had ever
dissolved automatically, as opposed to being dissolved by Royal
ProclamationWEAK_YES …

In the above example, the answer is ambiguous because it can be inferred that
the Queen dissolved the parliament by Royal Proclamation even though it does
not say so explicitly. Examples for the other reasons evidence is annotated as
weak are available in the Appendix.

Our MRC system provides a confidence score for the answer. We expect that the
confidence score could be used to indicate the degree of relevance of the
evidence to the answer. During our user study we also asked the users to mark
the evidence as relevant, irrelevant, or weakly relevant as shown in Table 6.
We mapped the relevance to the confidence scores and found that on average the
score for relevant answers is highest indicating there is some signal in the
confidence scores. A user interface could indicate the confidence by making
the highlighted text lighter if it is weakly relevant and darker if it is
relevant. We leave further exploration of using confidence to indicate degree
of relevance as future work.

##  8 Conclusion

In this paper we explore the importance of providing a YES/NO response and
evidence for boolean questions to the end user of machine reading
comprehension systems. We provide detailed evidence annotations on top of the
boolean questions in the popular BoolQ and TyDi QA datasets. We analyze
several aspects that have been previously overlooked: the occurrence of
supporting evidence, the degree of relevance of answers, and existence of
conflicting answers in a single passage. We will be releasing our annotations
for research purposes.

We show that providing evidence not only increases the model performance but
also improves the user experience as measured in our user study. Our user
study shows that receiving concise evidence in addition to the YES/NO answer
and paragraph increases the confidence of the user in the answer. In the
future we plan to explore degree of relevance and conflicting answers on a
larger scale.

## References

  * Alberti et al. (2019) Chris Alberti, Kenton Lee, and Michael Collins. 2019.  A bert baseline for the natural questions.
  * Bajaj et al. (2018) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018.  Ms marco: A human generated machine reading comprehension dataset.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  Quac: Question answering in context.  In _EMNLP_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.
  * Dzendzik et al. (2020) Daria Dzendzik, Carl Vogel, and Jennifer Foster. 2020.  Q. can knowledge graphs be used to answer Boolean questions? A. it’s complicated! In _Proceedings of the First Workshop on Insights from Negative Results in NLP_ , pages 6–14, Online. Association for Computational Linguistics.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021.  Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies.  _Transactions of the Association for Computational Linguistics (TACL)_.
  * Glushkova et al. (2020) Taisia Glushkova, Alexey Machnev, Alena Fenogenova, Tatiana Shavrina, Ekaterina Artemova, and Dmitry I. Ignatov. 2020.  Danetqa: a yes/no question answering dataset for the russian language.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association of Computational Linguistics_.
  * Micheli et al. (2021) Vincent Micheli, Quentin Heinrich, François Fleuret, and Wacim Belblidia. 2021\.  Structural analysis of an all-purpose question answering model.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  In _CoCo@NIPS_.
  * Niu et al. (2020) Yilin Niu, Fangkai Jiao, Mantong Zhou, Ting Yao, Jingfang Xu, and Minlie Huang. 2020\.  A self-training method for machine reading comprehension with soft evidence extraction.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Ross et al. (2021) Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E. Peters, and Matt Gardner. 2021\.  Tailor: Generating and perturbing text with semantic controls.
  * Thorne et al. (2021) James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021.  Evidence-based verification for real world information needs.
  * Thorne et al. (2018a) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a.  FEVER: a large-scale dataset for fact extraction and VERification.  In _NAACL-HLT_.
  * Thorne et al. (2018b) James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b.  The Fact Extraction and VERification (FEVER) shared task.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_.
  * Wang et al. (2020a) Bingning Wang, Ting Yao, Qi Zhang, Jingfang Xu, and Xiaochuan Wang. 2020a.  Reco: A large scale chinese reading comprehension dataset on opinion.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34:9146–9153.
  * Wang et al. (2020b) Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, S. Sun, Yu Cheng, and Jingjing Liu. 2020b.  Cluster-former: Clustering-based sparse transformer for long-range dependency encoding.  _ArXiv_ , abs/2009.06097.
  * Wang et al. (2020c) Xuguang Wang, Linjun Shou, Ming Gong, Nan Duan, and Daxin Jiang. 2020c.  No answer is better than wrong answer: A reflection model for document level machine reading comprehension.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yin et al. (2020) Heju Yin, Feng Zhou, Xiaoyong Li, Junqiang Zheng, and Kai Liu. 2020.  Transfer learning on natural yes/no questions.  In _2020 IEEE 11th International Conference on Software Engineering and Service Science (ICSESS)_ , pages 1–6.
  * Zhang et al. (2021) Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021.  Poolingformer: Long document modeling with pooling attention.  In _ICML_.

##  Appendix A Annotation

Figure 3: Screenshot of the annotation tool used to find evidence.

###  A.1 Weakly Relevant Examples

The following are examples of questions with Weakly Relevant evidence:
Question Ambiguity, Missing Coref, Partial Answer, and Extra Info.

####  A.1.1 Question Ambiguity

Question: Is Creole a pidgin of French?

Title: French-based creole languages

Passage: A French creole, or French-based creole language, is a creole
language (contact language with native speakers) for which French is the
lexifier. Most often this lexifier is not modern French but rather a 17th-
century koiné of French from Paris, the French Atlantic harbors, and the
nascent French colonies. French-based creole languages are spoken natively by
millions of people worldwide, primarily in the Americas and on archipelagos
throughout the Indian Ocean. This article also contains information on French
pidgin languages, contact languages that lack native speakers. …

The above TyDi QA question provides an example of question ambiguity because a
Creole is a pidgin with native speakers and does not refer to a specific
language.

####  A.1.2 Missing Coref

Question: Did the equal pay act pass?

Title: The Equal Pay Act of 1963

Passage: The Equal Pay Act of 1963 is a United States labor law amending the
Fair Labor Standards Act, aimed at abolishing wage disparity based on sex (see
Gender pay gap). It was signed into law on June 10, 1963, by John F. Kennedy
as part of his New Frontier Program. [1] …

The highlighted evidence in the above BoolQ example is missing coref. The
Equal Pay Act is referenced in the first sentence.

####  A.1.3 Partial Answer

Question: Are they actually singing in Grey’s Anatomy?

Title: Song Beneath the Song

Passage: A vocal coach was enlisted to help the cast. [10] Music director
Chris Horvath was recruited to arrange the selected songs for the cast. The
arrangements took around two months, with vocals recorded over four days in
February 2011.[7][12] Horvath praised the cast’s response to the episode,
noting that only four performers had "serious vocal talent," while some had
"barely sung in the shower" before.[7] Those with professional singing
experience include Ramirez, who won a Tony Award for their role in the musical
Spamalot, and Wilson, who appeared in the Broadway production of Caroline …

The highlighted evidence spans in the above BoolQ example are partial answers
because the evidence does not satisfy the question completely. The evidence
indicates that the actors did sing but does not say so explicitly.

####  A.1.4 Extra Info

Question: Does the UK have universal health care?

Title: List of countries with universal health care

Passage: Virtually all of Europe has either publicly sponsored and regulated
universal health care or publicly provided universal healthcare. The public
plans in some countries provide basic or "sick" coverage only, with their
citizens being able to purchase supplemental insurance for additional coverage
. Countries with universal health care include Austria, Belarus,[55] Croatia,
Czech Republic, Denmark, Finland, France, Germany, Greece, Iceland, Ireland,
Italy, Luxembourg, Malta, Moldova,[56] the Netherlands, Norway, Portugal,[57]
Romania, Russia, Serbia, Spain, Sweden, Switzerland, Turkey, Ukraine,[58] and
the United Kingdom.[59] …

In the above TyDi QA example countries other than the UK are included to
ensure a complete evidence span is highlighted leading to extra information.

###  A.2 Data Release

We convert the BoolQ dataset, which is passage level, to a document level task
for our experiments. We segment the original Wikipedia document into rough
paragraphs by splitting it into sentences using the NLTK tokenizer. We keep
roughly 500 characters in each paragraph, while ensuring a cutoff does not
occur mid-sentence. We also ensure that the candidate answers are in a single
paragraph. The document level version of BoolQ-X is available in TyDi QA
format for easy replication of experiments.

##  Appendix B User Study

Figure 4: Screenshot of the user study tool used to find evidence. In addition
to the three randomized predictions shown here the users were also provided
with the gold answers and full document text to optionally use if additional
context was required.

Figure 4 shows a screenshot of the tool the participants in the user study
used to compare our various models.

The following are examples of good and poor highlighting from the user study
from the different models. The good highlighting is in bold and the poor
highlighting is underlined. (In some cases they overlap.)

Question: Is there a sequel to The Amazing Spider Man? Title: The Amazing
Spider-Man 2 Passage: The Amazing Spider-Man 2 (also released as The Amazing
Spider-Man 2: Rise of Electro in some markets) is a 2014 American superhero
film featuring the Marvel Comics character Spider-Man. The film was directed
by Marc Webb and was produced by Avi Arad and Matt Tolmach. It is the fifth
theatrical "Spider-Man" film produced by Columbia Pictures and Marvel
Entertainment, and is the sequel to 2012’s "The Amazing Spider-Man". It is
also the second and final film in "The Amazing Spider-Man" franchise. The
studio hired James Vanderbilt to write the screenplay and Alex Kurtzman and
Roberto Orci to rewrite it.

Question: Is Starbucks and Seattle’s Best same company?
Title: Seattle’s Best Coffee
Passage: Seattle’s Best Coffee LLC is an American coffee retailer and
wholesaler based in Seattle, Washington. Since 2003, they have been a
subsidiary of American coffeehouse chain Starbucks. Seattle’s Best Coffee has
retail stores and grocery sub-stores in 20 states and provinces and the
District of Columbia. Sub-stores can also be found at many other businesses
and college campuses, including JCPenney and Subway restaurants.

Question: Is Cargo an Australian movie?
Title: Cargo (2017 film)
Passage: Cargo is a 2017 Australian post-apocalyptic drama thriller film
directed by Ben Howling and Yolanda Ramke with a screenplay by Ramke based on
their 2013 short film of the same name.[1][2] The film stars Martin Freeman,
Anthony Hayes, Susie Porter, and Caren Pistorius.[3]. It premiered at the
Adelaide Film Festival on 6 October 2017 and was released in cinemas in
Australia on 17 May 2018, worldwide except for Australia on 18 May 2018 by
Netflix and on Netflix in Australia on 16 November 2018.[4]


# Novel Uncertainty Framework for Deep Learning Ensembles

Tal Kachman, Michal Moshkovitz, Michal Rosen-Zvi
IBM Research

###### Abstract

Deep neural networks have become the default choice for many of the machine
learning tasks such as classification and regression. Dropout, a method
commonly used to improve the convergence of deep neural networks, generates an
ensemble of thinned networks with extensive weight sharing. Recent studies
that dropout can be viewed as an approximate variational inference in Gaussian
processes, and used as a practical tool to obtain uncertainty estimates of the
network. We propose a novel statistical mechanics based framework to dropout
and use this framework to propose a new generic algorithm that focuses on
estimates of the variance of the loss as measured by the ensemble of thinned
networks. Our approach can be applied to a wide range of deep neural network
architectures and machine learning tasks. In classification, this algorithm
allows the generation of a don’t-know answer to be generated, which can
increase the reliability of the classifier. Empirically we demonstrate state-
of-the-art AUC results on publicly available benchmarks.

##  1 Introduction

Deep learning (DL) algorithms have successfully solved real-world
classification problems from a variety of fields, including recognizing
handwritten digits and identifying the presence of key diagnostic features in
medical images [18, 16]. A typical classification challenge for a DL algorithm
consists of training the algorithm on an example data set, then using a
separate set of test data to evaluate its performance. The aim is to provide
answers that are as accurate as possible, as measured by the true positive
rate (TPR) and the true negative rate (TNR). Many DL classifiers, particularly
those using a softmax function in the very last layer, yield a continuous
score, $h$; A step function is used to map this continuous score to each of
the possible categories that are being classified. TPR and TNR scores are then
generated for each separate variable that is being predicted by setting a
threshold parameter that is applied when mapping $h$ to the decision. Values
above this threshold are mapped to positive predictions, while values below it
are mapped to negative predictions. The ROC curve is then generated from these
pairs of TPR/TPN scores. The performance of binary classifiers, performance is
often evaluated by calculating the area under the ROC curve (AUC) [;,
hanley1982meaning] the higher the AUC, the better the performance of the
algorithm.

Many studies show that the AUC achieved by DL algorithms is higher than most,
if not all, of the alternative classifiers. Although they achieve high scores
for metrics such as AUC, DL algorithms are notorious for being “black box”
models, as it is difficult to obtain insight into how the algorithm arrived at
its conclusion. This makes them less reliable, particularly for applications
where data-based decisions are a firm requirement. One way to mitigate this
problem when applying DL algorithms to these data-driven applications is to
provide a measure of the classification uncertainty, or the confidence one has
in the classification prediction, along with the prediction of the outcome.
This can be accomplished by training a series of similar models that differ in
their initialization parameters, and then calculating the variance between the
probabilities for each possible outcome across the differently initialized
networks. The ground truth of the uncertainty, then, is the variance between
these similar models. Reporting a confidence level along side the prediction
helps the model’s end user to interpret and build trust in the model’s
performance. For Example, this is a critical requirement for algorithms
deployed in a medical setting [2], where a moderately positive prediction with
high uncertainty has very different prognosis and treatment implications than
does the same prediction when the uncertainty is low. We show an example of
both low variance and high variance predictions, calculated using the ground
truth as defined , in Figure 1a.

When assessing uncertainty in DL algorithms such classifiers are frequently
permitted to return a “don’t know” answer for very low confidence predictions
in the test data. This allows the algorithm to be judged only on the responses
in the “do know” portion of the data set. Consequently, the algorithm
generates overall higher quality predictions, while leaving humans to
interpret samples for which it would generate poor quality predictions,
similar to a triage system

The continuous variable $h$ that is output by the last layer of a DL with a
softmax function provides the likelihood of an outcome. However, this
likelihood estimation should not be mistaken for a confidence level in the
prediction (see [6]). Instead, recent studies have combined the DL
regularization technique of dropout with Bayesian modeling to derive
uncertainty estimates in DL classifiers [6, 7]. Bayesian approaches provides a
natural framework for estimating the uncertainty of the prediction.
Furthermore, dropout is a regularization technique that uses an ensemble of
models to create high-performance classifiers [22, 15]. The interpretation of
dropout through a Bayesian lens enables derivation of uncertainty estimates in
DL classifiers. Within this framework the classification score, $h$, describes
the probability of predicting the correct class. The ROC curve for the model
is measured on an averaged value of $h$, rather than a single value, and the
uncertainty is estimated based on the variance of $h$. This uncertainty
measure has been shown to correlate with the performance of the classifier.
Using the averaged $h$ to predict the outcome was sufficient to increase
classification accuracy in some cases [11].

In this paper, we present a novel method for estimating the uncertainty of a
DL classifier. We propose a framework that assigns probability distributions
to thinned networks,(i.e., neural networks with a subset of neurons removed),
based on the performance of their cross-entropy loss function across the test
data set. The main contributions of this work are threefold. First, we
introduce a statistical-mechanics framework that assigns probability
distributions over the ensemble of thinned networks of the dropout. This
framework has a flexible variable, $\beta$, which represents the inverse
temperature, and a statistical fluctuation scale. When set to zero, it results
in a uniform distribution assumption over the thinned networks and the
framework collapses to a Gaussian process. In contrast, a finite inverse
temperature $\beta$ results in a non-uniform distribution, and the framework
enables interpretation and reasoning regarding uncertainty. Second, we present
a new algorithm, called Loss Variance Monte Carlo Estimate (LoVME), which is
based on estimations of the loss variance in the case of a finite $\beta$
through Monte Carlo sampling. Finally, we illustrate the benefits of deriving
uncertainty through the LoVME algorithm in scenarios where the classifier can
yield a don’t-know answer. We use the MNIST [14] and CIFAR [13] data sets to
show the performance of our algorithm, and compare our results to state-of-
the-art algorithms for uncertainty in DL.

The rest of the paper is organized as follows: First, we introduce related
work. We focus on two related bodies of knowledge: recent proposed methods to
derive uncertainty measures for test data using the Bayesian interpretation of
dropout, and the existing statistical-mechanics frameworks for analyzing
distributions over an ensemble of models. Next, we introduce a new methodology
for interpreting the ensemble of Neural Networks (NNs) generated by dropout.
We present a derivation of a new algorithm, LoVME, which uses Monte Carlo
sampling to estimate the loss variance; through this loss variance, it
provides estimates of the predicted variable $h$ and the uncertainty of the
prediction. We evaluate the performance of the algorithm on multiple data sets
and compare it to state-of-the-art algorithms. We conclude with a discussion
of the advantages of the proposed framework and algorithm, as well as
potential extensions and open questions.

We conclude with a discussion of the advantages of the proposed framework and
algorithm, as well as potential extensions and open questions.

##  2 Related Work

###  2.1 Dropout as a Bayesian average

DL algorithms are based on large NNs with non-linear layers. Each layer
contains units that are composed of input variables, output variables, and the
weight vectors that connect them. While deep learning algorithms perform
exceptionally well as classifiers when trained on large data sets, they are
also known to suffer from overfitting. Different approaches have been proposed
to address this drawback, including dropout, which is among the most
successful. Dropout can be viewed as a mechanism for learning an ensemble of
closely related, partly overlapped (sometimes called ‘thinned’) neural
networks. During training, a single fixed architecture is perturbed through a
random process of pruning or thinning; this results in an ensemble of closely
related NNs. A simple version of dropout is based on a fixed random variable,
$p$. Each of the units (hidden and visible) in a NN has probability $p$ of
being retained during a given training iteration, and probability $1-p$ of
being dropped out of the network. For units that are dropped from the NN,
incoming and outgoing connections are also removed. The result of this random
pruning process is an ensemble of thinned NNs, where if the fixed original
architecture had $N$ units, the training updates generate NNs with about $pN$
units. This can create as many as $2^{N}$ different NNs, all with a common
base architecture. Predictions on test data should ideally use the trained
ensemble of thinned NNs to find an average prediction. However, such a
prediction method requires using an exponentially large collection of NNs, and
is therefore impractical. Instead, one common method for employing dropout is
to use the full learned NN, without any pruning but with down-scaled weights.
The original trained weights of all units are multiplied by the factor $1/p$,
and the original fixed architecture with the scaled weights is used for
inference of the test examples.

###  2.2 Bayesian uncertainty in deep learning

Recent breakthrough studies [6, 7] have added uncertainty estimation to DL
algorithms in a way that is both empirically and mathematically grounded. The
uncertainty evaluation uses _dropout inference_ , where means that the
ensemble of thinned NNs is used to find the variance of the prediction. The
method has been proven successful in a variety of domains, including
reinforcement learning [7] and computer vision [11, 12].From a theoretical
perspective, this method uses the Bayesian inference framework to evaluate the
uncertainty. In this framework, a prior over the model parameters and a
likelihood are defined. Using Bayes’ theorem, these two distributions yield a
posterior over the model parameters. Marginalizing the posterior leads to the
posterior predictive distribution, which encapsulates the uncertainty.
Unfortunately, marginalization is computationally intractable. To overcome
this difficulty, the approximation inference paradigm [5] suggests replacing
the true posterior by an approximate one that is restricted to belong to a
simpler form of distribution family. The dropout inference method uses the
approximation inference paradigm for NNs. They define the prior as a product
of multivariate normal distributions for each layer, and the likelihood as a
softmax for multiclass classification. The simpler form of distribution family
is a mixture of two Gaussians with small variances, and the mean of one of the
Gaussians is fixed at zero.

To evaluate the posterior predictive distribution, Kendall et al. [11, 12]
used Monte Carlo integration. This integration samples a few thinned NNs and
calculates their variance. This variance is used as an approximation for the
uncertainty. It is important to note that in calculating the variance, each
sampled thinned NN is of equal weight. In this paper, we claim that this
uniform averaging is only a crude estimation. Inspired by ideas from
statistical mechanics, we suggest taking into account the (estimated) loss of
each thinned NN. We demonstrate that the loss for different thinned NNs can
differ by an order of magnitude, thus using an appropriate weighting for each
thinned NN is extremely valuable

###  2.3 Statistical mechanics approach

Work on random networks in the context of Bayesian NNs has a long history [19,
3]. In the Bayesian approach, one can consider the different weights in the
network as drawn from a certain prior and trained towards a posterior. This
suggests that one can in fact look at the different thinned networks created
by dropout as an ensemble. The approach of viewing ensembles in order to
approximate the behavior and states of individual entities is a fundamental
approach in the field of statistical mechanics. Applying techniques and models
commonly used in statistical mechanics to problems in deep learning has seen a
recent resurgence. For example, Choromanska et al. [4] showed that random
rectified linear NNs could, with approximation, be mapped onto spin glasses.
Baity [1] explored the learning dynamics and effects of training deep networks
as glassy dynamics, and Schoenholz et al. [21] construct a full statistical
field theory for deep NNs. Consequently, drawing on methods and findings from
statistical mechanics can be a powerful tool for improving deep learning
methods.

Motived by this set of recent results, and the intrinsic link between
statistical mechanics and statistical properties of thinned NN, we formulate a
new approach to infer and obtain uncertainty measures for different models.

##  3 Statistical Mechanics Framework for Deep Learning Ensemble

###  3.1 Methodology

In this section, we derive an explicit connection between random NNs and
statistical mechanics. This link between the two methodologies will provide
insight into the different moments and cumulants of the the NN ensemble. One
can think of the uncertainty as the equivalent of the statistical properties
of the ensemble in the test phase.

To describe the stochastic ensemble of networks, we must design a measure for
the probability space and define the fluctuation size with its given
statistical properties, (i.e., its observable quantity). Following the well-
known methodology of Jaynes [9], we can write a maximum entropy argument that
allows us to obtain the maximally unbiased probability distribution. Motivated
by dropout, we define an ensemble that is defined by the full NN and its
associated thinned NNs. Our observables will be the average loss and for a
changing number of neurons average number of units.

For an ensemble of thinned networks, we define each thinned network $i$, is
defined by its loss $\mathcal{L}_{i}$ and the number of neurons that it
contains, $N_{i}$. We would like to find the distribution over the thinned NNs
which we denote $p(\mathcal{L_{\text{i}}},N_{i})$ or simply $p_{i}$. We can
estimate the expected size $N$, as it depends on the dropout value and the
expected loss using the training error. Thus, we can approximate the following
two expectations

$\mathbb{E}[\mathcal{L}]=\sum_{i}p\left(\mathcal{L}_{i},N_{i}\right)\mathcal{L}_{i}=\mathcal{L}$
(1)
$\mathbb{E}[N]=\sum_{i}p\left(\mathcal{L}_{i},N_{i}\right)N_{i}=\mathcal{N}$
(2)

According to the principle of maximum entropy, [9] the distribution with
maximal entropy best represents the current state of knowledge (i.e., $-\sum
p_{i}\log p_{i}$) when equations 1, 2 and the normalization constraint $\sum
p_{i}=1$ are satisfied. To find the maximally unbiased probability
distribution, we define the following Lagrangian

$\displaystyle J[p]$ $\displaystyle=-\sum_{i}p_{i}\log
p_{i}+\lambda_{0}\left(\sum_{i}p_{i}-1\right)+\beta\left(\sum_{i}p_{i}\mathcal{L}_{i}-\mathcal{L}\right)+$
$\displaystyle+\eta\left(\sum_{i}p_{i}N_{i}-\mathcal{N}\right)$

Setting the derivative $\frac{\partial J}{\partial p_{i}}$ to $0$, we have
that the probability of the $i$-th thinned NN is

$p_{i}=\frac{1}{Z}e^{-\left(\beta\mathcal{L}_{i}+\eta N_{i}\right)}$ (3)

where $Z$ is the well-known partition function that normalizes the probability
measure $p_{i}$

$Z=\sum_{i}e^{-\left(\beta\mathcal{L}_{i}+\eta N_{i}\right)}$ (4)

This is known as the Gibbs distribution in statistical mechanics, where
$\beta$ is known as the inverse temperature and $\eta$ is the chemical
potential.

We can deduce the variance of the loss from $Z$,

$\mathbb{V\text{ar}}[\mathcal{L}^{2}]=\frac{\partial^{2}\log
Z}{\partial\beta^{2}}$ (5)

This variance is a measure of the uncertainty. Thus, we want to compute the
partition function. However, the partition function contains an exponential
number of terms and therefore is impossible to calculate directly. In the next
section, we present an algorithm that approximates the value of this partition
function using an importance sampling technique; this technique gives a
measure of the uncertainty with respect to different cumulants. The partition
function 4 corresponds exactly to the well-known grand canonical partition
function[10], establishing the relationship between the two fields. While the
partition function 4 can generate moments to give a representation of the
uncertainty, obtaining this representation involves summing over an
exponential probability space. Sampling such a high dimensional space is
impractical; instead, we can use the partition function as a scale. In the
following sections, we present our algorithm, which combines the probability
measure 3 with importance sampling to obtain the variance in the loss
$\mathcal{L}$ and number of units $N$. This allows us to estimate the
uncertainty using the above moments.

###  3.2 Loss Variance Monte Carlo Estimate

It is impossible to calculate the cumulant analytically, but it is possible to
evaluate the cumulant numerically. We outline our Loss Variance Monte Carlo
Estimate (LoVME) algorithm to calculate the uncertainty. We can write the
estimator of $\mathbb{E}\left[Q\right]$ over $M$ samples for an observable $Q$
as $\left\langle
Q\right\rangle^{M}=\frac{\sum_{j=1}^{M}Q_{j}p_{j}}{\sum_{j=1}^{M}p_{j}}$ and
see that as $M\rightarrow\infty$ this equation gives the exact average of the
estimator. As with previous methodologies using dropout, one can choose to
uniformly sample the $M$ states in the estimator. Uniform sampling is
equivalent to generating a network through a Bernoulli trial much like
dropout, this network will by the nature of this trial take a long time to
converge, since one can get stuck in a local minima of this “state space” .

For more robust sampling of this estimator, we used the probability measure 3
as a non-uniform sampler. This approach gives less weight to highly loss-
thinned networks, and allows us to avoid sampling only in a small region of
thinned NN. To ensure that we pick each state with its associated probability
measure, we generate a Markov chain Monte Carlo in an ergodic way, such that
the rates obey a detailed balance. Imposing the condition of ergodicity on our
Monte Carlo allows us to make some of the transitions in the Markov chain
zero, while requiring that there be at least one path of non-zero transition
probabilities between any two configurations. This forces all states to be
accessible. Imposing a second condition of detailed balance, allows us to
sample thinned networks with Metropolis-Hastings algorithms. Requiring
detailed balance ensures that after enough sampling steps, we indeed sample
with 3. Mathematically, we can write down the transition dynamics between two
thinned NNs as

$\sum_{v}p_{\mu}P\left(\mu\rightarrow
v\right)=\sum_{v}p_{v}P\left(v\rightarrow\mu\right)$ (6)

here $P\left(\mu\rightarrow v\right)$ is the rate of transition. Detailed
balance ensures that all the thinned NN are accessible one from another

$p_{\mu}P\left(\mu\rightarrow v\right)=p_{v}P\left(v\rightarrow\mu\right)\
\forall\mu,v$ (7)

This tells us that, on average the system should go from $\mu$ to $v$ as much
as it goes from $v$ to $\mu$. In fact, we can now use the probability 3 to
write down the rates that will generate the desired Markov chain.

$\frac{P\left(\mu\rightarrow
v\right)}{P\left(v\rightarrow\mu\right)}=\frac{p_{v}}{p_{\mu}}=e^{-\eta\left(N_{v}-N_{\mu}\right)}e^{-\beta\left(\mathcal{L}_{v}-\mathcal{L_{\mu}}\right)}$
(8)

The constraint of 8 still leaves us a good deal of freedom over how to choose
the transition probabilities, since there are many ways to satisfy the
equality. Rewriting the detailed balance condition

$\frac{P\left(\mu\rightarrow
v\right)}{P\left(v\rightarrow\mu\right)}=\frac{g\left(\mu\rightarrow
v\right)}{g\left(v\rightarrow\mu\right)}\frac{A\left(\mu\rightarrow
v\right)}{A\left(v\rightarrow\mu\right)}$ (9)

we can introduce the acceptance function $A$ for the Monte Carlo move and
selection probabilities $g$. The following lets us look at the reversible
selection probabilities of $\frac{1}{N}$ and

$A\left(v\rightarrow\mu\right)=A_{0}e^{-\frac{1}{2}\eta\left(N_{\mu}-N_{v}\right)}e^{-\beta\left(\mathcal{L}^{\mu}-\mathcal{L}^{v}\right)}$
(10)

to obtain the acceptance rule in the form

$A\left(v\rightarrow\mu\right)=A_{0}e^{-\frac{1}{2}\eta\left(N_{\mu}-N_{v}\right)}e^{-\beta\left(\mathcal{L}^{\mu}-\mathcal{L}^{v}\right)}$
(11)

By building on 11 and using Metropolis rates we can write the full acceptance
rule for our simulations

$A\left(v\rightarrow\mu\right)=\begin{cases}e^{-\beta\left(\left(\mathcal{L}^{\mu}-\mathcal{L}^{v}\right)+\frac{\eta}{\beta}\left(N_{\mu}-N_{v}\right)\right)}&\text{if
}\mathcal{L}^{\mu}-\mathcal{L}^{v}+\frac{\eta}{\beta}\left(N_{\mu}-N_{v}\right)>0\\\
1&\text{otherwise }\end{cases}$ (12)

Using sequential dynamical Monte Carlo moves with the acceptance rate 12 we
can generate the ensemble of different thinned NNs. In Algorithm 2, we give
the flow and the details of the LoVME algorithm implemented with the above
rates.

1: Input: Oracle access to loss $\mathcal{L}_{i}$ for each thinned NN $i$

2: Output: Estimate $\mathbb{V}ar[\mathcal{L}_{i}]$

3: Parameter: $T$ \- number of transitions the algorithm performs

4: Initialization:

5: $1.L$: array of size $T$ of losses encountered throughout the run of the
algorithm

6: $2.N_{0},\mathcal{L}_{0}$: loss and number of neurons in the fully
connected trained NN

7: $3.N_{\mu}=N_{0}$: first NN in HM algorithm

8: $4.t=1$

9: while $t\leq T$ do

10: $N\in_{R}[N_{0}]$, random from the discrete uniform distribution

11: $v:=$ random thinned NN with $N$ neurons

12: $\theta\sim U(0,1)$, random from the continuous uniform distribution

13: if
$\mathcal{L}_{\mu}-\mathcal{L}_{v}+\frac{\eta}{\beta}(N_{\mu}-N_{v})<\theta$
then

14: $L[t]=\mathcal{L}_{v}$

15: $\mu:=v$

16: $t:=t+1$

17: end if

18: end while

19: return variance of array $L$

Algorithm 1 Loss Variance Monte Carlo Estimate ((LoVME) Algorithm 2 Loss
Variance Monte Carlo Estimate

##  4 Experiments

We now demonstrate the ability of the LoVMe algorithm to predict model
uncertainty with respect to several different metrics. For in-depth insight
into the structure of the uncertainty measure in our ensemble, we
qualitatively evaluated the higher order cumulates for the loss distribution.
In addition to evaluating several different metrics for the prediction, we
also evaluated a series of higher order cumulates and their effects on the
uncertainty. To estimate the uncertainty, we trained a LeNET [17] network in
PyTorch [20], using dropout with probability $p=0.5$ 111Code to be available
for both the network and the LoVMe. We used a cross-entropy softmax loss
function in both the training and testing phases. For the testing stage and to
calculate the Monte Carlo moves, we evaluated the loss function against a
randomly generated label set, for both the uncertainty measure against $N$ and
$\mathcal{L}$.

###  4.1 Data

To demonstrate the effectiveness of the LoVEMe algorithm 2, we applied it to
three different datasets: the widely adopted MNIST [14], Fashion-MNIST [23],
and CIFAR-10 [13] . For both the MNIST and Fashion-MNIST we also tested
against a partially perturbed subset, where we introduced both random rotation
and noise to randomly chosen $10\%$ of the images in the test set. For each
dataset, we generated an ensemble of thinned NNs using the LoVMe algorithm;
each of these ensembles typically converged within 300 Monte Carlo steps.

###  4.2 Obtaining a Ground Truth

(a) The Monte Carlo loss distribution for two data sets studied.
Each distribution is color coded to their image.

(b) Our “ground truth” uncertainty, the loss variance and average probability
correlation function

Figure 1: Monte Carlo results for uncertainty

Before describing our uncertainty measure in detail, it’s worthwhile to obtain
an understanding of the intrinsic or ground truth uncertainty that a model
possess. To gauge the basic uncertainty measure, we constructed a “ground
truth” measure of the uncertentity. This was done by training an ensemble of
$4000$ LeNET networks with different initializations of the weights. For each
ensemble we performed a classification of test portion in the dataset, and
took the probability outcome from the softmax cross-entropy for correct class.

Fig. 1b shows the scatter plot between the variance in the loss, the
uncertainty $\mathbb{{V}}ar[\mathcal{L}]$ and the average over the different
probabilities of the correct label $\mathbb{E}[P\left(\text{correct
label}\right)$. It is interesting to note that the point at which the
probability of the model returning the correct label equals 1 corresponds to a
transition point between the uncertainty and the probability being correlated
and uncorrelated. Furthermore, there is a region where for a high uncertainty
we have an ambiguous or almost random probability of the model returning the
correct prediction.

###  4.3 Estimating the Uncertainty using LoVME

We next estimate the uncertainty using the variance $\mathbb{{V}}ar[N]$ and
$\mathbb{{V}}ar[\mathcal{L}]$. In Fig. 1a we show two distributions of
$\mathcal{L}$ and their respective test examples for the CIFAR10 and MNIST
dataset. The MNIST data set quantitatively showed lower values of variance.
This is to be expected as the network learns and converges within a few epochs
to $99.7\%$ accuracy. Nonetheless, by plotting the distribution and
calculating its variance, we can see the variance and uncertainty of each
sample. Each color in Fig. 1a coincides with its distribution function, and
the figure legend shows the variance values. For the CIFAR data set, there is
a more distinct difference in the uncertainty values and in the spread of the
distribution. It is also informative to look at the structure of a few chosen
examples and their distributions, as shown in Fig. 1b shows the structure of a
few chosen examples and their distributions. Surprisingly, most distributions
either look bimodal or non-symmetric, which suggests that while the variance
measures the uncertainty, higher moments might also come into play while
considering a sample’s uncertainty. Our LoVME method demonstrates this
important fact by evaluating the distribution explicitly, as well as by
looking at higher moments.

###  4.4 Uncertainty in a Decision Metric

Estimating the uncertainty during the test stage also enables us to
incorporate uncertainty into the decision metric. To show the effect of
uncertainty on the prediction, we show the loss variance scores through both
Receiver Operating Characteristic (ROC) and Area Under the Curve (AUC) plots.

For a probability score $P$ of a sample, we use its associated uncertainty as
a binary confidence interval. Looking at
$P\pm\mathbb{V}ar\left[\mathcal{L}^{2}\right]$ could either shift the sample
into a TPR or a FPR, allowing us to use a type of confidence interval on top
of the ROC, as shown in Fig. 2.Generating the confidence interval of the score
allows us to set a certain threshold, above which a sample is too ambiguous to
be classified as either a false positive or true negative. Thus, it can be
eliminated from the ROC decision process. In Fig. 3 we show that removing high
uncertainty examples increases the AUC when using our LoVME method, Monte
Carlo dropout, and when calculating the naïve ground truth. Table 1 shows the
different values of the AUC for each of the three methods.

Figure 2: ROC for two classes, one of each dataset, with our “confidence
intervals” measures

Figure 3: ROC curves with improvements based on uncertainty measures  |  no uncertainty corrections |  ground truth uncertainty corrections |  MC dropout uncertainty corrections |  LoVME uncertainty corrections
---|---|---|---|---
MNIST AUC |  0.967 |  0.974 |  0.974 |  0.974
CIFAR10 AUC |  0.956 |  0.967 |  0.972 |  0.973
Table 1: AUC values for the set of experiments

## References

  * [1] M Baity-Jesi, L Sagun, M Geiger, S Spigler, G Ben Arous, C Cammarota, Y LeCun, M Wyart, and G Biroli.  Comparing dynamics: Deep neural networks versus glassy systems.  arXiv preprint arXiv:1803.06969, 2018.
  * [2] Riccardo Bellazzi and Blaz Zupan.  Predictive data mining in clinical medicine: current issues and guidelines.  International journal of medical informatics, 77(2):81–97, 2008\.
  * [3] Youngmin Cho and Lawrence K Saul.  Kernel methods for deep learning.  In Advances in neural information processing systems, pages 342–350, 2009.
  * [4] Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun.  The loss surfaces of multilayer networks.  In Artificial Intelligence and Statistics, pages 192–204, 2015\.
  * [5] Charles W Fox and Stephen J Roberts.  A tutorial on variational bayesian inference.  Artificial intelligence review, 38(2):85–95, 2012.
  * [6] Yarin Gal.  Uncertainty in deep learning.  University of Cambridge, 2016.
  * [7] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059, 2016.
  * [8] James A Hanley and Barbara J McNeil.  The meaning and use of the area under a receiver operating characteristic (roc) curve.  Radiology, 143(1):29–36, 1982.
  * [9] Edwin T Jaynes.  Information theory and statistical mechanics.  Physical review, 106(4):620, 1957.
  * [10] Mehran Kardar.  Statistical physics of fields.  Cambridge University Press, 2007.
  * [11] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.  Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding.  arXiv preprint arXiv:1511.02680, 2015.
  * [12] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  In Advances in Neural Information Processing Systems, pages 5580–5590, 2017.
  * [13] Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  2009\.
  * [14] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [15] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436, 2015.
  * [16] Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne E Hubbard, and Lawrence D Jackel.  Handwritten digit recognition with a back-propagation network.  In Advances in neural information processing systems, pages 396–404, 1990.
  * [17] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [18] Paulo J Lisboa and Azzam FG Taktak.  The use of artificial neural networks in decision support in cancer: a systematic review.  Neural networks, 19(4):408–415, 2006.
  * [19] David JC MacKay and Radford M Neal.  Near shannon limit performance of low density parity check codes.  Electronics letters, 32(18):1645, 1996.
  * [20] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * [21] Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein.  A correspondence between random neural networks and statistical field theory.  arXiv preprint arXiv:1710.06570, 2017.
  * [22] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research, 15(1):1929–1958, 2014\.
  * [23] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.