#  LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning
Capabilities for NLI

Ishan Tarunesh,1 Somak Aditya, 2 Monojit Choudhury 2

###### Abstract

Natural Language Inference (NLI) is considered a representative task to test
natural language understanding (NLU). In this work, we propose an extensible
framework to collectively yet categorically test diverse Logical reasoning
capabilities required for NLI (and by extension, NLU). Motivated by behavioral
testing, we create a semi-synthetic large test-bench ($363$ templates, $363k$
examples) and an associated framework that offers following utilities: 1)
individually test and analyze reasoning capabilities along 17 reasoning
dimensions (including pragmatic reasoning), 2) design experiments to study
cross-capability information content (leave one out or bring one in); and 3)
the synthetic nature enable us to control for artifacts and biases. The
inherited power of automated test case instantiation from free-form natural
language templates (using CheckList), and a well-defined taxonomy of
capabilities enable us to extend to (cognitively) harder test cases while
varying the complexity of natural language. Through our analysis of state-of-
the-art NLI systems, we observe that our benchmark is indeed hard (and non-
trivial even with training on additional resources). Some capabilities stand
out as harder. Further fine-grained analysis and fine-tuning experiments
reveal more insights about these capabilities and the models – supporting and
extending previous observations. Towards the end we also perform an user-
study, to investigate whether behavioral information can be utilised to
generalize much better for some models compared to others.

## Introduction

Recently the NLP community has observed that the measured performance
improvements in the popular benchmarks do not always translate to targeted
improvements in learning the capabilities expected for solving a task. This
gap was specifically highlighted as Richardson et al. (2020) and Ribeiro et
al. (2020) showed a collection of simple examples representing minimal
expected capabilities for a task are enough to completely degrade state-of-
the-art models’ performance. Such a performance degradation gap was
acknowledged and explicitly pointed out by Bowman and Dahl (2021), who
released a call for action for new benchmarks where the performance in
benchmark aligns with good task performance. Furthermore, in Schlangen (2021),
authors point out crucial differences between benchmarking in computing and
benchmarking as followed in NLP. Performing a task requires a collection of
cognitive capabilities for a human. In contrast, datasets are only a
reflection of the task, and models are often optimized on the train split of
the benchmark dataset, irrespective the required underlying cognitive
capabilities. Such a train-test setup, as highlighted by the authors, makes it
hard to know the connections between task and capabilities, and the possible
transfer of learnt capabilities from one task to the other without re-
training.

Several researchers believe a dynamic adversarially created benchmark will
properly reflect performance, explaining the performance degradation. Nie et
al. (2020) proposed an adversarial human-in-the-loop dataset creation process
where humans play a game with a state-of-the-art system(s) to create
increasingly complex examples. This however does not sufficiently address the
problem as the initial models still fail at simpler examples (Kaushik, Hovy,
and Lipton 2020; Richardson et al. 2020; Ribeiro et al. 2020), leaving the
question “can the system perform a particular type of reasoning?” unanswered.
Inspired from behavioral testing, Ribeiro et al. (2020) proposed the creation
of template-based test-suites (called CheckList), that has a broader coverage
ranging from the minimal expected functionality to more complicated tests
across a range of capabilities. Moving beyond phenomena-wise probing and
cloze-task formulations, the methodology produces a capability-wise behavioral
summary that aggregates different shortcomings of the SOTA models across
capabilities in a disentagled manner. In this work, we show that with the
right selection of required reasoning (or cognitive) capabilities, such
behavioral testing can help find out which reasoning abilities cause the
performance degradation gap; thereby also establishing a task-capability
relation (Schlangen 2021). A templated test-suite following the behavioral
testing methodology conforms to the traditional definition of benchmarking in
computing, where it is not necessary to provide a training split; rather might
be discouraged, as behavioral functional testing is often independent of
software development.

To this end, we propose a novel extensible Natural Language Inference
benchmark to explicitly test a diverse set of reasoning capabilities. We
create a CheckList that enables evaluation of whether NLI systems exhibit such
reasoning capabilities. In the process, we extend the list of capabilities in
Ribeiro et al. (2020) to cover more interesting linguistic and logical
reasoning phenomena (such as causal, spatial, pragmatic) required in NLI (and
similar tasks such as Question-answering). We expand on how we come up with
templates for such reasoning capabilities and discuss the different ways our
benchmarking dataset can be used. Through a short survey of related datasets
and our experiments, we show how the LoNLI dataset (and the associated
framework) uniquely enables fine-grained disentangled capability-wise analysis
along various dimensions; and to some extent, address the performance
degradation by underscoring the harder capabilities.

#### Contributions.

Our contributions are as follows. We create a semi-synthetic capability-rich
NLI test-suite LoNLI (17 capabilities, 363 templates, 363k examples) by
extending the reasoning taxonomy of Joshi et al. (2020). We show that LoNLI is
hard, extensible by nature, balanced across known dimensions of bias, and
enables fine-grained analysis of reasoning abilities. From our analysis, we
find 1) LoNLI is hard to solve even with additional targeted resources (such
as ImpPres (Jeretic et al. 2020)), 2) quantifier, implicature and deduction
capabilities (spatial, temporal, numerical) stand out as harder across
systems; and 3) also orthogonal as they are hard to learn by solely training
on other capabilities. 4) Further, template-wise and intra-template
observations attest to previously found facts and discover new ones without
the hassle of targeted probing task creation.

## Survey of Existing Datasets

The two most widely used large-scale NLI datasets are the SNLI (Bowman et al.
2015) and the MultiNLI (MNLI) dataset (Williams, Nangia, and Bowman 2018). The
high performance of the Transformer-based pre-trained language models (PTLM)
on these datasets (and corresponding NLU benchmarks which include them) have
led researchers to scrutinize them further, and find annotation biases and
spurious artifacts in the datasets (Poliak et al. 2018b; Gururangan et al.
2018). These findings and performance degradation of trained PTLMs on targeted
phenomena-wise tests have led to different evaluation strategies such as i)
probing tasks, ii) challenge datasets, and iii) capability-wise fine-grained
evaluation. Extensive probing tasks have been implemented to investigate how
and where (within the model) linguistic information has been encoded (Tenney,
Das, and Pavlick 2019; Tenney et al. 2019; Hewitt and Manning 2019; Jawahar,
Sagot, and Seddah 2019; Liu et al. 2019a; Kim et al. 2019). However, the
effectiveness of probing tasks have come into question. For example,
Ravichander et al. (2020) cautions that BERT may not understand some
“concepts” even though probing studies may indicate otherwise. Several work
(McCoy, Pavlick, and Linzen 2019; Kaushik, Hovy, and Lipton 2020; Glockner,
Shwartz, and Goldberg 2018) also develop targeted evaluation sets to challenge
these large PTLMs and demonstrated shortcomings. Moreover, popular NLU
benchmarks (including MultiNLI, SNLI), as pointed by Schlangen (2021), lack
explicit task-capability connection which has led to attempts at building
datasets focusing on testing a range of reasoning capabilities (Poliak et al.
2018a; Richardson et al. 2019). Lastly, Ribeiro et al. (2020) proposed the
CheckList framework for NLP tasks that made the above performance degradation
explicit, as PTLMs failed at minimal expected capabilities.

Dataset |  Creation Methodology | Size |  | Reasoning
---
Annotations
SNLI (Bowman et al. 2015) |  Crowd-sourced | 570K | ✗
MNLI (Williams, Nangia, and Bowman 2018) |  Crowd-sourced | 433K | ✗
ANLI (Nie et al. 2020) |  Crowd-sourced with human in the loop | 162K | ✗
ConTRoL (Liu et al. 2021) |  Expert annotated | 8325 | ✓
CLCD (Salvatore 2019) |  Synthetic | 60K | ✓
ImpPres (Jeretic et al. 2020) |  Synthetic | 25K | ✓
TaxiNLI (Joshi et al. 2020) |  (Subset of) MNLI annotated with trained annotators | 10K | ✓
LoNLI (ours) |  Semi-Synthetic | 363K | ✓
Table 1: The position of LoNLI in the spectrum of (a subset of closely
related) NLI datasets.

Based on such observations, many calls (Schlangen 2021; Bowman and Dahl 2021)
for proposing improved NLP benchmarks have poured in. The following properties
for an ideal NLU benchmark seems essential: 1) Benchmarks should make the
connection between task and required capabilities explicit, where capabilities
should be transferable across tasks (NLI, QA). 2) Benchmarks should be hard to
solve, and 3) extensible. 4) Benchmarks should control for annotation
artifacts and known biases. 5) Benchmarks should enable an understanding of
the limits of current systems as well as whether systems exhibit minimal
expected functionalities.

Most recent datasets fall short on one or multiple of these criterion.
Capability-agnostic datasets such as QNLI, SciTail (Khot, Sabharwal, and Clark
2018), and FEVER (Thorne et al. 2018) focus on a specific aspect of knowledge
(such as knowledge of scientific phenomenon or world knowledge). Datasets such
as FEVER often exhibit biases (Schuster et al. 2019), possibly inherited from
the source. Moreover, most examples in these datasets (even targeted ones such
as WNLI) often require multiple capabilities. DialogueNLI (Welleck et al.
2019) features a persona-based dialogue structure for making inference on the
current utterance based on previous dialogue history. ANLI (Nie et al. 2020)
is created through a human-in-the-loop process where participants play an
adversarial game to fool a state-of-the-art system. While the methodology
yields a way to create dynamic and increasingly harder benchmark, the
resulting dataset becomes model-aware, and does not help address the
performance degradation on simpler examples. Examples in ANLI also lack
explicit capability information. Capability-aware datasets such as ConTRoL
(Liu et al. 2021), ImpPres (Jeretic et al. 2020), CLCD (Salvatore, Finger, and
Hirata Jr 2019), and TaxiNLI (Joshi et al. 2020) are proposed to evaluate
capability-wise performance. Examples in ConTRoL test understanding of
coreference resolution, logical and temporal reasoning alongwith information
integration over multiple sentences. The examples are expert-designed using
publicly available online practice tests as the source. AlphaNLI (Bhagavatula
et al. 2020) (cloze-style dataset rather than NLI) explores the problem of
abductive reasoning - finding the most plausible narrative context given the
outcome. CLCD and ImpPres use synthetically generated examples to test logical
(only connectives) and pragmatic capabilities respectively. In TaxiNLI,
authors annotate a subset of MNLI with taxonomic information rather than
creating a synthetic dataset. However, authors do not control for the biases
and artifacts that may have been inherited from the MNLI, and it is also hard
to extend as annotation require extensive training.

In contrast to existing datasets (Tab. 1), through the proposed LoNLI
benchmark, we enable evaluation at multiple granularity levels: intra-template
(grouping via lexicon), a template, a (reasoning) capability, or the entire
test bench level. We extend the capabilities proposed in Joshi et al. (2020),
covering a gamut of reasoning abilities required for the NLI task (and NLU in
general). Unlike challenge datasets, our dataset is not hard by design as the
templates are targeted to cover minimal functionalities. But upon evaluation,
we observe that our benchmark is difficult for most SOTA systems (and non-
trivial to solve even with additional resources). Despite having simple
templates we are able to attest existing observations from past works and also
find out new observations. This brings us to another merit, i.e.,
extensibility. LoNLI can be extended in multiple dimensions: (i) more
capability and template can be added for probing particular phenomena, and
(ii) templates involving multiple capabilities (hardness). Harder templates,
in a cognitive sense, can be created by proposing templates that require
multiple capabilities. Lastly, being semi-synthetic in nature, LoNLI is
balanced with respect to known dimensions of biases, such as gender-profession
bias (de Vassimon Manela et al. 2021), and lexical bias (Gururangan et al.
2018) such as “not” being strongly correlated with contradiction (see
Appendix).

## A Logical CheckList for NLI

The CheckList111https://github.com/marcotcr/checklist methodology (Ribeiro et
al. 2020) assists users in testing NLP models by creating templates for a
variety of linguistic capabilities. These templates can then be used to
generate multiple examples following the same structure but with different
values of placeholders. Table 2 shows an example CheckList template for NLI
task. Here NAME1 and NAME2 are placeholders of the same lexicon {NAME} =
{Alexia, John, Mia, …} and ADJ comes from the lexicon {ADJ} = {good, bad,
kind, …}.

Template | Expected
---|---
P: {NAME1} and {NAME2} are {ADJ}. H: {NAME1} is {ADJ}. | Entail
Example: P: John and Harry are apathetic. H: John is apathetic. | Entail

Table 2: An example NLI template testing connective and.

In tasks such as NLI, inferencing often requires (one or more) linguistic and
logical reasoning capabilities. Our goal is to test the systems against such
reasoning types. This presents us with two challenges for templated test-suite
generation for the NLI task (or tasks that require different types of
reasoning): 1) careful selection of capabilities that represent well-known
linguistic and logical taxonomy, and are easily extensible, 2) template
creation for such capabilities.

### Selection of Capabilities

According to the linguistics literature (Wittgenstein 1922; Jurafsky and
Martin 2009), deciphering meaning from natural language form often takes both
semantic and pragmatic understanding abilities. From the perspective of Logic
(Charles Pierce), there are three pre-dominant forms of reasoning: deductive,
inductive, and abductive; that can be employed by an agent to understand and
interact. Recently, Joshi et al. (2020) proposed an (extensible)
categorization of the reasoning tasks involved in NLI with a granularity finer
than the high-level categorizations (Language and Logic), based on their
relevance with respect to current public NLI datasets. Authors define three
broad groups of reasoning: Linguistic, Logical and Knowledge, which aligns
with our philosophy. Other categorizations (Nie et al. 2020; Wang et al.
2018), though relevant, are often incomplete as they are tuned towards
analyzing (model-specific) error cases.

Here, we introduce the categories from Joshi et al. (2020)222For detailed
definitions, please see Joshi et al. (2020). and show examples (and templates)
in Tab. 3. Linguistic covers examples where inference process is internal to
the provided text; further divided into capability lexical and syntactic.
Logical covers examples where the inference may involve processes external to
text, and grouped under Connectives and Deduction. Connectives involve
categories such as negation, boolean, quantifier, conditional and comparative.
Deduction comprises of reasoning pertaining to relational, spatial, temporal,
causal and coreference. Knowledge indicates examples where external (world) or
commonly assumed (we only consider taxonomic) knowledge is required for
inference. We extend the taxonomy by adding back the (pruned) numerical
category. We also add a high-level category Pragmatic, with two sub-categories
pre-supposition (subsuming factivity) and implicature.

# | Template & Examples | Annotations
---|---|---
T1 |  P: {NAME1} and {NAME2} are from {CTRY1} and {CTRY2} respectively. H: {NAME1} is from {CTRY1}. entail | E (1)
Bool. |  P: George and Michael are from Germany and Australia respectively. H: George is from Germany.
T2 |  P: {NAME1} and {NAME2} are from {CTRY1} and {CTRY2} respectively. H: {NAME1} is from {CTRY2}. contradict | C (1)
Bool. |  P: Helen and Barbara are from Canada and Brazil respectively. H: Helen is from Brazil.
T3 |  P: {MALE} and {FEMALE} are friends. He is {A} {PROF1} and she is {A} {PROF2}. H: {FEMALE} is {A} {PROF2}. entail | E (.8)
Coref |  P: Ricardo and Angelique are collegues. He is a minister and she is a model. H: Angelique is a model.
T4 |  P: {CITY1} is {N1} miles from {CITY2} and {N2} miles from {CITY3}. H : {CITY1} is nearer to {CITY2} than {CITY1} is to {CITY3}. entail | E (1)
Spatial |  P: Manchester is 67 miles from Pittsburg and 27 miles from Kansas. H: Manchester is nearer to Kansas than Manchester is to Pittsburg.
T5 |  P: {NAME} has {SUB1} class then {SUB2} class then {SUB3} class. H: {SUB1} class is the first class among the three. entail | E (1)
Temp. |  P: Jessica has latin class then geography class then french class. H: Latin class is the first class among the three.
T6 |  P: {NAME1} {CAUSAL_VERB} {CAUSAL_OBJ} to {NAME2}. H: {NAME2} {CAUSAL_VERB_OPP} {CAUSAL_OBJ} from {NAME1} entail | E (1)
Causal |  P: Fiona taught mathematics to Ruth. H: Ruth learnt mathematics from Fiona.
T7 |  P: {NAME1}, {NAME2}, {NAME3} are the only children of {NAME0}. H: {NAME0} has exactly 3 children. entail | E (1)
Num. |  P: Dorothy, Katie and Rose are the only children of Stephen. H: Stephen has exactly 3 children.
T8 |  P: {NAME1} is the child of {NAME0}, {NAME2} is the child of {NAME0}. H: {NAME0} has atleast 2 children. contradict | C (1)
Num. |  P: Martin is the child of Susan. Jill is the child of Susan. H: Susan has atleast 2 children.
T9 |  P: {NAME} lives in {CITY}. H: {NAME} lives in {CTRY}. entail | E (.8)
World |  P: Patrick lives in Lahore. H: Patrick lives in Pakistan
T10 |  P: {NAME}’s {RELATION} is {ADJ}. H: {NAME} has a {RELATION}. entail | E (1)
PreSup |  P: William’s wife is tactful. H: William has a wife.
T11 |  P: {NAME} has stopped {VERB_CONTINUOUS}. H: {NAME} used to {VERB}. entail | E (1)
PreSup |  P: Martin has stopped drinking. H: Martin used to drink.
T12 |  P: Some of the {OBJECTS} are {COLOUR} in colour. H: Not all of the {OBJECTS} are {COLOUR} in colour. entail | E (1)
Implic. |  P: Some of the banknotes are crimson in colour. H: Not all of the banknotes are crimson in colour.
T13 |  P: {OBJ1} and {OBJ2} lie on the table. {NAME} asked for {OBJ1}. H: {NAME} did not ask for the {OBJ2}. entail | N (.5) E (.5)
Implic. |  P: Toothpaste and eyeliner lie on the table. Jane asked for eyeliner. H: Jane did not ask for the toothpaste.
T14 |  P: {NAME1} asked {NAME2} for {N1} dollars. {NAME2} had {N2 $<$ N1} dollars. H: {NAME2} did not have {N1} dollars. entail | E (1)
Implic. |  P: Donald asked Chris for 200 dollars. Chris had 90 dollars. H: Chris didn’t have 200 dollars.
T15 |  P: {NAME1} is {COM ADJ} than {NAME2} but not {NAME3}. H: {NAME1} is ANTONYM ADJ than {NAME3}. entail | E (1)
Mult. |  P: Evelyn is weaker than Billy but not Michael. H: Evelyn is stronger than Michael.
T16 |  P: {NAME1} asked {NAME2} for {N1} dollars. {NAME2} refused. H: {NAME1} has higher chances of convincing {NAME2} for {N2 $>$ N1} dollars. contradict | C (1)
Mult. |  P: Grace asked Albert for 20 dollars. Albert refused. H: Grace has higher chances of convincing Albert for 40 dollars.

Table 3: We show a few representative Templates (examples), top human
annotated label, and associated confidence (0-1). T15 and T16 are examples of
relatively harder templates requiring multiple capabilities. T15 require
comparative, negation, and syntactic. T16 requires commonsense and numerical
reasoning.

### Template Generation for Reasoning Categories

Automatic template creation from public datasets is not straightforward, as
examples in public NLI datasets represent multiple capabilities (Joshi et al.
2020). Even targeted datasets such as Winograd Schema Challenge (Levesque,
Davis, and Morgenstern 2012) require careful re-annotation, as examples may
require lexical or boolean reasoning. Instead, we resort to manually creating
templates and use human annotations to verify the correctness of templated
instances. In the subsequent paragraphs, we highlight interesting templates
(Tab. 3) and discuss some challenges faced during template creation.
Linguistic. We test basic lexical phenomena such as synonyms, antonyms,
hypernyms and unrelated phrases through Lexical templates. Syntactic templates
test different forms of ellipsis. Paraphrasing is hard to test individually,
as most paraphrases in existing-paraphrase corpora (Dolan and Brockett 2005;
Iyer, Dandekar, and Csernai 2017) are not necessarily entailments, and
paraphrases may involve lexical changes as well.
Logical. Boolean templates test logical and ($\land$), or ($\lor$) and their
combination; ordered resolution (T1, T2) along with inclusion of negation.
Quantifier templates test the understanding of universal (all, $\forall$), and
existential (some, none; $\exists,\neg\exists$) quantification, and the
effects of interchanging them. Comparative templates require inference
involving two or three entities connected using comparative or superlative
adjectives (including templates of insufficient information). For Coreference,
we come up with templates to test gendered (T3), animate vs. inanimate
resolution. Spatial templates test understanding of spatial prepositions and
adverbs indicating relative positions (near, far, above, below, left, right).
We also include a set of templates testing cardinal directions (north, east,
south and west); and some requiring comparison of distances (T4, requiring
both Spatial and Numerical understanding). Temporal templates cover relative
occurrences of events using prepositions, such as before-after and earlier-
later. Another set of templates test the understanding of time in the day (8AM
comes before 2PM), month or year. We add templates for temporally ordered
events (T5) which require the inference of earliest or latest events. Causal
template creation is tricky since often reasoning beyond form is required.
However, with specific controls placed, we can generate accurate causal
premise-hypothesis pairs. We use a set of action-verb pairs which are
complementary (e.g. give-take, lend-borrow, etc.) to describe corresponding
actions between 2 entities and an object (T6). This still results in limited
test cases. Explorations into leveraging knowledge graphs such as ConceptNet,
ATOMIC (Speer, Chin, and Havasi 2016; Sap et al. 2019) to retrieve appropriate
causal action phrases could be a way to tackle the issue. Most Logical
category templates implicitly test the Relational (deductive reasoning with
relations in text) capability, and hence we add three representative
templates. Numerical templates test a basic understanding of counting (T7,
T8), addition, subtraction and numerical comparison.

Knowledge.  For Taxonomic, we create templates where a taxonomic hierarchy
(external to text) is implicitly required to infer the hypothesis – either
requiring to infer A is a type of B (A has some properties. $\Rightarrow$ A is
a type of B) or utilising that information (similar to Talmor et al. (2020b))
(B has property X $\Rightarrow$ A has property X). The scope of World
templates is vast. We add a representative template to test basic knowledge of
geography (city-country pairs, T9)
Pragmatic.  For Pragmatic, we add templates along the lines of Jeretic et al.
(2020). Pre-supposition templates test the existence of objects (T10),
occurrence of events, aspectual verbs (T11) and quantifiers. Implicature
templates are constructed by following Grice’s cooperative principle (Maxims
of quality, quantity, and relevance) (Grice 1975). Most Implicature templates
also require other capabilities such as quantifiers (T12), Boolean (T14), and
Numerical (T15).

####  LoNLI Dataset.

We created a total of 363 templates spanning all 17 capabilities. For each
template, we generate 1000 examples by careful instantiations of the
placeholders present in the template. We ask two independent annotators to
annotate the NLI label for 5 random examples from each template (1815
examples). The average Fleiss’ $\kappa$ $0.81$ (same as Cohen’s) shows very
high inter-annotator agreement. The dataset can be found
here333https://github.com/microsoft/lonli

## LoNLI: Observations and Analysis

The LoNLI dataset enables a multitude of analysis at different granularity.
Through our analysis, we observe following salient points: 1) MultiNLI fine-
tuned systems fare poorly most Logical, Knowledge, and implicature categories.
2) In a post-hoc setting, due to these harder categories, the dataset remains
non-trivial to solve even for systems trained on additional targeted
resources. 3) Most Logical categories require some unique information, that
may not be learnt solely from other capabilities. Apart from this, the
framework of LoNLI allows us to attest to previously found observations and
discover new phenomena-specific ones, such as systems oscillate inexplicably
between logical and pragmatic interpretation for implicature examples. Intra-
template variations help us find out model senstivity towards gendered names
and other lexicons (such as profession or adjectives).

### Analysis of NLI Systems

We analyze four state-of-the-art systems – BERT-base, DistilBERT-base,
RoBERTa-large (Devlin et al. 2019; Liu et al. 2019b) and DeBERTa-large (He et
al. 2020) fine-tuned on MNLI data. Whenever available, we use the MNLI
finetuned models from the Huggingface Transformers
repository444https://huggingface.co/models. Otherwise, we train it using the
script from the repository (Wolf et al. 2019), with the hyper-parameters
suggested by corresponding papers (details are in appendix).

Dataset | BERT | DistilBERT | RoBERTa | DeBERTa
---|---|---|---|---
MNLI-test | 84.5 | 82.2 | 90.2 | 91.3
LoNLI | 53.4 | 50.9 | 66.5 | 69.8
Table 4: Average accuracy on MNLI-test set and LoNLI.

Table 4 shows the accuracy of the models on MNLI test and LoNLI dataset.
Following the trend on MNLI, RoBERTa and DeBERTa outperform BERT and
DistilBERT on LoNLI. But the absolute results highlight a poor performance of
these MNLI-finetuned systems. Next, we analyze further at the level of i)
capabilities, ii) templates, and iii) intra-template granularity.

(a)  Figure 1: (Best viewed in color) For each of the 17 reasoning
capabilities, we show average accuracy for each model  (a)  Figure 2: Showing
fraction of total templates falling in the accuracy bins (0-20), (20-80) and
(80-100) for each model

#### Capability-wise Performance.

From the capability-wise average accuracy (Figure 1(a)), we observe that all
models perform well on Lexical, Syntactic and Presupposition capabilities. For
the Logical categories, the results are comparably poor and inconsistent
across both the capabilities and model dimension. The same holds for the
Knowledge categories and implicature templates. In short, a subset of Logical
categories (comparative, spatial, temporal, quantifier, numerical),
implicature, and taxonomic stands out as relatively harder capabilities across
systems. Through further template-wise analysis, we affirm many previous
observations and discover new ones related to these harder capabilities. For
the ensuing analysis, we mark a template as passed if the model’s accuracy is
above $80\%$, as unsure if the accuracy is between $20$ to $80\%$, and failed
if less than $20\%$.

#### Reproducing Previous Observations.

In temporal, models are sensitive to lexical substitutions (Glockner, Shwartz,
and Goldberg 2018) such as replacing “before” (“after”) with “earlier than”
(“later than”). The models show a label bias towards contradiction in presence
of “not” even if the true label is entailment (Naik et al. 2018). Similar to
Talmor et al. (2020a), in numerical, we find that all models struggle in
comparing age given year of birth. BERT and DistilBERT are unsure for most
templates. RoBERTa and DeBERTa perform better at counting while failing when
the hypothesis refers to an incorrect count. All models struggle with
templates pertaining to addition and subtraction (often showing label bias)
(Pikekos, Michalewski, and Malinowski 2021; Wallace et al. 2019; Naik et al.
2018). On quantifier templates with “some-all” the result of BERT and
DistilBERT are neither consistent with pragmatic nor logical mode of reasoning
(Jeretic et al. 2020). Failure on many templates (for example all models fail
to predict contradiction on P: Sue but not Harry is a dentist. H: Harry is a
dentist.) can be attributed to shallow, fallible syntactic heuristics rather
than “understanding” (McCoy, Pavlick, and Linzen 2019).

#### Uncovering New Findings.

On ordered resolution of varying lengths, we observe that RoBERTa and DeBERTa
are biased towards entailment (unsure for contradiction). Within comparative,
RoBERTa and DeBERTa fail on templates testing associativity of comparison (P:
A is X than B, B is X than C. H: A is X than C). All models fail on templates
where the information is insufficient (P: A is X than C, B is X than C H: A is
X than B). BERT and DistilBERT are unsure when premise or hypothesis requires
reasoning over a superlative adjective. On another set of templates for
counting (P: Mary and Billy are the only children of Joe. H: Joe has exactly 2
children.), we find even RoBERTa and DeBERTa struggle which is in contrast to
Talmor et al. (2020a). This indicates that RoBERTa and DeBERTa may not
actually know how to count, rather may rely on spurious correlations. Within
temporal, all models fail to detect the “first” or the “last” event in a
sequence of three events (A then B then C). On implicature templates, the
models oscillate unexplainably between logical (P: Silverware and plate lie on
the table. Barbara asked for the plate. H: Barbara also asked for the
silverware.; predicting neutral over contradiction) and pragmatic
interpretation (P: Some of the balls are purple in color. H: All of the balls
are purple in color.; predicting contradiction over neutral), depending on
template.
We also analyse intra-template variations, i.e., why some templates fall in
the middle bin ($20$-$80\%$; Fig. 2(b)). Accuracy of a template lying in the
middle bin can be explained using the choice of lexemes for the placeholders.
Even here, we attest to previously observed model sensitivity towards gendered
names (Vig et al. 2020; Bhardwaj, Majumder, and Poria 2020). We find that
models are sensitive in dimension of other lexicons as well such as profession
or adjectives (See Appendix for further analysis).

### Effect of Existing Resources

From Tab. 4, we observed that current SOTA models perform poorly on LoNLI, and
found certain capabilities compratively harder. Venturing further to assess
the hardness, we examine whether further finetuning on specific existing
datasets can help. We select two capability-aware resources, CLCD (aimed at
connectives), and ImpPres (aimed at Pragmatics), and an adversarial resource
ANLI. We use MNLI trained BERT and RoBERTa, and further train them on the
three resources (hyperparameters in Appendix). Fig. 3 summarizes the
capability-wise accuracy of the six variants on selected capabilities of
interest.

(a)  Figure 3: We show the increment in accuracy compared to MNLI variants,
for select capabilities. (C)=CLCD, (I)=ImpPres, (A)=ANLI.

Expectedly, training on ImpPres shows benefit (albeit marginal) on implicature
for both models over their MNLI-variants. ImpPres benefits BERT on most
capabilities. For RoBERTa, most other capabilities suffer except for
taxonomic. Training on CLCD benefits BERT marginally on boolean capability,
while preserving its performance on comparative and quantifier. Even here,
improvements for RoBERTa is minimal. We also see that continued finetuning on
capability-aware datasets is often detrimental to the rest of the
capabilities. Numerical capability suffers the most on both datasets, while
temporal suffers the most on CLCD. ANLI is capability-agnostic and contains
harder examples. We hardly observe any benefit of using both MNLI and ANLI for
both models (all capabilities suffer marginally). For ANLI, the decline in
performance is higher for BERT than RoBERTa. In summary, we observe that LoNLI
is non-trivial to solve even with additional resources.

### Few-shot and Cross-capability Finetuning

Finally, we use the LoNLI dataset (as train-test) to investigate whether the
harder capabilities can be learnt from others (in leave-one-out setting) or in
a few-shot manner.

Figure 4: (Few-shot) Figure shows how RoBERTa-large and RoBERTa-large-MNLI
performance varies with $1-10\%$ of LoNLI training data. For more than $10\%$,
the accuracy varies minimally.

#### Few-shot Finetuning.

We employ few-shot finetuning to gauge if some categories remain harder with
addition of data. We gradually increase training data from $1\%,2\%,10\%,20\%$
to $30\%$ (uniformly sampled for each capability); and finetune pretrained
RoBERTa-large and RoBERTa-large trained on MultiNLI555For each model, we
perform a hyperparameter search using different learning-rate
($1e-5,2e-5,5e-5$), warm-up steps ($0,500$), scheduler (linear, constant), and
training epoch ($3,5$) combinations; and choose the hyperparameters that
provides the highest LoNLI and MultiNLI validation accuracy.. We test on
$20\%$ of the dataset. As shown in Figure 4, pretrained RoBERTa-large performs
very low on all categories with $1\%$ training data, but quickly reaches near
100% accuracy with $2\%$ data. As expected, even with $1\%$ data, RoBERTa-
large-MNLI achieves 100% accuracy on all categories except implicature,
numerical, quantifier and taxonomic.
The curious case of implicature. Some implicature templates stand out as the
outcome may vary based on whether pragmatic or logical interpretation is
employed. Interestingly, RoBERTa-large-MNLI becomes somewhat consistent in
adapting logical interpretations for templates that may have different
outcomes for different interpretations. This is expected, as without
additional input context about which interpretation is expected, model should
be consistent.

#### Cross-capability Finetuning.

Next we design leave-one-out (loo) and bring-one-in (boi) experiments. For
loo, we remove a capability altogether from training and train using $80\%$ of
the remaining data (train/val/test: 64/16/20). For comparison, we create the
loo-all setting where no capability is left out. For the dual setting boi, we
train using $80\%$ of one particular capability (leaving others out). For
both, we test using $20\%$ of entire dataset.

Figure 5: (Cross-capability loo) Capability-wise accuracies after RoBERTa-
large, and RoBERTa-large-MNLI finetuning when an individual (X-axis)
capability is left out. The diag indicates the test accuracies of the left-out
capabilities.

We fine-tune and plot RoBERTa-large and RoBERTa-large-MNLI accuracies in the
loo setting in Figure 5 (BERT results in Appendix). The X-axis denotes the
capabilities that are removed. The Y-axis denotes test accuracy per capability
(left-out capability’s test accuracy for diag, average with error bars for
others). We observe the following patterns.
Can capabilities be acquired by training on others? For both models (in loo),
the average test performance of the left out capabilities remain low (RB-MNLI:
$70.5\pm 20.7$, RB: $64.8\pm 16.64$), showing most capabilities (and our
selected templates) need some unique reasoning or information not provided by
others. The capabilities comparative, conditional, coreference, implicature,
quantifier, spatial, and temporal shows most degradation (compared to loo-
all).
Are there any spurious correlations? Leaving most capabilities seem to improve
quantifier and implicature performance (compared to loo-all scenario). While
implicature performance can be explained, we experiment with RoBERTa-large-
MNLI in boi setting where we train on one capability and test on others. Other
than lexical, numerical, implicature and spatial, each category individually
improves on quantifier ($3.4\pm 10.7$) compared to RoBERTa-large-MNLI model. A
very similar, but prominent improvement happens on numerical ($12.94\pm
11.78$), and spatial ($10.36\pm 14.24$). Interestingly, for spatial and
numerical, there is hardly any information in other templates; showing how
easily these models pick up spurious patterns.
Does MNLI Finetuning help acquire the harder capabilities? For RoBERTa-large
and bert-base (in appendix), finetuning on MNLI followed by other categories
do not seem to help pick up comparative, quantifier, and implicature. From
both models, we observe that for comparative, quantifier, and implicature; the
left-out capability’s accuracy (in loo, see diag in Fig. 5) often degrade when
we use MNLI-finetuned model, compared to the pretrained model. This
underscores the necessity for category-specific evaluation, and enhancement of
NLI systems.

## Conclusion

Most recent NLI datasets lack an explicit connection to cognitive capabilities
(transferable across tasks); primarily due to complexity involved in eliciting
manual annotations about reasoning (or cognitive capability) required to solve
complex examples. Additionally, popular crowdsourced NLI datasets also suffer
from a lack of extensibility, unintended artifacts or bias, and do not enable
fine-grained capability-wise analysis. Using the CheckList framework, we
propose an extensible semi-synthetic capability-aware NLI benchmark ($363$
templates, $363$k examples), covering 17 reasoning types (extending a recently
proposed NLI taxonomy). SOTA NLI systems perform poorly in this dataset, even
with capability-specific existing resources as additional training data.
Further analysis leads us to find a subset of capabilities that are harder
across systems. Through fine-tuning experiments, we find some remain harder
even with addition of training data from the test-suite, and are hard to learn
solely from other capabilities. Furthermore, we design a human study where we
ask humans to predict model behavior given behavioral information about NLI
systems. A $12.5\%$ increase in human prediction accuracy for RoBERTa over
BERT provides an indication that, despite fine-grained inconsistencies,
RoBERTa is more predictable than BERT. Our work shows how behavioral
information may help quantify progress towards systems with more predictable
(therefore trusted; (Jacovi et al. 2021)) behavior.

## References

  * Bhagavatula et al. (2020) Bhagavatula, C.; Bras, R. L.; Malaviya, C.; Sakaguchi, K.; Holtzman, A.; Rashkin, H.; Downey, D.; tau Yih, W.; and Choi, Y. 2020.  Abductive Commonsense Reasoning.  In _ICLR_.
  * Bhardwaj, Majumder, and Poria (2020) Bhardwaj, R.; Majumder, N.; and Poria, S. 2020.  Investigating Gender Bias in BERT.  _CoRR_ , abs/2009.05021.
  * Bowman et al. (2015) Bowman, S.; Angeli, G.; Potts, C.; and Manning, C. D. 2015.  A large annotated corpus for learning natural language inference.  In _EMNLP 2015_ , 632–642.
  * Bowman and Dahl (2021) Bowman, S. R.; and Dahl, G. E. 2021.  What Will it Take to Fix Benchmarking in Natural Language Understanding?  In _NAACL-HLT 2021, Online, June 6-11, 2021_ , 4843–4855. Association for Computational Linguistics.
  * de Vassimon Manela et al. (2021) de Vassimon Manela, D.; Errington, D.; Fisher, T.; Breugel, B. V.; and Minervini, P. 2021.  Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models.  In _EACL_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT 2019, Volume 1 (Long and Short Papers)_ , 4171–4186.
  * Dolan and Brockett (2005) Dolan, B.; and Brockett, C. 2005.  Automatically Constructing a Corpus of Sentential Paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_. Asia Federation of Natural Language Processing.
  * Gilpin et al. (2018) Gilpin, L. H.; Bau, D.; Yuan, B. Z.; Bajwa, A.; Specter, M.; and Kagal, L. 2018\.  Explaining explanations: An overview of interpretability of machine learning.  In _2018 IEEE 5th International Conference on data science and advanced analytics (DSAA)_ , 80–89. IEEE.
  * Glockner, Shwartz, and Goldberg (2018) Glockner, M.; Shwartz, V.; and Goldberg, Y. 2018.  Breaking NLI Systems with Sentences that Require Simple Lexical Inferences.  In _ACL 2018 (Volume 2: Short Papers)_ , 650–655. Melbourne, Australia: Association for Computational Linguistics.
  * Grice (1975) Grice, H. P. 1975.  Logic and Conversation.  In Cole, P.; and Morgan, J. L., eds., _Syntax and Semantics: Vol. 3: Speech Acts_ , 41–58. New York: Academic Press.
  * Gururangan et al. (2018) Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S. R.; and Smith, N. A. 2018.  Annotation Artifacts in Natural Language Inference Data.  In _NAACL_.
  * He et al. (2020) He, P.; Liu, X.; Gao, J.; and Chen, W. 2020.  DeBERTa: Decoding-enhanced BERT with Disentangled Attention.  arXiv:2006.03654.
  * Hewitt and Manning (2019) Hewitt, J.; and Manning, C. D. 2019.  A Structural Probe for Finding Syntax in Word Representations.  In _NAACL-HLT 2019, Volume 1 (Long and Short Papers)_ , 4129–4138. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Iyer, Dandekar, and Csernai (2017) Iyer, S.; Dandekar, N.; and Csernai, K. 2017.  First Quora Dataset Release: Question Pairs.
  * Jacovi et al. (2021) Jacovi, A.; Marasović, A.; Miller, T.; and Goldberg, Y. 2021.  Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, 624–635. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450383097.
  * Jawahar, Sagot, and Seddah (2019) Jawahar, G.; Sagot, B.; and Seddah, D. 2019.  What Does BERT Learn about the Structure of Language?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 3651–3657. Florence, Italy: Association for Computational Linguistics.
  * Jeretic et al. (2020) Jeretic, P.; Warstadt, A.; Bhooshan, S.; and Williams, A. 2020.  Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 8690–8705. Online: Association for Computational Linguistics.
  * Joshi et al. (2020) Joshi, P.; Aditya, S.; Sathe, A.; and Choudhury, M. 2020.  TaxiNLI: Taking a Ride up the NLU Hill.  In _CoNLL_.
  * Jurafsky and Martin (2009) Jurafsky, D.; and Martin, J. H. 2009.  _Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition_.  Pearson Prentice Hall, second edition.
  * Kaushik, Hovy, and Lipton (2020) Kaushik, D.; Hovy, E. H.; and Lipton, Z. C. 2020.  Learning The Difference That Makes A Difference With Counterfactually-Augmented Data.  In _ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Khot, Sabharwal, and Clark (2018) Khot, T.; Sabharwal, A.; and Clark, P. 2018.  SciTaiL: A Textual Entailment Dataset from Science Question Answering.  In McIlraith, S. A.; and Weinberger, K. Q., eds., _AAAI 2018, New Orleans, Louisiana, USA, February 2-7, 2018_ , 5189–5197. AAAI Press.
  * Kim et al. (2019) Kim, N.; Patel, R.; Poliak, A.; Xia, P.; Wang, A.; McCoy, T.; Tenney, I.; Ross, A.; Linzen, T.; Van Durme, B.; Bowman, S. R.; and Pavlick, E. 2019.  Probing What Different NLP Tasks Teach Machines about Function Word Comprehension.  In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_ , 235–249. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Lage et al. (2019) Lage, I.; Chen, E.; He, J.; Narayanan, M.; Kim, B.; Gershman, S.; and Doshi-Velez, F. 2019.  An Evaluation of the Human-Interpretability of Explanation.  _CoRR_ , abs/1902.00006.
  * Levesque, Davis, and Morgenstern (2012) Levesque, H.; Davis, E.; and Morgenstern, L. 2012.  The winograd schema challenge.  In _Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning_. Citeseer.
  * Lipton (2018) Lipton, Z. C. 2018.  The mythos of model interpretability.  _Queue_ , 16(3): 31–57.
  * Liu et al. (2021) Liu, H.; Cui, L.; Liu, J.; and Zhang, Y. 2021.  Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , 13388–13396. AAAI Press.
  * Liu et al. (2019a) Liu, N. F.; Gardner, M.; Belinkov, Y.; Peters, M. E.; and Smith, N. A. 2019a.  Linguistic Knowledge and Transferability of Contextual Representations.  In _NAACL-HLT 2019, Volume 1 (Long and Short Papers)_ , 1073–1094.
  * Liu et al. (2019b) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * McCoy, Pavlick, and Linzen (2019) McCoy, T.; Pavlick, E.; and Linzen, T. 2019.  Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference.  In _ACL_ , 3428–3448. Florence, Italy: Association for Computational Linguistics.
  * Naik et al. (2018) Naik, A.; Ravichander, A.; Sadeh, N.; Rose, C.; and Neubig, G. 2018.  Stress Test Evaluation for Natural Language Inference.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , 2340–2353. Santa Fe, New Mexico, USA: Association for Computational Linguistics.
  * Nie et al. (2020) Nie, Y.; Williams, A.; Dinan, E.; Bansal, M.; Weston, J.; and Kiela, D. 2020.  Adversarial NLI: A New Benchmark for Natural Language Understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
  * Pikekos, Michalewski, and Malinowski (2021) Pikekos, P.; Michalewski, H.; and Malinowski, M. 2021.  Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning.  _ArXiv_ , abs/2106.03921.
  * Poliak et al. (2018a) Poliak, A.; Haldar, A.; Rudinger, R.; Hu, J. E.; Pavlick, E.; White, A. S.; and Van Durme, B. 2018a.  Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation.  In _EMNLP 2018_ , 67–81. Brussels, Belgium: Association for Computational Linguistics.
  * Poliak et al. (2018b) Poliak, A.; Naradowsky, J.; Haldar, A.; Rudinger, R.; and Durme, B. V. 2018b.  Hypothesis Only Baselines in Natural Language Inference.  In _*SEMEVAL_.
  * Ravichander et al. (2020) Ravichander, A.; Hovy, E.; Suleman, K.; Trischler, A.; and Cheung, J. C. K. 2020\.  On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT.  In _Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics_ , 88–102. Barcelona, Spain (Online): Association for Computational Linguistics.
  * Ribeiro, Singh, and Guestrin (2016) Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016.  ” Why should I trust you?” Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 1135–1144.
  * Ribeiro et al. (2020) Ribeiro, M. T.; Wu, T.; Guestrin, C.; and Singh, S. 2020.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 4902–4912. Online: Association for Computational Linguistics.
  * Richardson et al. (2020) Richardson, K.; Hu, H.; Moss, L.; and Sabharwal, A. 2020.  Probing Natural Language Inference Models through Semantic Fragments.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34: 8713–8721.
  * Richardson et al. (2019) Richardson, K.; Hu, H. N.; Moss, L. S.; and Sabharwal, A. 2019.  Probing Natural Language Inference Models through Semantic Fragments.  _ArXiv_ , abs/1909.07521.
  * Salvatore (2019) Salvatore, F. 2019.  Cross-Lingual Contradiction Detection.  https://github.com/felipessalvatore/CLCD.  Commit xxxxxxx.
  * Salvatore, Finger, and Hirata Jr (2019) Salvatore, F.; Finger, M.; and Hirata Jr, R. 2019.  A logical-based corpus for cross-lingual evaluation.  In _Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)_ , 22–30.
  * Sap et al. (2019) Sap, M.; Bras, R. L.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019.  ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning.  In _AAAI 2019_ , 3027–3035. AAAI Press.
  * Schlangen (2021) Schlangen, D. 2021.  Targeting the Benchmark: On Methodology in Current Natural Language Processing Research.  In _ACL: Short Papers_.
  * Schuster et al. (2019) Schuster, T.; Shah, D.; Yeo, Y. J. S.; Roberto Filizzola Ortiz, D.; Santus, E.; and Barzilay, R. 2019.  Towards Debiasing Fact Verification Models.  In _EMNLP-IJCNLP 2019_ , 3419–3425. Hong Kong, China: Association for Computational Linguistics.
  * Speer, Chin, and Havasi (2016) Speer, R.; Chin, J.; and Havasi, C. 2016.  ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.  _CoRR_ , abs/1612.03975.
  * Talmor et al. (2019) Talmor, A.; Elazar, Y.; Goldberg, Y.; and Berant, J. 2019.  oLMpics – On what Language Model Pre-training Captures.  arXiv:1912.13283.
  * Talmor et al. (2020a) Talmor, A.; Elazar, Y.; Goldberg, Y.; and Berant, J. 2020a.  oLMpics-On What Language Model Pre-training Captures.  _Transactions of the Association for Computational Linguistics_ , 8: 743–758.
  * Talmor et al. (2020b) Talmor, A.; Tafjord, O.; Clark, P.; Goldberg, Y.; and Berant, J. 2020b.  Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 20227–20237. Curran Associates, Inc.
  * Tenney, Das, and Pavlick (2019) Tenney, I.; Das, D.; and Pavlick, E. 2019.  BERT Rediscovers the Classical NLP Pipeline.  In _ACL_ , 4593–4601. Florence, Italy: Association for Computational Linguistics.
  * Tenney et al. (2019) Tenney, I.; Xia, P.; Chen, B.; Wang, A.; Poliak, A.; McCoy, R. T.; Kim, N.; Van Durme, B.; Bowman, S.; Das, D.; et al. 2019.  What do you learn from context? Probing for sentence structure in contextualized word representations.  In _ICLR 2019_.
  * Thorne et al. (2018) Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; and Mittal, A. 2018.  FEVER: a Large-scale Dataset for Fact Extraction and VERification.  In Walker, M. A.; Ji, H.; and Stent, A., eds., _NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , 809–819. Association for Computational Linguistics.
  * Vig et al. (2020) Vig, J.; Gehrmann, S.; Belinkov, Y.; Qian, S.; Nevo, D.; Singer, Y.; and Shieber, S. 2020.  Investigating Gender Bias in Language Models Using Causal Mediation Analysis.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M. F.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 12388–12401. Curran Associates, Inc.
  * Wallace et al. (2019) Wallace, E.; Wang, Y.; Li, S.; Singh, S.; and Gardner, M. 2019.  Do NLP Models Know Numbers? Probing Numeracy in Embeddings.  In _Empirical Methods in Natural Language Processing_.
  * Wang et al. (2018) Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2018.  GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding.  _EMNLP 2018_ , 353.
  * Welleck et al. (2019) Welleck, S.; Weston, J.; Szlam, A.; and Cho, K. 2019.  Dialogue Natural Language Inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 3731–3741. Florence, Italy: Association for Computational Linguistics.
  * Williams, Nangia, and Bowman (2018) Williams, A.; Nangia, N.; and Bowman, S. 2018.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference.  In _NAACL-HLT, Volume 1 (Long Papers)_ , 1112–1122. Association for Computational Linguistics.
  * Wittgenstein (1922) Wittgenstein, L. 1922.  Tractatus Logico-Philosophicus.  _London: Routledge, 1981_.
  * Wolf et al. (2019) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  _ArXiv_ , abs/1910.03771.

–

## Analysing bias in LoNLI

Exploiting the semi-synthetic nature, we carefully choose the lexicon (and
their combinations) to control for known dimensions of bias and avoid
stereotypes. Authors in de Vassimon Manela et al. (2021) highlighted the
presence of stereotype (such as gender-profession bias) and skew (such as male
pronouns occurring more than female pronouns). We balance LoNLI against such
stereotypes and get rid of the skew during template instantiation process.
Given a template with placeholders NAME and PROFESSION, the examples are
generated by sampling from the space of {NAME} $\times$ {PROFESSION}. We
manually balance for gender by including equal numbers of male and female
names.

The second class of biases as pointed out by Gururangan et al. (2018) is
certain purious correlations between occurrences of certain lexemes and the
corresponding ground-truth labels. In MNLI for example, use of negation words
such as not or no is often strongly correlated with contradiction. Analysis of
most frequently occurring words – auxiliary verb (is, are), prepositions (in,
to, of), negation (not), temporal (before, after), spatial (left, right),
quantifiers (some, all) roughly match the label-wise distribution of LoNLI as
a whole. The same result is observed for frequently occurring placeholders –
NAME, PRONOUNS, MALE NAME, FEMALE NAME, PROFESSION, COUNTRY, ADJECTIVE,
COMPARATIVE, SUPERLATIVE, YEAR, COUNT, that is label wise distribution of
these placeholders match that of the entire LoNLI dataset.

## Analysis of NLI Systems

### Experiment Setting

#### Details of MNLI trained models.

We use RoBERTa-large trained on MNLI from pytorch
hub666https://pytorch.org/hub/pytorch˙fairseq˙roberta/ and DeBERTa-large
trained on MNLI from huggingface model
hub777https://huggingface.co/microsoft/deberta-large-mnli. BERT and DistilBERT
were not publicly available. So we train bert-base-uncased and distilbert-
base-uncased with learning rate 5e-5, batch size 32, and sequence length 128
for 3
epochs888https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-
classification. The validation was done after each epoch and the best model
was saved.

#### Finetuning on additional resources.

CLCD dataset is a two-way classification task – contradiction and non-
contradiction. So, first we convert the examples to traditional 3-way labeling
task by labelling non-contradiction with neutral and entailment. As CLCD is
synthetic (and templated) in nature, we use the templates to re-label sets of
examples than individual ones. Also CLCD and ImpPres do not provide a separate
test split so we sample 10% data for testing and remove from training. ANLI
has three subsets R1, R2 and R3 which we use in a combined fashion by
contatenating the datasets. For finetuning we start with BERT-base and
RoBERTa-large fine-tuned on MNLI, and train them on these three datasets using
learning rate of 5e-5. Since the datasets are smaller, we experiment with
smaller batch sizes of 4 and 8. For CLCD and ImpPres the sequence length
remains 128 but for ANLI we increase the sequence length to 256 (following
(Nie et al. 2020)). For both, we train the model for 3 epochs and save the
model with the best validation score after each epoch.

### Further Observations and Analysis

#### Template-wise Observations.

We provide model-wise accuracies for certai interesting templates in Tab.
LABEL:tab:app:interestingtemplate. Next, we discuss the template-wise
observations.

T1: BERT, DistilBERT and RoBERTa fail on a simple template testing “but not”.
T2: The models are unsure in presence negation adverb such as hardly, rarely,
scarcely.
T3: All models fail on a simple template testing “or” possibly because of
large overlap in premise and hypothesis.
T4,T5: Between T4 and T5 we just switch the placeholder in hypothesis
retaining the target label but model result shows drastic drop. The means the
models are unsure of ordered resolution. The observation remains consistent
for chain of length 2, 3 and 4.
T6: When “not” is introduced in the hypothesis all models shift the prediction
towards contradiction label (bias also present in MNLI)
T7,T8,T9: Within the comparative capability, we observe that all models
struggle with templates where the information is insufficient to conclude the
hypothesis. We show three representative templates combining comparative and
superlative adjectives on which all models show poor results.
T10: DistilBERT fails on this very basic comparative template requiring
information from preceding sentence.
T11: Both BERT and DistilBERT fail (DeBERTa is unsure) on the quantifier
template testing the understanding of “none” and “some”
T12,T13: Here we replicate the observation from Talmor et al. (2019) and
indeed find that RoBERTa and DeBERTa seem to posses the ability to count (BERT
and DistilBERT fail). That is seen when the hypothesis contains the correct
count but incase of incorrect count we see that even RoBERTa and DeBERTa are
unsure.
T14,T15: We create a similar template to test basic counting. For these set of
templates we find RoBERTa and DeBERTa consistently failing (for both correct
and incorrect count). This observation is consistently seen for count of 2,3,4
and 5.
T16,T17: Through our observation and also pointed out in Talmor et al. (2019)
RoBERTa (and DeBERTa) are able to compare small numbers. However in these set
of spatial templates we find that the models fail to arrive at “near/far” by
comparing distance.
T18,T19,T20: RoBERTa seems to be able to compare year of birth to compare age
but the same can’t be said about BERT and DistilBERT. These three templates
test the exact same phenomenon with just hypothesis written slighlty
differently. We observe BERT being sensitive to the lexical substitution
“before” (or “after”) to “earlier than“ (or “later than”) and “elder” (or
“younger”).
T21,T22: Given three events ordered temporally (A then B then C) all models
fail to conclude the first or the last event.
T23,T24: Pair of exactly same template but differing only in the target label.
All models predict contradiction which is “implicative” reasoning.
T25,T26: Another pair templates pertaining to quantifiers but unlike the last
observation here the models are not unanimous in their prediction. DeBERTa is
“implicative” in its prediction of contradiction. The other three models fail
at both the templates meaning they are neither “implicative” or “logical” on
this template.
T27,T28: A pair of templates pertaining to boolean where we observe that
RoBERTa and DeBERTa are “logical” whereas BERT and DeBERTa fail on both the
pair.

#### Probing intra-template variations

{PROFESSION} | MALE | FEMALE
---|---|---
engineer | 0.00 | 0.01
poet | 0.00 | 0.04
entrepreneur | 0.01 | 0.01
politician | 0.02 | 0.35
writer | 0.03 | 0.07
banker | 0.06 | 0.08
dancer | 0.9 | 0.12
actor | 0.10 | 0.26
painter | 0.10 | 0.33
accountant | 0.12 | 0.35
businessman | 0.16 | 0.32
author | 0.33 | 0.45
singer | 0.49 | 0.77
teacher | 0.62 | 0.63
doctor | 0.80 | 0.77
professor | 0.92 | 0.83
Table 5: Variation in accuracy in BERT for different values of professions and gender {ADJ} | MALE | FEMALE
---|---|---
big | 0.00 | 0.03
sweet | 0.16 | 0.11
smart | 0.25 | 0.20
weird | 0.38 | 0.40
strong | 0.41 | 0.40
tough | 0.47 | 0.48
old | 0.56 | 0.55
tall | 0.56 | 0.61
tiny | 0.99 | 0.99
creepy | 1.00 | 1.00
Table 6: Variation in accuracy of BERT for different values of adjective and
gender

We expand on our analysis of variation within a template arising due to
lexicons by discussing two example templates.

#### Case Study 1 on PROFESSION.

The expected label for the template P: {NAME1}, but not {NAME2}, is a
{PROFESSION}. H: {NAME2} is a {PROFESSION}.  is “contradiction”. We create
examples from this template by varying on two dimensions, 1) first we vary the
gender of NAME2 (we keep NAME1 to be same gender as NAME2 to avoid any
confusion), and then 2) use different values of the PROFESSION lexicon, and
then vary only names. For each combination of the above two we sample 100
examples and report BERT’s accuracy in Table 5. As mentioned, accuracy varies
strongly with the change in lexicon for PROFESSION. Template accuracy varies
from low when profession is set to engineer ($0\%$), poet ($0\%$) to high when
profession is set to doctor ($0.82\%$), professor ($0.92\%$).

#### Case Study 2 on ADJECTIVE.

Next, we select another template: P: {NAME1} is {ADJ}. {NAME2} is {COM ADJ}.H:
{NAME2} is {COM ADJ} than {NAME1}. For this template, expected label is
entailment. Similar to before, we generate examples conditioned on ADJ and
gender. Table 6 shows the accuracy for different combinations of gender and
ADJ. Compared to the previous template, we notice that the accuracies are
almost unchanged across the gender dimension. This means that compared to ADJ,
BERT shows bias towards certain professions PROFESSION conditioned on gender.
Here also template accuracy varies from low when adjectives are bigger
($0\%$), sweeter ($0.16\%$) to high when adjectives are tiny ($0.99\%$),
creepier ($1.00\%$).

### Cross-Capability Finetuning

(a) RoBERTa-large and RoBERTa-large-MNLI

(b) BERT-base and BERT-base-MNLI

Figure 6: (Cross-capability loo) (a) Capability-wise accuracies after RoBERTa-
large, and RoBERTa-large-MNLI finetuning when an individual (X-axis)
capability is left out. In Y-axis we show the mean of the accuracies over all
other capabilities with hues that reflect error bars. The diag indicates the
test accuracies of the left-out capabilities. (b) Similar figure for bert-
base-uncased as the pre-trained model. We show loo results for both pre-
trained and MNLI-finetuned variants. In many capabilities, the MNLI-finetuned
variant does not improve over the pre-trained one on the diag accuracies,
which denotes the accuracies of the corresponding left-out capability.

In the Fig. 6, we show how RoBERTa-large and BERT-base behaves in the loo
setting (before and after finetuning on MNLI). Each experiment was run 5 times
with different seeds, and average was reported. Hyperparametr setting
discussion has been included in the main paper.

# | Template | Performance (Accuracy %)
---|---|---
| BERT | DistilBERT | RoBERTa | DeBERTa
T1 |  P: {NAME1} but not {NAME2} is {A} {PROFESSION}. H: {NAME2} is {A} {PROFESSION}. contradict | 22.50 | 31.70 | 19.20 | 82.60
Neg. |  Frances but not Jerry is a designer. H: Jerry is a designer.
T2 |  P: {NAME} {V5 VERB} {OBJECT}. H: {NAME} {NEGATION ADVERB} {VERB} {OBJECT}. contradict | 63.30 | 79.80 | 45.40 | 44.60
Neg. |  Alison helps the stranger. H: Alison rarely help the stranger.
T3 |  P: {NAME1} or {NAME2} are {ADJECTIVE}. H: {NAME1} is {ADJECTIVE}. neutral | 0.00 | 0.00 | 0.10 | 1.90
Bool. |  Kelly or Frederick are industrious. H: Kelly is industrious.
T4 |  P: {NAME1} and {NAME2} are from {COUNTRY1} and {COUNTRY2} respectively. H: {NAME1} is from {COUNTRY2}. contradict | 90.70 | 63.30 | 94.80 | 99.50
Bool. |  Katherine and Daniel are from Libya and India respectively. H: Katherine is from India.
T5 |  P: {NAME1} and {NAME2} are from {COUNTRY1} and {COUNTRY2} respectively. H: {NAME2} is from {COUNTRY1}. contradict | 0.10 | 10.70 | 1.40 | 24.60
Bool. |  Jack and Eleanor are from Fiji and Iran respectively. H: Eleanor is from Fiji.
T6 |  P: {NAME1} and {NAME2} are from {COUNTRY1} and {COUNTRY2} respectively. H: {NAME1} is not from {COUNTRY2}. entail | 0.00 | 0.00 | 0.00 | 0.00
Bool. |  Anthony and Karen are from Iran and Italy respectively. H: Anthony is not from Italy.
T7 |  P: {NAME1} is {COM ADJ} than {NAME2}. {NAME1} is {COM ADJ} than {NAME3}. H: {NAME2} is {COM ADJ} than {NAME3}. neutral | 0.00 | 0.00 | 0.00 | 0.00
Comp. |  P: James is hungry. Jessica is hungrier. H: James is hungrier than Jessica.
T8 |  P: Among {NAME1}, {NAME2} and {NAME3} the {SUP ADJ} is {NAME1}. H: {NAME2} is {COM ADJ} than {NAME3}. neutral | 16.70 | 2.00 | 0.30 | 3.80
Comp. |  Sophie is older than Steven. Sophie is older than Betty. H: Steven is older than Betty.
T9 |  P: {NAME1} is {COM ADJ} than {NAME3}. {NAME2} is {COM ADJ} than {NAME3}. H: Among {NAME1}, {NAME2} and {NAME3} the {SUP ADJ} is {NAME1}. neutral | 1.70 | 0.70 | 0.60 | 0.00
Comp. |  Among Edwin, Anna and Dave the richest is Edwin. H: Anna is richer than Dave.
T10 |  P: {NAME1} is {ADJ}. {NAME2} is {COM ADJ}. H: {NAME1} is {COM ADJ} than {NAME2}. contradict | 43.40 | 1.40 | 95.70 | 97.30
Comp. |  Larry is gloomier than Kelly. Ralph is gloomier than Kelly. H: Among Larry, Ralph and Kelly the gloomiest is Larry.
T11 |  P: None of the {OBJECTS} are {COLOUR} in colour. H: Some of the {OBJECTS} are {COLOUR} in colour. contradict | 0.00 | 0.00 | 99.10 | 69.20
Quan. |  None of the combs are red in colour. H: Some of the combs are red in colour.
T12 |  P: {NAME1} is the child of {NAME0}. {NAME2} is the child of {NAME0}. {NAME3} is the child of {NAME0}. {NAME4} is the child of {NAME0}. H: {NAME0} has atleast 4 children. entail | 0.00 | 0.00 | 100.00 | 100.00
Num. |  Christine is the child of Ron. Deborah is the child of Ron. Henry is the child of Ron. Carolyn is the child of Ron. H: Ron has atleast 4 children.
T13 |  P: {NAME1} is the child of {NAME0}. {NAME2} is the child of {NAME0}. {NAME3} is the child of {NAME0}. {NAME4} is the child of {NAME0}. H: {NAME0} has only {N $<$ 4} children. contradict | 16.70 | 55.00 | 49.90 | 73.20
Num. |  Rebecca is the child of Caroline. Joseph is the child of Caroline. Ron is the child of Caroline. Rachel is the child of Caroline. H: Caroline has only 3 children.
T14 |  P: {NAME1}, {NAME2}, {NAME3} and {NAME4} are the only children of {NAME0}. H: {NAME0} has exactly 4 children. entail | 0.00 | 0.00 | 0.50 | 0.00
Num. |  Michelle, Steve, Christopher and Mike are the only children of Sam. H: Sam has exactly 4 children.
T15 |  P: {NAME1}, {NAME2}, {NAME3} and {NAME4} are the only children of {NAME0}. H: {NAME0} has exactly {N $\neq$ 4} children. contradict | 18.30 | 100.00 | 59.60 | 90.90
Num. |  Arthur, Paul, Katherine and Stephen are the only children of Ian. H: Ian has exactly 1 children.
T16 |  P: {CITY1} is {MILE1} miles from {CITY2} and {MILE2 $<$ MILE1} miles from {CITY3}. H: {CITY1} is nearer to {CITY2} than {CITY1} is to {CITY3}. contradict | 0.00 | 0.00 | 0.00 | 19.00
Spat. |  Boston is 865 miles from New York and 420 miles from Houston. H: Boston is nearer to New York than Boston is to Houston.
T17 |  P: {CITY1} is located between {CITY2} and {CITY3}. H: {CITY3} is nearer to {CITY1} than {CITY3} is to {CITY2}. entail | 0.90 | 0.00 | 0.00 | 0.10
Spat. |  Madison is located between Dallas and San Jose. H: Dallas is nearer to Madison than Dallas is to San Jose.
T18 |  P: {NAME1} was born in {YEAR1} and {NAME2} was born in {YEAR2 $<$ YEAR1}. H: {NAME2} was born before {NAME1}. entail | 6.50 | 67.50 | 93.00 | 43.20
Temp. |  Alexandra was born in 1992 and Rebecca was born in 1991. H: Rebecca was born before Alexandra.
T19 |  P: {NAME1} was born in {YEAR1} and {NAME2} was born in {YEAR2 $<$ YEAR1}. H: {NAME2} is elder than {NAME1}. entail | 84.90 | 0.00 | 85.40 | 92.20
Temp. |  Patrick was born in 1996 and Thomas was born in 1995. H: Thomas is elder than Patrick.
T20 |  P: {NAME1} was born in {YEAR1} and {NAME2} was born in {YEAR2 $<$ YEAR1}. H: {NAME2} was born earlier than {NAME1}. entail | 92.30 | 98.20 | 98.70 | 99.80
Temp. |  Andrea was born in 2001 and Ruth was born in 2000. H: Ruth was born earlier than Andrea.
T21 |  P: {NAME} has {SUBJECT1} class then {SUBJECT2} class then {SUBJECT3} class. H: {SUBJECT1} class is the first class among the three. entail | 0.00 | 2.00 | 0.00 | 83.90
Temp. |  Larry has art class then political science class then chemistry class. H: Art class is the first class among the three.
T22 |  P: {NAME} has {SUBJECT1} class then {SUBJECT2} class then {SUBJECT3} class. H: {SUBJECT3} class is the last class among the three. entail | 0.00 | 0.00 | 0.00 | 98.50
Temp. |  Bob has history class then geography class then economics class. H: Economics class is the last class among the three.
T23 |  P: Some of the {OBJECTS} are {COLOUR} in colour. H: All of the {OBJECTS} are {COLOUR} in colour. contradict | 100.00 | 100.00 | 100.00 | 100.00
Impl. |  Some of the cases are purple in colour. H: All of the cases are purple in colour.
T24 |  P: Some of the {OBJECTS} are {COLOUR} in colour. H: All of the {OBJECTS} are {COLOUR} in colour. neutral | 0.00 | 0.00 | 0.00 | 0.00
Quan. |  Some of the coins are white in colour. H: All of the coins are white in colour.
T25 |  P: All of the {OBJECTS} are {COLOUR} in colour. H: Some of the {OBJECTS} are {COLOUR} in colour. contradict | 0.00 | 0.00 | 27.50 | 100.00
Impl. |  All of the postcards are pink in colour. H: Some of the postcards are pink in colour.
T26 |  P: All of the {OBJECTS} are {COLOUR} in colour. H: Some of the {OBJECTS} are {COLOUR} in colour. entail | 0.00 | 67.90 | 7.80 | 0.00
Quan. |  All of the lighters are brown in colour. H: Some of the lighters are brown in colour.
T27 |  P: {OBJECT1} and {OBJECT2} lie on table. {NAME} asked for {OBJECT1}. H: {NAME} also asked for {OBJECT2}. contradict | 0.00 | 0.00 | 0.00 | 0.00
Impl. |  Perfume and knife lie on table. Jane asked for perfume. H: Jane also asked for knife.
T28 |  P: {OBJECT1} and {OBJECT2} lie on table. {NAME} asked for {OBJECT1}. H: {NAME} also asked for {OBJECT2}. neutral | 13.40 | 3.90 | 100.00 | 100.00
Bool. |  Snack and spoon lie on table. Benjamin asked for snack. H: Benjamin also asked for spoon.

Table 7: We show some interesting Templates, and model accuracies on the
templates.

## Consulting Humans to Quantify Model Inconsistency

The inconsistencies observed for both BERT and RoBERTa begs the question as to
how to quantify the progress towards models with increased predictable
behavior999Training machine learning models to predict behavior adds more
confounders that can affect the analysis.. Inspired by the recent XAI
literature (Gilpin et al. 2018; Lipton 2018; Lage et al. 2019), we design a
human study where humans are shown different types of post-hoc behavioral
information and asked to predict system’s behavior on new examples. This
premise presents us with many dimensions of control, which may affect the
outcome of the study: 1) the level of abstraction of behavioral information,
2) example-based (local) vs. global summary, 3) interface design (or
presentation of such information), 4) baseline explanation, 5) choice of test
examples, 6) task (verification, simulation or counterfactual), 7) choice of
participants.

Figure 7: Landing Page Instructions to participants. Study in
https://bit.ly/3huowZG

#### Local Explanations and Interface Design.

From multiple pilot studies using global behavioral summary (such as Template-
wise accuracy scores), and relevant template-level local summaries
101010Current study (Stage B1): checklist-nli.herokuapp.com/A, we observe that
global template-wise accuracies have large cognitive load for participants,
and template-wise aggregate accuracies are hard to comprehend (and predict)
without the knowledge of the lexicon that the keywords represent (owing to
intra-template inconsistencies). Hence, in this study, we fixate on providing
local example-based explanations. For the explanations interface design, we
follow Lage et al. (2019). Through various pilot studies, authors in Lage et
al. (2019) found that the use of three intuitive boxes – namely, the input
box, the explanation box, and the question box; makes the information easier
to follow. We observe that this vastly improves relative to our earlier pilot
studies based on Google Forms-based questionnaire.

#### Test Example Selection and Baseline.

To choose test examples, we choose 25 test templates111111This study was done
on a smaller subset of LoNLI carefully by balancing templates representing
different accuracy buckets, and ensuring inclusion of multiple capability-
templates. For each template, we show 5 random test examples. As a baseline,
we consider example-based LIME (Ribeiro, Singh, and Guestrin 2016)
explanations. For each test example, five nearest neighbor examples are
chosen, where the top three attended words (from Premise and hypothesis) are
highlighted using LIME output. The variations are created by varying where the
nearest neighbors come from. In Stage 1, we choose the nearest neighbors from
the MultiNLI validation set and Stage 2 includes nearest neighbors from LoNLI
dataset respectively (barring the corresponding test template examples). To
calculate nearest neighbors, we use the underlying system’s (BERT/RoBERTa)
final hidden-layer embeddings (corresp. to [CLS] token) and calculate cosine
distances.

#### Task, Metrics and Participants.

In this study, we restrict ourselves to simulation questions, where the
participants are asked to simulate (anticipate) an underlying blackbox
system’s (instructions in Fig. 7) prediction given explanation and the input.
Since Transformers often show different types of bias and its not known
whether they follow human reasoning, we track two metrics: 1) prediction
accuracy, 2) mutual agreement score. Consuming such examples and being able to
generalize based on explanations on nearest neighbors require a certain amount
of analytical reasoning skills. Hence, instead of going to crowd-sourced
platforms, we choose a total of 10 International Linguistics Olympiad
participants121212https://www.ioling.org/. Such participants are trained on
analytical text-based puzzle solving, but not trained in formal Linguistics or
Logic.

### Findings and Observations

Figure 8: Average Accuracy and Mutual pairwise agreement for participants (out
of 125).

In Figure 8, we report results from Stages 1 and 2 for BERT (referred to as
B1, B2) and Stage 2 for RoBERTa (R2). The average prediction accuracy of the
participants increase from $58.7\pm 6.8$ (out of 125) to $61.4\pm 6.8$ from
Stage B1 to B2. Similarly, average mutual agreement among participants
increases from $74.75$ to $79.8$, showing that the nearest neighbors from
LoNLI (representing disentangled phenomena) enable better explanation. Upon
interviewing participants, some mentioned their responses in stage B1 was
often “random”. This is intuitive as MNLI examples are often quite complex
with long sentences. So, we repeat only stage 2 for RoBERTa (R2). For R2, we
clearly see a $12.5\%$ improvement in prediction accuracy with a $2.62\%$
improvement in agreement score. This indicates that even though
inconsistencies exists for both models, participants were able to anticipate
RoBERTa’s behavior better.

We also recorded responses about the different stages of study from
participants. Most participants found that nearest neighbor examples were most
relevant for B2 (5 out of 8) and R2 (4 out of 8). The LIME-based highlights
and predicted labels were both useful in B2 and R2. A participant commented
that ”The highlighted words and predictions were very helpful. Relied
completely on both”. Interestingly, the participants were only told that the
systems are different for B2 and R2, without revealing any further detail.
But, most people ($60\%$) found that its easier to predict the system’s
behavior for R2. Participants mentioned that task interface was easy to
navigate and they had no difficulty understanding the instructions.

## Summary of Pilot Studies

We carry out series of pilot studies varying user interface, explanation
granularity and type of behavioral information. We choose 4 (final-year)
undergraduate Computer Science students, who have some experience in NLP
through projects, but not trained on Linguistics and Logic.

### Phase 1: Global Explanation, Template-wise Accuracy Prediction

Stage | 1 | 2 | 3 | 1 | 2 | 3
---|---|---|---|---|---|---
| Average Score | Argmax Bin Accuracy
P1 | 1.4 | 1.08 | 1 | 0.28 | 0.24 | 0.16
P2 | 0.84 | 0.96 | 0.88 | 0.12 | 0.12 | 0.12
P3 | 1.16 | 1.68 | 1.64 | 0.24 | 0.40 | 0.40
P4 | 1.36 | 1.4 | 1.36 | 0.32 | 0.28 | 0.24
| Pearson Correlation | Argmax Bin Consensus
P1 & P2 | 0.41 | 0.14 | 0.29 | 0.52 | 0.52 | 0.60
P3 & P4 | 0.02 | 0.60 | 0.62 | 0.32 | 0.72 | 0.60
Table 8: Phase 1 Pilot Study Results using Global observation summaries.
Average score (out of 5) is the average bid on the right accuracy bin, across
25 test templates. Pearson Correlation is calculated between the bid put in
the correct bin by the two participants. “Argmax bin” indicates metrics where
we consider where the participants have placed their maximum bid, ignoring the
bid value. Consensus is the number of times both participants placed their
highest bid in the same bin (not necessarily the correct one).

The first phase of our pilot studies involved a Google Form-based
questionnaire as an interface131313https://forms.gle/kzqB2Luqub97mKQd6. We
envisioned a training stage, where the participants will be shown behavioral
summary of a blackbox NLI system; and a testing stage where participants will
be asked to place bids (out of 5) on the most accurate bin(s) for unseen
templates. First, we chose these 25 test templates, similarly in the final
study. Then, we design the three variants (or stages) by varying levels of
behavioral information (abstract to fine-grained). In Stage 1, we only
provided the NLI label-wise accuracies for the models. In Stage 2, we provided
Template wise accuracies for all remaining templates from the test-suite (at
that point 100). Then, on Stage 3, we carefully chose 30 related templates and
added short textual description for each. Since, the behavioral information
was at a global level, the participants found it very hard to successfully
utilize the information for each specific template. Moreover, the underlying
system BERT was severely inconsistent even within templates. The combined
effects of these factors resulted in particpants’ predictions to be near
random and the performance hardly improved over the first stage (shown in
Table. 8). However, in some cases, the relevant fine-grained behavioral
information seemed to improve agreement score among participants.

### Phase 2: Local Explanation, Template-level vs. Example-wise Questions

|  | Template-Based
---
Explanations
| Example-based
---
Explanations
| Example-based
---
Q+E
Accuracy | 14.5 | 17.0 | 22.0
Agreement | 15 | 19.0 | 20.0

Table 9: Pilot Study Phase 2 (Based on Website Interface): Average accuracy
and agreement scores for local question-wise explanations. For first two
columns, we stick to template-based questions, where participants are required
to predict the correct accuracy bin. For the last, we move to example-based
questions and explanations. All scores are out of 25.

Using the lessons from the first pilot study, we moved on to local
explanations. Using a web-based user interface designed according to (Lage et
al. 2019), we carried out pilot studies for overall three variations: 1)
template-based questions and explanations, 2) template-based questions and
example-based explanations, and 3) example-based questions and explanations.
We proceed with same way of selecting 25 test templates and then Template-
based questions are formulated in the similar manner as pilot study 1. In each
page (for a Test Template), the user is shown a template (and an example), and
asked to predict the most accurate bin for the underlying System X out of 5
accuracy bins. For template-based explanations, we manually chose three useful
templates and highlighted a total of 5 examples with LIME output (top 3 words
of the premise and the hypothesis). For example-based explanations, we used
nearest neighbor examples from LoNLI dataset to show 5 nearest examples (again
highlighted with LIME output). Lastly, for example-based questions, we removed
all information about templates (from instructions and pages). On the
questions box, we instead ask participants to predict the entailment labels
for 5 random examples from the chosen test template. In the left, we show
nearest neighbor example-based explanations, similar to the final study. We
clearly see from the pilot study (Tab. 9), that accuracies gradually improved
as we moved towards both example-based questions and explanations.