# Bertrand-DR: Improving Text-to-SQL using a Discriminative Re-ranker

Amol Kelkar, Rohan Relan, Vaishali Bhardwaj
Saurabh Vaichal, Chandra Khatri, Peter Relan
Got It AI R&D
<EMAIL_ADDRESS>
###### Abstract

To access data stored in relational databases, users need to understand the
database schema and write a query using a query language such as SQL. To
simplify this task, text-to-SQL models attempt to translate a user’s natural
language question to corresponding SQL query. Recently, several generative
text-to-SQL models have been developed. We propose a novel discriminative re-
ranker to improve the performance of generative text-to-SQL models by
extracting the best SQL query from the beam output predicted by the text-to-
SQL generator, resulting in improved performance in the cases where the best
query was in the candidate list, but not at the top of the list. We build the
re-ranker as a schema agnostic BERT fine-tuned classifier. We analyze relative
strengths of the text-to-SQL and re-ranker models across different query
hardness levels, and suggest how to combine the two models for optimal
performance. We demonstrate the effectiveness of the re-ranker by applying it
to two state-of-the-art text-to-SQL models, and achieve top 4 score on the
Spider leaderboard at the time of writing this article.

##  1 Introduction

Data is generally stored in structured form through relational databases Codd
(1970). SQL is the domain specific language used to query and manage data
stored in most commercially available relational databases Ramakrsihnan et al.
(1998). Although powerful, SQL is hard to master and thus out of reach for
many users who need to query data. As a result, various Natural Language
Interfaces to Databases (NLIs) Grosz et al. (1987) have been investigated. One
approach towards NLI is to use a text-to-SQL semantic parser to convert a
natural language utterance into a SQL query. With the recent developments
related to deep learning, a large number of neural network based models such
as IRNet Guo et al. (2019), and GNN Bogin et al. (2019a) have been developed
for this task.

Many recent text-to-SQL models use a sequence-to-sequence approach, encoding a
sequence of natural language tokens and then decoding a sequence of SQL tokens
Lin et al. (2019). During each decoding step, these models produce a
probability distribution over a set of tokens such as SQL query tokens,
production rules Bogin et al. (2019a) or IR tokens Guo et al. (2019). The
output SQL is produced either using a greedy best-first search Doran and
Michie (1966), i.e. producing the most probable token at each time step, or
using a beam search Reddy et al. (1977), i.e. keeping a set of most probable
sequences of tokens and selecting the most probable sequence at the end.

A text-to-SQL parser is a generative model of the joint probability
distribution Ng and Jordan (2002) between natural language questions, schema
information and correct SQL queries. This is a very complex task and models
have had limited success unless the problem space is constrained using various
techniques such as using intermediate representation Guo et al. (2019) or
using a grammar-based decoder Lin et al. (2019). Here we explore a novel
approach to improve the performance of text-to-SQL models.

Intuitively, it is significantly easier to recognize correct SQL from a set of
candidate queries compared to writing correct SQL for a given utterance and
schema. So, rather than putting all the burden on the generative model and the
greedy/beam search for producing the correct query, we can use a discriminator
to perform a better search among candidate queries produced by the generator.
Also, compared to a generative model, discriminators are relatively easier to
train and have better accuracy because they need to model only the conditional
probability distribution instead of the joint probability distribution Ng and
Jordan (2002).

When used with a beam search Reddy et al. (1977), a text-to-SQL model produces
a set of candidate SQL queries. It is observed that the correct query may be
in the set of candidates, but not at the top of the list when sorted by token
generation probabilities. We use a discriminative re-ranker Collins and Koo
(2005) to identify the correct query from the set of candidates. It
significantly improves the performance of text-to-SQL models. For example, our
re-ranker improves SQL logical form accuracy of GNN Bogin et al. (2019a) from
51.3% to 57.9%, when trained on the Spider Yu et al. (2018) dataset.

Some of the main contributions of this paper include: 1) We present a novel
discriminative re-ranker for the text-to-SQL task, 2) We show that the re-
ranker can improve multiple state-of-the-art text-to-SQL models, and 3) We
demonstrate how to pair a text-to-SQL model with corresponding re-ranker for
best performance.

We make the code for our model available at https://github.com/amolk/Bertrand-
DR.

##  2 Problem definition and notation

Given a natural language utterance $u$ and schema information $s$, a text-to-
SQL model generates a valid SQL query $q$ such that when executed against a
database conforming to schema $s$, the query $q$ would produce answer to the
question $u$.

text-to-SQL semantic parsing can be framed as a supervised learning task, to
induce a function $\mathcal{F}:(\mathcal{U},\mathcal{S})\to\mathcal{Q}$ given
each training example $(u,s,q)$, where $u\in\mathcal{U}$, the set of potential
natural language utterances, $s\in\mathcal{S}$, the set of potential database
schemas, and $q\in\mathcal{Q}$, the set of potential SQL queries. We define
$\mathcal{G}(u,s)\subset\mathcal{Q}$ to be the set of candidates SQL queries
generated for a given input $(u,s)$. The model assigns probability
$\mathcal{P}(u,s,q)$ for each $(u,s)$ and $q$ pair. The most likely parse for
each $(u,s)$ is

$q_{greedy}=\operatorname*{argmax}_{q\in\mathcal{G}(u,s)}\mathcal{P}(u,s,q)$
(1)

$\mathcal{P}(u,s,q)$ is defined as the cumulative probability of the sequence
of tokens $\\{q_{0},q_{1},\ldots,q_{t}\\}$ generated for $q$.

$\mathcal{P}(u,s,q)=\prod_{i=0\ldots
t}\mathcal{P}(q_{i}|u,s,\\{q_{0},\ldots,q_{i-1}\\})$ (2)

It is not practical to search through all possible candidate SQL queries
$\mathcal{G}(u,s)$, so a beam search is often used to produce a subset of
approximately best candidates $\mathcal{C}\subset\mathcal{G}(u,s)$. The
candidate with the highest probability $q_{top}\in\mathcal{C}$ is used as the
prediction of the model. The correct query $q_{gold}$ may be

  * •

at the top of $\mathcal{C}$, i.e. $q_{gold}=q_{top}$,

  * •

somewhere else within $\mathcal{C}$, i.e. $q_{gold}\in\mathcal{C}$ and
$q_{gold}\neq q_{top}$, OR

  * •

it may not be present in the candidates list, i.e.
$q_{gold}\notin\mathcal{C}$.

The first two cases together are considered a ”beam hit” and the last case is
a ”beam miss”. First case represents correct prediction by the text-to-SQL
model. A discriminative re-ranker can be trained to improve accuracy in the
second case, where $q_{gold}$ was predicted but it isn’t at the top of the
beam.

##  3 Our approach

We build Bertrand-DR, a Discriminative Re-ranker as a binary classifier by
fine tuning BERT Devlin et al. (2019), to predict whether a given candidate
query $q$ is the gold query for given utterance $u$ and schema information
$s$.

First, utterance $u$ and query $q$ are encoded using BERT. We use [SEP] to
separate utterance and the query. We use WordPiece Wu et al. (2016) to
tokenize utterance into utterance tokens $u_{1}...u_{N}$ and query into query
tokens $q_{1}...q_{M}$, where $N$ and $M$ are the token counts for utterance
and query, respectively. The tokens are combined as follows to form the input
token sequence:

[CLS], $u_{1}$, $u_{2}$, ... $u_{N}$, [SEP], $q_{1}$, $q_{2}$, ... $q_{M}$,
[SEP]

BERT[CLS] how manyQuestionClassifierGoldNot Gold$H_{[CLS]}$[SEP] select count
( SQL Query Figure 1: Bertrand-DR network architecture

[CLS] and [SEP] are special tokens for classification and context separation,
as used in Devlin et al. (2019). The token sequence is then encoded using
BERT. We take the last layer’s hidden state for the first token, $H_{[CLS]}$,
as the input embedding. $H_{[CLS]}$ is passed through a linear layer with tanh
activation function to form a pooled encoding. The resulting encoding is
passed through a linear classification layer and the model is trained using
binary cross entropy loss.

During inference, given utterance $u$ and query $q$, the model produces
$\mathcal{P}(q|u)$, the probability that the given query is correct for the
utterance. Note that schema information is not used. The probability is used
as the $score$ for the query.

The beam search candidates produced by a text-to-SQL model are sorted by the
generation probability of each candidate. This is the initial ranking of the
candidates. The goal of the re-ranker is to predict a high score value for
$q_{gold}$, if it is present in the candidates list, and lower score values
for non-gold queries, such that $q_{gold}$ would end up at the top of the list
when re-ordered by descending score values. An ideal re-ranker would ensure
that whenever $q_{gold}$ is present in the candidates list, it is re-ranked to
the top of the list. Thus, beam hit rate represents the theoretical maximum
possible performance of a re-ranker.

In some cases, the text-to-SQL model produces the gold query at the top of the
beam, but the re-ranker erroneously promotes a different candidate to the top.
Thus, fully relying on the re-ranker’s scores results in lower accuracy. We
use an empirically derived threshold value $t$, such that a candidate is moved
higher in order only if it has significantly larger score, i.e. larger at
least by threshold $t$ compared to the candidate immediately above it. See
Algorithm 1 and figure 2.

Data: List of candidate SQL queries $\mathcal{C}$, Utterance $u$, Schema
information $s$, a re-ranker $model$, and threshold $t$

Result: SQL queries list $\mathcal{C}$ with the best candidate at the top

foreach  _$c\in\mathcal{C}$_ do

$c.score={model}(u,c)$

end foreach

for  _i = C.length to 1_ do

if  _$C_{i}.score\geq C_{i-1}.score+t$_ then

swap($C_{i}$, $C_{i-1}$)

end if

end for

Algorithm 1 Re-ranking algorithm

##  4 Experiments and results

System | Dev | Test
---|---|---
GNN Bogin et al. (2019a) | 40.7 | 39.4
\+ Re-impl Bogin et al. (2019b) | 44.1 | –
\+ BERT (ours) | 51.3 | –
\+ Bertrand-DR (ours) | 57.9 | 54.6
EditSQL Zhang et al. (2019) | 36.4 | 32.9
\+ BERT Zhang et al. (2019) | 57.6 | 53.4
\+ Bertrand-DR (ours) | 58.5 | –
Table 1: Logical form accuracy of various models on Spider dev and test datasets. Note that results on the Spider test dataset are available only for Spider leaderboard submissions.  System | Threshold | Easy | Medium | Hard | Extra Hard | Overall
---|---|---|---|---|---|---
GNN Bogin et al. (2019a) |  |  |  |  |  |
\+ BERT (ours) |  | 70.4 | 54.1 | 35.6 | 28.2 | 50.7
\+ Bertrand-DR (ours) | 0.10 | 81.6 | 62.5 | 39.1 | 30.6 | 57.9
EditSQL |  |  |  |  |  |
\+ BERT Zhang et al. (2019) |  | 78.0 | 60.0 | 45.4 | 33.5 | 57.5
\+ Bertrand-DR (ours) | 0.68 | 81.6 | 59.5 | 47.1 | 33.5 | 58.5
Table 2: Logical form accuracy at various query hardness levels on Spider dev
dataset. The re-ranker improved GNN results at all hardness levels, but for
EditSQL, only the easy queries show improvement. The threshold levels were
determined empirically to achieve best Overall performance.

We apply our re-ranker on beam candidates generated by two state-of-the-art
Text-to-SQL models, GNN Bogin et al. (2019a) and EditSQL Zhang et al. (2019).

###  4.1 GNN

The GNN Bogin et al. (2019a) model is a grammar based semantic parser that
uses a Graph Neural Network Scarselli et al. (2008) to encode schema entity
relationships. The GNN model uses learned word embeddings. First, we modify
GNN to leverage BERT embeddings. We use the scheme proposed by Yu et al.
(2018) to compute question and schema entity embeddings using BERT. These
embeddings are then used by GNN to encode the question, schema entities and to
compute the schema linking. This improves the SQL logical form accuracy of GNN
on the Spider Yu et al. (2018) dev dataset from 44.1% to 51.3%.

We train GNN model on the Spider dataset and then collect all beam candidates
it produces on training and dev sets, using beam size 40. The collected
dataset of beam candidates serves as training and dev datasets for the re-
ranker.

When the correct query is present in the set of beam candidates, it is counted
as a beam hit. Beam hit rate is the percent examples that result in a beam hit
and it represents the theoretical maximum possible performance by the re-
ranker. Beam hit rate for GNN was 62.4% and the re-ranker produced accuracy
close to the theoretical max accuracy.

| Base | \+ Bertrand-DR | Beam hit
---|---|---|---
GNN | 51.3 | 57.9 | 62.3
EditSQL | 57.6 | 58.5 | 78.7
Table 3: Base indicates base model logical form accuracy on Spider dev
dataset. Beam hit is the theoretical maximum performance a re-ranker model
could achieve.

###  4.2 EditSQL

The EditSQL implementation by Zhang et al. (2019) uses greedy search strategy
for the decoder. We replace that with a beam search. Our implementation is
available at https://github.com/amolk/editsql/tree/beam_search. Using beam
search improved the accuracy on the Spider dev dataset by a small amount, from
57.6% to 57.7%. Beam hit rate for EditSQL was 78.7%.

Due to hardware constraints, instead of 40, we used beam size 10 on a trained
EditSQL model to generate a training and dev set for the re-ranker. This sub-
optimal dataset resulted in a re-ranker that did not make much progress
towards achieving accuracy close to the beam hit rate.

###  4.3 Re-ranker training

The re-ranker uses the BERT-base model Devlin et al. (2019) with a hidden size
of 768, 12 Transformer blocks Vaswani et al. (2017) and 12 self-attention
heads. We use a pre-trained BERT model from the PyTorch transformer repository
Wolf et al. (2019). We fine-tune the BERT model on a single V100 GPU with
dropout probability 0.1. We use Adam optimizer Kingma and Ba (2014) with a
learning rate 1e-3 for the linear classifier layer. A lower learning rate of
5e-6 is used to fine tune BERT layers. After training the re-ranker, we
empirically find a good threshold value to maximize the accuracy indicated by
the Spider Yu et al. (2018) evaluation scripts.

If the re-ranker is applied to other text-to-SQL models, such as those listed
on the Spider leaderboard, we expect that each model would have different
optimal threshold value.

###  4.4 Threshold tuning

$0$$0.2$$0.4$$0.6$$0.8$$1$$-2$$-1$$0$$1$$2$Threshold% improvement

Figure 2: Re-ranker performance improvement for EditSQL at various threshold
values. For EditSQL, threshold value 0.68 performs best.

Re-ranker’s performance depends on the threshold value. A threshold value of 0
allows re-ranker to promote a candidate if it scores more than the candidate
above it. A threshold of $\infty$ turns off the re-ranker by disallowing all
re-ranking decisions. A positive threshold value promotes a candidate only if
its score is at least threshold larger than the score of the candidate above,
thus allowing only high confidence re-ranking decisions to be executed. Figure
2 shows results of a threshold tuning experiment for EditSQL. Notice that the
re-ranker performs best at the empirically determined threshold value 0.68.

We realize that this tuning process uses validation data which makes it
vulnerable to overfitting, resulting in potentially poor performance on unseen
test data. We mitigate this risk by using half of the validation data for
tuning the threshold, while measuring performance on the other half.

###  4.5 Results

We compare results of BERT based text-to-SQL models with and without using the
re-ranker. Our implementation of BERT enabled GNN has 51.3 accuracy on Spider
dev set. Re-ranker improves that to 57.9, a 12.9% relative improvement. The
re-ranker, when trained using beam data from EditSQL, improves EditSQL’s Zhang
et al. (2019) 57.6 accuracy to 58.5, a 1.5% relative improvement.

The results of our model can be seen on the Spider leaderboard https://yale-
lily.github.io/spider. At the time of writing this artical, it was ranked 4th
on the leaderboard.

###  4.6 Analysis

As seen in table 2, the re-ranker improves GNN results at all hardness levels,
but for EditSQL, only the easy queries show improvement.

We trained the re-ranker using two different input configurations: 1) using
utterance, schema and query, and 2) using only utterance and query. We
observed that including schema information reduced performance at all hardness
levels. We hypothesize that the model is better able to use the syntactic
similarities between utterance and query and including schema information
breaks that mapping, leading to worse performance.

$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$$0.8$$0.9$$1$$-30$$-20$$-10$$0$$10$Threshold%
improvementEasyMediumHardExtra HardOverall Figure 3: Re-ranker performance
improvement for EditSQL at various threshold values for queries of various
hardness levels.

What threshold value works best depends on the relative strengths of the text-
to-SQL model and the corresponding re-ranker model. Spider evaluation scripts
Yu et al. (2018) categorize SQL queries into 4 hardness buckets - easy,
medium, hard and extra hard \- based on the set of SQL features that appear in
the queries. Figure 3 shows performance of the re-ranker for EditSQL for
queries of various hardness levels at various threshold values.

At low threshold values, performance of easy queries see most improvement,
while performance of extra hard queries suffers substantially. Looking at the
queries, we noticed that easy queries tend to look semantically similar to the
corresponding utterance. In contrast, hard and extra hard queries have
structures such as joins and sub-queries that depend on the database schema
and do not resemble the utterance semantically.

We hypothesize that the re-ranker does better with easy queries because it can
use the semantic similarities between such utterance and query pairs. On the
other hard, text-to-SQL models perform relatively better on harder queries
where schema based information is utilized to select appropriate columns,
joins and sub-queries. Also, text-to-SQL models trained on a complex dataset
such as Spider are probably biased towards more complex questions and queries
and thus perform worse on easy queries. The re-ranker may correct this bias.

The utterance words could be semantically related to the table and column
names used in the corresponding queries. BERT is pre-trained on various
language model tasks and is thus capable to mapping utterance words to table
and column names if they are semantically related. In easy queries, this
mapping is more apparent. In contrast, hard queries use constructs such as
table joins that may not have direct semantic relationship with utterance
words.

##  5 Conclusion and Future work

In this paper, we demonstrate a discriminative re-ranking based approach to
improving text-to-SQL performance. We make the code available as a library at
https://github.com/amolk/Bertrand-DR. We hope that the library would ease the
adoption of discriminative re-ranking technique for text-to-SQL models and
other encoder-decoder based generative models.

There are several potential avenues for further research. For instance, we
built a single re-ranker model for queries of all hardness levels. Given
various hardness level queries align differently in syntax and semantics with
corresponding utterances, building separate re-ranker models for different
hardness levels might allow the models to focus on relevant features. In such
an ensemble, appropriate re-ranker could be selected based on the hardness
level of the query at the top of the beam produced by the text-to-SQL model.
Alternately, a separate model can be trained to predict query hardness for a
given utterance and schema.

Another ensemble model approach would be to use other models that work well
with easy queries in conjunction with text-to-SQL models. WikiSQL dataset
contains queries without JOINs and sub-queries, and thus tend to be easy
queries. Models that perform well on WikiSQL dataset, such as X-SQL He et al.
(2019) and SQLova Hwang et al. (2019) would be good candidates as the easy
query model.

We plan to apply our re-ranker to other state-of-the-art text-to-SQL models at
the top of the Spider leaderboard. Current leading models such as IRNet Guo et
al. (2019) are well suited to be used with the re-ranker.

##  6 Acknowledgements

We thank Ben Bogin and Rui Zhang for helping us utilize GNN and EditSQL code
bases, respectively. We also appreciate the efforts by Tao Yu in managing the
Spider leaderboard and helping us with our leaderboard submission.

Bertrand AI is a suite of AI models developed by Got It Inc to tackle the
text-2-SQL problem. The model described in this paper, Bertrand-DR, is part of
that suite. We thank the Got It team for providing valuable feedback in the
process of productization of this work to build a conversational interface to
databases.

##  7 References

## References

  * Bogin et al. (2019a) Ben Bogin, Jonathan Berant, and Matt Gardner. 2019a.  Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4560–4565, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Bogin et al. (2019b) Ben Bogin, Matt Gardner, and Jonathan Berant. 2019b.  Global Reasoning over Database Structures for Text-to-SQL Parsing.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3650–3655, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Codd (1970) Edgar F Codd. 1970.  A relational model of data for large shared data banks.  _Communications of the ACM_ , 13(6):377–387.
  * Collins and Koo (2005) Michael Collins and Terry Koo. 2005.  Discriminative reranking for natural language parsing.  _Computational Linguistics_ , 31(1):25–70.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North_ , Mlm, pages 4171–4186, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Doran and Michie (1966) J. E. Doran and D. Michie. 1966.  Experiments with the Graph Traverser program.  _Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences_ , 294(1437):235–259.
  * Grosz et al. (1987) Barbara J Grosz, Douglas E Appelt, Paul A Martin, and Fernando CN Pereira. 1987\.  Team: an experiment in the design of transportable natural-language interfaces.  _Artificial Intelligence_ , 32(2):173–243.
  * Guo et al. (2019) Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019.  Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4524–4535, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * He et al. (2019) Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019.  X-SQL: reinforce schema representation with context.  _CoRR_ , abs/1908.08113.
  * Hwang et al. (2019) Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019.  A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization.  (2017).
  * Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * Lin et al. (2019) Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019.  Grammar-based neural text-to-sql generation.  _arXiv preprint arXiv:1905.13326_.
  * Ng and Jordan (2002) Andrew Y Ng and Michael I Jordan. 2002.  On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes.  In _Advances in neural information processing systems_ , pages 841–848.
  * Ramakrsihnan et al. (1998) R Ramakrsihnan, Donko Donjerkovic, Arvind Ranganathan, Kevin S Beyer, and Muralidhar Krishnaprasad. 1998.  Srql: Sorted relational query language.  In _Proceedings. Tenth International Conference on Scientific and Statistical Database Management (Cat. No. 98TB100243)_ , pages 84–95. IEEE.
  * Reddy et al. (1977) D Raj Reddy et al. 1977.  Speech understanding systems: A summary of results of the five-year research effort. department of computer science.
  * Scarselli et al. (2008) Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008.  The graph neural network model.  _IEEE Transactions on Neural Networks_ , 20(1):61–80.
  * Vaswani et al. (2017) Ashish Vaswani, Google Brain, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  _Advances in neural information processing systems_ , (Nips):5998–6008.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhieng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018\.  Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3911–3921, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Zhang et al. (2019) Rui Zhang, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019.  Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5341–5352, Stroudsburg, PA, USA. Association for Computational Linguistics.


# EQUATE : A Benchmark Evaluation Framework for Quantitative Reasoning in
Natural Language Inference

Abhilasha Ravichander , Aakanksha Naik11footnotemark: 1 ,
Carolyn Rose, Eduard Hovy
Language Technologies Institute, Carnegie Mellon University
{aravicha, anaik, cprose<EMAIL_ADDRESS>*The first two authors contributed
equally to this work.

###### Abstract

Quantitative reasoning is a higher-order reasoning skill that any intelligent
natural language understanding system can reasonably be expected to handle. We
present equate111Code and data available at
https://github.com/AbhilashaRavichander/EQUATE. (Evaluating Quantitative
Understanding Aptitude in Textual Entailment), a new framework for
quantitative reasoning in textual entailment. We benchmark the performance of
9 published NLI models on equate, and find that on average, state-of-the-art
methods do not achieve an absolute improvement over a majority-class baseline,
suggesting that they do not implicitly learn to reason with quantities. We
establish a new baseline Q-Reas that manipulates quantities symbolically. In
comparison to the best performing NLI model, it achieves success on numerical
reasoning tests (+24.2%), but has limited verbal reasoning capabilities
(-8.1%). We hope our evaluation framework will support the development of
models of quantitative reasoning in language understanding.

##  1 Introduction

Numbers play a vital role in our lives. We reason with numbers in day-to-day
tasks ranging from handling currency to reading news articles to understanding
sports results, elections and stock markets. As numbers are used to
communicate information accurately, reasoning with them is an essential core
competence in understanding natural language Levinson (2001); Frank et al.
(2008); Dehaene (2011). A benchmark task in natural language understanding is
natural language inference (NLI)(or recognizing textual entailment (RTE))
Cooper et al. (1996); Condoravdi et al. (2003); Bos and Markert (2005); Dagan
et al. (2006), wherein a model determines if a natural language hypothesis can
be justifiably inferred from a given premise222Often, this is posed as a
three-way decision where the hypothesis can be inferred to be true
(entailment), false (contradiction) or cannot be determined (neutral).. Making
such inferences often necessitates reasoning about quantities.

RTE-Quant
---
P: |  After the deal closes, Teva will generate sales of about $ 7 billion a year, the company said.
H: |  Teva earns $ 7 billion a year.
AWP-NLI
P: |  Each of farmer Cunningham’s 6048 lambs is either black or white and there are 193 white ones.
H: |  5855 of Farmer Cunningham’s lambs are black.
NewsNLI
P: |  Emmanuel Miller, 16, and Zachary Watson, 17, are charged as adults, police said.
H: |  Two teen suspects charged as adults.
RedditNLI
P: |  Oxfam says richest one percent to own more than rest by 2016.
H: |  Richest 1% To Own More Than Half Worlds Wealth By 2016 Oxfam.
Table 1: Examples from evaluation sets in EQUATE

Consider the following example from Table 1,

P: With 99.6% of precincts counted , Dewhurst held 48% of the vote to 30% for
Cruz .
H: Lt. Gov. David Dewhurst fails to get 50% of primary vote.

To conclude the hypothesis is inferable, a model must reason that since 99.6%
precincts are counted, even if all remaining precincts vote for Dewhurst, he
would fail to get 50% of the primary vote. Scant attention has been paid to
building datasets to evaluate this reasoning ability. To address this gap, we
present EQUATE (Evaluating Quantity Understanding Aptitude in Textual
Entailment) (§3), which consists of five evaluation sets, each featuring
different facets of quantitative reasoning in textual entailment (Table 1)
(including verbal reasoning with quantities, basic arithmetic computation,
dealing with approximations and range comparisons.).

We evaluate the ability of existing state-of-the-art NLI models to perform
quantitative reasoning (§4.1), by benchmarking 9 published models on EQUATE.
Our results show that most models are incapable of quantitative reasoning,
instead relying on lexical cues for prediction. Additionally, we build Q-REAS
, a shallow semantic reasoning baseline for quantitative reasoning in NLI
(§4.2). Q-REAS is effective on synthetic test sets which contain more
quantity-based inference, but shows limited success on natural test sets which
require deeper linguistic reasoning. However, the hardest cases require a
complex interplay between linguistic and numerical reasoning. The EQUATE
evaluation framework makes it clear where this new challenge area for textual
entailment stands.

##  2 Related Work

NLI has attracted community-wide interest as a stringent test for natural
language understanding Cooper et al. (1996); Fyodorov ; Glickman et al.
(2005); Haghighi et al. (2005); Harabagiu and Hickl (2006); Romano et al.
(2006); Dagan et al. (2006); Giampiccolo et al. (2007); Zanzotto et al.
(2006); Malakasiotis and Androutsopoulos (2007); MacCartney (2009); DeMarneffe
et al. (2009); Dagan et al. (2010); Angeli and Manning (2014); Marelli et al.
(2014). Recently, the creation of large-scale datasets Bowman et al. (2015);
Williams et al. (2017); Khot et al. (2018) spurred the development of many
neural models Parikh et al. (2016); Nie and Bansal (2017); Conneau et al.
(2017); Balazs et al. (2017); Chen et al. (2017a); Radford et al. (2018);
Devlin et al. (2018).

However, state-of-the-art models for NLI treat the task like a matching
problem, which appears to work in many cases, but breaks down in others. As
the field moves past current models of the matching variety to ones that
embody more of the reasoning we know is part of the task, we need benchmarks
that will enable us to mark progress in the field. Prior work on challenge
tasks has already made headway in defining tasks for subproblems such as
lexical inference with hypernymy, co-hyponymy, antonymy Glockner et al.
(2018); Naik et al. (2018). In this work, we specifically probe into
quantitative reasoning.

De Marneffe et al. (2008) find that in a corpus of real-life contradiction
pairs collected from Wikipedia and Google News, _29%_ contradictions arise
from numeric discrepancies, and in the RTE-3 (Recognizing Textual Entailment)
development set, numeric contradictions make up 8.8% of contradictory pairs.
Naik et al. (2018) find that model inability to do numerical reasoning causes
4% of errors made by state-of-the-art models. Sammons et al. (2010); Clark
(2018) argue for a systematic knowledge-oriented approach in NLI by evaluating
specific semantic analysis tasks, identifying quantitative reasoning in
particular as a focus area. Bentivogli et al. (2010) propose creating
specialized datasets, but feature only 6 examples with quantitative reasoning.
Our work bridges this gap by providing a more comprehensive examination of
quantitative reasoning in NLI.

While to the best of our knowledge, prior work has not studied quantitative
reasoning in NLI, Roy (2017) propose a model for a related subtask called
_quantity entailment_ , which aims to determine if a given quantity can be
inferred from a sentence. In contrast, our work is concerned with general-
purpose textual entailment which considers if a given _sentence_ can be
inferred from another. Our work also relates to solving arithmetic word
problems Hosseini et al. (2014); Mitra and Baral (2016); Zhou et al. (2015);
Upadhyay et al. (2016); Huang et al. (2017); Kushman et al. (2014a); Koncel-
Kedziorski et al. (2015); roy2016solving; Roy (2017); Ling et al. (2017a). A
key difference is that word problems focus on arithmetic reasoning, while the
requirement for linguistic reasoning and world knowledge is limited as the
text is concise, straightforward, and self-contained Hosseini et al. (2014);
Kushman et al. (2014b). Our work provides a testbed that evaluates basic
arithmetic reasoning while incorporating the complexity of natural language.

Recently, Dua2019DROPAR also recognize the importance of quantitative
reasoning for text understanding. They propose DROP, a reading comprehension
dataset focused on a limited set of discrete operations such as counting,
comparison, sorting and arithmetic. In contrast, EQUATE features diverse
phenomena that occur naturally in text, including reasoning with
approximation, ordinals, implicit quantities and quantifiers, requiring NLI
models to reason comprehensively about the interplay between quantities and
language. Additionally, through EQUATE we suggest the inclusion of controlled
synthetic tests in evaluation benchmarks. Controlled tests act as basic
validation of model behaviour, isolating model ability to reason about a
property of interest.

Source |  Test Set |  Size |  Classes |  Data Source |  Annotation Source |  Quantitative Phenomena
---|---|---|---|---|---|---
|  RTE-Quant |  166 |  2 |  RTE2-RTE4 |  Experts |  Arithmetic, Ranges, Quantifiers
Natural |  NewsNLI |  968 |  2 |  CNN |  Crowdworkers |  Ordinals, Quantifiers, Arithmetic, Approximation, Magnitude, Ratios
|  RedditNLI |  250 |  3 |  Reddit |  Experts |  Range, Arithmetic, Approximation, Verbal
|  Stress Test |  7500 |  3 |  AQuA-RAT |  Automatic |  Quantifiers
Synthetic |  AwpNLI |  722 |  2 |  Arithmetic Word Problems |  Automatic |  Arithmetic

Table 2: An overview of test sets included in EQUATE. RedditNLI and Stress
Test are framed as 3-class (entailment, neutral, contradiction) while RTE-
Quant, NewsNLI and AwpNLI are 2-class (entails=yes/no). RTE 2-4 formulate
entailment as a 2-way decision. We find that few news article headlines are
contradictory, thus NewsNLI is similarly framed as a 2-way decision. For
algebra word problems, substituting the wrong answer in the hypothesis
necessarily creates a contradiction under the event coreference assumption De
Marneffe et al. (2008), thus it is framed as a 2-way decision as well.

##  3 Quantitative Reasoning in NLI

Our interpretation of “quantitative reasoning” draws from cognitive testing
and education Stafford (1972); Ekstrom et al. (1976), which considers it
“verbal problem-solving ability”. While inextricably linked to mathematics, it
is an inclusive skill involving everyday language rather than a specialized
lexicon. To excel at quantitative reasoning, one must interpret quantities
expressed in language, perform basic calculations, judge their accuracy, and
justify quantitative claims using verbal and numeric reasoning. These
requirements show a reciprocity: NLI lends itself as a test bed for
quantitative reasoning, which conversely, is important for NLI Sammons et al.
(2010); Clark (2018). Motivated by this, we present the EQUATE (Evaluating
Quantity Understanding Aptitude in Textual Entailment) framework.

###  3.1 The EQUATE Dataset

EQUATE consists of five NLI test sets featuring quantities. Three of these
tests for quantitative reasoning feature language from real-world sources such
as news articles and social media (§3.2; §3.3; §3.4). We focus on sentences
containing quantities with numerical values, and consider an entailment pair
to feature quantitative reasoning if it is at least one component of the
reasoning required to determine the entailment label (but not necessarily the
only reasoning component). Quantitative reasoning features quantity matching,
quantity comparison, quantity conversion, arithmetic, qualitative processes,
ordinality and quantifiers, quantity noun and adverb resolution333Such as the
quantities represented in _dozen_ , _twice_ , _teenagers_. as well as verbal
reasoning with the quantity’s textual context444For example, $\langle$Obama
cuts tax rate to 28%, Obama wants to cut tax rate to 28% as part of
overhaul$\rangle$.. Appendix B gives some examples for these quantitative
phenomena. We further filter sentence pairs which require only temporal
reasoning, since specialized knowledge is needed to reason about time. These
three test sets contain pairs which conflate multiple lexical and quantitative
reasoning phenomena. In order to study aspects of quantitative reasoning in
isolation, EQUATE further features two controlled synthetic tests (§3.5;
§3.6), evaluating model ability to reason with quantifiers and perform simple
arithmetic. We intend that models reporting performance on any NLI dataset
additionally evaluate on the EQUATE benchmark, to demonstrate competence at
quantitative reasoning.

###  3.2 RTE-Quant

This test set is constructed from the RTE sub-corpus for quantity entailment
Roy (2017), originally drawn from the RTE2-RTE4 datasets Dagan et al. (2006).
The original sub-corpus conflates temporal and quantitative reasoning. We
discarded pairs requiring temporal reasoning, obtaining a set of 166
entailment pairs.

###  3.3 NewsNLI

This test set is created from the CNN corpus Hermann et al. (2015) of news
articles with abstractive summaries. We identify summary points with
quantities, filtering out temporal expressions. For a summary point, the two
most similar sentences555According to Jaccard similarity. from the article are
chosen, flipping pairs where the premise begins with a first-person pronoun
(eg:$\langle$“He had nine pears”, “Bob had nine pears”$\rangle$ becomes
$\langle$“Bob had nine pears”, “He had nine pears”$\rangle$). The top 50% of
similar pairs are retained to avoid lexical overlap bias. We crowdsource
annotations for a subset of this data from Amazon Mechanical Turk.
Crowdworkers666We require crowdworkers to have an approval rate of 95% on at
least 100 tasks and pass a qualification test. are shown two sentences and
asked to determine whether the second sentence is definitely true, definitely
false, or not inferable given the first. We collect 5 annotations per pair,
and consider pairs with lowest token overlap between premise and hypothesis
and least difference in premise-hypothesis lengths when stratified by
entailment label. Top 1000 samples meeting these criteria form our final set.
To validate crowdsourced labels, experts are asked to annotate 100 pairs.
Crowdsourced gold labels match expert gold labels in 85% cases, while
individual crowdworker labels match expert gold labels in 75.8%. Disagreements
are manually resolved by experts and examples not featuring quantitiative
reasoning are filtered, leaving a set of 968 samples.

###  3.4 RedditNLI

This test set is sourced from the popular social forum \reddit777According to
the Reddit User Agreement, users grant Reddit the right to make their content
available to other organizations or individuals.. Since reasoning about
quantities is important in domains like finance or economics, we scrape all
headlines from the posts on \r\economics, considering titles that contain
quantities and do not have meta-forum information. Titles appearing within
three days of each other are clustered by Jaccard similarity, and the top 300
pairs are extracted. After filtering out nonsensical titles, such as
concatenated stock prices, we are left with 250 sentence pairs. Similar to
RTE, two expert annotators label these pairs, achieving a Cohen’s kappa of
0.82. Disagreements are discussed to resolve final labels.

###  3.5 Stress Test

We include the numerical reasoning stress test from Naik et al. (2018) as a
synthetic sanity check. The stress test consists of 7500 entailment pairs
constructed from sentences in algebra word problems Ling et al. (2017b).
Focusing on quantifiers, it requires models to compare entities from
hypothesis to the premise while incorporating quantifiers, but does not
require them to perform the computation from the original algebra word problem
(eg: $\langle$“NHAI employs 100 men to build a highway of 2 km in 50 days
working 8 hours a day”,“NHAI employs less than 700 men to build a highway of 2
km in 50 days working 8 hours a day”$\rangle$).

###  3.6 AwpNLI

To evaluate arithmetic ability of NLI models, we repurpose data from
arithmetic word problems roy2016solving. They have the following
characteristic structure. First, they establish a world and optionally update
its state. Then, a question is posed about the world. This structure forms the
basis of our pair creation procedure. World building and update statements
form the premise. A hypothesis template is generated by identifying
modal/auxiliary verbs in the question, and subsequent verbs, which we call
secondary verbs. We identify the agent and conjugate the secondary verb in
present tense followed by the identified unit to form the final template (for
example, the algebra word problem ‘Gary had 73.0 dollars. He spent 55.0
dollars on a pet snake. How many dollars did Gary have left?’ would generate
the hypothesis template ‘Agent(Gary) Verb(Has) Answer(18.0) Unit(dollars)
left’). For every template, the correct guess is used to create an entailed
hypothesis. Contradictory hypotheses are created by randomly sampling a wrong
guess ($x\in\mathbb{Z}^{+}$ if correct guess is an integer, and
$x\in\mathbb{R}^{+}$ if it is a real number) 888From a uniform distribution
over an interval of 10 around the correct guess (or 5 for numbers less than
5), to identify plausible wrong guesses.. We check for grammaticality, finding
only 2% ungrammatical hypotheses, which are manually corrected leaving a set
of 722 pairs.

##  4 Models

We describe the 9 NLI models999Accuracy of all models on MultiNLI closely
matches original publications (numbers in appendix A). used in this study, as
well as our new baseline. The interested reader is invited to refer to the
corresponding publications for further details.

Figure 1: Overview of Q-REAS baseline.

###  4.1 NLI Models

1) Majority Class (MAJ):  Simple baseline that always predicts the majority
class in test set.
2) Hypothesis-Only (HYP):  FastText classifier Mikolov et al. (2018) trained
on only hypotheses Gururangan et al. (2018).
3) ALIGN:  A bag-of-words alignment model inspired by MacCartney
(2009).101010Model accuracy on RTE-3 test is 61.12%, comparable to the
reported average model performance in the RTE competition of 62.4% .
4) CBOW:  A simple bag-of-embeddings sentence representation model Williams et
al. (2017).
5) BiLSTM: The simple BiLSTM model described by Williams et al. (2017).
6) Chen (CH): Stacked BiLSTM-RNNs with shortcut connections and character word
embeddings Chen et al. (2017b).
7) InferSent:  A single-layer BiLSTM-RNN model with max-pooling Conneau et al.
(2017).
8) SSEN:  Stacked BiLSTM-RNNs with shortcut connections Nie and Bansal (2017).
9) ESIM:  Sequential inference model proposed by Chen et al. (2017a) which
uses BiLSTMs with an attention mechanism.
10) OpenAI GPT: Transformer-based language model Vaswani et al. (2017), with
finetuning on NLI Radford et al. (2018).
11) BERT: Transformer-based language model Vaswani et al. (2017), with a
cloze-style and next-sentence prediction objective, and finetuning on NLI
Devlin et al. (2018).

INPUT
---
$P_{c}$ |  Set of “compatible” single-valued premise quantities
$P_{r}$ |  Set of “compatible” range-valued premise quantities
$H$ |  Hypothesis quantity
$O$ |  Operator set $\\{+,-,*,/,=,\cap,\cup,\setminus,\subseteq\\}$
$L$ |  Length of equation to be generated
$SL$ |  Symbol list ($P_{c}\cup P_{r}\cup H\cup O$)
$TL$ |  Type list (set of types from $P_{c},P_{r},H$)
$N$ |  Length of symbol list
$K$ |  Index of first range quantity in symbol list
$M$ |  Index of first operator in symbol list
OUTPUT
$e_{i}$ |  Index of symbol assigned to $i^{th}$ position in postfix equation
VARIABLES
$x_{i}$ |  Main ILP variable for position $i$
$c_{i}$ |  Indicator variable: is $e_{i}$ a single value?
$r_{i}$ |  Indicator variable: is $e_{i}$ a range?
$o_{i}$ |  Indicator variable: is $e_{i}$ an operator?
$d_{i}$ |  Stack depth of $e_{i}$
$t_{i}$ |  Type index for $e_{i}$

Table 3: Input, output and variable definitions for the Integer Linear
Programming (ILP) framework used for quantity composition Definitional
Constraints
---
Range restriction |  $x_{i}<K$ or $x_{i}=M-1$ for $i\in[0,L-1]$ if $c_{i}=1$
|  $x_{i}\geq K$ and $x_{i}<M$ for $i\in[0,L-1]$ if $r_{i}=1$
|  $x_{i}\geq M$ for $i\in[0,L-1]$ if $o_{i}=1$
Uniqueness |  $c_{i}+r_{i}+o_{i}=1$ for $i\in[0,L-1]$
Stack definition |  $d_{0}=0$ (Stack depth initialization)
|  $d_{i}=d_{i-1}-2o_{i}+1$ for $i\in[0,L-1]$ (Stack depth update)
Syntactic Constraints
First two operands |  $c_{0}+r_{0}=1$ and $c_{1}+r_{1}=1$
Last operator |  $x_{L-1}\geq N-1$ (Last operator should be one of $\\{=,\subseteq\\}$)
Last operand |  $x_{L-2}=M-1$ (Last operand should be hypothesis quantity)
Other operators |  $x_{i}\leq N-2$ for $i\in[0,L-3]$ if $o_{i}=1$
Other operands |  $x_{i}<K$ for $i\in[0,L-3]$ if $c_{i}=1$
|  $x_{i}<M$ for $i\in[0,L-3]$ if $r_{i}=1$
Empty stack |  $d_{L-1}=0$ (Non-empty stack indicates invalid postfix expression)
Premise usage |  $x_{i}\neq x_{j}$ for $i,j\in[0,L-1]$ if $o_{i}\neq 1,o_{j}\neq 1$
Operand Access
Right operand |  $op2(x_{i})=x_{i-1}$ for $i\in[0,L-1]$ such that $o_{i}=1$
Left operand |  $op1(x_{i})=x_{l}$ for $i,l\in[0,L-1]$ where $o_{i}=1$ and $l$ is the largest index such that $l\leq(i-2)$ and $d_{l}=d_{i}$
Table 4: Mathematical validity constraint definitions for the ILP framework.
Functions $op1()$ and $op2()$ return the left and right operands for an
operator respectively. Variables defined in table 3. Type Consistency
Constraints
---
Type assignment |  $t_{i}=TL[k]$ for $i\in[0,L-1]$ if $c_{i}+r_{i}=1$ and $type(SL_{i})=k$
Two type match |  $t_{i}=t_{a}=t_{b}$ for $i\in[0,L-1]$ such that $o_{i}=1,x_{i}\in\\{+,-,*,/,=,\cap,\cup,\setminus,\subseteq\\},a=op1(x_{i}),b=op2(x_{i})$
One type match |  $t_{i}\in\\{t_{a},t_{b}\\},t_{a}\neq t_{b}$ for $i\in[0,L-1]$ such that $o_{i}=1,x_{i}=*,a=op1(x_{i}),b=op2(x_{i})$
|  $t_{i}=t_{a}\neq t_{b}$ for $i\in[0,L-1]$ such that
$o_{i}=1,x_{i}=/,a=op1(x_{i}),b=op2(x_{i})$
Operator Consistency Constraints
Arithmetic operators |  $c_{a}=c_{b}=1$ for $i\in[0,L-1]$ such that $o_{i}=1,x_{i}\in\\{+,-,*,/,=\\},a=op1(x_{i}),b=op2(x_{i})$
Range operators |  $r_{a}=r_{b}=1$ for $i\in[0,L-1]$ such that $o_{i}=1,x_{i}\in\\{\cap,\cup,\setminus\\},a=op1(x_{i}),b=op2(x_{i})$
|  $r_{b}=1$ for $i\in[0,L-1]$ such that
$o_{i}=1,x_{i}=\subseteq,b=op2(x_{i})$
Table 5: Linguistic consistency constraint definitions for the ILP framework.
Functions $op1()$ and $op2()$ return the left and right operands for an
operator respectively. Variables defined in table 3.

###  4.2 Q-REAS Baseline System

M D | RTE-Q $\Delta$ | NewsNLI $\Delta$ | RedditNLI $\Delta$ | NR ST $\Delta$ | AWPNLI $\Delta$ |  Nat. Avg. $\Delta$ |  Synth. Avg. $\Delta$ |  All Avg. $\Delta$
---|---|---|---|---|---|---|---|---
MAJ | 57.8 | 0.0 | 50.7 | 0.0 | 58.4 | 0.0 | 33.3 | 0.0 | 50.0 | 0.0 | +0.0 | +0.0 | +0.0
HYP | 49.4 | -8.4 | 52.5 | +1.8 | 40.8 | -17.6 | 31.2 | -2.1 | 50.1 | +0.1 | -8.1 | -1.0 | -5.2
ALIGN | 62.1 | +4.3 | 56.0 | +5.3 | 34.8 | -23.6 | 22.6 | -10.7 | 47.2 | -2.8 | -4.7 | -6.8 | -5.5
CBOW | 47.0 | -10.8 | 61.8 | +11.1 | 42.4 | -16.0 | 30.2 | -3.1 | 50.7 | +0.7 | -5.2 | -1.2 | -3.6
BiLSTM | 51.2 | -6.6 | 63.3 | +12.6 | 50.8 | -7.6 | 31.2 | -2.1 | 50.7 | +0.7 | -0.5 | -0.7 | -0.6
CH | 54.2 | -3.6 | 64.0 | +13.3 | 55.2 | -3.2 | 30.3 | -3.0 | 50.7 | +0.7 | +2.2 | -1.2 | +0.9
InferSent | 66.3 | +8.5 | 65.3 | +14.6 | 29.6 | -28.8 | 28.8 | -4.5 | 50.7 | +0.7 | -1.9 | -1.9 | -1.9
SSEN | 58.4 | +0.6 | 65.1 | +14.4 | 49.2 | -9.2 | 28.4 | -4.9 | 50.7 | +0.7 | +1.9 | -2.1 | +0.3
ESIM | 54.8 | -3.0 | 62.0 | +11.3 | 45.6 | -12.8 | 21.8 | -11.5 | 50.1 | +0.1 | -1.5 | -5.7 | -3.2
GPT | 68.1 | +10.3 | 72.2 | +21.5 | 52.4 | -6.0 | 36.4 | +3.1 | 50.0 | +0.0 | +8.6 | +1.6 | +5.8
BERT | 57.2 | -0.6 | 72.8 | +22.1 | 49.6 | -8.8 | 36.9 | +3.6 | 42.2 | -7.8 | +4.2 | -2.1 | +1.7
Q-REAS | 56.6 | -1.2 | 61.1 | +10.4 | 50.8 | -7.6 | 63.3 | +30 | 71.5 | +21.5 | +0.5 | +25.8 | +10.6

Table 6: Accuracies(%) of 9 NLI Models on five tests for quantitiative
reasoning in entailment. M and D represent _models_ and _datasets_
respectively. $\Delta$ captures improvement over majority-class baseline for a
dataset. Column Nat.Avg. reports the average accuracy(%) of each model across
3 evaluation sets constructed from natural sources (RTE-Quant, NewsNLI,
RedditNLI), whereas Synth.Avg. reports the average accuracy(%) on 2 synthetic
evaluation sets (Stress Test, AwpNLI). Column Avg. represents the average
accuracy(%) of each model across all 5 evaluation sets in equate.

Figure 1 describes the Q-REAS baseline for quantitative reasoning in NLI. The
model manipulates quantity representations symbolically to make entailment
decisions, and is intended to serve as a strong heuristic baseline for
numerical reasoning on the equate benchmark. This model has four stages:
Quantity mentions are extracted and parsed into semantic representations
called Numsets (§4.2.1, §4.2.2); compatible Numsets are extracted (§4.2.3) and
composed (§4.2.4) to form _justifications_ ; Justifications are analyzed to
determine entailment labels (§4.2.5).

####  4.2.1 Quantity Segmenter

We follow Barwise and Cooper (1981) in defining quantities as having a number,
unit, and an optional approximator. Quantity mentions are identified as least
ancestor noun phrases from the constituency parse of the sentence containing
cardinal numbers.

####  4.2.2 Quantity Parser

The quantity parser constructs a grounded representation for each quantity
mention in the premise or hypothesis, henceforth known as a Numset 111111A
Numset may be a composition of other Numsets .. A Numset is a tuple (val,
unit, ent, adj, loc, verb, freq, flux)121212As in Koncel-Kedziorski et al.
(2015) S denotes all possible spans in the sentence, ${\phi}$ represents the
empty span, and $S^{\phi}$=$S\cup\phi$ with:

1\. val $\in$ [$\mathbb{R},\mathbb{R}$]: quantity value represented as a range
2\. unit $\in S$: unit noun associated with quantity
3\. ent $\in S^{\phi}$: entity noun associated with unit (e.g., ‘ _donations_
worth 100$’)
4\. adj $\in S^{\phi}$: adjective associated with unit if any131313Extracted
as governing verb linked to entity by an _amod_ relation.,
5\. loc $\subseteq S^{\phi}$: location of unit (e.g.,’in the
bag’)141414Extracted as prepositional phrase attached to the quantity and
containing noun phrase.
6\. verb $\in S^{\phi}$: action verb associated with quantity151515Extracted
as governing verb linked to entity by _dobj_ or _nsubj_ relation..
7\. freq $\subseteq S^{\phi}$: if quantity recurs161616extracted using
keywords _per_ and _every_ (e.g, ’per hour’),
8\. flux $\in$ {increase to, increase from, decrease to, decrease
from$\\}^{\phi}$: if quantity is in a state of flux171717using gazetteer:
_increasing_ , _rising_ , _rose_ , _decreasing_ , _falling_ , _fell_ , _drop_.

To extract values for a quantity, we extract cardinal numbers, recording
contiguity. We normalize the number181818(remove “,”s, convert written numbers
to float, decide the numerical value, for example hundred fifty eight thousand
is 158000, two fifty eight is 258, 374m is 3740000 etc.). If cardinal numbers
are non-adjacent, we look for an explicitly mentioned range such as ‘to’ and
‘between’.. We also handle simple ratios such as quarter, half etc, and
extract bounds (eg: _fewer than 10 apples_ is parsed to $[-\infty,10]$
apples.)

To extract units, we examine tokens adjacent to cardinal numbers in the
quantity mention and identify known units. If no known units are found, we
assign the token in a _numerical modifier_ relationship with the cardinal
number, else we assign the nearest noun to the cardinal number as the unit. A
quantity is determined to be approximate if the word in an _adverbial
modifier_ relation with the cardinal number appears in a
gazetteer191919roughly, approximately, about, nearly, roundabout, around,
circa, almost, approaching, pushing, more or less, in the neighborhood of, in
the region of, on the order of,something like, give or take (a few), near to,
close to, in the ballpark of. If approximate, range is extended to (+/-)2% of
the current value.

####  4.2.3 Quantity Pruner

The pruner constructs “compatible” premise-hypothesis Numset pairs. Consider
the pair “Insurgents killed _7 U.S. soldiers_ , set off a car bomb that killed
_four Iraqi policemen_ ” and “ _7 US soldiers_ were killed, and _at least 10
Iraqis_ died”. Our parser extracts Numsets corresponding to _“four Iraqi
policemen”_ and _“7 US soldiers”_ from premise and hypothesis respectively.
But these Numsets should not be compared as they involve different units. The
pruner discards such incompatible pairs. Heuristics to identify unit-
compatible Numset pairs include three cases- 1) direct string match, 2)
synonymy/hypernymy relations from WordNet, 3) one unit is a nationality/
job202020Lists of jobs, nationalities scraped from Wikipedia. and the other
unit is synonymous with person Roy (2017).

####  4.2.4 Quantity Composition

The composition module detects whether a hypothesis Numset is justified by
composing “compatible” premise Numsets . For example, consider the pair “I had
3 apples but gave one to my brother” and “I have two apples”. Here, the
premise Numsets $P_{1}$ (“3 apples”) and $P_{2}$ (“one apple”) must be
composed to deduce that the hypothesis Numset $H_{1}$ (“2 apples”) is
justified. Our framework accomplishes this by generating postfix arithmetic
equations212121Note that arithmetic equations differ from algebraic equations
in that they do not contain unknown variables from premise Numsets , that
justify the hypothesis Numset 222222Direct comparisons are incorporated by
adding “=” as an operator.. In this example, the expression
$<P_{1},P_{2},-,H_{1},=>$ will be generated.

The set of possible equations is exponential in number of Numsets , making
exhaustive generation intractable. But a large number of equations are invalid
as they violate constraints such as unit consistency. Thus, our framework uses
integer linear programming (ILP) to constrain the equation space. It is
inspired by prior work on algebra word problems Koncel-Kedziorski et al.
(2015), with some key differences:

1\.  Arithmetic equations: We focus on arithmetic equations instead of
algebraic ones.
2\. Range arithmetic: Quantitative reasoning involves ranges, which are
handled by representing then as endpoint-inclusive intervals and adding the
four operators ($\cup,\cap,\setminus,\subseteq$)
3\. Hypothesis quantity-driven: We optimize an ILP model for each hypothesis
Numset because a sentence pair is marked “entailment” iff every hypothesis
quantity is justified.

Table 3 describes ILP variables. We impose the following types of constraints:
1\. Definitional Constraints: Ensure that ILP variables take on valid values
by constraining initialization, range, and update.
2\. Syntactic Constraints: Assure syntactic validity of generated postfix
expressions by limiting operator-operand ordering.
3\. Operand Access: Simulate stack-based evaluation correctly by choosing
correct operator-operand assignments.
4\. Type Consistency: Ensure that all operations are type-compatible.
5\. Operator Consistency: Force range operators to have range operands and
mathematical operators to have single-valued operands.

Definitional, syntactic, and operand access constraints ensure mathematical
validity while type and operator consistency constraints add linguistic
consistency. Constraint formulations are provided in Tables 4 and 5. We limit
tree depth to 3 and retrieve a maximum of 50 solutions per hypothesis Numset ,
then solve to determine whether the equation is mathematically correct. We
discard equations that use invalid operations (division by 0) or add
unnecessary complexity (multiplication/ division by 1). The remaining
equations are considered plausible justifications .

####  4.2.5 Global Reasoner

The global reasoner predicts the final entailment label as shown in Algorithm
1232323MaxSimilarityClass() takes two quantities and returns a probability
distribution over entailment labels based on unit match. Similarly,
ValueMatch() detects whether two quantities match in value (this function can
also handle ranges)., on the assumption that every Numset in the hypothesis
_has_ to be justified 242424This is a necessary but not sufficient condition
for entailment. Consider the example, $\langle$‘Sam believed Joan had 5
apples’, ‘Joan had 5 apples’$\rangle$. The hypothesis quantities of 5 apples
is justified but is not a sufficient condition for entailment. for entailment.

Algorithm 1 PredictEntailmentLabel($P,H,C,E$)

1:Premise quantities $P$, Hypothesis quantities $H$, Compatible pairs $C$,
Equations $E$

2:Entailment label $l\in$ { e, c, n }

3:if $C$ = $\emptyset$ then return $n$

4:$J\leftarrow\emptyset$

5:$L\leftarrow[]$

6:for $q_{h}\in H$ do

7: $J_{h}\leftarrow\\{q_{p}\mid q_{p}\in P,(q_{p},q_{h})\in C\\}$

8: $J\leftarrow J\cup\\{(q_{h},J_{h})\\}$

9: $L\leftarrow L+[false]$

10:for $(q_{h},J_{h})\in J$ do

11: if $J_{h}=\emptyset$ then return $n$

12: for $q_{p}\in J_{h}$ do

13: $s\leftarrow$ MaxSimilarityClass($q_{p},q_{h}$)

14: if $s=e$ then

15: if ValueMatch($q_{p},q_{h}$) then

16: $L$[$q_{h}$] = $true$

17: if !ValueMatch($q_{p},q_{h}$) then

18: $L$[$q_{h}$] = $false$

19: if $s=c$ then

20: if ValueMatch($q_{p},q_{h}$) then

21: $L$[$q_{h}$] = $c$

22:for $q_{h}\in H$ do

23: $E_{q}\leftarrow\\{e_{i}\in E\mid hyp(e_{i})=q_{h}\\}$

24: if $E_{q}\neq\emptyset$ then

25: $L$[$q_{h}$] = $true$

26:if $c\in L$ then return $c$

27:if count($L,true$) = len($L$) then return $e$

28:return $n$

##  5 Results and Discussion

Table 6 presents results on EQUATE. All models, except Q-REAS are trained on
MultiNLI. Q-REAS utilizes WordNet and lists from Wikipedia. We observe that
neural models, particularly OpenAI GPT excel at verbal aspects of quantitative
reasoning (RTE-Quant, NewsNLI), whereas Q-REAS excels at numerical aspects
(Stress Test, AwpNLI).

###  5.1 Neural Models on NewsNLI:

To tease apart contributory effects of numerical and verbal reasoning in
natural data, we experiment with NewsNLI. We extract all entailed pairs where
a quantity appears in both premise and hypothesis, and perturb the quantity in
the hypothesis generating contradictory pairs. For example, the pair
$\langle$‘In addition to 79 fatalities , some 170 passengers were
injured.’$\rangle$ ‘The crash took the lives of 79 people and injured some
170’, ‘entailment’ is changed to $\langle$‘In addition to 79 fatalities , some
170 passengers were injured.’, ‘The crash took the lives of 80 people and
injured some 170’, ‘contradiction’$\rangle$, assuming scalar implicature and
event coreference. Our perturbed test set contains 218 pairs. On this set,
GPT252525the best-performing neural model on EQUATE. achieves an accuracy of
51.18%, as compared to 72.04% on the unperturbed set, suggesting the model
relies on verbal cues rather than numerical reasoning. In comparison, Q-REAS
achieves an accuracy of 98.1% on the perturbed set, compared to 75.36% on the
unperturbed set, highlighting reliance on quantities rather than verbal
information. Closer examination reveals that OpenAI switches to predicting the
‘neutral’ category for perturbed samples instead of entailment, accounting for
42.7% of its errors, possibly symptomatic of lexical bias issues Naik et al.
(2018).

###  5.2 What Quantitative Phenomena Are Hard?

We sample 100 errors made by Q-REAS on each test in EQUATE, to identify
phenomena not addressed by simple quantity comparison. Our analysis of causes
for error suggest avenues for future research:

1\. Multi-step numerical-verbal reasoning: Models do not perform well on
examples requiring interleaved verbal and quantitative reasoning, especially
multi-step deduction. Consider the pair $\langle$“Two people were injured in
the attack”, “Two people perpetrated the attack”$\rangle$. Quantities “two
people” and “two people” are unit-compatible, but must not be compared.
Another example is the NewsNLI entailment pair in Table 1. This pair requires
us to identify that 16 and 17 refer to Emmanuel and Zachary’s ages
(quantitative), deduce that this implies they are teenagers (verbal) and
finally count them (quantitative) to get the hypothesis quantity “two teens”.
Numbers and language are intricately interleaved and developing a reasoner
capable of handling such complex interplay is challenging.

2\. Lexical inference: Lack of real world knowledge causes errors in
identifying quantities and valid comparisons. Errors include mapping
abbreviations to correct units (“m” to “meters”), detecting part-whole
coreference (“seats” can be used to refer to “buses”), and resolving
hypernymy/hyponymy (“young men” to “boys”).

3\. Inferring underspecified quantities: Quantity attributes can be implicitly
specified, requiring inference to generate a complete representation. Consider
“A mortar attack killed four people and injured 80”. A system must infer that
the quantity “80” refers to people. On RTE-Quant, 20% of such cases stem from
zero anaphora, a hard problem in coreference resolution.

4\. Arithmetic comparison limitations: These examples require composition
between incompatible quantities. For example, consider $\langle$“There were 3
birds and 6 nests”, “There were 3 more nests than birds”$\rangle$. To
correctly label this pair “3 birds” and “6 nests” must be composed.

##  6 Conclusion

In this work, we present EQUATE, an evaluation framework to estimate the
ability of models to reason quantitatively in textual entailment. We observe
that existing neural approaches rely heavily on the lexical matching aspect of
the task to succeed rather than reasoning about quantities. We implement a
strong symbolic baseline Q-REAS that achieves success at numerical reasoning,
but lacks sophisticated verbal reasoning capabilities. The EQUATE resource
presents an opportunity for the community to develop powerful hybrid neuro-
symbolic architectures, combining the strengths of neural models with
specialized reasoners such as Q-REAS . We hope our insights lead to the
development of models that can more precisely reason about the important,
frequent, but understudied, phenomena of quantities in natural language.

## Acknowledgments

This research was supported in part by grants from the National Science
Foundation Secure and Trustworthy Computing program (CNS-1330596,
CNS-15-13957, CNS-1801316, CNS-1914486) and a DARPA Brandeis grant
(FA8750-15-2-0277). The views and conclusions contained herein are those of
the authors and should not be interpreted as necessarily representing the
official policies or endorsements, either expressed or implied, of the NSF,
DARPA, or the US Government. The author Naik was supported by a fellowship
from the Center of Machine Learning and Health at Carnegie Mellon University.
The authors would like to thank Graham Neubig, Mohit Bansal and Dongyeop Kang
for helpful discussion regarding this work, and Shruti Rijhwani and Siddharth
Dalmia for reviews while drafting this paper. The authors are also grateful to
Lisa Carey Lohmueller and Xinru Yan for volunteering their time for pilot
studies.

## References

  * Angeli and Manning (2014) Gabor Angeli and Christopher D Manning. 2014.  Naturalli: Natural logic inference for common sense reasoning.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 534–545.
  * Balazs et al. (2017) Jorge Balazs, Edison Marrese-Taylor, Pablo Loyola, and Yutaka Matsuo. 2017.  Refining raw sentence representations for textual entailment recognition via attention.  In _Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP_ , pages 51–55, Copenhagen, Denmark. Association for Computational Linguistics.
  * Barwise and Cooper (1981) Jon Barwise and Robin Cooper. 1981.  Generalized quantifiers and natural language.  In _Philosophy, Language, and Artificial Intelligence_ , pages 241–301. Springer.
  * Bentivogli et al. (2010) Luisa Bentivogli, Elena Cabrio, Ido Dagan, Danilo Giampiccolo, Medea Lo Leggio, and Bernardo Magnini. 2010.  Building textual entailment specialized data sets: a methodology for isolating linguistic phenomena relevant to inference.  In _Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10)_ , Valletta, Malta. European Languages Resources Association (ELRA).
  * Bos and Markert (2005) Johan Bos and Katja Markert. 2005.  Recognising textual entailment with logical inference.  In _Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing_ , pages 628–635. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Chen et al. (2017a) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017a.  Enhanced lstm for natural language inference.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2017b) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b.  Recurrent neural network-based sentence encoder with gated attention for natural language inference.  In _Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP_ , pages 36–40, Copenhagen, Denmark. Association for Computational Linguistics.
  * Clark (2018) Peter Clark. 2018.  What knowledge is needed to solve the rte5 textual entailment challenge?  _arXiv preprint arXiv:1806.03561_.
  * Condoravdi et al. (2003) Cleo Condoravdi, Dick Crouch, Valeria De Paiva, Reinhard Stolle, and Daniel G Bobrow. 2003.  Entailment, intensionality and text understanding.  In _Proceedings of the HLT-NAACL 2003 workshop on Text meaning-Volume 9_ , pages 38–45. Association for Computational Linguistics.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Cooper et al. (1996) Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996\.  Using the framework.  Technical report.
  * Dagan et al. (2010) Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010.  The fourth pascal recognizing textual entailment challenge.  _Journal of Natural Language Engineering_.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The pascal recognising textual entailment challenge.  In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment_ , pages 177–190. Springer.
  * De Marneffe et al. (2008) Marie-Catherine De Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008\.  Finding contradictions in text.  In _Proceedings of ACL-08: HLT_ , pages 1039–1047, Columbus, Ohio. Association for Computational Linguistics.
  * Dehaene (2011) Stanislas Dehaene. 2011.  _The number sense: How the mind creates mathematics_.  OUP USA.
  * DeMarneffe et al. (2009) Marie-Catherine DeMarneffe, Sebastian Padó, and Christopher D Manning. 2009\.  Multi-word expressions in textual inference: Much ado about nothing?  In _Proceedings of the 2009 Workshop on Applied Textual Inference_ , pages 1–9. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Ekstrom et al. (1976) Ruth B Ekstrom, Diran Dermen, and Harry Horace Harman. 1976.  _Manual for kit of factor-referenced cognitive tests_ , volume 102\.  Educational Testing Service Princeton, NJ.
  * Frank et al. (2008) Michael C Frank, Daniel L Everett, Evelina Fedorenko, and Edward Gibson. 2008.  Number as a cognitive technology: Evidence from pirahã language and cognition.  _Cognition_ , 108(3):819–824.
  * (21) Yaroslav Fyodorov.  A natural logic inference system.  Citeseer.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third pascal recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9. Association for Computational Linguistics.
  * Glickman et al. (2005) Oren Glickman, Ido Dagan, and Moshe Koppel. 2005.  Web based probabilistic textual entailment.
  * Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018.  Breaking nli systems with sentences that require simple lexical inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 650–655, Melbourne, Australia. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018.  Annotation artifacts in natural language inference data.  _arXiv preprint arXiv:1803.02324_.
  * Haghighi et al. (2005) Aria Haghighi, Andrew Ng, and Christopher Manning. 2005.  Robust textual inference via graph matching.  In _Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing_.
  * Harabagiu and Hickl (2006) Sanda Harabagiu and Andrew Hickl. 2006.  Methods for using textual entailment in open-domain question answering.  In _Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics_ , pages 905–912. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems_ , pages 1693–1701.
  * Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014\.  Learning to solve arithmetic word problems with verb categorization.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 523–533.
  * Huang et al. (2017) Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017.  Learning fine-grained expressions to solve math word problems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 805–814.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  Scitail: A textual entailment dataset from science question answering.
  * Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015.  Parsing algebraic word problems into equations.  _Transactions of the Association for Computational Linguistics_ , 3:585–597.
  * Kushman et al. (2014a) Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014a.  Learning to automatically solve algebra word problems.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 271–281.
  * Kushman et al. (2014b) Nate Kushman, Luke S. Zettlemoyer, Regina Barzilay, and Yoav Artzi. 2014b.  Learning to automatically solve algebra word problems.  In _ACL_.
  * Levinson (2001) Stephen C Levinson. 2001.  Pragmatics.  In _International Encyclopedia of Social and Behavioral Sciences: Vol. 17_ , pages 11948–11954. Pergamon.
  * Ling et al. (2017a) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017a.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  _arXiv preprint arXiv:1705.04146_.
  * Ling et al. (2017b) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017b.  Program induction by rationale generation: Learning to solve and explain algebraic word problems.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 158–167, Vancouver, Canada. Association for Computational Linguistics.
  * MacCartney (2009) Bill MacCartney. 2009.  _Natural language inference_.  Stanford University.
  * Malakasiotis and Androutsopoulos (2007) Prodromos Malakasiotis and Ion Androutsopoulos. 2007.  Learning textual entailment using svms and string similarity measures.  In _Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing_ , pages 42–47. Association for Computational Linguistics.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014.  A sick cure for the evaluation of compositional distributional semantic models.
  * Mikolov et al. (2018) Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018.  Advances in pre-training distributed word representations.  In _Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Mitra and Baral (2016) Arindam Mitra and Chitta Baral. 2016.  Learning to use formulas to solve simple arithmetic problems.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , volume 1, pages 2144–2153.
  * Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018.  Stress test evaluation for natural language inference.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Nie and Bansal (2017) Yixin Nie and Mohit Bansal. 2017.  Shortcut-stacked sentence encoders for multi-domain inference.  In _Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP_ , pages 41–45, Copenhagen, Denmark. Association for Computational Linguistics.
  * Parikh et al. (2016) Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016\.  A decomposable attention model for natural language inference.  _arXiv preprint arXiv:1606.01933_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Romano et al. (2006) Lorenza Romano, Milen Kouylekov, Idan Szpektor, Ido Dagan, and Alberto Lavelli. 2006\.  Investigating a generic paraphrase-based approach for relation extraction.
  * Roy (2017) Subhro Roy. 2017.  _Reasoning about quantities in natural language_.  Ph.D. thesis, University of Illinois at Urbana-Champaign.
  * Sammons et al. (2010) Mark Sammons, VG Vydiswaran, and Dan Roth. 2010.  Ask not what textual entailment can do for you…  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 1199–1208. Association for Computational Linguistics.
  * Stafford (1972) Richard E Stafford. 1972.  Hereditary and environmental components of quantitative reasoning.  _Review of Educational Research_ , 42(2):183–201.
  * Upadhyay et al. (2016) Shyam Upadhyay, Ming-Wei Chang, Kai-Wei Chang, and Wen-tau Yih. 2016.  Learning from explicit and implicit supervision jointly for algebra word problems.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 297–306.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Zanzotto et al. (2006) F Zanzotto, Alessandro Moschitti, Marco Pennacchiotti, and M Pazienza. 2006.  Learning textual entailment from examples.  In _Second PASCAL recognizing textual entailment challenge_ , page 50. PASCAL.
  * Zhou et al. (2015) Lipu Zhou, Shuaixiang Dai, and Liwei Chen. 2015.  Learn to solve algebra word problems using quadratic programming.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 817–822.

## Appendix

##  Appendix A Baseline performance on MultiNLI-Dev Matched

Model | MultiNLI Dev
---|---
Hyp Only | 53.18%
ALIGN | 45.0%
CBOW | 63.5%
BiLSTM | 70.2%
Chen | 73.7%
NB | 74.2%
InferSent | 70.3%
ESIM | 76.2%
OpenAI Transformer | 81.35%
BERT | 83.8%
Table 7: Performance of all baseline models used in the paper on the matched
subset of MultiNLI-Dev

Table 7 presents classification accuracies of all baseline models used in this
work on the matched subset of MultiNLI-Dev. These scores are very close to the
numbers reported by the original publications, affirming the correctness of
our baseline setup.

##  Appendix B Examples of quantitative phenomena present in EQUATE

Table 8 presents some examples from EQUATE which demonstrate interesting
quantitative phenomena that must be understood to label the pair correctly.

Phenomenon |  Example
---|---
Arithmetic |  P: Sharper faces charges in Arizona and California H: Sharper has been charged in two states
Ranges |  P: Between 20 and 30 people were trapped in the casino H: Upto 30 people thought trapped in casino
Quantifiers |  P: Poll: Obama over 50% in Florida H: New poll shows Obama ahead in Florida
Ordinals |  P: Second-placed Nancy celebrated their 40th anniversary with a win H: Nancy stay second with a win
Approximation |  P: Rwanda has dispatched 1917 soldiers H: Rwanda has dispatched some 1900 soldiers
Ratios |  P: Londoners had the highest incidence of E. Coli bacteria (25%) H: 1 in 4 Londoners have E. Coli bacteria
Comparison |  P: Treacherous currents took four lives on the Alabama Gulf coast H: Rip currents kill four in Alabama
Conversion |  P: If the abuser has access to a gun, it increases chances of death by 500% H: Victim five times more likely to die if abuser is armed
Numeration |  P: Eight suspects were arrested H: 8 suspects have been arrested
Implicit Quantities |  P: The boat capsized two more times H: His sailboat capsized three times
Table 8: Examples of quantitative phenomena present in EQUATE