The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains unexplored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into LISP-like programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. Verifying whether a textual hypothesis is entailed or refuted by the given evidence is a fundamental problem in natural language understanding (Katz & Fodor, 1963; Van Benthem et al., 2008) . It can benefit many downstream applications like misinformation detection, fake news detection, etc. Recently, the first-ever end-to-end fact-checking system has been designed and proposed in Hassan et al. (2017) . The verification problem has been extensively studied under different natural language tasks such as recognizing textual entailment (RTE) (Dagan et al., 2005) , natural language inference (NLI) (Bowman et al., 2015) , claim verification (Popat et al., 2017; Hanselowski et al., 2018; Thorne et al., 2018) and multimodal language reasoning (NLVR/NLVR2) (Suhr et al., 2017; . RTE and NLI view a premise sentence as the evidence, claim verification views passage collection like Wikipedia 1 as the evidence, NLVR/NLVR2 views images as the evidence. These problems have been previously addressed using a variety of techniques including logic rules, knowledge bases, and neural networks. Recently large-scale pre-trained language models (Devlin et al., 2019; Peters et al., 2018; Yang et al., 2019; Liu et al., 2019) have surged to dominate the other algorithms to approach human performance on several textual entailment tasks (Wang et al., 2018; . However, existing studies are restricted to dealing with unstructured text as the evidence, which would not generalize to the cases where the evidence has a highly structured format. Since such structured evidence (graphs, tables, or databases) are also ubiquitous in real-world applications like database systems, dialog systems, commercial management systems, social networks, etc, we argue that the fact verification under structured evidence forms is an equivalently important yet underexplored problem. Therefore, in this paper, we are specifically interested in studying fact verification with semi-structured Wikipedia tables (Bhagavatula et al., 2013) 2 as evidences owing to its structured and ubiquitous nature (Jauhar et al., 2016; Zhong et al., 2017; Pasupat & Liang, 2015) . To this end, we introduce a large-scale dataset called TABFACT, which consists of 118K manually annotated statements with regard to 16K Wikipedia tables, their relations are classified as ENTAILED and REFUTED 3 . The entailed and refuted statements are both annotated by human workers. With some examples in Figure 1 , we can clearly observe that unlike the previous verification related problems, TABFACT combines two different forms of reasoning in the statements, (i) Linguistic Reasoning: the verification requires semantic-level understanding. For example, "John J. Mcfall failed to be re-elected though being unopposed." requires understanding over the phrase "lost renomination ..." in the table to correctly classify the entailment relation. Unlike the existing QA datasets (Zhong et al., 2017; Pasupat & Liang, 2015) , where the linguistic reasoning is dominated by paraphrasing, TABFACT requires more linguistic inference or common sense. (ii) Symbolic Reasoning: the verification requires symbolic execution on the table structure. For example, the phrase "There are three Democrats incumbents" requires both condition operation (where condition) and arithmetic operation (count). Unlike question answering, a statement could contain compound facts, all of these facts need to be verified to predict the verdict. For example, the "There are ..." in Figure 1 requires verifying three QA pairs (total count=5, democratic count=2, republic count=3). The two forms of reasoning are interleaved across the statements making it challenging for existing models. In this paper, we particularly propose two approaches to deal with such mixed-reasoning challenge: (i) Table-BERT, this model views the verification task completely as an NLI problem by linearizing a table as a premise sentence p, and applies state-of-the-art language understanding pre-trained model to encode both the table and statements h into distributed representation for classification. This model excels at linguistic reasoning like paraphrasing and inference but lacks symbolic reasoning skills. (ii) Latent Program Algorithm, this model applies lexical matching to find linked entities and triggers to filter pre-defined APIs (e.g. argmax, argmin, count, etc). We adopt bread-first-search with memorization to construct the potential program candidates, a discriminator is further utilized to select the most "consistent" latent programs. This model excels at the symbolic reasoning aspects by executing database queries, which also provides better interpretability by laying out the decision rationale. We perform extensive experiments to investigate their performances: the best-achieved accuracy of both models are reasonable, but far below human performance. Thus, we believe that the proposed table-based fact verification task can serve as an important new benchmark towards the goal of building powerful AI that can reason over both soft linguistic form and hard symbolic forms. To facilitate future research, we released all the data, code with the intermediate results. This paper investigates a very important yet previously under-explored research problem: semistructured fact verification. We construct a large-scale dataset and proposed two methods, Table- BERT and LPA, based on the state-of-the-art pre-trained natural language inference model and program synthesis. In the future, we plan to push forward this research direction by inspiring more sophisticated architectures which can perform both linguistic and symbolic reasoning. We list all the trigger words for different functions in Figure 8 Trigger Function 'average' average 'difference ', 'gap', 'than', 'separate' diff 'sum', 'summation', 'combine', 'combined', 'total', 'add', 'all', 'there are' ddd, sum 'not', 'no', 'never', "didn't", "won't", "wasn't", "isn't,"haven't", "weren't", "won't", 'neither', 'none', 'unable, 'fail', 'different', 'outside', 'unable', 'fail' not_eq, not_within, Filter_not_eq, none 'not', 'no', 'none' none 'first', 'top', 'latest', 'most' first 'last', 'bottom', 'latest', 'most' last 'RBR', 'JJR', 'more', 'than', 'above', 'after' filter_greater, greater 'RBR', 'JJR', 'less', 'than', 'below', 'under' filter_less, less 'all', 'every', 'each' all_eq, all_less, all_greater, ['all', 'every', 'each'] , ['not', 'no', 'never', "didn't", "won't", "wasn't"] 2. Negation: the negation operation refers to sentences like "xxx did not get the best score", "xxx has never obtained a score higher than 5". 3. Superlative: the superlative operation refers to sentences like "xxx achieves the highest score in", "xxx is the lowest player in the team". 4. Comparative: the comparative operation refers to sentences like "xxx has a higher score than yyy".
We propose a new dataset to investigate the entailment problem under semi-structured table as premise This paper proposes a new dataset for table-based fact verification and introduces methods for the task. The authors propose the problem of fact verification with semi-structured data sources such as tables, create a new dataset, and evaluate baseline models with variations.