In natural language inference, the semantics of some words do not affect the inference. Such information is considered superficial and brings overfitting. How can we represent and discard such superficial information? In this paper, we use first order logic (FOL) - a classic technique from meaning representation language – to explain what information is superficial for a given sentence pair. Such explanation also suggests two inductive biases according to its properties. We proposed a neural network-based approach that utilizes the two inductive biases. We obtain substantial improvements over extensive experiments. In natural language inference (Bowman et al., 2015) , the semantics of some words do not affect the inference. In figure 1a , if we discard the semantics of some words (e.g. Avatar, fun, adults, children) from s 1 and s 2 , we obtain s 1 and s 2 , respectively. Without figuring out the specific meaning of these words, one can still infer that they are contradictory. In this case, the semantics of Avatar, fun, adults, and children are superficial for the inference. Such superficial information brings overfitting to models. Recent studies already noticed that superficial information will hurt the generalization of the model (Jia and Liang, 2017) , especially in unseen domains . Without distinguishing the superficial semantics, an NLI model can learn to predict contradiction for sentence pairs with "children" or "adults" by example 1 in Figure 1a . On the other hand, if we discard the superficial information during inference, we can prevent such overfitting. s 1 : Avatar is fun for children, not adults. s 2 : Avatar is fun for adults, not children. Label: contradiction After discarding Avatar, fun, adults, children : s 1 : A is B for C, not D. s 2 : A is B for D, not C. Label: contradiction After discarding Avatar, fun, adults, children and their correspondence information: s 1 : − is − for −, not −. s 2 : − is − for −, not −. Label: unknown (a) s 3 : Avatar is fun for all people. s 4 : Avatar is fun for adults only. Common sense: People include adults and children. Label: contradiction (b) Figure 1: Examples. Some approaches have been proposed to reduce such overfitting. HEX identifies the superficial information by projecting the textural information out. HEX defines the textural information w.r.t. the background of images for image classification, which cannot be generalized to other tasks (e.g. NLP). For NLP, the attention mechanism (Bahdanau et al., 2015) is able to discard some words by assigning them low attention scores. But such mechanism is more about the semantic similarity or relatedness of the words, not the superficial semantics. In example 1 of figure 1, the two Avatar in the two sentences will have a high attention score, since their similarity is 1 (Vaswani et al., 2017) . But we have shown that these words are superficial for inference. So previous approaches cannot be applied to modeling the superficial information in natural language inference. On top of that, a more critical issue is the lack of mathematical definition of such superficial information in previous studies. Why do people think the semantics of adults and children are superficial? In this paper, we tackle this question via the toolkit of first-order logic (FOL). FOL is a classic technique of meaning representation language, which provides a sound computational basis for the inference. We explain such superficial information from the perspective of FOL. Furthermore, such explanation suggests two inductive biases, which are used to design our NLI model. By representing natural language sentences by FOL, the sentence pair and its FOLs are logically equivalent. The conversion of figure 1a is shown in figure 2a . The entailment (resp. contradiction) between s 1 and s 2 is equivalent to F OL(s 1 ) |= F OL(s 2 ) (resp. F OL(s 1 ) |= ¬F OL(s 2 )). Thus we successfully convert the problem of identifying superficial information in NLI to identifying the superficial information in FOL inference. The superficial information exists in the non-logical symbols in FOL. From the specification of the FOL representation (Russell and Norvig, 1995) , the symbols of FOL include the logical symbols and non-logical symbols. In figure 1a , the contradiction remains if we discard the semantics of Avatar, fun, adults, children, which are non-logical symbols. We can surely change these non-logical symbols to new symbols without changing the results of F OL(s 1 ) |= F OL(s 2 ) or F OL(s 1 ) |= ¬F OL(s 2 ). However, there is a big gap between the FOL representation and the natural language: people use common sense when understanding the natural language. For example, people are able to infer the contradiction between s 3 and s 4 in figure 1b, because they have the common sense that people include adults and children. The FOLs of s 3 , s 4 and the common sense are shown in figure 2b. With the common sense, the contradiction between s 3 and s 4 is equivalent to CS ∧ F OL(s 3 ) |= ¬F OL(s 4 ), where CS denotes the FOL of the common sense. With the common sense, some non-logical symbols in the two sentences are not superficial, because we need these non-logical symbols for joint inference with the common sense. For example, in figure 2b , the non-logical symbols Adult and P eople are not superficial. This brings the major challenge of using FOL to identify the superficial information, because the common sense can hardly be obtained. Since the common sense is unknown, we restrict the definition of superficial symbols. We regard a non-logical symbol as superficial, if it is superficial for all possible common sense. We show the necessary condition of the superficial symbols to avoid the effect of the common sense, which is unknown. We show that the necessary condition is related to the semantical formula-variable (FV) independence (Lang et al., 2003) , which is NP-complete. Nevertheless, the properties of the FOL suggest two inductive biases for superficial information identification: word information discard and correspondence information representation. We propose a neural network-based approach to incorporate such two inductive biases. We point out that we need to retain the correspondence information of the discarded words. From the perspective of FOL, although the semantics of some non-logical symbols are independent for inference, the correspondence information still affects the inference. More specifically, we need to represent the occurrence of one word in different positions in the sentence pair. This is also intuitive from the perspective of natural language inference. For example, in figure 1a, although adults and children are superficial, we need to be aware that for is followed by adults in s 1 , while for is followed by adults in s 2 . Otherwise, as illustrated in s 1 and s 2 , we cannot infer their relation. We summarize our contributions in this paper below: • We proposed the problem of identifying and discarding superficial information for robust natural language inference. We use FOL to precisely define what information is superficial. • We analyze the superficial information from the perspective of FOL. We show that the superficial non-logical symbols are related to the semantical formula-variable (FV) independence in reasoning. We give two properties of the superficial information, and design neural networks to reflect the two inductive biases accordingly. • We implement a neural network-based algorithm based on the two inductive biases. The experimental results over extensive settings verify the effectiveness of our proposed method. In this paper, we study the problem of projecting superficial information out for NLI. The projection prevents models from overfitting and makes them more robust. Specially, we explain the superficial information from the perspective of FOL, and project them out in a neural network-based architecture. We conduct extensive experiments to verify the effectiveness of our proposed approach. The results verify that our proposed approaches increase the baselines by a large margin.
We use neural networks to project superficial information out for natural language inference by defining and identifying the superficial information from the perspective of first-order logic. This paper tries to reduce superficial information in natural language inference to prevent overfitting, and introduces a graph neural network to model relation between premise and hypothesis.  An approach to treat natural language inference using first-order logic and to infuse NLI models with logical information to be more robust at inference.