11institutetext:  Ural Federal University, Yekaterinburg, Russia
22institutetext: Krasovskii Institute of Mathematics and Mechanics,
Yekaterinburg, Russia
22email<EMAIL_ADDRESS>33institutetext:
Universität Hamburg, Hamburg, Germany
33email<EMAIL_ADDRESS>
# Fighting with the Sparsity of Synonymy Dictionaries for Automatic Synset
Induction

Dmitry Ustalov 1122 Mikhail Chernoskutov 1122 Chris Biemann 33
Alexander Panchenko 33

###### Abstract

Graph-based synset induction methods, such as MaxMax and Watset, induce
synsets by performing a global clustering of a synonymy graph. However, such
methods are sensitive to the structure of the input synonymy graph: sparseness
of the input dictionary can substantially reduce the quality of the extracted
synsets. In this paper, we propose two different approaches designed to
alleviate the incompleteness of the input dictionaries. The first one performs
a pre-processing of the graph by adding missing edges, while the second one
performs a post-processing by merging similar synset clusters. We evaluate
these approaches on two datasets for the Russian language and discuss their
impact on the performance of synset induction methods. Finally, we perform an
extensive error analysis of each approach and discuss prominent alternative
methods for coping with the problem of sparsity of the synonymy dictionaries.

###### Keywords:

lexical semantics $\cdot$ word embeddings $\cdot$ synset induction $\cdot$
synonyms $\cdot$ word sense induction $\cdot$ synset induction $\cdot$ sense
embeddings

##  1 Introduction

A synonymy dictionary, representing synonymy relations between the individual
words, can be modeled as an undirected graph where nodes are words and edges
are synonymy relations.111In the context of this work, we assume that synonymy
is a relation of lexical semantic equivalence which is context-independent, as
opposed to “contextual synonyms” [32]. Such a graph, called a synonymy graph
or a synonymy network, tends to have a clustered structure [7]. This property
is exploited by various graph-based word sense induction (WSI) methods, such
as [22]. The goal of such WSI methods is to build a word sense inventory from
various networks, such as synonymy graphs, co-occurrence graphs, graphs of
distributionally related words, etc. (see a survey by Navigli [19]).

The clusters are densely connected subgraphs of synonymy graph that correspond
to the groups of semantically equivalent words or synsets (sets of synonyms).
Synsets are building blocks for WordNet [5] and similar lexical databases used
in various applications, such as information retrieval [14]. Graph-based WSI
combined with graph clustering makes it possible to induce synsets in an
unsupervised way [12, 30]. However, these methods are highly sensitive to the
structure of the input synonymy graph [30], which motivates the development of
synonymy graph expansion methods.

In this paper, we are focused on the data sparseness reduction problem in the
synonymy graphs. This problem is inherent to the majority of manually
constructed lexical-semantic graphs due to the Zipf’s law of word frequencies
[33]: the long tail of rare words is inherently underrepresented. In this
work, given a synonymy graph and a graph clustering algorithm, we compare the
performance of two methods designed to improve synset induction. The goal of
each method is to improve the final synset cluster structures. Both methods
are based on the assumption that synonymy is a symmetric relation. We run our
experiments on the Russian language using the Watset state-of-the-art
unsupervised synset induction method [30].

The contribution of this paper is a study of two principally different methods
for dealing with the sparsity of the input synonymy graphs. The former,
relation transitivity method, is based on expansion of the synonymy graph. The
latter, synset merging method, is based on the mutual similarity of synsets.

##  2 Related Work

Hope and Keller [12] introduced the MaxMax clustering algorithm particularly
designed for the word sense induction task. In a nutshell, pairs of nodes are
grouped if they have a maximal mutual affinity. The algorithm starts by
converting the undirected input graph into a directed graph by keeping the
maximal affinity nodes of each node. Next, all nodes are marked as root nodes.
Finally, for each root node, the following procedure is repeated: all
transitive children of this root form a cluster and the root are marked as
non-root nodes; a root node together with all its transitive children form a
fuzzy cluster.

Van Dongen[3] presented the Markov Clustering (MCL) algorithm for graphs based
on simulation of stochastic flow in graphs. MCL simulates random walks within
a graph by alternation of two operators called expansion and inflation, which
recompute the class labels. This approch has been successfully used for the
word sense induction task [4].

Biemann [1] introduced Chinese Whispers, a clustering algorithm for weighted
graphs that can be considered as a special case of MCL with a simplified class
update step. At each iteration, the labels of all the nodes are updated
according to the majority labels among the neighboring nodes. The author
showed usefulness of the algorithm for induction of word senses based on
corpus-induced graphs.

The ECO approach [8] was applied to induce a WordNet of the Portuguese
language.222http://ontopt.dei.uc.pt In its core, ECO is based on a clustering
algorithm that was used to induce synsets from synonymy dictionaries. The
algorithm starts by adding random noise to edge weights. Then, the approach
applies Markov Clustering of this graph several times to estimate the
probability of each word pair being in the same synset. Finally, candidate
pairs over a certain threshold are added to output synsets.

In our experiments, we rely on the Watset synset induction method [30] based
on a graph meta-clustering algorithm that combines local and global hard
clustering to obtain a fuzzy graph clustering. The authors shown that this
approach outperforms all methods mentioned above on the synset induction task
and therefore we use it as the strongest baseline to date.

Meyer and Gurevich [17] presented an approach for construction of an
ontologized version of Wiktionary, by formation of ontological concepts and
relationships between them from the ambiguous input dictionary, yet their
approach does not involve graph clustering.

##  3 Two Approaches to Cope with Dictionary Sparseness

We propose two approaches for dealing with the incompleteness of the input
synonymy dictionaries of a graph-based synset induction method, such as Watset
or MaxMax. First, we describe a graph-based approach that preprocesses the
input graph by adding new edges. This step is applied before the synset
induction clustering. Second, we describe an approach that post-processes the
synsets by merging highly semantically related synsets. This step is applied
after the synset induction clustering step, refining its results.

###  3.1 Expansion of Synonymy Graph via Relation Transitivity

Assuming that synonymy is an equivalence relation due to its reflexiveness,
symmetry, and transitivity, we can insert additional edges into the synonymy
graph between nodes that are transitively, synonymous, i.e. are connected by a
short path of synonymy links. We assume that if an edge for a pair of synonyms
is missing, the graph still contains several relatively short paths connecting
the nodes corresponding to these words.

Firstly, for each vertex, we extract its neighbors and the neighbors of these
neighbors. Secondly, we compute the set of candidate edges by connecting the
disconnected vertices. Then, we compute the number of simple paths between the
vertices in candidate edges. Finally, we add an edge into the graph if there
are at least $k$ such paths which lengths are in the range $[i,j]$.

Particularly, the algorithm works as follows:

  1. 1.

extract a first-order ego network $N_{1}$ and a second-order ego network
$N_{2}$ for each node;

  2. 2.

generate the set of candidate edges that connect the disconnected nodes in
$N_{1}$, i.e., the total number of the candidates is
$C_{|N_{1}|}^{2}-|E_{N_{1}}|$, where $C_{|N_{1}|}^{2}$ is the number of all
2-combinations over the $|N_{1}|$-element set and $E_{N_{1}}$ is the set of
edges in $N_{1}$;

  3. 3.

keep only those edge candidates that satisfy two conditions: 1) there are at
least $k$ paths $p$ in $N_{2}$, so no path contains the initial ego node, and
2) the length of each path belongs to the interval $[i,j]$.

The approach has two parameters: the minimal number of paths to consider $k$
and the path length interval $[i,j]$. It should be noted that this approach
processes the input synonymy graph without taking the polysemous words into
account. Such words are then handled by the Watset algorithm that induces word
senses based on the expanded synonymy graph.

###  3.2 Synset Merging based on Synset Vector Representations

We assume that closely related synsets carry equivalent meanings and use the
following procedure to merge near-duplicate synsets:

  1. 1.

learn synset embeddings for each synset using the SenseGram method by simply
averaging word vectors that correspond to the words in the synset [26];

  2. 2.

identify the closely related synsets using the $m\text{-}k\textit{NN}$
algorithm [20] that considers two objects as closely related if they are
mutual neighbors of each other;

  3. 3.

merge the closely related synsets in a specific order: the smallest synsets
are merged first, the largest are merged later; every synset can be merged
only once in order to avoid giant merged clusters.

This approach has two parameters: the number of nearest neighbors to consider
$k$ (fixed to $10$ in our experiments)333In general, the
$m\text{-}k\textit{NN}$ method can be parametrized by two different
parameters: $k_{ij}$ – the number of nearest neighbors from the word $i$ to
the word $j$ and $k_{ji}$ – the number of nearest neighbors from the word $j$
to the word $i$. In our case, for simplicity, we set $k_{ij}=k_{ji}=k$.  and
the maximal number of merged synsets $t$, e.g., if $t=1$ then only the first
mutual nearest neighbor is merged. It should be noted that this approach
operates on synsets that which are already have been discovered by Watset.
Therefore, the merged synsets are composed of disambiguated word senses.

##  4 Evaluation

We evaluate the performance of the proposed approaches using the Watset graph
clustering method that shows state-of-the-art results on synset induction
[30]. Watset is a meta-clustering algorithm that disambiguates a (word) graph
by first performing ego-network clustering to split nodes (words) into (word)
senses. Then a global clustering is used to form (syn)sets of senses. For both
clustering steps, any graph clustering algorithm can be employed; in [30], it
was shown that combinations of Chinese Whispers [1] (CW) and Markov Clustering
[3] (MCL) provide the best results. We also evaluated the same approaches with
the MaxMax [12] method, but the results were virtually the same, so we omitted
them for brevity.

###  4.1 Datasets

We evaluate the proposed augmentation approaches on two gold standard datasets
for Russian: RuWordNet [15] and YARN [2]. Both are analogues of the original
English WordNet [5].

We used the same input graph as in [30]; the graph is based on three synonymy
dictionaries, the Russian Wiktionary, the Abramov’s dictionary and the UNLDC
dictionary. The graph is weighted using the similarities from Russian
Distributional Thesaurus (RDT) [23].444http://russe.nlpub.ru/downloads. To
construct synset embeddings, we used word vectors from the RDT.

The lexicon of the input dictionary is different from the lexicon of RuWordNet
[15], which includes a lot of domain-specific synsets. At the same time, the
input dataset is the same as the data sources used for boostrapping YARN [2].

The summary of the datasets is shown in Table 1: the ‘‘# words’’ column
specifies the number of lexical units in the dataset (nodes of the input
graph), the ‘‘# synonyms’’ column indicates the number of synonymy pairs
appearing in the dataset (edges of the input graph). The problem of dictionary
sparsity is the fact that some edges (synonyms) are missing in the input
resource. Finally, the ‘‘# synsets’’ column specifies the number of resulting
synsets (if applicable).

Table 1: Summary of the datasets used in the experiments. Resource |  # words |  # synsets |  # synonyms
---|---|---|---
Input Synonymy Dictionary: Wiktionary |  $83\,092$ |  n/a |  $211\,986$
Induced Synsets: Watset MCL-MCL |  $83\,092$ |  $36\,217$ |  $406\,430$
Induced Synsets: Watset CW-MCL |  $83\,092$ |  $55\,369$ |  $355\,158$
Gold Synsets: RuWordNet |  $110\,242$ |  $49\,492$ |  $278\,381$
Gold Synsets: YARN |  $9\,141$ |  $2\,210$ |  $48\,291$

###  4.2 Quality Measures

We report results according to standard word sense induction evaluation
measures: paired precision, recall and F-score [16], i.e., each cluster of $n$
words yields $\frac{n(n-1)}{2}$ synonymy pairs. The exact same evaluation
protocol was used in the original Watset publication. We perform evaluation on
the intersection of gold standard lexicon and the lexicon of the induced
resource.

###  4.3 Results

The evaluation results are shown in Figure 1. As one may observe, in the case
of the RuWordNet dataset, the method based on the transitivity expansion
rendered almost no improvements in terms of recall while dramatically dropping
the precision. The second method, based on synset embeddings shows much better
results on this dataset: It substantially improves recall, yet at the cost of
a drop in precision.

In case of the YARN dataset, the results are similar with the graph-based
method significantly lagging behind the vector-based method. However, in this
case, the difference in the observed performance is smaller with some
configurations of the graph-based methods approaching the performance of the
vector-based method. Similarly to the first dataset, both methods trade off
gains in recall for the drops in precision. Note, however, that the vector-
based method can perform a shift of the ‘‘sweet spot’’ of the clustering
approach. While the F-measure remains at the same level, it is possible to
obtain higher levels of recall, which can be useful for some applications. In
the following section, we perform error analysis for each method.

(a) RuWordNet (b) Yet Another RussNet

Figure 1: Precision-recall plots built on two gold standard datasets for
Russian. The shapes and colors:
$\color[rgb]{0.13671875,0.546875,0.26953125}\definecolor[named]{pgfstrokecolor}{rgb}{0.13671875,0.546875,0.26953125}\blacktriangle$
original graph,
$\color[rgb]{0.12890625,0.44140625,0.7109375}\definecolor[named]{pgfstrokecolor}{rgb}{0.12890625,0.44140625,0.7109375}\blacksquare$
transitivity expansion with the allowed path lengths $2\leq i\leq j\leq 3$ and
the number of simple paths $k\leq 10$,
$\color[rgb]{0.796875,0.09375,0.11328125}\definecolor[named]{pgfstrokecolor}{rgb}{0.796875,0.09375,0.11328125}\bullet$
synset merging with the maximal number of merged mutual neighbours
$t\in\\{1,2,3,5,10\\}$ and the number of nearest neighbours set to $k=10$.
Chinese Whispers-based Watset configurations are hollow, while MCL-based are
solid.

##  5 Discussion

Perfect synonyms are very rare, which is confirmed by the precision-recall
plot in Figure 1. Both methods insert relations of other types, such as
association, co-hyponymy, hypernymy, etc. Recall increases with the level of
inclusiveness of the configuration; this also causes significant drops in
precision. The expansion methods presented in this paper could therefore be
more useful for generation of other types of symmetric semantic relations,
such as co-hyponymy.

###  5.1 Error Analysis: Synonymy Transitivity

We tried the following configurations of the approach: $2\leq i\leq j\leq 3$,
$k\leq 10$. However, only the variations with a small allowed length $i=j=2$
and a high number of found simple paths $k\geq 5$ yielded viable results.

We explain the quick drops in precision by the fact that no word is a perfect
synonym of another [10]. This results in the potential loss of the synonymy
relation on each additional transitive node. While having a lot of pertinent
edge insertions like ‘‘ оказываться–появляться (show up – appear)’’ or
‘‘подтрунивать–стебаться (prank – make jokes)’’, this method introduces such
false positives like ‘‘кий–хлыст (cue – whip)’’, ‘‘шеф–царь (boss – tsar)’’,
‘‘солидный–корректный (solid – correct)’’, etc.

One of the reasons of this outcome is that adding new edges increases the size
of the communities. They capture neighboring vertices and edges belonging to
other communities in the initial graph. Hence, on the one hand, we obtain
communities with excess elements, while on the other hand, we observe depleted
communities.

###  5.2 Error Analysis: Synset Merging

The different configurations of the vector-based method in this plot
correspond to the following values of the $t$ parameter (the maximum number of
merged synsets): 1, 2, 3, 5, 10. Merging more than one synset at a time
provides a substantial gain in the recall, yet again at the cost of the
precision drop.

Table 2 presents an example of correct merging of synsets. The results of the
clustering generate multiple small synsets that refer to the same meaning.
Such synsets tend to be mutual nearest neighbors. Top ten most similar synsets
to the synset ‘‘cynicism’’ are depicted. In this table, we also indicate
whether each neighbor is the mutual nearest neighbor or not. In this example,
the method of mutual nearest neighbors perfectly achieves its goal of merging
synonymous synsets.

Table 3 presents an example of a wrong merging of synsets on the example of
the synset ‘‘zinc, Zn’’. This sample illustrates the reasons behind the drops
in precision. While different chemical elements, such as zinc and cobalt are
strongly semantically related, they are co-hyponyms of the common hypernym
‘‘chemical element’’, and not synonyms, i.e. terms with equivalent meanings.
This result is in line with the prior results showing that the majority of the
nearest neighbors delivered by the distributional semantic models, such as the
skip-gram model [18] used in our experiments, tend to be co-hyponyms as shown
in prior studies [31, 11, 25, 24]. The results presented in both Table 2 and 3
have been manually annotated by a single expert.

Table 2: An example of correct synset merging, where predicted labels are equal to gold labels for the top $k=10$ nearest neighbours of the synset ‘‘цинизм, циничность (cynicism, cynicism)’’. In this table, the ‘‘Predicted’’ column contains mutually related synsets, while the ‘‘Gold’’ column lists expert judgments. $k$ | Similarity |  Related Synset | Predicted | Gold
---|---|---|---|---
1 | 0.866 |  беспринципность, цинизм (unprincipledness, cynicism) | true | true
2 | 0.856 |  беспринципность, циничность (unprincipledness, cynicism) | true | true
3 | 0.853 |  кинизм, беспардонность, цинизм (cynicism, shamelessness, cynicism) | true | true
4 | 0.734 |  нахрапистость, нахальство, нахальность, циничность, бесцеремонность, нецеремонность (сheekiness, impudence, cheekiness, cynicism, brusqueness, unceremoniousness) | false | false
5 | 0.677 |  грубость, примитивизм (rudeness, primitivism) | false | false
6 | 0.677 |  хамство, лапидарность, хамёж, топорность, грубость, прямолинейность (rudeness, conciseness, rudeness, clumsiness, rudeness, straightness) | false | false
7 | 0.674 |  безнравственность, беспринципность, злонравие, аморальность (wickedness, lack of principles, depravity, immorality) | false | false
8 | 0.671 |  бесстыдство, непристойность, бессовестность, нахрап (immorality, lack of principle, malice, immorality) | false | false
9 | 0.663 |  скепсис, скептичность (skepticism, skepticism) | true | false
10 | 0.661 |  фанатизм, ханжество (bigotry, bigotry) | false | false
Table 3: An example of wrong synset merging, where predicted labels are not equal to gold labels for the top $k=10$ nearest neighbours of the synset ‘‘цинка, Zn (zinc, Zn)’’. In this table, the ‘‘Predicted’’ column contains mutually related synsets, while the ‘‘Gold’’ column lists expert judgments  $k$ | Similarity |  Related Synset | Predicted | Gold
---|---|---|---|---
1 | 0.676 |  станнат кобальта, кобальт (cobalt stannate, cobalt) | true | false
2 | 0.673 |  Mg, магний (Mg, magnesium) | true | false
3 | 0.670 |  глиний, крылатый металл, алюминий, Al (сlay, winged metal, aluminum, Al) | false | false
4 | 0.663 |  фосфор, P (phosphorus, P) | true | false
5 | 0.646 |  оксид, окись (oxide, oxide) | false | false
6 | 0.631 |  гидрооксид, гидроокись, гидроксид (hydroxide, hydroxide, hydroxide) | false | false
7 | 0.630 |  ванадий, V (vanadium, V) | true | false
8 | 0.628 |  рибофлавин, лактофлавин, витамин В (riboflavin, lactoflavin, vitamin B) | false | false
9 | 0.624 |  иодат, йодид, йодат (iodate, iodide, iodate) | false | false
10 | 0.618 |  кремний, Si (silicon, Si) | false | false

###  5.3 Other Ways to Deal with Sparseness of the Input Dictionary

In this section, we discuss avenues for future work: the prominent approaches
that might be useful in addressing the sparseness of the synonym dictionaries.

##### Lexical-syntactic patterns for extraction of synonyms.

Hearst patterns are widely used to mine hypernymy relations from text [27].
Such patterns can be also learned automatically [29, 28]. In [21], seven
patterns for extraction of synonyms were proposed, which function in the same
was as the Hearst patterns for hypernymy extraction. Such synonymy extraction
patterns can be also learned automatically in the same fashion as patterns for
hypernymy extraction from text [29]. Finally, hypernyms, antonyms, and other
relations extracted from text can be used to filter our non-synonymous
candidates.

##### Global clustering of synsets.

It is possible to find groups of semantically related words (clique-like
structures) and expand synonyms only within such communities with a graph
clustering algorithm, such as Chinese Whispers [1]. The current graph-based
transitivity expansion method does not consider the structure of the
communities of the synonymy graph.

##### Synonymy detection as an anaphora resolution problem.

Another observation is that synonyms are often not used in the same sentence,
but instead used to ensure linguistic variance of the text. In this respect,
synonymy extraction task is similar to the anaphora resolution task [13]. This
line of work is related to prior work of [6] in detecting ‘‘bridging
mentions’’.

##### Crowdsourcing.

Finally, the last option is simply to improve the quality of the input
dictionaries by the means of crowdsourcing [9]. Namely, involving more people
to edit Wiktionary that we use as the input data will increase the coverage of
the extracted synsets, but large-scale crowdsourcing requires a set of
elaborated quality control measures.

##  6 Conclusion

In this paper, we explored two alternative strategies for coping with the
problem of inherent sparsity and incompleteness of the synonymy dictionaries.
These sparsity issues hamper performance of the methods for automatic
induction of synsets, such as MaxMax [12] and Watset [30]. One of the proposed
methods performs pre-processing of the graph of synonyms, while the second one
performs post-processing of the induced synsets.

Our experiments on two large scale datasets show that (1) both methods are
able to substantially improve recall, but at the cost of substantial drops of
precision; (2) the post-processing approach yields better results overall. We
conclude our study with an overview of prominent alternative approaches for
expansion of incomplete synonymy dictionaries.

We believe the results of our study will be useful for both enriching the
available lexical semantic resources like OntoWiktionary [17] as well as for
increasing the lexical coverage of the input data for the graph-based word
sense induction methods.

#### Acknowledgements.

We acknowledge the support of the Deutsche Forschungsgemeinschaft (DFG) under
the ‘‘JOIN-T’’ project, the DAAD, the RFBR under the projects no. 16-37-00203
мол_а and no. 16-37-00354 мол_а, and the RFH under the project no.
16-04-12019. The calculations were carried out using the supercomputer
‘‘Uran’’ at the Krasovskii Institute of Mathematics and Mechanics. Finally, we
also thank four anonymous reviewers for their helpful comments.

## References

  * [1] Biemann, C.: Chinese Whispers: An Efficient Graph Clustering Algorithm and Its Application to Natural Language Processing Problems. In: Proceedings of the First Workshop on Graph Based Methods for Natural Language Processing. pp. 73–80. TextGraphs-1, Association for Computational Linguistics, New York, NY, USA (2006)
  * [2] Braslavski, P., Ustalov, D., Mukhin, M., Kiselev, Y.: YARN: Spinning-in-Progress. In: Proceedings of the 8th Global WordNet Conference. pp. 58–65. GWC 2016, Global WordNet Association, Bucharest, Romania (2016)
  * [3] van Dongen, S.: Graph Clustering by Flow Simulation. Ph.D. thesis, University of Utrecht (2000)
  * [4] Dorow, B., Widdows, D.: Discovering Corpus-Specific Word Senses. In: Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics - Volume 2. pp. 79–82. EACL ’03, Association for Computational Linguistics, Budapest, Hungary (2003)
  * [5] Fellbaum, C.: WordNet: An Electronic Database. MIT Press (1998)
  * [6] Feuerbach, T., Riedl, M., Biemann, C.: Distributional Semantics for Resolving Bridging Mentions. In: Proceedings of the International Conference Recent Advances in Natural Language Processing. pp. 192–199. INCOMA Ltd. Shoumen, BULGARIA, Hissar, Bulgaria (2015)
  * [7] Gfeller, D., Chappelier, J.C., De Los Rios, P.: Synonym Dictionary Improvement through Markov Clustering and Clustering Stability. In: Proceedings of the International Symposium on Applied Stochastic Models and Data Analysis. pp. 106–113 (2005)
  * [8] Gonçalo Oliveira, H., Gomes, P.: ECO and Onto.PT: a flexible approach for creating a Portuguese wordnet automatically. Language Resources and Evaluation 48(2), 373–393 (2014)
  * [9] Gurevych, I., Kim, J. (eds.): The People’s Web Meets NLP. Springer Berlin Heidelberg (2013)
  * [10] Herrmann, D.J.: An old problem for the new psycho-semantics: Synonymity. Psychological Bulletin 85(3), 490–512 (1978)
  * [11] Heylen, K., Peirsman, Y., Geeraerts, D., Speelman, D.: Modelling Word Similarity: an Evaluation of Automatic Synonymy Extraction Algorithms. In: Proceedings of the Sixth International Conference on Language Resources and Evaluation. pp. 3243–3249. LREC 2008, European Language Resources Association, Marrakech, Morocco (2008)
  * [12] Hope, D., Keller, B.: MaxMax: A Graph-Based Soft Clustering Algorithm Applied to Word Sense Induction. In: Computational Linguistics and Intelligent Text Processing: 14th International Conference, CICLing 2013, Samos, Greece, March 24-30, 2013, Proceedings, Part I, pp. 368–381. Springer Berlin Heidelberg, Berlin, Heidelberg (2013)
  * [13] Lappin, S., Leass, H.J.: An Algorithm for Pronominal Anaphora Resolution. Computational Linguistics 20(4), 535–561 (1994)
  * [14] Loukachevitch, N.V.: Thesauri in information retrieval tasks. Moscow University Press, Moscow, Russia (2011), in Russian.
  * [15] Loukachevitch, N.V., Lashevich, G., Gerasimova, A.A., Ivanov, V.V., Dobrov, B.V.: Creating Russian WordNet by Conversion. In: Computational Linguistics and Intellectual Technologies: papers from the Annual conference ‘‘Dialogue’’. pp. 405–415. RSUH, Moscow, Russia (2016)
  * [16] Manandhar, S., Klapaftis, I., Dligach, D., Pradhan, S.: SemEval-2010 Task 14: Word Sense Induction & Disambiguation. In: Proceedings of the 5th International Workshop on Semantic Evaluation. pp. 63–68. Association for Computational Linguistics, Uppsala, Sweden (2010)
  * [17] Meyer, C.M., Gurevyich, I.: OntoWiktionary: Constructing an Ontology from the Collaborative Online Dictionary Wiktionary, pp. 131–161. IGI Global, Hershey, PA, USA (2012)
  * [18] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed Representations of Words and Phrases and their Compositionality. In: Advances in Neural Information Processing Systems 26, pp. 3111–3119. Curran Associates, Inc., Harrahs and Harveys, NV, USA (2013)
  * [19] Navigli, R.: A Quick Tour of Word Sense Disambiguation, Induction and Related Approaches. In: Proceedings of the 38th International Conference on Current Trends in Theory and Practice of Computer Science, pp. 115–129. SOFSEM’12, Springer-Verlag, Berlin, Heidelberg (2012)
  * [20] Panchenko, A., Adeykin, S., Romanov, A., Romanov, P.: Extraction of Semantic Relations between Concepts with KNN Algorithms on Wikipedia. In: Proceedings of the 2nd International Workshop on Concept Discovery in Unstructured Data. pp. 78–86. No. 871 in CEUR Workshop Proceedings, Leuven, Belgium (2012)
  * [21] Panchenko, A., Morozova, O., Naets, H.: A Semantic Similarity Measure Based on Lexico-Syntactic Patterns. In: Proceedings of KONVENS 2012. pp. 174–178. ÖGAI (2012)
  * [22] Panchenko, A., Simon, J., Riedl, M., Biemann, C.: Noun Sense Induction and Disambiguation using Graph-Based Distributional Semantics. In: Proceedings of the 13th Conference on Natural Language Processing. pp. 192–202. KONVENS 2016, Bochumer Linguistische Arbeitsberichte (2016)
  * [23] Panchenko, A., Ustalov, D., Arefyev, N., Paperno, D., Konstantinova, N., Loukachevitch, N., Biemann, C.: Human and Machine Judgements for Russian Semantic Relatedness, pp. 221–235. Springer International Publishing, Cham, Switzerland (2017)
  * [24] Panchenko, A.: Comparison of the baseline knowledge-, corpus-, and web-based similarity measures for semantic relations extraction. In: Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics. pp. 11–21. Association for Computational Linguistics, Edinburgh, UK (2011)
  * [25] Peirsman, Y., Heylen, K., Speelman, D.: Putting things in order. First and second order context models for the calculation of semantic similarity. In: Proceedings of the 9th Journées internationales d’Analyse statistique des Données Textuelles. pp. 907–916. JADT 2008, Lyon, France (2008)
  * [26] Pelevina, M., Arefyev, N., Biemann, C., Panchenko, A.: Making Sense of Word Embeddings. In: Proceedings of the 1st Workshop on Representation Learning for NLP. pp. 174–183. Association for Computational Linguistics, Berlin, Germany (2016)
  * [27] Seitner, J., Bizer, C., Eckert, K., Faralli, S., Meusel, R., Paulheim, H., Ponzetto, S.P.: A Large Database of Hypernymy Relations Extracted from the Web. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. pp. 360–367. LREC 2016, European Language Resources Association (ELRA), Portorož, Slovenia (2016)
  * [28] Shwartz, V., Goldberg, Y., Dagan, I.: Improving Hypernymy Detection with an Integrated Path-based and Distributional Method. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2389–2398. Association for Computational Linguistics, Berlin, Germany (2016)
  * [29] Snow, R., Jurafsky, D., Ng, A.Y.: Learning Syntactic Patterns for Automatic Hypernym Discovery. In: Proceedings of the 17th International Conference on Neural Information Processing Systems. pp. 1297–1304. NIPS’04, MIT Press, Vancouver, British Columbia, Canada (2004)
  * [30] Ustalov, D., Panchenko, A., Biemann, C.: Watset: Automatic Induction of Synsets from a Graph of Synonyms. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1579–1590. Association for Computational Linguistics, Vancouver, Canada (2017)
  * [31] Wandmacher, T.: How semantic is Latent Semantic Analysis? In: Proceedings of RÉCITAL 2005. pp. 525–534. Dourdan, France (2005)
  * [32] Zeng, X.M.: Semantic relationships between contextual synonyms. ERIC 4(9), 33–37 (2007)
  * [33] Zipf, G.K.: The psycho-biology off language. Boston: I-Ioughton-Mifflin (1935)


# Mind Reasoning Manners: Enhancing Type Perception for Generalized Zero-shot
Logical Reasoning over Text

Fangzhi Xu, Jun Liu, Qika Lin, Tianzhe Zhao, Jian Zhang, and Lingling Zhang
Fangzhi Xu, Qika Lin, Tianzhe Zhao, Jian Zhang are with School of Computer
Science and Technology, Xi’an Jiaotong University, Xi’an, China.
Jun Liu and Lingling Zhang are with Shaanxi Province Key Laboratory of
Satellite and Terrestrial Network Tech. R&D, National Engineering lab for Big
Data Analytics, Xi’an, China.
The corresponding author is Jun Liu.

###### Abstract

Logical reasoning task involves diverse types of complex reasoning over text,
based on the form of multiple-choice question answering. Given the context,
question and a set of options as the input, previous methods achieve superior
performances on the full-data setting. However, the current benchmark dataset
has the ideal assumption that the reasoning type distribution on the train
split is close to the test split, which is inconsistent with many real
application scenarios. To address it, there remain two problems to be studied:
(1) How is the zero-shot capability of the models (train on seen types and
test on unseen types)? (2) How to enhance the perception of reasoning types
for the models? For problem 1, we propose a new benchmark for generalized
zero-shot logical reasoning, named ZsLR. It includes six splits based on the
three type sampling strategies. For problem 2, a type-aware model TaCo is
proposed. It utilizes both the heuristic input reconstruction and the
contrastive learning to improve the type perception in the global
representation. Extensive experiments on both the zero-shot and full-data
settings prove the superiority of TaCo over the state-of-the-art methods.
Also, we experiment and verify the generalization capability of TaCo on other
logical reasoning dataset.

###### Index Terms:

Natural Language Processing, Logical Reasoning, Question Answering,
Generalized Zero-shot.

##  1 Introduction

Logical reasoning over text has aroused wide interest in the area of Machine
Reading Comprehension (MRC) [1] and Natural Language Processing (NLP) [2][3]
recently. In the form of the traditional multiple-choice question answering
(MCQA) [4][5][6], the task of logical reasoning requires the model to perform
complex reasoning and generalization. One of the main difficulties of the task
lies in addressing diverse reasoning types. Fig. 1 shows some examples of
reasoning types in the logical reasoning task. Given questions with different
reasoning types, humans tend to focus on the respective aspects of
interactions between the context and the option. For instance, for the type
_Identify the flaw_ (a), the option is strongly related to the detailed logic
flaws within the global idea. While for the type _Necessary assumption_ (b),
the focus may be switched to the premise of the arguments and detect the
missing assumption reflected by the option. Also, for the reasoning type of
_Parallel reasoning_ (c), it is required to consider the corresponding logical
structure of the context and the option, rather than the specific entities or
events. Therefore, the modeling of the specific reasoning type is intuitive
and necessary to the logical reasoning task.

Figure 1: Examples of different reasoning types.

Recent works have witnessed improvements in the logical reasoning tasks.
Generally, they can be categorized into two folds: graph-based and data-based.
In the graph-based family, DAGN [7], FocalReasoner [8] and Logiformer [9]
attempt to construct the context graphs from different levels, such as causal
and co-occurrence. And AdaLoGN [10] proposes a neural-symbolic system in an
adaptive manner. In the data-based family, previous works explore various data
augmentation strategies. For example, LReasoner [11] extracts the symbols from
the text and extends them with logical rules. MERIt [12] designs several data
generation methods to facilitate the training process. However, all of the
methods in the two families lack the modeling of the reasoning type features.
Although the above mentioned methods achieve superior results over strong
baselines, the current researches are based on the ideal setting that train
and test splits share the similar distribution of reasoning types.
Nevertheless, in the real application, we are mainly exposed to the common
types of questions while unfamiliar with the reasoning on the novel and
uncommon types. In another word, there is an obvious gap between the ideal
setting and the real scenarios. To address this issue, there exist two
problems to be studied: 1) What about the test performance on the unseen types
during training and how is the zero-shot capability of the models? 2) How to
enhance the perception of reasoning types for the models?

As for Problem 1, we propose a new benchmark for the generalized Zero-shot
Logical Reasoning, named ZsLR. Based on the ReClor dataset [13] with 17
reasoning types, we form 6 zero-shot data splits according to 3 strategies
(i.e., amount, randomness, and difficulty). To make comprehensive assessments,
we introduce the generalized zero-shot setting [14], which test on both seen
types and unseen types with two defined metrics. The necessity and meaning of
ZsLR is verified by the pilot experiments and it encourages more future works
to mind reasoning manners.

As for Problem 2, we propose a Type-aware reasoning network based on
Contrastive learning, named TaCo. First, we design a keyword-based extractor
to output the reasoning type. Then, through the two designed heuristic
strategies, we merge each question and the option into a unified _Q-A_ pair.
Based on the co-occurrence node extraction algorithm proposed in Logiformer
[9], we form the topology within context part and _Q-A_ pair part
respectively. To model the interaction of the context and _Q-A_ pair for
different reasoning types, we add a global node to connect all the nodes of
the two parts. Through the self-attention aggregation, the final
representation of the global node is obtained and utilized to predict the
answer. Meanwhile, we employ the sentence-BERT [15] to obtain type embeddings
based on their descriptions. The margin loss is applied to model the reasoning
type, where the global node serves as the anchor, ground truth type as
positive example and other types as negative examples. In this way, the global
node representation contains both the context and the reasoning type
semantics. Therefore, the perception of the reasoning types can facilitate the
zero-shot capability.

In all, the main contributions of this paper are summarized as follows:

(1) We are the first to focus on the issue of type-oriented reasoning manners.
To address the issue, we propose the first benchmark: generalized ZsLR.111The
ZsLR dataset and implementation of TaCo are public at
github.com/xufangzhi/TaCo It can well reflect the real scenarios and test the
zero-shot capabilities of the models.

(2) We propose to tackle the zero-shot task through the heuristic input
reconstruction and type-aware contrastive learning. The proposed model TaCo
can function as a strong baseline for the future works on the task of ZsLR.

(3) Extensive experiments on both zero-shot and full-data settings prove the
huge potential of the proposed issue, as well as the superiority of TaCo.
Also, we conduct additional experiments to further verify the generalization
capability of TaCo on other setting and dataset.

##  2 Related Work

###  2.1 Logical Reasoning

The logical reasoning task, which aims at testing the reasoning capability of
models, has aroused wide interest. Several representative datasets have been
proposed, such as ReClor [13] and LogiQA [16]. Current methods on the logical
reasoning task can be divided into graph-based methods and data-based methods.
The former focuses on the construction of the text graphs and leverage the
node connection to model the logical relations. Among this category, DAGN [7]
is the first work to split the text into EDUs and perform the reasoning with
graph neural networks [17][18], but it only builds a chain-type graph and
ignores the long distance interaction between nodes. FocalReasoner [8]
proposes to attend to the fact triplets [19] within the text and constructs a
supergraph based on the extracted triplets. However, it lacks the modeling of
the logical information in the text. To fully explore the logic, AdaLoGN [10]
is proposed to construct the adaptive neural-symbolic system to improve the
performances. However, its reasoning process is complex and costly.
Considering the above drawbacks, Logiformer [9] stresses much importance on
both the causal relations and the co-occurrence relations in a two-branch
graph transformer network. Yet it still lacks the perception of different
question types, which limits the zero-shot reasoning capability of the model.
The latter one is the data-based methods. These works aim to improve the
performance through the data augmentation strategy. One of the representatives
is LReasoner [11], which extends the symbolic expressions by logical rules and
templates. In addition, MERIt [12] designs a meta-path-guided contrastive
learning method to facilitate the training process, by utilizing the extra
data. However, both of them fail to model the reasoning type of each question
and lack the application value in zero-shot setting.

###  2.2 Current Machine Reading Comprehension Settings

Current machine reading comprehension (MRC) tasks [20] can be categorized into
two settings: full-data setting and low-resource (i.e., few-shot, zero-shot)
setting.

Full-data setting has aroused wide concerns in the area of MRC in recent
years. Popular benchmark SQuAD [21] [22], which is sourced from from Wikipedia
articles, contains over 100,000 questions. It forms an abundant database for
the model training. Similarly, HotpotQA [23] includes 113K questions with a
variety of reasoning strategies. It stresses more on the multi-hop reasoning
abilities of the model. RACE dataset [24] is specially designed to improve the
reading skills for both middle school and high school students, which contains
about 9.7K examples. Also, some domain-specific datasets like NewsQA [25],
TextbookQA [26], PubMedQA [27], aim at the area of journalism, education and
biology respectively. All of them have rich training examples to improve the
model capability. However, such full-data setting encourages the models to
depend on the ideal scenes with abundant training data, which may not fit in
some real cases.

Based on such concern, some previous works focus on the low-resource setting
MRC, including few-shot or zero-shot setting. For example, to evaluate the
robustness of the model, [28] reconstructs the data from knowledge sources for
zero-shot commonsense QA. Also, [29] proposes a neuro-symbolic approach to
boost the performance of zero-shot commonsense QA. Both of the methods are not
exposed to the training data, forming a zero-shot setting. However, they rely
on the extra knowledge bases, which limits their applications. There are also
some works related to the few-shot QA. In [30], a new pretraining strategy is
proposed to explore the realistic few-shot setting. Also, [31] tackles the
few-shot challenge on the task of Visual Question Answering. However, all of
the above methods simply obtain the few-shot splits according to the amount,
which treat all the samples equally.

To sum up, current MRC benchmarks mainly focus on the common types of
questions. However, in reality, it is more challenging when faced with some
uncommon types of questions. To make up for these drawbacks, we attend to the
reasoning types in the logical reasoning tasks and propose the first benchmark
for zero-shot logical reasoning based on type attributes.

##  3 The Benchmark of Generalized ZsLR

In this section, we introduce the generalized ZsLR benchmark. At the very
beginning, we obtain the statistical distribution of the number of reasoning
types on the train, development and test splits, shown in Fig. 2. We arrange
them in descending order of the number. The distributions are in the similar
form. It demonstrates that the full-data setting is based on such an ideal
assumption, which is insufficient to verify the zero-shot generalization
capability of the models. Therefore, we propose the benchmark of generalized
ZsLR and conduct the pilot experiments to verify the necessity.

###  3.1 Zero-shot Data Construction

The zero-shot logical reasoning datasets are split based on ReClor [13]
without shuffling. Considering the situation in reality, it is easier to learn
to reason on common types of samples while struggle with the rare ones. To
this end, we design three sampling strategies, namely _amount_ , _randomness_
and _difficulty_.

TABLE I: Details of the split datasets for the zero-shot setting. The third column presents the index of the seen types and the fourth column shows the number of the seen and unseen types. The last two columns present the number of training samples and test samples respectively. Especially, test samples contain both seen and unseen types for generalized zero-shot setting. Strategy | Split | Seen Type |  # Type Seen (Unseen) |  # Train Seen |  # Test Seen (Unseen)
---|---|---|---|---|---
Amount | v1 | {0,3,4,8,13} | 5 (12) | 2,190 | 475 (525)
v2 | {0,1,2,3,8,9,14,16} | 8 (9) | 2,700 | 595 (405)
Randomness | v3 | {0,2,3,13} | 4 (13) | 1,928 | 435 (565)
v4 | {0,2,3,5,7,8,13} | 7 (10) | 2,896 | 645 (355)
Difficulty | v5 | {0,2,4,6,8,13,15} | 7 (10) | 2,175 | 473 (527)
v6 | {1,3,5,7,9,10,11,12,14,16} | 10 (7) | 2,463 | 527 (473)

For _amount_ , we select top-k reasoning types as the seen types, merely by
the amount. It can be seen as a simple implementation to filter the uncommon
types of samples.

For _randomness_ , we arrange the reasoning types in descending order of
amount and select the seen ones based on the geometric distribution. The
discrete form of the geometric distribution is,

$P\left({X=k}\right)={\left({1-p}\right)^{k-1}}p,$ (1)

where $k$ is the sorted index of the reasoning type, $p$ is the hyper-
parameter set to 0.1 in our implementation. In this random setting, the type
with more training samples has a higher probability to be selected, which is
in parallel with the real situation.

For _difficulty_ , we first rank the difficulty of the reasoning types, based
on their performance with RoBERTa-Large single model. On one split, we select
some of the most difficult reasoning types as the seen ones. On the other
split, we select a part of the easiest types as the seen ones.

In total, 6 zero-shot splits are obtained (2 for each strategy) for ReClor.
The details of the splits are presented in Table I.

###  3.2 Generalized Zero-shot Setting

(a) The distribution of the occurrence frequency of different reasoning types
in the ReClor dataset.

(b) Pilot comparisons between full and zero-shot settings.

Figure 2: The visualization of the pilot experiments.

We propose the zero-shot setting for the logical reasoning task. Given the
context $C_{i}$, question $Q_{i}$ ($\emph{type}(Q_{i})\in\mathcal{T}$) and
option set $A_{i}$ of the $i^{th}$ question, the model is required to predict
the correct answer $a\in A_{i}$. In the definition, $\emph{type}(Q_{i})$
denotes the reasoning type of the question $Q_{i}$ and $\mathcal{T}$ is the
type set of the zero-shot splits. For each zero-shot split, a part of the
types are sampled as the seen types, while others are viewed as the unseen
ones. Only the questions of seen types exist during training and they consist
of the training examples. The type set seen during the training stage and the
set for the test stage are $\mathcal{T}^{train}$ and $\mathcal{T}^{test}$
respectively, thus they satisfy
$\mathcal{T}^{train}\cup\mathcal{T}^{test}=\mathcal{T}$.

To be closer to the real scenes, we consider the generalized zero-shot
setting. That is to say, the test scope is not limited to the unseen types. To
this end, we employ two metrics for the generalized zero-shot setting, _Test-
All_ and _Test-Unseen_. The former one denotes the exact match results on the
full-data test split, which contains both seen and unseen types, i.e.,
$\mathcal{T}^{train}\subseteq\mathcal{T}^{test}$. The latter one is the exact
match results only on the unseen types of the test split, which ignores the
performance on the seen types, i.e.,
$\mathcal{T}^{train}\cap\mathcal{T}^{test}=\emptyset$. In this way, we expect
to achieve comprehensive assessments of the model capabilities.

###  3.3 Pilot Experiments

Also, we are going to verify the necessity of the zero-shot setting from the
pilot experiments. For comparison with each split, we randomly sample the same
amount of training examples on both the seen and unseen types, forming the
comparison group. Take the zero-shot split v1 as an example, it has $2,190$
seen samples for training. Thus, its comparison training group also includes
$2,190$ examples, but are distributed over all types. The purpose of the pilot
experiments is to observe the performance changes of the two metrics on the
test split.

In the implementation, we utilize the RoBERTa single model [32] to conduct the
reasoning and maintain the same hyper-parameters of the zero-shot splits with
comparison groups. Especially, to avoid the noise brought by the random
sampling, we conduct the comparison experiments five times with different
random seeds. Final results are the average values of the five experiments.

We select the results on four of the split versions for illustration (i.e.,
v1-v4), shown in Fig. 2. The first and the third column in blue are the
performance of the comparison group, on the full-data test and the unseen
types of test respectively. The second and the fourth column represent the
performance on the zero-shot setting. With the same number of training
examples, training only on seen types (zero-shot setting) witnesses obvious
drops on the performance, especially on the unseen types of the test split.

Such observations are consistent among all the zero-shot splits. Zero-shot
pilot experiments based on reasoning types uncover the obvious drawback of the
current full-data setting. In another word, it verifies the necessity to
propose a new benchmark for the generalized ZsLR.

##  4 Methods

In this section, we introduce the proposed methods. To tackle the zero-shot
challenges, we propose a model named TaCo, which focuses on the reasoning type
perception in the logical reasoning task. The architecture of TaCo is shown in
Fig. 3. It mainly consists of three parts: (a) heuristic reconstruction to
acquire the type-aware input sequences; (b) text graph construction and
reasoning for the QA problems; (c) type-aware contrastive learning.

Figure 3: The architecture of TaCo. Taken the three basic parts (i.e.,
context, question and option) as the inputs, we reconstruct them with the
heuristic strategy (a). The text graph is built in module (b1) and the
reasoning based on the graph transformer is conducted in module (b2). The
updated representation of the global node is employed to perform the final
inference. Meanwhile in module (c), the contrastive learning is introduced to
improve the awareness of the reasoning type in the global node.

###  4.1 Heuristic Input Reconstruction

One of the common practices for MCQA problems is to concatenate three
sequences as inputs: context, question and option. But it is insufficient for
the zero-shot setting. There exist two main drawbacks: 1) the modeling of the
reasoning type is implicit in the sequence; 2) it is difficult to bridge the
interaction between context and options via the question sequence in the
middle since it is not natural for the language model (LM). To this end, this
paper introduces the heuristic reconstruction to the inputs based on the type
extraction.

To address the first drawback, we are required to label the reasoning type of
each question based on the limited inputs. Inspired by LReasoner [11], we
propose a simple but effective type extractor through keywords. The procedure
of the heuristic type extractor is presented in the pseudo-code of Algorithm
1.

Since the ReClor dataset only makes the reasoning types on the test split
public, we are required to collect the classification method from it. Before
the extraction, we conclude the keywords and phrases for each reasoning type,
forming the keywords base $\mathcal{B}$. Also, we include the maximum window
size W and the question sentence as the inputs. In Line 1, we first split the
question into words and form the word sequence $S$. Then we start the
iteration in the descending order of the window size. We slide the window over
the question sequence to obtain the sub-sequence (Line 5). Then we match the
sub-sequence $C$ with keywords base $\mathcal{B}_{k}$ for each reasoning type
$k$ to derive the number of exact matches (Line 7). After each window size
iteration, we judge the exit condition (Line 11). If there exists a unique
type with the maximum number of matches, we label the type as the ground
truth. Meanwhile, if we can not extract the type at the last round of
iteration, we label the instance as _Others_ type (Line 16, 17).

Input: keywords base $\mathcal{B}$, window size W, question sentence $q$

Output: reasoning type $T$

1 $S\leftarrow$ SplitByWords ($q$)

2 for  _$i=W,...,2,1$_ do

3 $\textrm{Count}\leftarrow\\{\\}$

4 for  _$j=0,...,length(S)-$ W_ do

5 $C\leftarrow S[j:\mathrm{W}]$

6 for  _$k=0,1,...,16$_ do

7 $\textrm{num}\leftarrow\texttt{NumOfMatch}(C,\mathcal{B}_{k})$

8 Count[k] += num

9 end for

10

11 end for

12 if  _max(Count) > 0 and is unique_ then

13 $T\leftarrow$ type of the max match count

14 return reasoning type $T$

15 end if

16 if  _i == 1_ then

/* classify to ‘Others’ */

17 $T\leftarrow`Others^{\prime}$

18 return reasoning type $T$

19

20 end if

21

22 end for

Algorithm 1 Reasoning Type Extraction

Then, we convert the type index of each question into the natural language
(e.g., _Implication_ , _Conclusion_). Meanwhile, to equip the LM with the type
information, we add a type-related prefix at the beginning of the sequence:

$\textrm{prefix}=\textrm{This is the task of [R-Type]}$ (2)

where [R-Type] denotes the natural language label of the specific type. Thus,
the type semantic is merged into the inputs in an explicit manner.

For the second drawback, one intuitive idea is to reconstruct the inputs of
question and option sequences, transforming them into a single sequence in a
declarative form, defined as _Q-A_ pair. In another word, we are required to
fill the option into the proper position in the question. According to our
observation, the question sentence is led by a trigger span. For example, in
the question _Which one of the following most weakens the arguments above ?_ ,
the span _Which one of the following_ serves as the trigger. Then, we can
replace the trigger span with the option sequence to obtain the _Q-A_ pair,
formulated as _[Option] most weakens the arguments above_.

Therefore, the core of the heuristic strategy is to pinpoint the precise
position of the triggers. Although the trigger span is similar to the basic
form of _Which of the following_ , it is not always the same. To further
improve the diversity of the trigger base, we propose two heuristic
strategies:

  * •

_Combination_ : We predefine a set of basic words (i.e., {which, one, of, the,
following}), and then randomly combine all or parts of them to form the new
triggers. For instances, _of which the following_ and _which of following_ are
two newly constructed triggers.

  * •

_Tolerance_ : More trigger spans are not limited to the combination of the
predefined words. To this end, we propose a tolerance strategy to contain the
extra words within the trigger span. For example, in the trigger _of the
following claims, which_ , we include the extra word _claims_ to the span.

In this way, the trigger base can be expanded to adapt to the complex
situations in reality. So far, the declarative _Q-A_ pair is obtained.

Concatenating the context sequence $C_{i}$ and _Q-A_ pair sequence $P_{i}$ of
the $i^{th}$ question, we send it into the pre-trained LM. In this paper, we
employ the RoBERTa-Large model [32] to obtain the token-level representations
as follows,

$\displaystyle{\left[\mathbf{h}_{<\mathrm{s}>};\mathbf{h}_{c_{0}};\ldots;\mathbf{h}_{</\mathrm{s}>};\mathbf{h}_{p_{0}};\ldots;\mathbf{h}_{</\mathrm{s}>}\right]}$
(3) $\displaystyle=$
$\displaystyle\operatorname{RoBERTa}\left(<\mathrm{s}>c_{0}\cdots
p_{0}\cdots</\mathrm{s}>\right),$

where the tokens $\\{c_{0},c_{1},\ldots,c_{|C_{i}|}\\}$ make up $C_{i}$, and
$\\{p_{0},p_{1},\ldots,p_{|P_{i}|}\\}$ make up $P_{i}$. The representation of
$C_{i}$, $P_{i}$ and the whole sequence $S_{i}$ (concatenation of prefix,
$C_{i}$ and $P_{i}$) can be obtained through the mean pooling strategy,

$\displaystyle\mathbf{h}_{C_{i}}=\frac{1}{|C_{i}|}$
$\displaystyle\sum_{k=1}^{|C_{i}|}\mathbf{h}_{c_{k}},\mathbf{h}_{P_{i}}=\frac{1}{|P_{i}|}\sum_{k=1}^{|P_{i}|}\mathbf{h}_{p_{k}},$
(4)
$\displaystyle\mathbf{h}_{S_{i}}=\frac{1}{|S_{i}|}\sum_{k=1}^{|S_{i}|}\mathbf{h}_{s_{k}}.$

###  4.2 Text Graph Construction and Reasoning

According to the previous analysis, the context and _Q-A_ pair may interact
differently under different reasoning types. Therefore, we consider to build
the topology structure of the two parts respectively and finally learn their
interactive semantic. For clearer illustration, we take the context part as an
example and present the graph construction process in Fig.4.

Figure 4: An example of the construction of context subgraph. The input
context can be split into 6 units (i.e., U1-U6). Based on the order relation
and overlap relation, the topology of the subgraph can be obtained.

Firstly, we split the text sequence into units (e.g., U1-U6 in Fig. 4) based
on the predefined explicit connectives or punctuation (marked in blue in Fig.
4). These text units function as the nodes during the graph reasoning process.
Therefore, two node sets for the context and _Q-A_ pair parts are obtained
respectively.

To form the dual subgraphs of the input sequence, we are also required to
define the edge relations, including a) order relation and b) overlap
relation. The former one models the original text order, where nodes are
connected in sequence. It forms a chain-type structure which maintains the
position perception. The latter one models the similarity between nodes, where
nodes with more vocabulary overlaps or similar semantics will be connected. In
the implementation, we will transform the text units into word sets, where the
stopwords are excluded. Through the calculation of the overlap ratio between
sets, we connect the node pairs that exceed the threshold.

In this way, the independent topology of the context part and the _Q-A_ pair
part are formed. To make the interaction of the two parts learnable, we add a
global node to link all the extracted nodes. The global node is initialized
with $\mathbf{h}_{S_{i}}$ and aggregates the two-way information flow.
Therefore, this node representation is expected to contain the type semantic
to improve the zero-shot performance.

Further, to overcome over-smoothing in graph neural networks [33] and fully
conduct node interactions, we employ the Graph Transformer Network [34, 9].
Let $\mathcal{N}$ be the node set in one graph, with the number of
$|\mathcal{N}|$. We feed the node sequence into the Transformer architecture
[35]. For each layer $l$, the updated information $\mathbf{e}_{j}$ of the node
$\mathcal{N}_{j}$ is formulated as follows:

$\mathbf{e}_{j}^{(l)}=\sigma(\mathbf{W}_{self}^{(l-1)}\mathbf{e}_{j}^{(l-1)}+\sum_{\mathcal{N}_{k}\in\mathcal{N}\backslash\\{\mathcal{N}_{j}\\}}\alpha_{j,k}\mathbf{e}_{k}^{(l-1)}).$
(5)

where $\mathcal{N}\backslash\\{\mathcal{N}_{j}\\}$ denotes the set without
$j^{th}$ node.

The core of the equation is the computation for the weighted attention
$\alpha_{j,k}$. It is obtained by the two matrices query $\mathbf{Q}$ and key
$\mathbf{K}$:

$\mathbf{A}=\mathtt{softmax}(\frac{\mathbf{QK}^{\textrm{T}}}{\sqrt{d}}+\mathbf{M}),$
(6)

where $\alpha_{j,k}$ is one of the elements in the matrix $\mathbf{A}$. $d$ is
the dimension of the hidden states.
$\mathbf{M}\in\mathbb{R}^{\mathcal{|N|}\times\mathcal{|N|}}$ is the attention
bias matrix to encode the structural semantic of the whole graph, which is the
adjacent matrix of the graph.

Thus, the final feature of the global node $\mathbf{h}_{g}$ is updated through
the graph reasoning network, containing the global semantic of the text
sequence. Before predicting the correct answer, we concatenate
$\mathbf{h}_{g}^{(j)}$, $\mathbf{h}_{C_{i}}^{(j)}$, $\mathbf{h}_{P_{i}}^{(j)}$
and $\mathbf{h}_{S_{i}}^{(j)}$ to obtain the score of the $j^{th}$ option
followed by softmax:

$\mathrm{z}_{i}^{(j)}=\mathtt{softmax}\left(\mathbf{W}\cdot\left[\mathbf{h}_{g}^{(j)};\mathbf{h}_{C_{i}}^{(j)};\mathbf{h}_{P_{i}}^{(j)};\mathbf{h}_{S_{i}}^{(j)}\right]\right),$
(7)

where $\mathbf{W}\in\mathbb{R}^{4d\times 1}$ denotes the linear projection.
Same with most of the MCQA methods, we adopt the cross-entropy loss function
for the optimization:

$\mathcal{L}_{MC}=\mathtt{CrossEntropy}\left(\mathrm{z}_{i},y_{i}\right),$ (8)

where $\mathrm{z}_{i}$ denotes the stack of each option score and $y_{i}$ is
the ground-truth label of the $i^{th}$ example.

TABLE II: The defined index and descriptions of 17 logical reasoning types in the dataset. Id | Type |  Descriptions
---|---|---
0 | Necessary Assumptions |  identify the claim that must be true or is required in order for the argument to work.
1 | Sufficient Assumptions |  identify a sufficient assumption, that is, an assumption that, if added to the argument, would make it logically valid.
2 | Strengthen |  identify information that would strengthen an argument.
3 | Weaken |  identify information that would weaken an argument.
4 | Evaluation |  identify information that would be useful to know to evaluate an argument.
5 | Implication |  identify something that follows logically from a set of premises.
6 | Conclusion and Main Point |  identify the conclusion/main point of a line of reasoning.
7 | Most Strongly Supported |  find the choice that is most strongly supported by a stimulus.
8 | Explain or Resolve |  identify information that would explain or resolve a situation.
9 | Principle |  identify the principle, or find a situation that conforms to a principle, or match the principles.
10 | Dispute |  identify or infer an issue in dispute.
11 | Technique |  identify the technique used in the reasoning of an argument.
12 | Role |  describe the individual role that a statement is playing in a larger argument.
13 | Identify a Flaw |  identify a flaw in an arguments reasoning.
14 | Match Flaws |  find a choice containing an argument that exhibits the same flaws as the passages argument.
15 | Match the Structure |  match the structure of an argument in a choice to the structure of the argument in the passage.
16 | Others |  other types of questions which are not included by the above.

###  4.3 Type-aware Contrastive Learning

In the upper part of the architecture, the interactive semantic of the context
and _Q-A_ pair is learned through one global node. Besides the implicit
awareness of the type, we also expect the model to distinguish the ground-
truth type from the negative ones for zero-shot logical reasoning.

Based on the heuristic extractor mentioned above, the reasoning type of each
example can be derived. For each reasoning type, the detailed description in
the form of natural language is utilized (listed in Table II).

Feeding all 17 reasoning type descriptions into the LM (i.e., Sentence-BERT
[15]), we obtain the sentence-level embeddings, which are fixed during
training.

For each example, we propose to attend to the type information through
contrastive learning. The final representation of the global node
$\mathbf{h}_{g}$ serves as the anchor. The ground-truth type functions as the
positive sample, while others are negative samples, with the embedding of
$\mathbf{h}_{gt}$ and $\mathbf{h}_{n}$ ($n$ is the index of the negative
samples) respectively. Our purpose is to close the distance between the global
node $\mathbf{h}_{g}$ and the positive sample $\mathbf{h}_{gt}$, while
distancing the negative samples.

For simplicity, we model the score of two vectors using Hadamard product. That
is:

$\displaystyle\mathrm{score}^{+}$
$\displaystyle=\mathbf{h}_{g}\odot\mathbf{h}_{gt},$ (9)
$\displaystyle\mathrm{score}^{-}_{n}$
$\displaystyle=\mathbf{h}_{g}\odot\mathbf{h}_{n},$

where $\mathrm{score}^{+}$ and $\mathrm{score}^{-}_{n}$ represent the score of
the positive sample and the scores of the negative ones respectively. We
employ the margin loss as the auxiliary optimization function for each
example:

$\mathcal{L}_{con}=\max\\{0,\gamma-\mathrm{score}^{+}+\underset{n}{\max}(\mathrm{score}^{-}_{n})\\},$
(10)

where $\gamma>0$ controls the difference between positive and negative scores.
We select the maximum score of the negative one for the loss computation.

To maximize the joint optimization performance of the two loss functions, we
set a trade-off coefficient $\alpha$ and transform the final loss function
into:

$\mathcal{L}=\mathcal{L}_{MC}+\alpha\mathcal{L}_{con}.$ (11)

In this way, the global semantic for reasoning reinforces mutually with the
type-aware semantic, which benefits the zero-shot performance. In addition, it
provides the interpretability for the test process, which will distinguish the
correct reasoning type from others.

##  5 Main Experiments

In this section, we will introduce the details of the experiment setup. Also,
extensive experiments on both the zero-shot and full-data setting will be
analyzed.

###  5.1 Current Full-data Benchmark

Currently, there exist two datasets in the logical reasoning task, ReClor [13]
and LogiQA [16]. Since the LogiQA datasets contain only 5 reasoning types, we
consider the ReClor dataset in the zero-shot experiments. And we take both of
them into account for the full-data setting to prove the generalization
capability of TaCo. The detailed information of the two datasets is presented
below.

ReClor is sourced from some standardized graduate admission examinations. It
consists of 6,138 examples in total, including 4,638 training samples, 500
validation samples, and 1,000 test samples. As is listed in Table II, ReClor
contains 17 reasoning types. Especially, its test split is divided into two
parts, which are _Test-E_ and _Test-H_. The former one is the easy splits,
which can be addressed without context. The latter one is the hard splits.

LogiQA is collected from National Civil Servants Examinations of China,
including 8,678 samples in total. Among them, 7,376 are training samples while
validation and test splits both contain 651 samples. Different from the
diverse reasoning types in ReClor, LogiQA contains only 5 reasoning types.

###  5.2 Baselines

We include all the previous SOTA baselines in the main paper for comparison,
as well as the results from some classical language models.

Random The results in this setting are based on the random predictions.

RoBERTa-Large [32] The results are obtained simply utilizing the RoBERTa
language model for the predictions. It is similar to the baseline of BERT-
Large [36] and XLNet-Large [37].

Human Performance [13] In ReClor, the average score of different graduate
students on the test split is utilized as the human performance.

DAGN [7] It is the first work to propose the construction of the text graph
based on the extracted EDUs. It mainly relies on the RoBERTa-Large [32] to
encode the tokens and graph neural network [17] to update the features.

FocalReasoner [8] It constructs a supergraph for reasoning, which consists of
the fact units extracted from the text.

LReasoner [11] It explores the context extension based on the defined logical
rules (e.g., De Morgan’s laws). Also, it employs the data augmentation method
to improve the performance. Since constructing more training data is proved to
be of great help to the zero-shot performance of most of the current models,
we do not consider the data augmentation strategy when reproducing.

MERIt [12] It proposes a meta-path-guided contrastive learning method to
reason over the text. The supervised pretraining is performed on the abundant
unlabeled text data. Due to the employment of the external data, we do not
consider this method in the zero-shot baselines for the fair comparison.

AdaLoGN [10] It proposes a neuro-symbolic system to adaptively update the text
graph. Additionally, a novel subgraph-to-node mechanism is utilized aggregate
the information.

Logiformer [9] It introduces two different strategies to construct the logical
graph and the syntax graph respectively. Through the two-branch graph
transformer network, the text features are updated to conduct the reasoning.

TABLE III: The tuned hyper-parameters with search scopes. ‘GT’ denotes the graph transformer network. Name of Param. | Search Scope | Best
---|---|---
# epoch | {10,15,20,25,30} | 30
# head in GT | {3,4,5,6} | 6
# layer in GT | {3,4,5,6} | 4
max sequence len | {128,256} | 256
learning rate | {4e-6. 5e-6, 6e-6} | 5e-6
margin $\gamma$ | {8,10,12,14} | 12
trade-off $\alpha$ | {0.1,0.2,0.5,1} | 0.2
TABLE IV: Experimental results on 6 zero-shot splits of ReClor dataset. The percentage signs (%) of accuracy values are omitted. The optimal and sub-optimal results are marked in bold and underline respectively (same for the following tables). _Test-A_ and _Test-U_ denote the abbreviations of the metrics _Test-All_ and _Test-Unseen_ respectively. Model | v1 | v2 | v3 | v4 | v5 | v6
---|---|---|---|---|---|---
Test-A | Test-U | Test-A | Test-U | Test-A | Test-U | Test-A | Test-U | Test-A | Test-U | Test-A | Test-U
BERT-Large | 38.00 | 34.36 | 42.00 | 33.39 | 37.50 | 31.61 | 38.00 | 33.26 | 29.60 | 28.02 | 28.80 | 32.24
RoBERTa-Large | 47.70 | 39.47 | 50.60 | 39.90 | 46.10 | 40.58 | 50.40 | 42.45 | 53.00 | 43.66 | 49.90 | 50.92
DAGN | 49.20 | 41.37 | 52.70 | 43.56 | 49.60 | 39.73 | 52.50 | 44.51 | 52.40 | 42.63 | 48.50 | 49.15
LReasoner | 46.90 | 40.60 | 50.20 | 43.49 | 48.40 | 42.76 | 49.20 | 44.12 | 51.90 | 42.02 | 46.30 | 44.93
Logiformer | 43.50 | 39.31 | 54.80 | 46.30 | 48.80 | 42.24 | 52.10 | 44.85 | 52.10 | 40.88 | 51.50 | 51.44
TaCo | 52.20 | 47.51 | 55.80 | 48.79 | 52.20 | 44.26 | 54.70 | 49.89 | 56.00 | 46.67 | 54.70 | 55.17

###  5.3 Implementation Details

In this paper, all experiments run on a single GPU of Tesla A100. We employ
RoBERTa-large model [32] as the text encoder for all previous methods for fair
comparison. For each split of the zero-shot setting, we train on the seen
types and select the best epoch on the seen types of the development set for
the test. We rerun other baselines on the zero-shot setting with the original
configuration and maintain their reported results on the full-data setting. As
for our proposed model, the hyper-parameters keep the same for both settings.
The training epoch is set to 30 and the batchsize is fixed to 1. For
optimization, we take Adam [38], with the peak learning rate as 5e-6. The
number of heads and layers in the graph transformer is set to 6 and 4
respectively. $\gamma$ in the margin loss is tuned as 12. The loss trade-off
$\alpha$ is 0.2. The selected five random seeds for the comparison groups are
{42, 12, 23, 234, 1234}. Additionally, some important hyper-parameters are
tuned for the best within a search scope. The details are included in the
Table III.

###  5.4 Main Results

The model TaCo is designed to bridge the gaps of the zero-shot logical
reasoning setting, thus we are going to evaluate its performance. We present
the results of 6 zero-shot splits in Table IV. We rerun five strong baselines
on the zero-shot setting, including (1) BERT-Large, (2) RoBERTa-Large, (3)
graph-based representative method DAGN [7], (4) data-based representative
model LReasoner [11], (5) SOTA model Logiformer [9].

It shows that TaCo outperforms these strong baselines with large margins on
all the zero-shot splits. Compared with the previous SOTA model Logiformer,
the averaged improvements among all the splits are 3.80% and 4.54%
respectively for the two metrics _Test-All_ and _Test-Unseen_. And compared
with all suboptimal results on the two metrics, TaCo still has great
superiority of 2.50% and 3.65% respectively. To explore the respective roles
of the two metrics, _Test-Unseen_ witnesses the greater improvements among all
the splits, especially 6.14% and 5.04% over the sub-optimal methods in split
v1 and v4 respectively. It illustrates that TaCo performs better on tackling
the unseen types of samples than the current methods, while maintaining the
superior performances on the seen ones. In addition, among all the zero-shot
splits, the performances within each method vary greatly. For example, the
previous SOTA Logiformer shows good performances in v2 but struggles a lot in
v1. It proves that the benchmark provides diverse distributions of the data
splits and encourages the consistent performances of the model. And
considering the experiment results, TaCo can function as a strong baseline for
the future work on the ZsLR benchmark.

TABLE V: Ablation Studies on zero-shot splits. The drops on the performances are marked in red. The darker ones represent the greater decline in values. Model | v1 | v2 | v3 | v4 | v5 | v6
---|---|---|---|---|---|---
Test-A | Test-U | Test-A | Test-U | Test-A | Test-U | Test-A | Test-U | Test-A | Test-U | Test-A | Test-U
TaCo | 52.20 | 47.51 | 55.80 | 48.79 | 52.20 | 44.26 | 54.70 | 49.89 | 56.00 | 46.67 | 54.70 | 55.17
a) Input Reconstruction |  |  |  |  |  |  |  |  |  |  |  |
w/o type prefix | 51.10 | 42.30 | 55.30 | 44.01 | 49.90 | 41.08 | 53.80 | 45.60 | 53.60 | 42.04 | 53.50 | 53.67
$\Delta$ | -1.10 | -5.21 | -0.50 | -4.78 | -2.30 | -3.18 | -0.90 | -4.39 | -2.40 | -4.63 | -1.20 | -1.50
w/o input reconstruction | 50.20 | 42.17 | 55.30 | 45.39 | 51.20 | 42.70 | 53.60 | 46.62 | 54.40 | 43.74 | 52.30 | 52.14
$\Delta$ | -2.00 | -5.34 | -0.30 | -3.40 | -1.00 | -1.56 | -1.10 | -3.27 | -1.60 | -2.93 | -2.40 | -3.03
b) Graph Construction & Reasoning |  |  |  |  |  |  |  |  |  |  |  |
w/o graph reasoning | 48.70 | 42.28 | 54.10 | 42.19 | 50.60 | 41.99 | 52.70 | 45.55 | 53.10 | 44.25 | 50.90 | 50.63
$\Delta$ | -3.50 | -5.23 | -1.70 | -6.60 | -1.60 | -2.27 | -2.00 | -4.34 | -2.90 | -2.42 | -3.80 | -4.54
w/o global node | 51.50 | 46.96 | 53.70 | 44.78 | 51.40 | 43.51 | 53.50 | 46.16 | 55.10 | 46.57 | 53.50 | 52.53
$\Delta$ | -0.70 | -0.55 | -2.10 | -4.01 | -0.80 | -0.75 | -1.20 | -3.73 | -0.90 | -0.10 | -1.20 | -2.64
c) Type-aware Contrastive Learning |  |  |  |  |  |  |  |  |  |  |  |
w/o type contrast | 51.20 | 42.70 | 55.00 | 44.62 | 50.90 | 43.96 | 54.40 | 46.19 | 55.70 | 45.89 | 53.80 | 53.94
$\Delta$ | -1.00 | -4.81 | -0.80 | -4.17 | -1.30 | -0.30 | -0.30 | -3.70 | -0.30 | -0.78 | -0.90 | -1.23

###  5.5 Ablation Studies

To illustrate the effectiveness of each module of TaCo in the zero-shot
setting, we conduct the following ablation studies. The results are listed in
Fig. V

Detailedly, in part a), _w/o type prefix_ and _w/o reconstruction_ are related
to the heuristic input reconstruction, where the former one ablates the type-
oriented prefix, and the latter one maintains the simple concatenation of the
question and option. Generally, the heuristic type prefix and input
reconstruction bring more improvements on the metric of _Test-Unseen_ with an
average of 3.95% and 3.26% respectively respectively on all six splits. It
shows that type-oriented prefix is more effective than input reconstruction in
most cases (i.e., v2-v5).

In part b), _w/o graph reason_ ablates the whole module of _graph construction
and reasoning_ , and replaces the global node feature with the pooled output
of the whole sequence. Since the reasoning process is the core of this task,
it contributes most to the performance on both the metrics, with the average
gains of 2.58% and 4.23%. While _w/o global node_ simply replaces the global
node feature with the pooled representation of all the other nodes. It can be
regarded as a small part of _graph construction and reasoning_. From the
results, it works on the splits v2, v4 and v6. As the training set of these
three splits are obviously larger, it can be concluded that _global node_ is
especially effective with more training samples.

In part c), we ablate the type-aware contrastive learning module. From the
experiments, the modeling of reasoning types does great help to the unseen
types of samples. In the unseen types of v1, v2 and v4, the margin loss brings
significant improvements, with gains of 4.81%, 4.17% and 3.70% respectively.

Overall, the three key modules (i.e., type prefix, input reconstruction, type
contrast) proposed to enhance the type perception all have positive effects on
the model performances, especially on the unseen parts of the test split.

###  5.6 Parameter Analysis

Further, we present the visualization of different selections for hyper-
parameters. All the hyper-parameters are selected in the full-data setting,
since the zero-shot splits are diverse and hard to unify. The first one is the
number of layers and the number of heads in the graph transformer network,
shown in Fig. 5. To make a comprehensive comparison, we search both the head
number and the layer number within {3,4,5,6}. As can be seen from the heatmap,
the darker color represents the higher performance. TaCo reaches the optimal
performance when the head number is 6 and the layer number is 4.

(a) The performance on the validation split.

(b) The performance on the test split.

Figure 5: The performances on the different margin $\gamma$.

In addition, we present the model performances under different margins
$\gamma$. The search scope of $\gamma$ is within {8,10,12,14,16}. And we
select the optimal performance on the test split. The optimal value of
$\gamma$ is 12.

(a) The performance on the validation split.

(b) The performance on the test split.

Figure 6: The performances on the different margin $\gamma$.

##  6 Analysis of Model Generalization

In this section, we will discuss the generalization capability of the proposed
model. Since TaCo has achieved superior performances on the zero-shot setting,
we further extend it to the full-data setting. Experiments are conducted on
two mainstream logical reasoning dataset, ReClor and LogiQA. In Fig. VI, we
present the comparison results.

###  6.1 Generalization on Full-data Setting of ReClor

TABLE VI: Experimental results on ReClor with the full-data setting. _Test-E_ and _Test-H_ denote the easy part and the hard part of the ReClor test split respectively. Model | ReClor
---|---
Dev | Test | Test-E | Test-H
Random | 25.00 | 25.00 | 25.00 | 25.00
Human Performance [13] | - | 63.00 | 57.10 | 67.20
BERT-Large [13] | 53.80 | 49.80 | 72.00 | 32.30
XLNet-Large [13] | 62.00 | 56.00 | 75.70 | 40.50
RoBERTa-Large [13] | 62.60 | 55.60 | 75.50 | 40.00
DAGN [7] | 65.80 | 58.30 | 75.91 | 44.46
FocalReasoner [8] | 66.80 | 58.90 | 77.05 | 44.64
LReasoner [11] | 66.20 | 62.40 | 81.40 | 47.50
MERIt [12] | 66.80 | 59.60 | 78.10 | 45.20
AdaLoGN [10] | 65.20 | 60.20 | 79.32 | 45.18
Logiformer [9] | 68.40 | 63.50 | 79.09 | 51.25
TaCo (Ours) | 69.00 | 63.30 | 81.36 | 49.11

Compared with all the current methods, TaCo achieves competitive results. In
comparison with the SOTA model Logiformer, TaCo is better on the metric of
_Dev_ and _Test-E_ with 0.60% and 2.27% gains. And it is only slightly lower
than SOTA with 0.20% in the test split. We argue that in the full-data
setting, the ideal distribution of types leads to the weakening of the type
modeling. Combined with previous experiments on the zero-shot setting, though
Logiformer does good at over-fitting the full-data setting, it loses some
generalization capability. From this perspective, TaCo also shows great
superiority on the model generalization, performing consistently for both
settings.

TABLE VII: Ablation Studies on the full-data setting. The drops on the performances are marked in red. The darker ones represent the greater decline in values. Model | ReClor
---|---
Valid | $\Delta$ | Test | $\Delta$ | Test-E | Test-H
TaCo (Ours) | 69.00 | - | 63.30 | - | 81.36 | 49.11
a) Heuristic Input Reconstruction |  |  |  |  |  |
w/o type prefix | 67.60 | -1.40 | 61.30 | -2.00 | 80.45 | 46.25
w/o input reconstruction | 67.80 | -1.20 | 61.10 | -2.20 | 79.09 | 51.25
b) Graph Construction & Reasoning |  |  |  |  |  |
w/o graph reasoning | 64.40 | -4.60 | 59.30 | -4.00 | 79.55 | 43.39
w/o global node | 65.60 | -3.40 | 61.90 | -1.40 | 81.82 | 46.25
c) Type-aware Contrastive Learning |  |  |  |  |  |
w/o type contrast | 67.80 | -1.20 | 62.90 | -0.40 | 79.55 | 49.82

Further we conduct the ablation studies to analyze the effectiveness of the
proposed modules in the full-data setting. From the results, the graph
reasoning contributes most to the performance on the test split. Also, we only
witness 0.40% improvements for the consideration of type-aware margin loss.
Since in the full-data setting, the train and test splits share the same
distributions, the effects of the reasoning types are reduced. Therefore, the
slight gain of the contrastive learning is reasonable. In all, the
generalization capability of TaCo is well verified.

###  6.2 Generalization on Other Dataset

To discuss the generalization capability of TaCo, we further conduct some
experiments on another logical reasoning dataset LogiQA[16]. Following the
method utilized for ReClor, we also label each example of LogiQA with one of
the 17 reasoning types, and consider the zero-shot setting. We apply the above
mentioned strategies to obtain three zero-shot splits (i.e., z1, z2 and z3)
for LogiQA.

In comparison, we take some previous logical reasoning models into account,
including two typical language model BERT-Large and RoBERTa-Large, the graph-
based method DAGN and previous SOTA model Logiformer. We rerun the zero-shot
setting for each baseline.

TABLE VIII: Experiments results on the zero-shot splits of the LogiQA dataset. Model | z1 | z2 | z3
---|---|---|---
Test-A | Test-U | Test-A | Test-U | Test-A | Test-U
BERT-L | 26.26 | 28.64 | 27.80 | 27.06 | 28.26 | 27.14
RoBERTa-L | 27.65 | 33.98 | 29.49 | 31.76 | 28.73 | 29.43
DAGN | 34.41 | 40.78 | 35.48 | 34.12 | 33.79 | 32.00
Logiformer | 35.33 | 36.41 | 33.18 | 35.29 | 34.56 | 33.43
TaCo (Ours) | 35.58 | 37.38 | 36.76 | 35.29 | 36.41 | 35.14

From the results shown in TABLE VIII, TaCo shows its superiority in most
cases, only except the _Test-Unseen_ metric of z1 split. Specifically,
compared with previous SOTA Logiformer, TaCo achieves an average gain of 1.89%
and 0.89% on the _Test-All_ and _Test-Unseen_ respectively. In general,
experiments on the LogiQA dataset prove the good generalization capability of
TaCo, which satisfies our expectations.

##  7 Case Study

In this section, we discuss the interpretability of TaCo for the perception of
reasoning types. Fig. 7 shows a successful case and a failure case. For each
case study, we present the visualization of the type perception via dimension
reduction and make the comparison with the SOTA model Logiformer. For the
successful case, TaCo can well distance the ground truth type _Implication_
with others and thus make the correct prediction. In this case, Logiformer
which lacks the type perception fails. For the failure one, TaCo obviously
struggles at distinguishing _Necessary assumption_ from other reasoning types.
It leads to the wrong prediction, which is in same with Logiformer. It
encourages us to explore the deeper modeling of types from the graph
construction. In all, the type-aware contrastive learning is helpful to the
answer prediction and provides the interpretability of the model.

(a) A successful case.

(b) A failure case.

Figure 7: Case Study.

##  8 Conclusion and Future Work

To study the zero-shot capability of the logical reasoning models, we propose
the first benchmark for the generalized zero-shot logical reasoning, named
ZsLR. It includes six splits sampled with three strategies and two metrics to
comprehensively evaluate the performances. Also, we propose a model TaCo to
enhance the reasoning type perception through the heuristic input
reconstruction and the type-aware contrastive learning. Also, we conduct
extensive experiments on the zero-shot splits, full-data setting as well as
other dataset. Superior results illustrate the effectiveness and
generalization capability of the proposed modules.

In the future, we encourage more works to mind reasoning manners in the
logical reasoning task. And we are also interested in exploring the data-
augmentation methods through logics.

## References

  * [1] S. Liu, X. Zhang, S. Zhang, H. Wang, and W. Zhang, “Neural machine reading comprehension: Methods and trends,” _Applied Sciences_ , 2019.
  * [2] J. Hirschberg and C. D. Manning, “Advances in natural language processing,” _Science_ , 2015.
  * [3] K. Chowdhary, “Natural language processing,” _Fundamentals of artificial intelligence_ , 2020.
  * [4] M. Richardson, C. J. Burges, and E. Renshaw, “MCTest: A challenge dataset for the open-domain machine comprehension of text,” in _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2013.
  * [5] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, “RACE: Large-scale ReAding comprehension dataset from examinations,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2017\.
  * [6] A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A question answering challenge targeting commonsense knowledge,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL)_ , 2019.
  * [7] Y. Huang, M. Fang, Y. Cao, L. Wang, and X. Liang, “Dagn: Discourse-aware graph network for logical reasoning,” in _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2021.
  * [8] S. Ouyang, Z. Zhang, and H. Zhao, “Fact-driven logical reasoning,” _arXiv preprint arXiv:2105.10334_ , 2021.
  * [9] F. Xu, Q. Lin, J. Liu, Y. Pan, and L. Zhang, “Logiformer: A two-branch graph transformer network for interpretable logical reasoning,” in _International Conference on Research and Development in Information Retrieval (SIGIR)_ , 2022.
  * [10] X. Li, G. Cheng, Z. Chen, Y. Sun, and Y. Qu, “Adalogn: Adaptive logic graph network for reasoning-based machine reading comprehension,” in _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2022.
  * [11] S. Wang, W. Zhong, D. Tang, Z. Wei, Z. Fan, D. Jiang, M. Zhou, and N. Duan, “Logic-driven context extension and data augmentation for logical reasoning of text,” in _Findings of the Association for Computational Linguistics (ACL Findings)_ , 2022.
  * [12] F. Jiao, Y. Guo, X. Song, and L. Nie, “Merit: Meta-path guided contrastive learning for logical reasoning,” in _Findings of the Association for Computational Linguistics (ACL Findings)_ , 2022.
  * [13] W. Yu, Z. Jiang, Y. Dong, and J. Feng, “Reclor: A reading comprehension dataset requiring logical reasoning,” in _International Conference on Learning Representations (ICLR)_ , 2019.
  * [14] F. Pourpanah, M. Abdar, Y. Luo, X. Zhou, R. Wang, C. P. Lim, X.-Z. Wang, and Q. J. Wu, “A review of generalized zero-shot learning methods,” _IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)_ , 2022.
  * [15] N. Reimers, I. Gurevych, N. Reimers, I. Gurevych, N. Thakur, N. Reimers, J. Daxenberger, I. Gurevych, N. Reimers, I. Gurevych _et al._ , “Sentence-bert: Sentence embeddings using siamese bert-networks,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2019.
  * [16] J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang, “Logiqa: a challenge dataset for machine reading comprehension with logical reasoning,” in _Proceedings of the International Conference on International Joint Conferences on Artificial Intelligence (IJCAI)_ , 2021.
  * [17] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, “The graph neural network model,” _IEEE Transactions on Neural Networks_ , 2008\.
  * [18] M. Welling and T. N. Kipf, “Semi-supervised classification with graph convolutional networks,” in _International Conference on Learning Representations (ICLR)_ , 2016.
  * [19] N. Nakashole and T. Mitchell, “Language-aware truth assessment of fact candidates,” in _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2014.
  * [20] C. Zeng, S. Li, Q. Li, J. Hu, and J. Hu, “A survey on machine reading comprehension—tasks, evaluation metrics and benchmark datasets,” _Applied Sciences_ , 2020.
  * [21] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2016.
  * [22] P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2018.
  * [23] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-hop question answering,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2018.
  * [24] G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy, “Race: Large-scale reading comprehension dataset from examinations,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2017\.
  * [25] A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman, “Newsqa: A machine comprehension dataset,” in _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , 2017.
  * [26] A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and H. Hajishirzi, “Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * [27] Q. Jin, B. Dhingra, Z. Liu, W. Cohen, and X. Lu, “Pubmedqa: A dataset for biomedical research question answering,” in _Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019.
  * [28] K. Ma, F. Ilievski, J. Francis, Y. Bisk, E. Nyberg, and A. Oltramari, “Knowledge-driven data construction for zero-shot evaluation in commonsense question answering,” in _Proceedings of the Conference on Artificial Intelligence (AAAI)_ , 2021.
  * [29] A. Bosselut, R. Le Bras, and Y. Choi, “Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering,” in _Proceedings of the Conference on Artificial Intelligence (AAAI)_ , 2021.
  * [30] O. Ram, Y. Kirstain, J. Berant, A. Globerson, and O. Levy, “Few-shot question answering by pretraining span selection,” in _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2021.
  * [31] D. Guo and D. Tao, “Learning compositional representation for few-shot visual question answering,” _arXiv preprint arXiv:2102.10575_ , 2021.
  * [32] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” _arXiv preprint arXiv:1907.11692_ , 2019.
  * [33] Q. Li, Z. Han, and X.-M. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” in _Proceedings of the Conference on Artificial Intelligence (AAAI)_ , 2018.
  * [34] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, and T.-Y. Liu, “Do transformers really perform badly for graph representation?” _Advances in Neural Information Processing Systems (NIPS)_ , vol. 34, 2021.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in Neural Information Processing Systems (NIPS)_ , 2017.
  * [36] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [37] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” _Advances in Neural Information Processing Systems (NIPS)_ , vol. 32, 2019\.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.