The major weakness of this paper is, in my eyes, the superficial modeling of logical relations. Deductive, Inductive, and Abductive reasoning types are only manifestations of logical reasoning, they do not touch down to the essence. Based on this classification, this paper only models the consequential relations in a sentence-level style, which ignores a broad range of other logical reasoning expressions such as negation, disjunction, and conjunction.
The so-called automatic identification of logical reasoning phenomena is in fact requires many manual labors. The authors have to pre-define words/expressions that indicate conclusion or premises, which greatly limits the universality of the proposed approach.
The authors only report the experimental results on the development set, which made the results less convincing. Also, the performance improvement over generation datasets is less noticeable than in classification datasets. It's hard to know that the model is really behaving as advertised rather than just adding capacity on top of the baselines.
In the related work section, especially Pre-training for Reasoning Ability Improvement, the authors describe several papers of other relation types that LogiGAN is not compared with. This makes it difficult to place this work in comparison with prior literature and understand the novelty of the proposed framework. Also, the key contribution of this paper could be blurred with improvement over GAN-like architecture rather than pre-training tasks or something else.
I do not see any open-source statements or code submissions in the supplementary materials, so I take issues with the reproducibility of this paper.
Minor Issues:
Line 19: “Learning without thought is labor lost” -> “learning without thinking is labor lost”
Caption of table 1: “development setsjudgement” -> “development set judgment”
Overall, I'm borderline with this paper. I think the paper tests three aspects: 1) GAN-style training in language models. 2) predict the masked-out logical statements as the pre-training task. 3) detection heuristics for logical reasoning phenomena. It certainly demonstrates great experimental results compared with vanilla T5 baselines. But none of them alone is novel enough. The most important part in my eyes, leveraging logical reasoning phenomena for pre-training, is after all a sententious logical relation extraction, whose effectiveness is already demonstrated in [1].
[1] Huang, Yinya, et al. "DAGN: Discourse-aware graph network for logical reasoning." NAACL 2021.
2) predict the masked-out logical statements as the pre-training task.