] See below for details of the following weaknesses: - Novelties of the paper are relatively unclear.
- No detailed error analysis is provided.
- A feature comparison with prior work is shallow, missing two relevant papers.
- The paper has several obscure descriptions, including typos.
[General Discussion:] The paper would be more impactful if it states novelties more explicitly. Is the paper presenting the first neural network based approach for event factuality identification? If this is the case, please state that.
The paper would crystallize remaining challenges in event factuality identification and facilitate future research better if it provides detailed error analysis regarding the results of Table 3 and 4. What are dominant sources of errors made by the best system BiLSTM+CNN(Att)? What impacts do errors in basic factor extraction (Table 3) have on the overall performance of factuality identification (Table 4)? The analysis presented in Section 5.4 is more like a feature ablation study to show how useful some additional features are.
The paper would be stronger if it compares with prior work in terms of features. Does the paper use any new features which have not been explored before? In other words, it is unclear whether main advantages of the proposed system come purely from deep learning, or from a combination of neural networks and some new unexplored features. As for feature comparison, the paper is missing two relevant papers: - Kenton Lee, Yoav Artzi, Yejin Choi and Luke Zettlemoyer. 2015 Event Detection and Factuality Assessment with Non-Expert Supervision. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1643-1648.
- Sandeep Soni, Tanushree Mitra, Eric Gilbert and Jacob Eisenstein. 2014.
Modeling Factuality Judgments in Social Media Text. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 415-420.
The paper would be more understandable if more examples are given to illustrate the underspecified modality (U) and the underspecified polarity (u). There are two reasons for that. First, the definition of 'underspecified' is relatively unintuitive as compared to other classes such as 'probable' or 'positive'.
Second, the examples would be more helpful to understand the difficulties of Uu detection reported in line 690-697. Among the seven examples (S1-S7), only S7 corresponds to Uu, and its explanation is quite limited to illustrate the difficulties.
A minor comment is that the paper has several obscure descriptions, including typos, as shown below: - The explanations for features in Section 3.2 are somewhat intertwined and thus confusing. The section would be more coherently organized with more separate paragraphs dedicated to each of lexical features and sentence-level features, by: - (1) stating that the SIP feature comprises two features (i.e., lexical-level and sentence-level) and introduce their corresponding variables (l and c) *at the beginning*; - (2) moving the description of embeddings of the lexical feature in line 280-283 to the first paragraph; and - (3) presenting the last paragraph about relevant source identification in a separate subsection because it is not about SIP detection.
- The title of Section 3 ('Baseline') is misleading. A more understandable title would be 'Basic Factor Extraction' or 'Basic Feature Extraction', because the section is about how to extract basic factors (features), not about a baseline end-to-end system for event factuality identification.
- The presented neural network architectures would be more convincing if it describes how beneficial the attention mechanism is to the task.
- Table 2 seems to show factuality statistics only for all sources. The table would be more informative along with Table 4 if it also shows factuality statistics for 'Author' and 'Embed'.
- Table 4 would be more effective if the highest system performance with respect to each combination of the source and the factuality value is shown in boldface.
- Section 4.1 says, "Aux_Words can describe the *syntactic* structures of sentences," whereas section 5.4 says, "they (auxiliary words) can reflect the *pragmatic* structures of sentences." These two claims do not consort with each other well, and neither of them seems adequate to summarize how useful the dependency relations 'aux' and 'mark' are for the task.
- S7 seems to be another example to support the effectiveness of auxiliary words, but the explanation for S7 is thin, as compared to the one for S6. What is the auxiliary word for 'ensure' in S7?
- Line 162: 'event go in S1' should be 'event go in S2'.
- Line 315: 'in details' should be 'in detail'.
- Line 719: 'in Section 4' should be 'in Section 4.1' to make it more specific.
- Line 771: 'recent researches' should be 'recent research' or 'recent studies'. 'Research' is an uncountable noun.
- Line 903: 'Factbank' should be 'FactBank'.
- The explanations for features in Section 3.2 are somewhat intertwined and thus confusing. The section would be more coherently organized with more separate paragraphs dedicated to each of lexical features and sentence-level features, by: