As a reader of a ACL paper, I usually ask myself what important insight can I take away from the paper, and from a big picture point of view, what does the paper add to the fields of natural language processing and computational linguistics. How does the task of lexical substitutability in general and this paper in particular help either in improving an NLP system or provide insight about language? I can't find a good answer answer to either question after reading this paper.
As a practitioner who wants to improve natural language understanding system, I am more focused on the first question -- does the lexical substitutability task and the improved results compared to prior work presented here help any end application? Given the current state of high performing systems, any discrete clustering of words (or longer utterances) often break down when compared to continuous representations words (see all the papers that utilitize discrete lexical semantics to achieve a task versus words' distributed representations used as an input to the same task; e.g. machine translation, question answering, sentiment analysis, text classification and so forth). How do the authors motivate work on lexical substitutability given that discrete lexical semantic representations often don't work well? The introduction cites a few papers from several years back that are mostly set up in small data scenarios, and given that this word is based on English, I don't see why one would use this method for any task. I would be eager to see the authors' responses to this general question of mine.
As a minor point, to further motivate this, consider the substitutes presented in Table 1.
1. Tasha snatched it from him to rip away the paper.
2. Tasha snatched it from him to rip away the sheet.
To me, these two sentences have varying meanings -- what if he was holding on to a paper bag? In that scenario, can the word "paper" be substituted by "sheet"? At least, in my understanding, it cannot. Hence, there is so much subjectivity in this task that lexical substitutes can completely alter the semantics of the original sentence.
Minor point(s): - Citations in Section 3.1.4 are missing.
Addition: I have read the author response and I am sticking to my earlier evaluation of the paper.
- Citations in Section 3.1.4 are missing. Addition: I have read the author response and I am sticking to my earlier evaluation of the paper.