. First, the classification model used in this paper (concat + linear classifier) was shown to be inherently unable to learn relations in "Do Supervised Distributional Methods Really Learn Lexical Inference Relations?" ( Levy et al., 2015). Second, the paper makes superiority claims in the text that are simply not substantiated in the quantitative results. In addition, there are several clarity and experiment setup issues that give an overall feeling that the paper is still half-baked.
= Classification Model = Concatenating two word vectors as input for a linear classifier was mathematically proven to be incapable of learning a relation between words (Levy et al., 2015). What is the motivation behind using this model in the contextual setting?
While this handicap might be somewhat mitigated by adding similarity features, all these features are symmetric (including the Euclidean distance, since |L-R| = |R-L|). Why do we expect these features to detect entailment?
I am not convinced that this is a reasonable classification model for the task.
= Superiority Claims = The authors claim that their contextual representation is superior to context2vec. This is not evident from the paper, because: 1) The best result (F1) in both table 3 and table 4 (excluding PPDB features) is the 7th row. To my understanding, this variant does not use the proposed contextual representation; in fact, it uses the context2vec representation for the word type.
2) This experiment uses ready-made embeddings (GloVe) and parameters (context2vec) that were tuned on completely different datasets with very different sizes. Comparing the two is empirically flawed, and probably biased towards the method using GloVe (which was a trained on a much larger corpus).
In addition, it seems that the biggest boost in performance comes from adding similarity features and not from the proposed context representation. This is not discussed.
= Miscellaneous Comments = - I liked the WordNet dataset - using the example sentences is a nice trick.
- I don’t quite understand why the task of cross-lingual lexical entailment is interesting or even reasonable.
- Some basic baselines are really missing. Instead of the "random" baseline, how well does the "all true" baseline perform? What about the context-agnostic symmetric cosine similarity of the two target words?
- In general, the tables are very difficult to read. The caption should make the tables self-explanatory. Also, it is unclear what each variant means; perhaps a more precise description (in text) of each variant could help the reader understand?
- What are the PPDB-specific features? This is really unclear.
- I could not understand 8.1.
- Table 4 is overfull.
- In table 4, the F1 of "random" should be 0.25.
- Typo in line 462: should be "Table 3" = Author Response = Thank you for addressing my comments. Unfortunately, there are still some standing issues that prevent me from accepting this paper: - The problem I see with the base model is not that it is learning prototypical hypernyms, but that it's mathematically not able to learn a relation.
- It appears that we have a different reading of tables 3 and 4. Maybe this is a clarity issue, but it prevents me from understanding how the claim that contextual representations substantially improve performance is supported.
Furthermore, it seems like other factors (e.g. similarity features) have a greater effect.
1) The best result (F1) in both table 3 and table 4 (excluding PPDB features) is the 7th row. To my understanding, this variant does not use the proposed contextual representation; in fact, it uses the context2vec representation for the word type.