- it is not clear what's the goal of the paper. Is the release of a challenging dataset or proposing an analysis of augmenting models with expert guided adversarial examples. If it is the first, ok, but the paper misses a lot of important information, and data analysis to give a sense of the quality and usefulness of such a dataset. If it is the second, it is not clear what's the novelty.
- In general, it seems the authors want to propose a way of create a challenging set. However, what they describe seems very specific and not scalable.
- The paper structure and writing is not sufficient
My main concern is that it's not clear what's the goal of the paper. Also, the structure and writing should greatly improve. I believe also that the choice to go for a short paper was penalizing the authors, as it seems clear that they cut out some information that could've been useful to better understand the paper (also given the 5 pages appendix).
Detailed comments/questions: - Line 107 data, -> data.
- Line 161-162: this sentence is not clear.
- Table 1: are these all the rules you defined? How the rule is applied? When you decide to make small changes to the context? For example, when you decide to add "and her team" as in the last example of Table 1? - Also, it seems that all the rules change a one-token entity to a multi-token one or vice-versa. Will models be biased by this?
- Line 183-197: not clear what you're doing here. Details cannot be in appendix.
- What is not clear also to me is how is used the Challenge Set. If I understood correctly, the CS is created by the linguistic experts and it's used for evaluation purposes. Is this used also to augment the training material? If yes, what is the data split you used? - Line 246-249: this sentence lacks the conclusion - Line 249: What are eligible and not eligible examples?
- Line 251: what is p?
- Line 253: The formula doesn't depend on p, so why the premise is "if p=100% of the eligible example"?
- Line 252: Not clear what is the subject of this sentence.
- What is not clear also to me is how is used the Challenge Set. If I understood correctly, the CS is created by the linguistic experts and it's used for evaluation purposes. Is this used also to augment the training material? If yes, what is the data split you used?