lack statistics of the datsets (e.g. average length, vocabulary size) the baseline (Moses) is not proper because of the small size of the dataset the assumption "sarcastic tweets often differ from their non sarcastic interpretations in as little as one sentiment word" is not supported by the data. - General Discussion: This discussion gives more details about the weaknesses of the paper. Half of the paper is about the new dataset for sarcasm interpretation.
However, the paper doesn't show important information about the dataset such as average length, vocabulary size. More importantly, the paper doesn't show any statistical evidence to support their method of focusing on sentimental words. Because the dataset is small (only 3000 tweets), I guess that many words are rare. Therefore, Moses alone is not a proper baseline. A proper baseline should be a MT system that can handle rare words very well. In fact, using clustering and declustering (as in Sarcasm SIGN) is a way to handle rare words.
Sarcasm SIGN is built based on the assumption that "sarcastic tweets often differ from their non sarcastic interpretations in as little as one sentiment word". Table 1 however strongly disagrees with this assumption: the human interpretations are often different from the tweets at not only sentimental words. I thus strongly suggest the authors to give statistical evidence from the dataset that supports their assumption. Otherwise, the whole idea of Sarcasm SIGN is just a hack.
-------------------------------------------------------------- I have read the authors' response. I don't change my decision because of the following reasons: - the authors wrote that "the Fiverr workers might not take this strategy": to me it is not the spirit of corpus-based NLP. A model must be built to fit given data, not that the data must follow some assumption that the model is built on.
- the authors wrote that "the BLEU scores of Moses and SIGN are above 60, which is generally considered decent in the MT literature": to me the number 60 doesn't show anything at all because the sentences in the dataset are very short. And that, if we look at table 6, %changed of Moses is only 42%, meaning that even more than half of the time translation is simply copying, the BLUE score is more than 60.
- "While higher scores might be achieved with MT systems that explicitly address rare words, these systems don't focus on sentiment words": it's true, but I was wondering whether sentiment words are rare in the corpus. If they are, those MT systems should obviously handle them (in addition to other rare words).
- "While higher scores might be achieved with MT systems that explicitly address rare words, these systems don't focus on sentiment words": it's true, but I was wondering whether sentiment words are rare in the corpus. If they are, those MT systems should obviously handle them (in addition to other rare words).