The overall result is not very useful for ML practioners in this field, because it merely confirms what has been known or suspected, i.e. it depends on the task at hand, the labeled data set size, the type of the model, etc. So, the result in this paper is not very actionable. The reviewer noted that this comprehensive analysis deepens the understanding of this topic.
- General Discussion: The paper's presentation can be improved. Specifically: 1) The order of the figures/tables in the paper should match the order they are mentioned in the papers. Right now their order seems quite random.
2) Several typos (L250, 579, etc). Please use a spell checker.
3) Equation 1 is not very useful, and its exposition looks strange. It can be removed, and leave just the text explanations.
4) L164 mentions the "Appendix", but it is not available in the paper.
5) Missing citation for the public skip-gram data set in L425.
6) The claim in L591-593 is too strong. It must be explained more clearly, i.e. when it is useful and when it is not.
7) The observation in L642-645 is very interesting and important. It will be good to follow up on this and provide concrete evidence or example from some embedding. Some visualization may help too.
8) In L672 should provide examples of such "specialized word embeddings" and how they are different than the general purpose embedding.
9) Figuer 3 is too small to read.
5) Missing citation for the public skip-gram data set in L425.