- The evaluation reported in this paper includes only intrinsic tasks, mainly on similarity/relatedness datasets. As the authors note, such evaluations are known to have very limited power in predicting the utility of embeddings in extrinsic tasks. Accordingly, it has become recently much more common to include at least one or two extrinsic tasks as part of the evaluation of embedding models.
- The similarity/relatedness evaluation datasets used in the paper are presented as datasets recording human judgements of similarity between concepts. However, if I understand correctly, the actual judgements were made based on presenting phrases to the human annotators, and therefore they should be considered as phrase similarity datasets, and analyzed as such.
- The medical concept evaluation dataset, ‘mini MayoSRS’ is extremely small (29 pairs), and its larger superset ‘MayoSRS’ is only a little larger (101 pairs) and was reported to have a relatively low human annotator agreement. The other medical concept evaluation dataset, ‘UMNSRS’, is more reasonable in size, but is based only on concepts that can be represented as single words, and were represented as such to the human annotators. This should be mentioned in the paper and makes the relevance of this dataset questionable with respect to representations of phrases and general concepts. - As the authors themselves note, they (quite extensively) fine tune their hyperparameters on the very same datasets for which they report their results and compare them with prior work. This makes all the reported results and analyses questionable.
- The authors suggest that their method is superb to prior work, as it achieved comparable results while prior work required much more manual annotation. I don't think this argument is very strong because the authors also use large manually-constructed ontologies, and also because the manually annotated dataset used in prior work comes from existing clinical records that did not require dedicated annotations.
- In general, I was missing more useful insights into what is going on behind the reported numbers. The authors try to treat the relation between a phrase and its component words on one hand, and a concept and its alternative phrases on the other, as similar types of a compositional relation. However, they are different in nature and in my mind each deserves a dedicated analysis. For example, around line 588, I would expect an NLP analysis specific to the relation between phrases and their component words. Perhaps the reason for the reported behavior is dominant phrase headwords, etc. Another aspect that was absent but could strengthen the work, is an investigation of the effect of the hyperparameters that control the tradeoff between the atomic and compositional views of phrases and concepts.
General Discussion: Due to the above mentioned weaknesses, I recommend to reject this submission. I encourage the authors to consider improving their evaluation datasets and methodology before re-submitting this paper.
Minor comments: - Line 069: contexts -> concepts - Line 202: how are phrase overlaps handled?
- Line 220: I believe the dimensions should be |W| x d. Also, the terminology ‘negative sampling matrix’ is confusing as the model uses these embeddings to represent contexts in positive instances as well.
- Line 250: regarding ‘the observed phrase just completed’, it not clear to me how words are trained in the joint model. The text may imply that only the last words of a phrase are considered as target words, but that doesn’t make sense. - Notation in Equation 1 is confusing (using c instead of o) - Line 361: Pedersen et al 2007 is missing in the reference section.
- Line 388: I find it odd to use such a fine-grained similarity scale (1-100) for human annotations.
- Line 430: The newly introduced term ‘strings’ here is confusing. I suggest to keep using ‘phrases’ instead.
- Line 496: Which task exactly was used for the hyper-parameter tuning?
That’s important. I couldn’t find that even in the appendix.
- Table 3: It’s hard to see trends here, for instance PM+CL behaves rather differently than either PM or CL alone. It would be interesting to see development set trends with respect to these hyper-parameters.
- Line 535: missing reference to Table 5.
- The similarity/relatedness evaluation datasets used in the paper are presented as datasets recording human judgements of similarity between concepts. However, if I understand correctly, the actual judgements were made based on presenting phrases to the human annotators, and therefore they should be considered as phrase similarity datasets, and analyzed as such.