- The paper hypothesizes that SimCSE suffers from the cue of sentence length and syntax. However, the experiments only targets sentence length but not syntax. - The writing of this paper can benefit by some work (see more below). Specifically, I find Section 3 difficult to understand as someone who does not directly work on this task. Specifically, a good mount of terminologies are introduced without explanations. I suggest a good effort of rewriting this section to be easier understandable by general NLP researchers.
- Though a universal issue in related papers and should not be blamed on the authors, why only consider BERT-base? It is known that other models such as BERT-large, RoBERTa, DeBERTa, etc. could produce better embeddings, and that the observations in these works do not necessarily hold in those larger and better models. - The introduction of the SRL-based discrete augmentation approach (line 434 onwards) is unclear and cannot be possibly understood by readers without any experience in SRL. I suggest at least discussing the following: - Intuitively why relying on semantic roles is better than work like CLEAR - What SRL model you use - What the sequence "[ARG0, PRED, ARGM − NEG, ARG1]" mean, and what these PropBank labels mean - What is your reason of using this sequence as opposed to alternatives
- (Line 3-6, and similarly in the Intro) The claim is a finding of the paper, so best prefix the sentence with "We find that". Or, if it has been discussed elsewhere, provide citations. - (7): semantics-aware or semantically-aware - (9-10): explore -> exploit - (42): works on -> works by - Figure 1 caption: embeddings o the : typo - Figure 1 caption: "In a realistic scenario, negative examples have the same length and structure, while positive examples act in the opposite way." I don't think this is true. Positive or negative examples should have similar distribution of length and structure, so that they don't become a cue during inference. - (99): the first mention of "momentum-encoder" in the paper body should immediately come with citation or explanation. - (136): abbreviations like "MoCo" should not appear in the section header, since a reader might not know what it means. - (153): what is a "key"?
- (180): same -> the same - (186-198): I feel that this is a better paragraph describing existing issue and motivation than (63-76). I would suggest moving it to the Intro, and just briefly re-mention the issue here in Related Work. - (245-246): could be -> should be - (248): a -> another - (252-55): Isn't it obvious that "positive examples in SimCSE have the same length", since SimCSE enocdes the same sentence differently as positive examples? How does this need "Through careful observation"?
- (288): "textual similarity" should be "sentence length and structure"? Because the models are predicting textual similarity, after all.
- (300-301): I don't understand "unchangeable syntax" - (310-314): I don't understand "query", "key and value". What do they mean here? Same for "core", "pseudo syntactic". - (376): It might worth mentioning SimCSE is the state-of-the-art method mention in the Abstract. - (392): Remove "In this subsection"
- (136): abbreviations like "MoCo" should not appear in the section header, since a reader might not know what it means.