The amount of background provided can be reduced, and consists of quite a few detailed descriptions of topics and experiments that are not directly related to the experiments of the paper (e.g. the Priming paragraph at L210, Novel verbs paragraph at L224). The space that is currently occupied by this extensive background could be used more efficiently, and cutting it down a bit opens up space for additional experiments. The ‘Jabberwocky constructions’ experiment is quite prone to several potential confounds that need to be explored in more detail in order to ensure that the current set of results truly hints at ‘the neural reality of argument structure constructions’. The fact that the contextualised embeddings of verbs in the same syntactic configuration is highly similar isn’t that surprising in itself (as is noted by the authors as well). The authors decided to drop the ‘priming’ component of the original paper in order to adapt the experiment to LMs, but there are other options that can be explored to align the setup more closely to the original (see section below for some ideas).
### Comments / Questions - Could the results of the Sentence Sorting be driven by the fact that sentence embeddings are obtained by averaging over word embeddings? It seems that this procedure would be quite prone to simply cluster based on features stemming from individual tokens, instead of a more general abstract signal. I could imagine that in a BERT-like architecture the representation at the [CLS] position might serve as a sentence representation as well.
- Alternatively, would it be possible to set up the sentence sorting experiment in such a way that the lexical overlap in between sentences is limited? This is common in structural priming experiments as well, and models are known to rely heavily on lexical heuristics. ,
- Did you consider different measures of similarity in the Jabberwocky experiment? Euclidean distance might not be the most perfect measure for expressing similarity, and I would suggest looking into alternatives as well, like cosine similarity. - A bit pedantic, but Jabberwocky words are non-existing nonce words, whereas the setup that the authors arrived at is only semantically nonsensical, yet still made up of existing words (a la ‘Colorless green ideas’). Referring to them as Jabberwocky (L.458) would give the impression of actually using nonce words.
- How many ASCs have been argued to exist (within English)? Is there a reason why the 4 constructions used in Case Study 1 (_transitive, ditransitive, caused-motion, resultative_; L.165), are slightly different from Case Study 2 (_ditransitive, resultative, caused-motion, removal_; Table 2)?
--- ### Suggestions: - L.490: “we take the embedding of the first subword token as the verb embedding.” It is also quite common in cases like that to average over the subword representations, which is done e.g. by [Hewitt and Manning (2019, footnote 4)](https://aclanthology.org/N19-1419.pdf).
- I suggest not phrasing the Jabberwocky experiment as “probing” (L.444 ‘method to probe LMs’, L.465 ‘probing strategy’, etc.), given the connotation this term has with training small classifiers on top of a model’s hidden state representations.
- Could the ‘priming’ aspect of Johnson and Goldberg (2013) in the Jabberwocky experiment perhaps be emulated more closely by framing it as a “Targeted Syntactic Evaluation” task (akin to Marvin & Linzen (2018), a.o.). In the context of priming, a similar setup has recently been utilised by [Sinclair et al. (2021)](https://arxiv.org/pdf/2109.14989.pdf). One could compare the probability of _P(gave | She traded her the epicenter. He)_ to that of _P(gave | He cut it seasonal. She)_, and likewise for the other 2 constructions. This way you wouldn’t run into the confounding issues that stem from using the contextualisation of ‘gave’. - An additional experiment that might be interesting to explore is by probing for construction type across layers at the position of the verb. In the ‘Jabberwocky’ setup one would expect that at the word embedding level construction information can’t be present yet, but as it is contextualised more and more in each layer the ASC information is likely to increase gradually. Would also be interesting than to see how the curve of a jabberwocky verb compares to that of a sensical/prototypical verb (like _gave_): there is probably _some_ degree of argument structure already encoded in the word embedding there (as a lexicalist would argue), so I would expect probing performance for such verbs to be much higher at lower levels already. - Adding to the previous point: probing in itself would not even be necessary to gain insight into the layerwise contextualisation; some of the current experiments could be conducted in such a fashion as well.
--- ### Typos/Style: Very well written paper, no remarks here.
- L.490: “we take the embedding of the first subword token as the verb embedding.” It is also quite common in cases like that to average over the subword representations, which is done e.g. by [Hewitt and Manning (2019, footnote 4)](https://aclanthology.org/N19-1419.pdf).