The main weak point of the paper is that at it is not super clear. There are many parts in which I believe the authors should spend some time in providing either more explanations or re-structure a bit the discussion (see the comments section).
- I suggest to revise a bit the discussion, especially in the modeling section, which in its current form is not clear enough. For example, in section 2 it would be nice to see a better formalization of the architecture. If I understood correctly, the Label Embeddings are external parameters; instead, the figure is a bit misleading, as it seems that the Label Embeddings are the output of the encoder.
- Also, when describing the contribution in the Introduction, using the word hypothesis/null hypothesis really made me think about statistical significance. For example, in lines 87-90 the authors introduce the hypothesis patterns (in contrast to the null hypothesis) referring to the way of representing the input labels, and they mention "no significant" difference, which instead is referring to the statistical significance. I would suggest to revise this part.
- It is not clear the role of the dropout, as there is not specific experiment or comment on the impact of such technique. Can you add some details?
- On which data is fine-tuned the model for the "Knowledge Distillation" - Please, add an intro paragraph to section 4.
- The baseline with CharSVM seems disadvantaged. In fact, a SVM model in a few-shot setting with up to 5-grams risks to have huge data-sparsity and overfitting problems. Can the authors explain why they selected this baseline? Is a better (fair) baseline available?
- In line 303 the authors mention "sentence transformers". Why are the authors mentioning this? Is it possible to add a citation?
- There are a couple of footnotes referring to wikipedia. It is fine, but I think the authors can find a better citation for the Frobenius norm and the Welch test.
- I suggest to make a change to the tables. Now the authors are reporting in bold the results whose difference is statistical significant. Would it be possible to highlight in bold the best result in each group (0, 8, 64, 512) and with another symbol (maybe underline) the statistical significant ones? -
- On which data is fine-tuned the model for the "Knowledge Distillation" - Please, add an intro paragraph to section 4.