The majority of the weaknesses in the paper seem to stem from confusion and inconsistencies between some of the prose and the results.
1. Figure 2, as it is, isn't totally convincing there is a gap in convergence times. The x-axis of the graph is time, when it would have been more convincing using steps. Without an efficient, factored attention for prompting implementation a la [He et al. (2022)](https://arxiv.org/abs/2110.04366) prompt tuning can cause slow downs from the increased sequence length. With time on the x-axis it is unclear if prompt tuning requires more steps or if each step just takes more time. Similarly, this work uses $0.001$ for the learning rate. This is a lot smaller than the suggested learning rate of $0.3$ in [Lester et al (2021)](https://aclanthology.org/2021.emnlp-main.243/), it would have been better to see if a larger learning rate would have closed this gap. Finally, this gap with finetuning is used as a motivating examples but the faster convergence times of things like their initialization strategy is never compared to finetuning.
2. Confusion around output space and label extraction. In the prose (and Appendix A.3) it is stated that labels are based on the predictions at `[MASK]` for RoBERTa Models and the T5 Decoder for generation. Scores in the paper, for example the random vector baseline for T5 in Table 2 suggest that the output space is restricted to only valid labels as a random vector of T5 generally produces nothing. Using this rank classification approach should be stated plainly as direct prompt reuse is unlikely to work for actual T5 generation.
3. The `laptop` and `restaurant` datasets don't seem to match their descriptions in the appendix. It is stated that they have 3 labels but their random vector performance is about 20% suggesting they actually have 5 labels?
4. Some relative performance numbers in Figure 3 are really surprising, things like $1$ for `MRPC` to `resturant` transfer seem far too low, `laptop` source to `laptop` target on T5 doesn't get 100, Are there errors in the figure or is where something going wrong with the datasets or implementation?
5. Prompt similarities are evaluated based on correlation with zero-shot performance for direct prompt transfer. Given that very few direct prompt transfers yield gain in performance, what is actually important when it comes to prompt transferability is how well the prompt works as an initialization and does that boost performance. Prompt similarity tracking zero-shot performance will be a good metric if that is in turn correlated with transfer performance. The numbers from Table 1 generally support that this as a good proxy method as 76% of datasets show small improvements when using the best zero-shot performing prompt as initialization when using T5 (although only 54% of datasets show improvement for RoBERTa). However Table 2 suggests that this zero-shot performance isn't well correlated with transfer performance. In only 38% of datasets does the best zero-shot prompt match the best prompt to use for transfer (And of these 5 successes 3 of them are based on using MNLI, a dataset well known for giving strong transfer results [(Phang et al., 2017)](https://arxiv.org/abs/1811.01088)). Given that zero-shot performance doesn't seem to be correlated with transfer performance (and that zero-shot transfer is relatively easy to compute) it seems like _ON_'s strong correlation would not be very useful in practice.
6. While recent enough that it is totally fair to call [Vu et al., (2021)](https://arxiv.org/abs/2110.07904) concurrent work, given the similarity of several approaches there should be a deeper discussion comparing the two works. Both the prompt transfer via initialization and the prompt similarity as a proxy for transferability are present in that work. Given the numerous differences (Vu et al transfer mostly focuses on large mixtures transferring to tasks and performance while this work focuses on task to task transfer with an eye towards speed. _ ON_ as an improvement over the Cosine similarities which are also present in Vu et al) it seems this section should be expanded considering how much overlap there is.
7. The majority of Model transfer results seem difficult to leverage. Compared to cross-task transfer, the gains are minimal and the convergence speed ups are small. Coupled with the extra time it takes to train the projector for _Task Tuning_ (which back propagation with the target model) it seems hard to imagine situations where this method is worth doing (that knowledge is useful). Similarly, the claim on line 109 that model transfer can significantly accelerate prompt tuning seems lie an over-claim.
8. Line 118 claims `embedding distances of prompts do not well indicate prompt transferability` but Table 4 shows that C$_{\text{average}}$ is not far behind _ON_. This claim seems over-reaching and should instead be something like "our novel method of measuring prompt similarity via model activations is better correlated with transfer performance than embedding distance based measures"
1. Line 038: They state that GPT-3 showed extremely large LM can give remarkable improvements. I think it would be correct to have one of their later citations on continually developed LM as the one that showed that. GPT-3 mostly showed promise for Few-Shot evaluation, not that it get really good performance on downstream tasks.
2. Line 148: I think it would make sense to make a distinction between hard prompt work updates the frozen model (Schick and Schütez, etc) from ones that don't.
3. Line 153: I think it makes sense to include [_Learning How to Ask: Querying LMs with Mixtures of Soft Prompts_ (Qin and Eisner, 2021)](https://aclanthology.org/2021.naacl-main.410.pdf) in the citation list for work on soft prompts.
4. Figure 3: The coloring of the PI group makes the text very hard to read in Black and White.
5. Table 1: Including the fact that the prompt used for initialization is the one that performed best in direct transfer in the caption as well as the prose would make the table more self contained.
6. Table 2: Mentioning that the prompt used as cross model initialization is from _Task Tuning_ in the caption would make the table more self contained.
7. Line 512: It is mentioned that _ON_ has a drop when applied to T$5_{\text{XXL}}$ and it is suggested this has to do with redundancy as the models grow. I think this section could be improved by highlighting that the Cosine based metrics have a similar drop (suggesting this is a fact of the model rather than the fault of the _ON_ method). Similarly, Figure 4 shows the dropping correlation as the model grows. Pointing out the that the _ON_ correlation for RoBERTA$_{\text{large}}$ would fit the tend of correlation vs model size (being between T5 Base and T5 Large) also strengths the argument but showing it isn't an artifact of _ON_ working poorly on encoder-decoder models. I think this section should also be reordered to show that this drop is correlated with model size. Then the section can be ended with hypothesizing and limited exploration of model redundancy.
8. Figure 6. It would have been interesting to see how the unified label space worked for T5 rather than RoBERTAa as the generative nature of T5's decoding is probably more vulnerable to issue stemming from different labels.
9. _ ON_ could be pushed farther. An advantage of prompt tuning is that the prompt is transformed by the models attention based on the value of the prompt. Without having an input to the model, the prompts activations are most likely dissimilar to the kind of activations one would expect when actually using the prompt.
10. Line 074: This sentence is confusing. Perhaps something like "Thus" over "Hence only"?
11. Line 165: Remove "remedy,"
2. Line 148: I think it would make sense to make a distinction between hard prompt work updates the frozen model (Schick and Schütez, etc) from ones that don't.