W1. Clarity
The organization of the paper is such that the reader has to refer to the appendix a lot. My biggest concern on clarity is on the “theoretical” results which are not rigorous and at times unsupported. Further, some statements/claims are not precise or clear enough for me to be convinced that the method is well-motivated and is doing what it is claimed to be doing.
W2. Soundness
I have a lot of concerns and questions here as I read through Sect. 3. At a high-level, I don’t see a clear connection between “improved variance control of prediction y^ or the smoothness of loss landscape” and “zero-shot learning effectiveness.” Details below. This is in part due to poor clarity.
W3. Experiments
IMO, if the main claim is really about the effectiveness of the two tricks and the proposed class normalization, then the experiments should go beyond one zero-shot learning starting point --- 3-layer MLP (Table 2).
If baseline methods already adopt some of these tricks, it should be made clear and see if removing these tricks lead to inferior performance.
If baseline methods do not adopt some of these tricks, these tricks, especially class normalization, could be applied to show improved performance. If it is difficult to apply these tricks, further explanation should be given (generally, also mention applicability of these tricks.)
This is done to some degree in the continual setting.
W4. Related work
As I mentioned in W3, it is unclear which methods are linear/deep, and which methods have already benefited from existing/proposed tricks.
Detailed comments (mainly to clarify my points about weaknesses)
Statement 1
The main claim for this part is that this statement provides “a theoretical understanding of the trick” and “allows to speed up the search [of the optimal value fo \gamma].”
However, I feel that we need further justifications on the correlation between Statement 1 (variance of y^_c, “better stability” and “the training would not stale”) and the zero-shot learning accuracy for this to be the “why normalization + scaling works.” My understanding is that the Appendix simply validates that Eq. (4) seems to hold in practice.
Moreover, is the usual search region [5,10] actually effective? Do we have stronger supporting empirical evidence than the three groups of practitioners (Li et al 2019, Zhang et al. 2019, Guo et al. 2020), who may have influenced each other, used it?
Finally, can the authors comment on the validity of multiple assumptions in Appendix A? To which degrees does each of them hold in practice?
Statement 2 and 3
Why wouldn’t the following statement in Sect. 3.3 invalidate Statement 1? “This may create an impression that it does not matter how we initialize the weights — normalization would undo any fluctuations. However it is not true, because it is still important how the signal flows, i.e. for an unnormalized and unscaled logit value”
It is unclear (at least not from the beginning) why understanding attribute normalization has to do with initialization of the weights.
Similar to my comments to Statement 1, why should we believe that the explanation in Sect. 3.3 and Sect. 3.4 is the reason for zero-shot learning effectiveness? In particular, the authors again claim that the main bottleneck in improving zero-shot learning is “variance control” (the end of Sect. 3.3).
I also have a hard time understanding some statements in Appendix H, which is needed to motivate the following statement in Sect. 3.3: “And these assumptions are safe to assume only for z but not for a_c, because they do not hold for the standard datasets (see Appendix H).” H1: Would this statement still be true after we transform a_c with an MLP? H2: Why is it not “a sensible thing to do” if we just want zero mean and unit variance? H3: Why is “such an approach far from being scalable”? H4: What if these are things like word embeddings? H5: Fig. 12 and Fig. 13 are not explained. H6: Histograms in Fig. 13 look quite normal.
How useful is Statement 2? Why is the connection with Xavier initialization important?
Why is “preserving the variance between z and y~” in Statement 3 important for zero-shot learning?
Improved smoothness
The claim “improved smoothness” at the end of Sect. 3 and Appendix F is really hard to understand. F1: How do the authors define “irregular loss surface”? F2: “Santurkar et al. (2018) showed that batch-wise standardization procedure decreases the Lipschitz constant of a model, which suggests that our class-wise standardization will provide the same impact.” This is not very precise and seems unsupported. Please make it clear how. If this is a hypothesis, please make it clear.
Similarly to my comments to Statement 1-3, how is improved smoothness related to zero-shot learning effectiveness?
Other more minor comments
Abstract: Are the authors the one to “generalize ZSL to a broader problem”? Please tone down the claim if not.
After Eq. (2): Why does attribute normalization look “inconsiderable” (possibly this is not the right word?) or why is it “surprising” that this is preferred in practice? Don’t most zero-shot learning methods use this (see for example Table 4 in [A])?
Suggestions for references for attribute normalization. This can be improved; I can trace this back to much earlier work such as [A] and [B] (though I think this fact is stated more explicitly in [A]).
Under Table 1 “These two tricks work well and normalize the variance to a unit value when the underlying ZSL model is linear (see Figure 1), but they fail when we use a multi-layer architecture.”: Could the authors provide a reference to evidence to support this? I think it is also important to provide a clear statement of what separates a “linear” or “multi-layer” model.
The first paragraph of Sect. 3: Could you provide references for motivations for different activation functions? Further, It is unclear that all of them perform normalization.
The second paragraph of Sect. 3: What exactly limits “the tools” for zero-shot learning vs. supervised learning? Further, it would also be nice to separate traditional supervised learning where classes are balanced and imbalanced; see, e.g., [C].
What is the closest existing zero-shot model to the one the authors describe in Sect. 3.1? Why is the described model considered/selected?
[A] Synthesized Classifiers for Zero-Shot Learning
[B] Zero-Shot Learning by Convex Combination of Semantic Embeddings
[C] Class-Balanced Loss Based on Effective Number of Samples
3. At a high-level, I don’t see a clear connection between “improved variance control of prediction y^ or the smoothness of loss landscape” and “zero-shot learning effectiveness.” Details below. This is in part due to poor clarity.