### Theory
The main theoretical results are Theorem 2.1 and 2.2. They state that if the "average last-layer feature norm and the last-layer weight matrix norm are both bounded, then achieving near-optimal loss implies that most classes have intra-class cosine similarity near one and most pairs of classes have inter-class cosine similarity near -1/(C-1)".
Qualitatively, this result is an immediate consequence of continuity of the loss function together with the fact that bounded average last-layer feature norm and bounded last-layer weight matrices implies NC.
Quantitatively, this work proves asymptotic bounds on the proximity to NC as a function of the loss. This quantitative aspect is novel. I am not convinced of its significance however, as I will outline below.
1. The result is only asymptotic, and thus it cannot be used to estimate proximity to NC from a given loss value.
2. The bound is used as basis to argue that *"under the presence of batch normalization
and weight decay of the final layer, larger values of weight decay provide stronger NC guarantees in the sense that the intra-class cosine similarity of most classes is nearer to 1 and the inter-class cosine similarity of most pairs of classes is nearer to -1/(C-1)."*
This is backed up by the observation, that the bounds get more tight if the weight decay parameter $\lambda$ increases. To be more specific, Theorem 2.2 shows that if $L< L{min}+\epsilon$, then the average intra class cosine similarity is smaller than $-1/(C-1) + O(f(C,\lambda,\epsilon,\delta))$ and $f$ decreases with $\lambda$.
The problem with this argument is that the loss function itself depends on the regularization parameter $\lambda$ and so it is a-priori not clear whether values of $\epsilon$ are comparable for different $\lambda$. For example, apply this argument to the more simple loss function $L(x,\lambda)=\lambda x^2$. As $L$ is convex, it is clear that the value of $\lambda>0$ is irrelevant for the minimum and the near optimal solutions. Yet, $L(x,\lambda)<\epsilon$ implies $x^2<\epsilon/\lambda$ which decreases with $\lambda$. By the logic given in this work, the latter inequality suggests that minimizing a loss function with a larger value of $\lambda$ provides stronger guarantees for arriving close to the minimum at $0$. Clearly, this is not the case and an artifact of quantifying closeness to the loss minimum by $\epsilon$, when it should have been adjusted to $\lambda \epsilon$ instead.
I have doubts on how batch normalization is handled. As far as I see, batch normalization enters the proofs only through the condition $\sum_i \| h_i \|^2 =\| h_i \|^2$ (see Prop 2.1). However, this is only an implication and batch normalization induces stronger constraints. The theorems assume that the loss minimizer is a simplex ETF in the presence of batch normalization. This is not obvious, and neither proven nor discussed. It is also not accounted for in the part of the proof of Theorem 2.2, where the loss minimum $m_{reg}$ is derived.
### Experiments
- Theorems 2.1 and 2.2 are not evaluated empirically. It is not tested, whether the average intra / inter class cosine similarities of near optimal solutions follow the exponential dependency in $\lambda$ and the square (or sixth) root dependency on $\epsilon$ as suggested by the theorems.
- Instead, the dependency of cosine similarities at the end of training (200 epochs) on weight decay strength is evaluated. As presumed by the authors, the intra class cosine similarities get closer to the optimum, if the weight decay strength increases. Yet, there are problems with this experiment. It is inconsistent with the setting of the theory part and thus only provides limited insight on if the idealized theoretical results transfer to practice.
1. The theory part depends only on the weight decay strength on the last layer parameters. Yet, in the experiments, weight decay is applied to all layers and its strength varies between experiments (when instead only the strength of the last layer should change).
2. The theorems assume near optimal training loss, but training losses are not reported. Moreover, the reported cosine similarities are far from optimal (e.g. intra class is around 0.2 instead of 1) which suggests that the training loss is also far from optimal. It also suggests that the models are of too small capacity to justify the 'unconstrained-features' assumption.
3. As (suboptimally) weight decay is applied to all layers, we would expect a large training loss and thus suboptimal cosine similarities for large weight decay parameters. Conveniently, cosine similarities for such large weight decay strengths are not reported and the plots end at a weight decay strength where cosine similarities are still close to optimal.
4. On real-world data sets, the inter class cosine similarity increases with weight decay (even for batch norm models VGG11), disagreeing with the theoretical prediction. This observation is insufficiently acknowledged.
### General
The central question that this work wants to answer **What is a minimal set of conditions that would guarantee the emergence of NC?"** is already solved in the sense that it is known that minimal loss plus a norm constraint on the features (explicit via feature normalization or implicit via weight decay) implies neural collapse. The authors argue to add batch normalization to this list but that contradicts minimality.
The first contribution listed by the authors is not a contribution.
1. *"We propose the intra-class and inter-class cosine similarity measure, a simple and geometrically intuitive quantity that measures the proximity of a set of feature vectors to several core
structural properties of NC. (Section 2.2)"*
Cosine similarity (i.e. the normalized inner product) is a well known and an extensively used distance measure on the sphere. In the context of neural collapse, cosine similarities were already used in the foundational paper by Papyan et al. (2020) to empirically quantify closeness to NC (cf. Figure 3 in this reference) and many others. Minor:
- There is a grammatical error in the second sentence of the second paragraph
- There is no punctuation after formulas; In the appendix, multiple rows start with a punctuation
- intra / inter is sometimes written in italics, sometimes upright
- $\beta$ is used multiply with a different meaning
- Proposition 2.1 $N$ = batch site, Theorem 2.2 $N$ = number of samples per class.
- As a consequence, it seems that $\gamma$ needs to be rescaled to account for the number of batches
3. As (suboptimally) weight decay is applied to all layers, we would expect a large training loss and thus suboptimal cosine similarities for large weight decay parameters. Conveniently, cosine similarities for such large weight decay strengths are not reported and the plots end at a weight decay strength where cosine similarities are still close to optimal.