Behavior of gradient descent iterates under the unhinged loss with different weight decay coefficients and $\eta_1(t)=\eta_2(t)=0.1$ (i.e., $s=1$), where $\mP$ denotes the component $\mW_1^+$ in the projection $\Pi_1^+\mZ_0$ calculated according to Lemma <ref>. (a) The logarithm of the norm of $\mW(t)$. As expected, the norm increases exponentially when $\lambda<\lambda^*$; (b) The difference between $\mW(t)$ and $\mP$. As expected (Corollary <ref>), $\mW(t)$ converges to $\mP$ when $\lambda=\lambda^*$, while other differences are dominated by $\|\mW(t)\|_2$; (c) The difference in $\ell_2$ norm between $\mH(t)$ and $\mW(t)$. The convergence is the same even if the weight decay is different.

[Train Accuracy]

Verification of the behavior of regularized gradient descent iterates in <Ref> with $\gamma\in\{0, 0.1, 0.01, 0.001, \frac{1}{C-1}\}$. We set $p=512$, $C=100$, $N=10$, $\lambda=\frac{(1+\gamma)}{C\sqrt{N}}$, $\eta_1(t)=\eta_2(t)=0.1$ (i.e., $s=1$), thus we have $\lim_{t\rightarrow\infty}\mZ(t)=\Pi_1^+\mZ_0$, according to Corollary <ref>, and then randomly initialize $\mH_0$ and $\mW_0$. (a) The training accuracy with the prediction rule $\argmax_c \vw_c^\top\vh$. As expected, the features align to their corresponding prototypes when $\gamma<\frac{2}{C-2}$. (b) The $\ell_2$ distance between $\mH(t)$ and $\mH_1^+$. As expected <ref>, the distance will decrease as exponential rate when $0<\gamma<\frac{2}{C-2}$.   (c) The $\ell_2$ distance between $\mW(t)$ and $\mW_1^+$.  (d) and (e) denote the norm of features and prototypes, respectively. As can be seen, $\|\mH\|_2$ and $\|\mW\|_2$ do not grow exponentially as in the unconstrained case, which confirms that weight decay can avoid excessive growth of feature norm and prototype norm.

[Train Accuracy]

Verification of the behavior of regularized gradient descent iterates in <Ref> with different learning rates ($\eta\in \{0.001, 0.01, 0.1, 0.2, 0.5\}$). We set $p=512$, $C=100$, $N=10$, $\eta_1=\eta_2=\eta$ ($s=1$), $\gamma=\frac{1}{C-1}$, and $\lambda=\frac{1+\gamma}{C\sqrt{N}}$. As can be seen, features and prototypes converge to $(\overline{\mH}, \overline{\mW})$ exponentially, and larger learning rates can accelerate convergence.

[Train accuracy]

Verification of the behavior of regularized gradient descent iterates in <Ref> with different weight decay coefficients ($\lambda=\{1e-5, 1e-4, 1e-3,5e-3,\frac{1+\gamma}{C\sqrt{N}}\}$).  We set $p=512$, $C=100$, $N=10$, $\eta_1(t)=\eta_2(t)=0.1$ (i.e., $s=1$), $\gamma=\frac{1}{C-1}$, where $\overline{\mH}=\pi_h^+\mH_1^++\pi_h^-\mH_1^-$ and $\overline{\mW}=\pi_w^+\mW_1^++\pi_w^-\mW_1^-$ in Corollary <ref>. (a) The logarithm of the norm of $\mW(t)$. As expected, the norm increases exponentially when $\lambda<\lambda^*=\frac{1+\gamma}{C\sqrt{N}}$; (b) The difference between $\mW(t)$ and $\mP$. As expected in Corollary <ref>, $\mW(t)$ converges to $\mP$ when $\lambda=\lambda^*$, while other differences are dominated by $\|\mW(t)\|_2$; (c) The difference in $\ell_2$ norm between $\mH(t)$ and $\mW(t)$. The convergence is the same even if the weight decay is different.

[Train accuracy]

Verification of the behavior of regularized gradient descent iterates in <Ref> with different scale parameters $s=\frac{\eta_1(0)}{\eta_2(0)} \in \{0.01, 0.05, 0.1, 0.5, 1.0\}$.  We set $p=512$, $C=100$, $N=10$, $\eta_2=0.5$, $\gamma=\frac{1}{C-1}$, where $\overline{\mH}=\pi_h^+\mH_1^++\pi_h^-\mH_1^-$ and $\overline{\mW}=\pi_w^+\mW_1^++\pi_w^-\mW_1^-$ in Corollary <ref>. As can be seen, larger scale parameter $s$ can achieve faster convergence speed.

[Train Accuracy]

[Train Accuracy]

Verification of the behavior of discrete gradient descent iterates in <Ref> under anchored prototypes with different learning rates $\eta\in\{0.1, 0.5, 1.0, 5.0, 10\}$ and without (a-c) or with (d-f) rescaled learning rates. We set $p=512$, $C=100$, and $N=10$. As expected in Theorem <ref>, the feature norm $\|\mH(t)\|_2$ is non-decreasing, and the error $\|\hat{\mH}(t)-\hat{\mW}(\mI_C\otimes \1_N^\top)\|_2$ shows exponential decrease.

Test accuracies on imbalanced CIFAR-10 under different explicit feature regularization.

Dataset     8cImbalanced CIFAR-10

Imbalance Type      4c|long-tailed     4cstep

Imbalance Ratio      1c|100     1c|50     1c|20     1c|10      1c|100     1c|50     1c|20     1c10

baseline     67.81     72.93     83.97     88.37     61.24     68.10     78.73     85.49

$\lambda=5e-6$     67.84     72.85     83.17     89.06     60.79     68.41     80.20     86.69

$\lambda=1e-5$     67.74     76.14     84.17     89.19     61.50     67.71     80.97     87.18

$\lambda=5e-5$     69.74     77.29     84.92     88.64     60.69     70.27     81.27     87.17

Test accuracies on imbalanced CIFAR-100 under different explicit feature regularization.

Dataset     8cImbalanced CIFAR-100

Imbalance Type      4c|long-tailed     4cstep

Imbalance Ratio      1c|100     1c|50     1c|20     1c|10      1c|100     1c|50     1c|20     1c10

baseline      33.37     39.40     42.96     56.38     40.89     42.69     51.92     57.52

$\lambda=5e-6$     36.00     41.92     50.75     60.13     41.90     43.85     47.80     56.74

$\lambda=1e-5$     36.61     42.36     49.21     58.91     41.48     43.77     49.64     56.49

$\lambda=5e-5$     34.88     42.74     54.72     60.84     40.97     43.20     48.96     57.97

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

Reliability diagrams of ResNet-34 [He et al., 2016] trained by CE on CIFAR-10-LT with imbalance ratio $\rho\in\{100, 50, 20, 10\}$ under different explicit feature regularization ($\lambda\in\{0, 5e-6, 1e-5, 5e-5\}$). As can be seen, an appropriate larger weight decay can improve both accuracy and confidence

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

[CE ($\lambda=0$)]

[CE ($\lambda=5e-6$)]

[CE ($\lambda=1e-5$)]

[CE ($\lambda=5e-5$)]

Reliability diagrams of ResNet-34 [He et al., 2016] trained by CE on CIFAR-100-LT with imbalance ratio $\rho\in\{100, 50, 20, 10\}$ under different explicit feature regularization ($\lambda\in\{0.0, 5e-6, 1e-5, 5e-5\}$), where ECE denotes the expected calibration error [Zhong et al., 2021].  As can be seen, an appropriate larger weight decay can improve both accuracy and confidence calibration.

[Feature Norm]

[Train Loss]

[Validation Accuracy]

[Feature Norm]

[Train Loss]

[Validation Accuracy]

[Feature Norm]

[Train Loss]

[Validation Accuracy]

Behavior of visual classification on CIFAR-10 with CE, the unhinged loss (PAL) and the unhinged loss (FNPAL) under different weight decay coefficients.

[Feature Norm]

[Train Loss]

[Validation Accuracy]

[Feature Norm]

[Train Loss]

[Validation Accuracy]

[Feature Norm]

[Train Loss]

[Validation Accuracy]

Behavior of visual classification on CIFAR-100 with CE, the unhinged loss (PAL) and the unhinged loss (FNPAL) under different weight decay coefficients.

[Train Loss]

[Feature Norm]

[Train Accuracy]

[Validation Accuracy]

Behavior of visual classification on ImageNet-100 with CE, the unhinged loss (PAL) and the unhinged loss (FNPAL) under different weight decay coefficients.

[Train Loss]

[Train Accuracy]

[Feature Norm]

Behavior of gradient descent iterates of the unhinged (PAL) loss in <ref> with different explicit feature regularization ($\lambda\in\{0, 0.001, 0.005, 0.01, 0.05, 0.1\}$). We set $p=512$, $C=100$, $N=10$, and $\eta=0.1$. We randomly initialize $\mH_0$ and $\mW$, and then anchor prototypes $\mW$ during training. As expected in <ref>, the error $\|\hat{\mH(t)}-\hat{\mW\mM}\|_2$ decreases as an exponential rate $O(e^{-\lambda \eta t})$, and a larger $\lambda$ can accelerate the convergence.

Distribution of softmax scores [Hendrycks and Gimpel, 2016] from models trained with different explicit feature regularization, where CE is the loss function.

Distribution of energy scores [Liu et al., 2020] from models trained with different explicit feature regularization, where CE is the loss function.

Distribution of feature norms from models trained with different explicit feature regularization, where CE is the loss function.

The behavior of features and prototypes when directly training ResNet-18 with the unhinged loss in <ref> on CIFAR-10. We set the weight decay coefficient as $5e-4$. (a) The train accuracy. (b) The feature norm. (c) the ratio $\frac{\max_i\|\vw_i\|_2}{\min_i\|\vw_i\|_2}$. (d) $\min_i\|\vw_i\|_2$. (e) The minimal angular between prototypes:  $\arccos\max_{i\neq j}\hat{\vw}_i^\top\hat{\vw}_j$. In these figures, we only show the curves for the first 21 epochs, since  “NaN” appears at the 22-th epoch. We can find that implicit penalization attached by other components (e.g., network architectures and weight decays) does not limit the rapid growth of the feature norm and prototype norm, indicating implicit penalization is fragile. Moreover, the ratio $\frac{\max_i\|\vw_i\|_2}{\min_i\|\vw_i\|_2}$ starts our very large and the minimal angular is very small, which indicates that there are two prototypes that are particular imbalanced.

§ EXPERIMENTS

In this section, we provide experimental details, including datasets, network architectures, optimization methods, hyperparameter settings, and more results.

§.§ Numerical Experiments

For numerical experiments in Figures <ref>, <ref>, <ref>, <ref>, <ref>, <ref>, and <ref>, we set $p=512$, $C=100$, $N=10$, and then randomly initialize $\mH_0$ and $\mW_0$. We use the SGD optimizer to optimize these free variables.

§.§ Visual Classification

For classification experiments in <ref>, <ref>, <ref>, <ref>, we experiment with ResNet-18, ResNet-34, and ResNet-50 [He et al., 2016] trained on CIFAR-10, CIFAR-100 [Krizhevsky and Hinton, 2009], and ImageNet-100 that takes the first 100 classes of ImageNet [Deng et al., 2009] , respectively. The networks are trained for 200 epochs and 100 epochs for CIFAR-10/-100 and ImageNet-100, respectively. For all training, we use SGD optimizer with momentum 0.9 and cosine learning rate annealing [Loshchilov and Hutter, 2017] with $T_{\max}$ being the corresponding epochs. The initial learning rate is set to $0.1$, weight decay is set to $5\times 10^{-4}$, and batch size is set to 256. Typical data augmentations including random width/height shift and horizontal flip are applied. Moreover, to use the PAL and FNPAL [Zhou et al., 2022] that anchors prototypes with a neural collapse solution, we remove the ReLU layer before the linear classifier in the last layer.

§.§ Imbalanced Classification

For the experiments of imbalanced learning in <ref>, <ref>, <ref>, <ref>, and <ref>, we utilize the same network architectures, and optimization settings as visual classification. We only use the imbalanced versions of CIFAR-10 and CIFAR-100 by following the setting in [Zhou et al., 2022]. The number of training examples is reduced for per class, and the test set keeps unchanged, where we use the imbalance ratio $\rho=\frac{\max_i n_i}{\min_i n_i}$ to denote the ratio between sample sizes of the most frequent and least frequent class. Moreover, long-tailed imbalance [Cui et al., 2019] that utilizes an exponential decay in samples sizes and step imbalance [Buda et al., 2018](that sets all minority classes to have the same number of samples, as do all majority classes) are considered.

For imbalanced learning, we utilize expected calibration error (ECE) to measure calibration of the models [Zhong et al., 2021], where all predictions are grouped into several interval bins of equal size and then calculate the error between the accuracy and confidence for each interval bin, i.e,

\begin{equation}
\mathop{\text{ECE}}=\sum_{b=1}^B\frac{|\gS_b|}{N}\left|\mathop{\text{acc}}(\gS_b)-\mathop{\text{conf}}(\gS_b)\right| \times 100\%,
\end{equation}

where $N$ denotes the number of predictions, $B$ is the number of interval bins, $\gS_b$ is the set of samples whose prediction scores fall into Bin-$b$,  $\mathop{\text{acc}}(\cdot)$ and $\mathop{\text{conf}}(\cdot)$ denote the accuracy and predicted confidence of $\gS_b$, respectively.

As shown in <ref> and <ref>, explicit feature regularization can improve imbalanced learning on CIFAR-10/-100 in most cases.

§.§ Out-of-Distribution Detection

For the experiments of OOD detection in <ref>, <ref>, <ref>, <ref>, and <ref>, we use a ResNet-18 on CIFAR-10 and a ResNet-34 on CIFAR-100 to train the classification models, and use their test dataset as the in-distribution data $\mathcal{D}_{\text{in}}^{\text{test}}$. For the OOD test dataset $\mathcal{D}_{\text{out}}^{\text{test}}$, we simply use a common benchmark: SVHN [Netzer et al., 2011]. We measure the performance with the following metrics: (1) the false positive rate (FPR95) of OOD examples when true positive rate of in-distribution examples is at $95\%$; (2) the area under the receiver operating characteristic curve (AUROC); and (3) the area under the precision-recall curve (AUPR). We then consider the softmax-based score [Hendrycks and Gimpel, 2016], energy-based score [Liu et al., 2020], and our proposed feature norm-based score to assessing the improvement of explicit feature regularization over the normal training.

OOD detection performance using softmax-based [Hendrycks and Gimpel, 2016], energy-based [Liu et al., 2020], and feature norm-based approaches while model training with feature regularization ($\lambda=\{0,1e-6,5e-6,1e-5\}$). We use ResNet-18 and ResNet-34 to train on the in-distribution datasets CIFAR-10 and CIFAR-100, respectively. We then use SVHN [Netzer et al., 2011] as the OOD dataset to evaluate the performance of OOD detection. All values are percentages. $\uparrow$ indicates large values are better, and $\downarrow$ indicates smaller values are better. The best results are underlined.

Dataset $\gD_{\text{in}}^{\text{test}}$     $\lambda$     FPR95 $\downarrow$      AUROC $\uparrow$     AUPR $\uparrow$

5*CIFAR-10     4cSoftmax-based / Energy-based / Feature Norm-based

0        52.09 / 43.04 / 52.10     91.67 / 91.94 / 89.54     84.11 / 82.80 / 77.06

1e-6     54.00 / 43.72 / 51.45     91.44 / 92.12 / 89.08     82.31 / 81.77 / 74.16

5e-6     45.37 / 33.92 / 26.93     93.08 / 93.78 / 94.03     84.31 / 83.73 / 82.79

1e-5     37.39 / 27.87 / 24.94     93.90 / 94.60 / 94.17     85.48 / 85.34 / 83.15

5*CIFAR-100     4cSoftmax-based / Energy-based / Feature Norm-based

0        87.75 / 89.84 / 95.54     71.01 / 71.94 / 59.54     55.42 / 56.69 / 43.21

1e-6     82.08 / 82.57 / 88.77     75.36 / 76.28 / 68.83     61.40 / 61.90 / 51.58

5e-6     79.01 / 78.68 / 85.94     78.70 / 79.15 / 70.32     62.58 / 62.39 / 48.39

1e-5     81.48 / 81.41 / 87.83     77.02 / 78.03 / 73.91     62.92 / 63.66 / 58.81

§ OTHER POTENTIAL INSIGHTS

§.§ A Good Initialization of Prototypes

As depicted in <ref> and <ref>, the dynamics under the unhinged loss is dependent on the initialization of both features and prototypes, such as $\Pi_1^+\mZ_0=(\frac{1}{\sqrt{N}}(\mP\otimes \1_N^\top),\mP)$, where $\mP=\frac{1}{2}\left(\frac{1}{\sqrt{N}}\mH_0(\mI_C\otimes \1_N)+\mW_0\right)(\mI_C-\frac{1}{C}\1_C\1_C^\top)$. However, these features $\mH_0$ extracted from a dataset by some nonlinear layers and parameterized layers are practically intractable, but we can elaborately initialize $\mW_0$ and highlight its role in the whole. To do this, we consider two ways: (1) Initializing the structure of $\mW_0$. Inspired by the neural collapse solution that maximizes class separation, we can initialize $\mW_0$ as this structure, i.e., $\hat{\vw}_i^\top\hat{\vw}_j=\frac{-1}{C-1}$, $\forall i\neq j$; (2) Increasing the importance of $\mW_0$. A simple strategy is scaling up $\mW_0$, thereby implicitly weakening the importance of $\mH_0$.
However, it is difficult to handle the initialization of features because they are obtained by a complex processing a large dataset, thus we seek to initialize the prototypes in the last layer of the network.

§.§ Refined Decision-makings

Recalling the rule—$\argmax_{c'}\langle\vw_{c'},\vh\rangle+b_{c'}$ that makes decision by selecting the class with the largest logit (where the inner product $\langle \vw_{c'},\vh\rangle$ is dominant), which may not be good to directly use the learned features and prototypes, since learning with the unhinged loss within limited iterations (that means $\zeta_1(t)<\infty$) will introduce some residual $\bm{\Delta}(t)$ caused by gradient descent regardless of the unconstrained case or regularized case.

If we add a perturbation $\Delta$ for all features while adding $s\Delta$ for all prototypes, then the perturbed decision-making will be $\arg\max_{c'}\langle \vw_{c'}+s\Delta,
\vh+\Delta\rangle+b_{c'}$, which may not be equivalent to $\argmax_{c'}\langle\vw_{c'},\vh\rangle+b_{c'}$.

§.§ Adjusted Sample Margin Loss

As aforementioned in <ref> and the proof of Theorem <ref>, we will encounter zero gradients when the cosine similarity $\hat{\vw}_y^\top\hat{\vh}$ is $-1$ or $1$, so we can adjust the loss to avoid the issue by to $\vw_y$ and accelerate convergence:

\begin{equation}
\begin{cases}
L_{\gamma}(\mW\hat{\vh},y) &  \text{if } \hat{\vw}_y^\top\hat{\vh}\ge -1+\varepsilon,\\
-(1+\gamma)(\vw_y+\bm{\delta})^\top\hat{\vh} &  \text{if } \hat{\vw}_y^\top\hat{\vh}<-1+\varepsilon,
\end{cases}
\end{equation}

where $\varepsilon\in(0,1)$ is a hyperparameter and $\bm{\delta}=-\left(1+\frac{\hat{\vw}_y^\top\hat{\vh}\sqrt{1-(1-\varepsilon)^2}}{(1-\varepsilon)\sqrt{1-(\hat{\vw}_y^\top\hat{\vh})^2}}\right)(\vw_y+\hat{\vh}\hat{\vh}^\top \vw_y)$ (performed with a stop-gradient) satisfying $\frac{(\vw_y+\bm{\delta})^\top\hat{\vh}}{\|\vw_y+\bm{\delta}\|_2}=-1+\varepsilon$.

[Arjovsky et al., 2017]
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan, 2017.
URL <https://arxiv.org/abs/1701.07875>.

[Auer et al., 2002]
Peter Auer, Nicolo Cesa-Bianchi, and Claudio Gentile.
Adaptive and self-confident on-line learning algorithms.
Journal of Computer and System Sciences, 640 (1):0 48–75, 2002.

[Bubeck and Sellke, 2021]
Sébastien Bubeck and Mark Sellke.
A universal law of robustness via isoperimetry.
Advances in Neural Information Processing Systems, 34:0 28811–28822, 2021.

[Buda et al., 2018]
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.
A systematic study of the class imbalance problem in convolutional neural networks.
Neural Networks, 106:0 249–259, 2018.

[Cao et al., 2019]
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.
Learning imbalanced datasets with label-distribution-aware margin loss.
Advances in neural information processing systems, 32, 2019.

[Chizat et al., 2019]
Lenaic Chizat, Edouard Oyallon, and Francis Bach.
On lazy training in differentiable programming.
Advances in Neural Information Processing Systems, 32, 2019.

[Cui et al., 2019]
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.
Class-balanced loss based on effective number of samples.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9268–9277, 2019.

[Demirkaya et al., 2020]
Ahmet Demirkaya, Jiasi Chen, and Samet Oymak.
Exploring the role of loss functions in multiclass classification.
In 2020 54th annual conference on information sciences and systems (ciss), pages 1–5. IEEE, 2020.

[Deng et al., 2009]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.

[Duchi et al., 2011]
John Duchi, Elad Hazan, and Yoram Singer.
Adaptive subgradient methods for online learning and stochastic optimization.
Journal of machine learning research, 120 (7), 2011.

[Fang et al., 2021]
Cong Fang, Hanze Dong, and Tong Zhang.
Mathematical models of overparameterized neural networks.
Proceedings of the IEEE, 1090 (5):0 683–703, 2021a.

[Fang et al., 2021]
Cong Fang, Hangfeng He, Qi Long, and Weijie J Su.
Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training.
Proceedings of the National Academy of Sciences, 1180 (43):0 e2103091118, 2021b.

[Galanti et al., 2021]
Tomer Galanti, András György, and Marcus Hutter.
On the role of neural collapse in transfer learning.
In International Conference on Learning Representations, 2021.

[Gao et al., 2023]
Peifeng Gao, Qianqian Xu, Peisong Wen, Huiyang Shao, Zhiyong Yang, and Qingming Huang.
A study of neural collapse phenomenon: Grassmannian frame, symmetry, generalization.
arXiv preprint arXiv:2304.08914, 2023.

[Gunasekar et al., 2018]
Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro.
Implicit bias of gradient descent on linear convolutional networks.
Advances in Neural Information Processing Systems, 31, 2018.

[Han et al., 2022]
XY Han, Vardan Papyan, and David L Donoho.
Neural collapse under mse loss: Proximity to and dynamics on the central path.
In International Conference on Learning Representations, 2022.

[Hardt et al., 2016]
Moritz Hardt, Ben Recht, and Yoram Singer.
Train faster, generalize better: Stability of stochastic gradient descent.
In International conference on machine learning, pages 1225–1234. PMLR, 2016.

[He et al., 2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.

[Hendrycks and Gimpel, 2016]
Dan Hendrycks and Kevin Gimpel.
A baseline for detecting misclassified and out-of-distribution examples in neural networks.
arXiv preprint arXiv:1610.02136, 2016.

[Hui and Belkin, 2021]
Like Hui and Mikhail Belkin.
Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks.
In International Conference on Learning Representations, 2021.

[Hui et al., 2022]
Like Hui, Mikhail Belkin, and Preetum Nakkiran.
Limitations of neural collapse for understanding generalization in deep learning.
arXiv preprint arXiv:2202.08384, 2022.

[Jacot et al., 2018]
Arthur Jacot, Franck Gabriel, and Clément Hongler.
Neural tangent kernel: Convergence and generalization in neural networks.
Advances in neural information processing systems, 31, 2018.

[Ji et al., 2022]
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su.
An unconstrained layer-peeled perspective on neural collapse.
In International Conference on Learning Representations, 2022.

[Ji and Telgarsky, 2019]
Ziwei Ji and Matus Telgarsky.
The implicit bias of gradient descent on nonseparable data.
In Conference on Learning Theory, pages 1772–1798. PMLR, 2019.

[Ji et al., 2020]
Ziwei Ji, Miroslav Dudík, Robert E Schapire, and Matus Telgarsky.
Gradient descent follows the regularization path for general losses.
In Conference on Learning Theory, pages 2109–2136. PMLR, 2020.

[Kasarla et al., 2022]
Tejaswi Kasarla, Gertjan J Burghouts, Max van Spengler, Elise van der Pol, Rita Cucchiara, and Pascal Mettes.
Maximum class separation as inductive bias in one matrix.
arXiv preprint arXiv:2206.08704, 2022.

[Kingma and Ba, 2015]
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
In ICLR, 2015.

[Koltchinskii and Panchenko, 2002]
Vladimir Koltchinskii and Dmitry Panchenko.
Empirical margin distributions and bounding the generalization error of combined classifiers.
The Annals of Statistics, 300 (1):0 1–50, 2002.

[Kornblith et al., 2021]
Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi.
Why do better loss functions lead to less transferable features?
Advances in Neural Information Processing Systems, 34:0 28648–28662, 2021.

[Kothapalli et al., 2022]
Vignesh Kothapalli, Ebrahim Rasromani, and Vasudev Awatramani.
Neural collapse: A review on modelling principles and generalization.
arXiv preprint arXiv:2206.04041, 2022.

[Krizhevsky and Hinton, 2009]
A. Krizhevsky and G. Hinton.
Learning multiple layers of features from tiny images.
Computer Science Department, University of Toronto, Tech. Rep, 1, 01 2009.

[Langer et al., 2021]
Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum.
What do we want from explainable artificial intelligence (xai)? – a stakeholder perspective on xai and a conceptual model guiding interdisciplinary xai research.
Artificial Intelligence, 296:0 103473, 2021.
ISSN 0004-3702.

[LeCun et al., 2015]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.
Deep learning.
nature, 5210 (7553):0 436–444, 2015.

[Lin et al., 2017]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.

[Liu et al., 2020]
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li.
Energy-based out-of-distribution detection.
Advances in Neural Information Processing Systems, 33:0 21464–21475, 2020.

[Liu et al., 2016]
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang.
Large-margin softmax loss for convolutional neural networks.
In International Conference on Machine Learning, pages 507–516. PMLR, 2016.

[Loshchilov and Hutter, 2017]
Ilya Loshchilov and Frank Hutter.
SGDR: Stochastic gradient descent with warm restarts.
In International Conference on Learning Representations, 2017.

[Lu and Steinerberger, 2022]
Jianfeng Lu and Stefan Steinerberger.
Neural collapse under cross-entropy loss.
Applied and Computational Harmonic Analysis, 59:0 224–241, 2022.

[Mei and Montanari, 2021]
Song Mei and Andrea Montanari.
The generalization error of random features regression: Precise asymptotics and the double descent curve.
Communications on Pure and Applied Mathematics, 75, 2021.

[Mixon et al., 2022]
Dustin G Mixon, Hans Parshall, and Jianzong Pi.
Neural collapse with unconstrained features.
Sampling Theory, Signal Processing, and Data Analysis, 200 (2):0 1–13, 2022.

[Nacson et al., 2019]
Mor Shpigel Nacson, J. Lee, Suriya Gunasekar, Nathan Srebro, and Daniel Soudry.
Convergence of gradient descent on separable data.
In AISTATS, 2019.

[Nakkiran et al., 2019]
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.
Deep double descent: Where bigger models and more data hurt.
In International Conference on Learning Representations, 2019.

[Netzer et al., 2011]
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.
Reading digits in natural images with unsupervised feature learning.
In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.

[Neyshabur et al., 2017]
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro.
Exploring generalization in deep learning.
Advances in neural information processing systems, 30, 2017.

[Papyan et al., 2020]
Vardan Papyan, XY Han, and David L Donoho.
Prevalence of neural collapse during the terminal phase of deep learning training.
Proceedings of the National Academy of Sciences, 1170 (40):0 24652–24663, 2020.

[Poggio and Liao, 2020]
Tomaso Poggio and Qianli Liao.
Explicit regularization and implicit bias in deep network classifiers trained with the square loss.
arXiv preprint arXiv:2101.00072, 2020.

[Qi et al., 2021]
Zhongang Qi, Saeed Khorram, and Li Fuxin.
Embedding deep networks into visual explanations.
Artificial Intelligence, 292:0 103435, 2021.
ISSN 0004-3702.

[Sekhari et al., 2021]
Ayush Sekhari, Karthik Sridharan, and Satyen Kale.
Sgd: The role of implicit regularization, batch-size and multiple-epochs.
Advances In Neural Information Processing Systems, 34:0 27422–27433, 2021.

[Shamir, 2021]
Ohad Shamir.
Gradient methods never overfit on separable data.
J. Mach. Learn. Res., 22:0 85:1–85:20, 2021.

[Soudry et al., 2018]
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro.
The implicit bias of gradient descent on separable data.
The Journal of Machine Learning Research, 190 (1):0 2822–2878, 2018.

[Thrampoulidis et al., 2022]
Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia.
Imbalance trouble: Revisiting neural-collapse geometry.
Advances in Neural Information Processing Systems, 35:0 27225–27238, 2022.

[Tian et al., 2021]
Yuandong Tian, Xinlei Chen, and Surya Ganguli.
Understanding self-supervised learning dynamics without contrastive pairs.
In International Conference on Machine Learning, pages 10268–10278. PMLR, 2021.

[Tirer and Bruna, 2022]
Tom Tirer and Joan Bruna.
Extended unconstrained features model for exploring deep neural collapse.
In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 21478–21505. PMLR, 2022.

[Tirer et al., 2023]
Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed.
Perturbation analysis of neural collapse.
In International Conference on Machine Learning, pages 34301–34329. PMLR, 2023.

[Van Rooyen et al., 2015]
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson.
Learning with symmetric label noise: The importance of being unhinged.
Advances in neural information processing systems, 28, 2015.

[Wang et al., 2017]
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille.
Normface: L2 hypersphere embedding for face verification.
In Proceedings of the 25th ACM international conference on Multimedia, pages 1041–1049, 2017.

[Wang et al., 2021]
Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis.
Benign overfitting in multiclass classification: All roads lead to interpolation.
Advances in Neural Information Processing Systems, 34:0 24164–24179, 2021.

[Xu et al., 2023]
Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio.
Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds.
Research, 6:0 0024, 2023.

[Yang and Hu, 2020]
Greg Yang and Edward J Hu.
Feature learning in infinite-width neural networks.
arXiv preprint arXiv:2011.14522, 2020.

[Yang and Hu, 2021]
Greg Yang and Edward J Hu.
Tensor programs iv: Feature learning in infinite-width neural networks.
In International Conference on Machine Learning, pages 11727–11737. PMLR, 2021.

[Yang et al., 2022]
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao.
Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network?
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.

[Zeiler, 2012]
Matthew D Zeiler.
Adadelta: an adaptive learning rate method.
arXiv preprint arXiv:1212.5701, 2012.

[Zhang et al., 2017]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
Understanding deep learning requires rethinking generalization.
In International Conference on Learning Representation, 2017.

[Zhong et al., 2021]
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia.
Improving calibration for long-tailed recognition.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16489–16498, 2021.

[Zhou et al., 2022]
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu.
On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features.
In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27179–27202. PMLR, 17–23 Jul 2022a.

[Zhou et al., 2022]
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu.
Are all losses created equal: A neural collapse perspective.
In Advances in Neural Information Processing Systems, 2022b.

[Zhou et al., 2022]
Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, Xin Gao, and Xiangyang Ji.
Prototype-anchored learning for learning with imperfect annotations.
In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 27245–27267. PMLR, 17–23 Jul 2022c.

[Zhou et al., 2022]
Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, Xin Gao, and Xiangyang Ji.
Learning towards the largest margins.
In International Conference on Learning Representations, 2022d.

[Zhou et al., 2023]
Xiong Zhou, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji.
Asymmetric loss functions for noise-tolerant learning: Theory and applications.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

[Zhu et al., 2021]
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu.
A geometric analysis of neural collapse with unconstrained features.
Advances in Neural Information Processing Systems, 34:0 29820–29834, 2021.