Heavy-Tailed Regularization of Weight Matrices in Deep Neural Networks
Unraveling the reasons behind the remarkable success and exceptional
generalization capabilities of deep neural networks presents a formidable
challenge. Recent insights from random matrix theory, specifically those
concerning the spectral analysis of weight matrices in deep neural networks,
offer valuable clues to address this issue. A key finding indicates that the
generalization performance of a neural network is associated with the degree of
heavy tails in the spectrum of its weight matrices. To capitalize on this
discovery, we introduce a novel regularization technique, termed Heavy-Tailed
Regularization, which explicitly promotes a more heavy-tailed spectrum in the
weight matrix through regularization. Firstly, we employ the Weighted Alpha and
Stable Rank as penalty terms, both of which are differentiable, enabling the
direct calculation of their gradients. To circumvent over-regularization, we
introduce two variations of the penalty function. Then, adopting a Bayesian
statistics perspective and leveraging knowledge from random matrices, we
develop two novel heavy-tailed regularization methods, utilizing Powerlaw
distribution and Frechet distribution as priors for the global spectrum and
maximum eigenvalues, respectively. We empirically show that heavytailed
regularization outperforms conventional regularization techniques in terms of
generalization performance.
Xuanzhe Xiao, Zeng Li, Chuanlong Xie, Fengwei Zhou
# Heavy-Tailed Regularization of Weight Matrices

###### Abstract

Unraveling the reasons behind the remarkable success and exceptional generalization capabilities of deep neural networks presents a formidable challenge. Recent insights from random matrix theory, specifically those concerning the spectral analysis of weight matrices in deep neural networks, offer valuable clues to address this issue. A key finding indicates that the generalization performance of a neural network is associated with the degree of heavy tails in the spectrum of its weight matrices. To capitalize on this discovery, we introduce a novel regularization technique, termed **Heavy-Tailed Regularization**, which explicitly promotes a more heavy-tailed spectrum in the weight matrix through regularization. Firstly, we employ the Weighted Alpha and Stable Rank as penalty terms, both of which are differentiable, enabling the direct calculation of their gradients. To circumvent over-regularization, we introduce two variations of the penalty function. Then, adopting a Bayesian statistics perspective and leveraging knowledge from random matrices, we develop two novel heavy-tailed regularization methods, utilizing Power-law distribution and Frechet distribution as priors for the global spectrum and maximum eigenvalues, respectively. We empirically show that heavy-tailed regularization outperforms conventional regularization techniques in terms of generalization performance.

Keywords:Heavy-Tailed Regularization Deep Neural Network Random Matrix Theory.

## 1 Introduction

Deep neural networks (DNN) have shown remarkable performance in recent years, achieving unprecedented success in various fields such as computer vision, natural language processing, and recommendation systems [6, 10, 11, 25]. However, there is still a lack of clear understanding of how neural networks generalize.
Efforts to construct a generalization framework for DNNs have incorporated various mathematical tools from conventional learning theory [3, 4, 5, 24]. Nevertheless, the majority of these approaches have been found to exhibit certain limitations. For example, the VC dimension and Rademacher complexity have been deemed inadequate in offering a satisfactory explanation for the generalization performance of DNNs [26]. The uniform-convergence-based generalization bounds may fail to elucidate generalization in deep learning due to their vacuous generalization guarantee [20].

One might consider that the weight matrices of a DNN serve as a representation of its generalization capabilities for the following reasons: from a theoretical standpoint, the parameters contained within the weight matrices are intricately connected to the model's output space, input data, optimization algorithm, etc; Additionally, in practical scenarios, access to trained models often comes with limited information regarding training and testing data, which can be attributed to the highly compartmentalized nature of the industry. Recently, Martin and Mahoney [16] introduced a perspective grounded in random matrix theory (RMT) to elucidate the generalization behavior of deep neural networks. They studied the empirical spectral distribution (ESD) of weight matrices in deep neural networks and observed a _5+1 phase of regularization_ - throughout the training process, the ESDs of the weight matrices initially conform well to the Marchenko-Pastur (MP) law, gradually deviate from it, and ultimately approach a Heavy-Tailed (HT) distribution[14, 16]. This regularization phenomenon is referred to as _Implicit Self-Regularization_. Furthermore, this theory suggests that large, well-trained DNN architectures should exhibit Heavy-Tailed Self-Regularization, meaning that the spectra of their weight matrices can be well-fitted by a heavy-tailed distribution. Building on Martin and Mahoney's work, Meng and Yao [19] discovered that the complexity of the classification problem could influence the weight matrices spectra of DNNs. These theories offer a novel perspective for exploring the generalization of DNNs.

In addition to the aforementioned studies, several works have advocated the positive impact of heavy tails of weight matrices on the generalization of neural networks from the perspective of stochastic gradient descent (SGD). Zhou et al. [27] pointed out that the time required for both SGD and Adam to escape sharp minima is negatively related to the heavy-tailedness of gradient noise. They further explained that the superior generalization of SGD compared to Adam in deep learning is due to Adam's gradient calculation being smoothed by the exponential moving average, resulting in lighter gradient noise tails compared to SGD. Hodgkinson et al. [12] presented a similar finding, demonstrating that, within a stochastic optimization problem, multiplicative noise and heavy-tailed stationary behavior enhance the capacity for basin hopping during the exploratory phase of learning, in contrast to additive noise and light-tailed stationary behavior. Simsekli et al. [22] approximated the trajectories of SGD using a Feller process and derived a generalization bound controlled by the Hausdorff dimension, which is associated with the tail properties of the process. Their results suggest that processes with heavier tails should achieve better generalization. Barsbey et al. [2]
argued that the heavy-tailed behavior present in the weight matrices of a neural network contributes to network compressibility, thereby enhancing the network's generalization capabilities. Taken together, these results suggest that the heavy tail of weight matrices is a fundamental factor for the improved generalization of DNNs under SGD.

An intuitive notion arising from these theories is that the presence of heavy tails of weight matrices during DNN training is crucial for achieving favorable generalization performance. However, previous studies provide a limited understanding of how to enhance the heavy-tailed behavior in neural networks. In this study, our focus lies in regularizing DNNs to facilitate more rapid and pronounced heavy-tailed behavior. To this end, we introduce an explicit regularization technique called **Heavy-Tailed Regularization**. We empirically demonstrate that models trained with heavy-tailed regularization display superior generalization performance compared to those trained with conventional methods.

#### 1.0.1 Contribution of this paper.

1. We propose a regularization framework termed _Heavy-Tailed Regularization_. This proposal is motivated by prior research, which has shown that the heavy-tailed behavior of weight matrices in neural networks can improve their generalization capabilities.
2. We develop four distinct heavy-tailed regularization methods, including (a) Weighted Alpha regularization, (b) Stable Rank regularization, (c) Power-law Prior, and (d) Frechet Prior. The first two methods are inspired by existing complexity measures for neural networks, while the latter two are informed by insights from random matrix theory (RMT) and Bayesian statistics.
3. We made comparison with conventional methods on widely used datasets including KMNIST and CIFAR10. Numerical experiments show that the heavy-tailed regularization approaches are efficient and outperform competing conventional regularization methods.

## 2 Heavy-Tailed Regularization

### Definition

Consider a DNN \(f_{\mathbf{W}}:\mathcal{X}\rightarrow\mathcal{Y}\) with \(L\) layers and with weight matrices of its fully connected layers \(\mathbf{W}=\left\{\mathbf{W}_{1},\mathbf{W}_{2},\cdots,\mathbf{W}_{L}\right\}\) and data sample set \(S=\left\{\left(x_{1},y_{1}\right),\left(x_{2},y_{2}\right)\cdots,\left(x_{N},y _{N}\right)\right\}\) with sample size \(N\).

Denote \(l(f(x),y)\) the loss of example \((x,y)\in\mathcal{X}\times\mathcal{Y}\) under model \(f_{\mathbf{W}}\). The optimization problem of the DNN can be viewed as a problem of minimizing the empirical risk with a penalty term:

\[\min_{\mathbf{W}}\quad\mathcal{L}\left(x,y\right)=\frac{1}{N}\sum_{i=1}^{N}l \left(f\left(x_{i}\right),y_{i}\right)+\lambda\sum_{l=1}^{L}p_{l}\left(\mathbf{ W}_{l}\right)\text{,} \tag{1}\]
where \(\lambda\) is a tuning parameter and \(p_{l}\left(\cdot\right)\) is a penalty function on the weight matrices.

Here, we propose a class of regularization methods called **Heavy-Tailed Regularization**, which refers to the regularization methods that are conducive to making the model's weight matrices more heavy-tailed. To achieve this goal, \(p_{l}\left(\cdot\right)\) is supposed to be a complex measure of the model that reflects the degree of the heavy-tailed behavior of the model, and it decreases as the tails get heavier.

To describe the degree of heavy tails of the spectra of the weight matrices, it is critical to estimate the tail index \(\alpha\). In statistics, estimating the tail index is a tricky issue. Denote the data points \(\left\{x_{i},1\leq i\leq n\right\}\) and assume the data come from a heavy-tailed distribution with density function \(p(x)\sim cx^{-\alpha}\), i.e., its p.d.f. is comparable with _power-law_\(x^{-\alpha}\) as \(x\rightarrow\infty\). A general method to estimate the tail index \(\alpha\) is the Hill estimator (HE), which can be used for general power-law settings. If the data is sorted in increasing order, the HE can be written by:

\[\hat{\alpha}=1+\frac{k}{\left(\sum_{i=1}^{k}\ln\frac{x_{n-i+1}}{x_{n-k}} \right)}, \tag{2}\]

where \(k\) is a tuning parameter. There is a trade-off depending on the value of \(k\) between the bias and variance of the estimator. In this study, we use HE with \(k=\frac{n}{2}\) for tail index estimation.

### Weighted Alpha Regularization

Motivated by Martin and Mahoney's work [15, 17, 18], the _Weighted Alpha_ (also called _AlphaHat_) is used in our regularization approach. In their theory, there is a strong linear correlation between the test accuracy and the Weighted Alpha of models. The Weighted Alpha is defined as:

\[\text{Weighted Alpha}(\mathbf{W})=\sum_{l=1}^{L}\alpha_{l}\log\lambda_{\max,l}, \tag{3}\]

where \(\alpha_{l}\) is the tail index of all the positive eigenvalues of \(\mathbf{S}_{l}=\mathbf{W}_{l}^{T}\mathbf{W}_{l}\), and \(\lambda_{\max,l}\) is the maximum eigenvalue of \(\mathbf{S}_{l}\). In Martin and Mahoney's theory, only the performance of Weighted Alpha on different architectures of large-scale, pre-trained, state-of-the-art models was discussed. While we are interested in how this metric changes during the training process of DNNs. Here, we conducted some experiments and obtained evidence that Weighted Alpha is _negatively correlated_ with test accuracy. Thus, the penalty function \(p_{l}(\cdot)\) can be written as

\[p_{l}\left(\mathbf{W}_{l}\right)=\alpha_{l}\cdot\log\lambda_{\max,l}. \tag{4}\]

In fact, we do not need to penalize the weighted alpha throughout the training process. Our goal is to impose a heavy-tailed perturbation in the stochastic optimization, which only requires us to activate the regularization in the early stages or intermittently. Otherwise, it will be over-regularized. On the other
hand, for practical reasons, we can terminate the regularization at some point to avoid high computational costs. Therefore, we provide two additional variants of the penalty function as follows:

1. Decay Weighted Alpha: \[p_{l}\left(\mathbf{W}_{l}\right)=d\left(\left\lfloor e/m\right\rfloor\right) \cdot\alpha_{l}\cdot\log\lambda_{\max,l},\] (5) where \(e\) is the current epoch number, \(m\) is the frequency of decay, and \(d(\cdot)\) is a decreasing function called the _decay function_. The decay function is called _power decay_ when \(d\left(x\right)=x^{-k}I_{\left\{x^{-k}>t\right\}}\) and called _exponential decay_ when \(d\left(x\right)=\exp\left(-kx\right)I_{\left\{\right.\right\}}\exp\left(-kx \right)>t\) for the hyperparameter \(k\) and \(t\). The adoption of this penalty function means that the regularization is activated only in the early epochs and becomes weaker with training.
2. Lower Threshold Weighted Alpha: \[p_{l}\left(\mathbf{W}_{l}\right)=\alpha_{l}\cdot\log\lambda_{\max,l}\cdot I \Bigg{\{}\sum_{l=1}^{L}\alpha_{l}\cdot\log\lambda_{\max,l}\geqslant t\Bigg{\}},\] (6) where \(t\) is a hyperparameter, \(I\left\{\cdot\right\}\) is the indicator function. The adoption of this penalty function means that the regularization is activated only when the Weighted Alpha is above a predetermined lower threshold \(t\), i.e., the model falls short of the degree of the heavy tail we expect.

### Stable Rank Regularization

Stable rank is a classical metric in deep learning, which is defined as

\[\text{stable}\left(\mathbf{W}_{l}\right)=\frac{\left\|\mathbf{W}_{l}\right\| _{F}^{2}}{\left\|\mathbf{W}_{l}\right\|_{2}^{2}}. \tag{7}\]

It has been verified that the stable rank decreases as the training of DNNs [16]. Several recent studies [4, 21] also shows that the generalization error can be upper bounded by \(O\left(\prod_{i}\left\|\mathbf{W}_{i}\right\|_{2}^{2}\sum_{i}\text{stable} \left(\mathbf{W}_{i}\right)\right)\), which implies that a smaller \(\sum_{i}\text{stable}\left(\mathbf{W}_{i}\right)\) leads to a smaller generalization error. In light of this, the penalty function for the stable rank regularization can be written as \(p_{l}\left(\mathbf{W}_{l}\right)=\text{stable}\left(\mathbf{W}_{l}\right)\), and thus the optimization problem can be written as

\[\min_{\mathbf{W}}\quad\mathcal{L}\left(x,y\right)=\sum_{i=1}^{N}l\left(f \left(x_{i}\right),y_{i}\right)+\lambda\sum_{l=1}^{L}\frac{\left\|\mathbf{W} _{l}\right\|_{F}^{2}}{\left\|\mathbf{W}_{l}\right\|_{2}^{2}} \tag{8}\]

Note that \(||\mathbf{W}||_{F}^{2}\) is the sum of square singular values of \(\mathbf{W}\) and \(||\mathbf{W}||_{2}\) is the maximum singular value of \(\mathbf{W}\). Recall the random matrix theory, when the matrix is heavy-tailed, the maximum eigenvalue is far off the global spectrum. Combined with Martin's 5+1 phase transition theory [16], a smaller stable rank of the weight matrix leads to stronger heavy-tailed self-regularization.

Similar to the weighted alpha regularization, in order to avoid over-regularization, we can also add decay and upper threshold to stable rank regularization as follows:
1. Decay Stable Rank: \[p_{l}\left(\mathbf{W}_{l}\right)=d\left(\left\lfloor e/m\right\rfloor\right)\cdot \frac{\left\|\mathbf{W}_{l}\right\|_{F}^{2}}{\left\|\mathbf{W}_{l}\right\|_{2}^ {2}}.\] (9)
2. Lower Threshold Stable Rank: \[p_{l}\left(\mathbf{W}_{l}\right)=\frac{\left\|\mathbf{W}_{l}\right\|_{F}^{2}}{ \left\|\mathbf{W}_{l}\right\|_{2}^{2}}\cdot I\Bigg{\{}\sum_{l=1}^{L}\frac{ \left\|\mathbf{W}_{l}\right\|_{F}^{2}}{\left\|\mathbf{W}_{l}\right\|_{2}^{2}} \geqslant t\Bigg{\}}.\] (10)

### Heavy Tailed Regularization from A Bayesian Perspective

Here, we propose two heavy-tailed regularization methods from a Bayesian perspective. Let us view the deep neural network as a probabilistic model \(P(\mathbf{y}|\mathbf{x},\mathbf{W})\), where \(\mathbf{x}\in\mathcal{X}=\mathbb{R}^{p}\) is the input and \(\mathbf{y}\in\mathcal{Y}\) is the output probability assigned by the neural network. \(\mathbf{W}=\left\{\mathbf{W}_{1},\cdots,\mathbf{W}_{L}\right\}\) is the set of weight matrices of the neural network. Given a training sample set \(S=\left\{\left(x_{1},y_{1}\right),\left(x_{2},y_{2}\right)\cdots,\left(x_{N}, y_{N}\right)\right\}\) with sample size \(N\), a common method for estimating the weights \(\mathbf{W}\) is the maximum likelihood estimation (MLE):

\[\mathbf{W}^{\mathrm{MLE}}=\operatorname*{arg\,max}_{\mathbf{W}}\sum_{i=1}^{N} \log P\left(y_{i}\left|x_{i},\mathbf{W}\right.\right). \tag{11}\]

Specifically, for a multi-classification task, the probabilistic model is usually a multinomial distribution, and then the MLE can be written as

\[\mathbf{W}^{\mathrm{MLE}}=\operatorname*{arg\,max}_{\mathbf{W}}\sum_{i=1}^{N} y_{i}\log f_{\mathbf{W}}\left(x_{i}\right). \tag{12}\]

From a Bayesian perspective, if we want to introduce the heavy-tailed regularization upon the model, we can assign a heavy-tailed prior upon the probabilistic model and then find the maximum a posteriori (MAP) rather than MLE:

\[\mathbf{W}^{\mathrm{MAP}}=\operatorname*{arg\,max}_{\mathbf{W}}\log P\left( \mathbf{y}\left|\mathbf{x},\mathbf{W}\right.\right)+\log P\left(\mathbf{W} \right). \tag{13}\]

Thus, it is important to choose a reasonable prior \(P(\mathbf{W})\) for the weights which can make the weights more heavy-tailed.

Recall the Random Matrix Theory for the heavy-tailed matrices, if a random matrix is heavy-tailed, its limiting spectral distribution (LSD) is supposed to be power-law [7, 8, 9] and its largest eigenvalue is supposed to be Frechet distribution [1, 23]. Therefore, it is easy to consider that prior distribution can be set as Power-law or Frechet distribution when we introduce prior knowledge of the global spectrum or maximum eigenvalue.

Now we introduce the heavy-tailed prior upon the model. Firstly we consider the prior of global spectra of weight matrices. When the weight matrices are
heavy-tailed, the LSD is power-law, so the prior distribution can be set as

\[P\left(\mathbf{W}\right)=\prod_{l=1}^{L}P\left(\mathbf{W}_{l}\right)\propto\prod _{l=1}^{L}\prod_{j=1}^{K_{l}}\lambda_{l,j}^{-\alpha_{l}}, \tag{14}\]

where \(\alpha_{l}\) is the tail index of the power-law of square singular value of weight matrix \(\mathbf{W}_{l}\) in the \(l\)-th layer of the neural network, \(\lambda_{l,j}\) is the \(j\)-th square singular value of \(\mathbf{W}_{l}\). \(K_{l}\) is the number of singular values of \(\mathbf{W}_{l}\) that is considered to be from a heavy-tailed distribution. \(K_{l}\) is a hyperparameter and we choose \(K_{l}\) as half the size of \(\mathbf{W}_{l}\) in our study. Substituting this into (13), we have the following optimization problem:

\[\mathbf{W}^{\mathrm{MAP}}=\operatorname*{arg\,max}_{\mathbf{W}}\sum_{i=1}^{N}y _{i}\log f_{\mathbf{W}}\left(x_{i}\right)-\sum_{l=1}^{L}\sum_{j=1}^{K_{l}} \alpha_{l}\log\lambda_{l,j} \tag{15}\]

Secondly, we consider the prior of the maximum square singular value of the weight matrices. When the weight matrices are heavy-tailed, the distribution of maximum square singular value is Frechet distribution, so the prior distribution can be set as

\[P\left(\mathbf{W}\right)=\prod_{l=1}^{L}P\left(\mathbf{W}_{l}\right)=\prod_{l =1}^{L}\exp\left(-\lambda_{\max,l}^{-\alpha_{l}}\right)\!. \tag{16}\]

where \(\alpha_{l}\) is the tail index of \(\mathbf{W}_{l}\) and \(\lambda_{\max,l}\) is the maximum square singular value. Similarly, substituting this into (13), we have the following optimization problem:

\[\mathbf{W}^{\mathrm{MAP}}=\operatorname*{arg\,max}_{\mathbf{W}}\sum_{i=1}^{N}y _{i}\log f_{\mathbf{W}}\left(x_{i}\right)-\sum_{l=1}^{L}\lambda_{\max,l}^{- \alpha_{l}}. \tag{17}\]

So far we derive two forms of MAP but we have some problems to solve: How to determine the hyperparameters \(\boldsymbol{\alpha}=\{\alpha_{l},1\leq l\leq L\}\), and how to solve this maximization problem. In Empirical Bayes, the hyperparameter is determined by maximizing the marginal likelihood of the data, that is

\[\boldsymbol{\alpha}=\operatorname*{arg\,max}_{\boldsymbol{\alpha}}\log\int P \left(\mathbf{y},\mathbf{W}\left|\mathbf{x},\boldsymbol{\alpha}\right.\right) \mathrm{d}\mathbf{W}. \tag{18}\]

It's apparently impossible since the integral is intractable. According to Mandt et al., [13], the SGD can be seen as a variational expectation maximization (VEM) method for Bayesian inference. The maximization problems in (15) and (17) are equivalent to the following minimization problem:

\[\min_{\mathbf{W}}\quad\mathcal{L}\left(x,y\right)=\frac{1}{N}\sum_{i=1}^{N}l \left(f\left(x_{i}\right),y_{i}\right)+\sum_{l=1}^{L}\sum_{j=1}^{K_{l}}\alpha _{l}\log\lambda_{l,j}, \tag{19}\]

\[\min_{\mathbf{W}}\quad\mathcal{L}\left(x,y\right)=\frac{1}{N}\sum_{i=1}^{N}l \left(f\left(x_{i}\right),y_{i}\right)+\sum_{l=1}^{L}\lambda_{\max,l}^{-\alpha _{l}}, \tag{20}\]
where \(l(f(x),y)=-y\log f(x)\) is the cross entropy loss. The hyperparameters \(\mathbf{\alpha}\) can be optimized when the SGD is seen as a type of VEM algorithm. Instead of the MLE, the Hill estimator is a better choice to estimate the hyperparameters \(\mathbf{\alpha}\). When the tuning parameter is added, the (19) and (20) can be modified as:

\[\min_{\mathbf{W}}\quad\mathcal{L}\left(x,y\right)=\frac{1}{N}\sum_{i=1}^{N}l \left(f\left(x_{i}\right),y_{i}\right)+\mu\sum_{l=1}^{L}\sum_{j=1}^{K_{l}} \hat{\alpha}_{l}\log\lambda_{l,j}, \tag{21}\]

\[\min_{\mathbf{W}}\quad\mathcal{L}\left(x,y\right)=\frac{1}{N}\sum_{i=1}^{N}l \left(f\left(x_{i}\right),y_{i}\right)+\mu\sum_{l=1}^{L}\lambda_{\max,l}^{- \hat{\alpha}_{l}}, \tag{22}\]

where \(\hat{\alpha}_{l}\) is the Hill estimator of the \(l\)-th layer weight matrix.

Note that (21) and (22) are the special cases of (1) when \(p_{l}(\mathbf{W})=\alpha_{l}\cdot\sum_{j}\log\lambda_{j,l}\) and \(p_{l}(\mathbf{W})=\lambda_{\max,l}^{\alpha_{l}}\). Since the penalty terms are similar to the weighted alpha, these regularization terms can be considered as variants of Weighted Alpha. According to the priors used in these regularizations, we call the (21) **Heavy-Tailed Regularization under Power-law Prior** and the (22) **Heavy-Tailed Regularization under Frechet Prior**.

## 3 Experiment

In this section, we experimentally demonstrate the performance of Heavy-Tailed Regularization on multi-classification tasks. To verify the effectiveness of heavy-tailed regularization, we employed the Weighted Alpha Regularization and Stable Rank Regularization with their variants, and the heavy-tailed regularization under the Power-law and Frechet spectral prior. Here all the tail index is replaced by its Hill estimator with \(k=\frac{n}{2}\), where \(n\) is the size of corresponding weight matrix. In our experiment, we used the following training methods to compare with the heavy-tailed regularization approach:

1. Vanilla problem (Base): We considered the original model without any explicit regularization.
2. Weight Decay: We considered the most commonly used explicit regularization in (1) where \(p_{l}(\mathbf{W})=\frac{1}{2}||\mathbf{W}||_{F}^{2}\).
3. Spectral Norm Regularization: We considered another explicit regularization method with regards to the spectrum which penalty function \(p_{l}(\mathbf{W})=\frac{1}{2}||\mathbf{W}||_{2}^{2}\).

All the experiments here are based on mini-batch SGD and learning rate decay. In our experiments, we used the following four settings on the model and dataset:

1. The Three Layer Neural Network (FC3) on KMNIST and CIFAR10.
2. The LeNet5 on CIFAR10.
3. The ResNet18 on CIFAR10.
### Fc3

In the first place, we train the neural network with three hidden layers on the KMNIST and CIFAR10 dataset for 200 epochs. The KMNIST dataset is adapted from Kuzushiji Dataset and is a drop-in replacement for the MNIST dataset. The image size of the KMNIST dataset is 28\(\times\)28. The CIFAR10 dataset consists of 60000 color images whose size is 32\(\times\)32. The CIFAR10 dataset is more complex than the KMNIST dataset so the CIFAR10 dataset is more difficult to be classified correctly. Because the image size of these two datasets varied, we use two types of three-layer neural networks with different sizes for each dataset. For the KMNIST dataset, we use the network with sizes of layers \(\mathbf{n}=[784,128,128,128,10]\); For the CIFAR10 dataset, we use the network with sizes of layers \(\mathbf{n}=[3072,512,256,256,10]\).

The results are shown in Figure 2, 2 and Table 1. The heavy-tailed regularizations all show better accuracies than the vanilla problem both on the KMNIST dataset and the CIFAR10 dataset. The Frechet prior achieves the best test accuracy on the KMNIST dataset, and the stable rank with lower threshold of \(t=15\) achieves the best test accuracy on the CIFAR10 dataset.
### LeNet5

Secondly, we train the LeNet5 on the CIFAR10 dataset for 200 epochs. The LeNet5 is a famous and classical convolutional neural network (CNN) architecture. The results are shown in Figure 3 and Table 2. The heavy-tailed regularization also all shows better accuracy than the vanilla problem both on the CIFAR10 dataset. As shown in the table, the stable rank with \(\beta=0.1\) achieves the best test accuracy.

### ResNet18

Thirdly, we train the ResNet18 on the CIFAR10 dataset for 200 epochs. The ResNet is a CNN architecture which greatly advanced the SOTA in various

\begin{table}
\begin{tabular}{c c c c} \hline \hline Network & Dataset & Method & \(\beta\) & Test accuracy \\ \hline \multirow{8}{*}{LeNet5} & base & & 89.19\(\pm\)0.020 \\  & weight decay & 5.00\(\times 10^{-4}\) & 89.44\(\pm\)0.037 \\  & spectral norm & 0.0001 & 89.27\(\pm\)0.013 \\  & KMNIST & weighted alpha\({}^{1}\) & 5.00\(\times 10^{-5}\) & 89.60\(\pm\)0.011 \\  & stable rank\({}^{2}\) & 1.00\(\times 10^{-4}\) & 89.48\(\pm\)0.008 \\  & Power-law prior & 5.00\(\times 10^{-4}\) & 89.58\(\pm\)0.175 \\  & **Frechet prior** & **2.00\(\times 10^{-5}\)** & **89.64\(\pm\)0.173** \\ \cline{2-4} \multirow{8}{*}{FC3} & base & & 54.97\(\pm\)0.039 \\  & weight decay & 5.00\(\times 10^{-4}\) & 55.56\(\pm\)0.092 \\  & spectral norm & 0.0001 & 55.27\(\pm\)0.003 \\  & weighted alpha\({}^{1}\) & 1.00\(\times 10^{-4}\) & 55.72\(\pm\)0.053 \\  & **stable rank\({}^{2}\)** & **1.00\(\times 10^{-4}\)** & **55.82\(\pm\)0.041** \\  & Power-law prior & 5.00\(\times 10^{-5}\) & 55.44\(\pm\)0.038 \\  & Fréchet prior & 5.00\(\times 10^{-5}\) & 55.45\(\pm\)0.029 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The average (\(\pm\) standard error) of test accuracy of FC3 with different regularization methods on the KMNIST and CIFAR10 dataset

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Network & Dataset & Method & \(\beta\) & Test Accuracy \\ \hline \multirow{8}{*}{LeNet5} & base & & 72.42\(\pm\)0.213 \\  & weight decay & 5.00\(\times 10^{-4}\) & 72.62\(\pm\)0.277 \\  & spectral norm & 0.0001 & 71.98\(\pm\)0.275 \\  & weighted alpha\({}^{1}\) & 0.004 & 72.61\(\pm\)0.300 \\  & **stable rank** & **0.1** & **73.63\(\pm\)0.193** \\  & Power-law prior & 7.00\(\times 10^{-4}\) & 72.61\(\pm\)1.061 \\  & Fréchet prior & 5.00\(\times 10^{-5}\) & 72.58\(\pm\)0.270 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The average (\(\pm\) standard error) of test accuracy of LeNet5 with different regularization methods on the CIFAR10 dataset
computer vision tasks. In this experiment, we add one linear layer with size of 512\(\times\)128 before the linear layer in the origin ResNet18 architecture. The results are shown in Figure 4 and Table 3. As shown in the table, the stable rank with \(\beta=5\times 10^{-4}\) achieves the best test accuracy.
