Learning Sparse Neural Networks with Identity Layers
The sparsity of Deep Neural Networks is well investigated to maximize the
performance and reduce the size of overparameterized networks as possible.
Existing methods focus on pruning parameters in the training process by using
thresholds and metrics. Meanwhile, feature similarity between different layers
has not been discussed sufficiently before, which could be rigorously proved to
be highly correlated to the network sparsity in this paper. Inspired by
interlayer feature similarity in overparameterized models, we investigate the
intrinsic link between network sparsity and interlayer feature similarity.
Specifically, we prove that reducing interlayer feature similarity based on
Centered Kernel Alignment (CKA) improves the sparsity of the network by using
information bottleneck theory. Applying such theory, we propose a plug-and-play
CKA-based Sparsity Regularization for sparse network training, dubbed CKA-SR,
which utilizes CKA to reduce feature similarity between layers and increase
network sparsity. In other words, layers of our sparse network tend to have
their own identity compared to each other. Experimentally, we plug the proposed
CKA-SR into the training process of sparse network training methods and find
that CKA-SR consistently improves the performance of several State-Of-The-Art
sparse training methods, especially at extremely high sparsity. Code is
included in the supplementary materials.
Mingjian Ni, Guangyao Chen, Xiawu Zheng, Peixi Peng, Li Yuan, Yonghong Tian
# Learning Sparse Neural Networks with Identity Layers

###### Abstract

The sparsity of Deep Neural Networks is well investigated to maximize the performance and reduce the size of overparameterized networks as possible. Existing methods focus on pruning parameters in the training process by using thresholds and metrics. Meanwhile, feature similarity between different layers has not been discussed sufficiently before, which could be rigorously proved to be highly correlated to the network sparsity in this paper. Inspired by interlayer feature similarity in overparameterized models, we investigate the intrinsic link between network sparsity and interlayer feature similarity. Specifically, we prove that reducing interlayer feature similarity based on Centered Kernel Alignment (CKA) improves the sparsity of the network by using information bottleneck theory. Applying such theory, we propose a plug-and-play **CKA**-based **S**parsity **R**egularization for sparse network training, dubbed CKA-SR, which utilizes CKA to reduce feature similarity between layers and increase network sparsity. In other words, layers of our sparse network tend to have their own identity compared to each other. Experimentally, we plug the proposed CKA-SR into the training process of sparse network training methods and find that CKA-SR consistently improves the performance of several State-Of-The-Art sparse training methods, especially at extremely high sparsity. Code is included in the supplementary materials.

Keywords:Network sparsity Inter-layer feature similarity Network compression.

## 1 Introduction

Deep Neural Networks (DNNs) achieve great success on many important tasks, including but not limited to computer vision and natural language processing. Such accurate solutions highly rely on overparameterization, which results in a tremendous waste of resources. A variety of methods are proposed to solve such issues, including model pruning [26, 56, 29] and sparse training [38, 33, 1, 40]. Sparse training aims to train a sparse network from scratch, which reduces both training and inference expenses.
A recent study [15] shows the close relation between overparameterization and interlayer feature similarity (_i.e._ similarity between features of different layers, as shown in Figure 1(a) ). Specifically, overparameterized models possess obviously greater similarity between features of different layers. Concluding from the facts above, we know that both interlayer feature similarity and network sparsity are deeply related to overparameterization. Inspired by this, we utilize the interlayer feature similarity to increase network sparsity and preserve accuracy at a high level, namely by adopting similarity methods to solve sparsity problems.

Following this path, we survey similarity measurements of features, including Canonical Correlation Analysis (CCA) [42; 41; 18] and Centered Kernel Alignment (Linear-CKA and RBF-CKA) [9], etc. Among these measurements, CKA measurement is advanced and robust, for it reliably identifies correspondences between representations in networks with different widths trained from different initializations. Theoretically, CKA measurement has many good properties, including invariance to orthogonal transform and isotropic scaling, and close correlation with mutual information [25]. The advantages of CKA make it possible to propose robust methods to solve sparsity problems with interlayer feature similarity.

To this end, we propose **CKA**-based **S**parsity **R**egularization (CKA-SR) by introducing the CKA measurement into training loss as a regularization term, which is a plug-and-play term and forces the reduction of interlayer feature similarity. Besides, we further prove that the proposed CKA-SR increases the sparsity of the network by using information bottleneck(IB) theory [7; 4; 6; 25]. Specifically, we mathematically prove that our CKA-SR reduces the mutual information between the features of the intermediate and input layer, which is one of the optimization objectives of the information bottleneck method. Further, we prove that reducing the mutual information above is equivalent to increasing network sparsity. By these proofs, we demonstrate the equivalence of reducing interlayer feature similarity and increasing network sparsity, which heuristically investigates the intrinsic link between interlayer feature similarity and network sparsity.

To validate the proposed CKA-SR, we conduct experiments on several advanced sparse training methods, such as Lottery Ticket Hypothesis (LTH) [33], Gradient Signal Preservation (GraSP) [40], Dual Lottery Ticket Hypothesis (DLTH) [38], and Random Sparse Training [1]. Specifically, we introduce our CKA-SR regularization to the training process of these sparse training methods and thus achieve consistent performance gains across these methods. Moreover, we introduce CKA-SR to the training and finetuning process of network pruning methods such as l1-norm filter pruning [3], non-structured weight-level pruning [56], and knapsack channel pruning [2], and thus achieve performance improvements. In short, CKA-SR boosts the performance of sparse training and network pruning methods. Appendix and codes are included in the supplementary materials. See them in [https://anonymous.4open.science/r/Learning-Sparse-Neural-Networks-with-Identity-Layers-9369](https://anonymous.4open.science/r/Learning-Sparse-Neural-Networks-with-Identity-Layers-9369).
Our contributions are four-fold:

* We heuristically investigate the intrinsic link between interlayer feature similarity and network sparsity. To the best of our knowledge, we are the first to find that reducing interlayer feature similarity directly increases network sparsity.
* Theoretically, we prove the equivalence of interlayer feature similarity reduction, interlayer mutual information reduction, and network sparsity increment.
* We proposed Identity Layers Regularization (ILR) with few-shot samples increases network sparsity and weakens overparameterization by explicitly reducing interlayer feature similarity. Specifically, we implement ILR as CKA-SR.
* Experimentally, our CKA-SR regularization term increases network sparsity and improves the performance of multiple sparse training methods and several pruning methods.

## 2 Related Works and Preliminaries

### Centered Kernel Alignment

Here we provide the formalization of Centered Kernel Alignment (CKA). For the feature map \(X\in\mathbb{R}^{n\times p_{1}}\) and feature map \(Y\in\mathbb{R}^{n\times p_{2}}\) (where \(n\) is the number of examples, while \(p_{1}\) and \(p_{2}\) are the number of neurons), we use kernels \(k\) and \(l\) to transform \(X\) and \(Y\) into \(K\) and \(L\) matrices, where the elements are defined as: \(K_{ij}=k(x_{i},x_{j}),L_{ij}=l(y_{i},y_{j})\). Further, the formalization of CKA-based similarity measurement \(\mathcal{F}\) of \(K\) and \(L\) matrices could be formulated as:

\[\mathbf{CKA}(K,L)=\frac{\text{HSIC}(K,L)}{\sqrt{\text{HSIC}(K,K)\text{HSIC}(L,L)}} \tag{1}\]

Figure 1: Reduction of interlayer feature similarity with CKA-SR. (a) Interlayer feature similarity visualization of baseline models. (b) Interlayer feature similarity visualization of models pre-trained with CKA-SR. (c) Comparison of weight distribution between baseline and CKA-SR models.
where HSIC is the empirical estimator of Hilbert-Schmidt Independence Criterion [19]. Then, the formalizations of CKA-based similarity measurement for linear kernel \(k(x,y)=x^{T}y\) is as follows:

\[\mathbf{CKA}_{Linear}(X,Y)=\frac{||Y^{T}X||_{F}^{2}}{||X^{T}X||_{F}||Y^{T}Y||_{F}} \tag{2}\]

### Interlayer feature similarity of overparameterized models

Nguyen _et al._[15] investigate the relationship between overparameterized models and similar feature representations. Specifically, wide ResNets, deep ResNets and ResNets trained on small datasets possess extremely similar feature representations between adjacent layers, named block structure. Then they infer an empirically verified hypothesis that _overparameterized models possess similar feature representations_. Besides, similar observations also appear in ViT [22] based architectures. We may conclude that such block structure is a common problem in different architectures. This prompts us to explore the potential benefits of reducing interlayer feature similarity and learning sparse neural networks with identity layers.

## 3 Methodology

### Sparsity regularization based on Centered Kernel Alignment

As discussed above, the interlayer feature similarity of overparameterized models motivates us to learn sparse neural networks with identity layers. We choose Centered Kernel Alignment (CKA) as the basis of our method, for it's widely applied to measuring feature similarity of different layers. On the other side, the high similarity of layers indicates the overparameterization of Deep Neural Networks. Hence, CKA similarity measurement could be regarded as a scale of overparameterization. This reminds us of directly reducing this measurement to solve overparameterization problems. Even more remarkable, CKA owns many excellent properties, including robustness, invariance to orthogonal transformation, and invariance to scale transformation. These properties make CKA ideal for designing a regularization term to solve overparameterization problems.

Specifically, we add a CKA-based regularization term to the training loss function. For a model with empirical loss (cross-entropy loss) \(\mathcal{L}_{\mathcal{E}}\), the training loss with CKA-SR is formalized as:

\[\mathcal{L}=\mathcal{L}_{\mathcal{E}}+\mathcal{L}_{\mathcal{C}}=\mathcal{L}_{ \mathcal{E}}+\beta\cdot\sum_{s=1}^{S}\sum_{i=0}^{N_{s}}\sum_{j=0,j\neq i}^{N_{ s}}w_{ij}\mathbf{CKA}_{Linear}(X_{i},X_{j}) \tag{3}\]

where \(\mathcal{L}_{\mathcal{C}}\) is CKA-SR and \(\beta\) is the weight of \(\mathcal{L}_{\mathcal{C}}\). \(S\) is the number of stages in the network. For networks with only one stage such as DeiTs, \(N_{s}\) is the total number of layers. And for networks with several stages such as ResNets, \(N_{s}\) is the number
of layers in each stage \(s\). \(w_{ij}\) is the weight of CKA measurement between the \(i^{th}\) and the \(j^{th}\) layer, and it's optional. \(X_{0}\) is the input representation and \(X_{i}\) is the output representation of the \(i^{th}\) layer.

The \(\mathcal{L_{C}}\) part in Eq.(3) forcibly reduces the sum of the pairwise similarity of all layers in the network, _i.e._ forcibly reduces the interlayer similarity of the network.

### Theoretical analysis

#### 3.2.1 Approximate sparsity.

To further explore the relationship between the Frobenius norm of weight matrix and network sparsity, we expand sparsity to approximate sparsity. We define \(\epsilon\)-sparsity (_i.e._, approximate sparsity) of a neural network as follows:

\[S_{\epsilon}=\frac{|\{w|w\in\mathbb{W}\wedge|w|<\epsilon\}|}{|\mathbb{W}|} \tag{4}\]

where \(\epsilon\) is a number close to zero, \(\mathbb{W}\) is the set consisting of all parameters of the network's weight matrix, \(|\mathbb{W}|\) is the total number of parameters, and \(\{w|w\in\mathbb{W}\wedge|w|<\epsilon\}\) is the set consisting of small parameters (_i.e._, parameters with an absolute value smaller then \(\epsilon\)) of the weight matrix.

In Eq. (4), \(S_{\epsilon}\) represents the proportion of network parameters that approach 0. We define this as \(\epsilon\)-sparsity of the network. Further, we prove that \(\epsilon\)-sparsity and sparsity (_i.e._, proportion of network parameters that equal 0) of neural networks are approximately equivalent in practice. Our theory is formulated as Theorem 3.1. See the detailed proof of Theorem 3.1 in the Appendix.

Theorem 3.1: _The \(\epsilon\)-sparsity and the sparsity of neural networks are approximately equivalent._

#### 3.2.2 Information bottleneck.

The information bottleneck (IB) theory proposed by Tishby _et al._[4] is an extension of the rate distortion theory of source compression. This theory shows a trade-off between preserving relevant label information and obtaining efficient compression. Tishby _et al._[6] further research the relationship between information bottleneck theory and deep learning. They interpret the goal of deep learning as an information-theoretic trade-off between compression and prediction. According to the principles of information bottleneck theory, for a neural network \(Y=f(X)\) with input \(X\) and output \(Y\), the best representation of intermediate feature map \(\hat{X}\) captures the relevant features and ignores the irrelevant features (features that have little contribution to the prediction of \(Y\)) at the same time. This process is called "compression". One of its minimization objectives is as follows:

\[L=I(X;\hat{X})-\alpha I(\hat{X};Y)\]
where \(I(X;\hat{X})\) is the mutual information between input \(X\) and intermediate representation \(\hat{X}\), \(I(\hat{X};Y)\) is the mutual information between intermediate representation \(\hat{X}\) and output \(Y\), and \(\alpha\) is a weight parameter for adjusting their proportions.

#### 3.2.2 Minimizing the mutual information.

Firstly, we prove that our CKA-SR is continuous and optimizable in Theorem 3.1, which makes it possible to minimize CKA-SR in machine learning. See the detailed proof of Theorem 3.1 in the Appendix. Then we prove that minimizing CKA-SR minimizes the mutual information \(R=I(X;\hat{X})\) between the intermediate and input representation. Besides, the \(\alpha I(\hat{X};Y)\) part of Eq. (5) is implicitly optimized through the cross entropy loss \(\mathcal{L}_{\mathcal{E}}\). Thus, we prove that our method minimizes the optimization objective in Eq. (5), _i.e._, our CKA-SR method conforms to the principles of information bottleneck theory, and it's beneficial to the representation compression process. Our theory is formulated as Theorem 3.1.

Theorem 3.1: \(\mathcal{L}_{\mathcal{C}}\) _is continuous and optimizable._

Theorem 3.2: _Minimizing \(\mathcal{L}_{\mathcal{C}}\) minimizes the mutual information \(R=I(X;\hat{X})\) between intermediate representation \(\hat{X}\) and input representation \(X\)._

To prove Theorem 3.1, we first review Lemma 1 and Lemma 2 from [25] as follows. Following [25], we assume that \(X\sim\mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}_{X})\) and \(Y\sim\mathcal{N}(\mathbf{0},\boldsymbol{\Sigma}_{Y})\), _i.e._, feature maps \(X\) and \(Y\) follow Gaussian distribution.

Lemma 1: _Minimizing the distance between \(X^{T}Y\) and zero matrix is equivalent to minimizing the mutual information \(I(X;Y)\) between representation \(X\) and \(Y\)._

Lemma 2: _Minimizing \(\mathbf{CKA}_{Linear}(X,Y)\) is equivalent to minimizing \(I(X;Y)\)._

These two lemmas illustrate the relationship between the CKA similarity measurement and information theory. That is, _minimizing the CKA similarity between two feature representations is equivalent to minimizing the mutual information between them_. Based on these two lemmas, we prove Theorem 3.1. See the detailed proof of the two lemmas and Theorem 3.1 in the Appendix.

Theorem 3.1 connects CKA-SR with information bottleneck theory. In short, _minimizing CKA-SR is equivalent to optimizing the optimization objective \(I(X;\hat{X})\) of information bottleneck theory_.

#### 3.2.3 Increasing the sparsity of neural networks.

Further, starting from the information bottleneck theory, we prove that CKA-SR increases the network sparsity, formulated as Theorem 3.1.

Theorem 3.2: _Minimizing \(R=I(X;\hat{X})\Leftrightarrow\) Minimizing \(||W||_{F}^{2}\Leftrightarrow\) Increasing the approximate sparsity of network \(\Leftrightarrow\) Increasing network sparsity._
Proof: According to Theorem 3.3, CKA-SR minimizes \(R=I(X;\hat{X})\) for any \(X\). Further, combining this with Lemma 1, for any \(X\), CKA-SR minimizes the distance between \(X^{T}\hat{X}\) and \(0\) matrix. For a fully-connected layer, we have \(\hat{X}=W^{T}X+b\). Hence, due to the discussions above, we have: for any \(X\), CKA-SR minimizes the distance between \(X^{T}(W^{T}X+b)=X^{T}W^{T}X+X^{T}b\) and \(0\) matrix. We take an orthogonalized \(X\). Due to the unitary invariance (_i.e._, orthogonal invariance in the real number field) of Frobenius norm, \(||W||_{F}^{2}\) equals to \(||X^{T}W^{T}X||_{F}^{2}\). Therefore, minimizing the distance between \(X^{T}W^{T}X+X^{T}b\) and \(0\) matrix is equivalent to minimizing \(||X^{T}W^{T}X||_{F}^{2}\) and further equivalent to minimizing \(||W||_{F}^{2}\).

The above minimization of \(||W||_{F}^{2}\) minimizes the norm of parameter values in weight matrix \(W\), thus making the values more concentrated around \(0\) value. This increases the network's approximate sparsity (defined earlier in this article). Further, according to Theorem 3.1, the approximate sparsity and sparsity are approximately equivalent. So we prove that the above minimization of \(||W||_{F}^{2}\) increases the network sparsity.

Theorem 3.4 connects the optimization objective of information bottleneck theory with network sparsity, thus connecting CKA-SR with network sparsity. In short, _CKA-SR models are more sparse_. We validate this conclusion with our experimental results. Fig.1(c) compares parameter distribution between CKA-SR and baseline models. It's evident that the absolute value of CKA-SR network parameters is more concentrated around \(0\).

## 4 Experiments

### Implementations

#### 4.1.1 Datasets and backbone models.

We validate the effectiveness of our CKA-SR method on image classification, network pruning, and advanced sparse training. We use ResNet18, ResNet20, ResNet32 and ResNet50 [20] as backbones to conduct extensive experiments on CIFAR-10, CIFAR-100 and ImageNet datasets.

#### 4.1.2 Implementations.

We implement our CKA-SR as a regularization of the loss function. We develop a plug-and-play CKA-SR class in PyTorch and plug it into various pre-training and sparse training codes. Because CKA-SR is a regularization of layerwise parameters instead of feature maps themselves, we could utilize few-shot samples of each batch (_generally 8 samples when the batch size is 128 or 256_) to compute CKA-SR. This reduces the computational complexity, thus reducing training expenses. Precisely, we strictly follow the experimental settings of the pruning [2, 56, 3] and sparse training methods [38, 33, 1, 40] and make fair comparisons with them using CKA-SR. The total number of epochs, batch size, optimizer, weight decay, and learning rates all stay the same with the methods to be compared with.
### Pre-Training with CKA-SR

As previously proved, our CKA-SR increases network sparsity. So we validate the performance of CKA-SR in network pruning tasks. We directly prune models pre-trained with CKA-SR on large-scale datasets such as ImageNet. We carry out experiments on several pruning methods and find that our method is effective. As shown in Figure 2, at the same pruning ratio, CKA-SR models outperform baseline models.

#### 4.2.1 Structured pruning.

Following the setting of [3], we perform filter pruning on models pre-trained with CKA-SR without finetuning. Specifically, we prune the filter according to the L1-Norm. The relationship between the pruning ratio and performance is shown in Figure 2(a). When a few filters are pruned, the performance reduction of CKA-SR models is significantly smaller than that of baseline models.

As a State-Of-The-Art method for channel pruning, we perform Knapsack channel pruning [2] on models pre-trained with CKA-SR and achieve higher classification accuracy. The results of Knapsack pruning (w/o finetuning) are shown in Figure 2(b). When a few channels are pruned, the performance reduction of CKA-SR models is much smaller than that of baseline models, which means CKA-SR models possess much higher sparsity.

#### 4.2.2 Non-structured pruning.

We perform non-structured weight-level pruning [56] according to the absolute values of individual weights and compare the performance between baseline ResNet models and pre-trained ResNets with CKA-SR. The relationship between pruning ratio and performance is shown in Figure 2(c). It could be concluded that when massive weights are pruned, the performance reduction of CKA-SR models is smaller than that of baseline models.

Figure 2: Performances of several pruning methods with CKA-SR. The red lines represent CKA-SR models and the blue lines represent baseline models. (a) Performances of L1-norm filter pruning with ResNet18 on ImageNet. (b) Performances of knapsack channel pruning with ResNet50 on ImageNet. (c) Performances of non-structured weight-level pruning with ResNet50 on ImageNet.
Generally, pre-trained models with CKA-SR outperform baseline models in both structured and non-structured pruning methods.

### Sparse network training with CKA-SR

We conduct extensive experiments on several State-Of-The-Art sparse training methods. For fair comparisons, our experiments follow the same settings and backbones of these methods [38, 33, 1, 40]. Note that we conduct experiments on extremely high sparsity (such as 99.8%) settings in GraSP [40], Random sparse training [1], and DLTH [38]. From Table 1, we can find that CKA-SR consistently improves the performance at different levels of sparsity ratios in LTH [33], GraSP [40], Random sparse training [1], and DLTH [38].

#### 4.3.1 Lth.

Lottery Ticket Hypothesis (LTH) [33] is proposed to train a sparse network from scratch, which states that any randomly initialized dense network contains sub-networks achieving similar accuracy to the original network. We plug our CKA-SR into the training process of LTH. We use the code implemented for LTH by [39], adopt ResNet32 as the backbone, and apply sparsity ratios from 0.70 to 0.98 for fair comparisons. The results are given in the first row of Table 1.

#### 4.3.2 GraSP.

Gradient Signal Preservation (GraSP) [40] proposes to preserve the gradient flow through the network during sparse training. We plug our CKA-SR into the sparse training process of GraSP, adopt ResNet32 as the backbone, and apply sparsity ratios from 0.70 to 0.998. The results are given in the second row of Table 1.

#### 4.3.3 Random sparse training.

As one of the newest and State-Of-The-Art sparse training methods, it has been proven that sparse training of randomly initialized networks can also achieve remarkable performances [1]. We plug our CKA-SR into the random sparse training process, adopt ResNet20 as the backbone, and apply sparsity ratios from 0.70 to 0.998. The results are given in the third row of Table 1.

#### 4.3.4 Dlth.

As one of the newest and State-Of-The-Art LTH-based sparse training methods, Dual Lottery Ticket Hypothesis (DLTH) [38] proposes to randomly select subnetworks from a randomly initialized dense network, which can be transformed into a trainable condition and achieve good performance. We apply our CKA-SR to the training process of the DLTH method, adopt ResNet20 as the backbone, and apply sparsity ratios from 0.70 to 0.998. The results are given in the final row of Table 1.

As shown in Table 1, our CKA-SR can be plugged into multiple sparse training methods and improves the model performance consistently. The CKA-SR is effective consistently at different sparse networks, especially at extremely high
sparsity. For GraSP, CKA-SR achieves more than 4.0% of performance improvement at sparsity 99.5% and 6.0% at sparsity 99.8%.

### Ablation studies

#### 4.4.1 Ablation study of regularization term.

Savarese _et al._[34] develop a regularization-based sparse network searching method named Continuous Sparsification. This method introduces \(L_{0}\) Regularization into sparse training. We compare our CKA-SR with \(L_{0}\) Regularization theoretically and experimentally. Theoretically, CKA-SR and \(L_{0}\) regularization regularize networks from different granularity levels. \(L_{0}\) regularization regularizes networks from the individual parameter level, while CKA-SR regularizes networks from the layer level. These regularizations from different granularity levels could work together. Experimentally, we conduct sparse training experiments with ResNet18 on CIFAR-10 using the official code of the CS method. We find that our CKA-SR is able to replace \(L_{0}\) regularization and achieves better performance. Besides, combining CKA-SR and \(L_{0}\) improves performance by 0.4%, demonstrating that our CKA-SR could cooperate with other regularizations. The results are shown in Table 2.

#### 4.4.2 Ablation study of hyperparameter \(\beta\).

We conduct the ablation study of hyperparameter \(\beta\) with Random Sparse Training [1] method on CIFAR-10 dataset. Taking ResNet20 model at a sparsity of 0.95 and adjusting the weight hyperparameter \(\beta\) of our CKA-SR, we get the results shown in Table 3.

\begin{table}
\begin{tabular}{|l|l|c|c|c|c|c|c|} \hline \multirow{2}{*}{Backbone} & \multirow{2}{*}{Method} & \multicolumn{6}{c|}{Sparsity} \\ \cline{3-8}  & & _0.70_ & _0.85_ & _0.90_ & _0.95_ & _0.98_ & _0.998_ \\ \hline \multirow{2}{*}{ResNet32} & LTH[33] & 72.28 & 70.64 & 69.63 & 66.48 & 60.22 & ✗ \\  & +CKA-SR & **72.67** & **71.90** & **70.11** & **67.07** & **60.36** & ✗ \\ \hline \multirow{2}{*}{ResNet32} & GraSP[40] & 71.98 & 70.22 & 69.19 & 65.82 & 59.46 & 12.19 \\  & +CKA-SR & **72.19** & **70.25** & **69.28** & **66.29** & **59.49** & **18.44** \\ \hline \multirow{2}{*}{ResNet20} & Random[1] & 65.42 & 60.37 & 56.96 & 47.27 & 33.74 & 2.95 \\  & +CKA-SR & **65.60** & **60.86** & **57.25** & **48.26** & **34.44** & **3.32** \\ \hline \multirow{2}{*}{ResNet20} & DLTH[38] & 67.63 & 65.33 & 62.90 & 57.33 & 48.08 & 19.32 \\  & +CKA-SR & **67.95** & **65.80** & **63.19** & **57.99** & **49.26** & **20.81** \\ \hline \end{tabular}
\end{table}
Table 1: The accuracy (%) when plugging CKA-SR to different sparse training methods on CIFAR-100 from scratch. (LTH is broken when sparsity ratio is larger than 0.99 due to destruction of the structure.)

\begin{table}
\begin{tabular}{|l|c|c|c|} \hline Settings & CKA-SR Only & \(L_{0}\) Only & CKA-SR+\(L_{0}\) \\ \hline Top1-Acc & 91.63 & 91.56 & 91.92 \\ \hline \end{tabular}
\end{table}
Table 2: Ablation study of regularization terms
We conclude that multiple values of hyperparameter \(\beta\) between 1e-05 and 1e-03 increase the performance of sparse networks. However, when the hyperparameter \(\beta\) becomes too large, it would weaken the succession of information through layers, thus causing a reduction in performance. That is to say, there is a trade-off between the identity of layers and the succession of information through layers. In the view of sparsity, there is a trade-off between high sparsity and ideal performance.

## 5 Conclusion

Our work reveals the relationship between overparameterization, network sparsity, and interlayer feature similarity. We thus propose to use the robust and advanced CKA similarity measurement to solve the overparameterization issue. Specifically, we propose a plug-and-play sparsity regularization named CKA-SR which explicitly reduces interlayer similarity. Theoretically, we reveal the equivalence of reducing interlayer similarity and increasing network sparsity, thus proving the CKA-SR increases network sparsity. Experimentally, our CKA-SR consistently improves the performances of several State-Of-The-Art sparse training methods and several pruning methods. Besides, our CKA-SR outperforms former regularization methods. In the future, considering our limitations of expenses to manually select hyperparameters and calculate loss, we will continue to investigate the cooperation of multiple regularizations in sparse training and reduce the expenses of sparse training.
