HyperSparse Neural Networks: Shifting Exploration to Exploitation
  through Adaptive Regularization
Sparse neural networks are a key factor in developing resource-efficient
machine learning applications. We propose the novel and powerful sparse
learning method Adaptive Regularized Training (ART) to compress dense into
sparse networks. Instead of the commonly used binary mask during training to
reduce the number of model weights, we inherently shrink weights close to zero
in an iterative manner with increasing weight regularization. Our method
compresses the pre-trained model knowledge into the weights of highest
magnitude. Therefore, we introduce a novel regularization loss named
HyperSparse that exploits the highest weights while conserving the ability of
weight exploration. Extensive experiments on CIFAR and TinyImageNet show that
our method leads to notable performance gains compared to other sparsification
methods, especially in extremely high sparsity regimes up to 99.8 percent model
sparsity. Additional investigations provide new insights into the patterns that
are encoded in weights with high magnitudes.
Patrick Glandorf, Timo Kaiser, Bodo Rosenhahn
# HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization

###### Abstract

Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model "knowledge" into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to \(99.8\%\) model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.1

Footnote 1: Code available at [https://github.com/GreenAutoML4FAS/HyperSparse](https://github.com/GreenAutoML4FAS/HyperSparse)

## 1 Introduction

Recent years have shown tremendous progress in the field of machine learning based on the use of neural networks (NN). Alongside the increasing accuracy in nearly all tasks, also the computational complexity of NNs increased, _e.g._, for Transformers [5, 7] or Large Language Models [2]. The complexity causes high energy costs, limits the applicability for cost efficient systems [10], and is counterproductive for the sake of fairness and trustworthiness due to dwindling interpretability [38].

Facing these issues, recent years have also led to a growing community in the field of sparse NNs [12]. The goal is to find small subgraphs (_a.k.a_ sparse NNs) in well performing NNs that have similar or comparable capabilities regarding the main tasks while being significantly less complex and therefore cheaper and potentially better interpretable. Standard methods usually create sparse NNs by obtaining a binary mask that limits the number of used weights in a NN [20, 34, 42]. The most prominent method is _Iterative Magnitude Pruning (IMP)_[16] that is based on the _Lottery Ticket Hypothesis (LTH)_[9]. Assuming that important weights have high magnitudes after training, it trains a dense NN and removes an amount of elements from the mask that correspond to the lowest weights. Afterward, the sparse NN is reinitialized and retrained from scratch. The process is iterated until a sparsity level is reached.

The assumption of magnitude pruning that highest weights in dense NNs encode most important decision rules for a diverse set of classes is problematic, because it is not guaranteed. Removed weights th

Figure 1: Magnitude of weights with their corresponding gradients at different epochs derived from our _HyperSparse_ loss sorted by the weight magnitude. The weights and gradients belong to a ResNet-32 trained on Cifar-100, where the desired pruning rate is \(\kappa=90\%\). The smallest weight \(w_{\kappa}\) that remains after pruning is marked by a dashed line. Note that we added the gradient for the \(\mathcal{L}_{1}\) loss in green.
to the prediction can no longer be reactivated during fine-tuning. In the worst case, a "layer collapse" can prohibit a useful forward propagation [37]. The lack of exploration ability still persists in the more accurate but resource consuming iterative _IMP_ approach.

Reviving the key ideas of _Han et al_. [10] and _Narang et al_. [27] (comparable to [26]), we introduce a lightweight and powerful method called _Adaptive Regularized Training (ART)_ to obtain highly sparse NNs, which implicitly "removes" weights with increasing regularization until a desired sparsity level is reached. _ART_ strongly regularizes the weights before magnitude pruning. First, a dense NN is pre-trained until convergence. In the second stage, the NN is trained with an increasing and weight decaying regularization until the hypothetical magnitude pruned NN performs on par with the dense counterpart. Lastly, we apply magnitude pruning and fine-tune the NN without regularization. Avoiding binary masks in the second stage allows exploration and regularization forces the exploitation of weights that remain in the sparse NN. We introduce the new regularization approach _HyperSparse_ for the second stage that overcomes static regularization like _Lasso_[39] or _Weight Decay_[44] and adapts to the weight magnitude by penalizing small weights. _HyperSparse_ balances the exploration/exploitation tradeoff and thus increases the accuracy while leading to faster convergence in the second stage. The combination of our regularization schedule and _HyperSparse_ improves the classification accuracy and optimization time significantly, especially in high sparsity regimes with up to \(99.8\%\) zero weights. We evaluate our method on CIFAR-10/100 [19] and TinyImageNet [6] with ResNet-32 [11] and VGG-19 [33].

Moreover, we analyze the gradient and weight distribution during regularized training, showing that _HyperSparse_ leads to faster convergence to sparse NNs. The experiments also shows that the claim of [34], that optimal sparse NNs can be obtained via simple weight distribution heuristics, does not hold in general. Finally, we analyze the process of compressing dense NNs into sparse NNs and show that the highest weights in NNs do not encode decision rules for a diverse set of classes with equal priority.

**In summary**, this paper

* introduces _HyperSparse_, a superior adaptive regularization loss that implicitly promotes configurable network sparsity by balancing the exploration and exploitation tradeoff.
* introduces the novel framework _ART_ to obtain sparse networks using regularization with increasing leverage, which improves the optimization time and classification accuracy of sparse neural networks, especially in high sparsity regimes.
* analyzes the continuous process of compressing patterns from dense to sparse neural networks.

## 2 Related Work

Sparse Learningmethods that find binary masks to remove a predefined amount of weights can be categorized as static or dynamic (_e.g_., in [14, 4, 12]). According to [4], in dynamic sparse training _"[...] removed elements [from masks] have chances to be grown back if they potentially benefit to predictions"_ whereas static training incorporates fixed masks.

Static methods are usually based on _Frankle et al_. [9], who introduce _LTH_ and show that well performing sparse NNs in random initialised NNs can be found after dense training via magnitude pruning. The magnitude pruning method is improved by _IMP_[16] that iterates the process. Replacing the time consuming training procedure, methods like _SNIP_[20] or _GraSP_[42] find sparse NNs in randomized dense NNs using a single network prediction and its gradients. To also address the risk of layer collapse during pruning, _SynFlow_[37] additionally conserves the total flow in the network. Contrary to the latter works, _Su et al_. [34] claim that appropriate sparse NNs do not depend on data or weight initialization and provide a general heuristic for the distribution of weights.

Different from static methods, dynamic methods prune and re-activate zero elements in the binary mask. The weights that are reactivated can be selected randomly [25] or determined by the gradients [3, 4, 8]. For example, _RigL_[8] iteratively prunes weights with low magnitude and therefore reactivates weights with highest gradient. Also, modern dynamic methods utilize continuous masks. For example, _Tai et al_. [36] relax the _IMP_ framework by introducing a parameterized softmax to obtain a weighted average between _IMP_ and _Top-KAST_[15]. Similar, [24, 31] relaxes the binary mask and optimizes its \(\mathcal{L}_{0}\)-norm. Another way is to inherently prune the model, _e.g_., by reducing the gradients of weights with small magnitude [32]. Compared to static methods, _Liu et al_. [22, 23] show that dynamic sparse training methods overcomes most static methods by allowing weight exploration.

Another property to distinguish modern sparse learning methods is the complexity during mask generation, _e.g_., as done by _Schwarz et al_. [32]. The more resource efficient _sparse_\(\rightarrow\)_sparse_ methods sustain sparse NNs during training [4, 8, 20, 32, 34, 42, 37], whereas _dense_\(\rightarrow\)_sparse_ methods utilize all parameters before finding the final mask [9, 15, 16, 36, 24, 31].

However, as explained later, our approach belongs to _dense_\(\rightarrow\)_sparse_ methods that inherently reduce the model complexity without masking before magnitude pruning to obtain a static sparse mask for fine-tuning. We want to mention the primary works of _Han et al_. [10], _Narang et al_. [27] and _Molchanov et al_. [26] whose combination is a role model for us. _Han et al_. use \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\) regularization to reduce the number of non-zero elements during
training. Their early framework uses regularization without bells and whistles and has no ability to control the sparsity level. _Narang et al_. and _Molchanov et al_. remove weights in fine-grained portions with an increasing removal-threshold, but do not incorporate weight exploration.

Interpretability and Understandingof machine learning is closely related to sparse learning and is also addressed in this paper. There is an increasing number of works in recent years that utilize sparse learning for other benefits, for example, to find interpretable correlations between feature- and image-space [38] or to visualize inter-class ambiguities [18]. The work of _Paul et al_. [28] gives details about the early learning stage which is crucial, _e.g_., to determine memorization of label noise [17]. They show that most data is not necessary to obtain suitable subnetworks. The general relationship between _LTH_ and generalization is investigated in [30]. _Varma et al_. [35] show that sparse NNs are better suited in data limited and noisy regimes. On the other hand _Hooker et al_. [13] show that sparse NNs have non-trivial impact on ethical bias by investigating which samples are "forgotten" first during network compression. The underlying research question of the latter work is altered to _"Which samples are compressed first?"_ and discussed in this paper.

## 3 Method

Sparsification aims to reduce the number of non-zero weights in a NN. To address this problem, we use a certain schedule for regularization such that small weights converge to zero and our model implicitly becomes sparse. In Sec. 3.1, we formally define the sparsification problem. Then, we present _Adaptive Regularized Training_ (ART) in Sec. 3.2, which iteratively increases the leverage of regularization to maximize the number of close-to-zero weights. Moreover, we introduce our regularization loss _HyperSparse_ in Sec. 3.3 that is integrated in _ART_. It simultaneously allows the exploration of new topologies while exploiting weights of the final sparse subnetwork.

### Preliminaries

We consider a NN \(f(W,x)\) with topology \(f\) and weights \(W\) that is trained to classify images from a dataset \(S=\{(x_{n},y_{n})\}_{n=1}^{N}\), where \(y_{n}\) is the ground truth class to an image sample \(x_{n}\). The training is structured in epochs, which are iterative optimizations of the weights \(W=\{w_{1},\ldots,w_{D}\}\) over all samples in \(S\) to minimize the loss objective \(\mathcal{L}\). The obtained weights after epoch \(e\) are denoted as \(W_{e}\), with \(W_{0}\) denoting the weights before optimization. Furthermore, the classification accuracy of a NN is measured by a rating function \(\psi(W)\).

The goal in sparsification is to reduce the cardinality of \(W\) by removing a pre-defined ratio of weights \(\kappa\), while maximizing \(\psi(W)\). The network is pruned by the Hadamard product \(m\odot W\) of a binary mask \(m\in[0,1]^{D}\) and the model-weights \(W\). The mask is usually created by applying magnitude pruning \(m=\nu(W)\)[3, 4, 9, 16], which is a technique that sets the \(\kappa\)-lowest weights to zero.

### Adaptive Regularized Training (ART)

Regularization losses like the \(L_{1}\)-norm (\(L_{\text{asso}}\)-regression) [39] or \(L_{2}\)-norm [44] are used to prevent overfitting by shrinking the magnitude of weights. We use this effect in _ART_ for sparsification, as weights with low magnitude have low effect on changing the output and thus can be removed with only little impact on \(\psi(W)\).

Regularization during training can be expressed as a mixed loss

\[\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{class}}+\lambda_{\text{init}} \cdot\eta^{e}\cdot\mathcal{L}_{\text{reg}}\, \tag{1}\]

where \(\mathcal{L}_{\text{class}}\) is the classification loss and \(\mathcal{L}_{\text{reg}}\) the regularization loss. The gradient of \(\mathcal{L}_{\text{reg}}\) shrinks a set of weights to approximately zero and creates a inherent sparse network of an undefined pruning rate [39]. Increasing \(\eta\) leverages the regularization \(\mathcal{L}_{reg}\) in an ascending manner, but current approaches use a fixed regularization rate \(\eta=1\)[4, 10, 26].

After unregularized training of a dense NN to convergence, _ART_ employs the standard regularization framework and modifies it by setting \(\eta>1\) and a low initialisation of \(\lambda_{\text{init}}\). Subsequently, the regularization loss \(\mathcal{L}_{\text{reg}}\) has almost no effect on \(\mathcal{L}_{\text{total}}\) in the beginning, but starts to shrink weights without much impact on \(\mathcal{L}_{\text{class}}\) to zero. However, it allows every weight \(w_{i}\) to potentially get a high magnitude such that \(w_{i}\) is shifted into the sparse NN of highest weights (exploration). With increasing regularization, the influence of the gradient \(\frac{d\mathcal{L}_{\text{reg}}}{dw_{i}}\) on \(w_{i}\) increases and is more likely to overcome the gradient \(\frac{d\mathcal{L}_{\text{dim}}}{dw_{i}}\). Regularization impedes proper exploration of small weights by pulling the magnitude to zero. On the other hand, the larger weights
need to be exploited to conserve the classification results. Therefore, our increasing regularization continually shifts the exploration/exploitation tradeoff from exploration to exploitation. The method allows reordering weights to find better topologies, but forces to exploit the highest weights regarding the classification task. Due to the increasing number of weights that are approximately zero, the dense model converges to a inherently sparse model. We stop the regularized training if the NN with best pruned weights \(\psi(\nu(W_{\text{best}})\odot W_{\text{best}})\) has higher accuracy than with the latest unpruned weights \(\psi(W_{e})\) and choose \(W_{\text{best}}\) as our candidate for fine-tuning.

The overall training pipeline is defined as follows:

1. Pre-train dense model until convergence without regularization.
2. Remove weights implicitly using _ART_ as described in algorithm 1.
3. Apply magnitude pruning and fine-tune pruned network until convergence.

_ART_ relaxes the iterative _IMP_ approach that prunes the least important weights over certain iterations. Analogous to the increasing pruning ratio in standard iterative methods, we iteratively increase the amount of weights that are close to zero and thus approximate a binary mask implicitly.

### HyperSparse Regularization

The latter Section 3.2 describes the process of shrinking weights in \(W\) by penalizing with ascending regularization. A drawback of this procedure is that also weights that remain after pruning are penalized by the regularization. This negatively affects the exploitation regarding the main task. Thus, remaining weights should not be penalized. On the other hand, if small weights are strongly penalized, the desired exploration property of dynamic pruning methods to "grow" back these elements is restricted. To address this tradeoff between exploitation and exploration, we introduce the sparsity inducing adaptive regularization loss _HyperSparse_.

Incorporating the _Hyper_bolic Tangent function applied on the magnitude denoted as \(t(\cdot)=\tanh(|\cdot|)\) for simplicity, the _HyperSparse_ loss is defined as

\[\begin{split}\mathcal{L}_{\text{HS}}(W)=&\frac{1}{A }\sum_{i=1}^{|W|}\bigg{(}|w_{i}|\sum_{j=1}^{|W|}t(s\cdot w_{j})\bigg{)}-\sum_{i =1}^{|W|}|w_{i}|\\ &\text{with}\quad A:=\sum_{w\in W}t(s\cdot w)\\ &\text{and}\quad\forall w\in W:\quad\frac{dA}{dw}=0,\end{split} \tag{2}\]

where \(A\) is treated as a pseudo-constant in the gradient computation and \(s\) is an alignment factor that is described later. The regularization penalizes weights depending on the gradient and can vary for different weights. The gradient of _HyperSparse_ with respect to a weight \(w_{i}\) is approximately

\[\begin{split}\frac{d\mathcal{L}_{\text{HS}}(W)}{dw_{i}}=\text{ sign}(w_{i})\cdot\frac{t^{\prime}(s\cdot w_{i})\cdot\sum_{j=1}^{|W|}|w_{j}|}{ \sum_{j=1}^{|W|}t(s\cdot w_{j})},\\ \text{with}\quad w_{i},w_{j}\in W,\quad t^{\prime}(\cdot)\in(0,1 ].\end{split} \tag{3}\]

The derivative \(t^{\prime}=\frac{dt}{dw_{i}}\) converges towards \(1\) for small magnitudes \(|w_{i}|\approx 0\) and towards \(0\) for large magnitudes \(|w_{i}|\gg 0\). Thus, the second term in Eq. (3) is adaptive to the weights and highly penalizes small magnitudes, but is breaking down to zero for large ones. Details for the gradient calculation and analysis can be found in the supplementary material, Sec. D.

The alignment factor \(s\) is mandatory to exploit the aforementioned properties for the sparsification task with a specific pruning rate \(\kappa\). Since \(\mathcal{L}_{\text{HS}}\) is dependent on the weights magnitude, but there is no determinable value range for weights, our loss \(\mathcal{L}_{\text{HS}}\) is not guaranteed to adapt reasonably to a given \(W\). For example, considering a fixed \(s=1\) and all weights in \(W\) are close to zero, the gradient from Eq. (3) results into nearly the same value for every weight. Therefore, we adapt \(s\) to the smallest weight \(|w_{\kappa}|\) that would remain after magnitude pruning, such that \(t^{\prime\prime\prime}(s\cdot w_{\kappa})=0\), which is the point of inflection of \(t^{\prime}\). According to this alignment, the gradients in Eq. (3) of remaining weights \(|w|\geq|w_{\kappa}|\) are shifted closer to 1 and are increased for weights \(|w|\leq|w_{\kappa}|\), while adhering a smooth gradient from remaining to removed weights. Moreover, the denominator in Eq. (3) decreases over time, if more weights in \(W\) are close to zero subsequent to ascending regularization. The gradient for different weight distributions of a NN based on _HyperSparse_ is shown in Fig. 1 and visualizes the described gradient behavior of adaptive weight regularization.

## 4 Experiments

This section presents experiments showing that our proposed method _ART_ outperforms comparable methods, especially in extreme high sparsity regimes. Our experimental setup is described in Sec. 4.1. In the subsequent section, we show that _HyperSparse_ has a large positive impact on the optimization time and classification accuracy. This improvement is explained by analyzes of the tradeoff between exploration and exploitation, the gradient and weight distribution in Sec. 4.3 and 4.4. Finally, we analyze and discuss the compression behaviour during regularized training and derive further insights about highest magnitude weights in Sec. 4.5.

### Experimental Setup

We evaluate _ART_ on the datasets CIFAR-10/100 [19] and TinyImageNet [6] to cover different complexities, given by
a varying number of class labels. Furthermore, we use different model complexities, where ResNet-32 [11] is a simple model with 1.8 M parameters and VGG-19 [33] is a complex model with 20 M parameters. Note that we use the implementation given in [34]. As explained in Sec. 3.2, we group our training in 3 steps. First we train our model for 60 epochs until convergence (step 1), using a constant learning rate of \(0.1\). In the following regularization step, we initialize the regularization with \(\lambda_{\text{init}}=5\cdot 10^{-6}\), \(\eta=1.05\), and use the same learning rate as used in pre-training. The fine-tuning-step (step 3) is similar to [34], as we train for 160 epochs in CIFAR-10/100 and for 300 epochs on Tiny-ImageNet, using a learning rate of \(0.1\) and apply a multiplied decay of 0.1 at 2/4 and 3/4 of the total number of epochs. We also adapt the batch size of 64 and weight-decay of \(10^{-4}\). All experiments are averaged over 5 runs.

We compare our method _ART_ to _SNIP_[20], _Grasp_[42], _SRatio_[34], and _LTH_[9] similar as done in [34, 41]. In addition we evaluate _IMP_[16] and _RigL_[8] as dynamic pruning methods. For comparability, all competitors in our experiments are trained with the same setup as given in the fine-tuning-step. To improve the performance of _RigL_, we extend the training duration by 360 epochs. Further details are given in the supplementary material, Sec. A.

### Sparsity Level

In this section, we compare the performances of _ART_ to other methods on different sparsity levels \(\kappa\in\{90\%,95\%,98\%,99\%,99.5\%,99.8\%\}\), using different datasets and models. To demonstrate the advantages of our novel regularization loss, we additionally substitute _HyperSparse_ with \(\mathcal{L}_{1}\)[39] and \(\mathcal{L}_{2}\)[44]. Table 1 shows the resulting accuracies with standard deviations.

Our method _ART_ combined with _HyperSparse_ outper

\begin{table}
\begin{tabular}{c c c c c c c|c c c c c c} \hline \hline \multirow{2}{*}{**\#Epodes \(\downarrow\)**} & \multicolumn{6}{c}{**ResNet-32 \(\rightarrow\)**} & \multicolumn{6}{c}{**VGG-19 \(\rightarrow\)**} \\ \cline{3-13}  & & \(\kappa\): 90\% & 98\% & 99\% & 99\% & 90\% & 98\% & 99\% & 99\% \\ \hline \multirow{6}{*}{**ART + \(\mathcal{L}_{1}\)**} & 34.23.51 & 68.24.8 & 94.26.80 & 5.80.44 & 24.24.24 & 56.01.30 & \\  & \(C\) & 57.66.46 & 49.62.20 & 11.66.68 & 11.62.63 & 17.58.13 & 17.88.48 & 17.84.37 & 16.28 & 17.88.43 & 17.86 & 0.51.00 \\  & \(C\) & **26.61.15** & 53.22 & 40.32.28 & **37.18.14** & **40.60** & **18.21.42** & **42.16** & **42.16** & **42.16** & **42.16** \\  & \(C\) & 53.22.88 & 77.57.41 & 101.34.96 & 11.22.04 & 47.72.21 & 66.54.46 & \\  & \(C\) & **11.20.74** & **11.48.12** & **11.20.04** & **11.53.25** & **11.86.35** & **11.86.35** & **11.86.35** & **11.86.35** & **11.86.35** & **11.86.35** & **11.86.35** \\  & \(C\) & **39.20.58** & **63.45** & **88.21.48** & **88.24** & **88.24** & **88.24** & **88.24** & **85.34** & **88.58** & **55.88** \\ \cline{2-13}  & \(L\) & 52.01.34 & 61.65 & 31.61.35 & 101.21.08 & 20.81.04 & 43.22.53 & 59.01.80 & 5.26.33 & 11.02.47 & 5.39.53 & 5.90.10 \\  & \(C\) & 56.94.50 & 31.40.32 & 32.51.99 & 89.11.6 & 93.24.63 & 27.66.81 & 11.84.42 & 11.07.44 & 9.66 & **30.32** & **69.21** \\
**ART + \(\mathcal{L}_{2}\)** & **57.66.36** & **57.87.38** & **53.92.14** & **47.97.04** & **40.68.05** & **28.95** & **61.55** & **43.26** & **43.04** & **52.31** & **43.69** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Number of epochs with regularization to obtain the final mask, evaluated for multiple datasets, network topologies, and pruning rates \(\kappa\). It shows that our _HyperSparse_\(\mathcal{L}_{\text{HS}}\) loss reduces the training time significantly.

\begin{table}
\begin{tabular}{c c c c c c c|c c c c c} \hline \hline \multirow{2}{*}{**\#Epodes \(\downarrow\)**} & \multicolumn{6}{c}{**ResNet-32 \(\rightarrow\)**} & \multicolumn{6}{c}{**VGG-19 \(\rightarrow\)**} \\ \cline{3-13}  & & \(\kappa\): 90\% & 98\% & 99\% & 99\% & 90\% & 98\% & 99\% \\ \cline{2-13}  & \(L\) & 34.23.51 & 68.24.8 & 94.26.80 & 5.80.44 & 24.24.24 & 56.01.30 \\  & \(C\) & 57.66.46 & 49.62.20 & 11.66.68 & 11.62.63 & 11.62.35 & 17.85.18 & 17.84.37 & 17.84.37 \\  & \(C\) & **26.61.15** & 53.22.00 & 40.32.22 & **38.61.35** & **40.60** & **18.21.42** & **42.16** & **42.16** \\  & \(C\) & 53.22.88 & 77.57.41 & 101.34.96 & 11.22.04 & 47.72.21 & 66.54.46 \\  & \(C\) & 11.20.74 & 11.48.12 & 11.39.87 & 12.00.14 & 15.32.95 & 16.86.59 & 75.42.12 \\  & \(C\) & **39.20.58** & **63.45** & **88.21.48** & **88.24.24** & **88.24** & **85.36** & **55.88** & **55.88** \\ \cline{2-13}  & \(L\) & 52.01.34 & 61.65.35 & 101.21.08 & 20.81.04 & 43.22.53 & 59.01.80 & 5.26.33 & 11.02.47 & 5.39.13 \\  & \(C\) & 11.20.27.33 & 12.99.81 & 93.24.62 & 19.32.45 & 27.69.81 & 11.08.48 & 44.12.35 & 107.44 & 9.66.95 \\
**ART + \(\mathcal{L}_{2}\)** & **56.97.36** & **57.01.03** & **53.72.03** & **47.19.04** & **40.63.32** & **28.96.66** & **40.95** & **59.67** & **59.67** & **56.72** & **43.08** & **53.23** \\ \cline{2-13}  & \(L\) & **53.22.85** & 77.57.41 & 101.34.96 & **40.63.65** & **11.21.08** & **47.79** & **40.63** & **11.20** & **47.72** & **66.55** & **44.66** \\  & \(C\) & **53.22.85** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **64.79** & **19.90** & **31.00** \\  & \(C\) & **53.28** & **63.45** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** & **63.25** \\ \cline
formes the methods _SNIP_[20], _Grasp_[42], _SRatio_[34], _LTH_[9] and _RigL_[8] on all sparsity levels. Considering the high sparsity of \(99\%\), \(99.5\%\) and \(99.8\%\), all competitors drop drastically in accuracy, even to the minimal classification bound of random prediction for SNIP and _LTH_ using VGG-19. However, _ART_ is able keep high accuracy even on extreme high sparsity levels. In comparison to the regularization losses \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\), our _HyperSparse_ loss achieves higher accuracy in nearly all settings and even minimizes the variance. If we skip the Pre-train-step (step 1) of _ART_, the performance slightly drops. However, _ART_ without pre-training still has good results.

Moreover, we present the number of trained epochs for the regularization phase (step 2) in Tab. 2. In almost all cases, _HyperSparse_ requires less epochs to terminate compared to \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\) and converges faster to a well performing sparse model. As a second aspect, _ART_ dynamically varies the training-length to the sparsity level, model and data complexity. Thus, _ART_ trains longer if higher sparsity is required or the model has more parameters and is more complex like VGG-19. In comparison of the two datasets CIFAR-10 and CIFAR-100, which have the same number of training samples and thus the same number of optimization steps per epoch, _ART_ extends the training-length for the more complex classification problem in CIFAR-100.

_ART_ trains the model for 60 epochs in pre-training (step 1) and 160 epochs in fine-tuning (step 3). Considering the dynamic training-length in step 2, the epochs of _ART_ using \(\mathcal{L}_{\text{HS}}\) sum up from \(226.2\) to \(301.2\) epochs in mean. In comparison, iterative pruning methods are computationally much more expensive, since each model is trained multiple times. For example, IMP [16] requires 860 epochs on CIFAR-10/100 in our experiments.

### Exploration and Exploitation aware Gradient

The training-schedule of _ART_ allows to explore new topologies of sparse networks, while compressing the dense network into the remaining weights that are exploited to minimize the loss \(\mathcal{L}_{\text{class}}\). To reduce the tradeoff between exploration and exploitation, our regularization loss _HyperSparse_ penalizes small weights with a higher regularization and forces the most weights to be close to zero, while preserving the magnitude of weights that remain after pruning. To highlight the beneficial behaviour of _HyperSparse_, this section visualizes and analyzes the gradient. Fig. 1 shows the values and the corresponding gradients of all weights, sorted by the weights magnitude. Note that we only focus on the second step of _ART_, where the regularization is incorporated. Epoch \(0\) represents the first epoch using regularization.In the lower subfigure, we observe that the gradient of _HyperSparse_ with respect to weights larger than \(|w_{\kappa}|\) is closer to 0 than for smaller weights. In comparison, \(\mathcal{L}_{1}\) remains constantly 1 for all weights. The effect of increasing regularization of small weights is stronger for networks with more weights close to zero and therefore amplifies over time, since increasing regularization shrinks the weights magnitude. For example, epoch 40 shows higher gradients for small weights compared to epoch 0, while having more weights with lower magnitude. The pruning-rate \(\kappa\) dependent \(\mathcal{L}_{\text{HS}}\) increases the gradient for small weights \(|w|<|w_{\kappa}|\) over time but conserves the low gradient of larger weights \(|w|>|w_{\kappa}|\) approximately at 0 to favor

Figure 3: Distribution of weights per layer after pruning in a ResNet-32 model that is trained on CIFAR-100 with pruning rate \(\kappa=98\%\). Layer index \(i\) describes the execution order. We group the model in residual blocks (RES), downsampling blocks (DS) and the linear layer (LL). Our method distributes the weights comparable to _IMP_[16], but it has more weights in the downsampling layers.

Figure 2: Intersection of the set of weights with highest magnitude during training and the final mask measured during _ART_ with ResNet-32, CIFAR-100, pruning rate \(\kappa=98\%\) and different regularization losses. Horizontal bars mark the intersection one epoch before pruning and the dashed line at epoch \(60\) indicates the start of regularization. Our _HyperSparse_ loss reduces the optimization time and the high intersection before pruning suggests a higher stability during regularization, which leads to better exploitation.
exploitation. During optimization, the gradient remains smooth and increases slowly for weights that are smaller, but close to \(|w_{\kappa}|\). This favors exploration in the domain of weights close to \(w_{\kappa}\). Therefore, the model becomes inherently sparse and the behaviour shifts continuously from exploration to exploitation.

### Reordering Weights

We use the regularization loss with ascending leverage to find a reasonable set of weights, that remain after pruning. We implicitly do this by shrinking small weights close to zero. During training, weights are reordered and thus can change the membership from the set of pruned to remaining weights, and vice versa. We analyze the reordering procedure in Fig. 2, which shows the intersection of the intermediate and final mask over all epochs, using different regularization losses in _ART_. The model is pre-trained to convergence without regularization for the first 60 epochs (step 1) and with regularization in further epochs (step 2). Fine-tuning is not visualized (step 3). After pre-training, the highest weights only intersects up to \(20\%\) with the final mask obtained by \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\), while _HyperSparse_ leads to an intersection of approximately \(35\%\). This results show _HyperSparse_ changes less parameter while reordering weights, which implies that more structures from the dense model are exploited. It also shows that _HyperSparse_ has a significantly smaller learning duration than \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\). The horizontal bars point to the intersection before last training epoch and show that \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\) only intersect by \(60\%\) and \(50\%\), while _HyperSparse_ is getting very close to the final mask with more than \(90\%\) intersection. This indicates that _HyperSparse_ finds a more stable set of high valued weights and reduces exploration, as the mask has less variation in the final epochs. More results for other training settings are shown in the supplementary material, Sec. B.

Moreover, we analyze the resulting weight distribution of our method and compare it to _IMP_[16] and _SRatio_[34]. Fig. 3 shows the number of remaining weights per layer for ResNet-32 that consists of three scaling levels, which end up with the linear layer (LL). Each scaling level consists of four residual blocks (RES), which are connected by a downsampling-block (DS). The basic topology of _ART_ and _IMP_ looks similar, since both methods show a constant keep-ratio over the residual blocks. Furthermore, _ART_ and _IMP_ use more parameters in downsampling and linear layers. We conclude that these two layer types require more weights and consequently are more important to the model. The higher accuracy discussed earlier suggest that our method exploit these weights better. To show that this results are also obtained on other datasets, models, and sparsity levels, we describe further weight distributions in the supplementary material, Sec. C and show that the number of parameters in the linear layer decreases drastically for a small set of classes in CIFAR-10. Moreover, the compared

\begin{table}
\begin{tabular}{c c|c c c c c c} \hline \multirow{2}{*}{**CP**} & \multicolumn{2}{c}{**Human**} & \multicolumn{5}{c}{**CIFAR-10 Class**} \\  & **Label** & & & & & & \\ \multirow{4}{*}{**Errors**} & 0 & 0.482 & 0.554 & 0.639 & 0.400 & 0.452 & 0.429 \\  & 1 & 0.548 & 0.632 & 0.722 & 0.479 & 0.547 & 0.493 \\  & 2 & 0.653 & 0.741 & 0.808 & 0.588 & 0.676 & 0.710 \\  & 3 & 0.760 & 0.823 & 0.862 & 0.695 & 0.783 & 0.769 \\ \hline \multirow{4}{*}{**Front**} & 0 & 0.166 & 0.499 & 0.494 & 0.567 & 0.580 & 0.524 \\  & 1 & 0.217 & 0.573 & 0.601 & 0.613 & 0.671 & 0.570 \\  & 2 & 0.292 & 0.678 & 0.721 & 0.676 & 0.790 & 0.737 \\  & 3 & 0.392 & 0.772 & 0.796 & 0.710 & 0.841 & 0.808 \\ \hline \multirow{4}{*}{**Front**} & 0 & 0.056 & 0.296 & 0.387 & 0.622 & 0.798 & 0.824 \\  & 1 & 0.061 & 0.317 & 0.411 & 0.647 & 0.835 & 0.835 \\  & 2 & 0.069 & 0.342 & 0.439 & 0.677 & 0.882 & 0.893 \\  & 3 & 0.081 & 0.363 & 0.469 & 0.699 & 0.902 & 0.929 \\ \hline \end{tabular}
\end{table}
Table 3: _Compression Position_ (see Sec. 4.5) for dense NNs (during pre-training) and \(\kappa\) pruned NNs (during regularization) for six CIFAR-10 classes. Samples of a class are split into 4 subsets according to the number of human label errors in CIFAR-N to indicate the difficulty. In sparse networks, different classes are compressed at different times and difficult samples are compressed later. All classes and pruning rates can be found in the supplementary material, Tab. 2.

Figure 4: First 5% CIFAR-10 samples that are compressed into the remaining highest weights after pruning with \(\kappa\in\{0\%,90\%,99.8\%\}\) deduced by the CP-metric. While dense networks learn samples approximately uniform-distributed over classes, the highest weights compress decision rules only for a subset of classes in the early learning stage. Note that we sampled by factor 10 for visualization purposes and ellipses represent the double standard deviation of cluster centers.
method _SRatio_ assumes that suitable sparse networks can be obtained using handcrafted keep-ratios per layer. It has a quadratic decreasing keep-ratio that can be observed in Fig. 3. As shown in Tab. 1, our method _ART_ performs significantly better than _SRatio_ and therefore we deduce that fixed keep-ratios have an adverse effect on performance. Reordering weights during training favors well performing sparse NNs, especially in high sparsity regimes.

### What do networks compress first?

Along with the introduction of _ART_, we are faced with the question of which patterns are compressed first into the large weights that remain after magnitude pruning during regularization. This question is in contrast to Hooker's question _"What Do Compressed Deep Neural Networks Forget?"_[13] and challenges the fundamental assumption of magnitude pruning, which assumes large weights to be most important. In this section, we analyze the chronological order of how samples are compressed and introduce the metric _Compression Position_ (CP) to determine it.

According to our method, regularization starts at epoch \(e_{S}\) and ends at \(e_{E}\) and therefore the weights \(W\) have different states \(\mathcal{W}=\{W_{e}\}_{e=e_{S}}^{e_{E}}\) during training. We measure the individual accuracy over time \(\psi_{1}\) reached by the sparse network for a training sample \((x,y)\in S\), defined by

\[\psi_{1}\!\big{(}x,y,f,\mathcal{W}\big{)}=\frac{\big{|}\big{\{}W_{e}\in \mathcal{W}\mid f(\nu(W_{e})\odot W_{e},x)=y\big{\}}\big{|}}{e_{E}-e_{S}}. \tag{4}\]

After computing the individual accuracy for all samples \(\Psi=\big{\{}\psi_{1}\!\big{(}x_{n},y_{n},f,\mathcal{W}\big{)}\big{\}}_{n=1} ^{N}\) and sorting \(\Psi\) in descending order, the metric \(\text{CP}\big{(}x,y,f,\mathcal{W}\big{)}\) describes the relative position of \(\psi_{1}\!\big{(}x,y,f,\mathcal{W}\big{)}\) in sort\((\Psi)\). In other words, early compressed and correctly classified samples obtain a low CP close to \(0\), and those compressed later closer to \(1\).

We calculate the CP metric for all samples in CIFAR-10 during training of dense, low, and high sparsity NNs. The compression behaviour for dense NNs is measured during the pre-training phase (\(e_{S}=0\) and \(e_{E}=60\)) and for sparse NNs during regularization phase (\(e_{S}=60\) and \(e_{E}=e_{\text{max}}\)).

To show, which samples are compressed first into the remaining highest weights, the \(5\%\) samples with lowest CP are visualized in Fig. 4 in the latent space of the well known _CLIP_ framework [29] mapped by _t-SNE_[40]. As commonly known, the dense model compresses easy samples of all classes in the early stages [17, 21], while the low sparsity model already loses some. In the high sparsity regime no discriminative decision rules are left at beginning of training, and the remaining classes are compressed step by step as the training continues (see supplementary material, Sec. E). In our experiments, we have seen continuously that there is a bias towards the class _deer_. We call this effect _"the deer bias"_, which must be reduced with regularisation. The _deer_ bias suggests that large weights in dense NNs do not encode decision rules for all classes.

To quantify the above results, Tab. 3 shows the average CP for all samples belonging to a specific class. Additionally, we split the class sets into four subsets according to their difficulty. We estimate the difficulty of a sample by counting the human label errors that are made from three human annotators derived from CIFAR-N [43], _e.g_., 2 means that two of three persons mislabeled the sample. The first observation is that the above mentioned separation of classes is confirmed, since CP values are similar in dense NNs, but diverge in sparse NNs. In high sparsity regimes, the _deer_ bias is persistent before first samples of other classes are compressed. The classes _horse_ and _airplane_ are only included at the end of the training. The second observation is, that within a closed set of samples belonging to a class, difficult samples are compressed later. This nature is similar to the training process of dense NNs.

Implementation details and more fine-grained results are available in the supplementary material, Sec. E.

## 5 Conclusion

Our work presents _Adaptive Regularized Training_ (_ART_), a method that utilizes regularization to obtain sparse neural networks. The regularization is amplified continuously and used to shrink most weight magnitudes close to zero. We introduce the novel regularization loss _HyperSparse_ that induces sparsity inherently while maintaining a well balanced tradeoff between exploration of new sparse topologies and exploitation of weights that remain after pruning. Extensive experiments on CIFAR and TinyImageNet show that our novel framework outperforms sparse learning competitors. _HyperSparse_ is superior to standard regularization losses and leads to impressive performance gains in extremely high sparsity regimes and is much faster. Additional investigations provide new insights about the weight distribution during network compression and about patterns that are encoded in high valued weights.

Overall, this work provides new insights into sparse neural networks and helps to develop sustainable machine learning by reducing neural network complexity.

## 6 Acknowledgments

This work was supported by the Federal Ministry of Education and Research (BMBF), Germany under the project AI service center KISSKI (grant no. 01IS22093C), the Deutsche Forschungsgemeinschaft (DFG) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC 2122), and by the Federal Ministry of the Environment, Nature Conservation, Nuclear Safety and Consumer Protection, Germany under the project GreenAutoML4FAS (grant no. 67KI32007A).