Sparsified Model Zoo Twins: Investigating Populations of Sparsified
  Neural Network Models
With growing size of Neural Networks (NNs), model sparsification to reduce
the computational cost and memory demand for model inference has become of
vital interest for both research and production. While many sparsification
methods have been proposed and successfully applied on individual models, to
the best of our knowledge their behavior and robustness has not yet been
studied on large populations of models. With this paper, we address that gap by
applying two popular sparsification methods on populations of models (so called
model zoos) to create sparsified versions of the original zoos. We investigate
the performance of these two methods for each zoo, compare sparsification
layer-wise, and analyse agreement between original and sparsified populations.
We find both methods to be very robust with magnitude pruning able outperform
variational dropout with the exception of high sparsification ratios above 80%.
Further, we find sparsified models agree to a high degree with their original
non-sparsified counterpart, and that the performance of original and sparsified
model is highly correlated. Finally, all models of the model zoos and their
sparsified model twins are publicly available: modelzoos.cc.
Dominik Honegger, Konstantin Schürholt, Damian Borth
# Sparsified Model Zoo Twins:

###### Abstract

With growing size of Neural Networks (NNs), model sparsification to reduce the computational cost and memory demand for model inference has become of vital interest for both research and production. While many sparsification methods have been proposed and successfully applied on individual models, to the best of our knowledge their behavior and robustness has not yet been studied on large populations of models. With this paper, we address that gap by applying two popular sparsification methods on populations of models (so called model zoos) to create sparsified versions of the original zoos. We investigate the performance of these two methods for each zoo, compare sparsification layer-wise, and analyse agreement between original and sparsified populations. We find both methods to be very robust with magnitude pruning able outperform variational dropout with the exception of high sparsification ratios above 80%. Further, we find sparsified models agree to a high degree with their original non-sparsified counterpart, and that the performance of original and sparsified model is highly correlated. Finally, all models of the model zoos and their sparsified model twins are publicly available: modelzoos.cc.

Machine Learning, Sparsified Model, Sparsified Model, Sparsified Model, Sparsified Model

## 1 Introduction

In recent years, deep neural networks have gained significant momentum and popularity with the general trend of growing in size. This is mainly due to the observed relationship between model size and performance i.e. larger models tend to have an improved performance over their smaller counterparts as reported by (Kaplan et al., 2020; Tan and Le, 2019; Brock et al., 2018). Unfortunately, the increasing performance results in very high computational and environmental costs for training and inference, as the size of the models continuous to increases (Hoefler et al., 2021; Strubell et al., 2019). As an example, the image classification model CoCa, which currently achieves the highest accuracy (91.0%) on the ImageNet dataset, has 2.1 billion parameters (Yu et al., 2022). Forecasts predict that by 2025 models will exist able to achieve a performance of 95% on the ImageNet object classification but demand as much electricity for training as New York City emits in \(CO_{2}\) in a month (Thompson et al., 2022).

One approach to tackle this issue is to exploit the over-parameterization of large models of neural networks (NNs) to successfully train, but to reduce their size significantly for inference. According to (Hoefler et al., 2021) sparsification of neural network models can achieve reductions of 10-100x without significant losses in performance, even for extremely large models (Frantar and Alistarh, 2023). By pruning parameters after training, it becomes possible to reduce the required computational power for inference, save energy or deploy models on mobile devices, on embedded systems or satellites with limited storage capabilities (Giaffrida et al., 2021; Hoefler et al., 2021; Howard et al., 2017).

Related work has investigated individual methods of sparsification extensively (Blalock et al., 2020; Hoefler et al., 2021) and run large scale studies rigorously evaluating performance differences between different methods (Gale et al., 2019). Contrary to (Gale et al., 2019), who evaluate sparsification on fixed seeds and optimize hyper-parameters for best sparsification, this work evaluates the effect on specification on populations of neural network models (so called "model zoos"). Since neural networks follow a non-convex optimization and are sensitive to hyper-parameter selection, to achieve more robust results in studying sparsity we propose to shift the focus from individual models to populations of neural networks (Schurholt et al., 2021; Schurholt et al., 2022), which are trained according to controlled generating factors i.e., selection of hyper-parameters, seeds, initialization methods. To the best of our knowledge, there are no studies on sparsification on a population of neural network models.

Our contributions:(1) We generate a sparsified version of an available model zoo (Schurholt et al., 2022) using two popular sparsification methods, namely Variational Dropout
(VD) (Molchanov et al., 2017) and Magnitude Pruning (MP) (Han et al., 2015; Strom, 1997) and thus generate a dataset consisting of 33'920 trained and sparsified CNNs with 1'721'600 unique model states representing their sparsification trajectories. (2) We conduct an in-depth analysis and comparison of the sparsified model zoos and the utilized sparsification methods and find that i) both methods perform robustly on all populations, ii) MP outperforms VD except for some very high sparsity ratios and iii) higher sparsity ratios are achieved in larger layers consistently in the populations of the model zoos. Since for each individual model a dense and fully parameterised as well as a sparsified version exists, their relationships can be investigated. Particular attention is paid to investigating how robustly the methods perform on the model zoos trained on different datasets and with varying hyperparameter configurations. (3) As expected, on average with increased sparsification, we observe a performance drop in the populations. However, within the population, we can find individual models, which are less prone to the performance drop (they are sparsification-friendly) or vice verse, are affected stronger by the performance drop (they are sparsification-hard). (4) Furthermore, we examine the weight spaces of the sparsified model zoos by learning hyper-representations of the individual model parameters and are able to show that model properties such as accuracy and sparsity disentangle very well in the latent space and can be predicted from its latent representation.

## 2 Related Work

Sparsification of Neural NetworksModel sparsification has been studied in depth, (Hoefler et al., 2021) provides a survey over the different approaches. Most sparsification approaches can be categorized as 'data-free' or 'training-aware'. Data-free approaches prune models based on the structure of the neural networks. Magnitude Pruning (MP) (Han et al., 2015; Strom, 1997) as the most common representative uses the absolute value or parameters as indicator for importance, but several other approaches have been proposed (Kusupati et al., 2020; Bellec et al., 2017). Training-aware rely on data to identify parameters that have the least impact on the output, based on, e.g., first (Xiao et al., 2019; Ding et al., 2019; Lis et al., 2019; Lee et al., 2018; Srinivas & Babu, 2015) or second order (Hassibi et al., 1993; Cun et al., 1990; Dong et al., 2017; Wang et al., 2019; Theis et al., 2018; Ba et al., 2016; Martens & Grosse, 2015) approximations of the loss functions. Variational methods like Variational Dropout (VD) (Molchanov et al., 2017) explicitly model the distribution of parameters and remove those with high amount of noise.

A large comparative study of sparsification methods found that simple MP can match or outperform more complicated VD on large models (Gale et al., 2019). Similarly, (neuralmagic) offers a selection of sparsified large-scale NNs. Despite the great diversity of sparsification methods and the application of those methods to a diverse range of NNs, sparsification has not yet been applied and studied on a large population of CNNs.

Populations of Neural NetworksRecently, populations of models have become an object of study. Several approaches predict model properties from model features (Yak et al., 2019; Jiang et al., 2019; Corneanu et al., 2020; Martin & Mahoney, 2019; Unterthiner et al., 2020; Eilertsen et al., 2020) or compare models based on their activations (Raghu et al., 2017; Morcos et al., 2018; Nguyen et al., 2020). Other methods leverage zoos for transfer or meta learning (Liu et al., 2019; Shu et al., 2021; Ramesh & Chaudhari, 2022).

Another line of work investigates the weight space of trained models (Lucas et al., 2021; Benton et al., 2021; Ainsworth et al., 2022; Ilharco et al., 2022). Recently,

Figure 1: An overview of the approach. (Left:) A population of neural network models is trained according to latent generating factors such as dataset, architecture and hyperparameter. (Middle:) The given model zoos are sparsified given magnitude pruning and variational dropout. (Right:) Models in the populations are analyzed and entire model zoos are compared with each other and its sparsified counterpart. Additional representation of the zoos are trained to analyse the underlying structure of the sparsified zoos.
several methods have been proposed to learn representations of trained models (Denil et al., 2013; Berardi et al., 2022; Peebles et al., 2022; Ashkenazi et al., 2022; Wang et al., 2023; Navon et al., 2023). (Schurholt et al., 2021; 2022a) proposed a self-supervised approach to learn representations of populations of models, which they dub hyperrepresentations and show to disentangle model properties and be useful to generate new models. Nonetheless, there are only few structured datasets of model zoos. (Gavrikov & Keuper, 2022) publish and analyse a dataset of convolutional filters. (Schurholt et al., 2022c) provide a large dataset of diverse, pre-trained models, which form the basis for our sparsification work.

## 3 Generating Sparsified Model Zoo Twins

To analyse sparsity on populations, we apply two sparsification methods on existing pre-trained model zoos, as outlined in Figures 1 and 2. We select magnitude pruning and variational dropout as representative for data-free and training-aware methods, since they can be applied to small or medium sized CNNs that were already trained to convergence and the methods are appropriate for scaling to large populations of models.

The model zoos of (Schurholt et al., 2022c) serve as a starting point of the sparsification process. We refer to these model zoos as original model zoos. (Schurholt et al., 2022c) establish a setting of varying architectures \(\mathcal{A}\) and hyperparameters \(\lambda\) on different datasets \(\mathcal{D}\) for the generation of their zoos, which we adopt for this work. The zoos were trained on MNIST (LeCun et al., 1998), Fashion-MNIST (Xiao et al., 2017), SVHN (Netzer et al., 2011), USPS (Hull, 1994), CIFAR-10 (Krizhevsky, 2009) and STL-10 (Coates et al., 2011) using a small CNN architecture. To sparsify the model zoos, we apply both sparsification methods to the last state of each model in the zoos. To ensure that the sparsified versions of the CNNs can be compared with their original versions, the generating factors \(\mathcal{A}\), \(\lambda\) and \(\mathcal{D}\) of the original models remain unchanged, except for the learning rate.

Magnitude PruningTo sparsify model zoos with MP, we select several sparsity ratios and sparsify each model in the zoo accordingly. The corresponding fraction of weights with smallest absolute value is set to zero and removed from the set of learnable parameters. We use global unstructured MP and rely on the pytorch implementation (Paganini, 2019). MP generally hurts the performance, so we fine-tune the pruned models on their original dataset to recover for a fixed number of epochs. During fine-tuning we document each epoch by saving the current state dict of the model and report the test accuracy and generalization gap.

Variational DropoutFollowing a similar setup, we apply VD for defined number of epochs on the last state of every model in the model zoos. Following (Gale et al., 2019), we reduce the learning rate compared to the original zoos. After training, the parameters with high variance (\(\alpha\geq 3\)) are removed from the NNs. As VD includes training, we do not fine-tuning the models further. We document each training epoch by saving the state dict as well es the accuracy ratio, test accuracy and generalization gap.

Figure 2: (Left:) Test accuracy over the initial training over a fixed number of epochs in the original MNIST Seed model zoo. (Right:) Test accuracy and sparsity over the 25 epochs of VD sparsification starting from the last epoch of the original training.
## 4 Experiments

This section outlines the experimental setup, evaluation, and analysis of generated sparsified populations of NN models.

### Experimental Setup

We sparsify 14 model zoos with VD and 10 model zoos with MP using the the methods introduced above. In the case of MP, we sparsify each zoo with sparsity levels \([10,20,30,40,50,60,70,80,90]\)%. This is followed by 15 epochs of fine-tuning, in which the pruned weights do not receive a weight update. For our experiments, we use the pruning library of PyTorch (Paganini, 2019). For VD, each weight parameter of the model receives an additional parameter \(\sigma\). Each model is trained for 25 epochs and the learnable parameters \(\mathbf{w}\) and \(\sigma\) are optimized. Both \(\mathbf{w}\) and \(\sigma\) are aggregated in a per-parameter value \(\alpha\). Weights are pruned for \(\alpha>3\). For the implementation of VD, we adapted the fully-connected and convolutional layers of PyTorch based on the code of several previous works (Ryzhikov, 2021; Gale et al., 2019; Molchanov et al., 2017).

Computing Infrastructure:The model zoos were sparsified on nodes with up to 4 CPUs and 64g RAM. Sparsifying a zoo of 1000 models takes 2-3 days. Large and more complex model zoos consisting of roughly 2600 models and greater diversity in terms of hyperparameters may take up to 11 days. Hyper-representations are trained on a GPU of a NVIDIA DGX2 station for up to 12 hours.

### Evaluation

For every model at every state, we record test accuracy, generalization gap and sparsity ratio as fundamental meters to evaluate models. Further, we compute the agreement between original and sparsified models and learn hyper-representations, to evaluate the structure of populations of sparsified models.

Model AgreementAs one measure for evaluation, we compute the pairwise agreement of models within the sparsified and original model zoos. The models agree when both predict the same class given same test data. Per model pair (\(k\) and \(l\)) this is summed up as follows:

\[\kappa_{aggr}=\frac{1}{N}\sum_{i=1}^{N}\lambda_{y_{i}}, \tag{1}\]

for test samples \(i=1,...,N\), where \(\lambda_{y_{i}}\) = 1, if \(y_{i}^{k}\) = \(y_{i}^{l}\) and \(\lambda_{y_{i}}\) = 0 otherwise.

Hyper-Representation LearningFor a deeper understanding of the weight spaces of the model zoos created with VD, we train a attention based auto-encoder (AE) proposed by (Schurholt et al., 2022; \(\mathbf{b}\)). We learn task-agnostic hyper-representations in a self-supervised learning setting. Such representations can provide a proxy to how structured the sparsification process is. Explicitly, it provides insights in how well weights and alphas can be compressed and how well the latent space disentangles model properties like accuracy or sparsity. We adapt the AE to take non-sparsified weights as input and reconstruct to weights and sparsification maps (\(\alpha\)). To improve the reconstruction quality, we introduce a new loss normalisation for the reconstruction of the alpha parameters defined as

\[\mathcal{L}_{MSE}^{\alpha}=\frac{1}{M}\sum_{i=1}^{M}\Big{|}\Big{|}\tanh\bigl{(} \frac{\hat{\alpha}_{i}-t}{r}\bigr{)}-\tanh\bigl{(}\frac{\alpha_{i}-t}{r} \bigr{)}\Big{|}\Big{|}_{2}^{2}, \tag{2}\]

where \(t\) refers to the pruning threshold and \(r\) to the selected range of interest. With that, we force the model to pay attention to the active range around the threshold that determines sparsification. Details of the model are shown in Appendix G and G.

### Experimental Results and Analysis

In this section we analyze the 24 sparsified model zoos. Due to the large scope of the results we only show highlights here and provide full details in Appendix B and C.

Robust Performance on Population Level:As previous work investigated sparsification on single models, or hyperparameter optimization of sparsification, the robustness of sparsification methods on populations has not yet been evaluated. Related work indicates that pruning the excess parameters of a model reduces overfitting and thus improves test accuracy and generalization (Hoefler et al., 2021; Bartoldson et al., 2020). With further increasing sparsity, functional parts of the models are removed and the performance drops. To investigate the performance of the methods we consider the sparsity-ratio, test accuracy and generalization gap (train accuracy - test accuracy) as metrics. In our experiments, magnitude pruning and variational dropout have showed remarkably robust sparsification performance on a population basis, preserving the original accuracy for considerable levels of sparsity. As illustrated in 2, the distribution of the performance metrics of the individual models in the zoo is very consistent and the variation from the top to the worst performing models is low. Although the standard deviation of the performance is higher on model zoos trained on a more sophisticated image dataset (e.g. CIFAR-10), comparable results are achieved. The results furthermore confirm on a population level, that the generalization gap of is lower for models with moderate sparsification levels.
**Larger Layers Achieve Higher Sparsity Ratios** The previous results indicate considerable robustness and consistency in the sparsification results within and between model zoos. To shed further light on sparsification patterns, we investigate the sparsification per layer. Within zoos, the sparsification ratios per layer are remarkably consistent, see Figure 3. Across all zoos, our experiments show that larger layers are more strongly pruned, since a positive relationship between the number of parameters of a layer and the corresponding sparsity ratio exists. This relationship is shown in Figure 4. Detailed results regarding the sparsity per layer can be found in Appendix 4, D and E. This may indicate that the allocation of parameters in the architecture for the original model zoos of (Schurholt et al., 2022c) was not optimal. This is in line with the literature (Hoefler et al., 2021), which states that pruning works particularly well for over-parameterized models.

**Magnitude Pruning outperforms Variational Dropout** Related work found that MP can outperform VD, especially for moderate sparsity ratios (Gale et al., 2019). Our results confirm that on population level. MP outperforms VD for sparsification levels of up to 80% consistently, see Figure 5, Appendix B and C. At higher sparsification levels, MP shows steep drops in performance. VD on some zoos is more stable and thus shows higher performance at higher sparsification levels, justifying the larger parameter count and computational load.

**Agreement between Twin and Original Model Zoos** By analysing the agreement between the original and twin models, we investigate how well the two methods preserve the behavior of the original models, beyond loss or accuracy. The agreement is evaluated for six model zoos at the sparsity ratio 60%, 70%, 80% or 90%. The sparsity ratio was selected such that a favourable accuracy-sparsity trade-off is achieved in variational dropout.

The results show relatively high levels of agreement between 60 and 80 % for both methods. Unsurprisingly, the agreement is higher for overall higher levels of accuracy. Generally, MP achieves higher accuracy and agreement, and appears to therefore preserve the original behavior of the model better. Our results indicate that simple performance metrics like accuracy may be a good proxy to estimate preserved behavior like agreement.

\begin{table}
\begin{tabular}{l|c c c|c c c} \hline \hline  & \multicolumn{3}{c|}{**Magnitude Pruning**} & \multicolumn{3}{c}{**Variational Dropout**} \\
**Model Zoo** & **Accuracy** & **Sparsity** & **Agreement** & **Accuracy** & **Sparsity** & **Agreement** \\ \hline MNIST (s) & 83.7 (13.5) & 80.0 (0.0) & 82.1 (13.0) & 87.6 (1.2) & 78.0 (1.1) & **83.4 (1.4)** \\ USS (s) & 73.8 (1.7) & 90.0 (0.0) & **73.1** (1.3) & 82.3 (1.5) & 85.8 (6.8) & **86.6 (1.0)** \\ SVHN (s) & 70.7 (7.7) & 60.0 (0.0) & **74.9** (1.5) & 62.2 (2.2) & 52.8 (2.9) & 57.5 (5.0) \\ FMNIST (s) & 73.6 (1.3) & 70.0 (0.0) & **70.7 (1.9)** & 69.3 (1.2) & 72.0 (1.0) & 76.2 (2.1) \\ CIFAR-10 (s) & 47.3 (1.2) & 70.0 (0.0) & **78.4 (1.2)** & 40.5 (1.5) & 67.1 (2.8) & 61.0 (2.7) \\ STL-10 (s) & 40.6 (0.8) & 70.0 (0.0) & **85.9 (2.8)** & 35.9 (1.2) & 66.5 (1.3) & 54.4 (2.8) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Agreement overview between original and twin Seed model zoos. The values mean (std) are reported in %. Higher values indicate higher agreement. _Agreement_ denotes Agreement.

Figure 4: Binned scatter plot of all model zoos sparsified with variational dropout at epoch 5, 10, 15 and 20. Epoch 25 is not shown because certain model zoos collapsed at high sparsity ratios and this would distort the plot. The x-axis shows the logarithmized layer size, the y-axis the logarithmized mean sparsity level. The error bar represents the standard deviation of the sparsity.

Figure 5: Mean accuracy per zoo over sparsity for a selection of model zoos sparsified with VD and MP. MP outperforms VD up to sparsity levels of 80%. At higher sparsity, MP performance drops, VD performance is more stable.

Figure 3: Sparsification Frequency per weight for the MNIST zoo at different VD epochs. Within layers, there is remarkable consistency. Further, different layers are pruned in different phases
### Performance of Original and Sparsified Models are Correlated

The sparsification of populations show remarkably robust results, as indicated above. Nonetheless, there is a spread in the performance of sparsified models, see Figure 2. In practice, it is relevant to identify candidates for high performance at high sparsity before sparsification. As first approximation, we compute the correlation between model performance before and after sparsification. We use Pearson's r as well as Kendall's tau coefficients, the former measures the covariance normalized by the product of variances, the latter measures agreement in rank order between the two paired samples. The results show a remarkable high correlation between original and sparsified models, see Table 2. For fixed sparsity levels with MP, the Pearson's r correlation is above 90% with a single exception. The Kendall's tau is similarly high, indicating that the rank order of samples remains preserved to a high degree. Since the sparsification levels of VD zoos are not as consistent, the correlation values are lower, but confirm the finding. Consequently, based on the results of the sparsified populations, the best performing models will likely be the best or among the best sparsified models.

### Disentangled Representations learned from Weight Space

With the revised AE and its novel loss normalisation we are able to not only reconstruct the weight spaces of the CNNs but also the alpha parameters needed for the pruning decision in VD. The results are remarkable in that both accuracy and sparsity are highly predictable and thus disentangled very well in latent space. What is more, both weights as well as alphas are reconstructed well, indicating a high degree of structure in populations of sparsified models. This opens the door for future attempts to zero-shot sparsify models impressing such structure on pre-trained models. The results are shown in Table 3.

## 5 Conclusion

In this work, we have analyzed sparsification on large populations of neural networks. Using magnitude pruning and variational dropout as underlying sparsification approach, we have created ten sparsified model zoo twins representing common computer vision datasets. In total, we have created 23'920 sparsified models with 1'726'000 documented model states. We can confirm, that both approaches - magnitude pruning (MP) and variational dropout (VD) - perform well on population level with respect to sparsification ratio and accuracy. For sparsification ratios below 80%, MP outperforms VD. At higher sparsification ratios, both methods degrade, but VD is more stable. Sparsified models show high agreement with their original models, with no clear preference between the two sparsification approaches. We further find that performance before and after sparsification is highly correlated, indicating that the best performing model is the best candidate for sparsification. The sparsification characteristics per layer within the zoos are surprisingly consistent. This gives rise to learning hyper-representations on sparsified model zoos, which shows to be unexpectedly successful. That indicates that sparsification is highly structured, which may be exploited for zero-shot sparsification.
