Why is the State of Neural Network Pruning so Confusing? On the
  Fairness, Comparison Setup, and Trainability in Network Pruning
The state of neural network pruning has been noticed to be unclear and even
confusing for a while, largely due to "a lack of standardized benchmarks and
metrics" [3]. To standardize benchmarks, first, we need to answer: what kind of
comparison setup is considered fair? This basic yet crucial question has barely
been clarified in the community, unfortunately. Meanwhile, we observe several
papers have used (severely) sub-optimal hyper-parameters in pruning
experiments, while the reason behind them is also elusive. These sub-optimal
hyper-parameters further exacerbate the distorted benchmarks, rendering the
state of neural network pruning even more obscure.
  Two mysteries in pruning represent such a confusing status: the
performance-boosting effect of a larger finetuning learning rate, and the
no-value argument of inheriting pretrained weights in filter pruning.
  In this work, we attempt to explain the confusing state of network pruning by
demystifying the two mysteries. Specifically, (1) we first clarify the fairness
principle in pruning experiments and summarize the widely-used comparison
setups; (2) then we unveil the two pruning mysteries and point out the central
role of network trainability, which has not been well recognized so far; (3)
finally, we conclude the paper and give some concrete suggestions regarding how
to calibrate the pruning benchmarks in the future. Code:
https://github.com/mingsun-tse/why-the-state-of-pruning-so-confusing.
Huan Wang, Can Qin, Yue Bai, Yun Fu
Why is the State of Neural Network Pruning _so Confusing_? On the Fairness, Comparison Setup, and Trainability in Network Pruning

###### Abstract

The state of neural network pruning has been noticed to be unclear and even confusing for a while, largely due to "a lack of standardized benchmarks and metrics" [3]. To standardize benchmarks, first, we need to answer: **what kind of comparison setup is considered fair**?_ This basic yet crucial question has barely been clarified in the community, unfortunately. Meanwhile, we observe several papers have used (severely) sub-optimal hyper-parameters in pruning experiments, while the reason behind them is also elusive. These sub-optimal hyper-parameters further exacerbate the distorted benchmarks, rendering the state of neural network pruning even more obscure.

Two mysteries in pruning represent such a confusing status: the performance-boosting effect of a larger finetuning learning rate, and the no-value argument of inheriting pre-trained weights in filter pruning.

In this work, we attempt to explain the confusing state of network pruning by demystifying the two mysteries. Specifically, (1) we first clarify the fairness principle in pruning experiments and summarize the widely-used comparison setups; (2) then we unveil the two pruning mysteries and point out the central role of **network trainability**, which has not been well recognized so far; (3) finally, we conclude the paper and give some concrete suggestions regarding how to calibrate the pruning benchmarks in the future. Code: [https://github.com/ningsun-tse/why-the-state-of-pruning-so-confusing](https://github.com/ningsun-tse/why-the-state-of-pruning-so-confusing).

## 1 Introduction

The past decade has witnessed the great success of deep learning, empowered by deep neural networks [39, 70]. The success comes at the cost of over-parameterization [5, 12, 27, 32, 33, 37, 63, 64, 68, 71, 77], causing prohibitive model footprint, slow inference/training speed, and rapid-growing energy consumption. Neural network pruning, an old model parameter reduction technique that used to be employed for improving generalization [66, 2, 6], now is mostly used for model size compression or/and speed acceleration [7, 8, 11, 21, 75, 31].

The prevailing pipeline of pruning comprises three steps: 1) **pretraining**: train a dense model; 2) **pruning**: prune the dense model based on certain rules; 3) **finetuning**: retrain the pruned model to regain performance. Most existing research focuses on the second step, seeking better criteria to remove unimportant weights so as to incur as less performance degradation as possible. This 3-step pipeline has been practiced for more than 30 years [2, 6, 40, 58] and is still extensively adopted in today's pruning methods [31, 75].

Despite the long history of network pruning, recent progress seems a little bit unclear. As we quote from the

\begin{table}
\begin{tabular}{l c c} \hline \hline Method & Pruned acc. (\%) & Speedup \\ \hline SFP [29],JCAMT18 & 74.61 & 1.72\(\times\) \\ DCP [90],JCAMT18 & 74.95 & 2.25\(\times\) \\ GAL-0.5 [48],CWP19 & 71.95 & 1.76\(\times\) \\ Taylor-FO [56],CWP19 & 74.50 & 1.82\(\times\) \\ CCP-AC [62],JCAMT19 & 75.32 & 2.18\(\times\) \\ ProvableFP [46],JCAMT20 & 75.21 & 1.43\(\times\) \\ HRank [47],CWP20 & 74.98 & 1.78\(\times\) \\ GReg-1 [81],JCAMT21 & 75.16 & 2.31\(\times\) \\ GReg-2 [81],JCAMT21 & 75.36 & 2.31\(\times\) \\ CC [45],CWP21 & **75.59** & 2.12\(\times\) \\ \(L_{1}\)-norm [43],JCAMT17 (**our relimpl.**) & 75.24 & **2.31\(\times\)** \\ GAL-1 [48],CWP19 & 69.88 & 2.59\(\times\) \\ Factorized [44],CWP21 & 74.55 & 2.33\(\times\) \\ LFPC [28],CWP20 & 74.46 & 2.55\(\times\) \\ GReg-1 [81],JCAMT21 & 74.85 & 2.56\(\times\) \\ GReg-2 [81],JCAMT21 & **74.93** & 2.56\(\times\) \\ CC [45],CWP21 & 74.54 & **2.68\(\times\)** \\ \(L_{1}\)-norm [43],JCAMT17 (**our relimpl.**) & 74.77 & 2.56\(\times\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Top-1 accuracy (%) benchmark of filter pruning with **ResNet50**[27] on **ImageNet**[10]. Simply by using a better fine-tuning LR schedule, we manage to revive a _6-year-ago_ baseline filter pruning method, \(L_{1}\)-_norm pruning_[43], making it _match or beat_ many filter pruning papers published in recent top-tier venues. Note, we achieve this simply by using the common step-decay LR schedule, 90-epoch finetuning, and standard data augmentation, _no_ any advanced training recipe (like cosine annealing LR) used. This paper studies the reasons and lessons behind this pretty confounding benchmark situation in filter pruning.
abstract of a recent paper [3], _"After aggregating results across 81 papers and pruning hundreds of models in controlled conditions, our clearest finding is that the community suffers from a lack of standardized benchmarks and metrics. This deficiency is substantial enough that it is hard to compare pruning techniques to one another or determine how much progress the field has made over the past three decades"_. We also sense this kind of confusion. Particularly, two mysteries in the area represent such confusion:

* **Mystery 1 (M1): The performance-boosting effect of a larger finetuning learning rate (LR)**. It _was_ broadly believed that finetuning the pruned model should use a _small_ LR, _e.g._, in the famous \(L_{1}\)-norm pruning [43], finetuning LR 0.001 is used for the ImageNet experiments. However, many other papers choose a larger LR, _e.g._, 0.01, which delivers significantly better performance than 0.001. For a pretty long time, pruning papers do not officially investigate this critical performance-boosting effect of a larger finetuning LR (although they may have _already used_ it in their experiments; see Tab. 3), until this paper [38]. [38] formally studies how different finetuning LR schedules affect the final performance. They find random pruning and magnitude-pruning (the two most basic pruning methods) armed with a good finetuning LR schedule (CLR, or Cyclic Learning Rate Restarting) can counter-intuitively rival or even surpass other more sophisticated pruning algorithms. Unfortunately, they do not give explanations for this phenomenon except for calling upon comparing pruning algorithms in the same retraining configurations.
* **Mystery 2 (M2): The value of network pruning**. The value of network pruning seems unquestionable given the development history of over 30 years. However, two papers [9, 51] empirically find that training the pruned model from scratch can match pruning a pre-trained model, thus radically challenging the necessity of pretraining a big model first in the conventional 3-step pruning pipeline.

Tab. 1 shows a concrete example that M1 makes the pruning benchmarks unclear. After using an improved finetuning LR schedule (see Tab. 3), we make \(L_{1}\)-norm pruning [43], which is broadly considered the _most basic baseline_ approach, _match or beat_ many top-performing pruning methods published in top-tier conferences in the past several years. Such a situation really bewilders us, especially for those not in this area trying to borrow the most advanced pruning methods in their projects - _"Has the area really developed in the past several years?"_ they might question.

This paper is meant to unveil these mysteries. Over this process, we hope to offer some helpful thoughts about why we have run into the current chaotic benchmark situation and how we can avoid it in the future.

Specifically, when we try to unveil the mysteries, only to find there are various comparison setups enforced in the area, many of the conclusions actually _hinge on which comparison setup is used_. To decide which comparison setup is more trustworthy, we have to be clear with the _fairness principle_ in pruning experiments first (_i.e._, what kind of comparison setup is considered fair?). After we sort out the fairness principle and comparison setups, we empirically examine the two mysteries and find M2 reduces to M1. When examining M1, we find the role of _network trainability_ in network pruning, through which we can easily explain M1.

Therefore, our investigation path and the contributions in this paper can be summarized as follows.

* We first clarify the fairness principle in pruning experiments and summarize the outstanding comparison setups (Sec. 3), which have been unclear in the literature for a long time.
* Then, we start to unveil M1[38] and M2[51]. As we shall show (Sec. 4), the conclusion of M2 actually varies up to which comparisons setup is used: If a larger finetuning LR is allowed to be used, the no-value-of-pruning argument cannot hold; otherwise, it mostly holds. Thus, to unveil M2, we have to unveil M1 first.
* to our best knowledge, we are the _first_ to clarify this mystery in the area.
* Finally, we summarize the major reasons that have led to the confusing benchmark situation of network pruning up to now, and give some concrete suggestions about how to avoid it in the future (Sec. 6).

## 2 Prerequisites

### Taxonomy of Pruning and Related Work

A pruning algorithm _has and only has_ five exclusive aspects to be specified: (1) Base model (when to prune): is pruning conducted on a pretrained model or a random model? (2) Sparsity granularity (what to prune): what is the smallest weight group in pruning? (3) Pruning ratio (how many to prune): how many weights are to be pruned? (4) Pruning criterion (by what to prune): what measure is used to select the important weights (_i.e._, those to be kept) _vs._ the unimportant weights (_i.e._, those to be pruned)? (5) Pruning schedule (how to schedule): How is the sparsity scheduled over the pruning process?
These five mutually orthogonal questions can shatter most (if not all) pruning papers in the literature. Researchers typically use (1), (2), and (4) to classify different pruning methods. We give the major backgrounds in these three axes.

**Pruning after training _vs._ pruning at initialization**. Pruning has been mostly conducted on a _pretrained_ model over the past 30 years, which is thus called _pruning after training_ or _post-training pruning_. This fashion has been the unquestioned norm until (at least) two papers in 2019, SNIP [42] and LTH [18]. They argue pruning can be conducted on _randomly initialized_ models and can achieve promising performance (allegedly matching the dense counterpart), too. This new fashion of pruning is called _pruning at initialization_ (PaI). Existing PaI approaches mainly include [20, 41, 42, 65, 79] and the series of lottery ticket hypothesis [18, 19]. PaI is not very relevant to this work because the benchmarking chaos and the mysteries are mostly discussed in the PaT context, so here we would not discuss in length the specific PaI techniques. Interested readers may refer to [82] for a comprehensive summary.

**Sparsity structure: structured pruning _vs._ unstructured pruning.** If the smallest weight group in pruning is a single weight element, this kind of pruning is called unstructured (or fine-grained) pruning [23, 18, 24], because the resulting zero-weight (_i.e._, pruned-weight) locations are typically irregular (if no extra regularization is enforced). If the smallest weight group in pruning presents some structure, this kind of pruning is called structured (or structural/coarse-grained) pruning [30, 43, 54, 84]. In the area, structured pruning typically narrowly refers to filter pruning or channel pruning if not explained otherwise. Structured pruning benefits more acceleration because the regular sparsity pattern is more hardware-friendly; meanwhile, the regularity imposes more constraints on the network, so given the same sparsity level, structured pruning typically underperforms unstructured pruning.

Note, the definition of "structured" sparsity is severely hardware-dependent and thus can vary as the hardware condition changes. _E.g._, the _N:M sparsity1_ pioneered by NVIDIA Ampere architecture was considered as unstructured sparsity, but since NVIDIA has launched new library support to exploit such kind of sparsity for acceleration, now it can be considered as structured sparsity (called fine-grained structured sparsity [59, 88]), too.

Footnote 1: [https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/](https://developer.nvidia.com/blog/accelerating-inference-with-sparsity-using-ampere-and-tensorrt/)

This paper focuses on _filter pruning_ for now because the two aforementioned mysteries (the effect of finetuning LR and the value of network pruning) are mainly discussed in this context.

**Pruning criterion: Importance-based _vs._ regularization based**. The former prunes weights based on some established importance criteria, such as magnitude (for unstructured pruning) [23, 24] or \(L_{1}\)-norm (for filter pruning) [43], saliency based on 2nd-order gradients (_e.g._, Hessian or Fisher) [76, 72, 40, 78, 25]. The latter adds a penalty term to the objective function, drives unimportant weights toward zero, then removes those with the smallest magnitude. Two notable points: (1) Even in a regularization-based pruning method, after the regularization process, the weights are still removed by a certain importance (typically magnitude). Namely, regularization-based pruning inherently embeds importance-based pruning. (2) The two paradigms are not exclusive; they can be employed simultaneously. _E.g._, [83, 13, 81] select unimportant weights by magnitude while also employing the regularization to penalize weights.

Finally, for more comprehensive literature on network pruning, we refer interested readers to several surveys: an early one [66], some recent surveys of pruning alone [31, 3, 21] or pruning as a sub-topic under the general umbrella of model compression and acceleration [7, 8, 11, 75].

**Most relevant papers**. The most relevant works to this paper are

* [3], which is the first to systematically report the frustrating state of network pruning benchmarks, identify some causes (such as the lack of standard benchmarks and metrics in pruning), and give concrete remedies. However, they do not go deeper and analyze why the standard benchmarks are hard to achieve. Our paper succeeds [3] and will elaborate more on this aspect and point out the central role of trainability within.
* [38], which officially reports the performance-boosting effect of a larger finetuning LR and calls upon comparing pruning algorithms under the same retraining configurations. [38] is actually motivated by [67], which proposes _LR rewinding vs._ the _weight rewinding_[19] proposed for finding winning ticket on large-scale networks and datasets. [67] virtually takes advantage of the performance-boosting effect of a larger finetuning LR, yet we do not know if they were aware at that point that the performance boosting is not because of the magic rewound LR schedule but simply because of a larger finetuning LR (_i.e._, any appropriately larger LR would do, even not a rewound one), as later clarified by [38]; so we tentatively consider [38] as the _first_ work to systematically report the performance-boosting effect of a larger finetuning LR.
* [51], which brings forward the argument regarding the value of network filter pruning against scratch training. We will re-evaluate the major claim (scratch training
can match filter pruning) of this paper under our more strictly controlled comparison setups.

### Terminology Clarification

First, we make some critical concepts clear to lay down the common ground for discussion. Although they are pretty simple concepts, misinterpreting them will twist our discussions.

* _Pruning pipeline_ **vs.**_pruning method_ **vs.**_sparsifying action_. Some papers refer to _pruning_ as the whole pruning algorithm, _i.e_., the 2nd step in the pruning pipeline; while others may mean a pruning paper or the instant sparsifying action. We realize such a vague conception definition is one reason causing confusion, so we make them exact here. We use _pruning pipeline_ to mean all the three steps in a pruning paper. We can consider _pruning pipeline_ to be interchangeable with a _pruning paper_. Then, we use _pruning method = pruning algorithm_ to mean the 2nd step of the pruning pipeline. Finally, the instant pruning action (_i.e_., rezoing out weights or physically taking away weights from a network) is referred to as _sparsifying action_. To summarize, in terms of concept scope, pruning paper = pruning pipeline = pruning \(>\) pruning method = pruning algorithm \(>\) sparsifying action2. Footnote 2: The notation “pruning paper = pruning pipeline \(>\) pruning method” means, in this paper we consider _pruning paper_ interchangeable with _pruning pipeline_, which includes _pruning method_ as one part.
* _Training from scratch_. _Training from scratch_ = _scratch training_, means to train a randomly initialized model to convergence. Scratch training of a _pruned model_ means, we already know the network architecture of the pruned model; the weights are randomly initialized; train this network from scratch using _the same_ training recipe as training the dense model. Notably, for filter pruning, when the architecture of the pruned model is known, the model should be implemented as a small-dense model, _not a large-sparse model_ (with structural masks). The reason is, the widely-used parameter initialization schemes (_e.g_., He initialization [26], the default initialization scheme for CONV and Linear layers in PyTorch [61]) depend on the parameter shape. The large-sparse implementation is _not_ equivalent to (often underperforms) the small-dense implementation. For unstructured pruning, the standard implementation scheme is large-sparse weights (with unstructured masks) [51, 81].
* _Finetuning_. After the sparsifying action action, the subsequent training process is called _finetuning_ or _retraining_. We notice the community seems to have different interpretations about these two terms. _E.g_., in [38], finetuning is a _sub-concept_ of retraining, specifically meaning retraining with the last (smallest) learning rate of original training3; while many

Figure 1: Overview of this paper. We are motivated by unveiling two mysteries (M1, M2) in filter pruning, which represent the confusing pruning benchmark situation. We first clarify the _fairness principle_ and summarize outstanding _comparison setups_ to lay down the discussion foundation (the notation “S1 \(<\) S2” means S1 is _less strict_ than S2; others can be inferred likewise). Then we start to unveil M1 and M2. M2 will be shown to reduce to M1, actually. To unveil M1, we introduce _network trainability_ as an effective perspective to demystify M1. The unawareness of the role of network trainability in pruning has actually led to several sub-optimal hyper-parameter settings, which exacerbates the chaotic benchmark status. We finally give some concrete suggestions to calibrate the pruning benchmarks.
more papers [51, 57, 78, 84, 81] consider finetuning _the same as_ retraining, meaning the 3rd step of the pruning pipeline. In this paper, we take the more common stance: considering finetuning and retraining interchangeable, and in the end of this paper, we will show the term "finetuning" should be deprecated in favor of "retraining".
* _Scratch-E, Scratch-B_. These two terms are from [51], denoting two scratch training schemes. "E" is short for epochs, "B" short for (computation) budget. In practice, [51] uses FLOPs as an approximation for the computation budget. In Scratch-E, the point is to maintain the same _total epochs_ when comparing scratch training to pruning. In Scratch-B, the point is to maintain the same _total FLOPs_ when comparing scratch training to pruning. Here is a concrete example of Scratch-B: a dense model has FLOPs \(F_{1}\); pretraining the dense model takes \(K_{1}\) epochs; it is pruned by \(L_{1}\)-norm pruning, giving a pruned model with FLOPs \(F_{2}\); the finetuning takes another \(K_{2}\) epochs; then the scratch training should take \((K_{1}F_{1}+K_{2}F_{2})/F_{2}=K_{1}(F_{1}/F_{2})+K_{2}\) epochs. The ratio \(F_{1}/F_{2}\) is typically called _speedup_ in network pruning.
* _Value of network pruning_. This term comes from [51]. The conclusion of [51] is that scratch training can match the performance of the 3-step pruning pipeline if Scratch-B is adopted, for filter pruning. Therefore, they argue there is no value for filter pruning algorithms that _use predefined layerwise pruning ratios_. For the filter pruning algorithms that do not use predefined layerwise pruning ratios, their role is to decide the favorable network architectures, akin to NAS [91, 15]. As for unstructured pruning, [43] shows scratch training _cannot_ match pruning. Therefore, rigorously, the argument about the value of network pruning means the value of inheriting pretrained weights in filter pruning with predefined layerwise pruning ratios. We will use the short notion, _value of network pruning_, without mentioning its much richer context.

Footnote 3: [https://github.com](https://github.com)
base model reported by ThiNet [53] has top-5 accuracy 88.44% while CP [30], a concurrent work with ThiNet, reported 89.9% (for those who are not familiar with these numbers, 1.5% top-5 accuracy is a _very significant_ gap for ImageNet-1K classification).

As a remedy, to make the results comparable, many papers report the relative _performance drop_, namely, base model accuracy minus final model accuracy. Such an idea is still broadly practiced at present [51, 81], _esp._ when comparing methods that are implemented under quite different conditions.

S2. Later, as the DL community develops, more DL platforms _e.g._, PyTorch [61] and TensorFlow [1] mature. There is usually a well-accepted model zoo (such as torchvision models5) for others to use. As a result, more and more pruning papers adopt them as the base models, such as [43, 81], which has become the mainstream practice at present. Thus, the S2 comparison setup arises.

Footnote 5: [https://pytorch.org/vision/stable/models.html](https://pytorch.org/vision/stable/models.html)

At this stage, few researchers have noticed the importance of finetuning. This makes sense since, in the pruning pipeline, only the pruning method part (_i.e._, the 2nd step) is regarded as the central one. The finetuning process is often so _downplayed_ that many papers do not even clearly report the hyper-parameters, as also noted by [3].

S3. Later, in the endless pursuit of higher and higher performance, there is a clear trend that the finetuning epochs become longer and longer (see Tab. 3 for an incomplete summary). This in effect renders the comparison more and more unfair. Besides, the finetuning LR has been noticed to have a significant impact on the final performance, as formally studied by [38] (although [38] is the first one to formally study this phenomenon, a larger finetuning LR has been employed by many papers even before). Because of these, the finetuning process must be taken into account to maintain fairness.

The most exact way to rule out the impact of finetuning is to use exactly the same finetuning process - the same LR schedule (including the same epochs; we omit the hyper-parameters, like weight decay, momentum, _etc._, and assume they are maintained the same), _i.e._, the S3.2 in Tab. 2. However, due to various objective or subjective reasons (_e.g._, prior papers may not release their finetuning details, making the follow-ups unable to reproduce the same finetuning), S3.2 is often impractical. Thence comes a weaker setup S3.1, which only keeps the same epochs of finetuning. It is allowed to use different finetuning LR schedules (_e.g._, different initial LR) - this is where M1, the mystery of the finetuning LR effect, arises_.

Several papers (such as [80, 81, 86, 87]) have ablative analysis experiments on small-scale datasets (_e.g._, CIFAR10 [36]) using the setup S3.2, while the main benchmark experiments (_e.g._, with ResNet-50 on ImageNet) using S2. The primary reason is that, following the setup S3.2 means re-running the experiments for other comparison methods in the large-scale benchmarks, which is usually impractical (too costly) or even impossible (_e.g._, the comparison methods do not release usable code).

S4. The S3.2 is still not the most strictly fair setup since it does not consider the cost (measured by the number of epochs) of the pruning method. For one-shot pruning (such as \(L_{1}\)-norm pruning [43]), the cost of pruning is zero; while for a regularization-based method (such as GReg [81]), it may take another few epochs for regularized training. Considering these cases, S4.2 comes out: it builds upon S3.2 and demands the same LR schedule for the pruning algorithm - _as far as we know, this is the most strict comparison setup_. In practice, again, for various reasons, we may not know the LR schedule of a pruning

\begin{table}
\begin{tabular}{l l} \hline \hline
**No.** & **Comparison setups** \\ \hline S1 & Compare performance or performance drop on the same \\  & dataset and network at the same compression or speedup rate \\ \hline S2 & +Same base model \\ S3.1 & +Same fineetuning epochs \\ \hline S3.2 & +Same base model \\  & +Same fineetuning LR schedule \\ \hline S4.1 & +Same fineetuning LR schedule \\  & +Same fineetuning epochs \\  & +Same base model \\  & +Same base model \\ \hline S4.2 & +Same fineetuning LR schedule \\  & +Same fineetuning LR schedule \\ \hline S4.2 & +Same epochs of “pretraining + pruning + finetuning” \\  & +Same fineetuning LR schedule \\ \hline SX-A & +Same epochs of “pretraining + pruning + finetuning” \\ SX-B & +Same FLOPs of “pretraining + pruning + finetuning” \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of popular comparison setups in pruning papers. It is helpful to review them along with the 3-step pruning pipeline: pretraining (output: base model) \(\Rightarrow\) pruning (output: pruned model) \(\Rightarrow\) finetuning (output: final small model). In terms of strictness, S1 \(<\) S2 \(<\) S3.1 \(<\) S3.2 \(<\) S4.1 \(<\) S4.2 (the notation “S1 \(<\) S2” means S1 is _less strict_ than S2; others can be inferred likewise). Most existing pruning papers follow the S2 comparison setup.

\begin{table}
\begin{tabular}{l c c} \hline \hline Method & \#Epochs & LR schedule \\ \hline SSL [84]\({}_{\
algorithm. Then, S4.2 degrades to S4.1, which only demands the same epochs.

**SX.** In setups S2 to S4.2, when comparing pruning to scratch training in obtaining the same pruned (small) model, the scratch training employs the same training recipe of obtaining the base (big) model. [51] challenges this practice. They argue, the scratch training scheme spends less cost than pruning, so the comparison is unfair. As a remedy, they propose to take into account the cost of the _pretraining_ stage, which gives the SX-A and SX-B setups. About the _cost_, one way to measure it is to use the number epochs (hence the SX-A); another is to consider the same computation and they approximate computation with FLOPs (hence the SX-B).

It is hard to say if considering the cost of the pretraining stage is really necessary and practical. Advocates of the older practice may list reasons, _e.g_., pretrained models often exist already (like those pretrained on ImageNet [10] and shared on HuggingFace6), so we do not need to consider the cost of pretraining. However, advocates of SX may argue that not all pretrained models are available; for many tasks, we still need to train the pretrained models first, so the cost of scratch training should be considered.

Footnote 6: [https://huggingface.co/](https://huggingface.co/)

We have no inclination here regarding which one is more correct. We make two points that we are fairly certain about: (1) In the pruning literature, most papers still follow the _older_ practice when reporting the scratch training results of the pruned model. (2) Given the recent rise of foundation models [4] (_e.g_., Bert [12], GPT-3 [5], CLIP [63], diffusion models [68, 73]), common researchers barely have the resources to train a model from scratch, so pruning would inevitably be conducted on the pretrained model, probably, for those big models.

**What comparison setup is mostly used now?** Unfortunately, S2 is the most prevailing comparison setup at present [3]. This setup ignores at least one important factor that, we now know [38], has a significant impact on the final performance: the finetuning LR schedule.

In the following sections, we start our empirical investigation of unveiling M1 and M2. We study M2 first and then M1, because the conclusion about M2 actually depends on M1, as we are about to show.

## 4 Reexamining the Value of Pruning

The rethinking paper [51] presents many valuable thoughts regarding the value of the 3-step pruning pipeline against scratch training. However, there are a few potential concerns in their experiments that may shake the validity of their conclusion. _First_, they directly cite the results of a few pruning papers and compare the relative performance drop. Because of the stark differences between different DL platforms, such a comparison (_e.g_., comparing methods that use different base models) may not be convincing enough for rigorous analysis. _Second_, when reproducing the \(L_{1}\)-norm pruning [43], they use fixed LR 0.001 and 20 epochs, following [43], for the finetuning stage, which is now known to be severely sub-optimal (see Tab. 4, a larger finetuning LR 0.01 can significantly boost performance).

It is thereby of interest whether the no-value-of-pruning argument would change if the comparison is conducted under a strictly controlled condition and a better finetuning LR is employed. This section attempts to answer this question. Three comparison setups (SX-A, SX-B, and S4.2) are considered since they are the _most strict_ setups up to date.

**Pruning method**. We choose \(L_{1}\)_-norm pruning_[43] because it is the most representative pruning method and easy to control at a strict comparison setup. Specifically, \(L_{1}\)-norm pruning prunes the filters of a pretrained model with the smallest \(L_{1}\)-norms to a predefined pruning ratio. After pruning, the pruned model is finetuned for a few epochs to regain performance. Other pruning methods, such as regularization-based methods (_e.g_., [81, 50, 84]), introduce many factors that are hard to control for rigorous analysis, so we do not adopt them here. We will discuss how the findings can generalize to those cases later.

**Networks and datasets**. The network used for analysis is ResNet34 [27], following [43]. For standard benchmarks (_e.g_., Tab. 1), we use ResNet50 [27] because it is one of the most representative benchmark networks in filter pruning. The datasets are ImageNet100 and the full ImageNet [10]. ImageNet100 is a randomly drawn 100-class subset of ImageNet. We use it for _faster_ analysis given our limited resource budget. The full ImageNet is used for benchmarks.

**Implementation details of pruning**. For analysis, pruning is conducted on the 1st CONV layer (the 2nd CONVs are not pruned, following \(L_{1}\)-norm pruning [43]) in all residual blocks of ResNet34. The first CONV and all FC layers are spared, also following the common practice [81, 89, 21]. Uniform layerwise pruning ratio is employed (which usually under-performs a tuned non-uniform layerwise pruning ratio scheme; but since this paper does not target the best performance but explanation, we adopt it for easy analysis). We conduct pruning at a wide sparsity spectrum (10% to 95%) in the hopes of comprehensive coverage.

**One table to show them all**. The results are presented in Tab. 4. Before we present the analyses, we introduce a notion, _pruning epoch_, which is defined as the epoch when the sparsifying action is physically enforced. _E.g_., if a model is trained for 30 epochs and then the sparsifying action is enforced, the pruning epoch is 30. We observe:

**(1)** For the S4.2 setup (rows marked by ), we are not
allowed to change the LR schedule. The only thing we can change is the pruning epoch. As seen, the best pruning epoch varies _w.r.t._ the sparsity level - at a small pruning ratio, different pruning epochs give a similar performance; while as the pruning ratio arises, the performance becomes more sensitive to the pruning epoch, _e.g._, for pruning ratio 95%, P90F30, le-4 severely underperforms P30F90, le-2. Notably, a clear trend is, when the pruning ratio is large (70% to 95%), it is better to have a smaller pruning epoch.

Under this setup, only at pruning ratios of 30%-70%, pruning surpasses scratch training by a statistically significant gap. Therefore, we can only say pruning has a marginal advantage over scratch training here.

**(2)** Then we look at the setup SX-A (rows marked by ). Under this setup, we are allowed to adjust the finetuning LR as long as the total epochs are kept the same. We increase the initial finetuning LR. As seen, it significantly improves the accuracies, _e.g._, (P30F90, le-1) improves the accuracy by nearly 2% at pruning ratio 95%, against (P30F90, le-2). This is the performance-boosting effect aforementioned [38]. We also apply the larger LR trick to another two settings P60F60 and P90F30. In all of them, we see a larger finetuning LR improves performance by an obvious margin.

Now, the gap between pruning and scratch training becomes much more significant. Pruning is more surely valuable under this setup.

**(3)** Next, we use the comparison setup SX-B (rows marked by ), which maintains the total FLOPs. We apply this scheme to the best pruning setup P30F90, le-1 in S4.1 in the hopes of better performances. Since the dense model is trained for 30 epochs, to compensate for the FLOPs, the pruning epoch should be squeezed by the speedup ratio \(k\). _E.g._, for pruning ratio 10%, the speedup ratio is 1.11, then the pruning epoch should be adjusted to \(30/k\approx 27\).

As seen, the squeezing of the pruning epoch does close the gap between pruning and scratch training: At pruning ratios of 10% to 70%, pruning is still better; while for 90% and 95%, pruning only matches or underperforms scratch training - this is a concrete example that we do _not_ have a once-for-all answer to questions like "_is pruning better than scratch training?_"

We also try a smaller finetuning LR in this setup, as shown in the row (P30/\(k\)F90, le-2). The LR effect also translates to this case - a smaller finetuning LR degrades the performance.

**Short summary**. As seen, the argument about the value of network pruning severely hinges on which comparison setup is employed and the pruning ratio. For the setup SX-A, where pruning outperforms scratch training obviously, the advantage comes from a better finetuning LR. Yet, we are not sure if such better LR schedules also exist for the scratch training; if so, scratch training can be further boosted, too - as such, this kind of "competition" can be _endless_. There are two kinds of attitudes toward this situation: _(1)_ Do not consider the performance improvement from a better finetuning LR as a fair/valid performance advantage as it is _not_ from the pruning algorithm. _(2)_ Still consider it as a valid performance advantage but will meet the "endless competition" challenge we just mentioned. The community now is mostly using _(2)_. We suggest using _(1)_, following our fairness definition clarified in Sec. 3.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Pruning ratio & 10\% & 30\% & 50\% & 70\% & 90\% & 95\% \\ FLOPs (G, speedup: \(k\times\)) & 3.30 (1.11\(\times\)) & 2.59 (1.41\(\times\)) & 1.90 (1.93\(\times\)) & 1.19 (3.09\(\times\)) & 0.48 (7.68\(\times\)) & 0.30 (12.06\(\times\)) \\ \hline Scratch training & 83.68\({}_{\pm 0.38}\) & 83.31\({}_{\pm 0.13}\) & 82.90\({}_{\pm 0.16}\) & 82.45\({}_{\pm 0.13}\) & 79.37\({}_{\pm 0.76}\) & 76.67\({}_{\pm 0.90}\) \\ \hline  & 83.95\({}_{\pm 0.17}\) & **84.01\({}_{\pm 0.23}\)** & **83.87\({}_{\pm 0.44}\)** & **82.93\({}_{\pm 0.10}\)** & 79.86\({}_{\pm 0.11}\) & 77.41\({}_{\pm 0.11}\) \\  & 83.88\({}_{\pm 0.07}\) & 84.00\({}_{\pm 0.22}\) & 83.29\({}_{\pm 0.14}\) & 82.61\({}_{\pm 0.07}\) & **80.41\({}_{\pm 0.32}\)** & **77.64\({}_{\pm 0.39}\)** \\  & 83.56\({}_{\pm 0.03}\) & 83.95\({}_{\pm 0.14}\) & 83.28\({}_{\pm 0.08}\) & 82.47\({}_{\pm 0.12}\) & 79.88\({}_{\pm 0.10}\) & 76.17\({}_{\pm 0.21}\) \\  & 84.21\({}_{\pm 0.07}\) & 83.87\({}_{\pm 0.09}\) & 82.90\({}_{\pm 0.10}\) & 81.24\({}_{\pm 0.17}\) & 77.29\({}_{\pm 0.05}\) & 70.53\({}_{\pm 0.37}\) \\  & 84.24\({}_{\pm 0.04}\) & 83.47\({}_{\pm 0.12}\) & 82.45\({}_{\pm 0.14}\) & 80.81\({}_{\pm 0.09}\) & 73.94\({}_{\pm 0.24}\) & 64.98\({}_{\pm 0.31}\) \\  & 84.09\({}_{\pm 0.07}\) & 82.47\({}_{\pm 0.02}\) & 79.70\({}_{\pm 0.00}\) & 74.87\({}_{\pm 0.19}\) & 49.23\({}_{\pm 0.21}\) & 29.89\({}_{\pm 0.26}\) \\  & 85.27\({}_{\pm 0.13}\) & **85.37\({}_{\pm 0.19}\)** & **85.48\({}_{\pm 0.18}\)** & **83.83\({}_{\pm 0.17}\)** & **81.56\({}_{\pm 0.29}\)** & **79.57\({}_{\pm 0.15}\)** \\  & 83.72\({}_{\pm 0.14}\) & 83.88\({}_{\pm 0.07}\) & 83.67\({}_{\pm 0.11}\) & 82.96\({}_{\pm 0.23}\) & 80.78\({}_{\pm 0.23}\) & 77.81\({}_{\pm 0.25}\) \\  & 83.91\({}_{\pm 0.08}\) & 84.02\({}_{\pm 0.20}\) & 83.41\({}_{\pm 0.15}\) & 82.91\({}_{\pm 0.12}\) & 79.43\({}_{\pm 0.07}\) & 75.20\({}_{\pm 0.23}\) \\  & 83.41\({}_{\pm 0.09}\) & 84.02\({}_{\pm 0.20}\) & 84.85\({}_{\pm 0.31}\) & **83.64\({}_{\pm 0.09}\)** & **79.65\({}_{\pm 0.31}\)** & **75.79\({}_{\pm 0.28}\)** \\  & 83.40\({}_{\pm 0.04}\) & 82.69\({}_{\pm 0.27}\) & 82.16\({}_{\pm 0.03}\) & 79.97\({}_{\pm 0.16}\) & 74.76\({}_{\pm 0.24}\) & 70.61\({}_{\pm 0.52}\) \\ \hline \hline \end{tabular}

* Under comparison setup S4.2 (same overall LR schedule),  Under comparison setup SX-A (same total epochs; finetuning LR increased),  Under comparison setup SX-B (same total FLOPs).

\end{table}
Table 4: Top-1 accuracy (%) comparison between \(L_{1}\)-norm pruning [43] and training from scratch with **ResNet34** on **ImageNet100**. Each result is averaged by at least three random runs. The learning rate (LR) schedule of scratch training is: Initial LR 0.1, decayed at epoch 30/60/90/105 by multiplier 0.1, total: 120 epochs (top-1 accuracy of dense ResNet34: 84.56%, FLOPs: _”P30F90, 1e-1” means the model is pruned at epoch 30 and finetuned for another 90 epochs with initial finetune LR 1e-1_ (please refer to our supplementary material for the detailed LR schedule); the others can be inferred likewise. The **best** result within each comparison setup is highlighted **hold**.
Despite many uncertainties, we are certain about one thing from Tab. 4: Whichever setup is favored, the fine-tuning LR holds a critical role to performance. Even for the comparisons setup S4.2, where the finetuning LR does not change, by changing the pruning epoch, implicitly, we change the finetuning LR, and it has been shown very pertinent to the final performance as well. In this sense, the two mysteries of pruning actually boil down to one (M1): _Why does finetuning LR have such a great impact on the performance?_

This is the next question we would like to answer. LR, (arguably) as the most influential hyper-parameter in training neural networks, has a significant impact on performance - this is definitely not surprising; what is really surprising might be, why the prior pruning works (_e.g_., the original \(L_{1}\)-norm pruning [43] adopts LR 0.001 in finetuning for their ImageNet experiments) did not realize that such a simple "trick" is so important to performance? This question is also worth our thinking.

## 5 Trainability in Network Pruning

### Background of Trainability

Trainability, by its name, means the ability (easiness) of training a neural network, _i.e_., the optimization speed (note, speed is not equal to quality, so we may see a network with good trainability turns out to have a bad generalization ability eventually).

Notably, essentially, the role of a pruning method is to provide the initial weights for the later finetuning process, that is, pruning is essentially a kind of _initialization_. In stark contrast to the broad awareness that initialization is very critical to neural network training [26, 35, 55, 74, 22], the initialization role of pruning has received negligible research attention, however. Trainability is also mostly studied for random initialization [85, 69].

A few recent works marry it with network pruning in some other similar forms like signal propagation [41] and gradient flows [79] (a good signal propagation or strong gradient flow usually suggests a good trainability). These works are inspiring, while they mostly stay in the domain of pruning at initialization (PaI). Few attempts before, to our best knowledge, tried to utilize the notion of trainability to examine pruning after training (PaT), at least, for the two mysteries we study here. This paper is meant to bridge this gap. The major difference between PaI and PaT is whether using a pretrained model as base. Such a context is essential to this paper since the above two mysteries are both brought forward in the context of PaT.

**Trainability accuracy**. Literally, a bad trainability implies the training is hard and the training performance will arise slowly. Per this idea, there is a straightforward metric to measure trainability - we introduce _trainability accuracy_, the average of the first \(N\) epochs,

\[T=\frac{1}{N}\sum_{i=1}^{N}\mathrm{Acc}_{i}. \tag{1}\]

Since the optimization speed depends on the LR used, when we calculate trainability accuracy, we must ensure they are under the same LR schedule. In this paper, we choose \(N\) as the number of the 1st LR stage, which characterizes the optimization speed in the early phase.

Next, we utilize trainability to explain the mysterious effect of the finetuning LR.

### Examining the Effect of Finetuning LR

**Two facts as foundation**. We first lay down two facts as the common ground for the discussion of this section. We will show the mystery about the finetuning LR effect boils down to these two simple facts.

_First, pruning damages trainability_. This is an intuitively straightforward fact since pruning removes connections or neurons, which virtually makes the network harder to train. This fact holds for not only pruning a random network [41], but also for pruning a pretrained model here. Moreover, notably, more aggressive pruning leads to more damaged trainability. _Second, a model of worse trainability will need more effective updates to reach convergence_. More effective updates mean two cases: If LR is not changed, more epochs are needed; if the number of epochs does not change, a larger LR is needed. This is also easy to understand since trainability measures the easiness of optimization; a bad trainability implies harder optimization

\begin{table}
\begin{tabular}{l c c} \hline \hline Finetuning setup & Top-1 acc. (\%) & TA (\%) \\ \hline P30F90, 1e-1 & 79.57\({}_{\pm 0.15}\) & 88.00 \\ P30F90, 1e-2 & 77.64\({}_{\pm 0.39}\) & 77.45 \\ P30F90, 1e-2 (+30 epochs) & 79.12\({}_{\pm 0.19}\) & / \\ P30F90, 1e-2 (+60 epochs) & **79.59\({}_{\pm 0.25}\)** & / \\ \hline P60F60, 1e-2 & **77.81\({}_{\pm 0.25}\)** & 87.39 \\ P60F60, 1e-3 & 70.53\({}_{\pm 0.37}\) & 68.19 \\ P60F60, 1e-3 (+60 epochs) & 75.71\({}_{\pm 0.09}\) & / \\ P60F60, 1e-3 (+120 epochs) & 77.17\({}_{\pm 0.13}\) & / \\ P60F60, 1e-3 (+180 epochs) & 77.33\({}_{\pm 0.09}\) & / \\ \hline P90F30, 1e-2 & 75.20\({}_{\pm 0.23}\) & 84.83 \\ P90F30, 1e-4 & 29.89\({}_{\pm 0.26}\) & 37.93 \\ P90F30, 1e-4 (+60 epochs) & 60.69\({}_{\pm 0.17}\) & / \\ P90F30, 1e-4 (+270 epochs) & 70.78\({}_{\pm 0.16}\) & / \\ P90F30, 1e-4 (+1485 epochs) & **78.18** & / \\ \hline \hline \end{tabular}
\end{table}
Table 5: Top-1 accuracy (%) comparison of different setups of \(L_{1}\)-norm pruning [43] with **ResNet34** on **ImageNet100**. Pruning ratio: 95%. TA: trainability accuracy (the metric used to measure trainability; see Eq. (1)). This table shows, the performance gap between a smaller LR and a larger LR is not fundamental. It can be closed simply by training more epochs. The root cause that a smaller LR _appears_ to under-perform a larger LR is simply that the model trained by the smaller LR does _not_ fully converge.
literally, hence the more effective updates. Such observation has been made by some sparse training papers, _e.g._, RigL [16] notes that "_sparse training methods benefit significantly from increased training steps_".

When we observe that a larger LR improves the final test accuracy of the pruned model (_e.g._, Row P30F90, le-1 _vs_. Row P30F90, le-2 in Tab. 4), it is worthwhile to differentiate two subtle yet distinct possibilities:

* A larger LR helps the pruned model reach a solution that the smaller LR _cannot_ reach, _i.e._, a larger LR help the model located into a better local minimum basin.
* The smaller LR can also help the model reach the solution as the larger LR does; just, the larger LR helps the model get there faster.

The former implies the performance-boosting effect of a larger LR is _fundamental_; while the latter implies there is no fundamental gap between the two solutions; it is only an issue of optimization speed.

Let's analyze a concrete example of P60F60 in Tab. 5. For pruning ratio 95% (we use this for example because at larger sparsity, the performance boosting effect is most pronounced), using 1e-2 _vs_. 1e-3 improves the test accuracy from 70.53 to 77.81, a very significant jump. This improvement also translates to the rows of P30F90 and P90F30.

However, in Tab. 5, we note the performance improvement coincides with trainability accuracy improvement. We were wondering if the performance improvement is actually due to a better trainability.

Fig. 2(a) plots the test accuracy during the finetuning of P60F60, le-3. We notice before the 1st LR decay at epoch 30, the accuracy keeps arising even at epoch 30. This triggers a question: usually, we decay LR when the accuracy _saturates_; now, when the accuracy is still steadily rising, _is the LR decayed too early?_ This question matters because if the LR decays too early, the model is _forced_ to stabilize due to the small step size and insufficient updates, not because it gets close to the local minimum, _i.e._, the model may _not converge at all_.

To verify this, we extend the epochs of the LR 0.001 phase by 60/120/180 epochs. See the results in Tab. 5 (note the rows P60F60, le-3 (+60/120/180 epochs)) and Fig. 2(b). Now, the model finetuned by LR 1e-3 can reach 77.33, very close to 77.81 reached by LR 1e-2. The test accuracy plot in Fig. 2(b) also confirms that the seeming underperformance of LR 1e-3 is due to insufficient epochs - namely, **the advantage of a larger LR 0.01 is not some magic fundamental advantage, but a simple consequence of faster optimization.**

We also verify this on other cases (P30F90 and P90F30) that the smaller LR "underperforms" the larger LR. The results are also presented in Tab. 5. In _all_ of these cases, given abundant epochs, the gap between the larger LR and the smaller LR can be closed. Especially for P90F30, the smaller LR 1e-4 can achieve a much better result than LR 1e-2 (78.18 _vs_. 75.20). This strongly demonstrates the smaller LR can also achieve what the larger LR can do.

To summarize, our results suggest **a larger LR does not really "improve" the performance. What really happens is, a larger LR accelerates the optimization process, making the higher performance _observed earlier_**. In practice, when researchers tune different LR's, they usually keep the total epochs fixed (for the sake of fairness). Given the same total epochs, the pruned model using the smaller finetuning LR does not fully converge, making the performance _appear_ "worse".

**Further remarks.** It is worthwhile to note that such an experimenting trap is _so covert_ if we are unaware of the damaged trainability issue in pruning. We may never realize that the epochs should be increased properly if a smaller finetuning LR is used. What's even trickier, we do not know how many more epochs is the so-called _proper_ - Tab. 5 is a living example. For some cases (_e.g._, P30F90), 60 more epochs is enough, while for others (_e.g._, P60F60, P90F30), 180 epochs or more is not enough to bridge the performance gap. Clearly, there is still much work to be done here toward a more rigorous understanding of the influence of damaged

Figure 2: Test accuracy _vs_. epoch during finetuning of the setting P60F60, 1e-3 at pruning ratio 95% in Tab. 5. Red vertical lines mark the epoch of decaying LR by 0.1. Particularly note before the 1st LR decay, the accuracy keeps arising in (a), implying the 1st LR decay may be too early – this is confirmed in (b), where the red cross marker (\(\times\)) indicates the time point of the 1st LR decay in (a). See more similar plots in our supplementary material.
trainability on pruning.

**Retrospective remarks and the lessons.** It is worthwhile to ponder why [43] employed a _seriously sub-optimal_ fine-tuning LR scheme. This, we conceive, may originate from a long-standing _misunderstanding_ in the area of network pruning - many have believed that because pruning is conducted onto a _converged_ model, the retraining of the pruned model needs not to be long and the LR should be _small_ to avoid destroying the knowledge the model has acquired, _e.g._, in [67], the authors mentioned in their abstract "_The standard retraining technique, fine-tuning, trains the unpruned weights from their final trained values using a small fixed learning rate_", implying that such misconception spreads so widely that it is taken for "standard".7

Footnote 7: Actually, the 3rd-step of the pruning pipeline is broadly referred to as _finetuning_ – this term per se already implies the inclination of using a small LR. To rule out such conceptual bias, a more accurate way to phrase the 3rd step in the pruning pipeline may be _retraining_ the pruned model.

However, the results in Tab. 4 suggest, such thought only holds for the cases of low pruning ratios. For a moderate or large pruning ratio, this thought _hardly_ holds. What was neglected is that the sparsifying action damages network trainability, slowing down the optimization. As a compensation, it is supposed to use a larger LR to accelerate the optimization, not a smaller LR; similarly, more epochs are needed to compensate for the slow optimization. However, the original \(L_{1}\)-norm pruning [43] chose LR 0.001 and only 20 epochs for their ImageNet experiments, exactly the opposite of what is supposed. This, we conceive, is the reason that \(L_{1}\)-norm pruning has been underrated for a long time. Its real performance is actually pretty strong even compared with recent top-performing approaches (see Tab. 1).

Similarly, based on what we just learned about the truth of M1, if we examine the other filter pruning methods, _e.g._, GAL [48] (see Tab. 3), its reported results are probably also underrated, because it uses only 30 epochs for finetuning and the model may well not expose its full potential, as a result of the immature convergence. This implies a pretty disturbing concern - for many filter pruning papers, we have to calibrate their results for fairness. Directly citing the numbers may (well) not show the real performance comparison.

**How to make comparisons in a pruning paper?** As seen, different finetuning hyper-parameters can lead to very different conclusions, which do not reflect the true comparison of the pruning algorithm part. The ideal solution to this problem is to re-implement the existing pruning algorithms _under the same configuration_. This counts on the community efforts (_e.g._, ShrinkBench8[3] and Torch-Pruning9[17]) to standardize the benchmarks. Meanwhile, even with these efforts, we have to recognize that re-implementing all or most of the past pruning algorithms under the same condition is unrealistic, and sometimes unnecessary10.

Footnote 8: [https://github.com/JJGO/shrinkbench](https://github.com/JJGO/shrinkbench)

Footnote 9: [https://github.com/VainF/Torch-Pruning](https://github.com/VainF/Torch-Pruning)

Footnote 10: _Rigidly_ demanding _exact the same_ condition in defining fairness may be of little meaning for scientific development. Think about the surge of deep learning – compared to the past SVM era, deep learning nowadays definitely takes (way) more resources than SVM, _i.e._, unfair, while few has questioned the unprecedented value of deep learning as large language models like ChatGPT [60] surprise us on a daily basis.

We suggest a more practical solution here, highlighting two rules of thumb: _(1)_ cross-validating different hyper-parameters; _(2)_ better performance still weighs more when fairness is hard to figure out. For a concrete example, given two pruning algorithms A (with finetuning setup \(FT_{A}\)) and B (with finetuning setup \(FT_{B}\)), we can conduct two more experiments: A + \(FT_{B}\) and B + \(FT_{A}\); then compare A + \(FT_{A}\)_vs._B + \(FT_{A}\) and A + \(FT_{B}\)_vs._B + \(FT_{B}\).

* If one method consistently wins in both cases, we are more confident that the winner is really better due to its effective pruning design.
* If there is inconsistency between the two comparisons, _e.g._, A + \(FT_{A}>\) B + \(FT_{A}\), while A + \(FT_{B}<\) B + \(FT_{B}\), namely, different pruning methods favor different finetuning recipes. This means synergistic interaction exists between the pruning and finetuning stage and we cannot single out the finetuning when evaluating the pruning algorithms. In this case, we suggest that **the case delivering higher performance weighs more** as it advances the state-of-the-art.

## 6 Conclusion and Discussion

This paper attempts to figure out the confounding benchmark situation in filter pruning. Two particular mysteries are explored: the performance-boosting effect of a larger finetuning LR, the no-value-of-pruning argument. We present a clear fairness principle and sort out four groups of popular comparison setups used by many pruning papers. Under a strictly controlled condition, we examine the two mysteries and find they both boil down to the issue of damaged network trainability. This issue was not well recognized by prior works, leading to (severely) sub-optimal hyper-parameters, which ultimately exacerbates the confounding benchmark situation in filter pruning now. We hope this paper helps the community towards a clearer understanding of pruning and more reliable benchmarks of it.

**Takeaways and suggestions from this paper**:

* _Why is the state of neural network pruning so confusing?_ Non-standard comparison setups (and its fundamental reason: unclear fairness principle) and the unawareness of the role of trainability are the two major reasons. The latter further leads to sub-optimal hyper-parameter settings, inherited by many follow-up papers, exacerbating the messy benchmark situation.
- Higher comparison setup means stricter experiment control, also means more resources and efforts; so there would be inevitably a _trade-off_ between how fair we want to be and how much we can invest in.
* Reporting all the finetuning details (_esp._ the LR schedule) is rather necessary and should be standardized.
* Cross-validating finetuning setups is a practical suggestion to decide the true winner between two pruning algorithms with different finetuning recipes.
* Filter pruning can beat scratch training or not, up to the specific comparison setup and pruning ratio in consideration. Given the recent rise of large foundation models, pruning may still follow the conventional 3-step pipeline.
* the performance is not "improved"; what really happens is that the good performance is observed earlier because the larger LR accelerates the optimization. The fundamental factor playing under the hood is the network trainability damaged by the sparsifying action (or zeroing out) action in pruning.
* The damaged network trainability was not well recognized by prior pruning works, resulting in severely sub-optimal hyper-parameters, rendering the potential of a baseline method, \(L_{1}\)-norm pruning [43], _underestimated_ for a long time. This fact may spur us to _reevaluate_ the actual efficacy of (so) many sophisticated pruning methods against the simple \(L_{1}\)-norm pruning.
* we should firmly use _retraining_ instead.
