A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis,
  and Recommendations
Modern deep neural networks, particularly recent large language models, come
with massive model sizes that require significant computational and storage
resources. To enable the deployment of modern models on resource-constrained
environments and accelerate inference time, researchers have increasingly
explored pruning techniques as a popular research direction in neural network
compression. However, there is a dearth of up-to-date comprehensive review
papers on pruning. To address this issue, in this survey, we provide a
comprehensive review of existing research works on deep neural network pruning
in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to
prune, and 4) fusion of pruning and other compression techniques. We then
provide a thorough comparative analysis of eight pairs of contrast settings for
pruning and explore emerging topics, including pruning for large language
models, large multimodal models, post-training pruning, and different
supervision levels for pruning to shed light on the commonalities and
differences of existing methods and lay the foundation for further method
development. To facilitate future research, we build a curated collection of
datasets, networks, and evaluations on different applications. Finally, we
provide valuable recommendations on selecting pruning methods and prospect
several promising research directions. We build a repository at
https://github.com/hrcheng1066/awesome-pruning.
Hongrong Cheng, Miao Zhang, Javen Qinfeng Shi
# A Survey on Deep Neural Network Pruning: Taxonomy, Comparison, Analysis, and Recommendations

###### Abstract

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. More than a thousand pruning papers have been published each year from 2020 to 2022. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of seven pairs of contrast settings for pruning (e.g., unstructured/structured, one-shot/iterative, data-free/data-driven, initialized/pretrained weights, etc.) and explore several emerging topics, including post-training pruning, different levels of supervision for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. Finally, we provide some valuable recommendations on selecting pruning methods and prospect several promising research directions for neural network pruning. To facilitate future research on deep neural network pruning, we summarize broad pruning applications (e.g., adversarial robustness, natural language understanding, etc.) and build a curated collection of datasets, networks, and evaluations on different applications. We maintain a repository on [https://github.com/hrcheng1066/awesome-pruning](https://github.com/hrcheng1066/awesome-pruning) that serves as a comprehensive resource for neural network pruning papers and corresponding open-source codes. We will keep updating this repository to include the latest advancements in the field.

 deep neural network pruning, model compression, model acceleration, edge devices.

## 1 Introduction

Over the past several years, Deep Neural Networks (DNNs) have achieved conspicuous progress in various domains and applications, such as Computer Vision (**CV**) [1, 2, 3], Natural Language Processing (**NLP**) [4] and Audio Signal Processing (**ASP**) [5], so on. Although DNNs achieve remarkable success in various areas, their performance heavily relies on model parameters and computational cost. For example, the widely used ResNet-50 [6] takes over 95MB memory for storage, contains over 23 million trainable parameters, and requires 4 GFLOPs (Giga Floating Point Operations) of computations [7]. The size of VGG-16 [2] trained on ImageNet [1] is more than 500 MB [8]. The Transformer network GPT-3 model consists of up to 175 billion parameters [9], and GPT-4 model has even more. The current trend of enlarging neural network size is anticipated to persist.

However, the more parameters of DNNs, the more time and memory space they typically require for processing the inputs [10]. The high training and inference costs associated with these models present a significant challenge to their deployment on devices constrained by limited computational resources (such as CPU, GPU, and memory), energy, and bandwidth [11, 12, 13]. For example, real-life applications such as autonomous driving, field rescue, and bushfire prevention necessitate high accuracy and efficient resource usage, including fast real-time response and compact memory footprint. Deep neural networks' computational complexity and memory footprint can make them impractical for deployment on edge devices [14]. With the popularity of large language models in recent years, there is growing interest in compressing neural networks for computers with flexible hardware requirements [15]. In addition, deep neural networks that contain redundant features can undermine their robustness, elevating the risk of adversarial attacks [16]. For instance, high-dimensional feature spaces created by these networks can provide greater entry points for adversarial attacks, undermining the network's ability to generalize beyond its original training data.

To relieve this issue, researchers have proposed various neural network compression techniques to design lightweight models, including neural network pruning ([17]), low-rank factorizations of the weight matrices ([18, 19]), quantization ([11, 20]), knowledge distillation ([21]), neural architecture search ([22, 23]) and other compression techniques ([24, 25]). Among them, there are continuing interests in neural network pruning, which has been proven as a desirable and effective way to save memory space and computation time at inference while maintaining a comparable or even better performance compared to the
original DNNs. As shown in Fig. 11, the number of papers on pruning has been markedly increasing from 2015 to 2022. It presents more than half of the papers on neural network compression.

Footnote 1: The data is from [https://www.webotknowledge.com](https://www.webotknowledge.com).

The research on pruning can be traced back to literature as early as 1988 [26]. However, it is only until the emergence of [11] that the research community realizes the potential of pruning in removing significant redundancy in deep neural networks, and pruning begins to gain widespread attention. There are several pieces of literature that review prior work on deep neural network pruning, as shown in Table I. Although these works overview several aspects of pruning and provide helpful guidance for researchers, many of them ([8, 27, 28, 29]) focus on multiple compression techniques, such as pruning, quantization, and knowledge distillation, with only brief examination of each technique. For example, Mishra et al. [27] summarize the compression techniques, including pruning, quantization, low-rank factorization, and knowledge distillation, where pruning is primarily introduced from channel/filter pruning, and many essential pruning techniques (such as lottery ticket hypothesis) are not included. Some review works (such as [30]) focus on reviewing convolutional neural network pruning and lack a description of pruning for other deep neural networks, such as Recurrent Neural Networks (**RNNs**). The work in [31] provides a comprehensive review of sparsity in deep learning up to 2020, but with little studying on emerging pruning methods, such as pruning in contrastive learning [32] and self-supervised pruning [33], etc. Wang et al. [34] provide an overview only for pruning at initialization and does not include studies on pruning during training, pruning after training, etc. [35] is the most recent survey about pruning, while which only focuses on the structured pruning.

This survey aims to provide a comprehensive overview of deep neural network pruning to diverse readers. We review representative pruning methods, propose a new taxonomy, conduct a comprehensive analysis of how different pruning manners behave in practice and give practitioners who wish to utilize pruning recommendations on choosing a suitable pruning method for different requirements. Our contributions are as follows:

**(1) Comprehensive review.** To our best knowledge, this survey is the most comprehensive overview of modern deep neural network pruning techniques. It distills ideas from over 300 related academic papers and establishes a new taxonomy, as shown in Fig. 2. In addition, we provide detailed descriptions of the representative methods for each class of pruning methods.

**(2) Comparative experiments and analysis.** We conduct a comparative analysis of seven pairs of contrast settings for pruning and emerging advances, including different levels of supervision for pruning. Unlike the existing surveys on pruning, this paper conducts experiments and related discussions.

**(3) Collection of abundant resources.** We summarize miscellaneous pruning applications and provide benchmark datasets, networks, and evaluations on different applications. Our collected resources in Appendix B could guide researchers and practitioners to understand, utilize, and develop different network pruning methods for various requirements. The ongoing updates of representative pruning efforts are available at [https://github.com/hrcheng1066/a](https://github.com/hrcheng1066/a) wesome-pruning.

**(4) Recommendations and future directions.** This survey provides valuable recommendations on choosing an appropriate pruning method for different application requirements and highlights promising future research directions.

The remainder of this survey is organized as follows. First, in Section 2, we explain commonly used terms and establish a clear taxonomy of pruning. Section 3 - 6 offer an overview of speedup, when to prune, and how to prune, followed by a comprehensive comparative analysis of different kinds of pruning methods in Section 7. Section 8 discusses integrating pruning with other compression methods. Some practical recommendations for choosing pruning methods and future directions are provided in Section 9. We conclude this paper in Section 10.

## 2 Background

In this section, we first list the commonly used terms and notations in this literature. Then the hierarchical structure of this survey is shown in Fig. 2.

### _Terms and Notations_

The following subsection presents the commonly used terms in pruning literature. It is worth mentioning that some terms (e.g., compression ratio) have a variety of definitions in prior works. We then provide several definitions used in different literature. In addition, for better readability, we list the notations used in this paper in Table II.

\begin{table}
\begin{tabular}{l|c|c} \hline Survey & Main work description & Year \\ \hline
[36] & survey of pruning methods & 1993 \\
[37] & overview 81 papers and propose ShrinkBench, & 2020 \\  & an open-source framework for evaluation & 2020 \\
[28] & pruning-weight sharing+low-rank matrix+ & 2020 \\  & KD+quantization & 2020 \\
[30] & pruning criteria+procedure & 2020 \\
[8] & pruning+quantization+KD+low-rank factor. & 2020 \\
[27] & pruning+KD+NAS+Tensor-Decompose & 2020 \\  & +quantization+hardware acceleration & 2021 \\
[31] & work on sparsity in deep learning up to 2020 & 2021 \\
[34] & overview pruning at initialization & 2022 \\
[29] & pruning+KD+NAS+quantization & 2022 \\
[35] & +Tensor-Decompose+hardware accel. & 2023 \\ \hline \end{tabular}
\end{table} TABLE I: Representative surveys and descriptions.

Fig. 1: The number of papers for neural network pruning and compression from 1988-2022.
* **Prune Ratio**: Prune ratio [38] denotes the percentage of weights (or channels, filters, neurons, etc.) that are removed from the dense network. In general, it can be determined in two ways: pre-defined or learning-decide.
* **Compression Ratio**: Compression ratio in [39, 40] is defined as the ratio of the original weight numbers to the preserved weight numbers, but in [41] it is defined as the ratio of the preserved weight numbers to the original weight numbers. For example, if 10% of weights are preserved, then the compression ratio in [40] is 10, but it is 10% in [41].
* **Sparsity Ratio**: Sparsity ratio or sparsity denotes the portion of zero weights (or channels, filters, neurons, etc.) in networks after pruning [42, 43]. It equals to compression ratio in [41].
* **Speedup Ratio**: Speedup ratio is defined as the value of the pruned number of FLOPs in [12], or MACs in [44] divided by the original number of FLOPs or MACs correspondingly. In [45], the speedup ratio is calculated by dividing the pruned number of filters in one layer by the original number of filters in that layer.
* **One-shot Pruning**: One-shot pruning, also called single-shot pruning in [46], scores only once and then prunes the network to the target prune ratio [47, 48].
* **Iterative Pruning**: Iterative pruning [10], also called greedy pruning or oracle pruning in [49], repeatedly score-prune-retrain circle multiple rounds, and each round is one iteration.
* **Local Pruning**: Local pruning prunes a network by subdividing all weights (or filter, channels, etc.,) into subsets (e.g., layers) and then removing a percentage of each subset [50].
* **Global pruning**: In contrast to local pruning, global pruning removes structures from all available structures of a network until a specific prune ratio is reached [50].
* **Dynamic pruning**: Dynamic pruning depends on specific inputs [51], wherein different subnetworks will be generated for each input sample.
* **Static Pruning**: In contrast to dynamic pruning, the pruned model is shared by different samples for static pruning [51]. In other words, the model capacities are fixed for different inputs.
* **Lottery Ticket Hypothesis**: Lottery Ticket Hypothesis (**LTH**) [47] points out that a randomly-initialized dense network \(f(\mathbf{x};\mathbf{w}_{0})\) contains a sparse subnetwork \(f(\mathbf{x};\mathbf{w}_{0}\odot\mathbf{m})\) which is trainable with the original weights to achieve competitive performance compared to the original networks.
* **Winning Tickets**: For a randomly initialized network \(f(\mathbf{x};\mathbf{w}_{0})\), a winning ticket \(f(\mathbf{x};\mathbf{w}_{0}\odot\mathbf{m})\) is its subnetwork that once be trained for \(T\) epochs (i.e., \(f(\mathbf{x};\mathbf{w}_{T}\odot\mathbf{m})\) will match the performance of the trained network \(f(\mathbf{x};\mathbf{w}_{T})\) under a non-trivial prune ratio. [47].
* **Layer-collapse**: Layer-collapse is a phenomenon mentioned in [39], which occurs when all weights of a layer of a network are removed. It renders the network untrainable. Hayou et al. [52] given a formal definition (i.e., ill-conditioned NN) to this problem.
* **Weight Rewinding**: Weight rewinding [53] rewinds the weights of the subnetwork to the values in an earlier epoch in training \(\mathbf{w}_{t}\), where \(t<<T\).
* **Learning Rate Rewinding**: Learning rate rewinding, proposed in [40], trains the remained weights from the final values using the learning rate schedule for the specified number of epochs.

### _Taxonomy_

There are three critical questions when pruning a deep neural network. **(1) Whether to get universal or specific acceleration through neural network pruning? (2) When to prune the neural network? Specifically, is the neural network pruned before, during, or after training the network for static pruning or dynamic (i.e., run-time) pruning? (3) Whether to prune based on specific criteria or learn to prune?** The answers to the three questions correspond to the three primary aspects of deep neural network pruning as shown in Fig. 2, respectively.

**The first question** is whether speedup depends on specific hardware/software. It is usually divided into three types: unstructured ([39, 46, 47]), semi-structured (also called pattern-based) ([54, 55, 56]) and structured ([57, 58, 59]). Only structured pruning can achieve universal neural network acceleration and compression without requiring special hardware or software. Conversely, both unstructured and semi-structured pruning need the support of special hardware or software.

**The second question** especially indicates the arrangement between pruning weights and training weights of the neural network for static pruning. According to whether pruning is performed before, during, or after training the network, static pruning arrangement can be divided into three categories: pruning before training (**PBT**) ([39, 46, 52, 60, 61]), pruning during training (**PDT**) ([62, 63, 64]), and pruning after training (**PAT**) ([47, 65, 66, 67]). In dynamic pruning, subnetworks are generated at run-time for each input data point.

**The third question** considers whether to prune neural networks with specific criteria or by learning. Criteria rely

\begin{table}
\begin{tabular}{l|c} \hline Notation & Description \\ \hline \(\mathbf{x}\) & Input data \\ \(\mathcal{D}\) & Dataset \\ \(f\) & A network function \\ \(N\) & The number of samples in a dataset \\ \(\mathbf{w}\) & The model weights \\ \(\mathbf{w}_{t}\) & The weights after training \(t\) epochs \\ \(\mathcal{F}_{i,j}\) & The \(j\)-th filter of the \(i\)-th layer \\ \(c_{t}\) & The \(i\)-th channel of a network \\ \(\ell\) & Standard loss function, e.g., cross-entropy loss \\ \(\mathcal{L}\) & Target loss function \\ \(L\) & Total number of layers in a network \\ \(\odot\) & Element-wise multiplication \\ \(\lambda\) & A balance factor \\ \(\boldsymbol{\gamma}\) & A scaling factor vector \\ \(\boldsymbol{\mathrm{m}}\) & The masks of weights (or filters, channels, etc.) \\ \(\mathcal{R}(\cdot)\) & Regularization term \\ \(\Delta\mathcal{L}\) & Loss change \\ \(c_{out}^{(i)}\) & The number of filters at layer \(i\) \\ \(t\) & Threshold vector \\ \hline \end{tabular}
\end{table} TABLE II: Notations and descriptions.
on a heuristic formula to measure the importance of each weight (or filter, channel, and so on). The commonly used pruning criteria include magnitude, norm, loss change, etc. In addition, it is also possible to prune neural networks by learning, such as pruning through sparsity regularization training or dynamic sparse training [68], etc. Whether through criteria or learning, pruning aims to determine the weights of a network that should be pruned.

The above three aspects determine the main characteristics of a pruning algorithm. Different combinations of these three aspects form different pruning methods. We provide a new taxonomy of deep neural network pruning in Section 3-6, and Section 8, as shown in Fig. 2.

## 3 Specific or Universal Speedup

This section categorizes deep neural network pruning into unstructured, semi-structured, and structured. The first two types correspond to specific speedup, and the third corresponds to universal speedup. In the following, we give a detailed introduction to each category.

### _Unstructured Pruning_

Unstructured pruning, also called non-structured pruning or weight-wise pruning, is the finest-grained case.

**Definition 1** (Unstructured Pruning). Given neural network weights \(\mathbf{w}=\{w_{0},w_{1},...,w_{K}\}\), a dataset \(\mathcal{D}=\{(\mathbf{x}_{i},\mathbf{y}_{i})\}_{i=1}^{N}\) composed of input (\(\mathbf{x}_{i}\)), output (\(\mathbf{y}_{i}\)) pairs, and a desired total number of non-zero weights \(k\), unstructured pruning can be written as the following constrained optimization problem [46]:

\[\begin{split}\underset{\mathbf{w}}{\text{min}}& \mathcal{L}(\mathbf{w};\mathcal{D})=\underset{\mathbf{w}}{\text{min}}\, \frac{1}{N}\sum_{i=1}^{N}\ell(\mathbf{w};(\mathbf{x}_{i},\mathbf{y}_{i})),\\ &\text{s.t.}\ \|\mathbf{w}\|_{0}\leq k.\end{split} \tag{1}\]

In practice, unstructured pruning usually does not directly set the weights to 0 but sets their corresponding masks (or indicators) \(\mathbf{m}\) to 0 [61, 46]. In this case, unstructured pruning is regarded as applying a binary mask to each weight. Then Eq.(1) is correspondingly changed as:

\[\begin{split}\underset{\mathbf{w},\mathbf{m}}{\text{min}}& \mathcal{L}(\mathbf{w}\odot\mathbf{m};\mathcal{D})=\underset{ \mathbf{w},\mathbf{m}}{\text{min}}\,\frac{1}{N}\sum_{i=1}^{N}\ell(\mathbf{w} \odot\mathbf{m};(\mathbf{x}_{i}.\mathbf{y}_{i})),\\ &\text{s.t.}\ \|\mathbf{m}\|_{0}\leq k.\end{split} \tag{2}\]

Generally, the network is retrained (i.e., fine-tuning or training-from-scratch) with fixed masks \(\mathbf{m}\), and the masked-out weights are not involved in retraining. Fig. 3 is an example of weight-wise pruning by removing the connections of the neurons (as shown in Fig. 3(a)) or masking the weights with their corresponding masks (as shown in Fig. 3(b)), respectively. Since it can remove weights anywhere, the irregular replacement of non-zero weights leads to actual acceleration requires the support of special software and/or hardware [69, 69, 11, 63, 51]. Therefore, we classify unstructured pruning as a specific speedup technique.

### _Structured Pruning_

**Definition 2** (Structured Pruning). Given a specific prune ratio and a neural network with \(S=\{s_{1},s_{2},...,s_{L}\}\), where \(s_{i}\) can be the set of channels, filters, or neurons in layer \(i\). Structured pruning aims to search for \(S^{{}^{\prime}}=\{s^{{}^{\prime}}_{1},s^{{}^{\prime}}_{2},...,s^{{}^{\prime}}_{L}\}\) to minimize the performance degeneration and maximize the speed improvement under the given prune ratio, where \(s^{{}^{\prime}}_{i}\subseteq s_{i}\), \(i\in\{1,..,L\}\).

Structured pruning removes entire filters, channels, neurons, or even layers ([64]) as shown in Fig. 4(b) and can rebuild a narrow model with a regular structure. It does not need the support of special hardware and software (such as sparse convolution libraries) and can directly speed up networks and reduce the size of the neural networks [69, 71, 14, 50]. Besides, filter and channel pruning can be considered equivalent because pruning a filter in layer \(i\) is equivalent to pruning the corresponding channels in layer \(i+1\)[72], as shown in Fig. 4(a).

### _Semi-structured Pruning_

To improve the flexibility of structured pruning and achieve lower accuracy drop when the pruning rate is high, some

Fig. 2: An overview of the hierarchical structure of the survey.
recent works ([55, 56]) introduce semi-structured pruning that is called pattern-based pruning in [56] to achieve high accuracy and structural regularity simultaneously. Various patterns can be designed; some examples are shown in Fig. 4(c). In contrast, fully structured pruning (such as channel or filter pruning) is classified as coarse-grained structured pruning ([55, 56, 73]), while semi-structured pruning is classified as fine-grained structured pruning. For example, Meng et al. [74] treat a filter as several stripes and propose to prune stripes in each filter. However, patterns used for semi-structured pruning need to be carefully designed to alleviate performance degradation and conduct specific speedup like unstructured pruning.

## 4 When to Prune

This section distinguishes three pruning pipelines for static pruning, as illustrated in Fig. 5, and run-time pruning. The examples of the three pipelines of static pruning are shown in Fig. 6. The statistics of pruning literature on the three types of pipelines for static pruning are shown in Appendix B Fig. 12.

### _Pruning Before Training_

Pruning Before Training (**PBT**), also called foresight pruning [61] or pruning at initialization [39, 46], represents a class of pruning methods that use randomly initialized weights for pruning the network. The principal motivation of PBT methods is to eliminate the cost of pretraining. Without loss of generality, we define a neural network as a function \(f(\mathbf{x};\mathbf{w}\odot\mathbf{m})\). The mask \(\mathbf{m}\) is used for pruning initialized weights \(\mathbf{w}_{0}\) sampled from a specific initialization distribution. After pruning, the network \(f(\mathbf{x};\mathbf{w}_{0}\odot\mathbf{m}^{{}^{\prime}})\) is trained to converge \(f(\mathbf{x};\mathbf{w}_{T}\odot\mathbf{m}^{{}^{\prime}})\) after \(T\) epochs, where \(\mathbf{m}^{{}^{\prime}}\) indicates the sparsity results after pruning.
dynamic process than that of PBT and PAT methods. We summarize the main prior solutions as the three paradigms: (1) sparsity regularization based, (2) dynamic spare training based, and (3) score-based. The methods related to (1) or (3) take dense-to-sparse training, and the ones related to (2) conduct sparse-to-sparse training.

#### 4.2.1 Sparsity Regularization based Methods

Sparsity regularization technique is commonly used in PDT methods ([74, 88]). This category of methods starts with dense networks, imposes sparse constraints on loss functions, and usually zeros out some weights or their masks during training. The main effort is to design the effective target loss function \(\mathcal{L}\) with an advanced penalty scheme and efficient optimization algorithms. For example, Wen et al. [63] propose Structured Sparsity Learning (**SSL**) to learn a sparse structure by group LASSO [89] regularization during the training. However, SSL requires computing the gradients of the regularization term w.r.t. all the weights, which is non-trivial. Gordon et al. [90] propose **MorphNet** that reuses the parameters of BN and conducts sparsity regularization on these parameters. However, some networks (e.g., some VGGNets [2]) have no BN layers. Instead of reusing BN parameters, some works associate scaling factors with channels, filters, layers, etc. For example, Huang and Wang [64] propose Sparse Structure Selection (**SSS**) that associates scaling factors for CNN micro-structures (e.g., channels, residual blocks) and exploit sparsity regularization to force the output of the micro-structures to zero, rather than pushing the weights in the same group to zero in [63]. In addition, SSS does not require extra fine-tuning that is needed in [63]. Li et al. [91] propose factorized convolutional filter (**FCF**) which introduces a binary scalar to each filter and proposes a back-propagation with Alternating Direction Method of Multipliers (**ADMM**) [92] algorithm to train the weights and the scalars during training jointly.

#### 4.2.2 Dynamic Sparse Training based Methods

A class of the PDT methods ([84, 93, 94, 95, 96, 97, 98]) take randomly initialized sparse network rather than dense network as the input model. Subsequently, one common method is pruning a fraction of unimportant weights followed by regrowing the same number of new weights to adjust the sparse architecture. By repeating the prune-and-grow cycle during training, this kind of method keeps searching for better sparse architecture, which is classified as dynamic sparse training in [84].

\begin{table}
\begin{tabular}{l|c c c} \hline Method & Criteria & U/S & Date-Free (Y/N) & One-shot (Y/N) \\ \hline SNIP (2019) [46] & \([\mathbf{V_{w}}\mathcal{L}(\mathbf{w})\odot\mathbf{w}]\) & U & N & Y \\ Gersp (2020)[61] & \(-(H\mathbf{V_{w}}\mathbf{w}(\mathbf{w}))\odot\mathbf{w}\) & U & N & Y \\ Smart-Ratio (2020) [60] & keep-ratios & U & Y & Y \\ SynFlow (2020) [39] & \(\nabla_{w}\mathcal{R}_{SF}(\mathbf{w})\odot\mathbf{w}\) & U & Y & N \\ PFS (2020) [76] & \(l_{1}\)-norm based sparsity regularization & S & N & N \\ RST (2022) [65] & \(l_{2}\)-norm based sparsity regularization & U & N & N \\ \hline \end{tabular}
\end{table} TABLE III: Representative methods of pruning before training. “U/S” denotes unstructured or structured pruning.

Fig. 4: The visualization of structured and semi-structured pruning, where each Conv is composed of a specific number of little cubes, and each little cube indicates a weight. The light orange cubes denote the pruned weights.

Fig. 3: The visualization of unstructured pruning. The light orange circles denote neurons.
For example, Mocanu et al. [93] propose Sparse Evolutionary Training (**SET**) that removes the smallest positive and the most negative weights and grow new weights in random locations. Instead of pruning a fixed fraction of weights at each redistribution step, such as in SET [93], Mostafa and Wang [95] propose Dynamic Sparse Reparameterization (**DSR**) which uses an adaptive threshold for pruning. In addition, DSR [95] reallocates weights across layers and does not restrict to the inner-layer weight redistribution in SET [93]. Liu et al. [99] propose an ensemble method **FreeTickets** that ensemble sparse subnetworks created by sparse-to-sparse methods. Instead of using a random regeneration scheme, Dai et al. [94] propose a DNN synthesis tool (**NeST**) that stars training with a small network, adjust the subnetworks with gradient-based growth and magnitude-based pruning. Evci et al. [84] propose Rigged Lottery (**RigL**) that actives new connections during training by using gradient magnitude. Similarly, Liu et al. [96] propose Gradual Pruning with zero-cost Neurogeneration (**GraNet**) to remove connections of networks based on their weight magnitudes and regrow connections of networks based on their gradient. They argue that even the weights with zero gradient values indicate the connection importance. Sokar et al. [100] pioneer to explore dynamic sparse training in Reinforcement Learning (**RL**). Graesser et al. [101] systematically investigate some dynamic sparse training methods (such as RigL [84], SET [93]) in RL.

Evci et al. [102] analyze the reasonability of dynamic sparse training. The authors find sparse networks have poor gradient flow at initialization, but dynamic sparse training significantly improves gradient flow. It could be the reason for their success.

#### 4.2.3 Score-based Methods

Some PDT methods exploit scoring criteria to prune the networks during training. He et al. [86] propose Soft Filter Pruning (**SFP**) filter pruning method which can train the networks from scratch and prune the networks simultaneously by using the \(l_{2}\) norm of each filter as its importance. Instead of associating masks with filters, they directly set the pruned filter weights as zeros, which can be updated from zeros through the forward-backward process. Hence, the pruned filters in this epoch can be recovered in the next epoch. However, SFP requires manually preset prune ratios for each layer. He et al. [103] propose Filter Pruning via Geometric Median (**FPGM**) to prune the redundant filters which are nearest to the geometric median [104] of the filters within the same layer. However, FPGM also requires a pre-defined prune ratio for each layer. Liu et al. [62] propose a method called **Network Slimming** which introduces a scaling factor for each channel and jointly trains the weights and the scaling factors by adding the regular loss \(\ell\) with sparsity regularization on the factors, and the magnitudes of these scaling factors are used as the filter scores. In practice, they directly reuse the \(\gamma\) parameters in Batch Normalization (**BN**) [105] layers as the scaling factors.

### _Pruning After Training_

Pruning After Training (**PAT**) is the most popular type of pruning pipeline because it is commonly believed that pretraining the dense network is necessary to obtain an efficient subnetwork [38]. This class of pruning methods generally follows a Pretrain-Prune-Retrain process as shown in Fig. 5 (c). (1) Pretrain a randomly initialized dense network \(f(\mathbf{x};\mathbf{w}_{0})\) to converge \(f(\mathbf{x};\mathbf{w}_{T})\). (2) Prune the weights (or

\begin{table}
\begin{tabular}{l|c c c} \hline Method & Object function/Criteria & Retrain (Y/N) & U/S \\ \hline Network Slimming (2017) [62] & \(\min_{\mathbf{w},f}\ell(\mathbf{y},f(\mathbf{x};\mathbf{w}))+\lambda\| \boldsymbol{\gamma}\|_{1}\) & Y & S \\ SSS (2018) [64] & \(\min_{\mathbf{w},f}\ell(\mathbf{y},f(\mathbf{x};\mathbf{w},\boldsymbol{\gamma })+\mathcal{R}(\mathbf{w})+\lambda\|\boldsymbol{\gamma}\|_{1}\) & N & S \\ SET (2018) [93] & \(\|\mathbf{w}\|\) for drop and random for grow & N & U \\ DST (2020) [68] & \(\min_{\mathbf{w},f}\ell(\mathbf{y},f(\mathbf{x};\mathbf{w}))+\lambda\sum_{i=1}^ {L}\sum_{j=1}^{c(i)}exp(-\mathbf{t}_{j})\) & N & S \\ GraNet (2021) [96] & \(\|\mathbf{w}\|\) for drop and gradient for grow & N & U\&S \\ FreeTickets (2022) [99] & \(\|\mathbf{w}\|\) for drop and gradient for grow & N & U \\ \hline \end{tabular}
\end{table} TABLE IV: Representative methods of pruning during training. “retrain” refers to training from scratch or fine-tuning, “U/S” denotes unstructured or structured pruning.

Fig. 5: The typical pruning pipelines of static pruning. Dashed boxes represent models and solid boxes denote actions.
filters, neurons, etc.) that have the least influence on the performance and fine-tune the pruned network \(f(\mathbf{x};\mathbf{w}_{T}^{{}^{\prime}}\odot\mathbf{m}^{{}^{\prime}})\) for several iterations, where \(\mathbf{w}_{T}^{{}^{\prime}}\) and \(\mathbf{m}^{{}^{\prime}}\) are the weights and masks after pruning, respectively. Process (2) is processed at least once (i.e., one-shot pruning) or multiple times (i.e., iterative pruning). (3) Train the remained weights from scratch \(f(\mathbf{x};\mathbf{w}_{0}\odot\mathbf{m}^{{}^{\prime}})\) or fine-tune \(f(\mathbf{x};\mathbf{w}_{T}^{{}^{\prime\prime}}\odot\mathbf{m}^{{}^{\prime \prime}})\) to recover the performance [40], where \(\mathbf{w}_{T}^{{}^{\prime\prime}}\) and \(\mathbf{m}^{{}^{\prime\prime}}\) are the final results of weights and masks after the overall pruning process, respectively. During the pruning process, sparsity is gradually increased until it achieves the target.

#### 4.3.1 LTH and its Variants

Lottery Ticket Hypothesis (**LTH**) [47] is one of the most influential hypotheses in the neural network pruning domain. Given a pretrained network, LTH iteratively removes a percentage of the weights based on their magnitudes. After pruning, the remaining weights are retrained from scratch with the original initialization, rather than random reinitialization, to match the original networks' accuracies. It challenges the commonly believed paradigm that the pretrained weights must be used for retraining and conjectures the existence of an independently trainable sparse subnetwork from a dense network. Inspired by LTH, there are various follow-up works to identify wider tickets and understand LTH better, which can be classified into four main classes: (1) proposing a stronger lottery ticket hypothesis, (2) exploring the transferability of LTH, (3) generalizing LTH to other contexts, (4) theoretical justification, and (5) revisiting and questioning LTH.

(1) Some recent works [106, 107, 108]) prove stronger hypothesis than LTH [47]. For example, Diffenderfer and Kailkhura [108] propose a stronger Multi-Prize LTH, which claims that winning tickets can be robust to extreme forms of quantization (i.e., binary weights and/or activations). Based on this, they propose a Multi-Prize Tickets (**MPTs**) algorithm to find MPIs on binary neural networks for the first time.

(2) Some literature ([67, 109, 110, 111]) studies the transferability of a winning ticket found in a source dataset to another dataset, which provides insights into the transferability of LTH. For example, S. Morcos et al. [109] find OneTicket that can generalize across a variety of datasets and optimizers within the natural image domain. Mehta [110] propose the ticket transfer hypothesis and transfer winning tickets for different image classification datasets.

(3) In addition to image classification, LTH is extended to numerous other contexts, such as node classification and link prediction ([67]) and vision-and-language ([111]). For example, Chen et al. [67] pioneer to generalize LTH to Graph Neural Networks (**GNNs**) [112] and propose Graph Lottery Ticket (**GLT**).

(4) On one hand, some literature ([117, 118]) analyzes the reasons why LTH [47] is able to win. For example, Zhang et al. [117] exploit dynamical systems theory and inertial manifold to theoretically verify the validity of LTH. Evci et al. [102] observe that the success of LTH lies in effectively re-learning the original pruning solution they are derived. Zhang et al. [118] pioneer to provide formal justification of the improved generalization of winning tickets observed from experimental results in LTH.

(5) On the other hand, some recent works ([119, 38, 120]) revisit and challenge the existence of LTH. For example, Ma et al. [119] provide a more rigorous definition of LTH for precisely identifying winning tickets and find that whether and when the winning tickets can be identified highly replies on the training settings, such as learning rate, training epochs, and the architecture characteristics, like network capacities and residual connections. It is more likely to find winning tickets by using a small learning rate or an insufficient number of training epochs.

It is worth pointing out that in some works [16, 34], LTH [47] is classified as a PBT method. However, LTH selects masks based on a pretrained network, which does not conform to the definition of PBT that attempts pruning the initialized network before training. Therefore, it is more reasonable to classify LTH as a PAT method.

#### 4.3.2 Other score-based Methods

The most straightforward and intuitive way to select pruning candidates is to evaluate them based on their norms. For example, Han et al. [11] propose to measure the weight importance by its absolute value. Li et al. [69] score every filter in each layer by calculating the sum of its weights' \(l_{1}\)-norm. In addition to norm-based criteria, evaluating loss change with and without the weights is also popular. For example, Nonnenmacher et al. [50] propose Second-order Structured Pruning (**SOSP**) to selectively zero out filter masks to minimize the effects of the loss change from removing some filters. Discrimination-aware Channel Pruning (**DCP**) [121] selects the most discriminative channels by minimizing a joint loss which includes the regular loss with the discrimination-aware loss. Many works ([14, 58, 113, 116]) evaluate the importance of weights (or filters, neurons, etc.) locally within each layer. The key drawback of these methods is that the local ranking makes it hard to decide the overall optimal sparsity, and the pre-defining prune ratio per layer may be non-trivial and sub-optimal. Some works ([122, 123, 43]) are presented to cope with this problem. For example, Chin et al. [122] proposes Learned Global Ranking (**LeGR**) to learn the filter ranking globally across layers. Liu et al. [71] propose Group Fisher Pruning (**GFP**) to apply the Fisher information to evaluate the importance of a single channel and coupled channels.

\begin{table}
\begin{tabular}{l|c c} \hline \hline Method & Object function/Criteria & U/S \\ \hline Auto-Balance (2018) [113] & score(\(\mathcal{P}_{i,j}\))=\(|\mathcal{F}_{i,j}|\)\(|_{1}\) & S \\ LTH (2019) [47] & score(\(w_{i}\))=\(\|w_{i}\|_{1}\) & U \\ Taylor-CR-BN (2019) [114] & score(\(w_{i}\))=\((\nabla_{w_{i}}\!\!
#### 4.3.3 Sparsity Regularization based Methods

Some works ([125, 124, 126, 124]) exploit sparsity regularization technique. For example, He et al. [124] propose an alternative two-step algorithm that introduces a scalar mask to each channel of a pretrained CNN model and selects redundant channels based on LASSO regression [127] and reconstructs the outputs with unpruned channels with linear least squares. Energy-Constrained Compression (ECC) [128] builds the energy consumption model via a bilinear regression function. Network Pruning via Performance Maximization (**NPPM**) [129] trains a performance prediction network and uses it as a proxy of accuracy to guide searching for subnetworks based on regularization penalty. Fang et al. [44] develop a general method called **DepGraph** to analyze dependencies of various network structures (e.g., CNNs, RNNs, GNNs, Transformers) and propose a structured pruning based on sparsity regularization. Some methods ([44, 58, 113, 116]) select important weights (or filters, neurons, etc.) by combining norm-based criteria and sparsity regularization.

#### 4.3.4 Pruning in Early Training

Instead of fully training a network from \(f(\mathbf{x};\mathbf{w}_{0})\) to \(f(\mathbf{x};\mathbf{w}_{T})\), this class of methods explore the network architecture by training a network only for a few iterations or epochs, i.e., \(f(\mathbf{x};\mathbf{w}_{t})\), where \(t<<T\). For example, You et al. [130] propose Early-Bird (**EB**) tickets which indicate that winning tickets can be identified at the early training stage via inexpensive training schemes (e.g., early stopping and low-precision training) at large learning rates and achieve similar performance to the dense network. Inspired by [130], Chen et al. [131] propose **EarlyBERT** that identifies structured winning tickets in the early stage of BERT [132] training. Frankle et al. [53] find that, in large-scale settings (such as ResNet-50 and Inception-v3 on ImageNet), the subnetworks that exhibit stability to SGD noise are able to reach full accuracy early in training.

#### 4.3.5 Post-Training Pruning

In contrast to the general PAT methods that follow the Pretrain-Prune-Retrain procedure, recently proposed post-training pruning simplifies the three-step process as Pretrain-Prune. It involves pruning a pre-trained model \(f(\mathbf{x};\mathbf{w}_{T})\) without retraining, with a negligible accuracy loss. This class of pruning methods is particularly attractive for billion-parameter models because retraining such pruned models after pruning is still very expensive. For example, Frantar and Alistarh [15] propose an unstructured post-training pruning method called **SparseGPT**. By simplifying the pruning problem to an approximate sparsity regression, the authors prune GPT family models at least 50% sparsity at minor accuracy loss without retraining. To our best knowledge, it is the first pruning method specifically designed for GPT family models. Kwon et al. [133] propose a structured post-training pruning framework for Transformers, which includes three steps: Fisher-based mask search, Fisher-based mask rearrangement, and mask tuning. Without retraining, this framework can prune Transformers on one GPU in less than 3 minutes.

### _Run-time Pruning_

The prior works on pruning usually focus on static pruning methods where the pruned model is reused for different inputs. In contrast, some methods prune neural networks according to individual inputs dynamically, namely runtime pruning [134]. This line of work is based on the premise that for a given task, the difficulty of producing accurate output can vary, implying the necessity of model capacities for different inputs is different [51]. For example, Rao et al. [134] propose a Runtime Network Routing (**RNR**) framework to conduct dynamic routing based on the input image and current feature maps and select an optimal path subset for compression. Tang et al. [51] point out that the importance of channels highly depends on the input data and propose to generate different subnetworks for each instance. At inference, only channels with saliencies larger than the threshold need to be computed, and the redundant features are skipped. Hua et al. [135] exploit input-specific characteristics and propose **CGNets** to predict the unimportant regions by the partial sum of the output activation by performing convolution on a subset of input channels. Gao et al. [136] propose Feature Boosting and Suppression (**FBS**) to predict the saliency of channels and skip those with less contribution to the classification results at run-time. Fire Together Wire Together [33] poses the dynamic model pruning as a self-supervised binary classification problem. Meng et al. [137] propose Contrastive Dual Gating (**CDG**) that is another self-supervised dynamic pruning method by using contrastive learning [138].

Fig. 6: The illustration of typical sparsity changes for PBT, PDT, and PAT (best view in color and zoom in). PBT is usually the cheapest, and PAT is the most expensive.
## 5 Pruning Criteria

In this section, we summarize some commonly used pruning criteria that are used to evaluate the importance of weights (or filters, neurons, etc.) from different perspectives, including magnitude ([139, 140, 141]), norm ([69, 86]), saliency and/or sensitivity ([142, 143, 85, 13]), loss change ([144, 50, 71, 13, 13]). There is no rigid boundary between these criteria but a different emphasis.

### _Magnitude-based Pruning_

[26] is one of the earliest works that propose magnitude-based pruning to reduce hidden units. Han et al. [11] popularize magnitude-based pruning for deep neural networks, which prunes the lowest-magnitude weights. It is based on the assumption that the weights with smaller absolute values tend to have the least influence on the network's output. The formulation is defined as

\[m_{i}=\begin{cases}1:if\ \|w_{i}\|_{1}\geq a\\ 0:if\ \|w_{i}\|_{1}<a\end{cases}, \tag{3}\]

where \(a\) is a threshold.

Magnitude-based criteria can be applied for either unstructured ([147, 146, 145, 147, 139, 10]) or structured pruning ([69, 148, 94]). For example, Li et al. [69] score the filters by calculating the sum of their absolute magnitude of weights. In addition, magnitude-based criteria can be combined with global/local, one-shot/iterative schedules. For example, the works in [149] and [150] propose magnitude-based iterative global pruning methods. Singh Lubana and P. Dick [140] argue that magnitude-based pruning results in faster model convergence than magnitude-agnostic methods.

### _\(l_{p}\) Norm_

Some methods ([69, 86]) use the \(l_{p}\) norm to evaluate the importance of weights (or filters, channels, etc.). For example, He et al. [86] exploit \(l_{p}\) norm to evaluate the importance of the filter \(\mathcal{F}_{i,j}\) as Eq.(4).

\[\|\mathcal{F}_{i,j}\|_{p}=\left(\sum_{n=1}^{c_{in}^{(i)}}\sum_{k_{1}=1}^{k^{( i)}}\sum_{k_{2}=1}^{k^{(i)}}|\mathcal{F}_{i,j}(n,k_{1},k_{2})|^{p}\right)^{ \frac{1}{p}}, \tag{4}\]

where \(k^{(i)}\) is the kernel size of layer \(i\) in a network and \(c_{in}^{(i)}\) is the number of channels at layer \(i\). The filters with small \(l_{p}\) norm are more likely pruned than those of higher \(l_{p}\) norm. Some norm-based pruning methods ([69]) use \(l_{1}\) norm as the importance metric. In this case, these methods also belong to magnitude-based pruning methods. In addition, the importance value is often optimized with norm-based sparsity regularization ([113]) that is discussed in 6.1.

### _Sensitivity and/or Saliency_

Some works ([143, 85, 151]) utilize sensitivity and/or saliency to evaluate the importance of weights (or channels, filters, etc.). The definition of sensitivity or saliency may be different in prior works. For example, LeCun et al. [151] define weight saliency as the loss change induced by pruning that weight. SNIP [46] proposes a saliency criterion called connection sensitivity criterion as the normalized magnitude of the derivatives \(g_{j}\):

\[s_{j}(\mathbf{w};\mathcal{D})=\frac{|g_{j}(\mathbf{w};\mathcal{D})|}{\sum_{k= 1}^{m}|g_{k}(\mathbf{w};\mathcal{D})|}, \tag{5}\]

where \(s_{j}\) is the sensitivity of the weight \(j\), \(\mathbf{w}\) is the networks weights, \(g_{j}\) is the derivative of the loss \(L(\mathbf{w}\odot\mathbf{m})\) w.r.t. \(m_{j}\). The higher the sensitivity of weight, the more important it is. VCP [85] reformulates the BN layer by extending the scale factor \(\gamma\) on shift term \(\beta\) that is treated as channel saliency. They reformulate BN as follows:

\[\mathbf{x}_{out}=\gamma\cdot BN(\mathbf{x})+\tilde{\beta}, \tag{6}\]

where \(\tilde{\beta}=\gamma\cdot\beta\). Rather than relying on the value of \(\gamma\), VCP prunes unimportant channels based on \(\gamma\)distributions.

### _Loss Change_

It is a popular criterion to measure the importance of weight (or filter, channel, etc.) by evaluating the loss change of the network with and without it. For simplicity, we take filter importance evaluation as an example. Since reevaluating the whole network's performance after removing filters one by one is nontrivial, some researchers ([143, 50, 71]) approximate a filter's importance in a Taylor expansion based way.

**The first-Order Taylor expansion** is the most commonly used method for measuring the loss change. The loss change with a small perturbation at \(\mathbf{w}\) is defined as follows:

\[\Delta\mathcal{L}=\mathcal{L}(\mathbf{w}+\Delta\mathbf{w})-\mathcal{L}( \mathbf{w})=\nabla_{\mathbf{w}}\mathcal{L}\ \Delta\mathbf{w}. \tag{7}\]

For example, GBN [13] introduces the scaling factors \(\boldsymbol{\lambda}\) to the BN and exploits the first-order Taylor expansion to estimate the loss change \(\Delta\mathcal{L}\) caused by setting some scaling factors to zero as follows:

\[\Delta\mathcal{L}(\boldsymbol{\lambda})=|\boldsymbol{\lambda}\nabla_{\boldsymbol {\lambda}}\mathcal{L}-R_{1}(\boldsymbol{\lambda})|\approx|\boldsymbol{\lambda} \nabla_{\boldsymbol{\lambda}}\mathcal{L}|=\left|\frac{\partial\mathcal{L}}{ \partial\boldsymbol{\lambda}}\boldsymbol{\lambda}\right|, \tag{8}\]

where \(R_{1}(\boldsymbol{\lambda})\) is the Lagrange remainder. The importance score of the \(i\)-th filter is defined as

\[\text{Score}(\mathcal{F}_{i})=\sum_{(\mathbf{x},\mathbf{y})\in\mathcal{D}} \left|\frac{\partial\mathcal{L}(\mathbf{y},f(\mathbf{x};\mathbf{w})}{\partial \lambda_{i}}\lambda_{i}\right|, \tag{9}\]

where \(\lambda_{i}\) is the scalar factor of the \(i\)-th filter.

**The second-order Taylor expansion** of the loss function is early used in [151] for removing unimportant weights and gradually exploited in many subsequent methods ([50, 71]), which includes the first-order (gradient) term, the second-order (Hessian) term, and the higher-order terms are neglected. Without loss of generality, the approximation of the loss change leads to

\[\mathcal{L}(\mathbf{w}+\Delta\mathbf{w})-\mathcal{L}(\mathbf{w})=\nabla_{ \mathbf{w}}\mathcal{L}\ \Delta\mathbf{w}+\frac{1}{2}\Delta\mathbf{w}^{T}H\Delta\mathbf{w}, \tag{10}\]

where \(H=\nabla_{\mathbf{w}}^{2}\mathcal{L}(\mathbf{w})\).

For example, GFP [71] applies the second-order Taylor expansion to approximate the loss change when removing a channel (setting its mask to 0):

\[\begin{split} s_{i}&=\Delta\mathcal{L}=\mathcal{L}( \mathbf{m}-\mathbf{e}_{i})-\mathcal{L}(\mathbf{m})\approx-\mathbf{e}_{i}^{T} \nabla_{\mathbf{m}}\mathcal{L}+\frac{1}{2}\mathbf{e}_{i}^{T}(\nabla_{ \mathbf{m}}^{2}\mathcal{L})\mathbf{e}_{i}\\ &=-\mathbf{e}_{i}^{T}g+\frac{1}{2}\mathbf{e}_{i}^{T}H\mathbf{e}_{ i}=-g_{i}+\frac{1}{2}H_{ii},\end{split} \tag{11}\]
where \(\mathbf{e_{i}}\) is the one-hot vector of which the \(i\)-th entry equals one, \(g\) is the gradient of loss function \(\mathcal{L}\) w.r.t. \(\mathbf{m}\).

## 6 Learn to Prune

In this section, we present some methods through learning to prune networks, including sparsity regularization ([62, 64, 124, 152, 153]) and some pruning methods based on meta-learning ([87, 152]), graph neural network ([154]), and reinforcement-learning ([134, 43]).

### _Sparsity Regularization based Pruning_

Sparsity regularization based pruning learns the weights and their masks by solving the following problem:

\[\min_{\mathbf{w},\mathbf{m}}\mathcal{L}(\mathbf{w},\mathbf{m}), \tag{12}\]

where \(\mathcal{L}=\ell(\mathbf{w},\mathbf{m})+\lambda\mathcal{R}(\cdot)\). One common way of this class of pruning methods is introducing scaling factor vector \(\boldsymbol{\gamma}\) for weights (or channels, filters, etc.). The network weights and the scaling factors \(\boldsymbol{\gamma}\) are trained jointly with sparsity regularization imposed on the latter. The magnitude of the scaling factors is treated as the important scores. Specifically, \(\mathcal{L}\) in Eq. 12 can be exemplified as follows:

\[\mathcal{L}=\frac{1}{N}\sum_{i=1}^{N}\ell(\mathbf{y}_{i},f(\mathbf{x}_{i}; \mathbf{w},\boldsymbol{\gamma}))+\lambda\sum_{\gamma_{i}\in\boldsymbol{ \gamma}}\mathcal{R}(\gamma_{i}). \tag{13}\]

For example, He et al. [124] cast channel selection as reconstruction error minimization of feature maps and formulate the channel pruning problem as follows:

\[\begin{split}&\min_{\beta,\mathbf{w}}\frac{1}{2N}\|\mathbf{y}- \sum_{i=1}^{c}\beta_{i}X_{i}W_{i}^{T}\|_{F}^{2}+\lambda\|\beta\|_{1},\\ &\text{subject to }\|\beta\|_{0}\leq c^{\prime},\forall i\;\|W_{i} \|_{F}=1.\end{split} \tag{14}\]

where \(\|\cdot\|_{F}\) is Frobenius norm, \(X_{i}\) is \(N\times k_{h}k_{w}\) matrix from \(i\)-th channel of input \(\mathbf{x}\), \(W_{i}\) is \(n\times k_{h}k_{w}\) weights from \(i\)-th channel of \(\mathbf{w}\), \(k_{h}\) and \(k_{w}\) are the kernel height and width, respectively. \(N\), \(c\), \(c^{\prime}\), and \(n\) are the number of samples, channels, retained channels, and filters, respectively. To solve this problem, He et al. [124] use LASSO regression [127] and greedy strategy to select the unimportant channels.

### _Meta-Learning based Pruning_

Some works ([87, 152]) adopt meta-learning to prune models. For example, MetaPruning [87] trains a meta network, PruningNet, to predict weights for different pruned networks. The PruningNet takes a network encoding vector \((v_{1},v_{2},...,v_{L})\) as input and outputs the weights \(\mathbf{w}\) of the pruned network:

\[\mathbf{w}=\text{PruningNet}(v_{1},v_{2},...,v_{L}), \tag{15}\]

where \(v_{i}\) is the number of the channels for \(i\)-th layer. The weights and the corresponding accuracy of each pruned network are obtained by inputting the network encoding into the fully trained PruningNet. Considering the huge search space of network encoding vectors, the pruned network is found by evolutionary search under the constraints.

### _Graph Neural Network based Pruning_

Any network can be viewed as a graph. GraphPruning [154] applies graph convolution to model compression for the first time. Specifically, GraphPruning designs a graph aggregator \(G\) with weights \(\theta_{G}\) combined with the Fully Connected (FC) layers to generate the weights \(\mathbf{w}=(\mathbf{w}^{(1)},\mathbf{w}^{(2)},...,\mathbf{w}^{(L)})\) of the Pruned Network as follow:

\[\begin{split}&(n_{1},n_{2},...,n_{L})=G(b_{1},b_{2},...,b_{L}| \theta_{G}),\\ &\mathbf{w}^{(i)}=FC_{i}(n_{i}|\theta_{i}),\end{split} \tag{16}\]

where \(b_{i}\in R^{1\times 7}\) denotes the embedding features of \(i\)-th node, \(n_{i}\) is the \(i\)-th column of output with the graph aggregator, \(\theta_{i}\) is the weights of the \(i\)th FC layer, and \(\mathbf{w}^{(i)}\) is the weights of the \(i\)-th pruned layer of the pruned network. Then the pruned network is fully trained on the dataset. Herein, the graph aggregator is responsible for extracting high-level features for each node, while each FC layer is used to generate reasonable weights for the pruned network. Afterward, the best configuration of the pruned network under computational constraints is searched by RL methods, during which the weights of the graph aggregator and FCs are not updated.

### _Reinforcement Learning based Pruning_

Rather than using RL to search the best configurations of the pruned networks in [154], some AutoML pruning methods ([134, 133]) adopt RL to compress models automatically. For example, He et al. [43] propose AutoML for model compression (**AMC**) method that is based on Q-learning, a kind of RL, to concern how an agent should take actions in order to maximize the cumulative reward. Specifically, He et al. [43] design the Deep Deterministic Policy Gradient (**DDPG**) agent to receive an embedding state \(s_{i}\) of layer \(l_{i}\) from the environment and output a sparsity ratio as action \(a_{i}\). Then layer \(l_{i}\) is compressed with \(a_{i}\) using a specific compression method (such as a channel pruning method). After that, the agent moves to layer \(l_{i+1}\) and does the same work as layer \(l_{i}\) until the final layer \(l_{L}\). The update process is as follows:

\[\begin{split} Loss=\frac{1}{N}\sum_{i=1}^{N}(y_{i}-Q(s_{i},a_{i}| \mathbf{w}^{Q}))^{2},\\ y_{i}=r_{i}-b+\gamma Q(s_{i+1},\mu\;(s_{i+1})|\mathbf{w}^{Q}), \end{split} \tag{17}\]

where \(b\) is the baseline reward, \(\gamma\) is a discount factor used to avoid over-prioritizing short-term rewards, \(\mathbf{w}^{Q}\) is the weights of the network \(Q\) following Block-QNN [155], and \(r_{i}\) is the reward of the whole trajectory for the \(i\)-th sample.

He et al. [43] observe that Error is inversely-proportional to \(log(\text{FLOPs})\) or \(log(\#\text{Param})\). Based on this observation, the reward function is defined as:

\[\begin{split}&\text{R}_{\text{FLOPs}}=-\text{Error}\cdot log( \text{FLOPs}),\\ &\text{R}_{\text{Param}}=-\text{Error}\cdot log(\#\text{Param}). \end{split} \tag{18}\]

This reward function provides an incentive for reducing FLOPs or the number of network parameters.

## 7 A Comprehensive Comparative Analysis

In this section, we focus on a comparative analysis of how to behave in different pruning manners, including seven pairs
of contrast settings for pruning, the different layer-wise densities of networks, and different levels of supervision for pruning. We also conduct comparative experiments to examine some pruning methods for VGGNet [2]/ResNet [6] on CIFAR-10 [156]/CIFAR-100 [156] under different contrast settings. The experimental settings refer to Appendix A.

### _Unstructured vs. Structured Pruning_

Unstructured pruning methods ([70, 11]) remove weights anywhere and can achieve high prune ratios with little impact on accuracy. In contrast, structured pruning ([13, 50, 71]) conducts pruning at entire filters (or channels, neurons, blocks, layers, etc.), which results in really compressed network and accelerated inference, but the accuracy is more likely lower than that of unstructured pruning under the same prune ratio, weight-level scoring, pipeline, and learning schemes. The possible reason is that unstructured pruning only focuses on the importance of a single weight, while structured pruning forces structural coupling, which demands the simultaneous pruning of multiple layers and expects all removed weights to be consistently unimportant. However, meeting the consistency of unimportant weights under the structural coupling constraints is challenging. The advantages and disadvantages of unstructured and structured pruning are briefly summarized in Table VI. We prune VGG-16 on CIFAR-10 [156] in 3 random runs and report the best results to compare unstructured and structured pruning of SNIP [46] and GraSP [61]. The vanilla SNIP and GraSP are unstructured pruning, denoted by SNIP-unstruct and GraSP-unstruct in Table VII, respectively. To prune channels, after scoring each weight by vanilla SNIP or GraSP, the sum of the weight-level scores in each channel is calculated as its importance score. As shown in Table VII, for SNIP and GraSP, Top-1 accuracy of unstructured pruning is generally better than their corresponding structured pruning under the same prune ratio of weights.

### _One-shot vs. Iterative Pruning_

One-shot pruning methods score once and then prune the network to a target prune ratio. Conversely, the iterative pruning methods alternately process the score-prune-retrain cycle until achieving the target prune ratio. As a result, the pruning cost in one-shot methods usually can be negligible, which greatly saves pruning efforts. However, this kind of method is not friendly to those significant weights whose importance is not immediately apparent at the beginning [109]. Hence, one-shot pruning generally requires more carefully designed scoring criteria to match the performance of the original network. In addition, the results in [39] show that one-shot pruning may more easily suffer from layer-collapse, resulting in a sharp accuracy drop. In contrast, iterative methods require more pruning cost but generally yield better accuracy [13, 157, 110, 69, 120].

Lin et al. [97] analyze the difference between one-shot and iterative pruning methods from the perspective of stochastic gradient. Their results show that the iterative pruning method computes a stochastic gradient at the pruned model and takes a step that best suits the compressed model. In contrast, one-shot pruning method computes a stochastic gradient at the original weights and moves towards the best dense model.

We prune VGG-16 and ResNet-32 on CIFAR-10 by SynFlow [39] and Magnitude-based pruning (For simplicity, it will be referred to as Magnitude in the following.) under one-shot or iterative pruning, respectively. As illustrated in Fig. 7, Top-1 accuracy of iterative pruning is generally better than the corresponding one-shot pruning.

### _Data-free vs. Data-driven Pruning_

Most of the existing pruning works ([71, 50, 46]) belong to data-driven methods, and only a few methods ([39, 60, 158, 159]) are data-free. Data is generally believed to be essential for finding good subnetworks. We prune VGG-16 on CIFAR-10 and SVHN by two data-free pruning methods (i.e., SynFlow [39] and Smart-Ratio [60]) and two data-driven methods (i.e., SNIP [46] and Magnitude). Both CIFAR-10 and SVHN are 10-class datasets. As shown in Fig. 8, the prune ratio per layer of VGG-16 by SynFlow or Smart-Ratio on CIFAR-10 is the same as the prune ratio of the corresponding layer on SVHN. However, the prune ratio per layer differs when SNIP or Magnitude prunes VGG-16 on CIFAR-10 and SVHN. Especially for Magnitude, the prune ratios of lower layers on CIFAR-10 are different from those on SVHN. The results in Table VIII are the best in three random runs and show that although data-free methods achieve good Top-1 accuracy on CIFAR-10 [156], SVHN

\begin{table}
\begin{tabular}{l|c|c} \hline  & Unstructured & Structured \\ \hline High sparsity with & \(\blacktriangledown\) & hard \\ minor accuracy drop & \(\blacktriangledown\) & \(\blacktriangledown\) \\ \hline Speedup w/o specific hardware & & \\ (e.g., FPGAs or ASICs) & hard & \\ \hline Speedup w/o specific software & & \\ (e.g., sparsify CNNs libraries) & hard & \\ \hline Really compressed with & & \\ significant acceleration & hard & \\ \hline Structure coupling & \(\bigstar\) & \(\bigstar\) \\ \hline \end{tabular}
\end{table} TABLE VI: Advantages and disadvantages of unstructured and structured pruning.

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline Dataset & CIFAR-10 & CIFAR-100 \\ \hline Parameters \(\downarrow\) (\%) & 10 & 40 & 10 & 50 \\ \hline SNIP-unstruct [46] & **93.34** & **93.73** & **73.09** & **72.67** \\ SNIP-struct & 93.74 & 93.71 & 72.82 & 70.68 \\ \hline GraSP-unstruct [61] & 93.58 & **93.18** & **72.46** & **71.42** \\ GraSP-struct & **93.67** & 93.04 & 72.43 & 71.30 \\ \hline \end{tabular}
\end{table} TABLE VII: Top-1 accuracy of unstructured and structured pruning for VGG-16. The best results are shown in **bold**.

Fig. 7: One-shot vs. iterative pruning on CIFAR-10. Shaded regions indicate standard deviation based on three independent runs (best view in color).
and CIFAR-100 [156] with 50% or 90% parameters pruned, the data-driven methods outperform in most cases. Among these methods, SNIP performs best. Although it only uses a small number of randomly chosen samples (i.e., one sample per class) to calculate weight scores, the data is essential for SNIP. When it gets random inputs, as denoted by SNIP (data-free), its Top-1 accuracy has generally dropped.

### _Pruning on Initialized vs. Pretrained Weights_

Frankle et al. [48] find the subnetworks obtained by pruning on randomly initialized weights (such as SNIP [46], GraSP [61], SynFlow [39]) are robust to the ablation treatments (i.e., randomly shuffling the mask positions within each layer or reinitializing weights while keeping masks unchanged). To find the reason behind such immunity, Singh and Liu [160] use the Wasserstein distance to measure distributional similarity and find that the remained weights' distribution is changed minimally with these ablations, which reinforces keeping similar performances. In contrast, Su et al. [60] find the subnetworks achieved by pruning on pretrained weights, such as LTH [47], are sensitive to these ablations. Qiu and Suda [161] claim that training weights can be decoupled into two dimensions: the locations of weights and their exact values, among which the locations of weights hold most of the information encoded by the training. Wolfe et al. [162] theoretically analyze the impact of pretraining on the performance of a pruned subnetwork. Although pretraining is crucial for PAT methods, it does not always bring benefits. We conduct some experiments to explore whether the pretrained weights facilitate PBT methods to find better subnetworks. The results in Fig. 9 show that for both SynFlow [39] and SNIP [46], pruning on the pretrained weights does not guarantee improved Top-1 accuracy. The results are worse for some prune ratios of weights.

### _Training from Scratch vs. Fine-tuning_

After pruning, many pruning methods require training the subnetwork for several epochs to regain performance. Le and Hua [163] argue that retraining is essential to recover loss accuracy in pruning. Generally, retraining can be divided into two types: training from scratch or fine-tuning. There has been debate over whether fine-tuning is more powerful than training from scratch to recover accuracy. On the one hand, Liu et al. [38] find that for ResNet, VGG, and other standard structures on ImageNet, training the subnetworks with new random initialization can achieve better performance than fine-tuning them. On the other hand, Li et al. [69] observe that training a subnetwork from scratch performs worse than fine-tuning it. Liu et al. [120] investigate pruning ResNet20 on CIFAR-10 with ADMM-based [164] one-shot pruning method and find that pruning & fine-tuning outperforms LTH (pruning & training from scratch) over various prune ratios. In addition, the results in [129, 165] show that fine-tuning is necessary to achieve better performance on sparse mobile networks than training from scratch. In recent years, some compromise methods (such as weight rewinding [53]) have been proposed. The results in [53, 40] show that weight rewinding can achieve higher accuracy than fine-tuning. We prune ResNet-32 on CIFAR-32 by Magnitude and SNIP [46], and then train the pruned network from scratch or fine-tune it. The results in Fig. 10 show that fine-tuning outperforms training from scratch for Magnitude and SNIP.

### _Original Task vs. Transfer Pruning_

In recent literature, pruning is combined with transfer learning [166] that can dramatically improve accuracy and speed up convergence. For ease of distinction, in this survey, the original task pruning is used to denote the whole pruning pipeline directly performing the target task. In contrast, transfer pruning performs on the source task and then transfers the subnetwork to the target task. Specifically, transfer pruning is divided into two types: dataset transfer and architecture transfer. The former prunes networks on the source dataset and transfers the subnetwork to the target dataset, and the latter prunes on one architecture and transfers the subnetwork to another.

Some works ([110, 123, 167, 168]) study the transferability of sparsity masks across datasets. For example, Tiwari

Fig. 8: An illustration of the architectures of VGG-16 with 90% prune ratio of weights through data-free and data-driven pruning on CIFAR-10 and SVHN.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline Dataset & \multicolumn{2}{c|}{CIFAR-10} & \multicolumn{2}{c|}{SVHN} & \multicolumn{2}{c}{CIFAR-100} \\ \hline Parameters \(\downarrow\) (\%) & 50 & 90 & 50 & 90 & 90 & 95 \\ \hline Smart-Ratio [60] & 93.71 & 93.20 & 96.34 & 96.23 & 71.37 & 70.16 \\ SynFlow [39] & 93.94 & 93.26 & 96.28 & 96.34 & 71.03 & 69.16 \\ SNIP (data-free) & **93.90** & 93.20 & 96.37 & 96.29 & 71.46 & 70.23 \\ \hline SNIP [46] & 93.70 & **93.40** & **96.39** & **96.60** & **71.65** & 70.32 \\ Magnitude & 93.65 & 93.18 & 96.23 & 96.26 & 71.15 & **70.35** \\ \hline \end{tabular}
\end{table} TABLE VIII: Top-1 accuracy (%) of data-free and data-driven pruning methods for VGG-16. The original Top-1 accuracy on CIFAR-10, SVHN, and CIFAR-100 are 93.76%, 96.36%, and 73.22%, respectively. The best results are shown in **bold**.

Fig. 9: Initialized vs. pretrained weights on CIFAR-10. Shaded regions indicate standard deviation based on three independent runs (best view in color).
et al. [123] compare the Top-1 accuracy of the pruned network on CIFAR-100 with the one wherein the masks are learned on Tiny ImageNet and transferred to CIFAR-100. The results show that the transferred masks perform better than those directly obtained from the original task, with 40% of the channels removed. S. Morcos et al. [109] observe that for image classification, winning tickets generated on larger datasets (such as bigger training set size and/or more number of classes) consistently transferred better than those generated with smaller datasets. For example, winning tickets generated on ImageNet and Places365 demonstrate better performance across other smaller target datasets such as CIFAR-10 and CIFAR-100. Iofirova et al. [169] present a pioneer study of the transfer performance of subnetworks and find that pruning methods with similar Top-1 accuracy on ImageNet [1] can have surprisingly different Top-1 accuracy when used for transfer learning. For architecture transfer pruning, Elastic Ticket Transformations (**ETTs**) [42] transforms winning tickets found in one kind of network (ResNet [6], or VGG [2]) to another deeper or shallower one from the same model family.

### _Static vs. Dynamic Pruning_

Static pruning [12] uses static pruning criteria and permanently removes components. In contrast, dynamic pruning ([135, 136, 51, 170]) exploits input-specific characteristic pruning criteria and preserves the entire network structures and accelerates the networks by dynamically skipping unimportant components. However, dynamic pruning generally does not perform run-time fine-tuning or retraining. The difference between static and dynamic pruning is mainly reflected in the pruning criteria and the pruned model. The advantages and disadvantages of static and dynamic pruning are shown in Table IX.

### _Layer-wise Weight Density Analysis_

Some works ([119, 154, 171, 71]) study the distributions of layer-wise weight density of a subnetwork, and their results show that different layers can have very different weight densities. The difference is produced by the joint action of structural characteristics of the dense network and the pruning methods. Zhang et al. [172] empirically study the heterogeneous characteristic of layers and divide the layers into either "ambient" or "critical". The ambient layers are not sensitive to changing weights to the initial values, while critical layers are. It seems that the ambient layers should be pruned heavily, and the corresponding weight densities will be lower.

Pruning methods also can cause different weight densities. Take pruning for image classification as an example. Some pruning methods tend to assign more weights to earlier layers than later ones. For example, Ma et al. [119] investigate the layer-wise keep-ratios of the subnetworks obtained by GraSP, SNIP [46], LTH on VGG and ResNet, and observe the common trend of declining in layer-wise keep-ratios except for some special layers (such as the downsampling layers in ResNet). In contrast, Liu et al. [71] find their pruned networks keep higher percentages of channels in the deeper layers than those in the lower layers for image classification. The results in [171] show that global magnitude pruning tends to prune layers of Transformer uniformly [173], while global first-order methods heavily prune the deeper layers.

### _Pruning with Different Levels of Supervision_

In descending order of supervision level during neural network pruning, pruning can be divided into supervised, semi-supervised, self-supervised, and unsupervised pruning [174]. Self-supervised learning can be divided into two classes: generative or contrastive learning [175]. Like supervised learning, supervised pruning works on fully labeled datasets. Most of the current pruning methods fall into supervised pruning. However, supervised pruning suffers from similar bottlenecks as supervised learning (such as expensive manual labeling). As a promising alternative, semi-supervised, self-supervised, and unsupervised pruning have drawn massive attention 2.

Footnote 2: The differences between supervised, semi-supervised, unsupervised, self-supervised learning refer to [175].

For example, Caron et al. [176] observe the different results in self-supervision pruning from in supervision pruning, where winning tickets initialization only introduce a slight performance improvement compared to random re-initialization. Pan et al. [177] claim that unsupervised pruning usually fails to preserve the accuracy of the original model. Notably, label supervision for network pruning and training can be independent. For example, Chen et al. [168] use supervised pruning method **IMP** (i.e., Iterative Magnitude Pruning) to explore the subnetworks of self-supervised pretrained models (simCLR [178] and MoCo [179]) on ImageNet. Similarly, Jeff Lai et al. [180] exploit the supervised pruning method IMP to prune self-supervised speech recognition models.

## 8 Fusion of Pruning and other Compression Techniques

In this section, we provide a review of the fusion of neural network pruning with other network compression techniques, such as quantization [11], tensor decomposition

Fig. 10: Training from scratch vs. fine-tuning for ResNet-32. Shaded regions indicate standard deviation based on three independent runs (best view in color).

\begin{table}
\begin{tabular}{l|c c} \hline \hline Type & Advantages & Disadvantages \\ \hline Static Pruning & model size reduced & fixed subnetwork \\ \hline Dynamic Pruning & flexible subnetwork & model size unreduced \\ \hline \hline \end{tabular}
\end{table} TABLE IX: Advantages and disadvantages of static and dynamic pruning.
[144], knowledge distillation [21], and network architecture search [73]. However, on the one hand, fusion provides more choices for network compression. On the other hand, the combined compression techniques can complement each other to improve the performance and prune ratio further.

**Pruning & Quantization:** Quantization [147] is a compression technique to reduce the number of bits used to represent the network weights and/or activations, which can significantly reduce the model size and memory footprint but with a minor performance drop. To obtain more compacted models as well as model acceleration, Han et al. [11] pioneer to prune the redundant network connections and quantize the weights. CLIP-Q [147] jointly perform pruning and quantization during the fine-tuning stage. MPTs [108] integrates pruning and quantizing randomly weighted full-precision neural networks to obtain binary weights and/or activations. EB [130] applies 8-bit low-precision training to the stage of searching EB tickets.

**Pruning & Tensor Decomposition:** Tensor decomposition [19] decomposes the convolutions into a sequence of tensors with fewer parameters. In contrast to pruning, it explores the original weights' low-rank structure, keeping the dimension of the convolutional output unchanged. CC [144] combines channel pruning and tensor decomposition to compress CNN models by simultaneously learning the model sparsity and low rankness. Hinge [181] introduces group sparsity to fuse filter pruning and decomposition under the same formulation.

**Pruning & NAS:** Neural Architecture Search (**NAS**) provides a mechanism to automatically discover the best architecture for the problem of interest, which provides a new idea for pruning to find suitable network depth and width. NPAS [73] performs a compiler-aware joint network pruning and architecture search, determining the filter type (different kernel sizes), the pruning scheme, and the rate for each layer. TAS [182] exploits NAS to search for the depth and width of a network to obtain the pruned networks and uses knowledge distillation to train these pruned networks.

**Pruning & Knowledge Distillation:** Knowledge Distillation (**KD**) [21] guides the student to effectively inherit knowledge from the teacher and mimic the output of the teacher. Some works ([183, 184]) exploit pruning before KD to boost the KD quality. For example, Liu et al. [183] prune unimportant channels to the contents of interest and focus the distillation on the interest regions. Park and No [184] prune the teacher network first to make it more transferable and then distill it to the student. Some works ([185, 66, 135]) use KD to train the pruned networks. The results in [185] show that the pruned network recovered by KD performs better than it regained by fine-tuning. Zou et al. [186] propose a data-free deraining model compression method that distills the pruned model to fit the pretrained model.

**Pruning & Multi-compression Techniques:** Some works ([187, 188]) explore the fusion of pruning with more than one compression technique. For example, GS [189] fuses pruning, quantization, and KD for Generative Adversarial Networks (**GANs**) compression. Joint-DetNAS [188] joins pruning, NAS, and KD for image translation. LadaBERT [187] integrates pruning, matrix factorization, and KD to compress Bidirectional Encoder Representations from Transformers (**BERTs**) [132] for natural language understanding.

## 9 Suggestions and Future Directions

In this section, we discuss how to choose different pruning methods and provide promising directions for future work.

### _Recommendations on pruning method selection_

After years of research and exploration, there are many off-the-shelf pruning methods. However, no golden standard exists to measure which one is the best. Different suitable pruning methods exist to compact deep neural networks to the specific application requirements and the hardware and software resources. Here are some general recommendations for choosing an appropriate pruning method.

(1) If you do not have special hardware (e.g., FPGAs or ASICs) or software (such as sparsity convolutional libraries) but need actual neural network acceleration and compression, structured pruning is more suitable than unstructured pruning because most software frameworks and hardware cannot accelerate sparse matrices' computation.

(2) If you have enough computational resources in the pruning stage, iterative PAT methods may be considered. Overall, this class of methods can minimize the impact on performance under the same prune ratio. On the other hand, if you have limited computational resources in both the pruning and inference stages, one-shot PBT or one-shot post-training pruning methods may be considered.

(3) If you have enough labeled examples on the target task, supervised pruning methods may be considered. However, if only a few examples on the target task are labeled, semi-supervised or transfer pruning methods may be considered. If the examples on the target task are not labeled, self-supervised, unsupervised, or transfer pruning methods may be considered.

(4) If you have enough memory footprint during pruning for NLP tasks, heavily compressing large models can be considered rather than lightly compressing smaller models to meet the same budgets. Some results in [190] show that for NLP tasks finding pruned models from large dense networks outperform small dense networks with a size comparable to pruned models.

(5) If you have enough memory footprint to store the dense neural network in the inference stage and hope to provide run-time flexible computational cost allocation for different inputs, dynamic pruning methods can be considered where inputs with smaller shapes can allocate less computational cost to perform the task and if the input shape is bigger more computational cost can be allocated.

(6) If you need to trim down neural networks in multiple dimensions, you can comprehensively consider layerwise pruning (decreasing the model's depth), channel pruning (reducing the model's width), and image resolution pruning (scaling down the model's input resolution). In addition, pruning can be integrated with quantization to further reduce the memory footprint and the neural networks' size.

(7) If you want to reach a better tradeoff between speed and accuracy, the following settings may help: use a pre-trained model; set a more appropriate learning rate (if any) in both the pruning and the retraining stages; fine-tune the pruned models for several epochs; integrate pruning, knowledge distillation, NAS or other compression methods
to achieve complementarity; adversarial training may have some help [111].

(8) If you need to train a subnetwork to recover loss accuracy, the results in [119] show that when residual connections exist, the subnetwork achieves higher accuracy at a relatively small learning rate. In contrast, a larger learning rate is preferable in training a subnetwork without residual connections.

### _Future Directions_

We discuss four promising directions for the further development of neural network pruning, namely, (1) theories, (2) techniques, (3) applications, and (4) evaluation.

**Theories:** Despite the existing works, several fundamental questions about pruning still need to be answered. For example, prior works demonstrate that network layers contain irreplaceable information as long as redundant ones. Does a theoretical upper bound of the prune ratio exist for a given network that still keeps the matching performance of its dense equivalent? In other words, how heavily can a network be pruned theoretically without accuracy loss? It is a tricky question because of the intricate relationships between network layers. Besides, is pruning explainable? A common belief is deep neural network is hard to interpret. As such, making explainable pruning is an uphill task. However, the interpretability of pruning is vital for understanding the factors behind pruning (e.g., model structure and weights) and exploring more effective pruning methods.

**Techniques:** To obtain better algorithm designs whose architectures are learned in an economical, efficient, and effective manner, it is a trend to extend Automated Machine Learning (**AutoML**) methods and NAS to pruning. Furthermore, pruning also begins to combine with various learning contexts, such as lifelong learning [66], continual learning [191], contrast learning [32], federated learning [192], etc. In addition, networks' rising energy consumption requires more attention to energy-aware pruning. However, preliminary efforts mainly focus on reducing computation and memory costs, which may not necessarily reduces the most energy consumption. Besides, incorporating pruning into the hardware to help deploy pruned networks is also an emerging trend. For example, Sui et al. [54] propose a hardware-friendly pruning method and deploy the pruned models on the FPGA platform.

**Applications:** The prior mainstream research targets pruning studies in CV, especially in image classification. Recently, pruning has begun to draw attention to more complex applications, like object tracking, visual question answering, natural language understanding, speech recognition, etc. For example, in recent years, foundation models such as GPT-4 [193] might be the possible way to Artificial General Intelligence (**AGI**). However, its huge model hinders its application in many downstream tasks. Therefore, colossal foundation models can benefit from pruning research to be more compact and efficient [15].

**Evaluation:** With the emergence of many pruning methods, standardized benchmarks and metrics are required to provide a fair evaluation. Different pruning techniques, network architectures, tasks, and experimental settings lead to incomparable results and make it hard to compare pruning methods fairly [194]. ShrinkBench [37] takes the first step and provides a benchmark of pruning methods for image classification. As pruning is applied to applications other than image classification, standardized benchmarks and metrics for other applications are needed.

## 10 Conclusion

As an essential compression technique, deep neural network pruning has attracted increasing research attention with the recent emergence of a wide variety of pruning methods and applications. This survey conducts a comprehensive review on the following four scopes: 1) universal/specific speedup, with a systematic review from unstructured, structured, and semi-structured pruning; 2) when to prune, including pruning before/during/after training for static pruning and run-time pruning; 3) how to prune, including pruning by heuristic criteria and by learning; 4) fusion of pruning with other compression techniques, such as KD, NAS, so on. A comprehensive comparative analysis, including seven pairs of contrast settings for pruning, layer-wise weight density, and different supervision levels, can help researchers to efficiently and effectively grasp the characteristics of different pruning methods. Additionally, recommendations on pruning method selection and future research directions are highlighted and discussed. To facilitate future research, real-world miscellaneous applications and commonly used resources of datasets, networks, and evaluation metrics in different applications are summarized in Appendix B. To help researchers and practitioners keep up with the development of pruning technologies, we continue updating the representative research efforts and open-source codes for pruning at [https://github.com/hrcheng1066/awesome-pruning](https://github.com/hrcheng1066/awesome-pruning).
