Structured Pruning for Deep Convolutional Neural Networks: A survey
The remarkable performance of deep Convolutional neural networks (CNNs) is
generally attributed to their deeper and wider architectures, which can come
with significant computational costs. Pruning neural networks has thus gained
interest since it effectively lowers storage and computational costs. In
contrast to weight pruning, which results in unstructured models, structured
pruning provides the benefit of realistic acceleration by producing models that
are friendly to hardware implementation. The special requirements of structured
pruning have led to the discovery of numerous new challenges and the
development of innovative solutions. This article surveys the recent progress
towards structured pruning of deep CNNs. We summarize and compare the
state-of-the-art structured pruning techniques with respect to filter ranking
methods, regularization methods, dynamic execution, neural architecture search,
the lottery ticket hypothesis, and the applications of pruning. While
discussing structured pruning algorithms, we briefly introduce the unstructured
pruning counterpart to emphasize their differences. Furthermore, we provide
insights into potential research opportunities in the field of structured
pruning. A curated list of neural network pruning papers can be found at
https://github.com/he-y/Awesome-Pruning . A dedicated website offering a more
interactive comparison of structured pruning methods can be found at:
https://huggingface.co/spaces/he-yang/Structured-Pruning-Survey .
Yang He, Lingao Xiao
# Structured Pruning for Deep Convolutional Neural Networks: A survey

###### Abstract

The remarkable performance of deep Convolutional neural networks (CNNs) is generally attributed to their deeper and wider architectures, which can come with significant computational costs. Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. In contrast to weight pruning, which results in unstructured models, structured pruning provides the benefit of realistic acceleration by producing models that are friendly to hardware implementation. The special requirements of structured pruning have led to the discovery of numerous new challenges and the development of innovative solutions. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning. While discussing structured pruning algorithms, we briefly introduce the unstructured pruning counterpart to emphasize their differences. Furthermore, we provide insights into potential research opportunities in the field of structured pruning. A curated list of neural network pruning papers can be found at

[https://github.com/he-y/Awesome-Pruning](https://github.com/he-y/Awesome-Pruning).

 Computer Vision, Deep Learning, Neural Network Compression, Structured Pruning, Unstructured Pruning.

## 1 Introduction

Deep convolutional neural networks (CNNs) have shown exceptional performance in a wide variety of applications, including image classification [1], object detection [2], and image segmentation [3], amongst others. Numerous CNN structures including AlexNet [4], VGGNet [5], Inceptions [6], ResNet [7] and DenseNet [8] have been proposed. These architectures contain millions of parameters and require large computing power, making deployments on resource-limited hardware challenging. Model compression is a solution for this problem, aiming to reduce the number of parameters, computational cost, and memory consumption. As such, its study has gained importance.

To generate more efficient models, model compression techniques including pruning [9], quantization [10], decomposition [11], and knowledge distillation [12] have been proposed. The term "pruning" refers to removing components of a network to produce sparse models for acceleration and compression. The objective of pruning is to minimize the number of parameters without significantly affecting the performance of the models. Most research on pruning has been conducted on CNNs for the image classification task, which is the foundation for other computer vision tasks.

Pruning can be categorized into unstructured [9] and structured pruning [13]. **Unstructured pruning** removes connections (weights) of neural networks, resulting in unstructured sparsity. Unstructured pruning often leads to a high compression rate, but requires specific hardware or library support for realistic acceleration. **Structured pruning** removes entire filters of neural networks, and can achieve realistic acceleration and compression with standard hardware by taking advantage of a highly efficient library such as the Basic Linear Algebra Subprograms (BLAS) library.

Revisiting the property of CNNs from the perspective of structured pruning is meaningful in the era of Transformers [14]. Recently, there has been an increasing trend of incorporating the architectural design of CNNs into Transformer models [15, 16, 17, 18, 19]. Although the self-attention [20] in Transformers is effective in computing a representation of the sequence, an enormous amount of training data is still needed since Transformers lack induction biases [17, 21, 22]. In contrast, the structure of CNNs enforces two key inductive biases on the weights: locality and weight sharing, to improve the generalization of learning algorithms and are independent of data [17]. This survey provides a better understanding of CNNs and provides hints to efficiently design architecture for the future.

In this survey, we focus on structured pruning. Existing surveys on related compression studies are shown in Table I. Some surveys covers orthogonal fields including quantization [23], knowledge distillation [24], and neural architecture search [25]. Some surveys [26] provide a broader overview. Although some surveys focus on pruning, they pay more attention to unstructured pruning and cover a small number of studies on structured pruning. The numbers of structured pruning papers cited by [27, 28, 29, 30, 31, 32, 33] are 1, 11, 15, 55, 38, 10, and 20, respectively. We provide a more comprehensive survey with more than 200 structured pruning papers. For example, [30] can be covered by Section 2.1 2.2 2.3 2.4.1 2.7.1 3.1.

The survey is arranged as follows. In the taxonomy (Fig. 1), we group the structured pruning methods into different categories. Every subsection of Section 2 corresponds
to a category of structured pruning methods. Most methods are first developed in an unstructured manner and then extended to meet structural constraints. Although some studies cover multiple categories, we place them in the most appropriate categories that serve this survey. Section 3 then introduces some potential and promising future directions. Due to the length constraints, only the most representative studies are discussed in detail.

## 2 Methods

**Preliminaries:** A deep convolutional neural network \(\mathcal{N}\) can be parameterized by \(\{\mathbf{W}^{l}\in\mathbb{R}^{N_{l+1}\times N_{l}\times K_{l}\times K_{l}},\,1 \leq l\leq L\}\). At \(l\)-th layer, the input tensor \(I^{l}\) has shape \(N_{l}\times H_{l}\times W_{l}\), and the output tensor \(O^{l}\) has shape \(N_{l+1}\times H_{l+1}\times W_{l+1}\). \(N_{l}\) and \(N_{l+1}\) denote the channel number of input and output tensors in the \(l\)-th layer, respectively. \(\mathbf{W}^{l}\) represents the connections (weights) between the input tensor \(\mathbf{I}^{l}\) and the output tensor (feature map) \(\mathbf{Q}^{l}\). The weight matrix \(\mathbf{W}^{l}\) is made of \(N_{l+1}\) 3-D filters \(\mathbf{F}^{l}\). Specifically, the \(i\)-th filter in \(l\)-th layer can be denoted as \(\{\mathbf{F}^{l}_{i}\in\mathbb{R}^{N_{l}\times H_{l}\times W_{l}},1\leq i\leq N _{l+1}\}\). In this paper, we call \(\mathbf{K}^{l}\in\mathbb{R}^{K_{l}\times K_{l}}\) 2-D kernels, so a filter has \(N_{l}\) kernels of kernel size \(K_{l}\). To express a single weight, we use \(\left\{w=\mathbf{F}^{l}_{i}(n,k_{1},k_{2}),1\leq n\leq N_{l},1\leq k_{1},k_{2 }\leq K_{l}\right\}\). A typical convolution operation of \(l\)-th layer can be expressed as:

\[\mathbf{O}^{l}_{i}=\mathbf{I}^{l}*\mathbf{F}^{l}_{i}\ \text{\bf for}\ 1\leq i\leq N_{l+1} \tag{1}\]

where \(*\) denotes the convolution operator.

Structured pruning, such as filter pruning, aims to:

\[\begin{split}\min_{\mathbf{F}}\mathcal{L}(\mathbf{F};\mathcal{ D})=\min_{\mathbf{F}}\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\left(\mathbf{F};( \mathbf{x}_{i},\mathbf{y}_{i})\right),\\ \text{s.t.}\ \ \mathrm{Card}(\mathbf{F})\leq\kappa,\end{split} \tag{2}\]

where \(\mathcal{L}(\cdot)\) is the loss function (e.g., cross-entropy loss) and \(\mathcal{D}=\left\{(\mathbf{x}_{i},\mathbf{y}_{i})\right\}_{i=1}^{N}\) is a dataset. \(\mathrm{Card}(\cdot)\) is the cardinality of the filter set, and \(\kappa\) is the target sparsity level such as the number of remaining nonzero filters.

### _Weight-Dependent_

Weight-dependent criteria evaluate the importance of filters based on the weights of filters. Compared to activation-based methods, weight-dependent methods do not involve input data. As such, weight-dependent methods are considered straightforward and require lower computational costs. There are two subcategories of weight-dependent criteria: **filter norm** and **filter correlation**. Calculating the norm of a filter is done independently of the norm of other filters, while calculating filter correlation involves multiple filters.

#### 2.1.1 Filter Norm

In contrast to unstructured pruning which uses the magnitude of the weights as the metric, structured pruning computes the filter norm values to be the metric. The \(\ell_{p}\)-norm of a filter can be written as:

\[\left\|\mathbf{F}^{l}_{i}\right\|_{p}=\sqrt[]{\sum_{n=1}^{N_{l}}\sum_{k_{1}=1 }^{K_{l}}\sum_{k_{2}=1}^{K_{l}}\left|\mathbf{F}^{l}_{i}\left(n,k_{1},k_{2} \right)\right|^{p}}, \tag{3}\]

where \(i\in N_{l+1}\) represents the \(i\)-th filter in \(l\)-th layer, \(N_{l}\) is the input channel size, and \(K_{l}\) is the kernel size. \(p\) is the order of the norm, and the most common norms are \(\ell_{1}\)-norm (Manhattan norm) and \(\ell_{2}\)-norm (Euclidean norm).

Pruning Filter for Efficient ConvNets (PFEC) [13] first computes the importance of filters based on \(\ell_{1}\)-norm. Li _et al._ consider filters with smaller norms to have a weak activation and contribute less to the final classification decision [13].

Soft Filter Pruning (SFP) [115] empirically finds that \(\ell_{2}\)-norm works slightly better than \(\ell_{1}\)-norm. This is discussed further in Section 2.5.1.

#### 2.1.2 Filter Correlation

Filter Pruning via Geometric Median (FPGM) [51] reveals the "smaller-norm-less-important" assumption to not always be true, based on the real distribution of the neural networks. Instead of pruning away unimportant filters, it finds redundant filters by exploiting relationships among filters of the same layer. He _et al._ consider the filters close to the geometric median to be redundant because they represent common information shared by all filters in the same layer [51]. These redundant filters can be removed without significantly influencing the performance.

RED [52] uses a data-free structured compression method. It consists of three steps. First, scalar hashing is conducted on weights in each layer. Second, redundant filters are merged based on the relative similarity of the filters. Third, a novel uneven depthwise separation technique is used to prune layers. In RED++ [53], the third step is replaced with an _input-wise splitting_ technique to remove redundant operations such as multiplication and addition. The reason behind this is that mathematical operations are more of a bottleneck compared to memory allocation.

Unlike FPGM [51], which measures filter importance within layers, Correlation-based Pruning (COP) [54] compares the importance of the cross-layer filters. To determine the redundancy among filters within a layer, COP [54] first conducts a Pearson correlation test. Next, a layer-wise max-normalization is used to address the scaling effect of the correlation-based importance metric in order to rank the filters across layers. Lastly, a cost-aware regularization term is added to the global filter-importance calculation to allow users to have finer control over the budget.

Structural Redundancy Reduction (SRR) [55] exploits the structural redundancy by looking for the most redundant

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Papers & P. & Q. & D. & KD & NAS \\ \hline
[23, 34, 35] & & ✓ & & & \\ \hline
[24, 36] & & & & ✓ & & \\ \hline
[25, 37, 38, 39] & & & & & ✓ & \\ \hline
[27, 28, 29, 30, 31, 32, 33] & ✓ & & & & \\ \hline
[40, 41] & ✓ & ✓ & & & & \\ \hline
[42, 43, 44, 45] & ✓ & ✓ & ✓ & ✓ & \\ \hline
[26, 46, 47, 48, 49, 50] & ✓ & ✓ & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table} TABLE I: Categorize existing surveys into Pruning (P), Quantization (Q.), Decomposition (D.), Knowledge Distillation (KD), and Neural Architecture Search (NAS).
layer, instead of the least ranked filters among all layers. First, filters in each layer are established as a graph. The redundancy of a graph can be evaluated by its two associated properties, i.e., quotient space size and \(\ell\)-covering number. The two properties with large values indicate a complex, and hence less redundant, graph. Within the most redundant layer, a filter norm can be applied to prune the least important filters. Finally, the graph of the layer is is re-established, and the layers' redundancy is re-evaluated.

Fig. 1: Taxonomy for structured pruning.
### _Activation-Based_

Activation maps are the convolutional results of input data and filters (Eq. 1). Apart from using the weights of filters, activation maps can be used for filter pruning. Activation channel pruning is another name for filter pruning since removing the channels of activation maps is equivalent to removing the filters. In addition to the effect of the current layer, filter pruning also influences the next layer's filter through feature maps.

To evaluate filters in layer \(l\), we can exploit the information on activation maps of:

1. **the current layer** - the channel importance can be evaluated by using the reconstruction error [58], the decomposition of the activation map [59], utilization of channel-independence [61], and post-activations [62, 63];
2. **the adjacent layers** - redundant channels can be effectively identified by exploiting the dependency between the current layer and the next layer [66, 67]. In addition, activation maps of the previous layer can also be utilized to guide the pruning decision [123, 124];
3. **all layers** - the holistic effect of removing a filter can be evaluated by minimizing the construction error of the Final Response Layer [69] and considering the discriminative power of all layers [70].

#### 2.2.1 Current Layer

Channel Pruning (CP) [58] uses layer \(l\)'s (current) activation maps to guide the pruning of layer \(l\)'s filters. It models layer-wise channel pruning as an optimization problem that minimizes the reconstruction error of sparse activation maps. Solving the optimization problems involve two alternating steps. (1) To find which channels to prune, CP explicitly solves the LASSO regression by fixing the weights rather than imposing a sparsity regularization to training loss. (2) To minimize the reconstruction error of layer \(l\)'s feature map, weights are fine-tuned with the fixed pruning decision.

HRank [59] uses the average rank of the current layer's activation maps from a small set of input data as the filter importance. An important finding is that regardless of the data received, a single filter generates activation maps with the same average rank. To find the average rank, the Singular Value Decomposition (SVD) is adopted. The decomposition conducted here is to find the rank rather than to reduce computational cost. After determining the average rank, a layer-wise pruning algorithm is then proposed to retain top-\(k\) filters.

Coreset-Based Compression (CBC) [60] adopts filter pruning to pre-process filters for coreset-based compression [225]. The scoring of the filters is based on the mean of activation norms over the entire training set. A binary search is then used to find the smallest number of filters that satisfy the accuracy constraint. After pruning, three coreset-based compression techniques are discussed, including \(k\)_-Means, Structured Sparse PCA, and Activation-Weighted coresets_. By using Deep Compression [1], _Activation-Weighted coresets_ outperforms the rest.

CHannel Independence (CHIP) has been used by Sui _et al._ to evaluate channel importance [61]. Channel independence is determined by the cross-channel correlation, indicating whether a channel is linearly dependent on other channels. The greater the independence of the channel, the higher its importance. Channel importance is determined by measuring the activation maps' nuclear norm change.

Average Percentage of Zero (APoZ) [62] utilizes the current layer's **post-activation** maps, which are the activation maps after activation functions such as ReLU. The _average percentages of zeros (APoZ)_ in the post-activation maps are used to measure the importance of channels. A small _APoZ_ value means that most parts of the activation maps are being activated, so these activation maps contribute more to the final results and are more important.

DropNet [63] utilizes the post-activation maps' **average magnitude** as the metric. Under this metric, a small non-zero activation value, which is considered important by APoZ [62], is no longer important in DropNet [63]. There are two reasons for the use of this metric. First, a small **average magnitude** indicates the presence of many inactive nodes. Second, the small magnitude also means these nodes are less adaptive to learning.

#### 2.2.2 Adjacent Layer

ThiNet [66] uses layer \(l+1\)'s (the next layer) activation maps to guide the pruning of the layer \(l\) (the current layer). The main idea is to approximate layer \(l+1\)'s activation maps with **subsets** of layer \(l\)'s activation maps. Channels outside these subsets are pruned. To find these subsets, a greedy algorithm is proposed. Specifically, the algorithm greedily adds channels to an initially empty set and measures the reconstruction error. Subsets that have the least reconstruction error and meet the sparsity constraint will be selected.

Approximated Oracle Filter Pruning (AOFP) [67] uses layer \(l+1\)'s activations and targets at pruning without heuristic knowledge which is often required by _Oracle Pruning_ methods [13, 62, 95]. **Firstly**, the concept of _damage isolation_ is introduced to avoid using heuristic importance metrics. _Damage isolation_ refers to the isolation of the damage caused by pruning layer \(l\) by layer \(l+1\), rendering it invisible to \(l+2\). **Secondly**, a multi-path framework is used to benefit from parallel scoring and fine-tuning. **Thirdly**, the _binary filter search_ method is used to solve problems of the multi-path framework.

In addition to using the next layer's activation maps, runtime neural pruning (RNP) [123] and feature boosting and suppression (FBS) [124] utilize the layer \(l-1\)'s (previous layer) activation maps to guide the pruning of the current layer. Both methods use the global average pooling result of the previous layer as the filter importance. This is further discussed in Section 2.5.2, since both methods conduct dynamic inference pruning.

#### 2.2.3 All Layer

Despite the success of existing methods, proponents of Neuron Importance Score Propagation (NISP) [69] argue that most methods did not consider the reconstruction error propagation. NISP proposes to use the _Final Response Layer (FRL)_ to determine the neuron importance because reconstruction errors from all previous layers will eventually be propagated to the _FRL_. Initially, the importance score of _FRL_ can be determined by any feature ranking technique, i.e., Inf-FS [226]. The neuron importance is then propagated
backward from \(\mathit{FRL}\) to the previous layers. Lastly, neurons with low importance scores in the layer are pruned. Pruned neurons will no longer back-propagate scores to the previous layers.

Discrimination-aware Channel Pruning (DCP) [70] aims to keep discriminative channels that substantially change the final loss in their absence. However, pruning shallow layers often triggers a smaller decrease in final loss due to the long propagation path. To resolve the problem, Zhuang _et al._ introduce discrimination-aware losses to every last layer of the intermediate layers. A greedy algorithm is then used to select channels based on the discrimination-aware loss and reconstruction loss between the baseline and pruned networks.

### _Regularization_

Regularization can be used for learning structured sparse networks by adding different sparsity regularizers \(R_{s}(\cdot)\). The sparsity regularizer can be applied to **BN parameters** if the networks contain batch normalization layers. To achieve structured sparsity, BN parameters are used to indicate the pruning decision of structures such as channels or filters. **Extra parameters** working as learnable gates have been introduced to guide pruning. With these extra parameters, networks no longer require batch normalization layers. Sparsity regularizers can also be directly applied to **filters**. Group Lasso regularization is commonly used to sparsify filters in a structured manner.

The general Group Lasso is defined as the solution to the following convex optimization problem [31],

\[\min_{\beta\in\mathbb{R}^{p}}\left(\left\|\mathbf{y}-\sum_{g=1}^{G}\mathbf{X }_{\mathbf{g}}\beta_{g}\right\|_{2}^{2}+\lambda\sum_{g=1}^{G}\sqrt{n_{g}} \left\|\beta_{g}\right\|_{2}\right) \tag{4}\]

where the feature matrix is divided into \(G\) groups, forming the matrix \(\mathbf{X}_{g}\) that contains only examples of group \(g\) as well as the corresponding coefficient vector \(\beta_{g}\). \(n_{g}\) indicates the size of group \(g\), and \(\lambda\geq 0\) is a tuning parameter. In the context of filter pruning, the first term can be viewed as the reconstruction error of the feature map, and the second term can be rewritten as:

\[\lambda\left\|\mathbf{F}^{l}\right\|_{2,1}=\lambda\sum_{i=1}^{N_{l+1}}\left\| \mathbf{F}_{i}^{l}\right\|_{2} \tag{5}\]

where groups \(G\) are replaced by output channels \(N_{l+1}\), and coefficients vectors \(\beta_{g}\) are replaced by filters. In addition, differences exist among the use of \(\ell_{1}\)-norm, \(\ell_{2}\)-norm and \(\ell_{2,1}\)-norm as the penalty function (Fig. 2).

#### 2.3.1 Regularization on BN Parameters

BN layers have been widely used in many modern CNNs to improve model generalization. By normalizing each training mini-batch, the internal covariate shift is addressed. The following describes the operations of a BN layer (NS [73]),

\[\hat{\mathbf{z}}=\frac{\mathbf{z}_{\mathrm{in}}-\mu_{\mathcal{B}}}{\sqrt{ \sigma_{\mathcal{B}}^{2}+\epsilon}};\quad\mathbf{z}_{\mathrm{out}}=\gamma \hat{\mathbf{z}}+\beta \tag{6}\]

where \(N_{l+1}\)-dimension \(\mathbf{z}_{\mathrm{in}}\) and \(\mathbf{z}_{\mathrm{out}}\) are the inputs and outputs of a BN layer, \(\mu_{\mathcal{B}}\) and \(\sigma_{\mathcal{B}}\) are the mean and standard deviation over the current mini-batch \(\mathcal{B}\). \(\epsilon\) is a small number to prevent zero-division. \(\gamma\in\mathbb{R}^{N_{l+1}}\) and \(\beta\in\mathbb{R}^{N_{l+1}}\) are the learnable parameters, indicating scale and shift, respectively. BN parameters are used as gates for filter pruning because the number of learnable parameters is equal to the number of feature maps and filters.

Network Slimming (NS) [73] directly uses the scaling parameter \(\gamma\) in BN to control output channels. The channel-level sparsity-induced \(\ell_{1}\) regularization is then introduced to jointly train the weights together with the \(\gamma\). After training, the corresponding channels that have close-to-zero \(\gamma\) are pruned. To optimize the non-smooth \(\ell_{1}\) penalty term, a subgradient descent [228] is used.

Similar to the situation in NS [73], Gated Batch Normalization (GBN) [74] uses \(\gamma\) as the channel-wise gate and \(\ell_{1}\)-norm of \(\gamma\) as the regularization term. A _Tick-Tock_ pruning framework is proposed to boost accuracy by iterative pruning. The _Tick_ phase trains the network with little data, and only the gates and final linear layer are allowed to be updated during training for one epoch. Meanwhile, channel importance is computed by the first-order Taylor Expansion for a global filter ranking. The _Tock_ phase then fine-tunes the sparse network with sparsity constraints.

Polarization Regularization (PR) [75] provides a variant \(\ell_{1}\)-based regularizer to polarize the scaling factors \(\gamma\). It contends that in the majority of the sparsity regularization methods, such as NS [73], a naive \(\ell_{1}\) regularizer converges all scaling factors to zero indiscriminately. A more reasonable approach is to push the scaling factors of unimportant neurons to zero and those of important neurons to a large value. To achieve the polarization effect, another penalty term is added to the naive \(\ell_{1}\) term, separating \(\gamma\) from their mean as far as possible. Similar to NS [73], subgradients [228] are used on non-differentiable points to solve the non-smooth regularizer.

Rethinking Smaller-Norm-Less-Informative (RSNLI) [76] is developed from the basis of previous methods tending to suffer from the Model Reparameterization problem and the Transform Invariance problem. As there are doubts whether smaller-norm parameters are less informative; Ye _et al._ propose to train with ISTA [229] to enforce sparsity on \(\gamma\). The channel whose \(\gamma\) is equal to zero is pruned. Then the \(\gamma-W\) rescaling trick is used on \(\gamma\) and weights to quickly start the sparsification process.

Operation-aware Soft Channel Pruning (SCP) [77] considers both BN and ReLU operations. In contrast to NS [73] making decision merely on the channel scaling \(\gamma\), Kang _et al._

Fig. 2: The left image shows the \(\ell_{1}\) penalty used in Lasso. The middle image shows the \(\ell_{2,1}\) penalty function used in Group Lasso, and the right image shows the \(\ell_{2}\) penalty function used in Ridge. \(\beta_{11}\) and \(\beta_{12}\) are grouped together for Group Lasso. (The image is taken from [227].)
also consider shifting parameters \(\beta\). Specifically, channels with large negative \(\beta\) and large \(\gamma\) are considered unimportant since these channels will become zero after ReLU. To account for BN's large negative mean values, the cumulative distribution function (CDF) of a Gaussian distribution parameterized by \(\beta\) and \(\gamma\) is used as the indicator function. To optimize \(\beta\) and \(\gamma\), a sparsity-loss-inducing large CDF value is designed to encourage the network to be more sparse.

EagleEye [78] proposes a three-stage pipeline. **First**, pruning strategies (i.e., layer-wise pruning ratio) are generated by simple random sampling. **Second**, sub-networks are generated according to the pruning strategies and \(\ell_{1}\)-norm of filters. **Third**, an _adaptive-BN-based candidate evaluation module_ is used to evaluate the performance of the sub-networks. Li _et al._ contend that outdated BN statistics are unfair to sub-network evaluations, and that BN statistics for every candidate should be re-calculated on a small part of the training dataset [78]. After sub-networks are evaluated with the _adaptive BN statistics_, the best-performing one is selected as the final pruned model.

#### 2.3.2 Regularization on Extra Parameter

Although some studies [74, 76] make special adjustments for networks without BN layers, introducing extra parameters is a more general solution. The extra parameters \(\theta\) are trainable and parameterize the gates \(g(\theta)\) in determining the pruning results.

To find a sparse structure, Sparse Structure Selection (SSS) [79] attempts to force the output of structures to zero. A scaling factor \(\theta\) is introduced after each structure, i.e., neuron, group or residual block. When \(\theta\) is lower than a threshold, the corresponding structure is removed. The gate \(g(\theta)\) function is:

\[g(\theta)=\begin{cases}0,&\theta<\text{Threshold}\\ 1,&\text{otherwise}\end{cases}. \tag{7}\]

It adopts a convex relaxation \(\ell_{1}\)-norm as the sparsity regularization on the extra parameter \(\theta\). To update \(\theta\), a modified Accelerated Proximal Gradient [230] is used.

Generative Adversarial Learning (GAL) [80] jointly prunes structures and adopts a Generative Adversarial Network (GAN) to achieve label-free learning. Extra scaling factors are introduced after each structure in the generator, forming soft masks. During training, a special regularization term that contains three regularizers is proposed: 1) a \(\ell_{2}\) weight decay regularizer on generators, 2) a \(\ell_{1}\) sparsity regularizer on the mask, and 3) an _adversarial regularization_ on discriminator. Furthermore, FISTA [229] is used to iteratively update the generator and discriminator, and the mask is updated together with the generator.

Discrete Model Compression (DMC) [81] explicitly introduces discrete (binary) gates after the feature map to precisely reflect the pruned channels' impact on the loss function. First, it samples subnetworks with stochastic discrete gates:

\[g(\theta)=\begin{cases}1,&\text{w.p. }\theta\\ 0,&\text{w.p. }1-\theta\end{cases} \tag{8}\]

where w.p. stands for "with probability". The stochastic nature of the gate ensures that every channel has the chance to be sampled if \(\theta\neq 0\), so different subnetworks can be produced. To update the non-differentiable binary gates, a Straight-Through Estimator [231] is adopted.

Similar to PR [75], Gates with Differentiable Polarization (GDP-Guo) [82] aims to polarize gates. Designing a gate with a polarization effect yields the property of smoothed \(\ell_{0}\) formulation [232]:

\[g(\theta)=\frac{\theta^{2}}{\theta^{2}+\epsilon}\xrightarrow{\text{has property}}\quad g(\theta)=\begin{cases}\approx 1,&\theta\neq 0\\ =0,&\theta=0\end{cases} \tag{9}\]

where \(\epsilon\) is a small positive value to prevent zero division. This property polarizes \(\theta\) to either exactly \(0\) or values close to \(1\). The gate itself is differentiable. However, the sparsity regularization on \(\theta\) involves \(\ell_{0}\)-norm and renders the objective function non-differentiable. Thus, proximal-SGD [233] is used to update \(\theta\).

Convolutional **Re-**parameterization and Gradient **Res**etting (ResRep) [83] re-parameterizes a CNN into two parts. The first "remembering part" learns to maintain the model's performance and will not be pruned. The second "forgetting part" inserts \(1\times 1\) CONV layers, or the _compactors_, after the BN layers. During training, a modified SGD update rule is applied to update the _compactors_ only. Thus, only _compactors_ are allowed to forget (forgetting) while other CONV layers are kept untouched (remembering).

Scientific Control Pruning (SCOP) [84] believes that the importance of a filter may be disturbed by _potential factors_ such as input data. For example, the filter importance ranking may vary if input data is slightly changed for data-dependent methods. To minimize the effect of _potential factors_, it prunes under scientific control by creating knockoff counterparts [234]. Knockoff features are identical to the real features except for not knowing the true label. Two introduced scaling factors \(\theta\) and \(\tilde{\theta}\) are then introduced to control the participation of real and knockoff features, respectively. The two parameters are complementary that \(\theta+\tilde{\theta}=\mathbf{1}\). If \(\theta\) cannot suppress \(\tilde{\theta}\), the real features are deemed to have little or no association with the true output. Thus, the filter importance score is defined as \(\mathcal{I}=\theta-\tilde{\theta}\), and filters with small importance scores are deemed redundant.

To directly control the model budget, Budget-Aware Regularization (BAR) [85] uses a prior loss and introduces a learnable dropout parameter \(\theta\)[235]. The prior loss is the product of two functions. The first function is an approximation of budgets, that is differentiable w.r.t. \(\theta\). The second function is a variant log-barrier function [236] that employs a sigmoidal schedule. The novel objective function consists of the prior loss, enabling simultaneous training and pruning according to the budget. Knowledge distillation is then used to improve accuracy.

#### 2.3.3 Regularization on Filters

Structured Sparsity Learning (SSL) [91] uses Group Lasso to prune channels. Removing layer \(l^{\prime}\)s channel will cause the removal of layer \(l^{\prime}\)s filters and layer \(l+1\)'s input channels. Hence, it adds two separate regularization terms for filter-wise and channel-wise pruning:

\[\sum_{n_{l}=1}^{N_{l+1}}\left\|\mathbf{W}_{n_{l},...,..}^{(l)}\right\|_{2}, \qquad\sum_{c_{l}=1}^{N_{l}}\left\|\mathbf{W}_{:c_{l},..,..}^{(l)}\right\|_{2}, \tag{10}\]
where \(\mathbf{W}^{(l)}\in\mathbb{R}^{N_{l+1}\times N_{l}\times K_{l}\times K_{l}}\).

Out-In-Channel Criterion Regularization (OICSR) [92] uses Group Lasso to **jointly** regularize filters that work cooperatively. The regularization term is:

\[\sum_{i=1}^{N}\left\|\mathbf{W}^{(l)}_{i,:}\oplus\mathbf{W}^{(l+1)}_{,:}\right\| _{2} \tag{11}\]

where \(\oplus\) denotes concatenation of the out-channel filters \(\mathbf{W}^{(l)}_{i,:}\in\mathbb{R}^{N_{l+1}\times(N_{l}K_{l})}_{i,:}\) and the in-channel filters \(\mathbf{W}^{(l+1)}_{:,i}\in\mathbb{R}^{(N_{l+2}K_{l+1}K_{l+1})\times N_{l+1}}\). It uses the premise that out-channel filters of layer \(l\) are interdependent of in-channel filters of layer \(l+1\), so these filters should be regularized together.

Only Train Once (OTO) [93] contends that even if all filters weights are zeros, the activations will be non-zero because of three parameters: 1) convolution bias, 2) BN mean, and 3) BN variance. Instead of grouping only filters, this method groups all parameters that cause a non-zero activation into a group named the zero-invariant group. Structured sparsity is then introduced to this group by applying the mixed \(\ell_{1}/\ell_{2}\)-norm. To solve the non-smooth mixed-norm regularization, a stochastic optimization algorithm named _Half-Space Stochastic Projected Gradient_ is used.

Growing Regularization (GREG) [94] exploits regularization under a growing penalty and uses two algorithms. The **first algorithm** focuses on the pruning schedule and adopts \(\ell_{1}\)-norm [13] to obtain a mask for pruning. Instead of immediately removing unimportant filters, a growing \(\ell_{2}\) penalty is used to gradually drive them to zero. The **second algorithm** uses a growing regularization to exploit the underlying Hessian information. The authors observe weight discrepancy increases as the regularization parameter increases, then weights will naturally separate. If the discrepancy is large enough, even a simple \(\ell_{1}\)-norm can be an accurate criterion.

### _Optimization Tools_

#### 2.4.1 Taylor Expansion

Taylor Expansion [237] expands a function into the Taylor Series which is an infinite sum of terms. The Taylor Expansion of a function \(f(x)\) expands at some point \(a\):

\[f(x)=\overbrace{f(a)+\frac{f^{\prime}(a)}{1!}(x-a)}^{\text{second-order approximation}}+\frac{f^{\prime\prime}(a)}{2!}(x-a)^{2}+\underbrace{R_{2}(x)}_{\text{reminders}}, \tag{12}\]

where \(f^{\prime}(\cdot)\) and \(f^{\prime\prime}(\cdot)\) is the first-order derivative and second-order derivative with respect to \(x\), respectively.

In structured pruning, Taylor Expansion is used to approximate the change in the loss \(\Delta\mathcal{L}\) of pruning structures such as filters or channels. Since pruned weights are set to 0, the loss function \(\mathcal{L}(w)\) of weights \(w\) can be evaluated at \(a=0\) using Taylor Expansion. By manipulating Eq. 12, we can get

\[\mathcal{L}(w)-\mathcal{L}(0)=\frac{\mathcal{L}^{\prime}(0)}{1!}(w-0)+\frac{ \mathcal{L}^{\prime\prime}(0)}{2!}(w-0)^{2}+R_{2}(w). \tag{13}\]

Let \(\mathcal{L}(w)-\mathcal{L}(0)=\Delta\mathcal{L}\) be the change in the loss of removing some weights, and \(w-0=\Delta w\) be the change in the weights. We can get the following equation based on [98],

\[\Delta\mathcal{L}=\frac{\partial\mathcal{L}^{\top}}{\partial w}\Delta w+\frac{ 1}{2}\Delta w^{\top}\mathbf{H}\Delta w+R_{2}(w) \tag{14}\]

where \(\frac{\partial\mathcal{L}}{\partial w}\) is the first-order gradient of loss function w.r.t. weights, and \(\mathbf{H}\) is the Hessian matrix containing second-order derivatives. Compared to regularization-based methods, pruning with Taylor Expansion does not need to wait until the activations are trained sufficiently small [96].

First-order and second-order Taylor expansions have their own characteristics. The second-order expansion contains more information, but it requires calculating the second-degree derivatives that are computationally prohibitive. On the contrary, the first-order expansion can be obtained from backpropagation without requiring additional memory, but this provides less information.

**First-order-Taylor:** Mol-16 [95] uses the first-order information to estimate the change in the loss of pruning activation maps. The higher-order remainders, including the second-order term, are discarded since they are computationally intractable and are encouraged to be small by the widely-used ReLU activation function. Thus, the absolute change in loss approximated by the first-order term is used as the metric for feature map importance:

\[\left|\frac{1}{M}\sum_{m}\frac{\partial\mathcal{L}}{\partial z_{m}}z_{m}\right|, \tag{15}\]

where \(M\) is the length of a flattened feature map and \(z\) is an activation in a feature map. After determining the feature map importance, the lowest ranked maps will be pruned.

Compared to Mol-16 [95] which fails with skip connections, Mol-19 [96] proposes a more general method that uses Taylor expansion to approximate the squared change in the final loss. Unlike Mol-16's [95] use of activations that increase memory consumption, Mol-19 [96] computes the importance \(\mathcal{I}\) based on weights:

\[\mathcal{I}_{\mathcal{S}}(\mathbf{W})\triangleq\sum_{s\in\mathcal{S}}\left( \frac{\partial\mathcal{L}}{\partial w_{s}}w_{s}\right)^{2}, \tag{16}\]

where \(\mathcal{S}\) is a structural set of parameters such as a convolutional filter, \(w_{s}\) is the individual weight of the filter, and \(\frac{\partial\mathcal{L}}{\partial w_{s}}\) represents the gradient. The first-order expansion performs significantly faster than the second-order expansion with a slightly higher accuracy drop.

Due to the simplicity and efficiency of computing, the first-order expansion is widely adopted by many methods such as GBN [74] and GDP-Lin [116] which are discussed in other sections.

**Second-order-Taylor:** In this sub-section, the Hessian matrix \(\mathbf{H}\) that contains second-order information from Eq. 14 is exploited. First, pioneering unstructured studies are briefly introduced. Second, the structured pruning methods are addressed.

**Pioneering unstructured studies:** Most current structured pruning methods that use second-order Taylor expansion are based on two pioneering studies on unstructured pruning: Optimal Brain Damage (OBD) [238] and Optimal Brain Surgeon (OBS) [239]. OBD [238] assumes that \(\mathbf{H}\) is
diagonal to ease computation. The diagonal \(\mathbf{H}\) is then used to compute the parameter importance. However, OBS [239] finds that most of the Hessian matrices \(\mathbf{H}\) are strongly non-diagonal. Thus, the full \(\mathbf{H}\) is used and the parameter importance is calculated by \(\mathbf{H}^{-1}\).

**Structured pruning methods**: With the success of OBD [238] and OBS [239] in unstructured pruning, the second-order expansion is applied to structured pruning. As deep CNNs have millions of parameters, computing and storing the Hessian \(\mathbf{H}\) become challenging [97]. Recent methods aim to approximate the Hessian matrix for structured pruning.

Collaborative Channel Pruning (CCP) [97] approximates the Hessian matrix by only using the first-order derivative of a pre-trained model. The first-order information can be retrieved from backpropagation, and no additional storage is needed. In addition, Peng _et al._ exploit the effect of removing multiple channels instead of a single channel. The non-diagonal element in \(\mathbf{H}\) reflects the interaction between two channels and hence exploits the inter-channel dependency. CCP models the channel selection problem as a constrained 0-1 quadratic optimization problem to evaluate the joint impact of pruned and unpruned channels.

Eigen Damage (ED) [98] introduces a baseline method that is the naive structured extension of OBD [238] and OBS [239]. Two algorithms are then applied to improve the baseline method. The **baseline method** sums up the individual change in parameters over a filter, raising the granularity to the filter level. However, computing and storing the Hessian \(\mathbf{H}\) is intractable. Wang _et al._ propose the use of the **first algorithm**[98] that applies K-FAC [240] approximation to decompose filters. As the naive extensions and the first algorithm both fail at capturing the correlation between filters, a **second algorithm** which decorrelates the weights before pruning is applied. This second algorithm adopts K-FAC [240] and projects weight space to a Kronecker-Factored eigenspace (KFE) [241] where there is little correlation.

Group Fisher Pruning (GFP) [99] addresses the difficulties other pruning methods face when channels from multiple layers are coupled and need to be pruned simultaneously. First, a layer grouping algorithm is used to automatically identify coupled channels. Second, the Hessian information is used as a unified importance criterion of a single channel and coupled channels. With the help from the Fisher information, the Hessian matrix is transformed into the square of first-order derivatives.

Hessian-Aware Pruning (HAP) [100] uses the Hessian information to measure the sensitivity of channels, and the average Hessian trace to approximate the Hessian. This approximation costs the same as backpropagating gradients [242]. The moderately sensitive channels are replaced with efficient low-rank decomposition, rather than removed. The premise behind this is when insensitive channels are pruned, moderately sensitive channels will be critical to the final accuracy.

Second-Order Structured Pruning (SOSP) [101] proposes two methods, SOSP-I and SOSP-H, to approximate the \(\mathbf{H}\). SOSP-I is based on the Gauss-Newton approximation [243] which uses first-order information. SOSP-H uses a fast Hessian-vector product approximation to enable saliency evaluation. SOSP-H considers the full Hessian matrix and has the complexity of first-order methods. Both SOSP-I and SOSP-H perform similarly while the latter scales better on large-scale vision tasks.

#### 2.4.2 Variational Bayesian

Bayesian inference [244] is a method to infer the posterior probability distribution \(p(\theta|x)\) with the known prior probability distribution \(p(\theta)\) of parameters \(\theta\) and the observed data \(x\). The formula to compute the posterior distribution over \(\theta\) is given:

\[\widehat{p(\theta|x)}=\underbrace{\overbrace{p(x|\theta)}^{\text{Likelihood}} \cdot\overbrace{p(\theta)}^{\text{Prior}}}_{\text{Evidence}},\text{where }p(x)=\int p(x|\theta)p(\theta)d\theta. \tag{17}\]

However, computing the evidence often requires computationally intractable integrals when large amounts of data are involved. Variational Bayesian (VB) methods [245] are used to approximate the posterior distribution \(p(\theta|x)\) by a variational distribution \(q(\theta)\). Specifically, \(q(\theta)\) is optimized by minimizing the Kullback-Leibler (KL) divergence which measures the "similarity" between \(q(\theta)\) and \(p(\theta|x)\). Since the computation of KL-divergence involves the intractable posterior distributed \(p(\theta|x)\), the optimization problem is solved by equivalent conversion to maximize the Evidence Lower BOund (ELBO).

Variational Pruning (VP) [102] is based on channel importance being indicated by random variables, as pruning by deterministic channel importance is inherently improper and unstable. Thus, BN's parameter \(\gamma\) is used to indicate the channel saliency and model \(\gamma\) by Gaussian distribution \(\mathcal{N}\left(\mu,\sigma\right)\). To introduce sparsity, VP utilizes the centrality property of the Gaussian distribution \(\mathcal{N}\) and samples \(\gamma\) from \(\mathcal{N}\left(\mu=0,\sigma\right)\) as the sparse prior distribution. After optimizing ELBO, distributions of \(\gamma\) with close-to-zero mean and small variance are considered safe for pruning, since such distributions are less likely to have salient parameters.

To find redundant channels, Recursive Bayesian Pruning (RBP) [103] targets the posterior of redundancy which assumes an inter-layer dependency among channels. First, each input channel is scaled by a dropout noise \(\theta\)[235, 246] with a dropout rate \(r\). Second, the dropout noises are then modeled across layers as a Markov chain to exploit the inter-layer dependency among channels. To get the posterior of redundancy, a sparsity-inducing Dirac-like prior is chosen. In addition, RBP [103] adopts the reparameterization trick [235] to scale the noise \(\theta\) on the corresponding channels to consider data fitness. As a result, the dropout rate \(r\) can be updated along with weights in a gradient-based manner. Pruning is conducted by setting the corresponding channels to zero when \(r\) is greater than a threshold.

A few other studies also use the Bayesian point of view. VIBNet [104] uses a _variational information bottleneck_, which is the informational theoretical measure of redundancy between adjacent layers. Louizos _et al._[105] utilize the horseshoe prior to efficiently approximate channel redundancy. Neklyudov _et al._[106] use a log-normal prior, resulting in a tractable and interpretable log-normal posterior.
#### 2.4.3 Others

**SGD-based:** Instead of zeroing out filters, Centripetal SGD (C-SGD) [107] makes redundant filters identical and merges identical filters into one filter. Regularized Modernized Dual Averaging (RMDA) [108] adds momentum to the RDA algorithm [247] and ensures that the trained model has the same structure as the original model. To prune the model, RMDA adopts Group Lasso to promote structured sparsity.

**ADMM-based:** Alternating Direction Method of Multipliers (ADMM) [248] is an optimization algorithm used to decompose initial problem into two smaller, more tractable subproblems. StructADMM [111] studies the solution of different types of structured sparsity such as filter-wise and shape-wise. Zhang _et al._ use a progressive and multi-step ADMM framework: at each step, it uses ADMM to prune and masks out zero weights, leaving the remaining weights as the optimization space for the next step.

**Bayesian Optimization:** Bayesian optimization (BO) [249] is a sequential design strategy for global optimization of black-box functions that does not assume any functional forms. When pruning a model which involves an extremely large design space, the curse of dimensionality problem occurs [250]. Rollback [113] adopts the RL-style automatic channel pruning [130] and uses BO to determine the optimal pruning policy.

**Soft Threshold:** Soft threshold [251] is a de-noising operator in the area of digital signal processing. Soft Threshold Reparameterization (STR) [114] adopts this operator to learn non-uniform sparsity budget across layers that is optimized per-layer.

### _Dynamic Pruning_

Dynamic pruning can be conducted during training and inference. **Dynamic during training** aims to preserve the models' representative capacity by maintaining a dynamic pruning mask during training. It is also called soft pruning to ensure that improper pruning decisions can be recovered later. On the other hand, hard pruning permanently removes weights with a fixed mask. **Dynamic during inference** indicates networks are pruned dynamically according to different input samples. For instance, a simple image that contains clear targets requires less model capacity compared to a complex image [108]. Hence, dynamic inference provides better resource-accuracy trade-offs.

#### 2.5.1 Dynamic during Training

**Weight-level dynamic:** The concept of training-time dynamic pruning was first introduced in Dynamic Network Surgery (DNS) [252], an unstructured pruning approach. To clarify the difference between other approaches, its formulation is:

\[\mathbf{W}_{i,j}^{l}\leftarrow\mathbf{W}_{i,j}^{l}-\eta\frac{\partial\mathcal{ L}(\mathbf{W}^{l}\odot\mathbf{T}^{l})}{\partial(\mathbf{W}_{i,j}^{l}\mathbf{T}_{ i,j}^{l})}, \tag{18}\]

where \(\eta\) is the learning rate and \(\odot\) denotes the Hadamard Product operator. \(\mathbf{W}^{l}\) is the weight matrix in the \(l\)-th layer with all weights of kernels unfolded and concatenated together. \(\mathbf{T}^{l}\) is the binary mask that has the same shape as the weight matrix, indicating the importance of the weights. \(\mathcal{L}(\cdot)\) is the loss function. \(\mathbf{W}_{i,j}^{l}\) and \(\mathbf{T}_{i,j}^{l}\) indicate a single element in \(\mathbf{W}^{l}\) and \(\mathbf{T}^{l}\). \(\mathbf{W}\) and \(\mathbf{T}\) are updated alternatively. All the weights are updated, so wrongly pruned parameters have a chance to regrow.

**Filter-level dynamic:** Soft filter Pruning (SFP) [115] adopts the idea of dynamic pruning in a structured way. Its use is based on the premise that hard pruning that uses a fixed mask throughout the training will reduce the optimization space. Therefore, it dynamically generates the masks based on the \(\ell_{2}\)-norm of filters at every epoch. Soft pruning means setting the values of filters to zero instead of removing filters. Previously soft-pruned filters are allowed to be updated at the next epoch, during which masks will be reformed based on new weights. The update rule is:

\[\mathbf{W}^{l}\leftarrow\mathbf{W}^{l}-\eta\frac{\partial\mathcal{L}\left( \mathbf{W}^{l}\odot\mathbf{m}^{l}\right)}{\partial\left(\mathbf{W}^{l}\odot \mathbf{m}^{l}\right)}, \tag{19}\]

where \(\mathbf{W}^{l}\in\mathbb{R}^{N_{l+1}\times N_{l}KK}\) and \(\mathbf{m}^{l}\in\mathbb{R}^{N_{l+1}}\).

Globally Dynamic Pruning (GDP-Lin) [116] also maintains a binary dynamic mask during training based on the filter importance. GDP-Lin adopts the first-order Taylor expansion to approximate the global discriminative power of each filter. In addition, the authors argue that frequently changing masks cannot effectively guide the pruning. Hence, masks are updated every \(e\) iteration, where \(e\) is set as a decreasing value to accelerate the convergence.

In addition to maintaining a sparse dynamic mask, Dynamic Pruning with Feedback (DPF) [117] simultaneously maintains a dense model. The premise behind its use is that a sparse model can be considered a dense model with compression error. The error can be used as feedback to ensure correct direction in the gradient updates. For this goal, gradients computed from the sparse model are used to update the dense model. The advantage of applying gradients on a dense model is that it helps weights to recover from errors.

CHannel EXploration (CHEX) [118] uses two processes to dynamically adjust filter importance. The first process is the channel pruning process with _Column Subset Selection criterion_[253]. The second process is the regrowing process, which is based on _orthogonal projection_[254] to avoid regrowing redundant channels and to explore channel diversity. Regrowing channels are restored to the _most recently used (MRIU)_ parameters rather than zeros. To better preserve model accuracy, the remaining channels are re-distributed dynamically among all the layers.

Dynamic Sparse Graph (DSG) [119] activates a small number of critical neurons with the constructed sparse graph at every iteration dynamically. DSG is developed from the argument that direct computation according to output activations is very costly for finding critical neurons. The _dimension reduction search_ is formulated to forecast the activation output by computing input and filters in a lower dimension. Furthermore, to prevent BN layers from damaging sparsity, Liu _et al._ introduce the _double-mask selection_ that uses the same selection mask before and after BN layers [119].

Other methods such as SCP [77] and DMC [81] also maintain the mask dynamically. These are discussed in Section 2.3.
#### 2.5.2 Dynamic during Inference

Runtime Neural Pruning (RNP) [123] is based on the premise that the static model fails to exploit the different properties of input images by using the same weights for both easier-recognized and complex pattern images. It uses a framework with CNN as the backbone and RNN as a decision network. The network pruning is modeled as a Markov decision process, and models are trained by reinforcement learning (RL). The RL agent evaluates filter importance and performs channel-wise pruning according to the relative difficulty of samples for a task. Thus, when the image is easier for the task, a sparser network is generated.

Feature Boosting and Suppression (FBS) [124] predicts output channel importance based on inputs by introducing a tiny and differentiable auxiliary network. In the auxiliary network, a 2D global average pooling is used to subsample the activation channels in the previous layer to scalars. The advantage of this is that it to reduce the computational overhead in the auxiliary network. A salient predictor then uses the subsampled scalars to generate the predicted saliency scores through a fully connected layer followed by an activation function. The top-\(k\) salient channels are finally involved in the inference-time calculation.

Deep Reinforcement Learning pruning (DRLP) [126] learns both the runtime (dynamic) and the static importance of channels. The runtime importance measures the importance of channels specific to an input. In contrast, the static importance measures the channel importance for the whole dataset. Like RNP [123], DRLP uses reinforcement learning (RL). The RL framework contains two parts: one static and one runtime. Each part contains an importance predictor and an agent, providing static/runtime importance and layer-wise pruning ratio respectively. In addition, a _trade-off pruner_ generates a unified pruning decision based on the outputs from the two parts.

Dynamic Dual Gating (DDG) [127] uses two separate gating modules (spatial and channel gating) to determine inference-time importance according to inputs. The spatial module consists of an adaptive average pooling followed by a 3x3 CONV layer to extract informative spatial features. By having a spatial mask, only informative features are allowed to pass to the next layer. The channel gating module works in a manner similar to existing methods such as FBS [124] by using global average pooling followed by the application of FC layers. To enable gradient flow, the Gumbel-Softmax reparameterization [255] is used for both spatial and channel gating modules.

Fire Together Wire Together (FTWT) [128] models the dynamic pruning task as a self-supervised binary classification problem. The proposed framework use 1) a prediction head to generate learnable binary masks and 2) a ground truth mask to guide the learning after each convolutional layer. The prediction head consists of a global max pooling layer, a 1x1 CONV layer, and a softmax layer. Then the outputs from the prediction head are rounded to generate a binary mask. To achieve self-supervised learning, ground truth binary masks are needed; by ranking the norm of input activations, ground truth pruning decisions are made to guide the learning.

At the same time, Contrastive Dual Gating (CDG) [129] implements another self-supervised dynamic pruning framework by using contrastive learning [256]. Contrastive learning uses transformed images such as flipped images as inputs and uses two contrastive branches to exploit the high-level features. Then a similarity-based contrastive loss [257] is used for gradient-based learning without the labeled data. Empirical studies show that pruning decisions cannot be transferred between contrastive branches. Thus, dual gating is introduced, and the two branches have distinct pruning masks. CDG [129] uses a framework similar to that of CGNet [258] for each contrastive branch. Like DDG [127], a 3x3 CONV layer is used to extract spatial features for unpruned features.

### _Neural Architecture Search_

As it is cumbersome to manually determine pruning-related hyperparameters such as the layer-wise pruning ratio, the Neural Architecture Search (NAS) [25] has been proposed to automatically find pruned structures.

Based on the survey of NAS [25], we categorize NAS for pruning into three methods. NAS can be modeled as:

1. [leftmargin=*]
2. **reinforcement learning (RL) problems**, in which RL agents find sparse subnetworks by searching over the action space such as pruning ratios.
3. **gradient-based methods** that modify the gradient update rule to make optimization problems with sparsity constraints differentiable to weights.
4. **evolutionary methods** that adopt evolutionary algorithms to explore and search the sparse subnetworks.

#### 2.6.1 Reinforcement Learning-Based

AutoML for Model Compression (AMC) [130] uses RL to automatically select the appropriate layer-wise pruning ratio without the manual sensitivity analysis. To search over a continuous action space, the deep deterministic policy gradient (DDPG) algorithm [259] is adopted. The actor receives 11 layer-dependent states such as FLOPS and outputs a continuous action about the pruning ratio. For accuracy-guaranteed pruning, the reward for the DDPG agent is modified from Eq. 20 to Eq. 21:

\[R_{\mathrm{err}}=-Error \tag{20}\] \[R_{\mathrm{FLOPs}}=-Error\cdot\log(\mathrm{FLOPs})\] (21) \[R_{\mathrm{Param}}=-Error\cdot\log(\#\mathrm{Param})\]

Automatic Graph encoder-decoder Model Compression (AGMC) [131] adopts the premise that AMC [130] still requires manual selection of the fixed 11 states fed into the RL agent. In addition, the fixed environment states ignore the rich information within the computational graphs. Hence, the DNNs are modeled as computational graphs and fed into a GCN-based graph encoder-decoder [260] to automatically learn the RL agent's input states. In contrast to AMC [130], AGMC rescales the pruning ratios to compensate for any unmet sparsity constraint rather than applying the FLOPs into the reward function (Eq. 21).

DECORE [132] adopts multi-agent learning. The presence of multiple agents comes from assigning dedicated agents to all channels in a layer, with each agent only learning one parameter representing a binary decision. In contrast to AMC [130] and AGMC [131], DECORE models
the state of the channel itself. To learn the action parameters, a higher reward is given when the accuracy retention task and compression task are both well-completed as shown in Eq. 22.

\[R_{\mathrm{comp.}}^{l}=\sum_{j=1}^{N_{l+1}}1-a_{j}^{l},\quad R_{ \mathrm{acc}}=\left\{\begin{array}{cc}1&\text{if }\hat{y}=y\\ -\lambda&\text{else}\end{array}\right. \tag{22}\] \[R^{l}=R_{\mathrm{comp.}}^{l}*R_{\mathrm{acc}}\]

where \(N_{l+1}\) represents the output channel size of layer \(l\), \(a^{l}\) is the action vector of layer \(l\), \(y\) and \(\hat{y}\) are the true label and predicted label, and \(-\lambda\) is a large penalty for a wrong prediction. The reward function does not consider the FLOPs; instead, the reward for the compression task simply takes into account the number of channels remaining.

GNN-RL [133] first models the DNN into a multi-stage graph neural network (GNN) to learn the global topology. The the generated hierarchical computational graph is then used as the environmental state of the agent. In contrast to AMC [130] and AGMC [131] that both use DDPG, this method uses the Proximal Policy Optimization (PPO) algorithm [261] as the policy since PPO gives a much better experiment result. In addition, due to the specially designed graph environment, the reward for the RL system does not need to contain sparsity constraints. Accuracy is the only metric for the reward described by Eq. 20.

#### 2.6.2 Gradient-Based

To search for the layer-wise sparsity, Differentiable Markov Channel Pruning (DMCP) [136] models channel pruning as a Markov process. Here, the state means retaining the channel during pruning, and transitions between states represent the pruning process. DMCP parameterizes both the transition and budget loss by learnable architecture parameters. The final loss is thus differentiable w.r.t. architecture parameters. To stages are used to finish pruning in a gradient-based manner. The first stage updates weights of unpruned networks with a variant sandwich rule [262]. The second stage updates the architecture parameters.

Differentiable Sparsity Allocation (DSA) [137] optimizes the sparsity allocation in continuous space using a gradient-based method that is more effective than a discrete search. The validation loss is adopted as a differentiable surrogate to render the evaluation of validation accuracy differentiable. Next, a probabilistic differentiable pruning process is used as a replacement for the non-differentiable hard pruning process. Finally, sparsity allocation is obtained under a budget constraint. An ADMM-inspired optimization method [248] is used to solve the constrained non-convex optimization problem.

Differentiable Hyper Pruning (DHP) [138] uses hypernetworks that generate weights for backbone networks. The designed hypernetwork consists of three layers. The **1) the latent layer** first takes two learnable latent vectors \(\{\mathbf{z}^{l}\text{ and }\mathbf{z}^{l-1},\mathbf{z}^{l}\in\mathbb{R}^{N_{l+1}}\}\) and forms the latent matrix. Then **2) the embedding layer** projects the latent matrix to embedding space, forming embedding vectors. Finally, **3) the explicit layer** takes embedding vectors and outputs weights that can be explicitly used as weights for CONV layers. Pruning is done by regularizing the latent vectors with \(\ell_{1}\) sparsity regularization, and proximal gradient algorithms [236] are used to update the latent vectors. This method is approximately differentiable since the proximal operation step has a closed-form solution.

To search for compact subnetworks, Pruning as Search (PaS) [139] adds a depth-wise binary convolution (DBC) layer after each convolutional layer. The DBC works as a pruning indicator, and Straight-Through Estimator [231] is used to allow gradient flow. Furthermore, a zero pruning indicator will zero out the gradient of weights, preserving the weights of pruned layers for ease of re-evaluation. Lastly, a structural reparameterization allows any width for the network. As an example, weights in the identity path are merged into CONV weights in a mathematically equivalent way to deal with residual connections.

Learning Filter Pruning Criteria (LFPC) [140] searches for layer-wise pruning criteria instead of pruning ratios. The basis for its use is that filters have differeing distributions for extracting coarse and fine-level features. This makes using the same pruning criterion for all layers in-appropriate. To explore the criteria space, criteria parameters are introduced to guide the choice of criteria. To enable gradient flow, this method uses the Gumbel-Softmax reparameterization [255] to make the loss differentiable to criteria parameters.

Transformable Architecture Search (TAS) [141] searches the width and depth of a network. Two learnable parameters \(\alpha\) and \(\beta\) are introduced to indicate the distribution of the possible number of channels and layer, respectively. Subnetworks can be sampled based on \(\alpha\) and \(\beta\). Applying the Gumbel-Softmax reparameterization [255] makes the sampling process differentiable. The search objectives are minimizing the validation loss while encouraging smaller computational costs and penalizing the unmet resources budget. Knowledge Distillation is further used to boost the performance of the pruned networks.

Exploration and Estimation (EE) [142] achieves channel pruning in a two-step gradient-based manner. The first step is **exploration** of subnetworks to allow larger search space with a fast sampling technique (_Stochastic Gradient Hamiltonian Monte Carlo_[263]). To bring sparsity, this method applies a FLOPs-aware prior distribution to the exploration process. The second step, **estimation**, is used to guide the generation of high-quality subnetworks.

#### 2.6.3 Evolutionary-Based

MetaPruning [148] aims to find the optimal layer-wise channel number with a two-stage framework. The first stage trains a meta network named _PruningNet_ to generate various structures. _PruningNet_ takes randomly sampled encoding vectors representing structures and learns in an end-to-end manner. In the second stage, an evolutionary search algorithm is deployed to search for the optimal structures under constraints. No fine-tuning is needed at search time since _PruningNet_ predicts the weights for all the pruned nets.

In contrast to MetaPruning [148], ABCPruner [149] looks for the optimal layer-wise channel number with a one-stage approach and requires no extra supporting network. In addition, it drastically reduces the combinations of pruned structures by limiting the preserved channels to a given
space. To search for the optimal pruned structures, an evolutionary algorithm based on the _Artificial Bee Colony algorithm (ABC)_[264] is applied.

Instead of using evolutionary algorithms (EA) on the entire network, Cooperative CoEvolution algorithm for Pruning (CCEP) [150] uses _cooperative coevolution_[265] that groups the network by layer and applies EA on each layer. By decomposing the networks into multiple groups, the search space is drastically decreased. The overall framework uses an iterative prune-and-fine-tune strategy. In each iteration, multiple individual candidates (parents) are generated by randomly pruning filters. Offsprings are generated by conducting parents with bit-wise mutations, where each bit represents the presence of a filter. Individuals are then evaluated with accuracy and FLOPs, and only top-\(k\) are retained for the next iteration.

### _Extensions_

#### 2.7.1 Lottery Ticket Hypothesis

The Lottery Ticket Hypothesis (LTH) [266] claims that "dense, randomly-initialized, feed-forward networks contain subnetworks (winning tickets) that -- when trained in isolation -- reach test accuracy comparable to the original network in a similar number of iterations." _Weight rewinding_ is always used in LTH-based papers. In the study by Frankle & Carbin [266], weights and the learning rate schedule are rewound to the values at epoch \(k=0\), in order to find the sparse subnetworks at initialization. Frankle _et al._[267] propose to rewind the weights and learning rate schedule to \(k>0\) epoch to deal with SGD randomness (noise). Different means of achieving unstructured LTH have been proposed: SNIP [268] by preserving the training loss; GraSP [269] by preserving the _gradient flow_; Neural Tangent Kernel (NTK) [270] by capturing the _training dynamics_; GF [271] by pruning weights that cause the least change to the _gradient flow_.

Renda _et al._[272] propose to rewind the learning rate schedule but not the weight value (_learning rate rewinding_). Experiments show that rewinding techniques consistently outperform fine-tuning, in which _learning rate rewinding_ outperforms or matches _weight rewinding_ in all scenarios.

Rethinking the Value of Network Pruning (RVNP) [152] re-evaluates the value of network pruning. The argument is that traditional fine-tuning techniques work no better than pruning from scratch. Extending LTH in a structured pruning setting for large-scale datasets fails to yield the lottery ticket.

Early Bird (EB) [153] proposes to find the winning ticket in the early stages of training, rather than fully training the dense network. To find the EB ticket, the mask for filters is computed based on the Hamming distance between two subnetworks pruned from the same model.

Prospect Pruning (ProsPr) [154] argues that the _trainability_ of the pruned networks should be considered. _Trainability_ exists as the model is going to be trained after pruning. Thus, _meta-gradients_ (gradient-of-gradients) have been proposed as a measure of _trainability_. Instead of estimating the changes in the loss at initialization, the effect of pruning on the loss over several steps of gradient descent at the beginning of training is estimated in ProsPr.

Early Compression via Gradient Flow Preservation (EarlyCroP) [155] achieves early pruning by solving three key problems. The problem **1) why to prune** is solved by extending a GF-based pruning criterion [271] to structured pruning. **2) How to prune** is solved by preserving the training dynamics leveraging using a connection between the _gradient flow_ and NTK [270]. Lastly, **3) when to prune** is solved by finding the _lazy kernel regime_, which is the phase where pruning has little effect on the _training dynamics_.

Pruning-aware Training (PaT) [156] tries to resolve the problem of _when early pruning should begin_? PaT is based on the premise that sub-networks having the same number of remaining neurons can have very different architectures. To determine when the architectures are stable, a novel metric called the _early pruning indicator (EPI)_ that computes the structure similarity of two subnetworks has been proposed.

#### 2.7.2 Joint Compression

Some prevailing techniques, i.e., architecture search, decomposition and quantization, are orthogonal to pruning methods for neural network compression. Applying these techniques in sequence may seem like a natural extension but can lead to sub-optimal solutions due to different optimization objectives [165]. Therefore, researchers have proposed to apply these techniques jointly.

**Pruning and NAS**: NPAS [161] develops a framework of joint network pruning and architecture search. This method is compiler-aware and replaces computational activation functions such as the sigmoid function with compiler-friendly ones.

**Pruning and Quantization:** DJPQ [162] jointly performs structured pruning and mixed-bit precision quantization in a gradient-based manner. This structured pruning is based on the variational information bottleneck [104], and mixed-bit precision quantization is extended to power-of-two bit-restricted quantization. Bayesian Bits (BB) [163] unifies the view of mixed precision quantization and pruning, and introduces mixed precision gates on filters for structured pruning. IODF [164] utilizes integer-only arithmetic based on 8-bit quantization and has learnable binary gates to eliminate redundant filters during inference.

**Pruning, NAS, and Quantization:** APQ [165] performs these tasks jointly. A once-for-all network [273] supporting a large search space for channel number is trained. Next, a quantization-aware accuracy predictor is designed to evaluate the accuracy of the selected structure with mixed-precision quantization [274]. This is followed by the use of an evolutionary-based architecture search [275] to find the best-accuracy model under latency or energy constraints.

**Pruning and Decomposition:** Some studies have exploited structured pruning and low-rank decomposition within a unified framework. The authors of Hinge [166] contend that pruning techniques cannot deal with the last CONV layer in a residual block that low-rank decomposition is able to. The two techniques are hinged by introducing _sparsity-inducing matrices_ after filters. Imposing group sparsity on the columns and rows of _sparsity-inducing matrices_ achieves filter pruning and decomposition respectively. Collaborative Compression (CC) [167] simultaneously learns the model sparsity and low-rankness to achieve channel pruning and tensor decomposition jointly.
#### 2.7.3 Special Granularity

Apart from unstructured pruning at the weight level and structured pruning at the filter level, there are also pruning methods at other granularities. To categorize these methods, we define three basic dimensions: output dimension \(N_{l+1}\), input dimension \(N_{l}\), and kernel dimension \(K^{2}\). Filter pruning can be viewed as pruning along dimension \(N_{l}\times K^{2}\), and channel pruning is along dimension \(N_{l+1}\times K^{2}\). They are similar since pruning filters in the \(l\)-th layer will remove the corresponding channels in the \(l+1\)-th layer. More different pruning granularities are shown in Fig. 3.

**Group along basic dimensions:** GBD [90] and SSL [91] introduce _group-wise pruning_ (Fig. 3c) which prunes weights located at the same position among all filters in a layer, that is, along the output dimension \(N_{l+1}\). SWP [170] prunes filters in a _stripe-wise_ (Fig. 3d) manner along input dimension \(N_{l}\). Compared to _group-wise pruning_, _stripe-wise pruning_ maintains filter independence, for there is no pattern across filters. PCONV [171] uses _kernel-wise pruning_ (Fig. 3e) to prune weights along kernel dimension \(K^{2}\).

**Diverse in basic dimensions:** Some researchers find making the same pruning decision along basic dimensions may not be optimal, and they propose to look into the basic dimensions. GKP-TMI [172] uses _grouped kernel pruning_ (Fig. 3f). On top of _kernel-wise pruning_, this method groups different filters and makes diverse pruning decisions in the output dimension \(N_{l+1}\). The remaining kernels are re-permuted to output a densely structured pruned network that benefits from parallel computing. 1xN [173] (Fig. 3f) uses a _1xN pruning pattern_ which treats \(N\) output kernels sharing the same input channel index as the basic pruning granularity. PCONV [171] uses _pattern-based pruning_ (Fig. 3g) by looking into the kernels and making different pruning decisions in the kernel dimension \(K^{2}\). _Block-wise pruning_[161] (Fig. 3h) generalizes _pattern-based pruning_[171] to share a same pattern for a kernel group, where kernels are grouped in both the output dimension \(N_{l+1}\) and the input dimension \(N_{l}\). **Other possible granularities.** There are some more possible structured granularities such as grouped stripe (extension of Fig. 3d) and grouped pattern (extension of Fig. 3g). Multi-granularity [178] also deserves investigation in structure pruning.

**Layer-level:** Besides breaking down filters into smaller granularies, Shallowing Deep Networks (SDN) [174] uses _layer-wise pruning_ to prune entire layers. Specifically, SDN places _a linear classifier probe_[276] after each layer to evaluate the effectiveness of the layer. The network is retrained with the knowledge distillation technique after pruning unimportant layers.

## 3 Future Directions

### _Pruning Topics_

Every subsection in Section 2 has the potential for further development in the future. Besides, structured pruning will continue to borrow ideas from unstructured pruning. In this section, we will discuss some promising topics directly related to pruning.

**Pruning theory:** Apart from utilizing optimization tools for pruning in Section 2.4, some works view pruning from the perspective of synaptic flow [179], signal propagation [180], and graph theory [181]. Besides, the pruning process can be guided by leveraging the interpretations of a model [182], the loss landscape [110], generalization-stability trade-off [183] and entropy of a model [184]. Moreover, the theory behind LTH is investigated with Logarithmic Pruning [185]. In addition, different training methods [186, 187, 188] can be considered together with pruning. The above directions have the potential in structure pruning.

**Pruning mechanism:** Researchers have new views on the prevailing three-stage training-pruning-retraining mechanism. First, the Lottery Ticket Hypothesis, which is proposed on unstructured pruning, is expected to extend on structured pruning. Second, single-shot pruning [189, 190] prunes only once to get the pruned models. Structured pruning can potentially benefit from this mechanism. Third, AC/DC training [191] is able to co-training the dense and

Fig. 3: Explain different pruning granularities in three basic dimensions: the output dimension \(N_{l+1}\), the input dimension \(N_{l}\), and the kernel dimension \(K^{2}\). “Along” means the same pruning decision made to all weights at the same position in this dimension. In contrast, “diverse” indicates pruning results are different in this dimension. _(a) Filter-wise_ and _(b) Channel-wise_ are the well-known structured pruning. Remaining _(c) - (h)_ are special granularities. _(c) Group-wise pruning_ groups weights along the output dimension \(N_{l+1}\). _(d) Stripe-wise pruning_ groups weights along the input dimension \(N_{l}\). _(e) Kernel-wise pruning_ groups weights along the kernel dimension \(K^{2}\). _(f) Grouped kernel pruning_ groups kernels across several different filters in \(N_{l+1}\) dimension. _(g) Pattern-based pruning_ utilizes pre-defined patterns to group weights in \(N_{l}\) dimension. _(h) Block-wise pruning_ groups kernels with the same pattern in both \(N_{l+1}\) and \(N_{l}\) dimensions.
sparse models. Therefore, handling multiple models during pruning and training is another promising direction for structured pruning.

**Pruning Rate:** Structured pruning can also extend the current weight pruning strategies on researching the layer-wise pruning ratios [192, 193].

**Pruning Domain:** Utilizing representation in the frequency domain [194, 64] to guide pruning is another interesting direction.

### _Pruning for Specific Tasks_

Pruning techniques can be applied to other tasks to achieve high computational efficiency. Here are some straightforward examples: super-resolution [202], personal re-id [203], medical imaging diagnostic [204], face attribute classification [205] and ensemble learning [206, 207]. Apart from tasks mentioned above, there are some emerging directions that are still in the early stages but are promising in the future.

**Pruning for Federated Learning:** Federated learning [277] aims to solve the problem of training a model without transferring data to a central location. Pruning [195, 196] helps alleviate communication costs required between devices and servers. Zhang _et al._[195] contend that the non-Independent and Identically Distributed (non-IID) degree for each device and server is different, so it proposes to compute the different pruning ratio from each device and an aggregated expected pruning ratio for the server.

**Pruning for Continual Learning:** Continual Learning tackles the problem of catastrophic forgetting [197]. Some pioneering works [198, 199, 200] use the pruned filters of the sparsified network to train a new task, so the training process do not cause deterioration to the performance of previous tasks.

**Limited Dataset:** Dataset compression and pruning [278] are emerging. The idea is to utilize part of the images from the training dataset to train the network. Structured pruning [201] has lots of topics to investigate along with this new trend.

### _Pruning Specific Networks_

Besides the prevailing CNNs, pruning is beneficial to other type of neural networks such as MLPs [216], rectifier neural networks [217], spiking neural networks [218, 219].

**Pruning GAN:** Generative adversarial network (GAN) [208] consists of a generative network and a discriminative network, and it requires large storage and FLOPs. Different pruning methods [209, 210, 201] are proposed achieve efficient GAN.

**Pruning Transformers:** Vision transformers [279] have achieved state-of-the-art (SOTA) performance in the area of computer vision. There has been a growing trend to incorporate the architectural design of CNNs into Transformer models [15, 16, 17, 18, 19]. Therefore, adopting these structured pruning techniques to compress these novel architecture designs is meaningful [14, 212].

**Pruning for AGI:** In recent years, foundation models and generalist agent [213] are the possible way to Artificial general intelligence (AGI). Foundation models such as GPT-3 [214] are trained on a huge amount of data and can be adapted to many downstream tasks. The generalist agent Gato [215] works as a multi-modal, multi-task, multi-embodiment generalist policy. These huge models can benefit from the research of structured pruning to achieve higher efficiency.

### _Pruning Targets_

During past years, the target of pruning has shifted from reducing the number of parameters in unstructured pruning to minimizing the FLOPs in structured pruning. Recently, the target of pruning has been evolving to meet the practical demands of real-world scenarios.

**Hardware:** Incorporate pruning to the hardware process is an emerging trend. Different types of hardware require different pruning techniques. Hardware compiler-aware pruning [220] conducts pruning based on the structural information of subgraphs constructed during compiler tuning.

**Energy:** As the deployment of AI models expands, the rising energy consumption of AI models needs greater attention. Considering energy during pruning needs to be investigated. Energy-aware pruning [221] greedily prunes the layer that consumes the most energy, for minimizing the MACs may not necessarily reduces the most energy consumption.

**Robustness:** The robustness of a network describes how easily a network is fooled to make a wrong prediction under attacks [222]. Consider the robustness and computational cost of models together during the model design process is a new trend. Researchers find sparsity brought by pruning can improve the adversarial robustness [280]. Linearity Grafting (Grafting) [223] uses the premise that network robustness favors linear functions and proposes the _linearity grafting_ method. ANP-VS [224] identifies the latent features with high vulnerability and proposes a Bayesian framework to prune these features by minimizing adversarial loss and feature-level vulnerability.

## 4 Related Works

**Unstructured Pruning (Weight Pruning):** The idea of pruning is first introduced by LeCun _et al._[238] who propose _saliency_ to measure the importance of each weight. The _saliency_ is computed by the diagonal elements of the Hessian matrix. Han _et al._[9] propose to iteratively prune weights with \(\ell_{1}\) regularization and retrain. Guo _et al._[252] propose _connection splicing_ which restores the wrongly pruned weights. In addition, Guo _et al._ incorporate the _connection splicing_ into the training process to dynamically learn the weights [252]. Han _et al._[1] adopt a 3-stage pipeline to prune weights and achieve a 49\(\times\) compression ratio with merely accuracy loss on VGG-16 [5], bringing back the research focus on network pruning.

**Quantization:** Quantization in model compression describes the process of approximating full precision weights (_32-bit floating point_) with lower precision weights (e.g., _8-bit integer_) to reduce computation and storage cost. The low-bit quantization includes binarization [281, 10] and ternarization [282, 283]. These methods can be roughly divided into two categories. First, heuristic quantization methods quantize weights by using mapping functions [281],
[282]. Second, non-heuristic methods formulate quantization as an optimization problem [283, 10]. In addition, the quantization process also includes clustering and parameter sharing [40], and quantization can be combined with pruning [284, 285, 1].

**Decomposition:** Decomposition is referring to matrix decomposition and tensor decomposition, where a matrix is just a 2-D tensor. Specifically, a tensor is decomposed or factorized into products of several low-rank tensors that approximate the original tensor. To decompose 2-D tensors (matrices), _full-rank decomposition_[286], _singular value decomposition (SVD)_[287], and _QR_[288] are arguably the most popular low-rank matrix decomposition techniques. To deal with 4-D tensors, methods or concepts such as _Tucker_[289], _classical prolongation (CP)_[290], and _tensor networks_[291] are applied. Furthermore, decomposition is orthogonal to pruning, and these two techniques are applied together [292, 293, 294].

**Knowledge Distillation:** The idea of transferring knowledge from a usually large network to a small network is first introduced by Bucilua _et al._[295], and the idea is later well-known as Knowledge Distillation [12]. The distillation process includes _(1) offline distillation_[12] which takes a pre-trained teacher model. Distillation can be _(2) online distillation_[296] which simultaneously updates the teacher model and the student model in an end-to-end manner. Furthermore, _(3) self-distillation_[297] requires only one model and distills the knowledge from deeper sections to shallow sections. In addition, knowledge distillation can work with pruning to achieve few-shot compression which requires few data [298, 299, 300].

**Neural Architecture Search:** The objective of neural architecture search [25] is to minimize human intervention in the search for neural network designs. In this regard, three factors play a significant role. (1) The search space defines potential candidates. Selecting candidates with prior knowledge reduces the frequently vast search space, but human bias may hinder performance. (2) Search strategies are the search space exploration algorithms. Simple taxonomy involves Bayesian optimization [301], evolutionary methods [39], gradient-based methods [302], and reinforcement learning [303]. (3) The evaluation strategy is used to assess the predictive ability of the model on unseen data. To address the prohibitive computational cost of the model's evaluation, efficient methods are developed.

## 5 Experiments

Table II provides straightforward comparisons among different structured pruning methods of ResNet-50 [7] on ImageNet [304]. This first trend is the target network changes from VGGNet [5] before 2017 to ResNet [7] after 2017. From 2017 to 2020, some studies on MobileNets [305] were proposed. Another obvious trend is to achieve a larger FLOPs reduction with a lower accuracy loss. For example, in 2018, SFP [115] reduced 41.8% FLOPs with 1.54% top-1 accuracy drop. In 2022, ResRep [83] achieved 51.2% FLOPs reduction without any accuracy drop.
