# Neural Network Compression via
Sparse Optimization

Tianyi Chen111Microsoft, Redmond, WA, USA. E-mail: {tiachen, yixshi, bif,
shengyi<EMAIL_ADDRESS>444These authors were equally contributed. ,
Bo Ji222National University of Singapore, Singapore. Email:
<EMAIL_ADDRESS>44footnotemark: 4 , Yixin Shi11footnotemark: 1 , Tianyu
Ding333Johns Hopkins University, Baltimore, MD, USA. Email<EMAIL_ADDRESS>,
Biyi Fang11footnotemark: 1 , Sheng Yi11footnotemark: 1 , Xiao
Tu11footnotemark: 1

###### Abstract

The compression of deep neural networks (DNNs) to reduce inference cost
becomes increasingly important to meet realistic deployment requirements of
various applications. There have been a significant amount of work regarding
network compression, while most of them are heuristic rule-based or typically
not friendly to be incorporated into varying scenarios. On the other hand,
sparse optimization yielding sparse solutions naturally fits the compression
requirement, but due to the limited study of sparse optimization in stochastic
learning, its extension and application onto model compression is rarely well
explored. In this work, we propose a model compression framework based on the
recent progress on sparse stochastic optimization. Compared to existing model
compression techniques, our method is effective and requires fewer extra
engineering efforts to incorporate with varying applications, and has been
numerically demonstrated on benchmark compression tasks. Particularly,
compared to the baseline heavy models, we achieve up to $\bm{7.2}\times$ and
$\bm{2.9}\times$ FLOPs reduction with competitive $\bm{91.2\%}$ and
$\bm{75.1}\%$ top-1 accuracy on VGG16 for CIFAR10 and ResNet50 for ImageNet,
respectively.

##  1 Introduction

In the past decade, building deeper and larger neural networks is a primary
trend for solving and pushing the accuracy boundary in varying artificial
intelligent scenarios (Long et al., 2015; Vaswani et al., 2017). However,
these high capacity networks typically possess significantly heavier inference
costs in both time and space complexity especially when used with embedded
sensors or mobile devices where computational resources are usually limited.
In addition, for cloud artificial intelligent service, the servers operating
on a limited latency budget (Shi et al., 2020) benefit remarkably from the
deployed models to be computationally efficient to maintain the reliable
service level agreement during heavy cloud traffic. For these applications, in
addition to satisfactory generalization accuracy, computational efficiency and
small network sizes are crucial factors to business success.

### Literature Review:

There have been numerous efforts devoted to network compression (Buciluǎ et
al., 2006) to achieve the speedup and efficient model inference, where the
studies are largely evolved into (i) weight pruning, (ii) quantization, and
(iii) knowledge distillation. The weight pruning mainly focuses on given a
heavy model with high generalization accuracy, filtering out redundant kernels
to achieve slimmer architectures (Gale et al., 2019), including Bayesian
pruning (Zhou et al., 2019; Neklyudov et al., 2017; Louizos et al., 2017),
ranking kernels importance (Luo et al., 2017; Hu et al., 2016; He et al.,
2018a; Li et al., 2019) and compression by reinforcement learning (He et al.,
2018b). The quantization is the process of constraining network parameters
onto lower precision (Wu et al., 2016). With network parameters quantized, the
majority of operations, e.g., convolution and matrix multiplication, can be
efficiently estimated via the approximate inner product computation (Jacob et
al., 2018; Zhou et al., 2016). The knowledge distillation constructs a
teacher-student training pipeline to transfer the knowledge from heavy and
accurate teacher network to a prescribed given smaller student network (Hinton
et al., 2015; Polino et al., 2018).

We have the perspective that these existing methods tackle the network
compression from different aspects and work as complementary to each other,
wherein we study the weight pruning method in this work. We note that the
existing weight pruning methods are perhaps either simple and heuristic (Li et
al., ; Luo et al., 2017), hence easily hurt generalization accuracy; or
sophisticated, so that numerous engineering efforts are required to reproduce
and employ on varying applications (He et al., 2018b), which may be
prohibitive to applications with complicated training procedures. Therefore,
establishing a simple and effective compression pipeline becomes indispensable
especially along with the irresistible popularization of various artificial
intelligent applications.

On the other hand, to remove redundancy of heavy models in convex learning,
sparse optimization, which augments the problem with a sparsity-inducing
regularization term to yield sparse solutions (including many zeros) (Beck &
Teboulle, 2009; Chen, 2018), serves as one of the most effective methods in
classical feature engineering (Tibshirani, 1996; Roth & Fischer, 2008), where
the redundancy is trimmed based on the distribution of zero entries on the
solution. However, its analogous applications in non-convex deep learning to
filter redundancy in networks is far away from well explored. In fact,
although there exist quite a few works formulating sparse optimization to
compress networks, they either do not actually utilize the sparsity ratio of
zero weight parameters, but rank their magnitudes to filter out weight
parameters of which norm are sufficiently small (Li et al., 2016); or perform
not comparable to other approaches (Wen et al., 2016; Lin et al., 2019; Gale
et al., 2019). As investigated in Xiao (2010); Xiao & Zhang (2014); Chen et
al. (2020a; b), the main reason regarding the limited performance of sparse
optimization on network compression is that effective sparsity exploration in
stochastic learning is hard to be achieved due to the randomness and limited
sparsity identification mechanism, i.e., the solutions computed by most of
stochastic sparse optimization algorithms are typically fully dense. Hence,
the previous compression techniques by sparse optimization may not utilize the
accurate sparsity signal to achieve effective model compression.

Moreover, the sparsity identification in stochastic non-convex learning are
improved to some extent by the recent progress made by orthant-face and half-
space projection for $\ell_{1}$-regularization and group sparsity
regularization problems respectively (Chen et al., 2020a; b). Therefore, it is
natural to ask if the recent progress on stochastic sparse optimization is
applicable and beneficial to the network compression scenario. In this work,
we attempt to re-investigate the utility of sparse optimization on the weight
pruning task to search approximately optimal model architectures.

Figure 1: Overview of alternating schema.

### Our Contributions:

We propose a novel network compression framework based on the recent sparse
optimization algorithm OBProx-SG for solving $\ell_{1}$-regularization, so-
called Recursive Sparse Pruning (RSP) method. Compared to other weight pruning
methods, RSP is easily plugged into various applications with competitive
performance. We now summarize our contributions as follows.

  * •

_Algorithmic Design:_ We formulate the model compression problem into an
optimization problem, and design an alternating recursive schema as shown in
Figure 1 to search an approximately optimal solution wherein the optimal model
architecture requires much fewer time and space cost for inference and
achieves competitive accuracy to the original full model. Compared to other
pruning methods, RSP is more friendly to incorporate with various applications
with much fewer efforts, as the fundamental herein step, i.e., redundancy
exploration, is tackled by a well-packaged optimization algorithm (OBProx-SG)
applicable for broad training pipelines.

  * •

_Numerical Experiments:_ We numerically demonstrate the effectiveness of our
compression approach on two benchmark model compression problems, i.e., VGG16
on CIFAR10 and ResNet50 on ImageNet (ILSVRC2012) with up to $7.2\times$ and
$2.9\times$ FLOPs reduction and only slight accuracy regression. Compared to
the results presented in Zhou et al. (2019), our approach outperforms all
other pruning methods on both FLOPs reduction and accuracy maintenance
significantly.

##  2 Notations and Preliminaries

Consider a CNN model consisting of $L$ convolutional layers, which are
interlaced with various activation functions, e.g., rectifier linear units,
and pooling operators, e.g., max pooling. For the convolution operation on
$l$-th convolutional layer, an input tensor $\mathcal{I}^{l}$ of size
$C^{l}\times H^{l}_{\mathcal{I}}\times W^{l}_{\mathcal{I}}$, is transformed
into an output tensor $\mathcal{O}^{l}$ of size $C^{l+1}\times
H^{l}_{\mathcal{O}}\times W^{l}_{\mathcal{O}}$ by the following linear
mapping:

$\mathcal{O}^{l}_{c^{\prime},h^{\prime},w^{\prime}}=\sum_{c=1}^{C_{l}}\sum_{i=1}^{H_{\mathcal{K}}^{l}}\sum_{j=1}^{W_{\mathcal{K}}^{l}}\mathcal{K}^{l}_{c^{\prime},c,i,j}\mathcal{I}_{c,h_{i},w_{j}}^{l},$
(1)

where the convolutional filter $\mathcal{K}^{l}$ at the $l$th layer is a
tensor of size $K^{l}\times C^{l}\times H_{\mathcal{K}}^{l}\times
W_{\mathcal{K}}^{l}$ with $K^{l}=C^{l+1}$. Particularly, we denote
$\mathcal{K}^{l}_{c^{\prime}}\in\mathbb{R}^{C^{l}\times
H_{\mathcal{K}}^{l}\times W_{\mathcal{K}}^{l}}$ as the $c^{\prime}$-th kernel
for the $l$-th convolutional layers. Similarly to Lin et al. (2019), we
flatten tensor $\mathcal{K}^{l}$ into a filter matrix $\mathcal{K}^{l}$ of
size $K^{l}\times{C^{l}H_{\mathcal{K}}^{l}W_{\mathcal{K}}^{l}}$. Hence, the
$c^{\prime}$-th row of $\mathcal{K}^{l}$ exactly store the parameters for the
$c^{\prime}$-th kernel.

Additionally, we consider several norms of the filter matrix $\mathcal{K}^{l}$
that are widely used to formulate sparsity inducing regularization problem.
Particularly, one is the $\ell_{1}$-norm of $\mathcal{K}^{l}$ as

$\left\|\mathcal{K}^{l}\right\|_{1}=\sum_{c^{\prime}}\left\|\mathcal{K}^{l}_{c^{\prime}}\right\|_{1},$
(2)

which induces a sparse solution while the zero elements may be distributed
randomly. To encode more sophisticated group sparsity structure, mixed
$\ell_{1}/\ell_{2}$ norm is introduced as

$\left\|\mathcal{K}^{l}\right\|_{1,2}=\sum_{c^{\prime}}\left\|\mathcal{K}^{l}_{c^{\prime}}\right\|_{2}$
(3)

which induces the entire rows of $\mathcal{K}^{l}$ to be zero during solving
sparse optimization problem. In this work, we will focus on leveraging the
sparse $\ell_{1}$-regularization optimization to compress heavy networks and
pay attention to mixed $\ell_{1}/\ell_{2}$-regularization to encode group
sparsity in the future work.

##  3 Network Compression Problem Formulation

Essentially, the network compression is to search a slimmer model architecture
given a prescribed network topology along with minimizing the scarification on
generalization accuracy, which can be formulated as the following constrained
optimization problem,

$\begin{split}{\displaystyle\mathop{\operator@font{minimize}}_{\bm{w},\mathcal{M}}}&\
f(\bm{w}~{}|~{}\mathcal{M},\mathcal{D})\\\ s.t.&\ |\mathcal{M}|\leq
T,\end{split}$ (4)

where $\mathcal{M}$ denotes the model architecture associated with its weight
parameter $\bm{w}$, $f$ is the loss function to measure the deviation from the
predicted output given evaluation dataset $\mathcal{D}$ on
$(\bm{w},\mathcal{M})$ to the ground truth output, and the constraint
$|\mathcal{M}|\leq T\in\mathbb{Z}^{+}$ restricts the size of model
$\mathcal{M}$. It is well-recognized that (4) is NP-hard and intractable in
general. But one can solve it approximately by transforming into non-
constraint optimization problem as

${\displaystyle\mathop{\operator@font{minimize}}_{\bm{w},\mathcal{M}}}\
f(\bm{w}~{}|~{}\mathcal{M},\mathcal{D})+\lambda\cdot\Omega(\mathcal{M}),$ (5)

where the constraint in (4) is augmented into the original objective as an
extra plenaty term $\Omega(\mathcal{M})$ associated with a Lagrangian
coefficient $\lambda$. The $\Omega$ serves as a regularization regarding the
size of model $\mathcal{M}$, which can be further relaxed to the
$\ell_{1}$-norm of weight parameters $\bm{w}$ on $\mathcal{M}$,

${\displaystyle\mathop{\operator@font{minimize}}_{\bm{w},\mathcal{M}}}\
f(\bm{w}~{}|~{}\mathcal{M},\mathcal{D})+\lambda\cdot\left\|\bm{w}\right\|_{1}.$
(6)

Compared to (4) and (5), problem (6) is tractable by our following proposed
alternating algorithm, where we iteratively update $\bm{w}$ given fixed
$\mathcal{M}$, then construct $\mathcal{M}$ given computed $\bm{w}$ till
convergence. The solution $(\bm{w}^{*},\mathcal{M}^{*})$ of problem (6) are
expected to possess smaller model size and similar evaluation accuracy
compared to the initial heavy model $\mathcal{M}^{0}$ to accelerate the
inference efficiency.

##  4 The Compression Method

In this section, we provide a comprehensive introduction of our compression
technique via sparse optimization as illustrated by Figure 1. Our RSP prunes
redundant filters from a high capacity model for computational efficiency
while avoiding sacrificing the generalization accuracy. In general, to solve
the formulated compression problem (6), our RSP as stated in Algorithm 1 is a
framework consisting of two stages, (i) an iteratively alternating compression
schema to search an approximately optimal model architecture, i.e.,
iteratively repeat updating $\bm{w}^{t}$ given $\mathcal{M}^{t}$ then updating
$\mathcal{M}^{t+1}$ given $\bm{w}^{t}$ till convergence as line 2 to 5 in
Algorithm 1; and (ii) fine-tune model parameters given searched optimal model
architecture as line 6 in Algorithm 1. In the reminder of this section, we
first present how the alternating compression schema works, then move on to
explain the fine-tuning of network weight parameters, and conclude this
section by revealing the Bayesian interpretation of our approach.

Algorithm 1 The Network Compression via Sparse Optimization Framework

1:Input: Initial high capacity CNN architectures $\mathcal{M}^{0}$ and dataset
$\mathcal{D}$.

2:while Network compression does not converge. do

3: Update $\bm{w}^{t}$ given $\mathcal{M}^{t}$: Train weight parameter
$\bm{w}^{t}$ on dataset $\mathcal{D}$ as
$\displaystyle\bm{w}^{t}\leftarrow\text{SparseOptimization}(\mathcal{M}^{t},\mathcal{D},\lambda^{t}),$
(Explore redundancy)

4: Update $\mathcal{M}^{t+1}$ given $\bm{w}^{t}$: Construct model
$\mathcal{M}^{t+1}$ by Algorithm 2,
$\mathcal{M}^{t+1}\leftarrow\text{Compression}(\mathcal{M}^{t},\bm{w}^{t})$
(Compress $\mathcal{M}^{t}$)

5: Update $\lambda^{t+1}$: $\lambda^{t+1}$ computes proportionally to the
model size ratio between $\mathcal{M}^{t+1}$ and $\mathcal{M}^{t}$,
$\lambda^{t+1}\leftarrow|\mathcal{M}^{t+1}|~{}/~{}|\mathcal{M}^{t}|\cdot\lambda^{t}.$
(7)

6:Global Fine Tuning: Retrain to attain weights $\bm{w}^{*}$ given the optimal
model $\mathcal{M}^{*}$.

7:Output: Pruned CNN architecture $\mathcal{M}^{*}$ with tuned weight
parameters $\bm{w}^{*}$.

### Explore Redundancy, update $\bm{w}^{t}$ given $\mathcal{M}^{t}$.

Given the current possibly heavy model architecture $\mathcal{M}^{t}$, we aim
at evaluating its redundancy of each layer in order to construct smaller
structures with the same prediction power. To achieve it, we first formulate
and train $\mathcal{M}^{t}$ under a sparsity-inducing regularization to attain
a highly sparse solution $\bm{w}^{t}$, i.e.,

$\bm{w}^{t}=\operatorname*{arg\,min}_{\bm{w}}F\left(\bm{w}~{}|~{}\mathcal{M}^{t},\mathcal{D}\right):=f(\bm{w}~{}|~{}\mathcal{M}^{t},\mathcal{D})+\lambda^{t}\cdot\left\|\bm{w}\right\|_{1},$
(8)

where the $\left\|\bm{w}\right\|_{1}$ penalizes on the density and magnitude
of the network weight parameters; theoretically, under proper $\lambda^{t}$,
the solution of problem (8) $\bm{w}^{t}$ tends to have both high sparsity and
evaluation accuracy. The problem as form of (8) is well-known as
$\ell_{1}$-regularization problem, which is widely used for feature selection
in convex applications to filter out redundant features by the sparsity of
solutions. Consequently, it is natural to consider to apply such
regularization to determine redundancy of deep neural networks. Unfortunately,
although problem as form of (8) has been well studied in convex deterministic
learning (Beck & Teboulle, 2009; Keskar et al., 2015; Chen et al., 2017), its
study in stochastic non-convex applications, e.g., deep learning, is quite
limited, i.e., the solutions computed by existing stochastic optimization are
typically dense. Such limitation on sparsity exploration results in that the
$\ell_{1}$-regularization is typically not used in a standard way to identify
redundancy of network compression, where it is leveraged to rank the
importance of filters only in the aspect of the magnitude but not on the
sparsity in the recent literatures (Li et al., 2016).

We note that a recent orthant-face prediction method (OBProx-SG) (Chen et al.,
2020a) made a breakthrough on the study of $\ell_{1}$-regularization
especially on the sparsity exploration compared to classical proximal methods
in stochastic non-convex learning (Xiao, 2010; Xiao & Zhang, 2014).
Particularly, the solutions computed by OBProx-SG typically have multiple-
times higher sparsity and competitive objective convergence than the solutions
computed by the competitors. Hence, we employ the OBProx-SG onto line 3 in
Algorithm 1 to seek highly sparse weight parameters $\bm{w}^{t}$ on model
architecture $\mathcal{M}^{t}$ without accuracy regression.

### Compress model, update $\bm{M}^{t+1}$ given $\bm{w}^{t}$ and
$\mathcal{M}^{t}$.

Given the highly sparse weight parameters $\bm{w}^{t}$ of model architecture
$\mathcal{M}^{t}$ yielded by OBProx-SG, as the high-level procedure presented
in Algorithm 2, we now perform a specified compression algorithm, e.g., filter
pruning, to construct a smaller model architecture $\mathcal{M}^{t+1}$,
wherein the sparsity ratio of each building block is leveraged to represent
the redundancy of corresponding layers. Particularly, taking the fully
convolution layers (no residual or skip connection) as a concrete example, as
stated in Algorithm 3, the sparsity ratio of each $\ell$-th convolutional
layer is computed as line 2 to 10 by counting the percentage of zero elements.
Then following the backward path of $\mathcal{M}^{t+1}$, the $\ell$-th
convolutional layer shrinks its number of filters proportionally to computed
sparsity ratio $s^{\ell+1,t}$ as line 5 to 11. Finally the architecture of
$\mathcal{M}^{t+1}$ is finalized after a calibration on the potential
structure inconsistency during pruning to guarantee the ultimate network
generating outputs in the desired shape.

### Update the regularization coefficient $\lambda^{t+1}$ given
$\lambda^{t},\mathcal{M}^{t}$ and $\mathcal{M}^{t+1}$.

The regularization coefficient $\lambda$ in problem (6) balances between the
sparsity level on the exact optimal solution and the generalization accuracy.
Larger $\lambda$ typically results in higher sparsity on the computed solution
$\bm{w}$ by OBProx-SG but sacrifices more on the bias of model estimation.
Hence, to achieve both low objective value and high sparse solution, $\lambda$
requires careful selection, of which setting is usually related to the number
of variables involved per iteration during optimization. Thus, we scale
$\lambda^{t}$ up according to the model size ratio between $\mathcal{M}^{t+1}$
and $\mathcal{M}^{t}$ to yield $\lambda^{t+1}$ as line 5 in Algorithm 1.

Algorithm 2 Compress Step for General CNNs

1:Input: CNN model $\mathcal{M}^{t}$ of $L^{t}$ building blocks, sparse
weights $\bm{w}^{t}$.

2:for $\ell=1,\cdots,L^{t}$ do

3: Compute the empirical sparsity ratio $s^{\ell,t}$ given $\bm{w}^{t}$ as
$s^{\ell,t}\leftarrow\frac{\text{\\# of zeros in}\
\mathcal{K}^{l,t}}{\text{\\# of elements in}\ \mathcal{K}^{l,t}}$ (9)

4:Remove redundancy in each building block by sparsity ratio
$\\{s^{\ell,t}\\}_{\ell=1}^{\ell=L^{t}}$ to form $\mathcal{M}^{t+1}$.

5:Output: The CNN model $\mathcal{M}^{t+1}$.

Algorithm 3 Compress Step for Fully Convolutional Layers

1:Input: Parameter $\bm{x}^{t}$, CNN model $\mathcal{M}^{t}$ of $L^{t}$
convolutional layers and weight parameters $\bm{w}^{t}$.

2:for $\ell=1,\cdots,L^{t}$ do

3: Compute the empirical sparsity ratio $s^{\ell,t}$ given $\bm{w}^{t}$ as
$s^{\ell,t}\leftarrow\frac{\text{\\# of zeros in}\
\mathcal{K}^{l,t}}{\text{\\# of elements in}\ \mathcal{K}^{l,t}}$ (10)

4:Initialize $\mathcal{M}^{t+1}$ with $L^{t+1}\leftarrow L^{t}$ convolution
layers of the same architectures as $\mathcal{M}^{t}$:

5:for $\ell=L^{t+1}-1,\cdots,1$ do

6: In $\ell$-th convolution layer, shrink the number of filters from
$K^{\ell,t}$ to  $K^{\ell,t+1}=\left\lceil
K^{\ell,t}\cdot(1-{s}^{\ell+1})\right\rceil$ (11)

7:Calibrate the architecture of $\mathcal{M}^{t+1}$.

8:Output: The CNN model $\mathcal{M}^{t+1}$.

### Convergence Criteria

The stopping criteria of the alternating schema is designed delicately to
ensure obtaining the approximate minimal model architecture without accuracy
regression. Our design monitors the several evaluation metric, e.g., sparsity,
model size and validation accuracy, during the whole procedure. In particular,
if the evolution of sparsity of $\bm{w}$ and model size $\mathcal{M}$ becomes
flatten or the validation accuracy regresses, the alternating schema is
considered as convergent and return the best checkpoint as the optimal model
$\mathcal{M}^{*}$ as line 2 in Algorithm 1.

### Global Fine-Tuning

After searching the optimal model $\mathcal{M}^{*}$, of which size tends to be
much smaller than the original $\mathcal{M}^{0}$. In the final step, we
retrain the network to strengthen the remaining neurons after compression to
enhance the generalization performance of the trimmed network as line 6 in
Algorithm 1.

### Bayesian Interpretation

It is well recognized that $\ell_{1}$-regularization in problem (8) is
equivalent to adding a Laplace prior onto the training problem to enhance the
occurrence of zero entries on the solutions. From this point of view, our
method can be categorized into weight pruning by Bayesian compression, where
other methods constructs other priors into the original objectives (Zhou et
al., 2019; Louizos et al., 2017; Neklyudov et al., 2017).

##  5 Numerical Experiments

In this section, we validate the effectiveness of the proposed method RSP on
two benchmark datasets CIFAR10 (Krizhevsky & Hinton, 2009) and ImagetNet
(ILSVRC2012) (Deng et al., 2009). The CNN architectures to prune include VGG16
(Simonyan & Zisserman, 2014) and ResNet50 (He et al., 2016). We mainly report
floating operations (FLOPs) to indicate acceleration effect. Compression rate
(CR) is also revealed as another criterion for pruning.

### Experimental Settings

For OBProx-SG called in Algorithm 1, the initial regularization coefficient
$\lambda^{0}$ is fine-tuned from all powers of 10 between $10^{-2}$ to
$10^{-5}$ as the one can achieve the same generalization accuracy as the
existing benchmark accuracy. The mini-batch sizes for CIFAR10 and ILSVRC2012
are selected as 64 and 512 respectively The learning rates follow the setting
as He et al. (2016) and decay periodically by factor $0.1$. In practice, due
to the expansiveness of training cost, we empirically set the number of
alternating update as one.

###  VGG16 on CIFAR10

We first compress VGG16 on CIFAR10, which contains $50,000$ images of size
$32\times 32$ in training set and $10,000$ in test set. The model performance
on CIFAR10 is qualified by the accuracy of classifying ten-class images.
Considering that VGG16 is originally proposed for large scale dataset, the
redundancy is naturally obvious. In realistic implementation, we add a lower
bound of compression ratio, i.e., $\epsilon>0$, into line 11 of Algorithm 3,
such that

$\begin{split}K^{\ell,t+1}=\left\lceil
K^{\ell,t}\cdot\max\\{1-{s}^{\ell+1},\epsilon\\}\right\rceil\end{split}$ (12)

to guarantee each layer maintain a fair number of kernels left after
compression and select $\epsilon$ as 0.1. We report the pruning results in
Table 1, including existing state-of-the-art compression techniques reported
in Zhou et al. (2019). Particularly, Table 1 shows that RSP is no doubt the
best compression method on the FLOPs reduction, by achieving $7.2\times$
speedups whereas others stuck on about $3\times$ speedups. Remark here that
although RSP reaches slight lower compression ratio than RBP because of the
universal setting of $\epsilon$ for both convolution and linear layers, the
FLOPs reduction of RSP is roughly 2 times higher due to the more significant
computational acceleration on the early layers. If we switch the gear to
compare the baseline validation accuracy, RSP only slightly regressed about
$0.4\%$, which outperforms the majority of other compression techniques as
well.

Table 1: Comparison of pruning VGG16 on CIFAR10. Convolutional layers are in
bold.

Method | Architecture | CR | FLOPs | ACC
---|---|---|---|---
Baseline |  64-64-128-128-256-256-256-512-512-512-512-512-512-512-512 | $1.0\times$ | $1.0\times$ | 91.6
SBP |  47-50-91-115-227-160-50-72-51-12-34-39-20-20-272 | $17.0\times$ | $3.2\times$ | 91.0
BC |  51-62-125-128-228-129-38-13-9-6-5-6-6-6-20 | $18.6\times$ | $2.6\times$ | 91.0
RBC |  43-62-120-120-182-113-40-12-20-11-6-9-10-10-22 | $25.7\times$ | $3.1\times$ | 90.5
IBP |  45-63-122-123-246-199-82-31-20-17-14-14-31-21-21 | $13.3\times$ | $2.3\times$ | 91.7
RBP |  50-63-123-108-104-57-23-14-9-8-6-7-11-11-12 | $39.1\times$ | $3.5\times$ | 91.0
RSP |  55-31-65-63-115-75-43-52-52-52-52-52-52-52-52 | $38.9\times$ | $\bm{7.2\times}$ | 91.2
|  |  |  |

###  ResNet50 on ILSVRC2012:

We now prune ResNet50 on ILSVRC2012 (Deng et al., 2009). ILSVRC2012, well
known as ImageNet, is a large-scale image classification dataset, which
contains $1,000$ classes, more than $1.2$ million images in training set and
$50,000$ in validation set. ResNet50 is a very deep CNN in the residual
network family. It contains 16 residual blocks (He et al., 2016), where around
50 convolutional layers are stacked. For ResNet, there exist some restrictions
due to its special structure. For example, the channel number of each block in
the same group needs to be consistent in order to finish the sum operation.
Thus it is hard to prune the last convolutional layer of each residual block
directly. Hence, we consider the residual block as the basic unit rather than
convolutional layer to proceed compression during Algorithm 2. Particularly,
the sparsity ratio of each residual block is computed and then leveraged to
prune redundant filters similarly to the strategy in Luo et al. (2017). The
compression results is described in Table 2, where the RSP definitely performs
the best, with remarkable FLOPs reduction and highest Top 1 and 5 accuracy.

Table 2: Comparison of pruning ResNet50 on ILSVRC2012. Method | FLOPs | Top-1 Acc. | Top-5 Acc.
---|---|---|---
Baseline | $1.0\times$ | $76.1$ | $92.9$
DDS | $1.7\times$ | $71.8$ | $91.9$
CP | $1.5\times$ | $72.3$ | $90.8$
ThiNet-50 | $2.3\times$ | $71.0$ | $90$
RBP | $2.3\times$ | $71.1$ | $90.0$
RSP | $\bm{2.9\times}$ | $\bm{75.1}$ | $\bm{91.9}$
|  |  |

##  6 Conclusions and Future Work

In this report, we propose a model compression framework based on the
$\ell_{1}$-regularized sparse optimization and the recent breakthrough on
sparsity exploration achieved by OBProx-SG. Our RSP contains a recursive
alternating schema to search an approximately optimal model architecture to
accelerate network inference without accuracy scarification by leveraging the
computed sparsity distribution during algorithmic procedure. Compared to other
Bayesian weight pruning methods, the RSP is no doubt the best in terms of
FLOPs reduction and accuracy maintenance on benchmark compression experiments,
i.e., ResNet50 on ImageNet by reducing FLOPs by $2.9\times$ and achieving
$75.1\%$ top-1 accuracy. As the future work, we remark here that how to
utilize the recent group-sparsity optimization algorithm HSPG (Chen et al.,
2020b) is an open problem to remove entire redundant hidden structures from
networks directly.

## References

  * Beck & Teboulle (2009) Amir Beck and Marc Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  _SIAM journal on imaging sciences_ , 2(1):183–202, 2009.
  * Buciluǎ et al. (2006) Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 535–541, 2006.
  * Chen (2018) Tianyi Chen.  _A Fast Reduced-Space Algorithmic Framework for Sparse Optimization_.  PhD thesis, Johns Hopkins University, 2018.
  * Chen et al. (2017) Tianyi Chen, Frank E Curtis, and Daniel P Robinson.  A reduced-space algorithm for minimizing $\ell_{1}$-regularized convex functions.  _SIAM Journal on Optimization_ , 27(3):1583–1610, 2017.
  * Chen et al. (2020a) Tianyi Chen, Tianyu Ding, Bo Ji, Guanyi Wang, Yixin Shi, Sheng Yi, Xiao Tu, and Zhihui Zhu.  Orthant based proximal stochastic gradient method for $ell\\_1$-regularized optimization.  _arXiv preprint arXiv:2004.03639_ , 2020a.
  * Chen et al. (2020b) Tianyi Chen, Guanyi Wang, Tianyu Ding, Bo Ji, Sheng Yi, and Zhihui Zhu.  A half-space stochastic projected gradient method for group-sparsity regularization.  _arXiv preprint arXiv:2009.12078_ , 2020b.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Gale et al. (2019) Trevor Gale, Erich Elsen, and Sara Hooker.  The state of sparsity in deep neural networks.  _arXiv preprint arXiv:1902.09574_ , 2019.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016.
  * He et al. (2018a) Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang.  Soft filter pruning for accelerating deep convolutional neural networks.  _arXiv preprint arXiv:1808.06866_ , 2018a.
  * He et al. (2018b) Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li, and Song Han.  Amc: Automl for model compression and acceleration on mobile devices.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 784–800, 2018b.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hu et al. (2016) Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang.  Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.  _arXiv preprint arXiv:1607.03250_ , 2016.
  * Jacob et al. (2018) Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2704–2713, 2018.
  * Keskar et al. (2015) Nitish Shirish Keskar, Jorge Nocedal, Figen Oztoprak, and Andreas Waechter.  A second-order method for convex $\ell_{1}$-regularized optimization with active set prediction.  _arXiv preprint arXiv:1505.04315_ , 2015.
  * Krizhevsky & Hinton (2009) A. Krizhevsky and G. Hinton.  Learning multiple layers of features from tiny images.  _Master’s thesis, Department of Computer Science, University of Toronto_ , 2009.
  * (17) H Li, A Kadav, I Durdanovic, H Samet, and HP Graf.  Pruning filters for efficient convnets. arxiv 2016.  _arXiv preprint arXiv:1608.08710_.
  * Li et al. (2016) Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.  Pruning filters for efficient convnets.  _arXiv preprint arXiv:1608.08710_ , 2016.
  * Li et al. (2019) Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji.  Exploiting kernel sparsity and entropy for interpretable cnn compression.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2800–2809, 2019.
  * Lin et al. (2019) Shaohui Lin, Rongrong Ji, Yuchao Li, Cheng Deng, and Xuelong Li.  Toward compact convnets via structure-sparsity regularized filter pruning.  _IEEE transactions on neural networks and learning systems_ , 31(2):574–588, 2019.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3431–3440, 2015.
  * Louizos et al. (2017) Christos Louizos, Karen Ullrich, and Max Welling.  Bayesian compression for deep learning.  In _Advances in neural information processing systems_ , pp. 3288–3298, 2017.
  * Luo et al. (2017) Jian-Hao Luo, Jianxin Wu, and Weiyao Lin.  Thinet: A filter level pruning method for deep neural network compression.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5058–5066, 2017.
  * Neklyudov et al. (2017) Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov.  Structured bayesian pruning via log-normal multiplicative noise.  In _Advances in Neural Information Processing Systems_ , pp. 6775–6784, 2017.
  * Polino et al. (2018) Antonio Polino, Razvan Pascanu, and Dan Alistarh.  Model compression via distillation and quantization.  _arXiv preprint arXiv:1802.05668_ , 2018.
  * Roth & Fischer (2008) Volker Roth and Bernd Fischer.  The group-lasso for generalized linear models: uniqueness of solutions and efficient algorithms.  In _Proceedings of the 25th international conference on Machine learning_ , pp. 848–855, 2008.
  * Shi et al. (2020) Yixin Shi, Aman Orazaev, Tianyi Chen, and Sheng YI.  Object detection and segmentation for inking applications, September 24 2020.  US Patent App. 16/360,006.
  * Simonyan & Zisserman (2014) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Tibshirani (1996) Robert Tibshirani.  Regression shrinkage and selection via the lasso.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 58(1):267–288, 1996.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Wen et al. (2016) Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li.  Learning structured sparsity in deep neural networks.  In _Advances in neural information processing systems_ , pp. 2074–2082, 2016.
  * Wu et al. (2016) Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng.  Quantized convolutional neural networks for mobile devices.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4820–4828, 2016.
  * Xiao (2010) Lin Xiao.  Dual averaging methods for regularized stochastic learning and online optimization.  _Journal of Machine Learning Research_ , 11(Oct):2543–2596, 2010.
  * Xiao & Zhang (2014) Lin Xiao and Tong Zhang.  A proximal stochastic gradient method with progressive variance reduction.  _SIAM Journal on Optimization_ , 24(4):2057–2075, 2014.
  * Zhou et al. (2016) Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou.  Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients.  _arXiv preprint arXiv:1606.06160_ , 2016.
  * Zhou et al. (2019) Yuefu Zhou, Ya Zhang, Yanfeng Wang, and Qi Tian.  Accelerate cnn via recursive bayesian pruning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 3306–3315, 2019.