Learning Instance-wise Sparsity for Accelerating Deep Models
Chuanjian Liu Yunhe Wang Kai Han Chunjing Xu Chang Xu
cs.CV cs.LG


Genetic Deep Learning for Lung Cancer Screening
    Hunter Park 

Innovation Dx Inc.

hpark@innovationdx.com
Connor Monahan 

Innovation Dx Inc. 

cmonahan@innovationdx.com

    today
=============================================================================================================================




	Exploring deep convolutional neural networks of high efficiency and low memory usage is very essential for a wide variety of machine learning tasks. Most of existing approaches used to accelerate deep models by manipulating parameters or filters without data, e.g., pruning and decomposition. In contrast, we study this problem from a different perspective by respecting the difference between data. An instance-wise feature pruning is developed by identifying informative features for different instances. Specifically, by investigating a feature decay regularization, we expect intermediate feature maps of each instance in deep neural networks to be sparse while preserving the overall network performance. During online inference, subtle features of input images extracted by intermediate layers of a well-trained neural network can be eliminated to accelerate the subsequent calculations.  We further take coefficient of variation as a measure to select the layers that are appropriate for acceleration. Extensive experiments conducted on benchmark datasets and networks demonstrate the effectiveness of the proposed method.




§ INTRODUCTION




Deep learning methods, especially convolutional neural networks have been successfully applied a number of computer vision tasks, such as image classification <cit.>, object detection <cit.>, image super-resolution <cit.>, . However, most of deep CNNs are well designed with huge parameters and computational complexities for the accuracy reason. For example, VGG16 <cit.> needs about 60 MB memory and 15.4 GFLOPs (floating-number operations) for a 224×224 image. This limits their usage in edge devices, , mobile phones, smart camera. Thus, effective methods for compressing and speeding-up these deep models are urgently required.

To this end, considerable methods have been proposed to reduce memory and FLOPs.  <cit.> employed vector quantization which makes use of a cluster center to represent similar connections. <cit.> exploited the singular value decomposition approach and decomposed the weight matrices of fully connect layers. Considering that 32-bit floating numbers are too fine for CNNs, <cit.> and <cit.> explored binarized neural networks, whose weights and activations are -1/1. <cit.> utilized pruning, quantization and Huffman coding together. In addition, <cit.> introduced the discrete cosine transform (DCT) bases and converted convolution filters into the frequency domain, thereby producing much higher compression ratio and speed improvement. Although these approaches can provide considerable compression and speed-up ratios for the original heavy CNNs, most of them are equally applied on all instances in the train and test set, which ignore the differences between instances.

Actually, natural images are of different complexities for a given neural network, , an image with clean background and moderate objects is easier to be accurately recognized than an image filled with complex textures and multiple objects as shown in Figure <ref>. To this end, some recent works proposed to explore accelerating methods for different instances,  <cit.> embedded a RNN to discard useless channels, <cit.> proposed a multi-branch scheme, . In fact, convolution filters are designed for learning some intrinsic patterns in natural images, , edges, blobs and color, and natural images are combined by these patterns. Thus, discarding same features for all instances is not the optimal solution for reducing the complexity of deep neural networks. Moreover, a higher sparsity ratio is possible for each instance than on entire dataset. Therefore, these methods have a very huge potential for real-world applications.

Although the above mentioned approaches have made tremendous efforts to address the existing problems for applying deep models, most of them discard useless features by introducing additional operations on the output feature maps such as RNN, RL, . Thus, we propose a new framework for learning the instance-wise sparsity in well designed neural networks by exploiting a feature decay regularization. In practice, a feature decay regularization is utilized to make features of different training instances sparse during the training procedure. Then, some features of instances can be discarded without significantly affecting the performance of the resulting network at runtime. Extensive experiments conducted on benchmark models and datasets demonstrate the proposed method can achieve competitive performance but with lower computational complexities.



§ RELATED WORKS

Conventional model compression and acceleration approaches are designed for removing redundant filters, weights and blocks to obtain compact architectures from pre-trained deep neural networks, which ignore the complexities of different instances in image datasets. 

In order to excavate the complexity of each instance, several works are proposed for assigning different parts of the designed network to different input data dynamically. For example, <cit.> utilized attention and gate layers to evaluate each channel and discard some of them with subtle importances during the inference phrase. <cit.> utilized a gate cell to discard some layers in pre-trained deep neural networks for efficient inference. <cit.> further proposed the branch selection operation to allow the learned neural networks to change themselves according to different input data. <cit.> applied the dynamic strategy on the activations of feature maps in neural networks.

In summary, most of the above mentioned approaches did not directly process convolution filters or features and introduce extra components, , gates, attention layers, importance predictors and multi-branches to original CNNs, which involves more computations and parameters. In this paper, we attempt to address the dynamic strategy in the feature pruning aspect, and discard redundant features to reduce the computational complexity during the online inference.



§ METHODS




In this section, we investigate the structural sparsity of deep neural networks by a feature decay regularization for obtaining efficient CNN models. In the training procedure, instance-wise ℓ_2,1-norm is included for making features of all instances sparse, while less important features will be eliminated to reduce computational complexity during the inference, as shown in Figure <ref>.



 §.§ Weight Pruning for Neural Networks

Given a deep convolutional neural network, denote weights of the l-th convolutional layer as a sequence of 4D tensors, , W^(l)∈ R^C_l+1× C_l× D_l × K_l, where C_l+1, C_l, D_l and K_l are the dimensions of the l-th weight tensor along the axes of filter, channel, height and width, respectively. The conventional method for compressing deep neural networks can be formulated as:

    min_Wℒ(W, 𝒳, 𝒴) + λ·∑_l=1^L R(W^(l)),

where ℒ(·) stands for classification loss for training sample (𝒳, 𝒴), R(·) is the weight regularization item in deep neural networks, and λ is a hyper parameter for seeking the trade-off between accuracy and model size.

According to different concerns, the weight regularization R(·) could have various formulations,

	
  - R(W^(l))=∑_c_l+1=1^C_l+1| W_c_l+1,:,:,:^(l)|_1 is utilized for removing useless convolution filters <cit.>;
	
  - R(W^(l))=∑_c_l=1^C_l| W_:,c_l,:,:^(l)|_1 can be applied to eliminate redundant channels <cit.>;
	
  - R(W^(l))=∑_d_l=1^D_l∑_k_l=1^K_l| W_:,:,d_l,k_l^(l)|_1 is often embedded into deep neural networks for discarding subtle weights in CNNs <cit.>.


Wherein, the ℓ_1-norm applied on each convolution filters, , filter pruning is the most effective scheme for removing useless filters in the pre-trained network and provide a new network with the same number of layers but significantly fewer parameters and FLOPs. However, pruning filters is equal to discard some of output features for all instances, which ignores the difference between complexities of various images. In fact, convolution filters are designed for learning some intrinsic patterns in natural images, , edges, blobs, and color, and images consist of various combinations of these patterns. Thus, discarding all features is not the optimal solution for reducing the complexity of deep neural networks.



 §.§ Instance Feature Sparse Regularization


As mentioned above, we can obtain a network with fewer convolution filters compared to the original network by minimizing the regularization of convolution filters in pre-trained deep neural networks. In fact, the regularization applied on convolution filters can be shifted to the corresponding features. For an optimal feature elimination, we propose to utilize the ℓ_2,1-norm to learn sparse features of different instances, which is a widely used regularization to select features <cit.>. In practice, the ℓ_2,1-norm for an arbitrary matrix M with n rows and m columns is defined as:

    | M|_2,1 = ∑_i=1^n√(∑_j=1^mM_ij^2),

which first calculates the ℓ_2-norm for each row and then stack them by utilizing the ℓ_1-norm.

Then, we incorporate Eq.<ref> in the training procedure of deep neural networks as shown in the left part in Figure <ref>. Specifically, let F_N× C_l× H_l× W_l^l be the features of l-th convolutional layer, N, C_l, H_l and W_l are batch size, channel number, height and width of this layer, respectively. For the c_l-th channel of l-th layer of n-th instance, we compute ℓ_2-norm for this channel: 

    N_n,c_l^l = √(∑_h_l=1^H_l∑_w_l=1^W_l(F_n,c_l,h_l,w_l^l)^2).

For each instance, we then compute its ℓ_1-norm over all layers in the network: 

    N_n = ∑_l=1^L∑_c_l=1^C_lN_n,c_l^l.

Thus, the entire objective function of the proposed method can be formulated as

    ℒ_task+λ·∑_n=1^NN_n,

where ℒ_task is final loss function of the given visual task such as classification or detection. λ is regularization parameter used for controlling the effect of corresponding feature regularization item. In contrast to weight sparsity, Eq.<ref> learns the feature sparsity of each instance in the training procedure simultaneously, which is a more accurate approach for removing redundancy in deep neural networks.



 §.§ Instance-wise Sparsity Inference

A novel framework for learning instance-wise feature redundancy is proposed in the above section. In order to implement the online inference efficiently, we further explore the testing procedure of the proposed method. The right part of Figure <ref> illustrates the detailed inference procedure of the proposed method. 
For an arbitrary input image, we first compute the ℓ_2-norm of feature maps of each convolutional layer using Eq.<ref>. 

The feature regularization in training leads to multi-polarization of features, so we utilize the coefficient of variation (CV) to calculate dispersion of norm distribution for each layer in the online inference. It is defined as the ratio of the standard deviation to the mean as follows:

    𝒞𝒱_l=√(1/C_l∑_c_l=1^C_l(N_n,c_l^l-μ_l)^2)/μ_l,

where

    μ_l=∑_c_l=1^C_lN_n,c_l^l/C_l.
In probability theory and statistics, CV is a standard measure of dispersion of a probability distribution, and it is independent of data scales. The larger CV value of some layer means greater changes in importance of channels, and we tend to drop channels from such layer. We set a global CV threshold as α, the layers whose CV values are larger than α could be pruned. 

After determined the prunable layers, we further find the specific channels that could be pruned in these layers. Since the magnitude of feature norms in different layers is various, we cannot use a global threshold to eliminate useless features. Alternatively, we compute the mean μ as Eq.<ref> of ℓ_2-norms for each layer and set a drop threshold β∈ [0,2). The channels whose ℓ_2-norm value is under β·μ are discarded, and only the remained channels will be transmitted into the following layers.



 §.§ Computation Complexity


Let F_N× C_l× H_l× W_l^l be feature map on the l-th layer, and the size of filters on the (l+1)-th layer is W_C_l+1× C_l× k× k. Standard convolutions have the computation cost of: C_l× C_l+1× H_l× W_l× k× k. The computation cost of L_2-norms is C_l× H_l× W_l, and the computation cost of mean and CV is about 4C_l. This added computation cost could be ignored in contrast to the convolution. As a result, if we discard C channels for a layer, then we can save C/C_l ratio of computation cost approximately at runtime.



§ EXPERIMENTS


In this section, we evaluate feature regularization on general image classification task. Detailed analysis on instance-wise feature sparsity brought by the feature regularization loss are presented. In addition, we demonstrate the instance-wise  channel pruning results and reduced FLOPs.



 §.§ Datasets and Experimental Settings


We extensively evaluate our methods on two popular classification datasets: CIFAR-10 <cit.> and Imagenet(ILSVRC2012) <cit.>. Three networks are considered: VGG16 <cit.>, resnet-18(Res18 for simplicity) <cit.> and mobilenet-v2 (MNet2 for simplicity) <cit.>. The MNet2 is more challenging to accelerate because of its compactness. Besides, to further verify our method's effectiveness, we did experiments on the network which is compressed by network slimming <cit.>. 

For VGG16, we substitute the fully connection layer with global average pooling in all experiments. We use filters with size 3×3 and stride 1 on the first convolutional layer for Res18 in CIFAR-10. For MNet2, we reset the strides in first convolutional layer and the second inverted residual block layer to 1. For preprocessing and other super-parameters in training,  we follow the same routines proposed for these networks.



 §.§ CIFAR-10


In this section, we make comprehensive experiments and analysis on the instance-wise feature sparsity on CIFAR-10 dataset. A number of ℓ_2,1-norm regularization factors are considered, λ=0, 1e-6, 1e-7, 1e-8 respectively. When λ=0, it is equivalent to standard approach. The baseline results are shown in Table <ref>. The feature regularization restricts the feature representation power of deep models, so larger λ leads to lower accuracy.





  
Variation of feature distribution. In our method, CV is a metric measuring the changes in importance of channels. In Table <ref>, we show the changes of the CV values of some convolutional layers. The standard VGG16 has small CV values usually, which means the importance of different channels are similar. In contrast to standard VGG16, we get higher CV values for VGG16 with proposed feature regularization. Additionally, the CV values increase with λ. With the increase of CV value, the importance of channels produces polarization. In the following, we will explore the trade-off between different CV value thresholds and prune ratio and accuracy.







  
Visualization of pruned channels. In Figure <ref> each grid in the heatmap represents the number of samples of one category that discard this channel (there are 1000 samples for each category in CIFAR-10 test set). The low level layers of VGG16 have the similar channel prune results for different categories. In our opinion, the low layers in CNN usually learn the pixel level information, which are shared by all categories. In the higher layer, more differences of dropped channels presented by this figure. The higher layer has more semantic information of categories, so they tend to discard different channels for different categories. In our model, the middle layers such as conv3_x have merely been pruned due to small CV value. The 10 easy samples that dropped more channels and 10 hard samples that dropped less channels during inference are shown in Figure <ref>. From these examples, we acknowledge that easy samples usually have pure background or texture, moderate size, . The hard samples present complex background and texture, truncated objects, .



  
Statistics of easy and hard samples. We group the samples from easy to hard in accordance with the number of pruned channels. From less to more, we compute the accuracy of every 100 samples by the number of pruned channels for VGG16 in Figure <ref>. Unusually, the first 100 easy samples have low accuracy in contrast to the global average accuracy. So we check all these images manually, and find the misclassified samples usually have small object or confusing background. The hard samples have poor accuracy as we believed, and more attention should be paid to these examples to improve accuracy. Besides, we find no significant difference in channel pruning ratio for different categories, the reason lies in that CIFAR-10 has few categories and the test set collects all kinds of instances for each category.





  
Results of instance-wise feature pruning of VGG16. We evaluate the proposed instance-wise feature pruning method with random pruning strategy and ℓ_2-norm based pruning. The random pruning strategy is randomly discarding fixed ratio of channels, while the ℓ_2-norm based pruning is discarding fixed ratio of channels with minimum ℓ_2-norm values. Our method is pruning channels that exceed predefined thresholds CV factor α and drop threshold β. From Table <ref> and <ref>, the model with large feature regularization factor λ=1e-6 gets 56.0% accuracy after pruning 95.6% channels for VGG16. The reason lies that large regularization factor limits the representation ability of features, which leads to low accuracy but high feature sparsity. The random method leads to bad accuracy under the same prune ratio, and the norm based method gets relative better results. Our methods has higher accuracy and pruned ratio simultaneously. We use different combinations of CV value α and drop threshold β. In Table <ref>, for low α and β, the standard VGG16 gets 94.0% accuracy and 20.7% pruned ratio, and the regularized VGG16 gets 93.6% accuracy and 46.9% prune ratio when λ=1e-7. With low α and high β, the standard VGG16 gets low accuracy 79.2%, and the regularized VGG16 still gets 92.6% accuracy after pruned 53.2% channels. This proves that feature regularization leads to the uneven distribution of features. Then for high α and β, the standard VGG16 gets 93.8% accuracy but very low prune ratio 4.2%, and the regularized VGG16 gets 93.3% accuracy under 45.1% pruned ratio. For all these methods, with the increase of λ(except λ=1e-6), the test accuracy grows under the same prune ratio. This certificates that large regularization factor λ produces large feature sparsity, and more channels could be pruned without sacrificing performance.







  
Results of instance-wise feature pruning of light-weight CNNs. To further verify the effectiveness of feature regularization, the results on light networks, MNet2 and pruned VGG16 by nets slimming <cit.>, are shown in Table <ref> and Table <ref>. For MNet2, the average prune ratio is bigger than 0.7, which suggests that the features in MNet2 are redundant. By check the prune ratio of each layer, we find that the expand layer(expand ratio is 6) in inverted residual block of MNet2 are pruned much more than the shrink layer, even exceed 95% channels are pruned in the higher expand convolutional layers. The depthwise separable convolution itself in standard MNet2 is beneficial to the feature sparsity, which leads to better accuracy and higher prune ratio in contrast to standard VGG16. The regularized MNet2 gains an advantage when CV thresh becomes larger. For pruned VGG16, the baseline of VGG16 with BN scale factor regularization is 0.938. The accuracy becomes 0.574 after prune 67% channels with minimum BN scale factor. The we fine-tune this pruned model with proposed feature regularization. In Table <ref>, the second column when α=0 and β=0 is the baseline  accuracy of fine-tuned on pruned VGG16 with different regularization factors. From the last column, we find that another 20.1% or 27.2% computations could be saved when the accuracy drops about 0.8% or 1.7%, respectively.









  
Comparative analysis.  We compare our methods with several dynamic accelerate methods.  <cit.> used low-cost collaborative layer(LCCL) to pre-compute the non-zero elements in the out feature maps, then ignored the locations with zeros. Channel gating(Gating) <cit.> worked by identifying the unimportant channels and turning off them. <cit.> proposed feature boosting and suppression, a new method to predictively amplify salient convolutional channels and skip unimportant ones at run-time. As shown in Table <ref>, the feature regularization method provides a better trade-off between error and FLOPs reduction under the similar network architecture. The feature regularization method is easy to implement in contrast to other methods which need extra network structures. These dynamic accelerated methods are complementary to static methods such as pruning, quantification and low rank decomposition. It is possible to obtain better accuracy and FLOPs reduction by combining these methods to statical models.



 §.§ Imagenet


ImageNet is a large dataset, which has 1.3M images and 1000 categories. Due to the diversity of categories and richness of images, more features are expected to be utilized by deep neural networks in order to get higher discriminability. 

Figure <ref> presents the easy and hard samples of Imagenet. There are more categories in Imagenet in contrast to CIFAR-10, the skew of channel prune ratio for different categories emerged. Some categories like 'window screen', 'space bar' and 'window shade' have more channels been dropped, these categories can be regarded as easy categories. The hard categories contains 'butcher shop', 'jinrikisha', 'oxcart', etc. 





From Table <ref>, some interesting results are found. Firstly, the MNet2 has near half of channels been dropped in spite of its compactness, which is less than the pruned channels in CIFAR-10. The inverted residual block in MNet2 expands the channels for the inputs initially, then shrink channels in the last convolutional layer of this block. Detailed analysis on the dropped channels show that almost all the dropped channels locate in the expand layer, which means that one small expand ratio in inverted residual block may get equivalent accuracy with large expand ratio. The result is consistent with CIFAR-10. Secondly, we find the standard MNet2 also has better accuracy under relatively lower prune ratio in contrast feature regularization method. But in the similar accuracy 69.4% and 69.2%, feature regularized model has more channels 47.5% to 38.5% been pruned.




§ CONCLUSION


The differences between instances make the dynamical pruning become possible. This paper introduces structural feature regularization on CNN models in the training procedure to get instance-wise sparsity, and prune useless channels during inference for each instance. The proposed method is not designed for reducing disk memory occupation and GPU space. In fact, we tend to accelerate the inference by selectively computing only a subset of channels that are important for the input. As a side effect, the amount of cached activations and the number of read, write and arithmetic operations would be largely decreased, which leads to the improvement of memory occupation and computation at runtime. Within 1% accuracy loss, we demonstrate that VGG16 can drop 50% channels on CIFAR-10, and MNet2 can drop 47% channels on Imagenet. In addition, the inference channel pruning method can be easily combined with other existing static network compression methods to get better speed-up ratio. Further to mine the restrictions on features maps to improve efficiency or performance of networks will be studied.



§ ACKNOWLEDGMENTS


Chang Xu was supported by the Australian Research Council under Project DE180101438.



	named
	


