Exploiting the Redundancy in Convolutional Filters for Parameter Reduction
Kumara Kahatapitiya Ranga Rodrigo
cs.CV

	
	
	
	
	
	
	
	
	
	
	
	

	
Exploiting the Redundancy in Convolutional Filters for Parameter Reduction 

    Kumara Kahatapitiyawork done at University of Moratuwa, Sri Lanka. 

		Stony Brook University, Stony Brook, NY

		kkahatapitiy@cs.stonybrook.edu
		Ranga Rodrigo

		University of Moratuwa, Sri Lanka

		ranga@uom.lk 
	
     ^⋆  Laboratoire d'Informatique et des Systèmes, Université de Toulon, Aix Marseille Univ, CNRS UMR 7020, LIS, Marseille, France

boizot@univ-tln.fr; 

^†  Department of Mathematics, Lehigh University, Bethlehem, PA, USA. 

lus219@lehigh.edu; 

 

This research has been partially supported by the ANR SRGI (reference ANR-15-CE40-0018). 


This is a post-peer-review, pre-copyedit version of an article published in Journal of Mathematical Imaging and Vision (JIMIV). The final authenticated version is available online at: <http://dx.doi.org/10.1007/s10851-020-00990-5>

   
      
   

 
 

====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

empty


	
 
	Convolutional Neural Networks (CNNs) have achieved state-of-the-art performance in many computer vision tasks over the years. However, this comes at the cost of heavy computation and memory intensive network designs, suggesting potential improvements in efficiency. Convolutional layers of CNNs partly account for such an inefficiency, as they are known to learn redundant features. In this work, we exploit this redundancy, observing it as the correlation between convolutional filters of a layer, and propose an alternative approach to reproduce it efficiently. The proposed `LinearConv' layer learns a set of orthogonal filters, and a set of coefficients that linearly combines them to introduce a controlled redundancy. We introduce a correlation-based regularization loss to achieve such flexibility over redundancy, and control the number of parameters in turn. This is designed as a plug-and-play layer to conveniently replace a conventional convolutional layer, without any additional changes required in the network architecture or the hyperparameter settings. Our experiments verify that LinearConv models achieve a performance on-par with their counterparts, with almost a 50% reduction in parameters on average, and the same computational requirement and speed at inference. Source is available at <https://github.com/kumarak93/LinearConv>.
	
	
	
	

 



§ INTRODUCTION



Deep Learning has been widely adopted in recent years over feature design and hand-picked feature extraction. This development was supported by the improvement in computational power <cit.> and large-scale public datasets <cit.>. In such a resourceful setting, the research community has put forth deep learning models with exceptional performance at the cost of heavy computation and memory usage <cit.>. Recent studies suggest—although the automated feature learning process captures more meaningful and high-level features—that the learned features of deep learning models inherit a considerable amount of redundancy <cit.>, which in turn, contributes to unwanted storage  and computations.

Such an inefficiency restricts the deployment of deep learning models in resource-constrained environments. To this end, it is interesting to investigate the possibility of exploiting the redundancy in feature extraction to improve the efficiency of deep networks.

CNNs have become the default backbone of deep neural networks with their success in feature extraction. A set of convolutional layers extracting rich localized features, followed by a set of fully-connected layers linearly combining these extracted features, makes up a general CNN architecture which can achieve state-of-the-art performance in most computer vision tasks. 


However, when we train CNNs, the weights of convolutional layers tend to converge such that there exists a considerable correlation between the learned filters <cit.>. As a consequence, the feature space spanned by these linearly dependent filters is 

a subspace of what could be represented by the corresponding number of filters. Therefore, if carefully optimized, it is possible to span the same feature space with a smaller number of filters, at least theoretically.

Otherwise speaking, the same performance could be achieved with fewer parameters. However, in practice, an over-complete spanning set of filters is allowed, to reach fine-grained performance improvements. Although this is the case, enabling a better control over this redundancy may reveal ways of efficiently replicating the same behavior.

To this end, recent literature has explored the possibility of inducing sparsity <cit.> and separability <cit.> in convolutional filters. Although some works consider the inherent correlation in learned convolutional filters for various improvements <cit.>, it has been overlooked for parameter reduction in deep networks. Moreover, previous works fall short in conveniently controlling the feature redundancy identified as the correlation between features.




In this paper, we discuss a method to exploit the feature redundancy observed as the correlation between convolutional filters for improving efficiency. The key idea is to restrict the inherent redundancy within the filters, and re-introduce a sufficient amount to maintain the performance, but with a smaller number of parameters.
To do this, we propose a novel convolutional layer LinearConv (Fig. <ref>, <ref>) which consists of two sets of weights: primary weights 𝐖^p_l , a set of convolutional filters learned with a regularized correlation, and secondary weights 𝐖^s_l, generated by linearly combining the former[Here, we use the terms filters and weights interchangeably. However, filters correspond to weights grouped as for each convolution, i.e., weight 𝐖_l contains f_l filters.]. The linear coefficients 𝐀_l are co-learned along with the primary weights. 

The set of secondary weights is spanned by a smaller number of parameters, i.e., linear coefficients, which accounts for the parameter reduction.


In contrast to methods which propose novel or significantly modified architectures for parameter reduction, we design ours to require no change in the architecture or the training process. This allows the LinearConv layer to be plugged into any widely-deployed and tested CNN architectures without any other modifications. We verify that our models can be trained from scratch to achieve a comparable performance, cutting down the number of parameters almost in half on average, and can be used in complement with other architectures for network compression.

The main contributions of this paper are as follows:  


	
  * We propose a novel convolutional layer LinearConv, which comprises a set of primary and secondary weights. The primary filters are regularized to be linearly independent, whereas the secondary filters are generated by a learnable linear combination of the former. The proposed layer is designed as a plug-and-play component which can replace the conventional convolutional layer without any additional change in the network architecture or the hyperparameter settings. 
	
	We experimentally validate LinearConv models to achieve a performance on-par with their counterparts, with a reduced number of parameters. 
	
	

	
  * We introduce a correlation-based regularization loss which gives the flexibility and control over the correlation between convolutional filters. This regularization is applied to the primary filters of LinearConv to make them linearly independent, which in turn, allows a better span for the secondary filters.
	
	
	


The rest of the paper is organized as follows: section <ref> presents previous works in the area, followed by method and implementation details in section <ref>, evaluation details in section <ref>, and finally, conclusion in section <ref>.  





§ RELATED WORK



The capacity of deep neural networks was widely recognized after the proposal of AlexNet <cit.>, which achieved state-of-the-art performance in ILSVRC-2012 <cit.>. Since then, CNN architectures have improved different facets of deep learning, suggesting deeper <cit.>  and wider <cit.> architectures. These networks have introduced not only better learning and performance, but also a high demand in computational and storage resources.




Parameter reduction has been of interest to the community over the years, in parallel to better architectures and optimization techniques. Group convolutions <cit.> and depth-wise separable convolutions <cit.> propose to reduce the channel-wise redundancy in learned feature maps.

In <cit.>, authors predict a majority of the weights using a small subset.
Network pruning techniques either remove non-significant features and fine-tune the network <cit.>, or choose only a set of representative channels <cit.>, whereas network quantization methods propose either weight sharing <cit.> or limited precision of weights <cit.> to control the memory requirement of deep networks. Furthermore, in <cit.>, authors enforce a low-rank structure or a filter decomposition to reduce network complexity.
More recently, methods such as neural architecture search <cit.>, SqueezeNet <cit.> and EfficientNet <cit.> have improved the state-of-the-art results with reduced complexity. All these works either propose novel network architectures or modify existing architectures. In contrast, we introduce a convenient replacement for convolutional layers, which is applicable to existing network architectures without additional changes.




Correlation in convolutional filters, and the resulting redundancy have been identified in recent literature. In <cit.>, authors observe a pair-wise negative correlation in low-level features of CNNs

. A similar property of correlation is identified in <cit.>, where authors suggest to generate such features based on a separate set of correlation filters instead of learning all the redundant features. Moreover, multiple correlation-based regularization methods have been proposed to de-correlate feature activations <cit.>. From these directions, we can see that the previous works have exploited feature correlation for efficiency.

In this work, we enforce a correlation-based regularization loss on convolutional filters rather than their activations, which restricts them to be a linearly independent set of basis filters.




Linear combinations in convolutional filters have been explored to improve CNNs in multiple aspects. Separable filters <cit.> and Sparselet models <cit.> propose the idea of approximating a set of convolutional filters as a linear combination of a smaller set of basis filters. One other direction suggests linearly combining activations instead of the filters which generate them, to efficiently recreate the feature redundancy <cit.>. In <cit.>, authors generate correlated filters as a matrix multiplication with a set of correlation matrices: either hand-designed or learned. Here, each primary filter is one-to-one mapped to a dependent filter. We follow a similar process, but instead of learning a one-to-one mapping, we learn a set of linear coefficients which combines a set of learnable filters to generate a set of correlated filters.

Our work is closely related but orthogonal to the recently proposed Octave convolutions <cit.> and Basis filters <cit.>, as well as concurrent work LegoNet <cit.> and GhostNet <cit.>. LegoNet <cit.> assembles a shared set of Lego filters as filter modules, whereas GhostNet <cit.> generates new activations based on cheap operations. Octave convolutions <cit.> separate convolutional filters into high-frequency and low-frequency components, processing the latter in low resolution to reduce computations, while keeping the same number of parameters. By design, OctConv is a plug-and-play component similar to ours. However, we follow an intuition based on correlation instead of frequency, reducing the number of parameters. Basis filters <cit.> learn a basis of split-weights and linearly combine the activations generated by them, achieving a lower complexity. In contrast, we linearly combine the filters themselves, not their activations. Moreover, Basis filters require the weights of original network for training, which means the cumulative computational and memory requirement is higher than the network it replaces. Our method has no such requirement, and can be trained from scratch without additional resources.





In essence, previous works have identified feature correlation and redundancy, utilizing them to improve the efficiency of CNNs. However, such approaches have limited control over the correlation and thus, a narrow outlook on the redundancy and its replication. In contrast, we achieve a finer control over the correlation based on the proposed regularization loss, and a flexible replication of the redundancy in a LinearConv layer. We design our method in a form that can be directly plugged into existing architectures without any additional modifications or hyperparameter tuning, which enables its fast and convenient adoption. 







§ METHOD






Our proposition is to control the correlation between convolutional filters in a layer, and in turn, its feature redundancy. To do this, first, we need to restrict the inherent correlation of the convolutional filters that is generally learned in training. Second, we should efficiently introduce a sufficient amount of redundancy to maintain the performance of the network. In this regard, we propose a novel convolutional layer LinearConv, with a primary set of filters which is learned to be linearly independent due to an associated regularization loss, and a secondary set of filters generated through a learnable linear combination of the former.







 §.§ Correlation-based Regularization Loss


KwInputInput                
KwOutputOutput              
KwParamParameters 










The intuition of the proposed correlation-based regularization loss is to reduce the inherent redundancy in the primary filters of LinearConv, which we observe as the correlation between the filters. When the correlation is penalized, the primary filters tend to become linearly independent, i.e., orthogonal. This enables the secondary filters to be spanned in a complete subspace which corresponds to the particular number of primary filters (refer subsection <ref>). Simply put, the secondary filters get more flexibility to learn the redundant features.


To do this, we define a loss which penalizes the correlation matrix between the flattened primary filters as shown in Algorithm <ref>. First, we reshape the input filters 𝐖^p_l which we wish to make orthogonal, to get a flattened set of filters 𝐕^p_l. Here, we want the correlation matrix to converge to the identity matrix, and hence, define the ℓ1-norm of the difference between the two to be our regularization loss L_c. In contrast to the conventional ℓ1 regularization, which is applied directly to the weights, our regularization is applied to the difference between the correlation and the identity matrix. At training, λ L_c is added to the target loss, where λ corresponds to the regularization strength. 





































 §.§ LinearConv operation










































Once we regularize the inherent redundancy in the convolutional filters, we need to re-introduce a sufficient redundancy to maintain the performance. We observe a considerable performance drop otherwise. The motivation behind removing and then re-applying the redundancy is to have a control over it. We do this by proposing the LinearConv layer, which has a set of orthogonal primary filters and a set of strictly linearly dependent secondary filters. 










[2]Note that the steps marked with ^* are performed in each forward pass only during training, to update 𝐀_l with backpropagation. At inference, these steps are performed only once at the layer initialization.
[3]Activation is shown here inside LinearConv layer to keep the consistency of input-output notation. In implementation, it is a separate component outside LinearConv. The LinearConv operation is presented in Algorithm <ref>. In the forward pass, it takes in the activations of the previous layer 𝐗_l-1, and outputs the activations to the next layer 𝐗_l, similar to Conv operation. The learnable parameters of LinearConv are 𝐖^p_l and 𝐀_l, in contrast to 𝐖_l in Conv. The secondary filters are generated as a linear combination of the primary filters. To update the parameters 𝐀_l during backpropagation, we have to perform this linear combination in every forward pass during training. However, at inference, this is only a one-time overhead at initialization. This means that LinearConv will have an increased computational cost during training, but a negligible change at inference. Fig. <ref> shows the LinearConv operation.

It is important to note that in LinearConv, the input 𝐗_l-1 goes through only a single convolution operation to produce  𝐗_l, similar to Conv operation. All other operations are performed on weights. Otherwise speaking, no additional operations are performed on data, having no change in the data flow through the network, compared to the architecture where Conv is replaced by LinearConv. The only difference is how the weights are defined and updated through backpropagation. The benefit of having a design with no change in the architecture is twofold: not only it can be plugged into existing widely-deployed and tested architectures, but also it can function in complement with other network compression methods.


The hyperparameter α defines the proportion of primary filters. The number of learnable parameters in LinearConv changes based on α. Thus, it plays an important role in parameter reduction, which we discuss further in the following subsection. In our experiments, we choose  α such that both α f_l and (1-α) f_l are integers, as they correspond to the number of primary and secondary filters.

We further define a rank-reduced version of LinearConv to reduce the computational cost. Here, the matrix of linear coefficients 𝐀_l is decomposed to two low-rank matrices: 𝐀_l1∈ℝ^α f_l× r and 𝐀_l2∈ℝ^r ×(1-α) f_l, where r<min{α f_l, (1-α) f_l}.

    𝐀_l =𝐀^_l1𝐀^_l2 ,  𝐔^s_l←𝐀^⊤_l2 (𝐀^⊤_l1𝐕^p_l)
























 §.§ Effect of hyperparameter α



The hyperparameter α∈ (0,1) plays an important role in LinearConv. It is defined as the proportion of primary filters, which directly affects the number of parameters. Moreover, it decides the nature of generated secondary filters as we show in the following formulation. 



On the number of parameters: Let us express the number of parameters in a Conv layer (p_l^C) and an equivalent LinearConv layer (p_l^LC) as,





    p_l^C = f_lh_lw_lc_l ,   
    p_l^LC (α) = α f_lh_lw_lc_l + α(1-α) f_l^2 ,

where p_l^LC depends on α. To have a parameter reduction in the proposed LinearConv layer, the following should hold:



    p_l^LC (α) ≤ p_l^C
    
    α≤h_lw_lc_l/f_l


In common CNN architectures, usually c_l(=f_l-1)/f_l is either 0.5 or 1, except at the input layer. Therefore, even with 1× 1 convolutions where h_l=w_l=1, the above condition is satisfied for our choice of α=0.5 (discussed below). However, for group convolutions, the effective c_l becomes c_l/g_l where g_l is the number of groups, in which case, this condition may not hold (discussed in subsection <ref>).
At the extreme, we would like to maximize the compression, i.e., minimize the number of parameters in a LinearConv layer. This can be formulated as,



    α^*
       =_α∈ (0,1) [α f_lh_lw_lc_l + α(1-α) f_l^2] ,
 
and considering the concavity of p_l^LC(α)

, α^* should be as small as possible. 




On the expressiveness of the filters: For Conv, let us define the filters 𝐖_l flattened as 𝐘_l∈ℝ^f_l× h_lw_lc_l, and for LinearConv, the primary filters 𝐖^p_l flattened as 𝐕^p_l∈ℝ^α f_l× h_lw_lc_l, the secondary filters 𝐖^s_l flattened as 𝐔^s_l∈ℝ^(1-α) f_l× h_lw_lc_l, and the matrix of linear coefficients as 𝐀_l∈ℝ^α f_l×(1-α) f_l. The filters 𝐕^p_l and 𝐔^s_l can be represented as,





    𝐕^p_l = [𝐯^p_l,1; ⋯; 𝐯^p_l,α f] ,  
    𝐔^s_l = [𝐮^s_l,1; ⋯; 𝐮^s_l,(1-α) f] ,


where 𝐯^p_l,i∈ℝ^1 × h_lw_lc_l and 𝐮^s_l,i = ∑_j=1^α f_l (𝐀_l^⊤)_i,j𝐯^p_l,j∈ℝ^1 × h_lw_lc_l. We see that the span of the filters in a particular layer is important to extract better features, because in subsequent layers, the activations of these filters are re-combined to generate new features. Now, let us consider the vector space spanned by the set of  Conv filters 𝐘_l. If we have a sufficient set of linearly independent filters, i.e., if f_l≥ h_lw_lc_l, we can span a space of ℝ^h_lw_lc_l. Otherwise, if f_l< h_lw_lc_l, the spanned subspace can be re-parameterized as a space of ℝ^f_l. However, since the Conv filters can be linearly dependent, the spanned subspace can be represented as,

    span(𝐘_l) ⊆ℝ^min{h_lw_lc_l, f_l}.


In contrast, since the proposed regularization (Algorithm <ref>) is applied to the primary filters in LinearConv, 𝐕^p_l can span a complete space given by,

    span(𝐕^p_l) = ℝ^min{h_lw_lc_l, α f_l}.









When the primary filters can span a complete space as above, their linear combinations, i.e., the secondary filters have the flexibility to learn any filter in the corresponding space, which in turn results in better activations.

However, the subspace spanned by 𝐔^s_l further depends on 

the rank of the matrix 𝐀_l, as we generate (1-α) f_l secondary filters based on α f_l primary filters.




    span(𝐔^s_l)⊆span(𝐕^p_l) = ℝ^min{h_lw_lc_l, α f_l}   if (1-α) ≥α
    ℝ^min{h_lw_lc_l, (1-α) f_l},    otherwise
       span(𝐔^s_l)⊆ℝ^min{h_lw_lc_l, α f_l, (1-α) f_l}


In common CNN architectures, as f_l<h_lw_lc_l, we can see that the subspace spanned by a LinearConv layer depends on α, based on Equations <ref> and <ref>. In other words, α acts as a decisive factor for the expressiveness of the feature space. Therefore, we want α to be as large as possible to have a good basis set of primary filters. At the same time, we want to generate a sufficient redundancy. Considering Equations <ref>, <ref> and <ref>, we can say that α=0.5 strikes a balance between the expressiveness of the filters and the number of parameters. In this case, the subspace spanned by the filters in a LinearConv layer becomes,



    span(𝐔^s_l)⊆span(𝐕^p_l)=ℝ^min{h_lw_lc_l, f_l/2},

and the corresponding number of learnable parameters becomes,

    p_l^LC = 1/2f_l(h_lw_lc_l + 1/2f_l).











§ EVALUATION






 §.§ Implementation details




C[1]>p#1
















































We want to evaluate the performance of LinearConv in common CNN architectures. Therefore, we replace the Conv layers in a variety of architectures with LinearConv (details in Appendix A). Except for a baseline CNN, all the others are well-known models proposed in the literature <cit.>. AllConv <cit.> is a fully convolutional network. The rest of the models have a single fully-connected (fc) layer each, which outputs logits. 

ResNeXt-29 is based on group convolutions <cit.>, whereas in MobileNetV2 <cit.>, it becomes depth-wise separable convolutions <cit.> as the number of groups is equal to the number of input feature maps. In all configurations, convolution is followed by batch normalization <cit.> and ReLU activations <cit.>, except for AllConv, which omits batch normalization. As for pooling, Base and VGG11 use max-pooling, whereas the others use strided convolutions with stride of 2. We carefully selected these specific models to show the behavior of LinearConv in vanilla CNNs, and the networks with skip-connections and group convolutions. All our models are implemented on PyTorch <cit.>.











We evaluate our models in multiple classification datasets: CIFAR-10,100 <cit.>, Street View House Numbers (SVHN) <cit.>, MNIST <cit.> and Fashion MNIST <cit.>. All datasets are channel-wise normalized at the input. The images in MNIST and Fashion MNIST are zero padded and randomly cropped. In CIFAR-10,100 and SVHN we perform random horizontal flips and random crops. All inputs are of 32×32 resolution.


We trained the models for 250 epochs with Adam optimizer <cit.> and cross-entropy loss. The proposed regularization loss is added to the target loss, with a regularization  strength of λ =10^-2. 

We use an input batch size of 64 in all experiments. The initial learning rate is selected to be one of {10^-3,  10^-4} by observation, with a step-wise decay of 10^-1 once every 100 epochs. 

For each configuration, we report the maximum accuracy on test data, model size (number of learnable parameters) and the computational requirement.





 §.§ Ablation study





In our ablation studies, we investigate different aspects of LinearConv. Specifically, we verify how the hyperparameter α affects the performance, which we discussed previously, and the significance of generating redundant features in the proposed layer.

The hyperparameter α controls the number of primary filters in a LinearConv layer. 
For smaller  α, LinearConv will have a larger proportion of secondary filters, extracting more redundant features.

When α=1, LinearConv without regularization is equivalent to Conv. Table <ref> presents the change in performance with α. We see a stable accuracy when α=0.5, which verifies our claim in subsection <ref>. This setting allows a better flexibility for the spanned redundant filters, striking a balance between parameter reduction and performance degradation. In terms of the computational requirement at training, we see a peak in the same setting as well. This is due to the increased dimensions of the matrix of linear coefficients 𝐀_l. As this is undesirable, we propose a rank-reduced version of the matrix in the following subsection. 

P[1]>p#1




We want to evaluate whether the proposed LinearConv layer and regularization can provide a performance stability in parameter reduction. To do this, we compare different versions of Conv and LinearConv models. We regularize Conv layers without regenerating any redundancy (-r) to see the importance of secondary filters in LinearConv. We consider Conv models with an equivalent amount of parameters as LinearConv (-e) to understand if the proposed layer is actually required in reducing parameters. To evaluate if the regularization provides any advantage in LinearConv, we consider un-regularized (-L) and regularized (-Lr) versions of LinearConv. Table <ref> shows the performance of these models. The regularized and equiv-parameter versions of Conv models verify that we need redundant features to maintain a stable performance, and simply reducing the number of parameters would not help. Among LinearConv models, the regularized version shows a better stability, verifying our claim in subsection <ref> that it provides a better flexibility in spanning redundant filters, and feature subspace in turn.







 §.§ Classification Results






Here, we compare a variety of CNN architectures which highlight different aspects of network design, replacing their Conv layers with the proposed LinearConv layers. For fair comparison, we evaluate all the configurations in our experiment settings, and present the maximum accuracies achieved in Table <ref>. The LinearConv models show a comparable performance with the respective baselines with a reduced number of parameters. Specifically, the regularized LinearConv (-Lr) models achieve an average of 44% and a minimum of 29% reduction in parameters (except in MobileNetV2) with performance variations ranging from -2.1% to +2.6%. However, in terms of the computational requirement, this version shows a maximum increment of ×13 in VGG11-Lr, which is undesirable. To remedy this, we introduce a rank-reduced, regularized version of LinearConv (-LRr) as described in Eq. <ref>. We set the rank of the matrix 𝐀_l to be 10 for all layers in every configuration by observation. These rank-reduced versions achieve an average of almost 50% and  a minimum of 41%  reduction in parameters with performance variations ranging from -3.1% to +1.5%. The increment in computational requirement for training is contained to a maximum of ×1 since the matrix multiplication which generates the secondary filters is decomposed to be lightweight. Under this setting, vanilla CNNs, and CNNs with skip connections, group convolutions and even depth-wise separable convolutions show a desirable parameter reduction with a comparable performance, when LinearConv is introduced. At inference, LinearConv models have the same computational requirement as the respective Conv models, since the extra cost will be only a one-time cost at initialization, which becomes negligible in the long run.  










It is worth noting only a 29% reduction in parameters in ResNeXt-29-Lr and an interesting 70% increment in MobileNetV2-Lr, both of which use group convolutions <cit.>. This behavior can be explained based on Eq. <ref>. The requirement for parameter reduction is adjusted as the effective input channels c_l becomes c_l/g_l where g_l is the number of groups. In MobileNetV2-Lr, this requirement is violated, resulting an increment in parameters. It further shows a notable increment of × 166 in FLOPs, since the matrix dimensions become very large in this architecture with a large number of filters per layer, which does not affect the cost of depth-wise separable convolutions <cit.> much. However, this behavior can still be contained in the rank-reduced version MobileNetV2-LRr, which shows a 41% reduction in parameters with × 8.5 increment in FLOPs. 

L[1]>p#1


We further compare LinearConv models with some methods in literature for network compression as presented  in Table <ref>. Our proposed method is intended for parameter reduction without having any change in the network architecture—in which Conv layers are replaced—or the hyperparameter settings. With such an implementation, our method is widely applicable to many existing architectures as a convenient plug-and-play module. Based on this design rule, we compare our work against the methods in literature with minimal changes in architecture <cit.>. However, we show that our method can also be complementary in architectures specifically designed for parameter reduction, such as SqueezeNet <cit.>. VGG11-OctConv <cit.> reduces computational requirement, having the same number of parameters, whereas VGG11-Basis <cit.> reduces both the number of parameters and FLOPs. However, VGG11-Basis requires the weights of the respective baseline for training, as it tries to approximate weights of the baseline. Therefore, in practice, the actual number of parameters and the computational requirement for training are the cumulative amounts including those of the baseline. In contrast, LinearConv models have no such requirement and can be trained from scratch with the presented complexity. SqueezeNet-LRr shows further parameter reduction compared to its baseline, verifying that LinearConv can be complementary to other architectures for parameter reduction. We further apply LinearConv in EfficientNet <cit.>, where it behaves as in MobileNetV2 due to depth-wise separable convolutions.





















Fig. <ref> (Left) visualizes the performance graphs of Conv and LinearConv models in the CNN architectures that we selected. LinearConv models achieve a similar performance as the respective baselines with almost a 50% reduction in parameters on average. The increment of computational requirement in LinearConv can be contained with the proposed rank-reduced implementation. Fig. <ref> (Right) shows the correlation matrices corresponding to a few layers in Conv and LinearConv models. This visualizes the effect of regularization in the primary filters of LinearConv. The secondary filters generated as linear combinations show increased correlations as expected. We observe some artifacts in the correlation matrix which corresponds to the primary filters in Layer 1. This is due to a small non-zero regularization loss at the end of training.





§ CONCLUSION



In this work, we exploited the feature redundancy in CNNs to improve their efficiency. Based on the intuition of restricting the inherent redundancy and re-applying a sufficient amount to maintain performance, we proposed a novel layer: LinearConv, which can replace the conventional convolutional layers with a smaller number of parameters. We design LinearConv as a plug-and-play module to allow convenient adoption in existing CNNs with no change in the architecture or the hyperparameter settings. In combination with the proposed correlation-based regularization loss, LinearConv layers flexibly reproduce the redundancy in convolutional layers with almost a 50% reduction in parameters on average, and the same computational requirement at inference. We evaluate its performance in common  architectures and show that it can function in complement with other methods in literature.  The control over feature correlation and redundancy opens-up room for improving the efficiency of CNNs. 




Acknowledgments: The authors thank D. Samaras, C.U.S. Edussooriya, P. Dharmawansa, M. Pathegama and D. Tissera for helpful discussions. The authors acknowledge the computational resources received from the QBITS Lab and the Faculty of Information Technology, University of Moratuwa. K. Kahatapitiya was supported by the Senate Research Committee Grant no. SRC/LT/2016/04.


	
ieee
	




§ APPENDIX A:



C[1]>p#1



	



