In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values. But it cannot provide useful gradients and therefore requires complex optimization techniques. The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant and causes the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant. Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element-wise and structural pruning. The use of deep neural network (DNN) models has been expanded from handwritten digit recognition (LeCun et al., 1998) to real-world applications, such as large-scale image classification (Simonyan & Zisserman, 2014) , self driving (Makantasis et al., 2015) and complex control problems (Mnih et al., 2013) . However, a modern DNN model like AlexNet (Krizhevsky et al., 2012) or ResNet (He et al., 2016) often introduces a large number of parameters and computation load, which makes the deployment and real-time processing on embedded and edge devices extremely difficult (Han et al., 2015b; a; . Thus, model compression techniques, especially pruning methods that increase the sparsity of weight matrices, have been extensively studied to reduce the memory consumption and computation cost of DNNs (Han et al., 2015b; a; Guo et al., 2016; Louizos et al., 2017b; Liu et al., 2015) . Most of the previous works utilize some form of sparsity-inducing regularizer in searching for sparse neural networks. The 1 regularizer, originally proposed by Tibshirani (1996) , can be easily optimized through gradient descent for its convex and almost everywhere differentiable property. Therefore it is widely used in DNN pruning: Liu et al. (2015) directly apply 1 regularization to all the weights of a DNN to achieve element-wise sparsity; present structural sparsity via group lasso, which applies an 1 regularization over the 2 norms of different groups of parameters. However, it has been noted that the value of the 1 regularizer is proportional to the scaling of parameters (i.e. ||αW || 1 = |α|·||W || 1 ), so it "scales down" all the elements in the weight matrices with the same speed. This is not efficient in finding sparsity and may sacrifice the flexibility of the trained model. On the other hand, the 0 regularizer directly reflects the real sparsity of weights and is scale invariant (i.e. ||αW || 0 = ||W || 0 , ∀α = 0), yet the 0 norm cannot provide useful gradients. Han et al. (2015b) enforce an element-wise 0 constraint by iterative pruning a fixed percentage of smallest weight elements, which is a heuristic method and therefore can hardly achieve optimal compression rate. Some recent works mitigate the lack of gradient information by integrating 0 regularization with stochastic approximation (Louizos et al., 2017b) or more complex optimization methods (e.g. ADMM) . These additional measures brought overheads to the optimization process, making the use of these methods on larger networks difficult. To achieve even sparser neural networks, we argue to move beyond 0 and 1 regularizers and seek for a sparsity-inducing regularizer that is both almost everywhere differentiable (like 1 ) and scale-invariant (like 0 ). Beyond the 1 regularizer, plenty of non-convex sparsity measurements have been used in the field of feature selection and compressed sensing (Hurley & Rickard, 2009; . Some popular regularizers like SCAD (Fan & Li, 2001) , MDP (Zhang et al., 2010) and Trimmed 1 (Yun et al., 2019 ) use a piece-wise formulation to mitigate the proportional scaling problem of 1 . The piece-wise formulation protects larger elements by having zero penalty to elements greater than a predefined threshold. However, it is extremely costly to manually seek for the optimal trimming threshold, so it is hard to obtain optimal result in DNN pruning by using these regularizers. The transformed 1 regularizer formulated as (a+1)|wi| a+|wi| manages to smoothly interpolate between 1 and 0 by tuning the hyperparameter a (Ma et al., 2019) . However, such an approximation is close to 0 only when a approaches infinity, so the practical formulation of the transformed 1 (i.e. a = 1) is still not scale-invariant. Particularly, we are interested in the Hoyer regularizer (Hoyer, 2004) , which estimates the sparsity of a vector with the ratio between its 1 and 2 norms. Comparing to other sparsity-inducing regularizers, Hoyer regularizer achieves superior performance in the fields of non-negative matrix factorization (Hoyer, 2004) , sparse reconstruction (Esser et al., 2013; Tran et al., 2018) and blend deconvolution (Krishnan et al., 2011; Repetti et al., 2015) . We note that Hoyer regularizer is both almost everywhere differentiable and scale invariant, satisfying the desired property of a sparsityinducing regularizer. We therefore propose DeepHoyer, which is the first Hoyer-inspired regularizers for DNN sparsification. Specifically, the contributions of this work include: • Hoyer-Square (HS) regularizer for element-wise sparsity: We enhance the original Hoyer regularizer to the HS regularizer and achieve element-wise sparsity by applying it in the training of DNNs. The HS regularizer is both almost everywhere differentiable and scale invariant. It has the same range and minima structure as the 0 norm. Thus, the HS regularizer presents the ability of turning small weights to zero while protecting and maintaining those weights that are larger than an induced, gradually adaptive threshold; • Group-HS regularizer for structural sparsity, which is extended from the HS regularizer; • Generating sparser DNN models: Our experiments show that the proposed regularizers beat state-of-the-arts in both element-wise and structural weight pruning of modern DNNs. In this work, we propose DeepHoyer, a set of sparsity-inducing regularizers that are both scaleinvariant and almost everywhere differentiable. We show that the proposed regularizers have similar range and minima structure as the 0 norm, so it can effectively measure and regularize the sparsity of the weight matrices of DNN models. Meanwhile, the differentiable property enables the proposed regularizers to be simply optimized with standard gradient-based methods, in the same way as the 1 regularizer is. In the element-wise pruning experiment, the proposed Hoyer-Square regularizer achieves a 38% sparsity increase on the LeNet-300-100 model and a 63% sparsity increase on the LeNet-5 model without accuracy loss comparing to the state-of-the-art. A 21.3× model compression rate is achieved on AlexNet, which also surpass all previous methods. In the structural pruning experiment, the proposed Group-HS regularizer further reduces the computation load by 24.4% from the state-of-the-art on LeNet-300-100 model. It also achieves a 8.8% increase from the 1 based method and a 110.6% increase from the 0 based method of the computation reduction rate on the LeNet-5 model. For CIFAR-10 and ImageNet dataset, the accuracy-FLOPs tradeoff achieved by training ResNet models with various strengths of the Group-HS regularizer constantly stays above the Pareto frontier of previous methods. These results prove that the DeepHoyer regularizers are effective in achieving both element-wise and structural sparsity in deep neural networks, and can produce even sparser DNN models than previous works.
We propose almost everywhere differentiable and scale invariant regularizers for DNN pruning, which can lead to supremum sparsity through standard SGD training. The paper proposes a scale-invariant regularizer (DeepHoyer) inspired by the Hoyer measure to enforce sparsity in neural networks. 