Modern deep neural networks (DNNs) require high memory consumption and large computational loads.   In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including the line of works on factorization methods. Factorization methods approximate the weight matrix of a DNN layer with multiplication of two or multiple low-rank matrices. However, it is hard to measure the ranks of DNN layers during the training process. Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step. The former approach usually induces a high accuracy loss while the latter prevents DNN factorization from efficiently reaching a high compression rate. In this work, we propose SVD training, which first applies SVD to decompose DNN's layers and then performs training on the full-rank decomposed weights. To improve the training quality and convergence, we add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding. Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer. Singular value pruning is applied at the end to reach a low-rank model. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods. The booming development in deep learning models and applications has enabled beyond human performance in tasks like large-scale image classification (Krizhevsky et al., 2012; He et al., 2016; Hu et al., 2018; Huang et al., 2017) , object detection (Redmon et al., 2016; Liu et al., 2016; He et al., 2017) , and semantic segmentation (Long et al., 2015; Chen et al., 2017) . Such high performance, however, comes with a high price of large memory consumption and computation load. For example, a ResNet-50 model needs approximately 4G floating-point operations (FLOPs) to classify a color image of 224 × 224 pixels. The computation load can easily expand to tens or even hundreds of GFLOPs for detection or segmentation models using state-of-the-art (SOTA) DNNs as backbones (Canziani et al., 2016 ). This is a major challenge that prevents the deployment of modern DNN models on resource-constrained platforms, such as phones, smart sensors, and drones. Model compression techniques for DNN models have been extensively studied. Some successful methods include element-wise pruning (Han et al., 2015; Liu et al., 2015; Zhang et al., 2018) , structural pruning (Wen et al., 2016; Luo et al., 2017; Li et al., 2019) , quantization (Liu et al., 2018; Wang et al., 2019) , and factorization (Jaderberg et al., 2014; Zhang et al., 2015; Yang et al., 2015; Xu et al., 2018) . Among these methods, quantization and element-wise pruning can effectively reduce model's memory consumption, but require specific hardware to realize efficient computation. Structural pruning reduces the computation load by removing redundant filters or channels. However, the complicated structures adopted in some modern DNNs (i.e., ResNet or DenseNet) enforce strict constraints on the input/output dimension of certain layers. This requires additional filter grouping during the pruning and filter rearranging after the pruning to make the pruned structure valid (Wen et al., 2017a; Ding et al., 2019) . Factorization method approximates the weight matrix of a layer with a multiplication of two or more low-rank matrices. It by nature keeps the input/output dimension of a layer unchanged, and therefore the resulted decomposed network can be supported by any common DNN computation architectures, without additional grouping and post-processing. The previous investigation show that it is feasible to approximate the weight matrices of a pretrained DNN model with the multiplication of low-rank matrices, but it may greatly degrade the performance (Jaderberg et al., 2014; Zhang et al., 2015; . Some other methods attempt to manipulate the "directions" of filters to implicitly reduce the rank of weight matrices (Wen et al., 2017b; Li et al., 2019) . However, the difficulties in training and the implicitness of rank representation prevent these methods from reaching a high compression rate. Nuclear norm regularizer (Xu et al., 2018) has been used to directly reduce the rank of weight matrices. Optimizing the nuclear norm requires back propagation through singular value decomposition (SVD). Applying such a numerical process on every training step is inefficient and unstable. Our work aims to explicitly achieve a low-rank DNN network during the training without applying SVD on every step. In particular, we propose SVD training by training the weight matrix of each layer in the form of its full-rank SVD. The weight matrix is decomposed into the matrices of left-singular vectors, singular values and right-singular vectors, and the training is done on the decomposed variables. Furthermore, two techniques are proposed to induce low-rank while maintaining high performance during the SVD training: (1) Singular vector orthogonality regularization which keeps the singular vector matrices close to unitary through the training. It mitigates gradient vanishing/exploding during the training, and provide a valid form of SVD to guarantee the effective rank reduction. (2) Singular value sparsification which applies sparsity-inducing regularizers on the singular values during the training to induce low-rank. The low-rank model is finally achieved through singular value pruning. We evaluate the individual contribution of each technique as well as the overall performance when putting them together via ablation studies. Results show that the proposed method constantly beats SOTA factorization and structural pruning methods on various tasks and model structures. In this work, we propose the SVD training framework, which incorporates the full-rank decomposed training and singular value pruning to reach low-rank DNNs with minor accuracy loss. We apply SVD to decompose each DNN layer before the training and directly train with the decomposed singular vectors and singular values, so we can keep an explicit measure of layers' ranks without performing the SVD on each step. Orthogonality regularizers are applied to the singular vectors during the training to keep the decomposed layers in a valid SVD form. And sparsity-inducing regularizers are applied to the singular values to explicitly induce low-rank layers. Thorough experiments are done to analyse each proposed technique. We demonstrate that the orthogonality regularization on singular vector matrices is crucial to the performance of the decomposed training process. For decomposition methods, we find that the spatial-wise method performs better than channel-wise in shallower networks while the performances are similar for deeper models. For the sparsity-inducing regularizer, we show that higher compression rate can be achieved by Hoyer regularizer comparing to that of the L 1 regularizer under low accuracy loss. We further apply the proposed method to various depth of ResNet models on both CIFAR-10 and ImageNet dataset, where we find the accuracy-#FLOPs tradeoff achieved by the proposed method constantly stays above the Pareto frontier of previous methods, including both factorization and structural pruning methods. These results prove that this work provides an effective way for learning low-rank deep neural networks. 1943-1955, 2015. A EXPERIMENT SETUPS Our experiments are done on the CIFAR-10 dataset (Krizhevsky & Hinton, 2009 ) and the ImageNet ILSVRC-2012 dataset (Russakovsky et al., 2015) . We access both datasets via the API provided in the "TorchVision" Python package. As recommended in the PyTorch tutorial, we normalize the data and augment the data with random crop and random horizontal flip before the training. We use batch size 100 to train CIFAR-10 model and use 256 for the ImageNet model. For all the models on CIFAR-10, both the full-rank SVD training and the low-rank finetuning are trained for 164 epochs. The learning rate is set to 0.001 initially and decayed by 0.1 at epoch 81 and 122. For models on ImageNet, the full-rank SVD training is trained for 90 epochs, with initial learning rate 0.1 and learning rate decayed by 0.1 every 30 epochs. The finetuning is done for 60 epochs, starting at learning rate 0.01 and decay by 0.1 at epoch 30. We use pretrained full-rank decomposed model (trained with the orthogonality regularizer but without sparsity-inducing regularizer) to initialize the SVD training. SGD optimizer with momentum 0.9 is used for optimizing all the models, with weight decay 5e-4 for CIFAR-10 models and 1e-4 for ImageNet models. The accuracy reported in the experiment is the best testing accuracy achieved during the finetuning process. During the SVD training, the decay parameter of the orthogonality regularizer λ o is set to 1.0 for both channel-wise and spatial-wise decomposition on CIFAR-10. On ImageNet, λ o is set to 10.0 for channel-wise decomposition and 5.0 for spatial-wise decomposition. The decay parameter λ s for the sparsity-inducing regularizer and the energy threshold used for singular value pruning are altered through different set of experiments to fully explore the accuracy-#FLOPs tradeoff. In most cases, the energy threshold is selected through a line search, where we find the highest percentage of energy that can be pruned without leading to a sudden accuracy drop. The λ s and the energy thresholds used in each set of the experiments are reported alongside the experiment results in Appendix B.
Efficiently inducing low-rank deep neural networks via SVD training with sparse singular values and orthogonal singular vectors. This paper introduces an approach to network compression by encouraging the weight matrix in each layer to have a low rank and explicitly factorizing the weight matrices into an SVD-like factorization for treatment as new parameters. Proposal to parametrize each layer of a deep neural network, before training, with a low-rank matrix decomposition, accordingly replace convolutions with two consecutive convolutions, and then train the decomposed method.