In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment. Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding. In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution. Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization. Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively. Compression of deep neural networks (DNNs) has been actively studied in deep learning to develop compact DNN models for memory-efficient and computation-efficient deployment. Han et al. BID0 showed impressive compression results by weight pruning, k-means clustering, and Huffman coding. It is further optimized in BID1 using Hessian-weighted k-means clustering. Recently, it is shown how soft weight sharing or soft quantization can be employed for DNN weight quantization in BID2 BID3 . On the other hand, weight pruning is also extensively studied, e.g., in BID4 BID5 BID6 BID7 BID8 . In this paper, we focus on DNN weight quantization, which can be used together with weight pruning to generate compressed models.Vector quantization reduces the gap to the rate-distortion bound by jointly quantizing multiple symbols. Since conjectured by Gersho in BID9 , lattice quantization has been presumed to be the most efficient entropy coded vector quantization in the high resolution regime asymptotically, as the rate goes to infinity and the distortion diminishes BID10 . Although lattice quantizers are simple and empirically shown to perform well even at finite rates, their efficiency depends on source statistics. Thus, we consider universal quantization that provides near-optimal performance for any source distribution BID11 . Of particular interest is randomized lattice quantization, where uniform random dithering makes the distortion independent of the source, and the gap of its rate from the rate-distortion bound at any distortion level is provably no more than 0.754 bits per sample for any finite dimension BID12 .From the classical lossy compression results, this paper establishes a universal DNN compression framework consisting of universal quantization and universal lossless source coding such as LempelZiv-Welch BID13 BID14 BID15 and the Burrows-Wheeler transform BID16 BID17 . In order to recover any accuracy loss resulting from weight quantization, we furthermore propose a fine-tuning algorithm for vector quantized DNNs. The gain of fine-tuning becomes larger as the vector dimension increases, due to the fact that the number of shared quantized values that are tunable (trainable) in a vector quantized model increases as the vector dimension increases. For unpruned models, we often have a high volume of weights concentrated around zero, and thus case (b) that assigns one bin to include all the weights near zero is expected to outperform case (a), which is aligned with our lattice quantization results in FIG1 . However, it is interesting to observe that randomized lattice quantization provides similarly good performance in both cases, which is the main benefit of randomizing the source by uniform dithering before quantization. FIG1 also shows that vector quantization provides additional gain over scalar quantization particularly when the compression ratio is large.Finally, TAB1 summarizes the compression ratios that we obtain from our universal DNN compression method for pruned ResNet-32 and AlexNet BID20 models. The proposed universal DNN compression scheme with the bzip2 BID17 universal source coding algorithm yields 47.10× and 42.46× compression for ResNet-32 and AlexNet, respectively. Compared with BID0 BID1 BID3 ] which need to optimize and/or calculate source statistics for compression, we achieved a better trade-off between rate (compression ratio) and distortion (loss in accuracy) through the universal compression of DNNs. FORMULA0 ); here, uniform quantization corresponds to lattice quantization with dimension n = 1. Given the vector dimension n, the weights from all layers of the pre-trained ResNet-32 model are vectorized as in FORMULA0 for vector quantization. Then, lattice quantization or randomized lattice quantization follows. In plain lattice quantization, no random dithering is added before quantization, i.e., we set u i = 0 for all i in (2). We fine-tune the quantization codebook as explained in Section 2. We simply use Huffman coding only in this experiment to get the compressed models.The gain of randomized lattice quantization over lattice quantization can be found in FIG1 (a) in particular for n ≥ 2 and large compression ratios. We note that randomized lattice quantizers provide similarly good performance in both cases (a) and (b). Lattice quantization performs well only in case (b), where the quantization bins are optimized for given weight distribution. We emphasize that randomized lattice quantization is applicable for any network models blindly, regardless of their weight distribution and with no optimization, while it is guaranteed to yield a good rate-distortion trade-off close to the optimum within a fixed gap BID11 .
We introduce the universal deep neural network compression scheme, which is applicable universally for compression of any models and can perform near-optimally regardless of their weight distribution. Introduces a pipeline for network compression that is similar to deep compression and uses randomized lattice quantization instead of the classical vector quantization, and uses universal source coding (bzip2) instead of Huffman coding.