Bag of Tricks with Quantized Convolutional Neural Networks for image
  classification
Deep neural networks have been proven effective in a wide range of tasks.
However, their high computational and memory costs make them impractical to
deploy on resource-constrained devices. To address this issue, quantization
schemes have been proposed to reduce the memory footprint and improve inference
speed. While numerous quantization methods have been proposed, they lack
systematic analysis for their effectiveness. To bridge this gap, we collect and
improve existing quantization methods and propose a gold guideline for
post-training quantization. We evaluate the effectiveness of our proposed
method with two popular models, ResNet50 and MobileNetV2, on the ImageNet
dataset. By following our guidelines, no accuracy degradation occurs even after
directly quantizing the model to 8-bits without additional training. A
quantization-aware training based on the guidelines can further improve the
accuracy in lower-bits quantization. Moreover, we have integrated a multi-stage
fine-tuning strategy that works harmoniously with existing pruning techniques
to reduce costs even further. Remarkably, our results reveal that a quantized
MobileNetV2 with 30\% sparsity actually surpasses the performance of the
equivalent full-precision model, underscoring the effectiveness and resilience
of our proposed scheme.
Jie Hu, Mengze Zeng, Enhua Wu
# Bag of tricks with quantized convolutional neural networks for image classification

###### Abstract

Deep neural networks have been proven effective in a wide range of tasks. However, their high computational and memory costs make them impractical to deploy on resource-constrained devices. To address this issue, quantization schemes have been proposed to reduce the memory footprint and improve inference speed. While numerous quantization methods have been proposed, they lack systematic analysis for their effectiveness. To bridge this gap, we collect and improve existing quantization methods and propose a gold guideline for post-training quantization. We evaluate the effectiveness of our proposed method with two popular models, ResNet50 and MobileNetV2, on the ImageNet dataset. By following our guidelines, no accuracy degradation occurs even after directly quantizing the model to 8-bits without additional training. A quantization-aware training based on the guidelines can further improve the accuracy in lower-bits quantization. Moreover, we have integrated a multi-stage fine-tuning strategy that works harmoniously with existing pruning techniques to reduce cost even further. Remarkably, our results reveal that a quantized MobileNetV2 with 30% sparsity actually surpasses the performance of the equivalent full-precision model, underscoring the effectiveness and resilience of our proposed scheme.

Jie Hu\({}^{1,2^{*}}\), Mengze Zeng\({}^{3^{*}}\), Enhua Wu\({}^{1,2,4}\)+\({}^{1}\)State Key Laboratory of Computer Science, ISCAS

\({}^{2}\)University of Chinese Academy of Sciences

\({}^{3}\)ByteDance Inc. \({}^{4}\)University of Macau

hujie@ios.ac.cn, zeitzmz@gmail.com, ehwu@um.edu.mo Model Quantization, Acceleration, Convolutional Neural Networks, Image Classification

Footnote †: 1}\)Corresponding author. This work is supported in part by NSFC Grants (62072449).

## 1 Introduction

Since the introduction of AlexNet [1], there has been an exponential increase in the number of exceptional convolutional neural networks proposed, resulting in promising outcomes for a variety of visual tasks [1, 2, 3, 4, 5]. Despite the remarkable results, deploying CNN models on embedded or mobile devices proves challenging as it poses an immense burden on computation and memory storage. To address this issue, a significant amount of research has been dedicated to reducing associated costs, thereby making CNN models more practical for real-world applications. Broadly speaking, this line of research can be categorized into three distinct areas: efficient structure design, network pruning, and network quantization.

Efficient structural design is a challenge in research, with introduction of the separated convolution [6] proposed as an effective technique. This method factorizes the standard convolution into a depthwise and pointwise convolution, reducing computation. Successful examples of its use in efficient networks include MobileNets [6, 7, 8] and ShuffleNets [9, 10]. These networks are widely used on resource-constrained devices and have shown promise in practical applications. Besides that, various pruning strategies [11] have also been proposed to reduce both the computational and storage burdens. However, these methods often incur accuracy degradation, making them less attractive for practical applications. As such, the focus has shifted towards developing efficient structural design methods that can provide high accuracy without compromising on computational efficiency.

Quantization is an incredibly efficient method for deploying models. Its effectiveness has been demonstrated in recent work [12, 13, 14, 15, 16] and has made it increasingly popular in the industry due to its hardware-friendly properties, which can significantly reduce computational and memory costs. However, despite its advantages, there exists a significant accuracy gap between a full-precision model and a quantized counterpart. This gap is especially pronounced in low-bitwidth quantization situations (e.g. 4-bits). Nevertheless, researchers are actively working on closing this gap and making quantization even more effective.

This paper presents a systematic exploration of the effects of each quantization factor on convolutional neural networks, forming a gold guideline for post-training quantization. Additionally, our proposed multi-stage fine-tuning strategy enables quantization to work in conjunction with existing pruning strategies. Exhaustive experiments with ResNet-50 and MobileNetV2, quantized using different bitwidths on ImageNet, showcase the effectiveness of our proposed method.