Network quantization is one of the most hardware friendly techniques to enable the deployment of convolutional neural networks (CNNs) on low-power mobile devices. Recent network quantization techniques quantize each weight kernel in a convolutional layer independently for higher inference accuracy, since the weight kernels in a layer exhibit different variances and hence have different amounts of redundancy. The quantization bitwidth or bit number (QBN) directly decides the inference accuracy, latency, energy and hardware overhead. To effectively reduce the redundancy and accelerate CNN inferences, various weight kernels should be quantized with different QBNs. However, prior works use only one QBN to quantize each convolutional layer or the entire CNN, because the design space of searching a QBN for each weight kernel is too large. The hand-crafted heuristic of the kernel-wise QBN search is so sophisticated that domain experts can obtain only sub-optimal results. It is difficult for even deep reinforcement learning (DRL) DDPG-based agents to find a kernel-wise QBN configuration that can achieve reasonable inference accuracy. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, to automatically search a QBN for each weight kernel, and choose another QBN for each activation layer. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy. Although convolutional neural networks (CNNs) have been the dominant approach Sandler et al. (2018) to solving a wide variety of problems such as computer vision and recommendation systems, it is challenging to deploy CNNs to mobile devices having only limited hardware resources and tight power budgets, due to their huge essential computing overhead, e.g., an inference of MobileNetV2 Sandler et al. (2018) involves 6.9M weights and 585M floating point operations. Several approaches such as pruning He et al. (2018) and low-rank approximation Denton et al. (2014) are proposed to reduce the inference computing overhead of CNNs. Network quantization ; becomes one of the most hardware friendly CNN acceleration techniques by approximating real-valued weights and activations to QBN -bit fixed-point representations, and performing inferences using cheaper fixed-point multiple-accumulation (MAC) operations, where QBN is the quantization bit number. Instead of using one QBN for the whole CNN, the layer-wise network quantization ; Elthakeb et al. (2018) assigns a QBN to the weights of each convolutional layer, and searches another QBN for the activations of the same layer to decrease the inference computing overhead. But the inference cost of the layer-wise quantized CNNs is still prohibitive for low-power mobile devices powered by batteries. Recent works Zeng et al. (2019) ; Choukroun et al. (2019b) ; Zhang et al. (2018) ; Li et al. (2019) ; Krishnamoorthi (2018) ; Sasaki et al. (2019) find that various weight kernels of a convolutional layer exhibit different variances shown in Figure 1 and hence have different amounts of redundancy. Therefore, they quantize each weight kernel independently for higher accuracy by calculating a QBN -element scaling factor vector for each kernel, rather than globally quantize all the kernels of a layer as a whole. To reduce different amounts of redundancy among different weight kernels, these kernel-wise network quantization techniques should have searched a QBN for each kernel of each layer in a CNN. However, the search space of choosing a QBN for each weight kernel is too large, so prior kernel-wise network quantization Zeng et al. (2019) ; Choukroun et al. (2019b) ; Zhang et al. (2018) ; Li et al. (2019) ; Krishnamoorthi (2018) ; Sasaki et al. (2019) still uses the same QBN for the entire CNN. As Figure 2 shows, compared to the layer-wise quantized model, on the same FPGA accelerator Umuroglu et al. (2019a) , the kernel-wise quantized model (assigning a QBN to each weight kernel and choosing a QBN for each activation layer) improves the inference accuracy by ∼ 2% with the same computing overhead (inference latency). How to decide a QBN for each weight kernel is the most important task of the kernel-wise network quantization, since the QBNs have a large impact on the inference accuracy, latency and hardware overhead. Determining a QBN for each weight kernel via hand-crafted heuristics is so sophisticated that even machine learning experts can obtain only sub-optimal results. Recent works ; Elthakeb et al. (2018) automatically select a QBN for each layer of a CNN through a deep reinforcement learning (DRL) agent without human intervention. However, it is still difficult for low-power mobile devices such as drones and smart glasses to adopt the layer-wise quantized CNN models. These mobile devices are very sensitive to the bit-width of fixed-point MAC operations and memory access during inferences due to their limited battery lifetime and hardware resources. Kernel-wise network quantization assigning a QBN to each weight kernel and searching a QBN for each activation layer of a CNN becomes a must to enable the efficient deployment of deep CNNs on mobile devices by reducing the inference computing overhead. Although it is straightforward to perform kernel-wise quantization via DRL, it takes ultra-long time for a DRL agent to find a proper QBN for each weight kernel of a CNN. As CNN architectures are becoming deeper, it is infeasible to employ rule-based domain expertise or conventional DRL-based techniques to explore the exponentially enlarging search space of kernel-wise network quantization. In this paper, we propose a hierarchical-DRL-based agent, AutoQ, to automatically and rapidly search a QBN for each weight kernel and choose a QBN for each activation layer of a CNN for accurate kernel-wise network quantization. AutoQ comprises a high-level controller (HLC) and a low-level controller (LLC). The HLC chooses a QBN for each activation layer and generates a goal, the average QBN for all weight kernels of a convolutional layer, for each layer. Based on the goal, the LLC produces an action, QBN, to quantize each weight kernel of the layer. The HLC and LLC simultaneously learn by trials and errors, i.e., penalizing inference accuracy loss while rewarding a smaller QBN. We also build a state space, a goal and an action space, an intrinsic reward and an extrinsic reward for AutoQ. Instead of proxy signals including FLOPs, number of memory access and model sizes, we design the extrinsic reward to take the inference latency, energy consumption and hardware cost into consideration. In this paper, we propose a hierarchical-DRL-based kernel-wise network quantization technique, AutoQ, consisting of a HLC and a LLC. The HLC automatically searches an average weight QBN and an average activation QBN for each convolutional layer. Based on the average weight QBN, the LLC generates a QBN for each weight kernel in each layer. We also create a state space, a goal and action space, an intrinsic reward and an extrinsic reward to support AutoQ. Particularly, our shaped intrinsic reward enables the LLC to learn efficiently from the environment by considering both the HLC goal completion and the environment extrinsic reward. Moreover, the extrinsic reward of AutoQ can balance the inference accuracy, latency, energy consumption and FPGA area. Compared to the models quantized by the state-of-the-art DRL-based schemes, on average, the same models quantized by AutoQ reduce the inference latency by 54.06%, and decrease the inference energy consumption by 50.69%, while achieving the same inference accuracy.
Accurate, Fast and Automated Kernel-Wise Neural Network Quantization with Mixed Precision using Hierarchical Deep Reinforcement Learning A method for quantizing neural network weights and activations that uses deep reinforcement learning to select bitwidth for individual kernels in a layer and that achieves better performance, or latency, than prior approaches. This paper proposes to automatically search quantization schemes for each kernel in the neural network, using hierarchial RL to guide the search. 