To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency.   To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision.   Here, for the first time, we demonstrate ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3, densenet-161, and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models.
 We also demonstrate ResNet-18, ResNet-34, and ResNet-50 4-bit models that match the accuracy of the full-precision baseline networks -- the highest scores to date. Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary.

 We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise. The number of iterations required by stochastic gradient descent to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates.   By drawing inspiration from this observation, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat noise introduced by quantizing weights and activations during training, by using larger batches along with matched learning rate annealing.   Sensitivity analysis indicates that these techniques, coupled with proper activation function range calibration, offer a promising heuristic to discover low-precision networks, if they exist, close to fp32 precision baseline networks.
 Weight discretization increases gradient noise for 8-, 4-, and 2-bit networks 7 . We define the increase in gradient noise due to weight discretization as the angular difference between the step taken by the learning algorithm, δw, on the float-point copy at iteration t − 1, w t−1 , and the actual step taken due to quantizing the weights, i.e. Q b,l (w t ) − Q b,l (w t−1 ). We measure this angle using cosine similarity (normalized dot-product) between the instantaneous δw and an exponential moving average of the actual step directions with smoothing factor 0.9 (Figure 1 ). Cosine similarity of 1.0 corresponds to an fp32 network and the absence of discretization-induced gradient noise. As bit precisions decrease, similarity decreases, signaling higher gradient noise.These results directly show discretization-induced gradient noise appreciably influences the finetuning and training trajectories of quantized networks. The increased noise (decreased similarity) of the 4-bit case compared to the 8-bit case possibly accounts for the difference in fine-tuning times required. Even the 8-bit case is significantly below unity, possibly explaining why training from scratch has not lead to the highest performance BID11 . The ResNet-18 4-bit solution after fine-tuning for 110 epochs was located relatively close to the initial high-precision solution used to initialize the network, indicating that training from scratch is unnecessary. Plotted is the mean, over all neurons in a ResNet-18 network, of the cosine similarity between the weights at the beginning of training from scratch, and the weights at epoch 110 (left bar). The minimum and maximum similarity measure is 0 and 1, respectively. The similarity between the random initial weights and the final solution is near 0 in this control experiment, indicating that the weights have moved far from the initial condition when training from scratch. The right bar shows the same measure between initial weights taken from the model zoo and the 4-bit solution after 100 epochs of FAQ training. The cosine similarity is close to 1, indicating that the 4-bit solution is close to the initial fp32 solution used for initialization. We show here that low-precision quantization followed by fine-tuning, when properly compensating for noise, is sufficient to achieve state of the art performance for networks employing 4-and 8-bit weights and activations. Compared to previous work, our approach offers a major advantage in the 8-bit space, by requiring only a single epoch of post quantization training to consistently exceed high-precision network scores, and a major advantage in the 4-bit space by matching high-precision baseline scores with a simpler approach, exceeding published results on ResNet-18, 34 and 50. We find support for the idea that overcoming noise is the main challenge in successful fine-tuning, given sufficient capacity in a network model: longer training times, exponential learning rate decay, very low final learning rate, and larger batch sizes all seem to contribute to improving the results of finetuning. SGD is faced with two sources of noise, one inherent to stochastic sampling, and the other due to quantization noise; these techniques may be reducing only one of the sources, or both, and we have not shown that FAQ is directly reducing quantization noise. Further experiments are warranted.We believe that the success of fine-tuning and the wide availability of pretrained models marks a major change in how low-precision networks will be trained. We conjecture that within every region containing a local minimum for a high-precision network, there exists a subregion(s) which also contains solutions to the lower precision 4-bit nets, provided that the network has sufficient capacity. The experiments reported herein provide support for this conjecture; if true, FAQ should generalize to any model. Fine-tuning for quantization has been previously studied. In BID27 , increasingly larger subsets of neurons from a pretrained network are replaced with low-precision neurons and finetuned, in stages. The accuracy exceeds the baseline for a range of networks quantized with 5-bit weights and 32-bit activations. Our results here with both fixed-precision weights and activations at either 8 or 4 bits suggest that incremental training may have been unnecessary. In BID0 , fine-tuning is employed along with a non-linear quantization scheme during training (see UNIQ in Table 1 ). We have shown that low-precision quantization followed by proper fine-tuning, is sufficient to achieve even greater accuracy when quantizing both weights and activations at 4 bits. Finally, using a combination of quantizing weights before activations, progressively lower precisions, fine-tuning, and a new loss function, BID28 are the first to show that a 4-bit ResNet network can match the top-1 accuracy of a baseline full-precision network. Our results show that a simpler method can achieve this for ResNet-18, 34, and 50.Future research includes combining FAQ with other approaches, new training algorithms designed specifically to fight the ill-effects of noise BID0 introduced by weight quantizaiton, and extending to quantize 2-bit networks. Training in the 2-bit case will be more challenging given the additional quantization noise FIG0 , and possible capacity limits with 2-bit quantization.FAQ is a principled approach to quantization. Ultimately, the goal of quantization is to match or exceed the validation score of a corresponding full-precision network. This work demonstrates that 8-bit and 4-bit quantized networks performing at the level of their high-precision counterparts can be obtained with a straightforward approach, a critical step towards harnessing the energy-efficiency of low-precision hardware.
Finetuning after quantization matches or exceeds full-precision state-of-the-art networks at both 8- and 4-bit quantization. This paper proposes to improve the performance of low-precision models by doing quantization on pre-trained models, using large batches size, and using proper learning rate annealing with longer training time. A method for low bit quantization to enable inference on efficient hardware that achieves full accuracy on ResNet50 with 4-bit weights and activations, based on observations that fine-tuning at low precision introduces noise in the gradient.