We analyze the trade-off between quantization noise and clipping distortion in low precision networks. We identify the statistics of various tensors, and derive exact expressions for the mean-square-error degradation due to clipping. By optimizing these expressions, we show marked improvements over standard quantization schemes that normally avoid clipping. For example, just by choosing the accurate clipping values, more than 40\% accuracy improvement is obtained for the quantization of VGG-16 to 4-bits of precision. Our results have many applications for the quantization of neural networks at both training and inference time. 
 A significant drawback of deep learning models is their computational costs. Low precision is one of the key techniques being actively studied recently to conquer the problem. With hardware support, low precision training and inference can compute more operations per second, reduce memory bandwidth and power consumption, and allow bigger network to fit into a device.In general, a low-precision scheme involves a floating-point to integer conversion, which introduces quantization noise into the network. This quantization noise is strongly linked to the dynamic range, defined as the range between the largest and smallest values that need to quantized. For a given N -bit integer representation, a smaller dynamic range leads to a smaller spacing between the 2 N quantization levels, enabling improved resolution and smaller quantization noise. To reduce this quantization noise, the dynamic range can be limited by clipping the values in the tensor. This clipping process introduces an additional noise because of the loss of information that otherwise would be carried by the clipped portion of the tensor. Hence, a trade-off between clipping and quantization effects exist. To find the best clipping value we need to minimize the information loss.In this paper, we study the effect of clipping with the aim of improving overall quantization noise. To this end, we first study the distribution of values within these tensors. In all our measurements, the statistical distributions of weights and activations are observed to follow a bell-curve. This indicates that large values occur very rarely compared to small values, and suggests that the loss of information due to the clipping process might be compensated by improving the resolution of the more common smaller values.To optimize this process further, it is essential to understand the underlying distribution of tensor elements before applying the clipping. By running a few statistical tests, we were able to see on a variety of convolution models that activation tensors follow either a Gaussian or Laplacian distributions with a high degree of certainty (p-value < 0.01). This modeling of activation tensors enables a clear formulation of the quantization process and constitutes the first step for its optimization.We turn to consider the objective we aim to optimize. It is well known that when batch norm is applied after a convolution layer, the output is invariant to the norm of the output on the proceeding layer BID4 ] i.e., BN (C · W · x) = BN (W · x) for any given constant C. This quantity is often described geometrically as the norm of the activation tensor, and in the presence of this invariance, the only measure that needs to be preserved upon quantization is the directionality of the tensor. Therefore, quantization preserves tensor information if the angle between the highprecision tensor and its quantized version is small. Recently, BID0 has shown that this angle depends only on the quantization error power (L2 -norm) and the power of original tensor. Therefore, minimizing the power of the quantization error constitutes a plausible goal for the optimization of the quantized network in terms of accuracy.In Section 4, we provide a rigorous formulation to optimize the quantization effect of activation tensors using clipping by analyzing both the Gaussian and the Laplace priors. This formulation is henceforth refered to as Analytical Clipping for Integer Quantization (ACIQ).These analytical results have many applications for the quantization of neural networks at both training and inference time. For example , a straightforward quantization of the weights and activations to 8-bit fixed point representation has been shown to have a negligible effect on the model accuracy. Yet, in the majority of the applications, further reduction of precision quickly degrades performance, calling for an optimal clipping scheme to minimize information-loss during quantization. On a more general level, exploiting the statistics of the activation tensors to minimize their quantization toll is orthogonal to other techniques for network quantization. It can work in synergy with other schemes to achieve more than could have been achieved by each individually. Finally, it is easy to implement and requires only the adjustment of clipping value according to an analytical formula.We further demonstrate the applicability and usability of our analytic terms on the following challenging problem. Given a pre-trained network , we would like to quantize the weights to 8-bit of precision and most activations to 4-bits of precision without any further processing (e.g., re-training). This specific setting is of a particular interest due to quantization of activations to 4-bits, which alleviates a major bottleneck in terms of memory bandwidth. Prior attempts using standard techniques BID8 show severe degradation on accuracy. While several recent works were able to overcome this issue by additional re-training BID12 , this is not feasible in many practical settings, e.g., we often do not have the dataset on which the network is working on.We compare ACIQ against two methods: (i) the traditional method that avoids clipping (also known by gemmlowp BID6 ), where values are uniformly quantized between the largest and smallest tensor values; (ii) the iterative method suggested by NVIDIA to search for a good clipping threshold based on the Kullback-Leibler Divergence (KLD) measure BID13 . Results are summarized in TAB1 . While both ACIQ and gemmlowp are fast non-iterative methods, ACIQ significantly outperforms in terms of validation accuracy. On the hand, KLD is an exhaustive timeconsuming procedure, which iteratively evaluates the KLD measure on a large candidate set of clipping values, and then returns the clipping value for which best evaluation is attained. In our simulations ACIQ and gemmlowp require a single pass over tensor values, while KLD requires 4000 passes. Nonetheless, excluding ResNet-101, ACIQ outperforms KLD in terms of validation accuracy.The methods introduced in this work may be additionally useful to current and future applications, such as the attempts to fully train in a low precision setting BID0 We introduce ACIQ -an optimized clipping framework for improved quantization of neural networks. Optimized clipping is shown to have a drastic impact on quantization in a variety of models. The underlying reason lies in the statistical dispersion of activations, where large values occur very rarely. We show the bell-curve statistics of activations are best fit as either Laplace or Gaussian distributions, and formulate the clipping process as an optimization problem. The solution to this optimization problem constitutes a polynomial-exponential equation that can be calculated numerically for a variety of statistical parameters, and stored in a lookup table for fast retrieval. This scheme is very simple and easy to implement either in software or in hardware.While results are very encouraging, this work is only the first step on the ladder for successful deployment of clipping in neural networks. First, our main focus in this work is quantization of activations, while similar evaluation still needs to be done for weights. On a more general level, our framework is not restricted to the inference settings and can be extended to training. For example, our preliminary results show that quantization of gradients might benefit from the clipping of small values (i.e., sparsification). Establishing the correct threshold for gradients is yet another important direction for future work. While much work still needs to be done with regards to optimized clipping, we believe our work clearly demonstrates the major importance of this concept for the quantization of neural networks. A PIECE-WISE LINEAR APPROXIMATIONHere we provide a more accurate analysis related to the qunatization noise (i.e., the second term in Equation 3), measured as the expected mean-square-error when the range [−α, α] is quantized uniformly to 2 M discrete levels. To that end, we approximate the density function f by a construction of a piece-wise linear function g such that f (q i ) = g(q i ) for each i ∈ [0, 2 M − 1]. Since we consider only smooth probability density functions (e.g., Gaussian or Laplace), the resulting approximation error is small for sufficient resolution i.e., small quantization step size ∆. In figure 1 we provide an illustration for this construction.We turn to calculate the linear equation for each line segment of the piece-wise linear function g, falling in the range [−α + i · ∆, −α + (i + 1) · ∆]. To that end, we consider the slope (derivative) and the value of the density function at the midpoint q i . With these two values we can define for each segment i ∈ [0, 2 M − 1] the corresponding form of linear approximation: DISPLAYFORM0 We now turn to calculate the second term in Equation 3. By equation 14, and since q i is defined to be the midpoint between the integration limits, the following holds true
We analyze the trade-off between quantization noise and clipping distortion in low precision networks, and show marked improvements over standard quantization schemes that normally avoid clipping Derives a formula for finding the minimum and maximum clipping values for uniform quantization which minimize the square error resulting from quantization, for either a Laplace or Gaussian distribution over pre-quantized value.