A White Paper on Neural Network Quantization
A White Paper on Neural Network Quantization

Date Published: 2021-06-15

Authors:
Markus  Nagel, markusn@qti.qualcomm.com
A I Qualcomm
  Research, Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc
Marios  Fournarakis
Rana Ali Amjad, ramjad@qti.qualcomm.com
Yelysei  Bondarenko
Mart  Van Baalen
Tijmen  Blankevoort, tijmen@qti.qualcomm.com
  Requantization
  # # !,$ # !,% # #,! # #,# # #,$ # #

Abstract

While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.

1 Introduction

With the rise in popularity of deep learning as a general-purpose tool to inject intelligence into electronic devices, the necessity for small, low-latency and energy efficient neural networks solutions has increased. Today neural networks can be found in many electronic devices and services, from smartphones, smart glasses and home appliances, to drones, robots and self-driving cars. These devices are typically subject to strict time restrictions on the execution of neural networks or stringent power requirements for long-duration performance.

One of the most impactful ways to decrease the computational time and energy consumption of neural networks is quantization. In neural network quantization, the weights and activation tensors are stored in lower bit precision than the 16 or 32-bit precision they are usually trained in. When moving from 32 to 8 bits, the memory overhead of storing tensors decreases by a factor of 4 while the computational cost for matrix multiplication reduces quadratically by a factor of 16. Neural networks have been shown to be robust to quantization, meaning they can be quantized to lower bit-widths with a relatively small impact on the network's accuracy. Besides, neural network quantization can often be applied along with other common methods for neural network optimization, such as neural architecture search, compression and pruning. It is an essential step in the model efficiency pipeline for any practical use-case of deep learning. However, neural network quantization is not free. Low bit-width quantization introduces noise to the network that can lead to a drop in accuracy. While some networks are robust to this noise, other networks require extra work to exploit the benefits of quantization.

In this white paper, we introduce the state-of-the-art in neural network quantization. We start with an introduction to quantization and discuss hardware and practical considerations. We then consider two different regimes of quantizing neural networks: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ methods, discussed in section 3, take a trained network and quantize it with little or no data, requires minimal hyperparameter tuning and no end-to-end training. This makes them a push-button approach to quantizing neural networks with low engineering effort and computational cost. In contrast, QAT, discussed in section 4, relies on retraining the neural networks with simulated quantization in the training pipeline. While this requires more effort in training and potentially hyperparameter tuning, it generally further closes the gap to the full-precision accuracy compared to PTQ for low-bit quantization. For both regimes, we introduce standard pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common computer vision and natural language processing models. We also propose a debugging workflow to identify and address common issues when quantizing a new model.

2 Quantization fundamentals

In this section, we introduce the basic principles of neural network quantization and of fixed-point accelerators on which quantized networks run on. We start with a hardware motivation and then introduce standard quantization schemes and their properties. Later we discuss practical considerations related to layers commonly found in modern neural networks and their implications for fixed-point accelerators.

2.1 Hardware background

Before diving into the technical details, we first explore the hardware background of quantization and how it enables efficient inference on device. Figure 1 provides a schematic overview of how a matrix-vector multiplication, y = Wx + b, is calculated in a neural network (NN) accelerator. This is the building block of larger matrix-matrix multiplications and convolutions found in neural networks. Such hardware blocks aim at improving the efficiency of NN inference by performing as many calculations as possible in parallel. The two fundamental components of this NN accelerator are the processing elements C n,m and the accumulators A n . Our toy example in figure 1 has 16 processing elements arranged in a square grid and 4 accumulators. The calculation starts by loading the accumulators with the bias value b n . We then load the weight values W n,m and the input values x m into the array and compute their product in the respective processing elements C n,m = W n,m x m in a single cycle. Their results are then added in the accumulators:

A n = b n + m C n,mThe above operation is also referred to as Multiply-Accumulate (MAC). This step is repeated many times for larger matrix-vector multiplications. Once all cycles are completed, the values in the accumulators are then moved back to memory to be used in the next neural network layer. Neural networks are commonly trained using FP32 weights and activations. If we were to perform inference in FP32, the processing elements and the accumulator would have to support floating-point logic, and we would need to transfer the 32-bit data from memory to the processing units. MAC operations and data transfer consume the bulk of the energy spent during neural network inference. Hence, significant benefits can be achieved by using a lower bit fixed-point or quantized representation for these quantities. Low-bit fixed-point representations, such as INT8, not only reduce the amount data transfer but also the size and energy consumption of the MAC operation (Horowitz, 2014) . This is because the cost of digital arithmetic typically scales linearly to quadratically with the number of bits used and because fixed-point addition is more efficient than its floating-point counterpart (Horowitz, 2014) .

3 Confidential and Proprietary -Qualcomm Technologies, Inc. and/or its affiliated companies. Source sample text

W !,! W !,# W !,$ W !,% x ! x # x $ x % # !,! # !,# # !,$ # !,% # #,! # #,# # #,$ # #,% # $,! # $,# # $,$ # $,% # %,! # %,# # %,$ # %,% $ ! $ # $ $ $ % Input values Weight values Accumulators W #,! W #,# W #,$ W #,% W $,! W $,# W $,$ W $,% W %,! W %,# W %,$ W %,%

Figure 1 : A schematic overview of matrix-multiply logic in neural network accelerator hardware.

To move from floating-point to the efficient fixed-point operations, we need a scheme for converting floating-point vectors to integers. A floating-point vector x can be expressed approximately as a scalar multiplied by a vector of integer values:

x = s x • x int ≈ xwhere s x is a floating-point scale factor and x int is an integer vector, e.g., INT8. We denote this quantized version of the vector as x. By quantizing the weights and activations we can write the quantized version of the accumulation equation:

Ân = b n + m W n,m x m = b n + m s w W int n,m s x x int m = b n + s w s x m W int n,m x int m (3)

Note that we used a separate scale factor for weights, s w , and activations, s x . This provides flexibility and reduces the quantization error (more in section 2.2). Since each scale factor is applied to the whole tensor, this scheme allows us to factor the scale factors out of the summation in equation ( 3) and perform MAC operations in fixed-point format. We intentionally ignore bias quantization for now, because the bias is normally stored in higher bit-width (32-bits) and its scale factor depends on that of the weights and activations (Jacob et al., 2018) .

Figure 2 shows how the neural network accelerator changes when we introduce quantization. In our example, we use INT8 arithmetic, but this could be any quantization format for the sake of this discussion. It is important to maintain a higher bit-width for the accumulators, typical 32-bits wide. Otherwise, we risk incurring loss due to overflow as more products are accumulated during the computation.

The activations stored in the 32-bit accumulators need to be written to memory before they can be used by the next layer. To reduce data transfer and the complexity of the next layer's operations, these activations are quantized back to INT8. This requires a requantization step which is shown in figure 2 .

2.2 Uniform affine quantization

In this section we define the quantization scheme that we will use in this paper. This scheme is called uniform quantization and it is the most commonly used quantization scheme because it permits efficient implementation of fixed-point arithmetic.

Figure 2 : A schematic of matrix-multiply logic in an neural network accelerator for quantized inference.

Uniform affine quantization, also known as asymmetric quantization, is defined by three quantization parameters: the scale factor s, the zero-point z and the bit-width b. The scale factor and the zero-point are used to to map a floating point value to the integer grid, whose size depends on the bit-width. The scale factor is commonly represented as a floating-point number and specifies the step-size of the quantizer. The zero-point is an integer that ensures that real zero is quantized without error. This is important to ensure that common operations like zero padding or ReLU do not induce quantization error.

Once the three quantization parameters are defined we can proceed with the quantization operation.

Starting from a real-valued vector x we first map it to the unsigned integer grid {0, . . . , 2 b -1}:

x int = clamp x s + z; 0, 2 b -1 , ()

where • is the round-to-nearest operator and clamping is defined as:

clamp (x; a, c) =    a, x < a, x, a ≤ x ≤ c, c, x > c.

(5)

To approximate the real-valued input x we perfrom a de-quantization step:

x ≈ x = s (x int -z)Combining the two steps above we can provide a general definition for the quantization function, q(•), as:

x = q(x; s, z, b) = s clamp x s + z; 0, 2 b -1 -z ,Through the de-quantization step, we can also define the quantization grid limits (q min , q max ) where q min = -sz and q max = s(2 b -1z). Any values of x that lie outside this range will be clipped to its limits, incurring a clipping error. If we want to reduce the clipping error we can expand the quantization range by increasing the scale factor. However, increasing the scale factor leads to increased rounding error as the rounding error lies in the range -1 2 s, 1 2 s . In section 3.1, we explore in more detail how to choose the quantization parameters to achieve the right trade-off between clipping and rounding errors.

2.2.1 Symmetric uniform quantization

Symmetric quantization is a simplified version of the general asymmetric case. The symmetric quantizer restricts the zero-point to 0. This reduces the computational overhead of dealing with zero-point offset during the accumulation operation in equation (3). But the lack of offset restricts the mapping between integer and floating-point domain. As a result, the choice of signed or unsigned integer grid matters:

x = s x int (8a)

x int = clamp x s ; 0, 2 b -1 for unsigned integers (8b) x int = clamp x s ; -2 b-1 , 2 b-1 -1 for signed integers (8c)

Unsigned symmetric quantization is well suited for one-tailed distributions, such as ReLU activations (see figure 3 ). On the other hand, signed symmetric quantization can be chosen for distributions that are roughly symmetric about zero.

13 Confidential and Proprietary -Qualcomm Technologies, Inc. and/or its affiliated companies.

0 0 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " K W F 1 W s L 3 C x 0 f 3 y Y p G D x U v z 2 0 H + g = " > A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K k k p 6 r H g x W M L 9 g P a U D b b S b t 2 s w m 7 G 6 G E / g I v H h T x 6 k / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a 1 n G q G L Z Y L G L V D a h G w S W 2 D D c C u 4 l C G g U C O 8 H k b u 5 3 n l B p H s s H M 0 3 Q j + h I 8 p A z a q z U 1 I N S 2 a 2 4 C 5 B 1 4 u W k D D k a g 9 J X f x i z N E J p m K B a 9 z w 3 M X 5 G l e F M 4 K z Y T z U m l E 3 o C H u W S h q h 9 r P F o T N y a Z U h C W N l S x q y U H 9 P Z D T S e h o F t j O i Z q x X v b n 4 n 9 d L T X j r Z 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T d G G 4 K 2 + v E 7 a 1 Y p 3 X a k 2 a + V 6 L Y + j A O d w A V f g w Q 3 U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 Y N J 5 8 5 g z 9 w P n 8 A 3 L m M 8 A = = < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " + F y 6 c P h / q q H p Q F s 8 d 3 c p W a y z o Q 8 = " > A A A B + X i c b V B N S 8 N A E N 3 U r 1 q / o h 6 9 L B b B U 0 m K q M e C F 4 8 V 7 Q e 0 o W w 2 k 3 b p 7 i b s b g o l 9 J 9 4 8 a C I V / + J N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y c q Z N p 7 3 7 Z Q 2 N r e 2 d 8 q 7 l b 3 9 g 8 M j 9 / i k r Z N M U W j R h C e q G x I N n E l o G W Y 4 d F M F R I Q c O u H 4 b u 5 3 J q A 0 S + S T m a Y Q C D K U L G a U G C s N X P d x K g Q Y x S j W b C g h G r h V r + Y t g N e J X 5 A q K t A c u F / 9 K K G Z A G k o J 1 r 3 f C 8 1 Q U 6 U Y Z T D r N L P N K S E j s k Q e p Z K I k A H + e L y G b 6 w S o T j R N m S B i / U 3 x M 5 E V p P R W g 7 B T E j v e r N x f + 8 X m b i 2 y B n M s 0 M S L p c F G c c m w T P Y 8 A R U 0 A N n 1 p C q G L 2 V k x H R B F q b F g V G 4 K / + v I 6 a d d r / n W t / l C v N q 6 K O M r o D J 2 j S + S j G 9 R A 9 6 i J W o i i C X p G r + j N y Z 0 X 5 9 3 5 W L a W n G L m F P 2 B 8 / k D n H a T m Q = = < / l a t e x i t > Symmetric signed < l a t e x i t s h a 1 _ b a s e 6 4 = " L A N 1 F z 9 J J p B s Y h c u R N m q r 7 w 7 H z c = " > A A A C C H i c b V D L S s N A F J 3 4 r P V V d e n C w S K 4 K k k p 2 m X B j c s K 9 g F N C J P J p B 0 6 e T B z I y 0 h S z f + i h s X i r j 1 E 9 z 5 N 0 7 b L L T 1 w M D h n H u Z e 4 6 X C K 7 A N L + N t f W N z a 3 t 0 k 5 5 d 2 / / 4 L B y d N x V c S o p 6 9 B Y x L L v E c U E j 1 g H O A j W T y Q j o S d Y z x v f z P z e A 5 O K x 9 E 9 T B P m h G Q Y 8 Y B T A l p y K 2 f K p n 4 M 2 A 4 J j L w g m + R u Z g O b Q M Y j a O a 5 W 6 m a N X M O v E q s g l R R g b Z b + b L 9 m K Y h i 4 A K o t T A M h N w M i K B U 8 H y s p 0 q l h A 6 J k M 2 0 D Q i I V N O N g + S 4 w u t + D i I p X 4 R 4 L n 6 e y M j o V L T 0 N O T s 3 v V s j c T / / M G K Q R N R 0 d K U m A R X X w U p A J D j G e t Y J 9 L R k F M N S F U c n 0 r p i M i C Q X d X V m X Y C 1 H X i X d e s 2 6 q t X v G t V W o 6 i j h E 7 R O b p E F r p G L X S L 2 q i D K H p E z + g V v R l P x o v x b n w s R t e M Y u c E / Y H x + Q M c w p q k < / l a t e x i t > s • x int8 < l a t e x i t s h a 1 _ b a s e 6 4 = " w Q V D M L D E U I q v t U S 9 d r D Q V Q O p Z c o = " > A A A C C X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 V w V Z J S t M u C G 5 c V 7 A O a E i a T S T t 0 M g k z N 9 I S s n X j r 7 h x o Y h b / 8 C d f + P 0 s d D W A x c O 5 9 z L v f f 4 i e A a b P v b K m x s b m 3 v F H d L e / s H h 0 f l 4 5 O O j l N F W Z v G I l Y 9 n 2 g m u G R t 4 C B Y L 1 G M R L 5 g X X 9 8 M / O 7 D 0 x p H s t 7 m C Z s E J G h 5 C G n B I z k l b F 2 a R A D d i M C I z / M J r m X u c A m k K V c Q i P P v X L F r t p z 4 H X i L E k F L d H y y l 9 u E N M 0 Y h K o I F r 3 H T u B Q U Y U c C p Y X n J T z R J C x 2 T I + o Z K E j E 9 y O a f 5 P j C K A E O Y 2 V K A p 6 r v y c y E m k 9 j X z T O T t Y r 3 o z 8 T + v n 0 L Y G G R c J i k w S R e L w l R g i P E s F h x w x S i I q S G E K m 5 u x X R E F K F g w i u Z E J z V l 9 d J p 1 Z 1 r q q 1 u 3 q l W V / G U U R n 6 B x d I g d d o y a 6 R S 3 U R h Q 9 o m f 0 i t 6 s J + v F e r c + F q 0 F a z l z i v 7 A + v w B A G + b I w = = < / l a t e x i t > s • x uint8 0 255 < l a t e x i t s h a 1 _ b a s e 6 4 = " K W F 1 W s L 3 C x 0 f 3 y Y p G D x U v z 2 0 H + g = " > A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K k k p 6 r H g x W M L 9 g P a U D b b S b t 2 s w m 7 G 6 G E / g I v H h T x 6 k / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a 1 n G q G L Z Y L G L V D a h G w S W 2 D D c C u 4 l C G g U C O 8 H k b u 5 3 n l B p H s s H M 0 3 Q j + h I 8 p A z a q z U 1 I N S 2 a 2 4 C 5 B 1 4 u W k D D k a g 9 J X f x i z N E J p m K B a 9 z w 3 M X 5 G l e F M 4 K z Y T z U m l E 3 o C H u W S h q h 9 r P F o T N y a Z U h C W N l S x q y U H 9 P Z D T S e h o F t j O i Z q x X v b n 4 n 9 d L T X j r Z 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T d G G 4 K 2 + v E 7 a 1 Y p 3 X a k 2 a + V 6 L Y + j A O d w A V f g w Q 3 U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 Y N J 5 8 5 g z 9 w P n 8 A 3 L m M 8 A = = < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " B X Q f Q 4 1 g D q S W S L O z Y 4 / / / I s z X 2 g = " > A A A B + 3 i c b V B N S 8 N A E N 3 4 W e t X r E c v i 0 X w V J I i 6 r H g x W N F + w F t K J v N t F 2 6 u w m 7 G z G E / h U v H h T x 6 h / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T D j T x v O + n b X 1 j c 2 t 7 d J O e X d v / + D Q P a q 0 d Z w q C i 0 a 8 1 h 1 Q 6 K B M w k t w w y H b q K A i J B D J 5 z c z P z O I y j N Y v l g s g Q C Q U a S D R k l x k o D t 3 K f C Q F G M Y p T q d l I Q j R w q 1 7 N m w O v E r 8 g V V S g O X C / + l F M U w H S U E 6 0 7 v l e Y o K c K M M o h 2 m 5 n 2 p I C J 2 Q E f Q s l U S A D v L 5 7 V N 8 Z p U I D 2 N l S x o 8 V 3 9 P 5 E R o n Y n Q d g p i x n r Z m 4 n / e b 3 U D K + D n M k k N S D p Y t E w 5 d j E e B Y E j p g C a n h m C a G K 2 V s x H R N F q L F x l W 0 I / v L L q 6 R d r / m X t f p d v d q 4 K O I o o R N 0 i s 6 R j 6 5 Q A 9 2 i J m o h i p 7 Q M 3 p F b 8 7 U e X H e n Y 9 F 6 5 p T z B y j P 3 A + f w B M Z Z S Q < / l a t e x i t > Symmetric unsigned < l a t e x i t s h a 1 _ b a s e 6 4 = " U w Z K x C f W W g L J S 3 6 4 f O k r k k i O z m s = " > A A A B 8 X i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j N F 1 G X F j c s K 9 o H t U D L p n T Y 0 k x m S O 0 I Z + h d u X C j i 1 r 9 x 5 9 + Y t r P Q 1 g O B w z n 3 k n N P k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V H J o 8 l r H u B M y A F A q a K F B C J 9 H A o k B C O x j f z v z 2 E 2 g j Y v W A k w T 8 i A 2 V C A V n a K X H G z O J I k A t e L 9 c c a v u H H S V e D m p k B y N f v m r N 4 h 5 G o F C L p k x X c 9 N 0 M + Y R s E l T E u 9 1 E D C + J g N o W u p Y h E Y P 5 s n n t I z q w x o G G v 7 F N K 5 + n s j Y 5 G x 0 Q I 7 G T E c m W V v J v 7 n d V M M r / 1 M q C R F U H z x U Z h K i j G d n U 8 H Q g N H O b G E c S 1 s V s p H T D O O t q S S L c F b P n m V t G p V 7 7 J a u 6 9 V 6 h d 5 H U V y Q k 7 J O f H I F a m T O 9 I g T c K J I s / k l b w 5 x n l x 3 p 2 P x W j B y X e O y R 8 4 n z / K F p D z < / l a t e x i t > Asymmetric 0 255 < l a t e x i t s h a 1 _ b a s e 6 4 = " K W F 1 W s L 3 C x 0 f 3 y Y p G D x U v z 2 0 H + g = " > A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K k k p 6 r H g x W M L 9 g P a U D b b S b t 2 s w m 7 G 6 G E / g I v H h T x 6 k / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a 1 n G q G L Z Y L G L V D a h G w S W 2 D D c C u 4 l C G g U C O 8 H k b u 5 3 n l B p H s s H M 0 3 Q j + h I 8 p A z a q z U 1 I N S 2 a 2 4 C 5 B 1 4 u W k D D k a g 9 J X f x i z N E J p m K B a 9 z w 3 M X 5 G l e F M 4 K z Y T z U m l E 3 o C H u W S h q h 9 r P F o T N y a Z U h C W N l S x q y U H 9 P Z D T S e h o F t j O i Z q x X v b n 4 n 9 d L T X j r Z 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T d G G 4 K 2 + v E 7 a 1 Y p 3 X a k 2 a + V 6 L Y + j A O d w A V f g w Q 3 U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 Y N J 5 8 5 g z 9 w P n 8 A 3 L m M 8 A = = < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 8 a d + I y s U X r T T s 3 E V 6 D C X U w R g C Q = " > A A A B 9 X i c b V B N S w M x E M 3 6 W e t X 1 a O X Y B G 8 W H Z L U S 9 C w Y v H C v Y D 2 r V k 0 2 w b m m S X Z F a t S / + H F w + K e P W / e P P f m L Z 7 0 N Y H A 4 / 3 Z p i Z F 8 S C G 3 D d b 2 d p e W V 1 b T 2 3 k d / c 2 t 7 Z L e z t N 0 y U a M r q N B K R b g X E M M E V q w M H w V q x Z k Q G g j W D 4 d X E b 9 4 z b X i k b m E U M 1 + S v u I h p w S s d N c B 9 g i p 5 G p 8 e W q e u o W i W 3 K n w I v E y 0 g R Z a h 1 C 1 + d X k Q T y R R Q Q Y x p e 2 4 M f k o 0 c C r Y O N 9 J D I s J H Z I + a 1 u q i G T G T 6 d X j / G x V X o 4 j L Q t B X i q / p 5 I i T R m J A P b K Q k M z L w 3 E f / z 2 g m E F 3 7 K V Z w A U 3 S 2 K E w E h g h P I s A 9 r h k F M b K E U M 3 t r Z g O i C Y U b F B 5 G 4 I 3 / / I i a Z R L 3 l m p f F M p V i t Z H D l 0 i I 7 Q C f L Q O a q i a 1 R D d U S R R s / o F b 0 5 D 8 6 L 8 + 5 8 z F q X n G z m A P 2 B 8 / k D 2 i G S s w = = < / l a t e x i t > min = sz < l a t e x i t s h a 1 _ b a s e 6 4 = " O X d v s 7 X 7 R d Z O Z A R T Q e Z F o g a w H d U = " > A A A B 6 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e w W U Y 8 F L x 4 r 2 g 9 o l 5 J N s 2 1 o k l 2 S r F i W / g Q v H h T x 6 i / y 5 r 8 x b f e g r Q 8 G H u / N M D M v T A Q 3 1 v O + U W F t f W N z q 7 h d 2 t n d 2 z 8 o H x 6 1 T J x q y p o 0 F r H u h M Q w w R V r W m 4 F 6 y S a E R k K 1 g 7 H N z O / / c i 0 4 b F 6 s J O E B Z I M F Y 8 4 J d Z J 9 5 I 8 9 c s V r + r N g V e J n 5 M K 5 G j 0 y 1 + 9 Q U x T y Z S l g h j T 9 b 3 E B h n R l l P B p q V e a l h C 6 J g M W d d R R S Q z Q T Y / d Y r P n D L A U a x d K Y v n 6 u + J j E h j J j J 0 n Z L Y k V n 2 Z u J / X j e 1 0 X W Q c Z W k l i m 6 W B S l A t s Y z / 7 G A 6 4 Z t W L i C K G a u 1 s x H R F N q H X p l F w I / v L L q 6 R V q / q X 1 d p d r V K / y O M o w g m c w j n 4 c A V 1 u I U G N I H C E J 7 h F d 6 Q Q C / o H X 0 s W g s o n z m G P 0 C f P 2 c I j d U = < / l a t e x i t > max < l a t e x i t s h a 1 _ b a s e 6 4 = " O X d v s 7 X 7 R d Z O Z A R T Q e Z F o g a w H d U = " > A A A B 6 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e w W U Y 8 F L x 4 r 2 g 9 o l 5 J N s 2 1 o k l 2 S r F i W / g Q v H h T x 6 i / y 5 r 8 x b f e g r Q 8 G H u / N M D M v T A Q 3 1 v O + U W F t f W N z q 7 h d 2 t n d 2 z 8 o H x 6 1 T J x q y p o 0 F r H u h M Q w w R V r W m 4 F 6 y S a E R k K 1 g 7 H N z O / / c i 0 4 b F 6 s J O E B Z I M F Y 8 4 J d Z J 9 5 I 8 9 c s V r + r N g V e J n 5 M K 5 G j 0 y 1 + 9 Q U x T y Z S l g h j T 9 b 3 E B h n R l l P B p q V e a l h C 6 J g M W d d R R S Q z Q T Y / d Y r P n D L A U a x d K Y v n 6 u + J j E h j J j J 0 n Z L Y k V n 2 Z u J / X j e 1 0 X W Q c Z W k l i m 6 W B S l A t s Y z / 7 G A 6 4 Z t W L i C K G a u 1 s x H R F N q H X p l F w I / v L L q 6 R V q / q X 1 d p d r V K / y O M o w g m c w j n 4 c A V 1 u I U G N I H C E J 7 h F d 6 Q Q C / o H X 0 s W g s o n z m G P 0 C f P 2 c I j d U = < / l a t e x i t > max < l a t e x i t s h a 1 _ b a s e 6 4 = " O X d v s 7 X 7 R d Z O Z A R T Q e Z F o g a w H d U = " > A A A B 6 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e w W U Y 8 F L x 4 r 2 g 9 o l 5 J N s 2 1 o k l 2 S r F i W / g Q v H h T x 6 i / y 5 r 8 x b f e g r Q 8 G H u / N M D M v T A Q 3 1 v O + U W F t f W N z q 7 h d 2 t n d 2 z 8 o H x 6 1 T J x q y p o 0 F r H u h M Q w w R V r W m 4 F 6 y S a E R k K 1 g 7 H N z O / / c i 0 4 b F 6 s J O E B Z I M F Y 8 4 J d Z J 9 5 I 8 9 c s V r + r N g V e J n 5 M K 5 G j 0 y 1 + 9 Q U x T y Z S l g h j T 9 b 3 E B h n R l l P B p q V e a l h C 6 J g M W d d R R S Q z Q T Y / d Y r P n D L A U a x d K Y v n 6 u + J j E h j J j J 0 n Z L Y k V n 2 Z u J / X j e 1 0 X W Q c Z W k l i m 6 W B S l A t s Y z / 7 G A 6 4 Z t W L i C K G a u 1 s x H R F N q H X p l F w I / v L L q 6 R V q / q X 1 d p d r V K / y O M o w g m c w j n 4 c A V 1 u I U G N I H C E J 7 h F d 6 Q Q C / o H X 0 s W g s o n z m G P 0 C f P 2 c I j d U = < / l a t e x i t > max DUPLICATE 2 (less high) 0 127 -128 < l a t e x i t s h a 1 _ b a s e 6 4 = " u s Q 8 n / D w T D C g a X u 4 j V 5 + g U 9 A Q f U = " > A A A C C X i c b V C 7 S g N B F J 2 N r x h f q 5 Y 2 g 0 G I h W E 3 B E 0 Z s L G M Y B 6 Q h D A 7 m U 2 G z M 4 u M 3 c l c d n W x l + x s V D E 1 j + w 8 2 + c P A p N P H D h c M 6 9 3 H u P F w m u w X G + r c z a + s b m V n Y 7 t 7 O 7 t 3 9 g H x 4 1 d B g r y u o 0 F K F q e U Q z w S W r A w f B W p F i J P A E a 3 q j 6 6 n f v G d K 8 1 D e w S R i 3 Y A M J P c 5 J W C k n o 0 1 L n Q C A k P P T 8 Z p L + k A G 0 M S c w m V N M U X D + c 9 O + 8 U n R n w K n E X J I 8 W q P X s r 0 4 / p H H A J F B B t G 6 7 T g T d h C j g V L A 0 1 4 k 1 i w g d k Q F r G y p J w H Q 3 m X 2 S 4 j O j 9 L E f K l M S 8 E z 9 P Z G Q Q O t J 4 J n O 6 d F 6 2 Z u K / 3 n t G P x K N + E y i o F J O l / k x w J D i K e x 4 D 5 X j I K Y G E K o 4 u Z W T I d E E Q o m v J w J w V 1 + e Z U 0 S k X 3 s l i 6 L e e r 5 U U c W X S C T l E B u e g K V d E N q q E 6 o u g R P a N X 9 G Y 9 W S / W u / U x b 8 1 Y i 5 l j 9 A f W 5 w + Q I Z o 1 < / l a t e x i t > s(x uint8 z)

Figure 3 : A visual explanation of the different uniform quantization grids for a bit-width of 8. s is the scaling factor, z the zero-point. The floating-point grid is in black, the integer quantized grid in blue.

2.2.2 Power-of-two quantizer

Power-of-two quantization is a special case of symmetric quantization, in which the scale factor is restricted to a power-of-two, s = 2 -k . This choice can bring hardware efficiencies because scaling with s corresponds to simple bit-shifting. However, the restricted expressiveness of the scale factor can complicate the trade-off between rounding and clipping error.

2.2.3 Quantization granularity

So far, we have defined a single set of quantization parameters (quantizer) per tensor, one for the weights and one for activations, as seen in equation ( 3). This is called per-tensor quantization. We can also define a separate quantizer for individual segments of a tensor (e.g., output channels of a weight tensor), thus increasing the quantization granularity. In neural network quantization, per-tensor quantization is the the most common choice of granularity due to its simpler hardware implementation: all accumulators in equation ( 3) use the same scale factor, s w s x . However, we could use finer granularity to further improve performance. For example, for weight tensors, we can specify a different quantizer per output channel. This is known as per-channel quantization and its implications are discussed in more detailed in section 2.4.2.

Other works go beyond per-channel quantization parameters and apply separate quantizers per group of weights or activations (Rouhani et al., 2020; Stock et al., 2019; Nascimento et al., 2019) . Increasing the granularity of the groups generally improves accuracy at the cost of some extra overhead. The overhead is associated with accumulators handling sums of values with varying scale factors. Most existing fixed-point accelerators do not currently support such logic and for this reason, we will not consider them in this work. However, as research in this area grows, more hardware support for these methods can be expected in the future.

2.3 Quantization simulation

To test how well a neural network would run on a quantized device, we often simulate the quantized behavior on the same general purpose hardware we use for training neural networks. This is called quantization simulation. We aim to approximate fixed-point operations using floating-point hardware. Such simulations are significantly easier to implement compared to running experiments on actual quantized hardware or using quantized kernels. They allow the user to efficiently test various quantization options and it enables GPU acceleration for quantization-aware training as described in section 4. In this section, we first explain the fundamentals of this simulation process and then discuss techniques that help to reduce the difference between the simulated and the actual on-device performance.

Previously, we saw how matrix-vector multiplication is calculated in dedicated fixed-point hardware.

In figure 4a , we generalize this process for a convolutional layer, but we also include an activation function to make it more realistic. During on-device inference, all the inputs (biases, weight and input activations) to the hardware are in a fixed-point format. However, when we simulate quantization using common deep learning frameworks and general-purpose hardware these quantities are in floating-point. This is why we introduce quantizer blocks in the compute graph to induce quantization effects.

Figure 4b shows how the same convolutional layer is modelled in a deep-learning framework. Quantizer blocks are added in between the weights and the convolution to simulate weight quantization, and after the activation function to simulate activation quantization. The bias is often not quantized because it is stored in higher-precision. In section 2.3.2, we discuss in more detail when it is appropriate to position the quantizer after the non-linearity. The quantizer block implements the quantization function of equation ( 7) and each quantizer is defined by a set of quantization parameters (scale factor, zero-point, bit-width). Both the input and output of the quantizer are in floating-point format but the output lies on the quantization grid.

2.3.1 Batch normalization folding

Batch normalization (Ioffe & Szegedy, 2015) is a standard component of modern convolutional networks. Batch normalization normalizes the output of a linear layer before scaling and adding an offset (see equation 9). For on-device inference, these operations are folded into the previous or next linear layers in a step called batch normalization folding (Krishnamoorthi, 2018; Jacob et al., 2018) . This removes the batch normalization operations entirely from the network, as the calculations are absorbed into an adjacent linear layer. Besides reducing the computational overhead of the additional scaling and offset, this prevents extra data movement and the quantization of the layer's output. More formally, during inference, batch normalization is defined as an affine map of the output x:

BatchNorm(x) = γ x -µ √ σ 2 + + β (9)

where µ and σ are the mean and variance computed during training as exponential moving average over batch-statistics, and γ and β are learned affine hyper-parameters per-channel. If batch normalization is applied right after a linear layer y = BatchNorm(Wx), we can rewrite the terms such that the batch normalization operation is fused with the linear layer itself. Assuming a weight matrix W ∈ R n×m we apply batch normalization to each output y k for k = {1, . . . , n}:

y k = BatchNorm(W k,: x) = γ k W k,: x -µ k σ 2 k + + β k = γ k W k,: σ 2 k + x + β k - γ k µ k σ 2 k + = W k,: x + b k (10)

where:

W k,: = γ k W k,: σ 2 k + , () b k = β k - γ k µ k σ 2 k + .2.3.2 Activation function fusing

In our naive quantized accelerator introduced in section 2.1, we saw that the requantization of activations happens after the matrix multiplication or convolutional output values are calculated. However, in practice, we often have a non-linearity directly following the linear operation. It would be wasteful to write the linear layer's activations to memory, and then load them back into a compute core to apply a non-linearity. For this reason, many hardware solutions come with a hardware unit that applies the non-linearity before the requantization step. If this is the case, we only have to simulate requantization that happens after the non-linearity. For example, ReLU non-linearities are readily modelled by the requantization block, as you can just set the minimum representable value of that activation quantization to 0.

Other more complex activation functions, such as sigmoid or Swish (Ramachandran et al., 2017) , require more dedicated support. If this support is not available, we need to add a quantization step before and after the non-linearity in the graph. This can have a big impact on the accuracy of quantized model. Although newer activations like Swish functions provide accuracy improvement in floating-point, these may vanish after quantization or may be less efficient to deploy on fixed-point hardware.

2.3.3 Other layers and quantization

There are many other types of layers being used in neural networks. How these are modeled depends greatly on the specific hardware implementation. Sometimes the mismatch between simulated quantization and on-target performance is down to layers not being properly quantized. Here, we provide some guidance on how to simulate quantization for a few commonly used layers:

Max pooling Activation quantization is not required because the input and output values are on the same quantization grid. Average pooling The average of integers is not necessarily an integer. For this reason, a quantization step is required after average-pooling. However, we use the same quantizer for the inputs and outputs as the quantization range does not significantly change. Element-wise addition Despite its simple nature, this operation is difficult to simulate accurately.

During addition, the quantization ranges of both inputs have to match exactly. If these ranges do not match, extra care is needed to make addition work as intended. There is no single accepted solution for this but adding a requantization step can simulate the added noise coarsely. Another approach is to optimize the network by tying the quantization grids of the inputs. This would prevent the requantization step but may require fine-tuning. Concatenation The two branches that are being concatenated generally do not share the same quantization parameters. This means that their quantization grids may not overlap making a requantization step necessary. As with element-wise addition, it is possible to optimize your network to have shared quantization parameters for the branches being concatenated.

2.4 Practical considerations

When quantizing neural networks with multiple layers, we are confronted with a large space of quantization choices including the quantization scheme, granularity, and bit-width. In this section, we explore some of the practical considerations that help reduce the search space.

Note that in this white paper we only consider homogeneous bit-width. This means that the bit-width chosen for either weights or activations remains constant across all layers. Homogeneous bit-width is more universally supported by hardware but some recent works also explore the implementation of heterogeneous bit-width or mixed-precision (van Baalen et al., 2020; Dong et al., 2019; Uhlich et al., 2020) .

2.4.1 Symmetric vs. asymmetric quantization

For each weight and activation quantization, we have to choose a quantization scheme. On one hand, asymmetric quantization is more expressive because there is an extra offset parameter, but on the other hand there is a possible computational overhead. To see why this is the case, consider what happens when asymmetric weights, W = s w (W int -z w ), are multiplied with asymmetric activations x = s x (x intz x ):

W x = s w (W int -z w )s x (x int -z x ) = s w s x W int x int -s w z w s x x int -s w s x z x W int + s w z w s x z x . ()

The first term is what we would have if both operations were in symmetric format. The third and fourth terms depend only on the scale, offset and weight values, which are known in advance. Thus these two terms can be pre-computed and added to the bias term of a layer at virtually no cost. The second term, however, depends on the input data x. This means that for each batch of data we need to compute an additional term during inference. This can lead to significant overhead in both latency and power, as it is equivalent to adding an extra channel.

For this reason, it is a common approach to use asymmetric activation quantization and symmetric weight quantization that avoids the additional data-dependent term.

2.4.2 Per-tensor and per-channel quantization

In section 2.2.3, we discussed different levels of quantization granularity. Per-tensor quantization of weights and activations has been standard for a while because it is supported by all fixed-point accelerators. However, per-channel quantization of the weights can improve accuracy, especially when the distribution of weights varies significantly from channel to channel. Looking back at the quantized MAC operation in equation ( 3), we can see that per-channel weight quantization can be implemented in the accelerator by applying a separate per-channel weight scale factor without requiring rescaling. Per-channel quantization of activations is much harder to implement because we cannot factor the scale factor out of the summation and would, therefore, require rescaling the accumulator for each input channel. Whereas per-channel quantization of weights is increasingly becoming common practice, not all commercial hardware supports it. Therefore, it is important to check if it is possible in your intended target device. be data-free or may require a small calibration set, which is often readily available. Additionally, having almost no hyperparameter tuning makes them usable via a single API call as a black-box method to quantize a pretrained neural network in a computationally efficient manner. This frees the neural network designer from having to be an expert in quantization and thus allows for a much wider application of neural network quantization.

3 Post-training quantization

A fundamental step in the PTQ process is finding good quantization ranges for each quantizer. We briefly discussed in section 2.2 how the choice of quantization range affects the quantization error. In this section, we start by discussing various common methods used in practice to find good quantization parameters. We then explore common issues observed during PTQ and introduce the most successful techniques to overcome them. Using these techniques we present a standard post-training quantization pipeline, which we find to work best in most common scenarios and, finally, we introduce a set of debugging steps to improve the performance of the quantized model.

3.1 Quantization range setting

Quantization range setting refers to the method of determining clipping thresholds of the quantization grid, q min and q max (see equation 7). The key trade-off in range setting is between clipping and rounding error, described in section 2.2, and their impact on the final task loss for each quantizer being configured. Each of the methods described here provides a different trade-off between the two quantities. These methods typically optimize local cost functions instead of the task loss. This is because in PTQ we aim for computationally fast methods without the need for end-to-end training.

Weights can usually be quantized without any need for calibration data. However, determining parameters for activation quantization often requires a few batches of calibration data.

Min-max To cover the whole dynamic range of the tensor, we can define the quantization parameters as follows

q min = min V, (14) q max = max V, ()

where V denotes the tensor to be quantized. This leads to no clipping error. However, this approach is sensitive to outliers as strong outliers may cause excessive rounding errors.

Mean squared error (MSE) One way to alleviate the issue of large outliers is to use MSE-based range setting. In this range setting method we find q min and q max that minimize the MSE between the original and the quantized tensor:

arg min qmin,qmax V -V(q min , q max ) 2 F ,where V(q min , q max ) denotes the quantized version of V and • F is the Frobenius norm. The optimization problem is commonly solved using grid search, golden section method or analytical approximations with closed-form solution (Banner et al., 2019) . Several variants of this range setting method exist in literature but they are all very similar in terms of objective function and optimization. Cross entropy For certain layers, all values in the tensor being quantized may not be equally important. One such scenario is the quantization of logits in the last layer of classification networks, in which it is important to preserve the order of the largest value after quantization. MSE may not be a suitable metric for this, as it weighs all the values in a tensor equally regardless of their order.

For a larger number of classes, we usually have a large number of small or negative logits that are unimportant for prediction accuracy and few larger values that matter. In this case, MSE would incur a large quantization error to the few larger important logits while trying to reduce the quantization error of the more populous smaller logits. In this specific case, it is beneficial to minimize the following cross-entropy loss function

arg min qmin,qmax H (ψ (v) , ψ (v(q min , q max )))where H(•, •) denotes the cross-entropy function, ψ is the softmax function, and v is the logits vector.

BN based range setting Range setting for activation quantizers often requires some calibration data. If a layer has batch-normalized activations, the per-channel mean and standard deviation of the activations are equal to the learned batch normalization shift and scale parameters, respectively. These can then be used to find suitable parameters for activation quantizer as follows (Nagel et al., 2019) :

q min = min (β -αγ) (18) q max = max (β + αγ)where β and γ are vectors of per-channel learned shift and scale parameters, and α > 0. Nagel et al. ( 2019) uses α = 6 so that only large outliers are clipped.

Comparison In table 1 we compare range setting methods for weight quantization. For high bitwidths, the MSE and min-max approaches are mostly on par. However, at lower bit-widths the MSE approach clearly outperforms the min-max. In table 2, we present a similar comparison for activation quantization. We note that MSE combined with cross-entropy for the last layer, denoted as MSE + Xent, outperforms other methods, especially at lower bit-widths. The table also clearly demonstrates the benefit of using cross-entropy for the last layer instead of the MSE objective.

3.2 Cross-Layer Equalization

A common issue for quantization error is that elements in the same tensor can have significantly different magnitudes. As discussed in the previous section, range setting for the quantization grid tries to find a good trade-off between clipping and rounding error. Unfortunately, in some cases, the difference in magnitude between them is so large that even for moderate quantization (e.g., INT8), we cannot find a suitable trade-off. Nagel et al. (2019) showed that this is especially prevalent in depth-wise separable layers since only a few weights are responsible for each output feature and this might result in higher variability of the weights. Further, they noted that batch normalization folding adds to this effect and can result in a strong imbalance between weights connected to various output channels (see figure 5 ). While the latter is less of an issue for a more fine-grained quantization granularity (e.g., per-channel quantization), this remains a big issue for the more widely used pertensor quantization. Several papers (Krishnamoorthi, 2018; Nagel et al., 2019; Sheng et al., 2018b) noted that efficient models with depth-wise separable convolutions, such as MobileNetV1 (Howard 

l u Y 9 Y G m C X 9 D x D N v 5 g 0 C s = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S a G E / I k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q G c x 9 U I 8 E i x g B G s j D W y 7 H 2 I 9 9 o O 0 k z 2 l V f c i G 9 g V p + b M g V a J W 5 A K F G g O 7 K / + M C J J S I U m H C v V c 5 1 Y e y m W m h F O s 3 I / U T T G Z I J H t G e o w C F V X j p P n q F z o w x R E E n z h E Z z 9 f d G i k O l Z q F v J v O c a t n L x f + 8 X q K D G y 9 l I k 4 0 F W R x K E g 4 0 h H K a 0 B D J i n R f G Y I J p K Z r I i M s c R E m 7 L K p g R 3 + c u r p F 2 v u V e 1 + s N l p X F b 1 F G C U z i D K r h w D Q 2 4 h y a 0 g M A U n u E V 3 q z U e r H e r Y / F 6 J p V 7 J z A H 1 i f P w w y k 0 c = < / l a t e x i t > W (1) < l a t e x i t s h a 1 _ b a s e 6 4 = " n K z Q n z 9 b X 9 A w U J l w 9 o N M O N 3 d Z i Y = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S a G E / I k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q G c x 9 U I 8 E i x g B G s j D W y 7 H 2 I 9 9 o O 0 k z 2 l 1 f p F N r A r T s 2 Z A 6 0 S t y A V K N A c 2 F / 9 Y U S S k A p N O F a q 5 z q x 9 l I s N S O c Z u V + o m i M y Q S P a M 9 Q g U O q v H S e P E P n R h m i I J L m C Y 3 m 6 u + N F I d K z U L f T O Y 5 1 b K X i / 9 5 v U Q H N 1 7 K R J x o K s j i U J B w p C O U 1 4 C G T F K i + c w Q T C Q z W R E Z Y 4 m J N m W V T Q n u 8 p d X S b t e c 6 9 q 9 Y f L S u O 2 q K M E p 3 A G V X D h G h p w D 0 1 o A Y E p P M M r v F m p 9 W K 9 W x + L 0 T W r 2 D m B P 7 A + f w A N u J N I < / l a t e x i t > W (2) < l a t e x i t s h a 1 _ b a s e 6 4 = " 2 1 y z a a 2 R 3 m R H u 9 W K m z G 0 l V / I w q Y = " > A A A B 8 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 2 K q M e i F 4 8 V 7 A e k o W w 2 m 3 b p Z j f s T o Q S + j O 8 e F D E q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B X c g O t + O 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / + g e n j U M S r T l L W p E k r 3 Q m K Y 4 J K 1 g Y N g v V Q z k o S C d c P x 3 c z v P j F t u J K P M E l Z k J C h 5 D G n B K z k 9 2 m k A H s X Z s A H 1 Z p b d + f A q 8 Q r S A 0 V a A 2 q X / 1 I 0 S x h E q g g x v i e m 0 K Q E w 2 c C j a t 9 D P D U k L H Z M h 8 S y V J m A n y + c l T f G a V C M d K 2 5 K A 5 + r v i Z w k x k y S 0 H Y m B E Z m 2 Z u J / 3 l + B v F N k H O Z Z s A k X S y K M 4 F B 4 d n / O O K a U R A T S w j V 3 N 6 K 6 Y h o Q s G m V L E h e M s v r 5 J O o + 5 d 1 R s P l 7 X m b R F H G Z 2 g U 3 S O P H S N m u g e t V A b U a T Q M 3 p F b w 4 4 L 8 6 7 8 7 F o L T n F z D H 6 A + f z B 2 f 8 k L A = < / l a t e x i t > •1/s i < l a t e x i t s h a 1 _ b a s e 6 4 = " i C 1 C O 8 R B S t l R Y C X n 0 L p s T N B q V a I = " > A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 m K q M e i F 4 8 V 7 A e 2 o W w 2 m 3 b p Z j f s T o Q S + i + 8 e F D E q / / G m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I B H c g O t + O 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / + g e n j U M S r V l L W p E k r 3 A m K Y 4 J K 1 g Y N g v U Q z E g e C d Y P J b e 5 3 n 5 g 2 X M k H m C b M j 8 l I 8 o h T A l Z 6 H N B Q A T Z D X h l W a 2 7 d n Q O v E q 8 g N V S g N a x + D U J F 0 5 h J o I I Y 0 / f c B P y M a O B U s F l l k B q W E D o h I 9 a 3 V J K Y G T + b X z z D Z 1 Y J c a S 0 L Q l 4 r v 6 e y E h s z D Q O b G d M Y G y W v V z 8 z + u n E F 3 7 G Z d J C k z S x a I o F R g U z t / H I d e M g p h a Q q j m 9 l Z M x 0 Q T C j a k P A R v + e V V 0 m n U v c t 6 4 / 6 i 1 r w p 4 i i j E 3 S K z p G H r l A T 3 a E W a i O K J H p G r + j N M c 6 L 8 + 5 8 L F p L T j F z j P 7 A + f w B v m O Q U A = = < / l a t e x i t > •s i < l a t e x i t s h a 1 _ b a s e 6 4 = " A T g U p w f G W o Z q d P 7 2 T 2 b j W z F e P X U = " > A A A B 8 X i c b V D L S g M x F L 3 j s 9 Z X 1 a W b Y B F c l Z k i 6 r L o x m U F + 8 C 2 l E x 6 p w 3 N Z I Y k I 5 a h f + H G h S J u / R t 3 / o 2 Z d h b a e i B w O O d e c u 7 x Y 8 G 1 c d 1 v Z 2 V 1 b X 1 j s 7 B V 3 N 7 Z 3 d s v H R w 2 d Z Q o h g 0 W i U i 1 f a p R c I k N w 4 3 A d q y Q h r 7 A l j + + y f z W I y r N I 3 l v J j H 2 Q j q U P O C M G i s 9 d E N q R n 6 Q P k 3 7 p b J b c W c g y 8 T L S R l y 1 P u l r + 4 g Y k m I 0 j B B t e 5 4 b m x 6 K V W G M 4 H T Y j f R G F M 2 p k P s W C p p i L q X z h J P y a l V B i S I l H 3 S k J n 6 e y O l o d a T 0 L e T W U K 9 6 G X i f 1 4 n M c F V L + U y T g x K N v 8 o S A Q x E c n O J w O u k B k x s Y Q y x W 1 W w k Z U U W Z s S U V b g r d 4 8 j J p V i v e R a V 6 d 1 6 u X e d 1 F O A Y T u A M P L i E G t x C H R r A Q M I z v M K b o 5 0 X 5 9 3 5 m I + u O P n O E f y B 8 / k D / 4 O R I w = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " 4 B S d + c d a 8 Y s x y 4 4 c g N a L W S X + h 7 I = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q N O Y e i E e C R Y w g r W R B r b d D 7 E e + 0 G W z p 6 y q n s x G 9 g V p + b M g V a J W 5 A K F G g O 7 K / + M C J J S I U m H C v V c 5 1 Y e x m W m h F O Z + V + o m i M y Q S P a M 9 Q g U O q v G y e f I b O j T J E Q S T N E x r N 1 d 8 b G Q 6 V S k P f T O Y 5 1 b K X i / 9 5 v U Q H N 1 7 G R J x o K s j i U J B w p C O U 1 4 C G T F K i e W o I J p K Z r I i M s c R E m 7 L K p g R 3 + c u r p F 2 v u V e 1 + s N l p X F b 1 F G C U z i D K r h w D Q 2 4 h y a 0 g M A U n u E V 3 q z M e r H e r Y / F 6 J p V 7 J z A H 1 i f P 0 C o k 2 k = < / l a t e x i t > y (1)

< l a t e x i t s h a 1 _ b a s e 6 4 = " 7 P et al., 2017) and MobileNetV2 (Sandler et al., 2018) , show a significant drop for PTQ or even result in random performance.

J E p 3 d N A 0 G E j 7 z W j I J F 3 s j C X E g = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q N O Y e i E e C R Y w g r W R B r b d D 7 E e + 0 G W z p 6 y a v 1 i N r A r T s 2 Z A 6 0 S t y A V K N A c 2 F / 9 Y U S S k A p N O F a q 5 z q x 9 j I s N S O c z s r 9 R N E Y k w k e 0 Z 6 h A o d U e d k 8 + Q y d G 2 W I g k i a J z S a q 7 8 3 M h w q l Y a + m c x z q m U v F / / z e o k O b r y M i T j R V J D F o S D h S E c o r w E N m a R E 8 9 Q Q T C Q z W R E Z Y 4 m J N m W V T Q n u 8 p d X S b t e c 6 9 q 9 Y f L S u O 2 q K M E p 3 A G V X D h G h p w D 0 1 o A Y E p P M M r v F m Z 9 W K 9 W x + L 0 T W r 2 D m B P 7 A + f w B C L p N q < / l a t e x i t > y (2) < l a t e x i t s h a 1 _ b a s e 6 4 = " t f E V Q C Q s f m Y q A U f L b 1 y I L P H p K N c = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s p u E f V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J u 1 7 z L m v 1 + 4 t K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z V i P j g = = < / l a t e x i t > f (•) < l a t e x i t s h a 1 _ b a s e 6 4 = " t f E V Q C Q s f m Y q A U f L b 1 y I L P H p K N c = " > A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s p u E f V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J u 1 7 z L m v 1 + 4 t K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z V i P j g = = < / l a t e x i t > f (•)

A solution to overcome such imbalances without the need to use per-channel quantization is introduced by Nagel et al. (2019) . A similar approach was introduced in concurrent work by Meller et al. (2019) . In both papers, the authors observe that for many common activation functions (e.g., ReLU, PreLU), a positive scaling equivariance holds:

f (sx) = sf (x)) for any non-negative real number s. This equivariance holds for any homogeneous function of degree one and can be extended to also hold for any piece-wise linear function by scaling its parameterization (e.g. ReLU6). We can exploit this positive scaling equivariance in consecutive layers in neural networks. Given two layers, h = f (W (1) x + b (1) ) and y = f (W (2) h + b (2) ), through scaling equivariance we have that:

y = f (W (2) f (W (1) x + b (1) ) + b (2) ) = f (W (2) S f (S -1 W (1) x + S -1 b (1) ) + b (2) ) = f ( W (2) f ( W (1) x + b (1) ) + b (2) )) where S = diag(s) is a diagonal matrix with value S ii denoting the scaling factor s i for neuron i. This allows us to reparameterize our model with W (2) = W (2) S, W (1) = S -1 W (1) and b (1) = S -1 b (1) . In case of CNNs the scaling will be per-channel and broadcast accordingly over the spatial dimensions. We illustrate this rescaling procedure in figure 6 .

To make the model more robust to quantization, we can find a scaling factor s i such that the quantization noise in the rescaled layers is minimal. ( Nagel et al., 2019) achieves this by equalizing dynamic ranges across consecutive layers. They prove that an optimal weight equalization is achieved by setting S such that:

s i = 1 r (2) i r (1) i r (2) i (22)

where r

(j)

i is the dynamic range of channel i of weight tensor j. The algorithm of Meller et al. ( 2019) introduces a similar scaling factor that also takes the intermediate activation tensor into account. However, they do not have a proof of optimality for this approach.

Absorbing high biases Nagel et al. (2019) further notice that in some cases, especially after CLE, high biases can lead to differences in the dynamic ranges of the activations. Therefore, they propose a procedure to, if possible, absorb high biases into the next layer. To absorb c from layer one (followed by a ReLU activation function f ) into layer two, we can do the following reparameterization:  and b (1) = b (1)c. In step two, we use the fact that for a layer with ReLU function f , there is a non-negative vector c such that r(Wx + bc) = r(Wx + b)c. The trivial solution c = 0 holds for all x. However, depending on the distribution of x and the values of W and b, there can be some values c i > 0 for which this equality holds for (almost) all x in the empirical distribution. This value is equal to

y = W (2) h + b (2) = W (2) (f (W (1) x + b (1) ) + c -c) + b (2) = W (2) (f (W (1) x + b (1) ) + c) + b (2) = W (2) h + b (2) (23) where b (2) = W (2) c + b (2) , h = h -c,

c i = max 0, min x W (1) i x + b (1) i . ()

where min x is evaluated on a small calibration dataset. To remove dependence on data, the authors propose to estimate the right hand side of ( 24) by the shift and scale parameters of the batch normalization layer which results3 in c i = max(0, β i -3γ i ).

Experiments In table 3 , we demonstrate the effect of CLE and bias absorption for quantizing MobileNetV2 to 8-bit. As skip connections break the equivariance between layers, we apply cross-layer equalization only to the layers within each residual block. Similar to Krishnamoorthi (2018) , we observe that model performance is close to random when quantizing MobileNetV2 to INT8. Applying CLE brings us back within 2% of FP32 performance, close to the performance of per-channel quantization. We note that absorbing high biases results in a small drop in FP32 performance, as it is an approximation, but it boosts quantized performance by 1% due to more precise activation quantization. Together, CLE and bias absorption followed by per-tensor quantization yield better results than per-channel quantization.

3.3 Bias correction

Another common issue is that quantization error is often biased. This means that the expected output of the original and quantized layer or network is shifted (E [Wx] = E Wx ). This kind of error is more pronounced in depth-wise separable layers with only a few elements per output channel (usually 9 for a 3 × 3 kernel). The main contributor to this error is often the clipping error, as a few strongly clipped outliers will likely lead to a shift in the expected distribution.

Several papers (Nagel et al., 2019; Meller et al., 2019; Finkelstein et al., 2019) noted this issue and introduce methods to correct for the expected shift in distribution. For a quantized layer W with quantization error ∆W = W -W, the expected output distribution is

E [ y] = E Wx = E [(W + ∆W)x] = E [Wx] + E [∆Wx] . ()

Thus the biased error is given by E

[∆Wx]. Since ∆W is constant, we have that E [∆Wx] = ∆W E [x]. In case ∆W E [x]

is nonzero, the output distribution is shifted. To counteract this shift we can substract it from the output:

E [y corr ] = E Wx -∆W E [x] = E [y] .Note, this correction term is a vector with the same shape as the bias and can thus be absorbed into the bias without any additional overhead at inference time. There are several ways of calculating the bias correction term, the two most common of which are empirical bias correction and analytic bias correction.

Empirical bias correction If we have access to a calibration dataset the bias correction term can simply be calculated by comparing the activations of the quantized and full precision model. In practice, this can be done layer-wise by computing

∆W E [x] = E Wx -E [Wx] .Analytic bias correction Nagel et al. (2019) introduce a method to analytically calculate the biased error, without the need for data. For common networks with batch normalization and ReLU functions, they use the BN statistics of the preceding layer in order to compute the expected input distribution E [x]. The BN parameters γ and β correspond to the mean and standard deviation of the BN layers output. Assuming input values are normally distributied, the effect of ReLU on the distribution can be modeled using the clipped normal distribution. They show that

E [x] = E [ReLU (x pre )] = γN -β γ + β 1 -Φ -β γ (28)

where x pre is the pre-activation output, which is assumed to be normally distributed with the perchannel means β and per-channel standard deviations γ, Φ(•) is the standard normal CDF, and the notation N (x) is used to denote the standard normal PDF. Note, all vector operations are elementwise (per-channel) operations. After calculating the input distribution E [x], the correction term can be simply derived by multiplying it with the weight quantization error ∆W.

Experiments In table 4 , we demonstrate the effect of bias correction for quantizing MobileNetV2 to 8-bit. Applying analytical bias correction improves quantized model performance from random to over 50%, indicating that the biased error introduced by quantization significantly harms model performance. When combining bias correction with CLE, we see that both techniques are complementary. Together, they achieve near FP32 performance without using any data.

3.4 AdaRound

Neural network weights are usually quantized by projecting each FP32 value to the nearest quantization grid point, as indicated by • in equation ( 4) for a uniform quantization grid. We refer to this quantization strategy as rounding-to-nearest. The rounding-to-nearest strategy is motivated by the fact that, for a fixed quantization grid, it yields the lowest MSE between the floating-point and quantized weights. However, Nagel et al. (2020) showed that rounding-to-nearest is not optimal in terms of the task loss when quantizing weights in the post-training regime. To illustrate this the authors quantized the weights of the first layer of ResNet18 to 4 bits using 100 different stochastic rounding samples (Gupta et al., 2015) and evaluated the performance of the network for each rounding choice. The best rounding choice among these outperformed rounding-to-nearest by more than 10%. Figure 7 illustrates this by plotting the performance of these rounding choices on the y-axis. In this section, we describe AdaRound (Nagel et al., 2020) , a systematic approach to finding good weight rounding choices for PTQ. AdaRound is a theoretically well-founded and computationally efficient method that shows significant performance improvement in practice.

As the main goal is to minimize the impact of quantization on the final task loss, we start by formulating the optimization problem in terms of this loss

arg min ∆w E [L (x, y, w + ∆w) -L (x, y, w)] ()

where ∆w denotes the perturbation due to quantization and can take two possible values for each weight, one by rounding the weight up and the other by rounding the weight down. We want to solve this binary optimization problem efficiently. As a first step, we approximate the cost function using a second-order Taylor series expansion. This alleviates the need for performance evaluation for each new rounding choice during the optimization. We further assume that the model has converged, implying that the contribution of the gradient term in the approximation can be ignored, and that the Hessian is block-diagonal, which ignores cross-layer correlations. This leads to the following Hessian based quadratic unconstrained binary optimization (QUBO) problem

arg min ∆w ( ) E ∆w ( ) T H (w ( ) ) ∆w ( )The clear correlation in figure 7 between the validation accuracy and objective of equation ( 30) indicates that the latter serves as a good proxy for the task loss (equation 29), even for 4-bit weight quantization. Despite the performance gains (see table 5 ), equation ( 30) cannot be widely applied for weight rounding for main two reasons:

• The memory and computational complexity of calculating the Hessian is impractical for general use-cases. Table 5 : Impact of various approximations and assumptions made in section 3.4 on the ImageNet validation accuracy (%) for ResNet18 averaged over 5 runs. N/A implies that the corresponding experiment was computationally infeasible.

• The QUBO problem of equation ( 30) is NP-Hard.

To tackle the first problem, the authors introduced additional suitable assumptions that allow simplifying the objective of equation ( 30) to the following local optimization problem that minimizes the MSE of the output activations for a layer.

arg min ∆W ( ) k,: E ∆W ( ) k,: x ( -1) 2Equation ( 31) requires neither the computation of the Hessian nor any other backward or forward propagation information from the subsequent layers. Note that the approximations and the analysis that have been used to link the QUBO problem of equation ( 30) with the local optimization problem of equation ( 31) is independent of the rounding problem. Hence this analysis also benefits the design of algorithms for other problems, including model compression and NAS (Moons et al., 2020) .

The optimization of ( 31) is still an NP-hard optimization problem. To find a good approximate solution with reasonable computational complexity, the authors relax the optimization problem to the following continuous optimization problem

arg min V Wx -Wx 2 F + λf reg (V) ,where • 2 F denotes the Frobenius norm and W are the soft-quantized weights defined as

W = s • clamp W s + h (V) ; n, p .We use n and p to denote integer grid limits, n = q min /s and p = q max /s. V i,j is the continuous variable that we optimize over and h can be any monotonic function with values between 0 and 1, i.e., h (V i,j ) ∈ [0, 1]. In Nagel et al. (2020) , the authors use a rectified sigmoid as h. The objective of (32) also introduces a regularizer term that encourages the continuous optimization variables h (V i,j ) to converge to either 0 or 1, so that they are valid solutions to the discrete optimization in (31). The regularizer used in Nagel et al. (2020) is

f reg (V) = i,j 1 -|2h (V i,j ) -1| β , ()

where β is annealed during the course of optimization to initially allow free movement of h (V i,j ) and later to force them to converge to 0 or 1. To avoid error accumulation across layers of the neural network and to account for the non-linearity, the authors propose the following final optimization problem

arg min V f a (Wx) -f a Wx 2 F + λf reg (V) ,where x is the layer's input with all preceding layers quantized and f a is the activation function. The objective of (35) can be effectively and efficiently optimized using stochastic gradient descent. This approach of optimizing weight rounding is known as AdaRound.

To summarize, the way we round weights during the quantization operation has a significant impact on the performance of the network. AdaRound provides a theoretically sound, computationally fast weight rounding method. It requires only a small amount of unlabeled data samples, no hyperparameter tuning or end-to-end finetuning, and can be applied to fully connected and convolutional layers of any neural network. 

3.5 Standard PTQ pipeline

In this section, we present a best-practice pipeline for PTQ based on relevant literature and extensive experimentation. We illustrate the recommended pipeline in figure 8 . This pipeline achieves competitive PTQ results for many computer vision as well as natural language processing models and tasks.

Depending on the model, some steps might not be required, or other choices could lead to equal or better performance.

Cross-layer equalization First we apply cross-layer equalization (CLE), which is a pre-processing step for the full precision model to make it more quantization friendly. CLE is particularly important for models with depth-wise separable layers and for per-tensor quantization, but it often also shows improvements for other layers and quantization choices. Add quantizers Next we choose our quantizers and add quantization operations in our network as described in section 2.3. The choice of quantizer might depend on the specific target HW; for common AI accelerators we recommend using symmetric quantizers for the weights and asymmetric quantizers for the activations. If supported by the HW/SW stack then it is favorable to use per-channel quantization for weights. Weight range setting To set the quantization parameters of all weight tensors we recommend using the layer-wise MSE based criteria. In the specific case of per-channel quantization, using the min-max method can be favorable in some cases. AdaRound In case we have a small calibration dataset4 available we next apply AdaRound in order to optimize the rounding of the weights. This step is crucial to enable low-bit weight quantization (e.g. 4 bits) in the PTQ. Bias correction In case we do not have such a calibration dataset and the network uses batch normalization, we can use analytical bias correction instead. Activation range setting As the final step, we determine the quantization ranges of all datadependent tensors in the network (i.e., activations). We use the MSE based criteria for most of the layers, which requires a small calibration set to find the minimum MSE loss. Alternatively, we can use the BN based range setting to have a fully data-free pipeline.

3.6 Experiments

We now evaluate the performance of the aforementioned PTQ pipeline on common computer vision and natural language understanding applications. Our results are summarized in table 6. For the task of semantic segmentation, we evaluate DeepLabV3 (with a MobileNetV2 backbone) (Chen et al., 2017) on Pascal VOC and for object detection, EfficientDet (Tan et al., 2020) on COCO 2017. The rest of the computer vision models are evaluated on the ImageNet classification benchmark.

For natural language understanding, we evaluate BERT-base on the GLUE benchmark (Wang et al., 2018) .

In all cases, we observe that 8-bit quantization of weights and activation (W8A8) leads to only marginal loss of accuracy compared to floating-point (within 0.7%) for all models. For W8A8 . We evaluate all models on the respective validation sets.

Higher is better in all cases. † A few quantized activations are kept in higher precision (16 bits).

quantization we also see no significant gains from using per-channel quantization. However, the picture changes when weights are quantized to 4 bits (W4A8). For ResNet18/50 and InceptionV3 the accuracy drop is still within 1% of floating-point for both per-tensor and per-channel quantization. However, for more efficient networks, such as MobileNetV2 and EfficientNet lite, the drop increases to 2.5% and 4.2% respectively for per-tensor quantization. This is likely due to the quantization of the depth-wise separable convolutions. Here, per-channel quantization can show a significant benefit, for example, in EfficientNet lite per-channel quantization increases the accuracy by 2.8% compared to per-tensor quantization, bringing it within 1.4% of full-precision accuracy. We see similar effects for EfficientDet-D1 and DeeplabV3 which both uses depth-wise separable convolutions in their backbone.

For BERT-base, we observe that a few activation tensors have extreme differences in their dynamic ranges. To make PTQ still work, we identified these layers using our debugging procedure outlined in section 3.7 and kept them in 16 bit. Otherwise BERT-base follows similar trends as most other models and our PTQ pipeline allows 4 bit weight quantization within 1.5% drop in GLUE score.

3.7 Debugging

We showed that the standard PTQ pipeline can achieve competitive results for a wide range of models and networks. However, if after following the steps of our pipeline, the model's performance is still not satisfactory, we recommend a set of diagnostics steps to identify the bottlenecks and improve the performance. While this is not strictly an algorithm, these debugging steps can provide insights on why a quantized model underperforms and help to tackle the underlying issues. These steps are shown as a flow chart in figure 9 and are described in more detail below:

FP32 sanity check An important initial debugging step is to ensure that the floating-point and quantized model behave similarly in the forward pass, especially when using custom quantization pipelines. Set the quantized model bit-width to 32 bits for both weights and activation, or by-pass the quantization operation, if possible, and check that the accuracy matches that of the FP32 model.

Weights or activations quantization

The next debugging step is to identify how activation or weight quantization impact the performance independently. Does performance recover if all weights are quantized to a higher bit-width while activations are kept in a lower bitwidth, or conversely if all activations use a high bit-width and activations a low bit-width? This step can show the relative contribution of activations and weight quantization to the overall performance drop and point us towards the appropriate solution.

Fixing weight quantization If the previous step shows that weight quantization does cause significant accuracy drop, then there are a few solutions to try:

• Apply CLE if not already implemented, especially for models with depth-wise separable convolutions.

• Try per-channel quantization. This will address the issue of uneven per-channel weight distribution.

• Apply bias correction or AdaRound if calibration data is available.

Fixing activation quantization To reduce the quantization error from activation quantization, we can also try using different range setting methods or adjust CLE to take activation quantization ranges into account, as vanilla CLE can lead to uneven activation distribution.

Per-layer analysis If the global solutions have not restored accuracy to acceptable levels, we consider each quantizer individually. We set each quantizer sequentially, to the target bit-width while keeping the rest of the network to 32 bits (see inner for loop in figure 9 ).

Visualizing layers If the quantization of a individual tensor leads to significant accuracy drop, we recommended visualizing the tensor distribution at different granularities, e.g. per-channel as in figure 5 , and dimensions, e.g., per-token or per-embedding for activations in BERT.

Fixing individual quantizers The visualization step can reveal the source of the tensor's sensitivity to quantization. Some common solutions involve custom range setting for this quantizer or allowing a higher bit-width for problematic quantizer, e.g., BERT-base from table 6 . If the problem is fixed and the accuracy recovers, we continue to the next quantizer. If not, we may have to resort to other methods, such as quantization-aware training (QAT), which is discussed in section 4.

After completing the above steps, the last step is to quantize the complete model to the desired bit-width. If the accuracy is acceptable, we have our final quantized model ready to use. Otherwise, we can consider higher bit-widths and smaller granularities or revert to more powerful quantization methods, such as quantization-aware training.

4 Quantization-aware training

The post-training quantization techniques described in the previous section are the first go-to tool in our quantization toolkit. They are very effective and fast to implement because they do not require retraining of the network with labeled data. However, they have limitations, especially when aiming for low-bit quantization of activations, such as 4-bit and below. Post-training techniques may not be enough to mitigate the large quantization error incurred by low-bit quantization. In these cases, we resort to quantization-aware training (QAT). QAT models the quantization noise source (see section 2.3) during training. This allows the model to find more optimal solutions than post-training quantization. However, the higher accuracy comes with the usual costs of neural network training, i.e., longer training times, need for labeled data and hyper-parameter search.

In this section, we explore how back-propagation works in networks with simulated quantization and provide a standard pipeline for training models with QAT effectively. We will also discuss the implications of batch normalization folding and per-channel quantization in QAT and provide results for a wide range of tasks and models.

4.1 Simulating quantization for backward path

In section 2.3, we saw how quantization can be simulated using floating-point in deep learning frameworks. However, if we look at the computational graph of figure 4 , to train such a network we need to back-propagate through the simulated quantizer block. This poses an issue because the gradient of the round-to-nearest operation in equation ( 4) is either zero or undefined everywhere, which makes gradient-based training impossible. A way around this would be to approximate the gradient using the straight-through estimator (STE, Bengio et al. 2013) , which approximates the gradient of the rounding operator as 1:

∂ y ∂y = 1Using this approximation we can now calculate the gradient of the quantization operation from equation ( 7). For clarity we assume symmetric quantization, namely z = 0, but the same result applies to asymmetric quantization since the zero-point is a constant. We use n and p to define the integer grid limits, such that n = q min /s and p = q max /s. The gradient of equation ( 7) w.r.t its input,

x i , is given by:

∂ x i ∂x i = ∂q(x i ) ∂x i = s • ∂ ∂x i clamp x i s ; n, p + 0 =                s • ∂ x i /s ∂(x i /s) ∂(x i /s) ∂x i if q min ≤ x i ≤ q max , s • ∂n ∂x i if x i < q min , s • ∂p ∂x i if x i > q max , = 1 if q min ≤ x i ≤ q max , 0 otherwise. ()

Using this gradient definition we can now back-propagate through the quantization blocks. Figure 10 shows a simple computational graph for the forward and backward pass used in quantization-aware training. The forward pass is identical to that of figure 4 , but in the backward pass we effectively skip the quantizer block due to the STE assumption. In earlier QAT work the quantization ranges for weights and activations were updated at each iteration most commonly using the min-max range (Krishnamoorthi, 2018) . In later work (Esser et al., 2020; Jain et al., 2019; Bhalgat et al., 2020) , the STE is used to calculate the gradient w.r.t. the quantization parameters, z and s. Using the chain rule and the STE, we first calculate the gradient w.r.t. the scale-factor:

∂ x i ∂s = ∂ ∂s s • clamp x i s ; n, p =    -x i /s + x i /s if q min ≤ x i ≤ q max , n if x i < q min , p if x i > q max .Originally, we restricted the zero-point to be an integer. To make zero-point learnable we convert into a real number and apply the rounding operator. The modified quantization function is defined as:

x = q(x; s, z) = s • clamp x s + z ; n, p -zThe gradient w.r.t. to z is calculated by applying the STE once again to the rounding operator: Note that in some QAT literature, the BN-folding effect is ignored. While this is fine when we employ per-channel quantization (more below in this section), keeping BN unfolded for per-tensor quantization will result in one of the two following cases:

∂ x i ∂z = 0 q min ≤ x i ≤ q max , -s otherwise. (1. The BN layer applies per-channel rescaling during inference. In this case we might as well use per-channel quantization in the first place. 2. We fold BN during deployment into the weight tensor and incur potentially significant accuracy drop as we trained the network to adapt to a different quantization noise.

A simple but effective approach to modeling BN-folding in QAT is to statically fold the BN scale and offset into the linear layer's weights and bias, as we saw in equations ( 11) and ( 12). This corresponds to re-parametrization of the weights and effectively removes the batch normalization operation from the network entirely. When starting from a converged pre-trained model, static folding is very effective, as we can see from the result of table 7 .

An alternative approach by Jacob et al. (2018) both updates the running statistics during QAT and applies BN-folding using a correction. This approach is more cumbersome and computationally costly because it involves a double forward pass: one for the batch-statistics and one for the quantized linear operation. However, based on our experiments (see table 7 ), static-folding performs on par or better despite its simplicity.

Per-channel quantization In section 2.4.2, we mentioned that per-channel quantization of the weights can improve accuracy when it is supported by hardware. The static folding re-parametrization is also valid for per-channel quantization. However, per-channel quantization provides additional flexibility as it allows us to absorb the batch normalization scaling operation into the per-channel scale-factor. Let us see how this is possible by revisiting the BN folding equation from section 2.3.1, but this time introduce per-channel quantization of the weights, such that W k,: = q(W k,: ; s w,k ) = s w,k W int k,: . By applying batch normalization to the output of a linear layer similar to equation (10), we get:

y k = BatchNorm W k,: x = γ k W k,: σ 2 k + x + β k - γ k µ k σ 2 k + = γ k s w,k σ 2 k + W int k,: x + b k = s w,k W int k,: x + b kWe can see that it is now possible to absorb the batch normalization scaling parameters into the per-channel scale-factor. For this modeling approach is on par or better for per-channel quantization compared to static folding as we can see from the last two rows of table 7.

4.3 Initialization for QAT

In this section, we will explore the effect of initialization for QAT. It is common practice in literature to start from a pre-trained FP32 model (Esser et al., 2020; Krishnamoorthi, 2018; Jacob et al., 2018; Jain et al., 2019) . While it is clear that starting from an FP32 model is beneficial, the effect of the quantization initialization on the final QAT result is less studied. Here we explore the effect of using several of our PTQ techniques as an initial step before doing QAT.

Effect of range estimation

To assess the effect of the initial range setting (see section 3.1) for weights and activations, we perform two sets of experiments, which are summarized in table 8. In the first experiment, we quantize the weights to 4-bits and keep the activations in 8-bits. We compare the min-max initialization with the MSE based initialization for the weights quantization range. While the MSE initialized model has a significantly higher starting accuracy, the gap closes after training for 20 epochs.

To explore the same effect for activation quantization, we perform a similar experiment, where we now quantize the activation to 4-bits and compare min-max initialization with MSE based initialization.

The observations from weight range initialization hold here as well. In figure 11 we show the full training curve of this experiment. In the first few epochs, there is a significant advantage for using MSE initialization, which almost vanishes in the later stage of training. In conclusion, a better initialization can lead to better QAT results, but the gain is usually small and vanishes the longer the training lasts. In conclusion, for models that have severe issues with plain PTQ we may need advanced PTQ techniques such as CLE to initialize QAT. In most other cases, an improved PTQ initialization leads only to a minor improvement in the final QAT performance.

4.4 Standard QAT pipeline

In this section, we present a best-practice pipeline for QAT based on relevant literature and extensive experimentation. We illustrate the recommended pipeline in figure 12 . This pipeline yields good QAT results over a variety of computer vision and natural language processing models and tasks, and can be seen as the go-to tool for achieving low-bit quantization performance. As discussed in previous sections, we always start from a pre-trained model and follow some PTQ steps in order to have faster convergence and higher accuracy.

Cross-layer equalization Similar to PTQ, we first apply CLE to the full precision model. As we saw in table 9, this step is necessary for models that suffer from imbalanced weight distributions, such as MobileNet architectures. For other networks or in the case of perchannel quantization this step can be optional. Add quantizers Next, we choose our quantizers and add quantization operations in our network as described in section 2.3. The choice for quantizer might depend on the specific target HW, for common AI accelerators we recommend using symmetric quantizers for the weights and asymmetric quantizers for the activations. If supported by the HW/SW stack, then it is favorable to use per-channel quantization for weights. At this stage we will also take care that our simulation of batch normalization is correct, as discussed in section 4.2. Range estimation Before training we have to initialize all quantization parameters. A better initialization will help faster training and might improve the final accuracy, though often the improvement is small (see table 8 ). In general, we recommend to set all quantization parameters using the layer-wise MSE based criteria. In the specific case of per-channel quantization, using the min-max setting can sometimes be favorable. Learnable Quantization Parameters We recommend making the quantizer paramaters learnable, as discussed in section 4.1. Learning the quantization parameters directly, rather than updating them at every epoch, leads to higher performance especially when dealing with low-bit quantization. However, using learnable quantizers requires special care when setting up the optimizer for the task. When using SGD-type optimizers, the learning rate for the quantization parameters needs to be reduced compared to the rest of the network parameters.

The learning rate adjustment can be avoided if we use optimizers with adaptive learning rates such as Adam or RMSProp.

4.5 Experiments

Using our QAT pipeline, we quantize and evaluate the same models we used for PTQ in section 3. Table 10 : Performance (average over 3 runs) of our standard QAT pipeline for various models and tasks. DeeplabV3 (MobileNetV2 backbone) is evaluated on Pascal VOC (mean intersection over union), EfficientDet-D1 on COCO 2017 (mean average precision), BERT-base on the GLUE benchmark and all other models on ImageNet (accuracy). We evaluate all models on the respective validation sets. Higher is better in all cases.

corresponding GLUE tasks for 3 to 12 epochs depending on the task and the quantization granularity. We use the Adam optimizer for all models. We present the results with the best learning rate per quantization configuration and perform no further hyper-parameter tuning.

We observe that for networks without depth-wise separable convolutions (first 3 rows of table 10 ), W8A8 and W4A8 quantization perform on par with and even outperform the floating-point model in certain cases. This could be due to the regularizing effect of training with quantization noise or due to the additional fine-tuning during QAT. For the more aggressive W4A4 case, we notice a small drop but still within 1% of the floating-point accuracy.

Quantizing networks with depth-wise separable layers (MobileNetV2, EfficientNet lite, DeeplabV3, EfficientDet-D1) is more challenging; a trend we also observed from the PTQ results in section 3.6 and discussed in the literature (Chin et al., 2020; Sheng et al., 2018a) . Whereas 8-bit quantization incurs close to no accuracy drop, quantizing weights to 4 bits leads to a larger drop, e.g. approximately 4% drop for EfficientNet lite with per-tensor quantization. Per-channel quantization can improve performance significantly bringing DeepLabV3 to floating-point accuracy and reducing the gap of MobileNetV2 and EfficientNet lite to less than 1.5%. Quantizing both weights and activations to 4-bits remains a challenging for such networks, even with per-channel quantization it can lead to a drop of up to 5%. EfficientDet-D1 remains more difficult to quantize than the other networks in this group.

For BERT-base we observe that QAT with range learning can efficiently deal with the high dynamic ranges allowing to keep all activations in 8 bits (unlike for PTQ). W4A8 stays within 1% of the original GLUE score, indicating that low bit weight quantization is not a problem for transformer models. We only notice a significant drop in performance when combining this with low bit activation quantization (W4A4).

5 Summary and Conclusions

Deep learning has become an integral part of many machine learning applications and can now be found in countless electronic devices and services, from smartphones and home appliances to drones, robots and self-driving cars. As the popularity and reach of deep learning in our everyday life increases, so does the need for fast and power-efficient neural network inference. Neural network quantization is one of the most effective ways of reducing the energy and latency requirements of neural networks during inference.

Quantization allows us to move from floating-point representations to a fixed-point format and, in combination with dedicated hardware utilizing efficient fixed-point operations, has the potential to achieve significant power gains and accelerate inference. However, to exploit these savings, we require robust quantization methods that can maintain high accuracy, while reducing the bit-width of weights and activations. To this end, we consider two main classes of quantization algorithms: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).

Post-training quantization techniques take a pre-trained FP32 networks and convert it into a fixedpoint network without the need for the original training pipeline. This makes them a lightweight, push-button approach to quantization with low engineering effort and computational cost. We describe a series of recent advances in PTQ and introduce a PTQ pipeline that leads to near floating-point accuracy results for a wide range of models and machine learning tasks. In particular, using the proposed pipeline we can achieve 8-bit quantization of weights and activations within only 1% of the floating-point accuracy for all networks. We further show that many networks can be quantized even to 4-bit weights with only a small additional drop in performance. In addition, we introduce a debugging workflow to effectively identify and fix problems that might occur when quantizing new networks.

Quantization-aware training models the quantization noise during training through simulated quantization operations. This training procedure allows for better solutions to be found compared to PTQ while enabling more effective and aggressive activation quantization. Similar to PTQ, we introduce a standard training pipeline utilizing the latest algorithms in the field. We also pay special attention to batch normalization folding during QAT and show that simple static folding outperforms other more computationally expensive approaches. We demonstrate that with our QAT pipeline we can achieve 4-bit quantization of weights, and for some models even 4-bit activations, with only a small drop of accuracy compared to floating-point.

The choice between PTQ and QAT depends on the accuracy and power requirements of the application. Both approaches are an essential part of any model efficiency toolkit and we hope that our proposed pipelines will help engineers deploy high-performing quantized models with less time and effort.



Assuming x is normally distributed, the equality will hold for approximately 99.865% of the inputs.

Usually, between

and 1000 unlabeled images are sufficient as a calibration set.

References

[1]Banner R, Nahshan Y, Soudry D. Post training 4-bit quantization of convolutional networks for rapid-deployment. Neural Information Processing Systems (NeuRIPS) 2019;:.

[2]Bengio Y, Léonard N, Courville A. Estimating or propagating gradients through stochastic neurons for conditional computation.  2013;:.

[3]Bhalgat Y, Lee J, Nagel M, Blankevoort T, Kwak N. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 1920;:.

[4]Chen L.-C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation.  2017;:.

[5]Chin T.-W, Chuang PI, Chandra -J, Marculescu V,  D. One weight bitwidth to rule them all. Computer Vision -ECCV 2020 Workshops 2020;:85--103.

[6]Dong Z, Yao Z, Gholami A, Mahoney MW, Keutzer K. HAWQ: hessian aware quantization of neural networks with mixed-precision. International Conference on Computer Vision (ICCV) 2019;:.

[7]Esser SK, Mckinstry JL, Bablani D, Appuswamy R, Modha DS. Learned step size quantization. International Conference on Learning Representations (ICLR) 2020;20:.

[8]Finkelstein A, Almog U, Grobman M. Fighting quantization bias with bias.  2019;:.

[9]Gupta S, Agrawal A, Gopalakrishnan K, Narayanan P. Deep learning with limited numerical precision. International Conference on Machine Learning 2015;:.

[10]Horowitz . 1 computing's energy problem (and what we can do about it). 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) 2014;2:.

[11]Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, et al.. Efficient convolutional neural networks for mobile vision applications.  2017;:.

[12]Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning 2015;37:7--09.

[13]Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, et al.. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Conference on Computer Vision and Pattern Recognition (CVPR) 2006;21:.

[14]Jain SR, Gural A, Wu M, Dick C. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware.  2019;:.

[15]Krishnamoorthi R. Quantizing deep convolutional networks for efficient inference: A whitepaper.  1920;21:.

[16]Meller E, Finkelstein A, Almog U, Grobman M. Same, same but different: Recovering neural network quantization error through weight factorization. Proceedings of the 36th International Conference on Machine Learning, ICML 2019;12:.

[17]Moons B, Noorzad P, Skliar A, Mariani G, Mehta D, Lott C, et al.. Distilling optimal neural networks: Rapid search in diverse spaces.  2020;:.

[18]Nagel M, Van Baalen M, Blankevoort T, Welling M. Data-free quantization through weight equalization and bias correction. International Conference on Computer Vision (ICCV) 2019;12:.

[19]Nagel M, Amjad RA, Van Baalen M, Louizos C, Blankevoort T. Up or down? Adaptive rounding for post-training quantization. Proceedings of the 37th International Conference on Machine Learning 2020;119:.

[20]Nascimento MG D, Fawcett R, Prisacariu VA, Dsconv . Efficient convolution operator. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2005;:.

[21]Ramachandran P, Zoph B, Le QV. Searching for activation functions.  2017;:.

[22]Rouhani B, Lo D, Zhao R, Liu M, Fowers J, Ovtcharov K, et al.. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Neural Information Processing Systems 2005;:.

[23]Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition 2018;:4510--4520.

[24]Sheng T, Feng C, Zhuo S, Zhang X, Shen L, Aleksic M. A quantization-friendly separable convolution for mobilenets. 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), Mar 2018a.  None;:.

[25]Sheng T, Feng C, Zhuo S, Zhang X, Shen L, Aleksic M. A quantization-friendly separable convolution for mobilenets. 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), 2018b None;:.

[26]Stock P, Joulin A, Gribonval R, Graham B, Jégou H. And the bit goes down: Revisiting the quantization of neural networks.  2019;:.

[27]Tan M, Pang R, Le QV. Efficientdet: Scalable and efficient object detection.  2020;:.

[28]Uhlich S, Mauch L, Cardinaux F, Yoshiyama K, Garcia JA, Tiedemann S, et al.. Mixed precision dnns: All you need is a good parametrization. International Conference on Learning Representations 2020;:.

[29]Van Baalen M, Louizos C, Nagel M, Amjad RA, Wang Y, Blankevoort T, et al.. Bayesian bits: Unifying quantization and pruning. Advances in Neural Information Processing Systems 2020;33:5741--5752.

[30]Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 2018;:18--5446.


A White Paper on Neural Network Quantization
Date Published: 2021-06-15
Authors:
Markus  Nagel, markusn@qti.qualcomm.com
A I Qualcomm
  Research, Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc
Marios  Fournarakis
Rana Ali Amjad, ramjad@qti.qualcomm.com
Yelysei  Bondarenko
Mart  Van Baalen
Tijmen  Blankevoort, tijmen@qti.qualcomm.com
  Requantization
  # # !,$ # !,% # #,! # #,# # #,$ # #
Abstract
While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.
1 Introduction
With the rise in popularity of deep learning as a general-purpose tool to inject intelligence into electronic devices, the necessity for small, low-latency and energy efficient neural networks solutions has increased. Today neural networks can be found in many electronic devices and services, from smartphones, smart glasses and home appliances, to drones, robots and self-driving cars. These devices are typically subject to strict time restrictions on the execution of neural networks or stringent power requirements for long-duration performance.
One of the most impactful ways to decrease the computational time and energy consumption of neural networks is quantization. In neural network quantization, the weights and activation tensors are stored in lower bit precision than the 16 or 32-bit precision they are usually trained in. When moving from 32 to 8 bits, the memory overhead of storing tensors decreases by a factor of 4 while the computational cost for matrix multiplication reduces quadratically by a factor of 16. Neural networks have been shown to be robust to quantization, meaning they can be quantized to lower bit-widths with a relatively small impact on the network's accuracy. Besides, neural network quantization can often be applied along with other common methods for neural network optimization, such as neural architecture search, compression and pruning. It is an essential step in the model efficiency pipeline for any practical use-case of deep learning. However, neural network quantization is not free. Low bit-width quantization introduces noise to the network that can lead to a drop in accuracy. While some networks are robust to this noise, other networks require extra work to exploit the benefits of quantization.
In this white paper, we introduce the state-of-the-art in neural network quantization. We start with an introduction to quantization and discuss hardware and practical considerations. We then consider two different regimes of quantizing neural networks: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ methods, discussed in section 3, take a trained network and quantize it with little or no data, requires minimal hyperparameter tuning and no end-to-end training. This makes them a push-button approach to quantizing neural networks with low engineering effort and computational cost. In contrast, QAT, discussed in section 4, relies on retraining the neural networks with simulated quantization in the training pipeline. While this requires more effort in training and potentially hyperparameter tuning, it generally further closes the gap to the full-precision accuracy compared to PTQ for low-bit quantization. For both regimes, we introduce standard pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common computer vision and natural language processing models. We also propose a debugging workflow to identify and address common issues when quantizing a new model.
2 Quantization fundamentals
In this section, we introduce the basic principles of neural network quantization and of fixed-point accelerators on which quantized networks run on. We start with a hardware motivation and then introduce standard quantization schemes and their properties. Later we discuss practical considerations related to layers commonly found in modern neural networks and their implications for fixed-point accelerators.
2.1 Hardware background
Before diving into the technical details, we first explore the hardware background of quantization and how it enables efficient inference on device. Figure 1 provides a schematic overview of how a matrix-vector multiplication, y = Wx + b, is calculated in a neural network (NN) accelerator. This is the building block of larger matrix-matrix multiplications and convolutions found in neural networks. Such hardware blocks aim at improving the efficiency of NN inference by performing as many calculations as possible in parallel. The two fundamental components of this NN accelerator are the processing elements C n,m and the accumulators A n . Our toy example in figure 1 has 16 processing elements arranged in a square grid and 4 accumulators. The calculation starts by loading the accumulators with the bias value b n . We then load the weight values W n,m and the input values x m into the array and compute their product in the respective processing elements C n,m = W n,m x m in a single cycle. Their results are then added in the accumulators:
A n = b n + m C n,mThe above operation is also referred to as Multiply-Accumulate (MAC). This step is repeated many times for larger matrix-vector multiplications. Once all cycles are completed, the values in the accumulators are then moved back to memory to be used in the next neural network layer. Neural networks are commonly trained using FP32 weights and activations. If we were to perform inference in FP32, the processing elements and the accumulator would have to support floating-point logic, and we would need to transfer the 32-bit data from memory to the processing units. MAC operations and data transfer consume the bulk of the energy spent during neural network inference. Hence, significant benefits can be achieved by using a lower bit fixed-point or quantized representation for these quantities. Low-bit fixed-point representations, such as INT8, not only reduce the amount data transfer but also the size and energy consumption of the MAC operation (Horowitz, 2014) . This is because the cost of digital arithmetic typically scales linearly to quadratically with the number of bits used and because fixed-point addition is more efficient than its floating-point counterpart (Horowitz, 2014) .
3 Confidential and Proprietary -Qualcomm Technologies, Inc. and/or its affiliated companies. Source sample text
W !,! W !,# W !,$ W !,% x ! x # x $ x % # !,! # !,# # !,$ # !,% # #,! # #,# # #,$ # #,% # $,! # $,# # $,$ # $,% # %,! # %,# # %,$ # %,% $ ! $ # $ $ $ % Input values Weight values Accumulators W #,! W #,# W #,$ W #,% W $,! W $,# W $,$ W $,% W %,! W %,# W %,$ W %,%
Figure 1 : A schematic overview of matrix-multiply logic in neural network accelerator hardware.
To move from floating-point to the efficient fixed-point operations, we need a scheme for converting floating-point vectors to integers. A floating-point vector x can be expressed approximately as a scalar multiplied by a vector of integer values:
x = s x • x int ≈ xwhere s x is a floating-point scale factor and x int is an integer vector, e.g., INT8. We denote this quantized version of the vector as x. By quantizing the weights and activations we can write the quantized version of the accumulation equation:
Ân = b n + m W n,m x m = b n + m s w W int n,m s x x int m = b n + s w s x m W int n,m x int m (3)
Note that we used a separate scale factor for weights, s w , and activations, s x . This provides flexibility and reduces the quantization error (more in section 2.2). Since each scale factor is applied to the whole tensor, this scheme allows us to factor the scale factors out of the summation in equation ( 3) and perform MAC operations in fixed-point format. We intentionally ignore bias quantization for now, because the bias is normally stored in higher bit-width (32-bits) and its scale factor depends on that of the weights and activations (Jacob et al., 2018) .
Figure 2 shows how the neural network accelerator changes when we introduce quantization. In our example, we use INT8 arithmetic, but this could be any quantization format for the sake of this discussion. It is important to maintain a higher bit-width for the accumulators, typical 32-bits wide. Otherwise, we risk incurring loss due to overflow as more products are accumulated during the computation.
The activations stored in the 32-bit accumulators need to be written to memory before they can be used by the next layer. To reduce data transfer and the complexity of the next layer's operations, these activations are quantized back to INT8. This requires a requantization step which is shown in figure 2 .
2.2 Uniform affine quantization
In this section we define the quantization scheme that we will use in this paper. This scheme is called uniform quantization and it is the most commonly used quantization scheme because it permits efficient implementation of fixed-point arithmetic.
Figure 2 : A schematic of matrix-multiply logic in an neural network accelerator for quantized inference.
Uniform affine quantization, also known as asymmetric quantization, is defined by three quantization parameters: the scale factor s, the zero-point z and the bit-width b. The scale factor and the zero-point are used to to map a floating point value to the integer grid, whose size depends on the bit-width. The scale factor is commonly represented as a floating-point number and specifies the step-size of the quantizer. The zero-point is an integer that ensures that real zero is quantized without error. This is important to ensure that common operations like zero padding or ReLU do not induce quantization error.
Once the three quantization parameters are defined we can proceed with the quantization operation.
Starting from a real-valued vector x we first map it to the unsigned integer grid {0, . . . , 2 b -1}:
x int = clamp x s + z; 0, 2 b -1 , ()
where • is the round-to-nearest operator and clamping is defined as:
clamp (x; a, c) =    a, x  c.
(5)
To approximate the real-valued input x we perfrom a de-quantization step:
x ≈ x = s (x int -z)Combining the two steps above we can provide a general definition for the quantization function, q(•), as:
x = q(x; s, z, b) = s clamp x s + z; 0, 2 b -1 -z ,Through the de-quantization step, we can also define the quantization grid limits (q min , q max ) where q min = -sz and q max = s(2 b -1z). Any values of x that lie outside this range will be clipped to its limits, incurring a clipping error. If we want to reduce the clipping error we can expand the quantization range by increasing the scale factor. However, increasing the scale factor leads to increased rounding error as the rounding error lies in the range -1 2 s, 1 2 s . In section 3.1, we explore in more detail how to choose the quantization parameters to achieve the right trade-off between clipping and rounding errors.
2.2.1 Symmetric uniform quantization
Symmetric quantization is a simplified version of the general asymmetric case. The symmetric quantizer restricts the zero-point to 0. This reduces the computational overhead of dealing with zero-point offset during the accumulation operation in equation (3). But the lack of offset restricts the mapping between integer and floating-point domain. As a result, the choice of signed or unsigned integer grid matters:
x = s x int (8a)
x int = clamp x s ; 0, 2 b -1 for unsigned integers (8b) x int = clamp x s ; -2 b-1 , 2 b-1 -1 for signed integers (8c)
Unsigned symmetric quantization is well suited for one-tailed distributions, such as ReLU activations (see figure 3 ). On the other hand, signed symmetric quantization can be chosen for distributions that are roughly symmetric about zero.
13 Confidential and Proprietary -Qualcomm Technologies, Inc. and/or its affiliated companies.
0 0 0  A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K k k p 6 r H g x W M L 9 g P a U D b b S b t 2 s w m 7 G 6 G E / g I v H h T x 6 k / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a 1 n G q G L Z Y L G L V D a h G w S W 2 D D c C u 4 l C G g U C O 8 H k b u 5 3 n l B p H s s H M 0 3 Q j + h I 8 p A z a q z U 1 I N S 2 a 2 4 C 5 B 1 4 u W k D D k a g 9 J X f x i z N E J p m K B a 9 z w 3 M X 5 G l e F M 4 K z Y T z U m l E 3 o C H u W S h q h 9 r P F o T N y a Z U h C W N l S x q y U H 9 P Z D T S e h o F t j O i Z q x X v b n 4 n 9 d L T X j r Z 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T d G G 4 K 2 + v E 7 a 1 Y p 3 X a k 2 a + V 6 L Y + j A O d w A V f g w Q 3 U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 Y N J 5 8 5 g z 9 w P n 8 A 3 L m M 8 A = =  s  A A A B + X i c b V B N S 8 N A E N 3 U r 1 q / o h 6 9 L B b B U 0 m K q M e C F 4 8 V 7 Q e 0 o W w 2 k 3 b p 7 i b s b g o l 9 J 9 4 8 a C I V / + J N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Y c q Z N p 7 3 7 Z Q 2 N r e 2 d 8 q 7 l b 3 9 g 8 M j 9 / i k r Z N M U W j R h C e q G x I N n E l o G W Y 4 d F M F R I Q c O u H 4 b u 5 3 J q A 0 S + S T m a Y Q C D K U L G a U G C s N X P d x K g Q Y x S j W b C g h G r h V r + Y t g N e J X 5 A q K t A c u F / 9 K K G Z A G k o J 1 r 3 f C 8 1 Q U 6 U Y Z T D r N L P N K S E j s k Q e p Z K I k A H + e L y G b 6 w S o T j R N m S B i / U 3 x M 5 E V p P R W g 7 B T E j v e r N x f + 8 X m b i 2 y B n M s 0 M S L p c F G c c m w T P Y 8 A R U 0 A N n 1 p C q G L 2 V k x H R B F q b F g V G 4 K / + v I 6 a d d r / n W t / l C v N q 6 K O M r o D J 2 j S + S j G 9 R A 9 6 i J W o i i C X p G r + j N y Z 0 X 5 9 3 5 W L a W n G L m F P 2 B 8 / k D n H a T m Q = =  Symmetric signed  A A A C C H i c b V D L S s N A F J 3 4 r P V V d e n C w S K 4 K k k p 2 m X B j c s K 9 g F N C J P J p B 0 6 e T B z I y 0 h S z f + i h s X i r j 1 E 9 z 5 N 0 7 b L L T 1 w M D h n H u Z e 4 6 X C K 7 A N L + N t f W N z a 3 t 0 k 5 5 d 2 / / 4 L B y d N x V c S o p 6 9 B Y x L L v E c U E j 1 g H O A j W T y Q j o S d Y z x v f z P z e A 5 O K x 9 E 9 T B P m h G Q Y 8 Y B T A l p y K 2 f K p n 4 M 2 A 4 J j L w g m + R u Z g O b Q M Y j a O a 5 W 6 m a N X M O v E q s g l R R g b Z b + b L 9 m K Y h i 4 A K o t T A M h N w M i K B U 8 H y s p 0 q l h A 6 J k M 2 0 D Q i I V N O N g + S 4 w u t + D i I p X 4 R 4 L n 6 e y M j o V L T 0 N O T s 3 v V s j c T / / M G K Q R N R 0 d K U m A R X X w U p A J D j G e t Y J 9 L R k F M N S F U c n 0 r p i M i C Q X d X V m X Y C 1 H X i X d e s 2 6 q t X v G t V W o 6 i j h E 7 R O b p E F r p G L X S L 2 q i D K H p E z + g V v R l P x o v x b n w s R t e M Y u c E / Y H x + Q M c w p q k  s • x int8  A A A C C X i c b V D L S s N A F J 3 U V 6 2 v q k s 3 g 0 V w V Z J S t M u C G 5 c V 7 A O a E i a T S T t 0 M g k z N 9 I S s n X j r 7 h x o Y h b / 8 C d f + P 0 s d D W A x c O 5 9 z L v f f 4 i e A a b P v b K m x s b m 3 v F H d L e / s H h 0 f l 4 5 O O j l N F W Z v G I l Y 9 n 2 g m u G R t 4 C B Y L 1 G M R L 5 g X X 9 8 M / O 7 D 0 x p H s t 7 m C Z s E J G h 5 C G n B I z k l b F 2 a R A D d i M C I z / M J r m X u c A m k K V c Q i P P v X L F r t p z 4 H X i L E k F L d H y y l 9 u E N M 0 Y h K o I F r 3 H T u B Q U Y U c C p Y X n J T z R J C x 2 T I + o Z K E j E 9 y O a f 5 P j C K A E O Y 2 V K A p 6 r v y c y E m k 9 j X z T O T t Y r 3 o z 8 T + v n 0 L Y G G R c J i k w S R e L w l R g i P E s F h x w x S i I q S G E K m 5 u x X R E F K F g w i u Z E J z V l 9 d J p 1 Z 1 r q q 1 u 3 q l W V / G U U R n 6 B x d I g d d o y a 6 R S 3 U R h Q 9 o m f 0 i t 6 s J + v F e r c + F q 0 F a z l z i v 7 A + v w B A G + b I w = =  s • x uint8 0 255  A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K k k p 6 r H g x W M L 9 g P a U D b b S b t 2 s w m 7 G 6 G E / g I v H h T x 6 k / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a 1 n G q G L Z Y L G L V D a h G w S W 2 D D c C u 4 l C G g U C O 8 H k b u 5 3 n l B p H s s H M 0 3 Q j + h I 8 p A z a q z U 1 I N S 2 a 2 4 C 5 B 1 4 u W k D D k a g 9 J X f x i z N E J p m K B a 9 z w 3 M X 5 G l e F M 4 K z Y T z U m l E 3 o C H u W S h q h 9 r P F o T N y a Z U h C W N l S x q y U H 9 P Z D T S e h o F t j O i Z q x X v b n 4 n 9 d L T X j r Z 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T d G G 4 K 2 + v E 7 a 1 Y p 3 X a k 2 a + V 6 L Y + j A O d w A V f g w Q 3 U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 Y N J 5 8 5 g z 9 w P n 8 A 3 L m M 8 A = =  s  A A A B + 3 i c b V B N S 8 N A E N 3 4 W e t X r E c v i 0 X w V J I i 6 r H g x W N F + w F t K J v N t F 2 6 u w m 7 G z G E / h U v H h T x 6 h / x 5 r 9 x 2 + a g r Q 8 G H u / N M D M v T D j T x v O + n b X 1 j c 2 t 7 d J O e X d v / + D Q P a q 0 d Z w q C i 0 a 8 1 h 1 Q 6 K B M w k t w w y H b q K A i J B D J 5 z c z P z O I y j N Y v l g s g Q C Q U a S D R k l x k o D t 3 K f C Q F G M Y p T q d l I Q j R w q 1 7 N m w O v E r 8 g V V S g O X C / + l F M U w H S U E 6 0 7 v l e Y o K c K M M o h 2 m 5 n 2 p I C J 2 Q E f Q s l U S A D v L 5 7 V N 8 Z p U I D 2 N l S x o 8 V 3 9 P 5 E R o n Y n Q d g p i x n r Z m 4 n / e b 3 U D K + D n M k k N S D p Y t E w 5 d j E e B Y E j p g C a n h m C a G K 2 V s x H R N F q L F x l W 0 I / v L L q 6 R d r / m X t f p d v d q 4 K O I o o R N 0 i s 6 R j 6 5 Q A 9 2 i J m o h i p 7 Q M 3 p F b 8 7 U e X H e n Y 9 F 6 5 p T z B y j P 3 A + f w B M Z Z S Q  Symmetric unsigned  A A A B 8 X i c b V D L S g M x F M 3 U V 6 2 v q k s 3 w S K 4 K j N F 1 G X F j c s K 9 o H t U D L p n T Y 0 k x m S O 0 I Z + h d u X C j i 1 r 9 x 5 9 + Y t r P Q 1 g O B w z n 3 k n N P k E h h 0 H W / n c L a + s b m V n G 7 t L O 7 t 3 9 Q P j x q m T j V H J o 8 l r H u B M y A F A q a K F B C J 9 H A o k B C O x j f z v z 2 E 2 g j Y v W A k w T 8 i A 2 V C A V n a K X H G z O J I k A t e L 9 c c a v u H H S V e D m p k B y N f v m r N 4 h 5 G o F C L p k x X c 9 N 0 M + Y R s E l T E u 9 1 E D C + J g N o W u p Y h E Y P 5 s n n t I z q w x o G G v 7 F N K 5 + n s j Y 5 G x 0 Q I 7 G T E c m W V v J v 7 n d V M M r / 1 M q C R F U H z x U Z h K i j G d n U 8 H Q g N H O b G E c S 1 s V s p H T D O O t q S S L c F b P n m V t G p V 7 7 J a u 6 9 V 6 h d 5 H U V y Q k 7 J O f H I F a m T O 9 I g T c K J I s / k l b w 5 x n l x 3 p 2 P x W j B y X e O y R 8 4 n z / K F p D z  Asymmetric 0 255  A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K k k p 6 r H g x W M L 9 g P a U D b b S b t 2 s w m 7 G 6 G E / g I v H h T x 6 k / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n Y 3 N r e 2 d 3 c J e c f / g 8 O i 4 d H L a 1 n G q G L Z Y L G L V D a h G w S W 2 D D c C u 4 l C G g U C O 8 H k b u 5 3 n l B p H s s H M 0 3 Q j + h I 8 p A z a q z U 1 I N S 2 a 2 4 C 5 B 1 4 u W k D D k a g 9 J X f x i z N E J p m K B a 9 z w 3 M X 5 G l e F M 4 K z Y T z U m l E 3 o C H u W S h q h 9 r P F o T N y a Z U h C W N l S x q y U H 9 P Z D T S e h o F t j O i Z q x X v b n 4 n 9 d L T X j r Z 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T d G G 4 K 2 + v E 7 a 1 Y p 3 X a k 2 a + V 6 L Y + j A O d w A V f g w Q 3 U 4 R 4 a 0 A I G C M / w C m / O o / P i v D s f y 9 Y N J 5 8 5 g z 9 w P n 8 A 3 L m M 8 A = =  s  A A A B 9 X i c b V B N S w M x E M 3 6 W e t X 1 a O X Y B G 8 W H Z L U S 9 C w Y v H C v Y D 2 r V k 0 2 w b m m S X Z F a t S / + H F w + K e P W / e P P f m L Z 7 0 N Y H A 4 / 3 Z p i Z F 8 S C G 3 D d b 2 d p e W V 1 b T 2 3 k d / c 2 t 7 Z L e z t N 0 y U a M r q N B K R b g X E M M E V q w M H w V q x Z k Q G g j W D 4 d X E b 9 4 z b X i k b m E U M 1 + S v u I h p w S s d N c B 9 g i p 5 G p 8 e W q e u o W i W 3 K n w I v E y 0 g R Z a h 1 C 1 + d X k Q T y R R Q Q Y x p e 2 4 M f k o 0 c C r Y O N 9 J D I s J H Z I + a 1 u q i G T G T 6 d X j / G x V X o 4 j L Q t B X i q / p 5 I i T R m J A P b K Q k M z L w 3 E f / z 2 g m E F 3 7 K V Z w A U 3 S 2 K E w E h g h P I s A 9 r h k F M b K E U M 3 t r Z g O i C Y U b F B 5 G 4 I 3 / / I i a Z R L 3 l m p f F M p V i t Z H D l 0 i I 7 Q C f L Q O a q i a 1 R D d U S R R s / o F b 0 5 D 8 6 L 8 + 5 8 z F q X n G z m A P 2 B 8 / k D 2 i G S s w = =  min = sz  A A A B 6 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e w W U Y 8 F L x 4 r 2 g 9 o l 5 J N s 2 1 o k l 2 S r F i W / g Q v H h T x 6 i / y 5 r 8 x b f e g r Q 8 G H u / N M D M v T A Q 3 1 v O + U W F t f W N z q 7 h d 2 t n d 2 z 8 o H x 6 1 T J x q y p o 0 F r H u h M Q w w R V r W m 4 F 6 y S a E R k K 1 g 7 H N z O / / c i 0 4 b F 6 s J O E B Z I M F Y 8 4 J d Z J 9 5 I 8 9 c s V r + r N g V e J n 5 M K 5 G j 0 y 1 + 9 Q U x T y Z S l g h j T 9 b 3 E B h n R l l P B p q V e a l h C 6 J g M W d d R R S Q z Q T Y / d Y r P n D L A U a x d K Y v n 6 u + J j E h j J j J 0 n Z L Y k V n 2 Z u J / X j e 1 0 X W Q c Z W k l i m 6 W B S l A t s Y z / 7 G A 6 4 Z t W L i C K G a u 1 s x H R F N q H X p l F w I / v L L q 6 R V q / q X 1 d p d r V K / y O M o w g m c w j n 4 c A V 1 u I U G N I H C E J 7 h F d 6 Q Q C / o H X 0 s W g s o n z m G P 0 C f P 2 c I j d U =  max  A A A B 6 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e w W U Y 8 F L x 4 r 2 g 9 o l 5 J N s 2 1 o k l 2 S r F i W / g Q v H h T x 6 i / y 5 r 8 x b f e g r Q 8 G H u / N M D M v T A Q 3 1 v O + U W F t f W N z q 7 h d 2 t n d 2 z 8 o H x 6 1 T J x q y p o 0 F r H u h M Q w w R V r W m 4 F 6 y S a E R k K 1 g 7 H N z O / / c i 0 4 b F 6 s J O E B Z I M F Y 8 4 J d Z J 9 5 I 8 9 c s V r + r N g V e J n 5 M K 5 G j 0 y 1 + 9 Q U x T y Z S l g h j T 9 b 3 E B h n R l l P B p q V e a l h C 6 J g M W d d R R S Q z Q T Y / d Y r P n D L A U a x d K Y v n 6 u + J j E h j J j J 0 n Z L Y k V n 2 Z u J / X j e 1 0 X W Q c Z W k l i m 6 W B S l A t s Y z / 7 G A 6 4 Z t W L i C K G a u 1 s x H R F N q H X p l F w I / v L L q 6 R V q / q X 1 d p d r V K / y O M o w g m c w j n 4 c A V 1 u I U G N I H C E J 7 h F d 6 Q Q C / o H X 0 s W g s o n z m G P 0 C f P 2 c I j d U =  max  A A A B 6 n i c b V B N S w M x E J 3 U r 1 q / q h 6 9 B I v g q e w W U Y 8 F L x 4 r 2 g 9 o l 5 J N s 2 1 o k l 2 S r F i W / g Q v H h T x 6 i / y 5 r 8 x b f e g r Q 8 G H u / N M D M v T A Q 3 1 v O + U W F t f W N z q 7 h d 2 t n d 2 z 8 o H x 6 1 T J x q y p o 0 F r H u h M Q w w R V r W m 4 F 6 y S a E R k K 1 g 7 H N z O / / c i 0 4 b F 6 s J O E B Z I M F Y 8 4 J d Z J 9 5 I 8 9 c s V r + r N g V e J n 5 M K 5 G j 0 y 1 + 9 Q U x T y Z S l g h j T 9 b 3 E B h n R l l P B p q V e a l h C 6 J g M W d d R R S Q z Q T Y / d Y r P n D L A U a x d K Y v n 6 u + J j E h j J j J 0 n Z L Y k V n 2 Z u J / X j e 1 0 X W Q c Z W k l i m 6 W B S l A t s Y z / 7 G A 6 4 Z t W L i C K G a u 1 s x H R F N q H X p l F w I / v L L q 6 R V q / q X 1 d p d r V K / y O M o w g m c w j n 4 c A V 1 u I U G N I H C E J 7 h F d 6 Q Q C / o H X 0 s W g s o n z m G P 0 C f P 2 c I j d U =  max DUPLICATE 2 (less high) 0 127 -128  A A A C C X i c b V C 7 S g N B F J 2 N r x h f q 5 Y 2 g 0 G I h W E 3 B E 0 Z s L G M Y B 6 Q h D A 7 m U 2 G z M 4 u M 3 c l c d n W x l + x s V D E 1 j + w 8 2 + c P A p N P H D h c M 6 9 3 H u P F w m u w X G + r c z a + s b m V n Y 7 t 7 O 7 t 3 9 g H x 4 1 d B g r y u o 0 F K F q e U Q z w S W r A w f B W p F i J P A E a 3 q j 6 6 n f v G d K 8 1 D e w S R i 3 Y A M J P c 5 J W C k n o 0 1 L n Q C A k P P T 8 Z p L + k A G 0 M S c w m V N M U X D + c 9 O + 8 U n R n w K n E X J I 8 W q P X s r 0 4 / p H H A J F B B t G 6 7 T g T d h C j g V L A 0 1 4 k 1 i w g d k Q F r G y p J w H Q 3 m X 2 S 4 j O j 9 L E f K l M S 8 E z 9 P Z G Q Q O t J 4 J n O 6 d F 6 2 Z u K / 3 n t G P x K N + E y i o F J O l / k x w J D i K e x 4 D 5 X j I K Y G E K o 4 u Z W T I d E E Q o m v J w J w V 1 + e Z U 0 S k X 3 s l i 6 L e e r 5 U U c W X S C T l E B u e g K V d E N q q E 6 o u g R P a N X 9 G Y 9 W S / W u / U x b 8 1 Y i 5 l j 9 A f W 5 w + Q I Z o 1  s(x uint8 z)
Figure 3 : A visual explanation of the different uniform quantization grids for a bit-width of 8. s is the scaling factor, z the zero-point. The floating-point grid is in black, the integer quantized grid in blue.
2.2.2 Power-of-two quantizer
Power-of-two quantization is a special case of symmetric quantization, in which the scale factor is restricted to a power-of-two, s = 2 -k . This choice can bring hardware efficiencies because scaling with s corresponds to simple bit-shifting. However, the restricted expressiveness of the scale factor can complicate the trade-off between rounding and clipping error.
2.2.3 Quantization granularity
So far, we have defined a single set of quantization parameters (quantizer) per tensor, one for the weights and one for activations, as seen in equation ( 3). This is called per-tensor quantization. We can also define a separate quantizer for individual segments of a tensor (e.g., output channels of a weight tensor), thus increasing the quantization granularity. In neural network quantization, per-tensor quantization is the the most common choice of granularity due to its simpler hardware implementation: all accumulators in equation ( 3) use the same scale factor, s w s x . However, we could use finer granularity to further improve performance. For example, for weight tensors, we can specify a different quantizer per output channel. This is known as per-channel quantization and its implications are discussed in more detailed in section 2.4.2.
Other works go beyond per-channel quantization parameters and apply separate quantizers per group of weights or activations (Rouhani et al., 2020; Stock et al., 2019; Nascimento et al., 2019) . Increasing the granularity of the groups generally improves accuracy at the cost of some extra overhead. The overhead is associated with accumulators handling sums of values with varying scale factors. Most existing fixed-point accelerators do not currently support such logic and for this reason, we will not consider them in this work. However, as research in this area grows, more hardware support for these methods can be expected in the future.
2.3 Quantization simulation
To test how well a neural network would run on a quantized device, we often simulate the quantized behavior on the same general purpose hardware we use for training neural networks. This is called quantization simulation. We aim to approximate fixed-point operations using floating-point hardware. Such simulations are significantly easier to implement compared to running experiments on actual quantized hardware or using quantized kernels. They allow the user to efficiently test various quantization options and it enables GPU acceleration for quantization-aware training as described in section 4. In this section, we first explain the fundamentals of this simulation process and then discuss techniques that help to reduce the difference between the simulated and the actual on-device performance.
Previously, we saw how matrix-vector multiplication is calculated in dedicated fixed-point hardware.
In figure 4a , we generalize this process for a convolutional layer, but we also include an activation function to make it more realistic. During on-device inference, all the inputs (biases, weight and input activations) to the hardware are in a fixed-point format. However, when we simulate quantization using common deep learning frameworks and general-purpose hardware these quantities are in floating-point. This is why we introduce quantizer blocks in the compute graph to induce quantization effects.
Figure 4b shows how the same convolutional layer is modelled in a deep-learning framework. Quantizer blocks are added in between the weights and the convolution to simulate weight quantization, and after the activation function to simulate activation quantization. The bias is often not quantized because it is stored in higher-precision. In section 2.3.2, we discuss in more detail when it is appropriate to position the quantizer after the non-linearity. The quantizer block implements the quantization function of equation ( 7) and each quantizer is defined by a set of quantization parameters (scale factor, zero-point, bit-width). Both the input and output of the quantizer are in floating-point format but the output lies on the quantization grid.
2.3.1 Batch normalization folding
Batch normalization (Ioffe & Szegedy, 2015) is a standard component of modern convolutional networks. Batch normalization normalizes the output of a linear layer before scaling and adding an offset (see equation 9). For on-device inference, these operations are folded into the previous or next linear layers in a step called batch normalization folding (Krishnamoorthi, 2018; Jacob et al., 2018) . This removes the batch normalization operations entirely from the network, as the calculations are absorbed into an adjacent linear layer. Besides reducing the computational overhead of the additional scaling and offset, this prevents extra data movement and the quantization of the layer's output. More formally, during inference, batch normalization is defined as an affine map of the output x:
BatchNorm(x) = γ x -µ √ σ 2 + + β (9)
where µ and σ are the mean and variance computed during training as exponential moving average over batch-statistics, and γ and β are learned affine hyper-parameters per-channel. If batch normalization is applied right after a linear layer y = BatchNorm(Wx), we can rewrite the terms such that the batch normalization operation is fused with the linear layer itself. Assuming a weight matrix W ∈ R n×m we apply batch normalization to each output y k for k = {1, . . . , n}:
y k = BatchNorm(W k,: x) = γ k W k,: x -µ k σ 2 k + + β k = γ k W k,: σ 2 k + x + β k - γ k µ k σ 2 k + = W k,: x + b k (10)
where:
W k,: = γ k W k,: σ 2 k + , () b k = β k - γ k µ k σ 2 k + .2.3.2 Activation function fusing
In our naive quantized accelerator introduced in section 2.1, we saw that the requantization of activations happens after the matrix multiplication or convolutional output values are calculated. However, in practice, we often have a non-linearity directly following the linear operation. It would be wasteful to write the linear layer's activations to memory, and then load them back into a compute core to apply a non-linearity. For this reason, many hardware solutions come with a hardware unit that applies the non-linearity before the requantization step. If this is the case, we only have to simulate requantization that happens after the non-linearity. For example, ReLU non-linearities are readily modelled by the requantization block, as you can just set the minimum representable value of that activation quantization to 0.
Other more complex activation functions, such as sigmoid or Swish (Ramachandran et al., 2017) , require more dedicated support. If this support is not available, we need to add a quantization step before and after the non-linearity in the graph. This can have a big impact on the accuracy of quantized model. Although newer activations like Swish functions provide accuracy improvement in floating-point, these may vanish after quantization or may be less efficient to deploy on fixed-point hardware.
2.3.3 Other layers and quantization
There are many other types of layers being used in neural networks. How these are modeled depends greatly on the specific hardware implementation. Sometimes the mismatch between simulated quantization and on-target performance is down to layers not being properly quantized. Here, we provide some guidance on how to simulate quantization for a few commonly used layers:
Max pooling Activation quantization is not required because the input and output values are on the same quantization grid. Average pooling The average of integers is not necessarily an integer. For this reason, a quantization step is required after average-pooling. However, we use the same quantizer for the inputs and outputs as the quantization range does not significantly change. Element-wise addition Despite its simple nature, this operation is difficult to simulate accurately.
During addition, the quantization ranges of both inputs have to match exactly. If these ranges do not match, extra care is needed to make addition work as intended. There is no single accepted solution for this but adding a requantization step can simulate the added noise coarsely. Another approach is to optimize the network by tying the quantization grids of the inputs. This would prevent the requantization step but may require fine-tuning. Concatenation The two branches that are being concatenated generally do not share the same quantization parameters. This means that their quantization grids may not overlap making a requantization step necessary. As with element-wise addition, it is possible to optimize your network to have shared quantization parameters for the branches being concatenated.
2.4 Practical considerations
When quantizing neural networks with multiple layers, we are confronted with a large space of quantization choices including the quantization scheme, granularity, and bit-width. In this section, we explore some of the practical considerations that help reduce the search space.
Note that in this white paper we only consider homogeneous bit-width. This means that the bit-width chosen for either weights or activations remains constant across all layers. Homogeneous bit-width is more universally supported by hardware but some recent works also explore the implementation of heterogeneous bit-width or mixed-precision (van Baalen et al., 2020; Dong et al., 2019; Uhlich et al., 2020) .
2.4.1 Symmetric vs. asymmetric quantization
For each weight and activation quantization, we have to choose a quantization scheme. On one hand, asymmetric quantization is more expressive because there is an extra offset parameter, but on the other hand there is a possible computational overhead. To see why this is the case, consider what happens when asymmetric weights, W = s w (W int -z w ), are multiplied with asymmetric activations x = s x (x intz x ):
W x = s w (W int -z w )s x (x int -z x ) = s w s x W int x int -s w z w s x x int -s w s x z x W int + s w z w s x z x . ()
The first term is what we would have if both operations were in symmetric format. The third and fourth terms depend only on the scale, offset and weight values, which are known in advance. Thus these two terms can be pre-computed and added to the bias term of a layer at virtually no cost. The second term, however, depends on the input data x. This means that for each batch of data we need to compute an additional term during inference. This can lead to significant overhead in both latency and power, as it is equivalent to adding an extra channel.
For this reason, it is a common approach to use asymmetric activation quantization and symmetric weight quantization that avoids the additional data-dependent term.
2.4.2 Per-tensor and per-channel quantization
In section 2.2.3, we discussed different levels of quantization granularity. Per-tensor quantization of weights and activations has been standard for a while because it is supported by all fixed-point accelerators. However, per-channel quantization of the weights can improve accuracy, especially when the distribution of weights varies significantly from channel to channel. Looking back at the quantized MAC operation in equation ( 3), we can see that per-channel weight quantization can be implemented in the accelerator by applying a separate per-channel weight scale factor without requiring rescaling. Per-channel quantization of activations is much harder to implement because we cannot factor the scale factor out of the summation and would, therefore, require rescaling the accumulator for each input channel. Whereas per-channel quantization of weights is increasingly becoming common practice, not all commercial hardware supports it. Therefore, it is important to check if it is possible in your intended target device. be data-free or may require a small calibration set, which is often readily available. Additionally, having almost no hyperparameter tuning makes them usable via a single API call as a black-box method to quantize a pretrained neural network in a computationally efficient manner. This frees the neural network designer from having to be an expert in quantization and thus allows for a much wider application of neural network quantization.
3 Post-training quantization
A fundamental step in the PTQ process is finding good quantization ranges for each quantizer. We briefly discussed in section 2.2 how the choice of quantization range affects the quantization error. In this section, we start by discussing various common methods used in practice to find good quantization parameters. We then explore common issues observed during PTQ and introduce the most successful techniques to overcome them. Using these techniques we present a standard post-training quantization pipeline, which we find to work best in most common scenarios and, finally, we introduce a set of debugging steps to improve the performance of the quantized model.
3.1 Quantization range setting
Quantization range setting refers to the method of determining clipping thresholds of the quantization grid, q min and q max (see equation 7). The key trade-off in range setting is between clipping and rounding error, described in section 2.2, and their impact on the final task loss for each quantizer being configured. Each of the methods described here provides a different trade-off between the two quantities. These methods typically optimize local cost functions instead of the task loss. This is because in PTQ we aim for computationally fast methods without the need for end-to-end training.
Weights can usually be quantized without any need for calibration data. However, determining parameters for activation quantization often requires a few batches of calibration data.
Min-max To cover the whole dynamic range of the tensor, we can define the quantization parameters as follows
q min = min V, (14) q max = max V, ()
where V denotes the tensor to be quantized. This leads to no clipping error. However, this approach is sensitive to outliers as strong outliers may cause excessive rounding errors.
Mean squared error (MSE) One way to alleviate the issue of large outliers is to use MSE-based range setting. In this range setting method we find q min and q max that minimize the MSE between the original and the quantized tensor:
arg min qmin,qmax V -V(q min , q max ) 2 F ,where V(q min , q max ) denotes the quantized version of V and • F is the Frobenius norm. The optimization problem is commonly solved using grid search, golden section method or analytical approximations with closed-form solution (Banner et al., 2019) . Several variants of this range setting method exist in literature but they are all very similar in terms of objective function and optimization. Cross entropy For certain layers, all values in the tensor being quantized may not be equally important. One such scenario is the quantization of logits in the last layer of classification networks, in which it is important to preserve the order of the largest value after quantization. MSE may not be a suitable metric for this, as it weighs all the values in a tensor equally regardless of their order.
For a larger number of classes, we usually have a large number of small or negative logits that are unimportant for prediction accuracy and few larger values that matter. In this case, MSE would incur a large quantization error to the few larger important logits while trying to reduce the quantization error of the more populous smaller logits. In this specific case, it is beneficial to minimize the following cross-entropy loss function
arg min qmin,qmax H (ψ (v) , ψ (v(q min , q max )))where H(•, •) denotes the cross-entropy function, ψ is the softmax function, and v is the logits vector.
BN based range setting Range setting for activation quantizers often requires some calibration data. If a layer has batch-normalized activations, the per-channel mean and standard deviation of the activations are equal to the learned batch normalization shift and scale parameters, respectively. These can then be used to find suitable parameters for activation quantizer as follows (Nagel et al., 2019) :
q min = min (β -αγ) (18) q max = max (β + αγ)where β and γ are vectors of per-channel learned shift and scale parameters, and α > 0. Nagel et al. ( 2019) uses α = 6 so that only large outliers are clipped.
Comparison In table 1 we compare range setting methods for weight quantization. For high bitwidths, the MSE and min-max approaches are mostly on par. However, at lower bit-widths the MSE approach clearly outperforms the min-max. In table 2, we present a similar comparison for activation quantization. We note that MSE combined with cross-entropy for the last layer, denoted as MSE + Xent, outperforms other methods, especially at lower bit-widths. The table also clearly demonstrates the benefit of using cross-entropy for the last layer instead of the MSE objective.
3.2 Cross-Layer Equalization
A common issue for quantization error is that elements in the same tensor can have significantly different magnitudes. As discussed in the previous section, range setting for the quantization grid tries to find a good trade-off between clipping and rounding error. Unfortunately, in some cases, the difference in magnitude between them is so large that even for moderate quantization (e.g., INT8), we cannot find a suitable trade-off. Nagel et al. (2019) showed that this is especially prevalent in depth-wise separable layers since only a few weights are responsible for each output feature and this might result in higher variability of the weights. Further, they noted that batch normalization folding adds to this effect and can result in a strong imbalance between weights connected to various output channels (see figure 5 ). While the latter is less of an issue for a more fine-grained quantization granularity (e.g., per-channel quantization), this remains a big issue for the more widely used pertensor quantization. Several papers (Krishnamoorthi, 2018; Nagel et al., 2019; Sheng et al., 2018b) noted that efficient models with depth-wise separable convolutions, such as MobileNetV1 (Howard 
l u Y 9 Y G m C X 9 D x D N v 5 g 0 C s = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S a G E / I k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q G c x 9 U I 8 E i x g B G s j D W y 7 H 2 I 9 9 o O 0 k z 2 l V f c i G 9 g V p + b M g V a J W 5 A K F G g O 7 K / + M C J J S I U m H C v V c 5 1 Y e y m W m h F O s 3 I / U T T G Z I J H t G e o w C F V X j p P n q F z o w x R E E n z h E Z z 9 f d G i k O l Z q F v J v O c a t n L x f + 8 X q K D G y 9 l I k 4 0 F W R x K E g 4 0 h H K a 0 B D J i n R f G Y I J p K Z r I i M s c R E m 7 L K p g R 3 + c u r p F 2 v u V e 1 + s N l p X F b 1 F G C U z i D K r h w D Q 2 4 h y a 0 g M A U n u E V 3 q z U e r H e r Y / F 6 J p V 7 J z A H 1 i f P w w y k 0 c =  W (1)  A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S a G E / I k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q G c x 9 U I 8 E i x g B G s j D W y 7 H 2 I 9 9 o O 0 k z 2 l 1 f p F N r A r T s 2 Z A 6 0 S t y A V K N A c 2 F / 9 Y U S S k A p N O F a q 5 z q x 9 l I s N S O c Z u V + o m i M y Q S P a M 9 Q g U O q v H S e P E P n R h m i I J L m C Y 3 m 6 u + N F I d K z U L f T O Y 5 1 b K X i / 9 5 v U Q H N 1 7 K R J x o K s j i U J B w p C O U 1 4 C G T F K i + c w Q T C Q z W R E Z Y 4 m J N m W V T Q n u 8 p d X S b t e c 6 9 q 9 Y f L S u O 2 q K M E p 3 A G V X D h G h p w D 0 1 o A Y E p P M M r v F m p 9 W K 9 W x + L 0 T W r 2 D m B P 7 A + f w A N u J N I  W (2)  A A A B 8 n i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 2 K q M e i F 4 8 V 7 A e k o W w 2 m 3 b p Z j f s T o Q S + j O 8 e F D E q 7 / G m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B X c g O t + O 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / + g e n j U M S r T l L W p E k r 3 Q m K Y 4 J K 1 g Y N g v V Q z k o S C d c P x 3 c z v P j F t u J K P M E l Z k J C h 5 D G n B K z k 9 2 m k A H s X Z s A H 1 Z p b d + f A q 8 Q r S A 0 V a A 2 q X / 1 I 0 S x h E q g g x v i e m 0 K Q E w 2 c C j a t 9 D P D U k L H Z M h 8 S y V J m A n y + c l T f G a V C M d K 2 5 K A 5 + r v i Z w k x k y S 0 H Y m B E Z m 2 Z u J / 3 l + B v F N k H O Z Z s A k X S y K M 4 F B 4 d n / O O K a U R A T S w j V 3 N 6 K 6 Y h o Q s G m V L E h e M s v r 5 J O o + 5 d 1 R s P l 7 X m b R F H G Z 2 g U 3 S O P H S N m u g e t V A b U a T Q M 3 p F b w 4 4 L 8 6 7 8 7 F o L T n F z D H 6 A + f z B 2 f 8 k L A =  •1/s i  A A A B 8 X i c b V B N S 8 N A E N 3 U r 1 q / q h 6 9 L B b B U 0 m K q M e i F 4 8 V 7 A e 2 o W w 2 m 3 b p Z j f s T o Q S + i + 8 e F D E q / / G m / / G T Z u D t j 4 Y e L w 3 w 8 y 8 I B H c g O t + O 6 W 1 9 Y 3 N r f J 2 Z W d 3 b / + g e n j U M S r V l L W p E k r 3 A m K Y 4 J K 1 g Y N g v U Q z E g e C d Y P J b e 5 3 n 5 g 2 X M k H m C b M j 8 l I 8 o h T A l Z 6 H N B Q A T Z D X h l W a 2 7 d n Q O v E q 8 g N V S g N a x + D U J F 0 5 h J o I I Y 0 / f c B P y M a O B U s F l l k B q W E D o h I 9 a 3 V J K Y G T + b X z z D Z 1 Y J c a S 0 L Q l 4 r v 6 e y E h s z D Q O b G d M Y G y W v V z 8 z + u n E F 3 7 G Z d J C k z S x a I o F R g U z t / H I d e M g p h a Q q j m 9 l Z M x 0 Q T C j a k P A R v + e V V 0 m n U v c t 6 4 / 6 i 1 r w p 4 i i j E 3 S K z p G H r l A T 3 a E W a i O K J H p G r + j N M c 6 L 8 + 5 8 L F p L T j F z j P 7 A + f w B v m O Q U A = =  •s i  A A A B 8 X i c b V D L S g M x F L 3 j s 9 Z X 1 a W b Y B F c l Z k i 6 r L o x m U F + 8 C 2 l E x 6 p w 3 N Z I Y k I 5 a h f + H G h S J u / R t 3 / o 2 Z d h b a e i B w O O d e c u 7 x Y 8 G 1 c d 1 v Z 2 V 1 b X 1 j s 7 B V 3 N 7 Z 3 d s v H R w 2 d Z Q o h g 0 W i U i 1 f a p R c I k N w 4 3 A d q y Q h r 7 A l j + + y f z W I y r N I 3 l v J j H 2 Q j q U P O C M G i s 9 d E N q R n 6 Q P k 3 7 p b J b c W c g y 8 T L S R l y 1 P u l r + 4 g Y k m I 0 j B B t e 5 4 b m x 6 K V W G M 4 H T Y j f R G F M 2 p k P s W C p p i L q X z h J P y a l V B i S I l H 3 S k J n 6 e y O l o d a T 0 L e T W U K 9 6 G X i f 1 4 n M c F V L + U y T g x K N v 8 o S A Q x E c n O J w O u k B k x s Y Q y x W 1 W w k Z U U W Z s S U V b g r d 4 8 j J p V i v e R a V 6 d 1 6 u X e d 1 F O A Y T u A M P L i E G t x C H R r A Q M I z v M K b o 5 0 X 5 9 3 5 m I + u O P n O E f y B 8 / k D / 4 O R I w = =  x  A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q N O Y e i E e C R Y w g r W R B r b d D 7 E e + 0 G W z p 6 y q n s x G 9 g V p + b M g V a J W 5 A K F G g O 7 K / + M C J J S I U m H C v V c 5 1 Y e x m W m h F O Z + V + o m i M y Q S P a M 9 Q g U O q v G y e f I b O j T J E Q S T N E x r N 1 d 8 b G Q 6 V S k P f T O Y 5 1 b K X i / 9 5 v U Q H N 1 7 G R J x o K s j i U J B w p C O U 1 4 C G T F K i e W o I J p K Z r I i M s c R E m 7 L K p g R 3 + c u r p F 2 v u V e 1 + s N l p X F b 1 F G C U z i D K r h w D Q 2 4 h y a 0 g M A U n u E V 3 q z M e r H e r Y / F 6 J p V 7 J z A H 1 i f P 0 C o k 2 k =  y (1)
 A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B H q p i R F 1 G X R j c s K 9 g F t L J P p p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c D A 4 Z x 7 u W e O H 3 O m t O N 8 W 2 v r G 5 t b 2 6 W d 8 u 7 e / s G h f X T c V l E i C W 2 R i E e y 6 2 N F O R O 0 p Z n m t B t L i k O f 0 4 4 / u c v 9 z p R K x S L x q N O Y e i E e C R Y w g r W R B r b d D 7 E e + 0 G W z p 6 y a v 1 i N r A r T s 2 Z A 6 0 S t y A V K N A c 2 F / 9 Y U S S k A p N O F a q 5 z q x 9 j I s N S O c z s r 9 R N E Y k w k e 0 Z 6 h A o d U e d k 8 + Q y d G 2 W I g k i a J z S a q 7 8 3 M h w q l Y a + m c x z q m U v F / / z e o k O b r y M i T j R V J D F o S D h S E c o r w E N m a R E 8 9 Q Q T C Q z W R E Z Y 4 m J N m W V T Q n u 8 p d X S b t e c 6 9 q 9 Y f L S u O 2 q K M E p 3 A G V X D h G h p w D 0 1 o A Y E p P M M r v F m Z 9 W K 9 W x + L 0 T W r 2 D m B P 7 A + f w B C L p N q  y (2)  A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s p u E f V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J u 1 7 z L m v 1 + 4 t K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z V i P j g = =  f (•)  A A A B 7 3 i c b V B N S w M x E J 2 t X 7 V + V T 1 6 C R a h X s p u E f V Y 9 O K x g v 2 A d i n Z b L Y N z S Z r k h X K 0 j / h x Y M i X v 0 7 3 v w 3 p u 0 e t P X B w O O 9 G W b m B Q l n 2 r j u t 1 N Y W 9 / Y 3 C p u l 3 Z 2 9 / Y P y o d H b S 1 T R W i L S C 5 V N 8 C a c i Z o y z D D a T d R F M c B p 5 1 g f D v z O 0 9 U a S b F g 5 k k 1 I / x U L C I E W y s 1 I 2 q f R J K c z 4 o V 9 y a O w d a J V 5 O K p C j O S h / 9 U N J 0 p g K Q z j W u u e 5 i f E z r A w j n E 5 L / V T T B J M x H t K e p Q L H V P v Z / N 4 p O r N K i C K p b A m D 5 u r v i Q z H W k / i w H b G 2 I z 0 s j c T / / N 6 q Y m u / Y y J J D V U k M W i K O X I S D R 7 H o V M U W L 4 x B J M F L O 3 I j L C C h N j I y r Z E L z l l 1 d J u 1 7 z L m v 1 + 4 t K 4 y a P o w g n c A p V 8 O A K G n A H T W g B A Q 7 P 8 A p v z q P z 4 r w 7 H 4 v W g p P P H M M f O J 8 / Z V i P j g = =  f (•)
A solution to overcome such imbalances without the need to use per-channel quantization is introduced by Nagel et al. (2019) . A similar approach was introduced in concurrent work by Meller et al. (2019) . In both papers, the authors observe that for many common activation functions (e.g., ReLU, PreLU), a positive scaling equivariance holds:
f (sx) = sf (x)) for any non-negative real number s. This equivariance holds for any homogeneous function of degree one and can be extended to also hold for any piece-wise linear function by scaling its parameterization (e.g. ReLU6). We can exploit this positive scaling equivariance in consecutive layers in neural networks. Given two layers, h = f (W (1) x + b (1) ) and y = f (W (2) h + b (2) ), through scaling equivariance we have that:
y = f (W (2) f (W (1) x + b (1) ) + b (2) ) = f (W (2) S f (S -1 W (1) x + S -1 b (1) ) + b (2) ) = f ( W (2) f ( W (1) x + b (1) ) + b (2) )) where S = diag(s) is a diagonal matrix with value S ii denoting the scaling factor s i for neuron i. This allows us to reparameterize our model with W (2) = W (2) S, W (1) = S -1 W (1) and b (1) = S -1 b (1) . In case of CNNs the scaling will be per-channel and broadcast accordingly over the spatial dimensions. We illustrate this rescaling procedure in figure 6 .
To make the model more robust to quantization, we can find a scaling factor s i such that the quantization noise in the rescaled layers is minimal. ( Nagel et al., 2019) achieves this by equalizing dynamic ranges across consecutive layers. They prove that an optimal weight equalization is achieved by setting S such that:
s i = 1 r (2) i r (1) i r (2) i (22)
where r
(j)
i is the dynamic range of channel i of weight tensor j. The algorithm of Meller et al. ( 2019) introduces a similar scaling factor that also takes the intermediate activation tensor into account. However, they do not have a proof of optimality for this approach.
Absorbing high biases Nagel et al. (2019) further notice that in some cases, especially after CLE, high biases can lead to differences in the dynamic ranges of the activations. Therefore, they propose a procedure to, if possible, absorb high biases into the next layer. To absorb c from layer one (followed by a ReLU activation function f ) into layer two, we can do the following reparameterization:  and b (1) = b (1)c. In step two, we use the fact that for a layer with ReLU function f , there is a non-negative vector c such that r(Wx + bc) = r(Wx + b)c. The trivial solution c = 0 holds for all x. However, depending on the distribution of x and the values of W and b, there can be some values c i > 0 for which this equality holds for (almost) all x in the empirical distribution. This value is equal to
y = W (2) h + b (2) = W (2) (f (W (1) x + b (1) ) + c -c) + b (2) = W (2) (f (W (1) x + b (1) ) + c) + b (2) = W (2) h + b (2) (23) where b (2) = W (2) c + b (2) , h = h -c,
c i = max 0, min x W (1) i x + b (1) i . ()
where min x is evaluated on a small calibration dataset. To remove dependence on data, the authors propose to estimate the right hand side of ( 24) by the shift and scale parameters of the batch normalization layer which results3 in c i = max(0, β i -3γ i ).
Experiments In table 3 , we demonstrate the effect of CLE and bias absorption for quantizing MobileNetV2 to 8-bit. As skip connections break the equivariance between layers, we apply cross-layer equalization only to the layers within each residual block. Similar to Krishnamoorthi (2018) , we observe that model performance is close to random when quantizing MobileNetV2 to INT8. Applying CLE brings us back within 2% of FP32 performance, close to the performance of per-channel quantization. We note that absorbing high biases results in a small drop in FP32 performance, as it is an approximation, but it boosts quantized performance by 1% due to more precise activation quantization. Together, CLE and bias absorption followed by per-tensor quantization yield better results than per-channel quantization.
3.3 Bias correction
Another common issue is that quantization error is often biased. This means that the expected output of the original and quantized layer or network is shifted (E [Wx] = E Wx ). This kind of error is more pronounced in depth-wise separable layers with only a few elements per output channel (usually 9 for a 3 × 3 kernel). The main contributor to this error is often the clipping error, as a few strongly clipped outliers will likely lead to a shift in the expected distribution.
Several papers (Nagel et al., 2019; Meller et al., 2019; Finkelstein et al., 2019) noted this issue and introduce methods to correct for the expected shift in distribution. For a quantized layer W with quantization error ∆W = W -W, the expected output distribution is
E [ y] = E Wx = E [(W + ∆W)x] = E [Wx] + E [∆Wx] . ()
Thus the biased error is given by E
[∆Wx]. Since ∆W is constant, we have that E [∆Wx] = ∆W E [x]. In case ∆W E [x]
is nonzero, the output distribution is shifted. To counteract this shift we can substract it from the output:
E [y corr ] = E Wx -∆W E [x] = E [y] .Note, this correction term is a vector with the same shape as the bias and can thus be absorbed into the bias without any additional overhead at inference time. There are several ways of calculating the bias correction term, the two most common of which are empirical bias correction and analytic bias correction.
Empirical bias correction If we have access to a calibration dataset the bias correction term can simply be calculated by comparing the activations of the quantized and full precision model. In practice, this can be done layer-wise by computing
∆W E [x] = E Wx -E [Wx] .Analytic bias correction Nagel et al. (2019) introduce a method to analytically calculate the biased error, without the need for data. For common networks with batch normalization and ReLU functions, they use the BN statistics of the preceding layer in order to compute the expected input distribution E [x]. The BN parameters γ and β correspond to the mean and standard deviation of the BN layers output. Assuming input values are normally distributied, the effect of ReLU on the distribution can be modeled using the clipped normal distribution. They show that
E [x] = E [ReLU (x pre )] = γN -β γ + β 1 -Φ -β γ (28)
where x pre is the pre-activation output, which is assumed to be normally distributed with the perchannel means β and per-channel standard deviations γ, Φ(•) is the standard normal CDF, and the notation N (x) is used to denote the standard normal PDF. Note, all vector operations are elementwise (per-channel) operations. After calculating the input distribution E [x], the correction term can be simply derived by multiplying it with the weight quantization error ∆W.
Experiments In table 4 , we demonstrate the effect of bias correction for quantizing MobileNetV2 to 8-bit. Applying analytical bias correction improves quantized model performance from random to over 50%, indicating that the biased error introduced by quantization significantly harms model performance. When combining bias correction with CLE, we see that both techniques are complementary. Together, they achieve near FP32 performance without using any data.
3.4 AdaRound
Neural network weights are usually quantized by projecting each FP32 value to the nearest quantization grid point, as indicated by • in equation ( 4) for a uniform quantization grid. We refer to this quantization strategy as rounding-to-nearest. The rounding-to-nearest strategy is motivated by the fact that, for a fixed quantization grid, it yields the lowest MSE between the floating-point and quantized weights. However, Nagel et al. (2020) showed that rounding-to-nearest is not optimal in terms of the task loss when quantizing weights in the post-training regime. To illustrate this the authors quantized the weights of the first layer of ResNet18 to 4 bits using 100 different stochastic rounding samples (Gupta et al., 2015) and evaluated the performance of the network for each rounding choice. The best rounding choice among these outperformed rounding-to-nearest by more than 10%. Figure 7 illustrates this by plotting the performance of these rounding choices on the y-axis. In this section, we describe AdaRound (Nagel et al., 2020) , a systematic approach to finding good weight rounding choices for PTQ. AdaRound is a theoretically well-founded and computationally efficient method that shows significant performance improvement in practice.
As the main goal is to minimize the impact of quantization on the final task loss, we start by formulating the optimization problem in terms of this loss
arg min ∆w E [L (x, y, w + ∆w) -L (x, y, w)] ()
where ∆w denotes the perturbation due to quantization and can take two possible values for each weight, one by rounding the weight up and the other by rounding the weight down. We want to solve this binary optimization problem efficiently. As a first step, we approximate the cost function using a second-order Taylor series expansion. This alleviates the need for performance evaluation for each new rounding choice during the optimization. We further assume that the model has converged, implying that the contribution of the gradient term in the approximation can be ignored, and that the Hessian is block-diagonal, which ignores cross-layer correlations. This leads to the following Hessian based quadratic unconstrained binary optimization (QUBO) problem
arg min ∆w ( ) E ∆w ( ) T H (w ( ) ) ∆w ( )The clear correlation in figure 7 between the validation accuracy and objective of equation ( 30) indicates that the latter serves as a good proxy for the task loss (equation 29), even for 4-bit weight quantization. Despite the performance gains (see table 5 ), equation ( 30) cannot be widely applied for weight rounding for main two reasons:
• The memory and computational complexity of calculating the Hessian is impractical for general use-cases. Table 5 : Impact of various approximations and assumptions made in section 3.4 on the ImageNet validation accuracy (%) for ResNet18 averaged over 5 runs. N/A implies that the corresponding experiment was computationally infeasible.
• The QUBO problem of equation ( 30) is NP-Hard.
To tackle the first problem, the authors introduced additional suitable assumptions that allow simplifying the objective of equation ( 30) to the following local optimization problem that minimizes the MSE of the output activations for a layer.
arg min ∆W ( ) k,: E ∆W ( ) k,: x ( -1) 2Equation ( 31) requires neither the computation of the Hessian nor any other backward or forward propagation information from the subsequent layers. Note that the approximations and the analysis that have been used to link the QUBO problem of equation ( 30) with the local optimization problem of equation ( 31) is independent of the rounding problem. Hence this analysis also benefits the design of algorithms for other problems, including model compression and NAS (Moons et al., 2020) .
The optimization of ( 31) is still an NP-hard optimization problem. To find a good approximate solution with reasonable computational complexity, the authors relax the optimization problem to the following continuous optimization problem
arg min V Wx -Wx 2 F + λf reg (V) ,where • 2 F denotes the Frobenius norm and W are the soft-quantized weights defined as
W = s • clamp W s + h (V) ; n, p .We use n and p to denote integer grid limits, n = q min /s and p = q max /s. V i,j is the continuous variable that we optimize over and h can be any monotonic function with values between 0 and 1, i.e., h (V i,j ) ∈ [0, 1]. In Nagel et al. (2020) , the authors use a rectified sigmoid as h. The objective of (32) also introduces a regularizer term that encourages the continuous optimization variables h (V i,j ) to converge to either 0 or 1, so that they are valid solutions to the discrete optimization in (31). The regularizer used in Nagel et al. (2020) is
f reg (V) = i,j 1 -|2h (V i,j ) -1| β , ()
where β is annealed during the course of optimization to initially allow free movement of h (V i,j ) and later to force them to converge to 0 or 1. To avoid error accumulation across layers of the neural network and to account for the non-linearity, the authors propose the following final optimization problem
arg min V f a (Wx) -f a Wx 2 F + λf reg (V) ,where x is the layer's input with all preceding layers quantized and f a is the activation function. The objective of (35) can be effectively and efficiently optimized using stochastic gradient descent. This approach of optimizing weight rounding is known as AdaRound.
To summarize, the way we round weights during the quantization operation has a significant impact on the performance of the network. AdaRound provides a theoretically sound, computationally fast weight rounding method. It requires only a small amount of unlabeled data samples, no hyperparameter tuning or end-to-end finetuning, and can be applied to fully connected and convolutional layers of any neural network. 
3.5 Standard PTQ pipeline
In this section, we present a best-practice pipeline for PTQ based on relevant literature and extensive experimentation. We illustrate the recommended pipeline in figure 8 . This pipeline achieves competitive PTQ results for many computer vision as well as natural language processing models and tasks.
Depending on the model, some steps might not be required, or other choices could lead to equal or better performance.
Cross-layer equalization First we apply cross-layer equalization (CLE), which is a pre-processing step for the full precision model to make it more quantization friendly. CLE is particularly important for models with depth-wise separable layers and for per-tensor quantization, but it often also shows improvements for other layers and quantization choices. Add quantizers Next we choose our quantizers and add quantization operations in our network as described in section 2.3. The choice of quantizer might depend on the specific target HW; for common AI accelerators we recommend using symmetric quantizers for the weights and asymmetric quantizers for the activations. If supported by the HW/SW stack then it is favorable to use per-channel quantization for weights. Weight range setting To set the quantization parameters of all weight tensors we recommend using the layer-wise MSE based criteria. In the specific case of per-channel quantization, using the min-max method can be favorable in some cases. AdaRound In case we have a small calibration dataset4 available we next apply AdaRound in order to optimize the rounding of the weights. This step is crucial to enable low-bit weight quantization (e.g. 4 bits) in the PTQ. Bias correction In case we do not have such a calibration dataset and the network uses batch normalization, we can use analytical bias correction instead. Activation range setting As the final step, we determine the quantization ranges of all datadependent tensors in the network (i.e., activations). We use the MSE based criteria for most of the layers, which requires a small calibration set to find the minimum MSE loss. Alternatively, we can use the BN based range setting to have a fully data-free pipeline.
3.6 Experiments
We now evaluate the performance of the aforementioned PTQ pipeline on common computer vision and natural language understanding applications. Our results are summarized in table 6. For the task of semantic segmentation, we evaluate DeepLabV3 (with a MobileNetV2 backbone) (Chen et al., 2017) on Pascal VOC and for object detection, EfficientDet (Tan et al., 2020) on COCO 2017. The rest of the computer vision models are evaluated on the ImageNet classification benchmark.
For natural language understanding, we evaluate BERT-base on the GLUE benchmark (Wang et al., 2018) .
In all cases, we observe that 8-bit quantization of weights and activation (W8A8) leads to only marginal loss of accuracy compared to floating-point (within 0.7%) for all models. For W8A8 . We evaluate all models on the respective validation sets.
Higher is better in all cases. † A few quantized activations are kept in higher precision (16 bits).
quantization we also see no significant gains from using per-channel quantization. However, the picture changes when weights are quantized to 4 bits (W4A8). For ResNet18/50 and InceptionV3 the accuracy drop is still within 1% of floating-point for both per-tensor and per-channel quantization. However, for more efficient networks, such as MobileNetV2 and EfficientNet lite, the drop increases to 2.5% and 4.2% respectively for per-tensor quantization. This is likely due to the quantization of the depth-wise separable convolutions. Here, per-channel quantization can show a significant benefit, for example, in EfficientNet lite per-channel quantization increases the accuracy by 2.8% compared to per-tensor quantization, bringing it within 1.4% of full-precision accuracy. We see similar effects for EfficientDet-D1 and DeeplabV3 which both uses depth-wise separable convolutions in their backbone.
For BERT-base, we observe that a few activation tensors have extreme differences in their dynamic ranges. To make PTQ still work, we identified these layers using our debugging procedure outlined in section 3.7 and kept them in 16 bit. Otherwise BERT-base follows similar trends as most other models and our PTQ pipeline allows 4 bit weight quantization within 1.5% drop in GLUE score.
3.7 Debugging
We showed that the standard PTQ pipeline can achieve competitive results for a wide range of models and networks. However, if after following the steps of our pipeline, the model's performance is still not satisfactory, we recommend a set of diagnostics steps to identify the bottlenecks and improve the performance. While this is not strictly an algorithm, these debugging steps can provide insights on why a quantized model underperforms and help to tackle the underlying issues. These steps are shown as a flow chart in figure 9 and are described in more detail below:
FP32 sanity check An important initial debugging step is to ensure that the floating-point and quantized model behave similarly in the forward pass, especially when using custom quantization pipelines. Set the quantized model bit-width to 32 bits for both weights and activation, or by-pass the quantization operation, if possible, and check that the accuracy matches that of the FP32 model.
Weights or activations quantization
The next debugging step is to identify how activation or weight quantization impact the performance independently. Does performance recover if all weights are quantized to a higher bit-width while activations are kept in a lower bitwidth, or conversely if all activations use a high bit-width and activations a low bit-width? This step can show the relative contribution of activations and weight quantization to the overall performance drop and point us towards the appropriate solution.
Fixing weight quantization If the previous step shows that weight quantization does cause significant accuracy drop, then there are a few solutions to try:
• Apply CLE if not already implemented, especially for models with depth-wise separable convolutions.
• Try per-channel quantization. This will address the issue of uneven per-channel weight distribution.
• Apply bias correction or AdaRound if calibration data is available.
Fixing activation quantization To reduce the quantization error from activation quantization, we can also try using different range setting methods or adjust CLE to take activation quantization ranges into account, as vanilla CLE can lead to uneven activation distribution.
Per-layer analysis If the global solutions have not restored accuracy to acceptable levels, we consider each quantizer individually. We set each quantizer sequentially, to the target bit-width while keeping the rest of the network to 32 bits (see inner for loop in figure 9 ).
Visualizing layers If the quantization of a individual tensor leads to significant accuracy drop, we recommended visualizing the tensor distribution at different granularities, e.g. per-channel as in figure 5 , and dimensions, e.g., per-token or per-embedding for activations in BERT.
Fixing individual quantizers The visualization step can reveal the source of the tensor's sensitivity to quantization. Some common solutions involve custom range setting for this quantizer or allowing a higher bit-width for problematic quantizer, e.g., BERT-base from table 6 . If the problem is fixed and the accuracy recovers, we continue to the next quantizer. If not, we may have to resort to other methods, such as quantization-aware training (QAT), which is discussed in section 4.
After completing the above steps, the last step is to quantize the complete model to the desired bit-width. If the accuracy is acceptable, we have our final quantized model ready to use. Otherwise, we can consider higher bit-widths and smaller granularities or revert to more powerful quantization methods, such as quantization-aware training.
4 Quantization-aware training
The post-training quantization techniques described in the previous section are the first go-to tool in our quantization toolkit. They are very effective and fast to implement because they do not require retraining of the network with labeled data. However, they have limitations, especially when aiming for low-bit quantization of activations, such as 4-bit and below. Post-training techniques may not be enough to mitigate the large quantization error incurred by low-bit quantization. In these cases, we resort to quantization-aware training (QAT). QAT models the quantization noise source (see section 2.3) during training. This allows the model to find more optimal solutions than post-training quantization. However, the higher accuracy comes with the usual costs of neural network training, i.e., longer training times, need for labeled data and hyper-parameter search.
In this section, we explore how back-propagation works in networks with simulated quantization and provide a standard pipeline for training models with QAT effectively. We will also discuss the implications of batch normalization folding and per-channel quantization in QAT and provide results for a wide range of tasks and models.
4.1 Simulating quantization for backward path
In section 2.3, we saw how quantization can be simulated using floating-point in deep learning frameworks. However, if we look at the computational graph of figure 4 , to train such a network we need to back-propagate through the simulated quantizer block. This poses an issue because the gradient of the round-to-nearest operation in equation ( 4) is either zero or undefined everywhere, which makes gradient-based training impossible. A way around this would be to approximate the gradient using the straight-through estimator (STE, Bengio et al. 2013) , which approximates the gradient of the rounding operator as 1:
∂ y ∂y = 1Using this approximation we can now calculate the gradient of the quantization operation from equation ( 7). For clarity we assume symmetric quantization, namely z = 0, but the same result applies to asymmetric quantization since the zero-point is a constant. We use n and p to define the integer grid limits, such that n = q min /s and p = q max /s. The gradient of equation ( 7) w.r.t its input,
x i , is given by:
∂ x i ∂x i = ∂q(x i ) ∂x i = s • ∂ ∂x i clamp x i s ; n, p + 0 =                s • ∂ x i /s ∂(x i /s) ∂(x i /s) ∂x i if q min ≤ x i ≤ q max , s • ∂n ∂x i if x i  q max , = 1 if q min ≤ x i ≤ q max , 0 otherwise. ()
Using this gradient definition we can now back-propagate through the quantization blocks. Figure 10 shows a simple computational graph for the forward and backward pass used in quantization-aware training. The forward pass is identical to that of figure 4 , but in the backward pass we effectively skip the quantizer block due to the STE assumption. In earlier QAT work the quantization ranges for weights and activations were updated at each iteration most commonly using the min-max range (Krishnamoorthi, 2018) . In later work (Esser et al., 2020; Jain et al., 2019; Bhalgat et al., 2020) , the STE is used to calculate the gradient w.r.t. the quantization parameters, z and s. Using the chain rule and the STE, we first calculate the gradient w.r.t. the scale-factor:
∂ x i ∂s = ∂ ∂s s • clamp x i s ; n, p =    -x i /s + x i /s if q min ≤ x i ≤ q max , n if x i  q max .Originally, we restricted the zero-point to be an integer. To make zero-point learnable we convert into a real number and apply the rounding operator. The modified quantization function is defined as:
x = q(x; s, z) = s • clamp x s + z ; n, p -zThe gradient w.r.t. to z is calculated by applying the STE once again to the rounding operator: Note that in some QAT literature, the BN-folding effect is ignored. While this is fine when we employ per-channel quantization (more below in this section), keeping BN unfolded for per-tensor quantization will result in one of the two following cases:
∂ x i ∂z = 0 q min ≤ x i ≤ q max , -s otherwise. (1. The BN layer applies per-channel rescaling during inference. In this case we might as well use per-channel quantization in the first place. 2. We fold BN during deployment into the weight tensor and incur potentially significant accuracy drop as we trained the network to adapt to a different quantization noise.
A simple but effective approach to modeling BN-folding in QAT is to statically fold the BN scale and offset into the linear layer's weights and bias, as we saw in equations ( 11) and ( 12). This corresponds to re-parametrization of the weights and effectively removes the batch normalization operation from the network entirely. When starting from a converged pre-trained model, static folding is very effective, as we can see from the result of table 7 .
An alternative approach by Jacob et al. (2018) both updates the running statistics during QAT and applies BN-folding using a correction. This approach is more cumbersome and computationally costly because it involves a double forward pass: one for the batch-statistics and one for the quantized linear operation. However, based on our experiments (see table 7 ), static-folding performs on par or better despite its simplicity.
Per-channel quantization In section 2.4.2, we mentioned that per-channel quantization of the weights can improve accuracy when it is supported by hardware. The static folding re-parametrization is also valid for per-channel quantization. However, per-channel quantization provides additional flexibility as it allows us to absorb the batch normalization scaling operation into the per-channel scale-factor. Let us see how this is possible by revisiting the BN folding equation from section 2.3.1, but this time introduce per-channel quantization of the weights, such that W k,: = q(W k,: ; s w,k ) = s w,k W int k,: . By applying batch normalization to the output of a linear layer similar to equation (10), we get:
y k = BatchNorm W k,: x = γ k W k,: σ 2 k + x + β k - γ k µ k σ 2 k + = γ k s w,k σ 2 k + W int k,: x + b k = s w,k W int k,: x + b kWe can see that it is now possible to absorb the batch normalization scaling parameters into the per-channel scale-factor. For this modeling approach is on par or better for per-channel quantization compared to static folding as we can see from the last two rows of table 7.
4.3 Initialization for QAT
In this section, we will explore the effect of initialization for QAT. It is common practice in literature to start from a pre-trained FP32 model (Esser et al., 2020; Krishnamoorthi, 2018; Jacob et al., 2018; Jain et al., 2019) . While it is clear that starting from an FP32 model is beneficial, the effect of the quantization initialization on the final QAT result is less studied. Here we explore the effect of using several of our PTQ techniques as an initial step before doing QAT.
Effect of range estimation
To assess the effect of the initial range setting (see section 3.1) for weights and activations, we perform two sets of experiments, which are summarized in table 8. In the first experiment, we quantize the weights to 4-bits and keep the activations in 8-bits. We compare the min-max initialization with the MSE based initialization for the weights quantization range. While the MSE initialized model has a significantly higher starting accuracy, the gap closes after training for 20 epochs.
To explore the same effect for activation quantization, we perform a similar experiment, where we now quantize the activation to 4-bits and compare min-max initialization with MSE based initialization.
The observations from weight range initialization hold here as well. In figure 11 we show the full training curve of this experiment. In the first few epochs, there is a significant advantage for using MSE initialization, which almost vanishes in the later stage of training. In conclusion, a better initialization can lead to better QAT results, but the gain is usually small and vanishes the longer the training lasts. In conclusion, for models that have severe issues with plain PTQ we may need advanced PTQ techniques such as CLE to initialize QAT. In most other cases, an improved PTQ initialization leads only to a minor improvement in the final QAT performance.
4.4 Standard QAT pipeline
In this section, we present a best-practice pipeline for QAT based on relevant literature and extensive experimentation. We illustrate the recommended pipeline in figure 12 . This pipeline yields good QAT results over a variety of computer vision and natural language processing models and tasks, and can be seen as the go-to tool for achieving low-bit quantization performance. As discussed in previous sections, we always start from a pre-trained model and follow some PTQ steps in order to have faster convergence and higher accuracy.
Cross-layer equalization Similar to PTQ, we first apply CLE to the full precision model. As we saw in table 9, this step is necessary for models that suffer from imbalanced weight distributions, such as MobileNet architectures. For other networks or in the case of perchannel quantization this step can be optional. Add quantizers Next, we choose our quantizers and add quantization operations in our network as described in section 2.3. The choice for quantizer might depend on the specific target HW, for common AI accelerators we recommend using symmetric quantizers for the weights and asymmetric quantizers for the activations. If supported by the HW/SW stack, then it is favorable to use per-channel quantization for weights. At this stage we will also take care that our simulation of batch normalization is correct, as discussed in section 4.2. Range estimation Before training we have to initialize all quantization parameters. A better initialization will help faster training and might improve the final accuracy, though often the improvement is small (see table 8 ). In general, we recommend to set all quantization parameters using the layer-wise MSE based criteria. In the specific case of per-channel quantization, using the min-max setting can sometimes be favorable. Learnable Quantization Parameters We recommend making the quantizer paramaters learnable, as discussed in section 4.1. Learning the quantization parameters directly, rather than updating them at every epoch, leads to higher performance especially when dealing with low-bit quantization. However, using learnable quantizers requires special care when setting up the optimizer for the task. When using SGD-type optimizers, the learning rate for the quantization parameters needs to be reduced compared to the rest of the network parameters.
The learning rate adjustment can be avoided if we use optimizers with adaptive learning rates such as Adam or RMSProp.
4.5 Experiments
Using our QAT pipeline, we quantize and evaluate the same models we used for PTQ in section 3. Table 10 : Performance (average over 3 runs) of our standard QAT pipeline for various models and tasks. DeeplabV3 (MobileNetV2 backbone) is evaluated on Pascal VOC (mean intersection over union), EfficientDet-D1 on COCO 2017 (mean average precision), BERT-base on the GLUE benchmark and all other models on ImageNet (accuracy). We evaluate all models on the respective validation sets. Higher is better in all cases.
corresponding GLUE tasks for 3 to 12 epochs depending on the task and the quantization granularity. We use the Adam optimizer for all models. We present the results with the best learning rate per quantization configuration and perform no further hyper-parameter tuning.
We observe that for networks without depth-wise separable convolutions (first 3 rows of table 10 ), W8A8 and W4A8 quantization perform on par with and even outperform the floating-point model in certain cases. This could be due to the regularizing effect of training with quantization noise or due to the additional fine-tuning during QAT. For the more aggressive W4A4 case, we notice a small drop but still within 1% of the floating-point accuracy.
Quantizing networks with depth-wise separable layers (MobileNetV2, EfficientNet lite, DeeplabV3, EfficientDet-D1) is more challenging; a trend we also observed from the PTQ results in section 3.6 and discussed in the literature (Chin et al., 2020; Sheng et al., 2018a) . Whereas 8-bit quantization incurs close to no accuracy drop, quantizing weights to 4 bits leads to a larger drop, e.g. approximately 4% drop for EfficientNet lite with per-tensor quantization. Per-channel quantization can improve performance significantly bringing DeepLabV3 to floating-point accuracy and reducing the gap of MobileNetV2 and EfficientNet lite to less than 1.5%. Quantizing both weights and activations to 4-bits remains a challenging for such networks, even with per-channel quantization it can lead to a drop of up to 5%. EfficientDet-D1 remains more difficult to quantize than the other networks in this group.
For BERT-base we observe that QAT with range learning can efficiently deal with the high dynamic ranges allowing to keep all activations in 8 bits (unlike for PTQ). W4A8 stays within 1% of the original GLUE score, indicating that low bit weight quantization is not a problem for transformer models. We only notice a significant drop in performance when combining this with low bit activation quantization (W4A4).
5 Summary and Conclusions
Deep learning has become an integral part of many machine learning applications and can now be found in countless electronic devices and services, from smartphones and home appliances to drones, robots and self-driving cars. As the popularity and reach of deep learning in our everyday life increases, so does the need for fast and power-efficient neural network inference. Neural network quantization is one of the most effective ways of reducing the energy and latency requirements of neural networks during inference.
Quantization allows us to move from floating-point representations to a fixed-point format and, in combination with dedicated hardware utilizing efficient fixed-point operations, has the potential to achieve significant power gains and accelerate inference. However, to exploit these savings, we require robust quantization methods that can maintain high accuracy, while reducing the bit-width of weights and activations. To this end, we consider two main classes of quantization algorithms: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).
Post-training quantization techniques take a pre-trained FP32 networks and convert it into a fixedpoint network without the need for the original training pipeline. This makes them a lightweight, push-button approach to quantization with low engineering effort and computational cost. We describe a series of recent advances in PTQ and introduce a PTQ pipeline that leads to near floating-point accuracy results for a wide range of models and machine learning tasks. In particular, using the proposed pipeline we can achieve 8-bit quantization of weights and activations within only 1% of the floating-point accuracy for all networks. We further show that many networks can be quantized even to 4-bit weights with only a small additional drop in performance. In addition, we introduce a debugging workflow to effectively identify and fix problems that might occur when quantizing new networks.
Quantization-aware training models the quantization noise during training through simulated quantization operations. This training procedure allows for better solutions to be found compared to PTQ while enabling more effective and aggressive activation quantization. Similar to PTQ, we introduce a standard training pipeline utilizing the latest algorithms in the field. We also pay special attention to batch normalization folding during QAT and show that simple static folding outperforms other more computationally expensive approaches. We demonstrate that with our QAT pipeline we can achieve 4-bit quantization of weights, and for some models even 4-bit activations, with only a small drop of accuracy compared to floating-point.
The choice between PTQ and QAT depends on the accuracy and power requirements of the application. Both approaches are an essential part of any model efficiency toolkit and we hope that our proposed pipelines will help engineers deploy high-performing quantized models with less time and effort.
Assuming x is normally distributed, the equality will hold for approximately 99.865% of the inputs.
Usually, between
and 1000 unlabeled images are sufficient as a calibration set.
References
[1]Banner R, Nahshan Y, Soudry D. Post training 4-bit quantization of convolutional networks for rapid-deployment. Neural Information Processing Systems (NeuRIPS) 2019;:.
[2]Bengio Y, Léonard N, Courville A. Estimating or propagating gradients through stochastic neurons for conditional computation.  2013;:.
[3]Bhalgat Y, Lee J, Nagel M, Blankevoort T, Kwak N. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 1920;:.
[4]Chen L.-C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation.  2017;:.
[5]Chin T.-W, Chuang PI, Chandra -J, Marculescu V,  D. One weight bitwidth to rule them all. Computer Vision -ECCV 2020 Workshops 2020;:85--103.
[6]Dong Z, Yao Z, Gholami A, Mahoney MW, Keutzer K. HAWQ: hessian aware quantization of neural networks with mixed-precision. International Conference on Computer Vision (ICCV) 2019;:.
[7]Esser SK, Mckinstry JL, Bablani D, Appuswamy R, Modha DS. Learned step size quantization. International Conference on Learning Representations (ICLR) 2020;20:.
[8]Finkelstein A, Almog U, Grobman M. Fighting quantization bias with bias.  2019;:.
[9]Gupta S, Agrawal A, Gopalakrishnan K, Narayanan P. Deep learning with limited numerical precision. International Conference on Machine Learning 2015;:.
[10]Horowitz . 1 computing's energy problem (and what we can do about it). 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC) 2014;2:.
[11]Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, et al.. Efficient convolutional neural networks for mobile vision applications.  2017;:.
[12]Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning 2015;37:7--09.
[13]Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, et al.. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Conference on Computer Vision and Pattern Recognition (CVPR) 2006;21:.
[14]Jain SR, Gural A, Wu M, Dick C. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware.  2019;:.
[15]Krishnamoorthi R. Quantizing deep convolutional networks for efficient inference: A whitepaper.  1920;21:.
[16]Meller E, Finkelstein A, Almog U, Grobman M. Same, same but different: Recovering neural network quantization error through weight factorization. Proceedings of the 36th International Conference on Machine Learning, ICML 2019;12:.
[17]Moons B, Noorzad P, Skliar A, Mariani G, Mehta D, Lott C, et al.. Distilling optimal neural networks: Rapid search in diverse spaces.  2020;:.
[18]Nagel M, Van Baalen M, Blankevoort T, Welling M. Data-free quantization through weight equalization and bias correction. International Conference on Computer Vision (ICCV) 2019;12:.
[19]Nagel M, Amjad RA, Van Baalen M, Louizos C, Blankevoort T. Up or down? Adaptive rounding for post-training quantization. Proceedings of the 37th International Conference on Machine Learning 2020;119:.
[20]Nascimento MG D, Fawcett R, Prisacariu VA, Dsconv . Efficient convolution operator. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2005;:.
[21]Ramachandran P, Zoph B, Le QV. Searching for activation functions.  2017;:.
[22]Rouhani B, Lo D, Zhao R, Liu M, Fowers J, Ovtcharov K, et al.. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Neural Information Processing Systems 2005;:.
[23]Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition 2018;:4510--4520.
[24]Sheng T, Feng C, Zhuo S, Zhang X, Shen L, Aleksic M. A quantization-friendly separable convolution for mobilenets. 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), Mar 2018a.  None;:.
[25]Sheng T, Feng C, Zhuo S, Zhang X, Shen L, Aleksic M. A quantization-friendly separable convolution for mobilenets. 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), 2018b None;:.
[26]Stock P, Joulin A, Gribonval R, Graham B, Jégou H. And the bit goes down: Revisiting the quantization of neural networks.  2019;:.
[27]Tan M, Pang R, Le QV. Efficientdet: Scalable and efficient object detection.  2020;:.
[28]Uhlich S, Mauch L, Cardinaux F, Yoshiyama K, Garcia JA, Tiedemann S, et al.. Mixed precision dnns: All you need is a good parametrization. International Conference on Learning Representations 2020;:.
[29]Van Baalen M, Louizos C, Nagel M, Amjad RA, Wang Y, Blankevoort T, et al.. Bayesian bits: Unifying quantization and pruning. Advances in Neural Information Processing Systems 2020;33:5741--5752.
[30]Wang A, Singh A, Michael J, Hill F, Levy O, Bowman S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP 2018;:18--5446.
A White Paper on Neural Network Quantization
https://arxiv.org/pdf/2106.08295.pdf