State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved. In this work, we propose a quantization strategy tailored to the Transformer architecture. We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline. We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU. The idea of using neural networks for machine translation was proposed only recently (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; . Nonetheless, the approach has reached impressive levels of translation. (Ahmed et al., 2017; . A key element of this success was to allow the decoder to attend to all hidden states of the encoder . A few variations to this additive attention mechanism were proposed, such as multiplicative attention and self-attention (Luong et al., 2015; Cheng et al., 2016; Lin et al., 2017) . The latter formed the basis of the Transformer network (Vaswani et al., 2017) , which achieved stateof-the-art machine translation. Inspiring a new wave of work, numerous natural language processing tasks reached new heights (Devlin et al., 2018; Liu et al., 2019) . Unfortunately, these models make use of an enormous amount of parameters. Inference on resource-limited hardware such as edgedevices is thus impractical. A solution to reduce the computational burden of these neural networks is to lower numerical precision. Consequently, numerical values can be represented using fewer bits (Tang & Kwan, 1993; Marchesi et al., 1993) . This method called quantization has the advantage of providing good compression rates with minimal loss in accuracy. It is also conveniently supported by most hardware. Properly quantizing the Transformer would allow computational speed gains at inference, as well as deployment on more constrained devices. In this work, we propose a custom quantization strategy of the entire Transformer architecture, where quantization is applied during the training process. Our method is easy to implement and results are consistent with the full-precision Transformer. We test our approach on multiple translation tasks such as WMT14 EN-FR and WMT14 EN-DE and obtain state-of-the-art quantization results. On most tasks, our quantized models score equal or higher BLEU compared to full-precision. We are, to the best of our knowledge, the first to fully quantize the Transformer architecture without impairing translation quality. We proposed a quantization strategy for the Transformer, quantizing all operations which could provide a computational speed gain, for a fully quantized architecture. All of our design decisions were aimed at maximizing computational efficiency while making sure our method would be compatible with as many different types of hardware as possible. With our method, we achieve higher BLEU scores than all other quantization methods for the Transformer on multiple translation tasks and avoid any loss in BLEU compared to full-precision. Specifically, out of 41 experiments, 8-bit quantization performed equal or better to full-precision in 36 cases. We are very excited about the possibilities this work opens and plan on applying our method to other tasks. We also intend to extend our work to variations of the Transformer, as well as further exploring the compression of these networks. We evaluated our quantization method on additional translation datasets (see Table 7 ). All models are trained following the same setup as in section 5.1, except the big model was only trained for one epoch. Vocabulary size is set to 30k for all models. Since there is no test set for WMT14 ES-EN, we used the validation set as a test set and omitted computing any validation epochs during training.
We fully quantize the Transformer to 8-bit and improve translation quality compared to the full precision model. An 8-bit quantization method to quantize the machine translation model Transformer, proposing to use uniform min-max quantization during inference and bucketing weigts before quantization to reduce quantization error. A method for reducing the required memory space by a quantization technique, focused on reducing it for Transformer architecture.