In this paper, we propose the use of in-training matrix factorization to reduce the model size for neural machine translation. Using in-training matrix factorization, parameter matrices may be decomposed into the products of smaller matrices, which can compress large machine translation architectures by vastly reducing the number of learnable parameters. We apply in-training matrix factorization to different layers of standard neural architectures and show that in-training factorization is capable of reducing nearly 50% of learnable parameters without any associated loss in BLEU score. Further, we find that in-training matrix factorization is especially powerful on embedding layers, providing a simple and effective method to curtail the number of parameters with minimal impact on model performance, and, at times, an increase in performance. While neural models for machine translation have realized considerable breakthroughs in recent years and are now state-of-the-art in many contexts, they are frequently expensive in resources, owing to their large number of parameters. In a context of democratization of deep learning tools, having smaller models that can be learned and/or applied offline on small devices would have immediate and important applications, for example for privacy reasons. However, it is often necessary to leverage deep networks with large layers in order to capture abstract and complex patterns and interactions over the data. We posit that the need for such large parameter matrices is not because they are essential to the representational power of the network. Rather, our hypothesis is that having many parameters can help neural architectures overcome limitations introduced with approximate optimization methods, noisy data supervision, and sub-optimal model architectures. Moreover, recent work has suggested many parameters in large neural architectures are largely superfluous, serving solely to accommodate the stochastic nature of modern machine learning optimization algorithms (Frankle & Carbin, 2019) . Motivated by those hypotheses, we study the application of in-training matrix factorization to neural machine translation. Traditional matrix factorization methods are a simple but powerful technique that has been used after training in other deep learning systems, such as Sainath et al. (2013) for speech recognition and Kim et al. (2015) for computer vision. In contrast to traditional matrix factorization techniques, in-training matrix factorization reduces the number of learnable parameters at training time, lessening the need for computational resources. The main contributions of this work are: 1. We formally define in-training matrix factorization and present a technique to utilize matrix factorization during training time. 2. We conduct sets of experiments on two standard neural machine translation architectures: the LSTM encoder-decoder and the transformer network. 3. We show that in-training factorization can decrease a model's size by half with no impact on performance, and at times, can improve model performance Figure 1 : A diagram of in-training factorization. A weight matrix is replaced by the product of two weight matrices, followed by the bias and activation. We have introduced a method to use matrix factorization at training time to reduce the parameter footprint of neural machine translation models. We compare in-training factorization to existing post-hoc parameter reduction methods, including parameter pruning and post-training factorization. We find that using factorization comes with significant gains in both final performance and number of required parameters. Lastly, we demonstrate the effectiveness of our in-training factorization technique to learn a model with fewer parameters, improved accuracy, and decreased training time.
This paper proposes using matrix factorization at training time for neural machine translation, which can reduce model size and decrease training time without harming performance. This paper proposes to compress models using matrix factorization during training for deep neural networks of machine translation.