Depthwise separable convolutions reduce the number of parameters and computation used in convolutional operations while increasing representational efficiency.
 They have been shown to be successful in image classification models, both in obtaining better models than previously possible for a given parameter count (the Xception architecture) and considerably reducing the number of parameters required to perform at a given level (the MobileNets family of architectures). Recently, convolutional sequence-to-sequence networks have been applied to machine translation tasks with good results. In this work, we study how depthwise separable convolutions can be applied to neural machine translation. We introduce a new architecture inspired by Xception and ByteNet, called SliceNet, which enables a significant reduction of the parameter count and amount of computation needed to obtain results like ByteNet, and, with a similar parameter count, achieves better results.
 In addition to showing that depthwise separable convolutions perform well for machine translation, we investigate the architectural changes that they enable: we observe that thanks to depthwise separability, we can increase the length of convolution windows, removing the need for filter dilation. We also introduce a new super-separable convolution operation that further reduces the number of parameters and computational cost of the models. In recent years, sequence-to-sequence recurrent neural networks (RNNs) with long short-term memory (LSTM) cells BID7 have proven successful at many natural language processing (NLP) tasks, including machine translation BID18 BID2 BID4 . In fact, the results they yielded have been so good that the gap between human translations and machine translations has narrowed significantly BID23 and LSTM-based recurrent neural networks have become standard in natural language processing.Even more recently, auto-regressive convolutional models have proven highly effective when applied to audio BID19 , image BID20 ) and text generation BID11 . Their success on sequence data in particular rivals or surpasses that of previous recurrent models BID11 BID6 . Convolutions provide the means for efficient non-local referencing across time without the need for the fully sequential processing of RNNs. However, a major critique of such models is their computational complexity and large parameter count. These are the principal concerns addressed within this work: inspired by the efficiency of depthwise separable convolutions demonstrated in the domain of vision, in particular the Xception architecture BID5 and MobileNets (Howard et al., 2017) , we generalize these techniques and apply them to the language domain, with great success. In this work, we introduced a new convolutional architecture for sequence-to-sequence tasks, called SliceNet, based on the use of depthwise separable convolutions. We showed how this architecture achieves results beating not only ByteNet but also the previous best Mixture-of-Experts models while using over two times less (non-embedding) parameters and floating point operations than ByteNet.Additionally, we have shown that filter dilation, previously thought to be a key component of successful convolutional sequence-to-sequence architectures, was not a requirement. The use of depthwise separable convolutions makes much larger convolution window sizes possible, and we found that we could achieve the best results by using larger windows instead of dilated filters. We have also introduced a new type of depthwise separable convolution, the super-separable convolution, which shows incremental performance improvements over depthwise separable convolutions.Our work is one more point on a significant trendline started with Xception and MobileNets, that indicates that in any convolutional model, whether for 1D or 2D data, it is possible to replace convolutions with depthwise separable convolutions and obtain a model that is simultaneously cheaper to run, smaller, and performs a few percentage points better. This trend is backed by both solid theoretical foundations and strong experimental results. We expect our current work to play a significant role in affirming and accelerating this trend. We only experimented on translation, but we expect that our results will apply to other sequence-to-sequence tasks and we hope to see depthwise separable convolutions replace regular convolutions in more and more use cases in the future.
Depthwise separable convolutions improve neural machine translation: the more separable the better. This paper proposes to use depthwise separable convolution layers in a fully convolutional neural machine translation model, and introduces a new super-separable convolution layer which further reduces computational cost.