Neural Machine Translation (NMT) is notorious for its need for large amounts of bilingual data. An effective approach to compensate for this requirement is Multi-Task Learning (MTL) to leverage different linguistic resources as a source of inductive bias. Current MTL architectures are based on the SEQ2SEQ transduction, and (partially) share different components of the models among the tasks. However, this MTL approach often suffers from task interference, and is not able to fully capture commonalities among subsets of tasks. We address this issue by extending the recurrent units with multiple blocks along with a trainable routing network. The routing network enables adaptive collaboration by dynamic sharing of blocks conditioned on the task at hand, input, and model state. Empirical evaluation of two low-resource translation tasks, English to Vietnamese and Farsi, show +1 BLEU score improvements compared to strong baselines.
Introduction Neural Machine Translation (NMT) has shown remarkable progress in recent years. However, it requires large amounts of bilingual data to learn a translation model with reasonable quality (Koehn and Knowles, 2017) . This requirement can be compensated by leveraging curated monolingual linguistic resources in a multi-task learning framework. Essentially, learned knowledge from auxiliary linguistic tasks serves as inductive bias for the translation task to lead to better generalizations. Multi-Task Learning (MTL) is an effective approach for leveraging commonalities of related tasks to improve performance. Various recent works have attempted to improve NMT by scaffolding translation task on a single auxiliary task (Domhan and Hieber, 2017; Zhang and Zong, 2016; Dalvi et al., 2017) . Recently, (Niehues and Cho, 2017) have made use of several linguistic tasks to improve NMT. Their method shares components of the SEQ2SEQ model among the tasks, e.g. encoder, decoder or the attention mechanism. However, this approach has two limitations: (i) it fully shares the components, and (ii) the shared component(s) are shared among all of the tasks. The first limitation can be addressed using deep stacked layers in encoder/decoder, and sharing the layers partially (Zaremoodi and Haffari, 2018) . The second limitation causes this MTL approach to suffer from task interference or inability to leverages commonalities among a subset of tasks. Recently, (Ruder et al., 2017) tried to address this issue; however, their method is restrictive for SEQ2SEQ scenarios and does not consider the input at each time step to modulate parameter sharing. In this paper, we address the task interference problem by learning how to dynamically control the amount of sharing among all tasks. We extended the recurrent units with multiple blocks along with a routing network to dynamically control sharing of blocks conditioning on the task at hand, the input, and model state. Empirical results on two low-resource translation scenarios, English to Farsi and Vietnamese, show the effectiveness of the proposed model by achieving +1 BLEU score improvement compared to strong baselines. SEQ2SEQ MTL Using Recurrent Unit with Adaptive Routed Blocks Our MTL is based on the sequential encoderdecoder architecture with the attention mecha-nism (Luong et al., 2015b; . The encoder/decoder consist of recurrent units to read/generate a sentence sequentially. Sharing the parameters of the recurrent units among different tasks is indeed sharing the knowledge for controlling the information flow in the hidden states. Sharing these parameters among all tasks may, however, lead to task interference or inability to leverages commonalities among subsets of tasks. We address this issue by extending the recurrent units with multiple blocks, each of which processing its own information flow through the time. The state of the recurrent unit at each time step is composed of the states of these blocks. The recurrent unit is equipped with a routing mechanism to softly direct the input at each time step to these blocks (see Fig 1) . Each block mimics an expert in handling different kinds of information, coordinated by the router. In MTL, the tasks can use different subsets of these shared experts. (Rosenbaum et al., 2018) uses a routing network for adaptive selection of non-linear functions for MTL. However, it is for fixed-size inputs based on a feed-forward architecture, and is not applicable to SEQ2SEQ scenarios such as MT. (Shazeer et al., 2017) uses Mixture-of-Experts (feed-forward sub-networks) between stacked layers of recurrent units, to adaptively gate state information vertically. This is in contrast to our approach where the horizontal information flow is adaptively modulated, as we would like to minimise the task interference in MTL. Assuming there are n blocks in a recurrent unit, we share n − 1 blocks among the tasks, and let the last one to be task-specific 1 . Task-specific block receives the input of the unit directly while shared blocks are fed with modulated input by the routing network. The state of the unit at each time-step would be the aggregation of blocks' states. Routing Mechanism At each time step, the routing network is responsible to softly forward the input to the shared blocks conditioning on the input x t , and the previous hidden state of the unit h t−1 as follows: s t = tanh(W x · x t + W h · h t−1 + b s ), τ t = softmax(W τ · s t + b τ ), where W 's and b's are the parameters. Then, the i-th shared block is fed with the input of the 1 multiple recurrent units can be stacked on top of each other to consist a multi-layer component Figure 1 : High-level architecture of the proposed recurrent unit with 3 shared blocks and 1 taskspecific. h (2) h (1) h (3) h t-1 x t h t h (4) h (2) h (1) h (3) h (4) t t t t t-1 t-1 t-1 t-1 unit modulated by the corresponding output of the routing networkx (i) t = τ t [i]x t where τ t [i] is the scalar output of the routing network for the i-th block. The hidden state of the unit is the concatenation of the hidden state of the shared and taskspecific parts h t = [h (shared) t ; h (task) t ]. The state of task-specific part is the state of the corresponding block h (task) t = h (n+1) t , and the state of the shared part is the sum of states of shared blocks weighted by the outputs of the routing network h (shared) t = n i=1 τ t [i]h (i) t . Block Architecture Each block is responsible to control its own flow of information via a standard gating mechanism. Our recurrent units are agnostic to the internal architecture of the blocks; we use the gated-recurrent unit  in this paper. For the i-th block the corresponding equations are as follows: z (i) t = σ(W (i) zx (i) t + U (i) z h (i) t−1 + b (i) z ), r (i) t = σ(W (i) rx (i) t + U (i) r h (i) t−1 + b (i) r ), h (i) t = tanh(W (i) hx (i) t + U (i) h h (i) t−1 + b (i) h ), h (i) t = z (i) t h (i) t−1 + (1 − z (i) t ) h (i) t . Training Objective and Schedule. The rest of the model is similar to attentional SEQ2SEQ model (Luong et al., 2015b) which computes the conditional probability of the target sequence given the source P θ θ θ (y|x) = j P θ θ θ (y j |y <j x). For the case of training M + 1 SEQ2SEQ transduction tasks, each of which is associated with a training set D m := {(x i , y i )} Nm i=1 , the parameters of MTL architecture Θ mtl = {Θ m } M m=0 are learned by maximizing the following objective: L mtl (Θ mtl ) := M m=0 γ m |D m | (x,y)∈Dm log P Θm (y|x) where |D m | is the size of the training set for the mth task, and γ m is responsible to balance the influence of tasks in the training objective. We explored different values in preliminary experiments, and found that for our training schedule γ = 1 for all tasks results in the best performance. Generally, γ is useful when the dataset sizes for auxiliary tasks are imbalanced (our training schedule handles the main task). Variants of stochastic gradient descent (SGD) can be used to optimize the objective function. In our training schedule, we randomly select a mini-batch from the main task (translation) and another mini-batch from a randomly selected auxiliary task to make the next SGD update. Selecting a mini-batch from the main task in each SGD update ensures that its training signals are not washed out by auxiliary tasks. Experiments Bilingual Corpora We use two language-pairs, translating from English to Farsi and Vietnamese. We have chosen them to analyze the effect of multi-task learning on languages with different underlying linguistic structures 2 . We apply BPE (Sennrich et al., 2016) on the union of source and target vocabularies for English-Vietnamese, and separate vocabularies for English-Farsi as the alphabets are disjoined (30K BPE operations). Further details about the corpora and their pre-processing is as follows: • The English-Farsi corpus has ∼105K sentence pairs. It is assembled from English-Farsi parallel subtitles from the TED corpus (Tiedemann, 2012) , accompanied by all the parallel news text in LDC2016E93 Farsi Representative Language Pack from the Linguistic Data Consortium. The corpus has been normalized using the Hazm toolkit 3 . We have removed sentences with more than 80 tokens in either side (before applying BPE). 3k and 4k sentence pairs were held out for the purpose of validation and test. • The English-Vietnamese has ∼133K training pairs. It is the preprocessed version of the IWSLT 2015 translation task provided by (Luong and Manning, 2015) . It consists of subtitles and their corresponding translations of a collection of public speeches from TED and TEDX talks. The "tst2012" and "tst2013" parts are used as validation and test sets, respectively. We have removed sentence pairs which had more than 300 tokens after applying BPE on either sides. Auxiliary Tasks We have chosen the following auxiliary tasks to leverage the syntactic and semantic knowledge to improve NMT: Named-Entity Recognition (NER). It is expected that learning to recognize named-entities help the model to learn translation pattern by masking out named-entites. We have used the NER data comes from the CONLL shared task. 4 Sentences in this dataset come from a collection of newswire articles from the Reuters Corpus. These sentences are annotated with four types of named entities: persons, locations, organizations and names of miscellaneous entities. Syntactic Parsing. By learning the phrase structure of the input sentence, the model would be able to learn better re-ordering. Specially, in the case of language pairs with high level of syntactic divergence (e.g. English-Farsi). We have used Penn Tree Bank parsing data with the standard split for training, development, and test (Marcus et al., 1993) . We cast syntactic parsing to a SEQ2SEQ transduction task by linearizing constituency trees (Vinyals et al., 2015) . Semantic Parsing. Learning semantic parsing helps the model to abstract away the meaning from the surface in order to convey it in the target translation. For this task, we have used the Abstract Meaning Representation (AMR) corpus Release 2.0 (LDC2017T10) 5 . This corpus contains natural language sentences from newswire, weblogs, web discussion forums and broadcast conversations. We cast this task to a SEQ2SEQ transduction task by linearizing the AMR graphs (Konstas et al., 2017) . Models and Baselines We have implemented the proposed MTL architecture along with the baselines in C++ using DyNet (Neubig et al., 2017) on top of Mantis (Cohn et al., 2016) which is an implementation of the attentional SEQ2SEQ NMT model. For our MTL architecture, we used the proposed recurrent unit with 3 blocks in encoder and decoder. For the fair comparison in terms the of number of parameters, we used 3 stacked layers in both encoder and decoder components for the baselines. We compare against the following baselines: • Baseline 1: The vanilla SEQ2SEQ model (Luong et al., 2015a) without any auxiliary task. • Baseline 2: The MTL architecture proposed in (Niehues and Cho, 2017) which fully shares parameters in components. We have used their best performing architecture with our training schedule. We have extended their work with deep stacked layers for the sake of comparison. • Baseline 3: The MTL architecture proposed in (Zaremoodi and Haffari, 2018) which uses deep stacked layers in the components and shares the parameters of the top two/one stacked layers among encoders/decoders of all tasks 6 . For the proposed MTL, we use recurrent units with 400 hidden dimensions for each block. The encoders and decoders of the baselines use GRU units with 400 hidden dimensions. The attention component has 400 dimensions. We use Adam optimizer (Kingma and Ba, 2014) with the initial learning rate of 0.003 for all the tasks. Learning 6 In preliminary experiments, we have tried different sharing scenarios and this one led to the best results. rates are halved on the decrease in the performance on the dev set of corresponding task. Mini-batch size is set to 32, and dropout rate is 0.5. All models are trained for 50 epochs and the best models are saved based on the perplexity on the dev set of the translation task. For each task, we add special tokens to the beginning of source sequence (similar to (Johnson et al., 2017) ) to indicate which task the sequence pair comes from. We used greedy decoding to generate translation. In order to measure translation quality, we use BLEU 7 (Papineni et al., 2002) and TER (Snover et al., 2006) scores. Table 1 reports the results for the baselines and our proposed method on the two aforementioned translation tasks. As expected, the performance of MTL models are better than the baseline 1 (only MT task). As seen, partial parameter sharing is more effective than fully parameter sharing. Furthermore, our proposed architecture with adaptive sharing performs better than the other MTL methods on all tasks, and achieve +1 BLEU score improvements on the test sets. The improvements in the translation quality of NMT models trained by our MTL method may be attributed to less interference with multiple auxiliary tasks. Figure 2 shows the average percentage of block usage for each task in an MTL model with 3 shared blocks, on the English-Farsi test set. We have aggregated the output of the routing network for the blocks in the encoder recurrent units over all the input tokens. Then, it is normalized by dividing on the total number of input tokens. Based on Figure 2 , the first and third blocks are more specialized (based on their usage) for the translation and NER tasks, respectively. The second block is mostly used by the semantic and syntactic parsing tasks, so specialized for them. This confirms our model leverages commonalities among subsets of tasks by dedicating common blocks to them to reduce task interference. Results and analysis Conclusions We have presented an effective MTL approach to improve NMT for low-resource languages, by leveraging curated linguistic resources on the source side. We address the task interference issue in previous MTL models by extending the recurrent units with multiple blocks along with a trainable routing network. Our experimental results on low-resource English to Farsi and Vietnamese datasets, show +1 BLEU score improvements compared to strong baselines.
1 2 2.1 2.2 2.3 3.1 3.2 3.3 4
Introduction SEQ2SEQ MTL Using Recurrent Unit with Adaptive Routed Blocks Routing Mechanism Block Architecture Training Objective and Schedule. Bilingual Corpora Auxiliary Tasks Models and Baselines Conclusions
Improving NMT in low Resource scenarios
Bilingually low-resource scenario: large amounts of bilingual training data is not available
IDEA: Use existing resources from other tasks and train one model for all tasks using multi-task learning
This effectively injects inductive biases to help improving the generalisation of
Auxiliary tasks: Semantic Parsing, Syntactic Parsing, Named Entity Recognition
Bilingually low-resource scenario: large amounts of bilingual training data is not available
IDEA: Use existing resources from other tasks and train one model for all tasks using multi-task learning
This effectively injects inductive biases to help improving the generalisation of
Auxiliary tasks: Semantic Parsing, Syntactic Parsing, Named Entity Recognition