In this paper, we describe NAVER machine translation system for English to Japanese and Korean to Japanese tasks at WAT 2015. We combine the traditional SMT and neural MT in both tasks.
Introduction This paper explains the NAVER machine translation system for the 2nd Workshop on Asian Translation (WAT 2015) (Nakazawa et al., 2015) . We participate in two tasks; English to Japanese (En-Ja) and Korean to Japanese (Ko-Ja). Our system is a combined system of traditional statistical machine translation (SMT) and neural machine translation (NMT). We adopt the tree-tostring syntax-based model as En-Ja SMT baseline, while we adopt the phrase-based model as Ko-Ja. We propose improved SMT systems for each task and an NMT model based on the architecture using recurrent neural network (RNN) (Cho et al., 2014; Sutskever et al., 2014) . We give detailed explanations of each SMT system in section 2 and section 3. We describe our NMT model in section 4. 2 English to Japanese Training data We used 1 million sentence pairs that are contained in train-1.txt of ASPEC-JE corpus for training the translation rule tables and NMT models. We also used 3 million Japanese sentences that are contained in train-1.txt, train-2.txt,train-3.txt of ASPEC-JE corpus for training the 5-gram language model. We also used 1,790 sentence pairs of dev.txt for tuning the weights of each feature of SMT linear model and as validation data of neural network. We filtered out the sentences that have 100 or more tokens from training data. Language Analyzer We used Moses tokenizer and Berkeley constituency parser 1 for tokenizing and parsing an English sentence. We used our own Japanese tokenizer and part-of-speech tagger for tokenizing and tagging a Japanese sentence. After running the tokenizer and the tagger, we make a token from concatenation of a word and its part-of-speech. Tree-to-string Syntax-based SMT To determining the baseline model, we first performed comparative experiments with the phrasebased, hierarchical phrase-based and syntax-based models. As a result, we chose the tree-to-string syntax-based model. The SMT models that consider source syntax such as tree-to-string and forest-to-string brought out better performance than the phrase-based and hierarchical phrase-based models in the WAT 2014 En-Ja task. The tree-to-string model was proposed by Huang (2006) and Liu (2006) . It utilizes the constituency tree of source language to extract translation rules and decode a target sentence. The translation rules are extracted from a source-parsed and word-aligned corpus in the training step. We use synchronous context free grammar (SCFG) rules. In addition, we used a rule augmentation method which is known as syntax-augmented machine translation (Zollmann and Venugopal, 2006) . Because the tree-to-string SMT makes some constraints on extracting rules by considering syntactic tree structures, it usually extracts fewer rules than hierarchical phrase-based SMT (HPBSMT) (Chiang, 2005) . Thus it is required to augment tree-to-string translation rules. The rule augmentation method allows the training system to extract more rules by modifying parse trees. Given a parse tree, we produce additional nodes by combining any pairs of neighboring nodes, not only children nodes, e.g. NP+VP. We limit the maximum span of each rule to 40 tokens in the rule extraction process. The tree-to-string decoder use a chart parsing algorithm with cube pruning proposed by Chiang (2005) . Our En-Ja SMT system was developed by using the open source SMT engines; Moses and Giza++. Its other specifications are as follows: • Grow-diag-final-and word alignment heuristic • Good-Turing discounting for smoothing probabilities • Minimum Error Rate Training (MERT) for tuning feature weights • Cube-pruning-pop-limit = 3000 Handling Out-of-Vocabulary In order to handle out-of-vocabulary (OOV) words, we use two techniques; hyphen word split and spell error correction. The former is to split a word with hyphen (-) to two separate tokens before running the language analyzer. The latter is to automatically detect and correct spell errors in an input sentence. We give a detailed description of spell error correction in section 2.4.1. English Spell Correction It is not easy to translate a word including errata, because the erroneous word has only a slim chance of occurrence in the training data. We discovered a lot of spell errors among OOV words that appear in English scientific text. We introduce English spell correction for reducing OOV words in input sentences. We developed our spell corrector by using Aspell 2 . For detecting a spell error, we skip words that have only capitals, numbers or symbols, because they are likely to be abbreviations or mathematic expressions. Then we regard words detected by Aspell as spell error words. For correcting spell error, we use only top-3 suggestion words from Aspell. We find that a large gap between an original word and its suggestion word makes wrong correction. To avoid excessive correction, we introduce a gap thresholding technique, that ignores the suggestion word that has 3 or longer edit distance and selects one that has 3 Korean to Japanese Training data We used 1 million sentence pairs that are contained in JPO corpus for training phrase tables and NMT models. We also used Japanese part of the corpus for training the 5-gram language model. We also used 2,000 sentence pairs of dev.txt for tuning the weights of each feature of SMT linear model and as validation data of neural network. We did not filter out any sentences. Language Analyzer We used MeCab-ko 3 for tokenizing a Korean sentence. We used Juman 4 for tokenizing a Japanese sentence. We did not perform part-of-speech tagging for both languages. Phrase-based SMT As in the En-Ja task, we first performed comparative experiments with the phrase-based and hierarchical phrase-based models, and then adopt the phrase-based model as our baseline model. For the Ko-Ja task, we develop two phrasebased systems; word-level and character-level. We use word-level tokenization for the word-based system. We found that setting the distortion limit to zero yields better translation in aspect of both BLEU and human evaluation. We use the 5-gram language model. We use character-level tokenization for character-based system. We use the 10gram language model and set the maximum phrase length to 10 in the phrase pair extraction process. We found that the character-level system does not suffer from tokenization error and out-ofvocabulary issue. The JPO corpus contains many technical terms and loanwords like chemical compound names, which are more inaccurately tokenized and allow a lot of out-of-vocabulary tokens to be generated. Since Korean and Japanese share similar transliteration rules for loanwords, the character-level system can learn translation of unseen technical words. It generally produces better translations than a table-based transliteration. Moreover, we tested jamo-level tokenization 5 for Korean text, however, the preliminary test did not produce effective results. We also investigated a parentheses imbalance problem. We solved the problem by filtering out parentheses-imbalanced translations from the nbest results. We found that the post-processing step can improve the BLEU score with low order language models, but cannot do with high order language models. We do not use the step for final submission. To boosting the performance, we combine the word-level phrase-based model (Word PB) and the character-level phrase-based model (Char PB). If there are one or more OOV words in an input sentence, our translator choose the Char PB model, otherwise, the Word PB model. Our Ko-Ja SMT system was developed by using the open source SMT engines; Moses and Giza++. Its other specifications are as follows: • Grow-diag-final-and word alignment heuristic • Good-Turing discounting for smoothing probabilities • Minimum Error Rate Training (MERT) for tuning feature weights Neural Machine Translation Neural machine translation (NMT) is a new approach to machine translation that has shown promising results compared to the existing approaches such as phrase-based statistical machine translation (Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015 ). An NMT system is a single neural network that reads a source sentence and generates its translation. Using the bilingual corpus, the whole neural network is jointly trained to maximize the conditional probability of a correct translation given a source sentence. NMT has several advantages over the existing statistical machine translation systems such as the phrase-based system. First, NMT uses minimal domain knowledge. Second, the NMT system is jointly trained to maximize the translation performance, unlike the existing phrase-based system which consists of many separately trained features. Third, the NMT system removes the need to store explicit phrase tables and language models. Lastly, the decoder of an NMT system is easy to implement. Despite these advantages and promising results, NMT has a limitation in handling a larger target vocabulary, as the complexity of training and decoding increases proportionally to the number of target words. In this paper, we propose a new approach to avoid the large target vocabulary problem by preprocessing the target word sequences, encoding them as a longer character sequence drawn from a small character vocabulary. The proposed approach removes the need to replace rare words with the unknown word symbol. Our approach is simpler than other methods recently proposed to address the same issue (Luong et al., 2015; Jean et al., 2015) . Model In this paper, we use our in-house software of NMT that uses an attention mechanism, as recently proposed by Bahdanau et al. (2015) . The encoder of NMT is a bi-directional recurrent neural network such that h t = [ h t ; h t ] (1) h t = f GRU (W s we x t , h t+1 ) (2) h t = f GRU (W s we x t , h t−1 ) (3) where h t is a hidden state of the encoder, x t is a one-hot encoded vector indicating one of the words in the source vocabulary, W s we is a weight matrix for the word embedding of the source language, and f GRU is a gated recurrent unit (GRU) (Cho et al., 2014) . At each time, the decoder of NMT computes the context vector c t as a convex combination of the hidden states (h 1 ,. . . ,h T ) with the alignment weights α 1 ,. . . ,α T : c t = T i=1 α ti h i (4) α ti = exp(e tj ) T j=1 exp(e tj ) (5) e ti = f F F N N (z t−1 , h i , y t−1 ) (6) where f F F N N is a feedforward neural network with a single hidden layer, z t−1 is a previous hidden state of the decoder, and y t−1 is a previous generated target word (one-hot encoded vector). A new hidden state z t of the decoder which uses GRU is computed based on z t−1 , y t−1 , and c t : z t = f GRU (y t−1 , z t−1 , c t ) (7) The probability of the next target word y t is then computed by (8) p(y t |y <t , x) = y T t f sof tmax {W z y z t + W zy z t + W cy c t + W yy (W t we y t−1 ) + b y } z t = f ReLU (W zz z t ) (9) where f sof tmax is a softmax function, f ReLU is a rectified linear unit (ReLU), W t we is a weight matrix for the word embedding of the target language, and b y is a target word bias. Settings We constructed the source word vocabulary with the most common words in the source language corpora. For the target character vocabulary, we used a BI (begin/inside) representation (e.g., 結/B, 果/I), because it gave better accuracy in preliminary experiment. The sizes of the source vocabularies for English and Korean were 245K and 60K, respectively, for the En-Ja and Ko-Ja tasks. The sizes of the target character vocabularies for Japanese were 6K and 5K, respectively, for the En-Ja and Ko-Ja tasks. We chose the dimensionality of the source word embedding and the target character embedding to be 200, and chose the size of the recurrent units to be 1,000. Each model was optimized using stochastic gradient descent (SGD). We did not use dropout. Training was early-stopped to maximize the performance on the development set measured by BLEU. Experimental Results All scores of this section are reported in experiments on the official test data; test.txt of the ASPEC-JE corpus. Table 1 shows the evaluation results of our En-Ja traditional SMT system. The first row in the table indicates the baseline of the tree-to-string systaxbased model. The second row shows the system that reflects the tree modification described in section 2.3. The augmentation method drastically increased both the number of rules and the BLEU score. Our OOV handling methods described in  The decoding time of the rule-augmented treeto-string SMT is about 1.3 seconds per a sentence in our 12-core machine. Even though it is not a terrible problem, we are required to improve the decoding speed by pruning the rule table or using the incremental decoding method (Huang and Mi, 2010) . Table 2 shows the evaluation results of our Ko-Ja traditional SMT system. We obtained the best result in the combination of two phrase-based SMT systems. Table 3 shows effects of our NMT model. "Human" indicates the pairwise crowdsourcing evaluation scores provided by WAT 2015 organizers. In the table, "T2S/PBMT only" is the final T2S/PBMT systems shown in section 5.1 and section 5.2. "NMT only" is the system using only RNN encoder-decoder without any traditional SMT methods. The last row is the combined system that reranks T2S/PBMT n-best translations by NMT. Our T2S/PBMT system outputs 100,000-best translations in En-Ja and 10,000best translations in Ko-Ja. The final output is 1best translation selected by considering only NMT score. En-Ja SMT Ko-Ja SMT NMT NMT outperforms the traditional SMT in En-Ja, while it does not in Ko-Ja. This result means that NMT produces a strong effect in the language pair with long linguistic distance. Moreover, the reranking system achieved a great synergy of T2S/PBMT and NMT in both task, even  if "NMT only" is not effective in Ko-Ja. From the human evaluation, we can be clear that our NMT model produces successful results. Conclusion This paper described NAVER machine translation system for En-Ja and Ko-Ja tasks at WAT 2015. We developed both the traditional SMT and NMT systems and integrated NMT into the traditional SMT in both tasks by reranking n-best translations of the traditional SMT. Our evaluation results showed that a combination of the NMT and traditional SMT systems outperformed two independent systems. For the future work, we try to improve the space and time efficiency of both the tree-to-string SMT and the NMT model. We also plan to develop and evaluate the NMT system in other language pairs.
1 2.1 2.2 2.3 2.4 2.4.1 3.1 3.2 3.3 4 4.1 4.2 5 5.3 6
Introduction Training data Language Analyzer Tree-to-string Syntax-based SMT Handling Out-of-Vocabulary English Spell Correction Training data Language Analyzer Phrase-based SMT Neural Machine Translation Model Settings Experimental Results NMT Conclusion
Experimental Results T2S Syntax based MT
+ Rule augmentation 1950M
+ Parameter modification 1950M
Rule augmentation increases both BLEU and #Rules
OOV handling improves the performance
+ Rule augmentation 1950M
+ Parameter modification 1950M
Rule augmentation increases both BLEU and #Rules
OOV handling improves the performance