Recurrent Continuous Translation Models - ACL ?· cently further developed in (Schwenk, 2012; Le et…

  • Published on

  • View

  • Download

Embed Size (px)


  • Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 17001709,Seattle, Washington, USA, 18-21 October 2013. c2013 Association for Computational Linguistics

    Recurrent Continuous Translation Models

    Nal Kalchbrenner Phil BlunsomDepartment of Computer Science

    University of Oxford{nal.kalchbrenner,phil.blunsom}


    We introduce a class of probabilistic con-tinuous translation models called Recur-rent Continuous Translation Models that arepurely based on continuous representationsfor words, phrases and sentences and do notrely on alignments or phrasal translation units.The models have a generation and a condi-tioning aspect. The generation of the transla-tion is modelled with a target Recurrent Lan-guage Model, whereas the conditioning on thesource sentence is modelled with a Convolu-tional Sentence Model. Through various ex-periments, we show first that our models ob-tain a perplexity with respect to gold transla-tions that is > 43% lower than that of state-of-the-art alignment-based translation models.Secondly, we show that they are remarkablysensitive to the word order, syntax, and mean-ing of the source sentence despite lackingalignments. Finally we show that they match astate-of-the-art system when rescoring n-bestlists of translations.

    1 Introduction

    In most statistical approaches to machine transla-tion the basic units of translation are phrases that arecomposed of one or more words. A crucial com-ponent of translation systems are models that esti-mate translation probabilities for pairs of phrases,one phrase being from the source language and theother from the target language. Such models countphrase pairs and their occurrences as distinct if thesurface forms of the phrases are distinct. Althoughdistinct phrase pairs often share significant similari-

    ties, linguistic or otherwise, they do not share statis-tical weight in the models estimation of their trans-lation probabilities. Besides ignoring the similar-ity of phrase pairs, this leads to general sparsity is-sues. The estimation is sparse or skewed for thelarge number of rare or unseen phrase pairs, whichgrows exponentially in the length of the phrases, andthe generalisation to other domains is often limited.

    Continuous representations have shown promiseat tackling these issues. Continuous representationsfor words are able to capture their morphological,syntactic and semantic similarity (Collobert and We-ston, 2008). They have been applied in continu-ous language models demonstrating the ability toovercome sparsity issues and to achieve state-of-the-art performance (Bengio et al., 2003; Mikolov etal., 2010). Word representations have also showna marked sensitivity to conditioning information(Mikolov and Zweig, 2012). Continuous repre-sentations for characters have been deployed incharacter-level language models demonstrating no-table language generation capabilities (Sutskever etal., 2011). Continuous representations have alsobeen constructed for phrases and sentences. The rep-resentations are able to carry similarity and task de-pendent information, e.g. sentiment, paraphrase ordialogue labels, significantly beyond the word leveland to accurately predict labels for a highly diverserange of unseen phrases and sentences (Grefenstetteet al., 2011; Socher et al., 2011; Socher et al., 2012;Hermann and Blunsom, 2013; Kalchbrenner andBlunsom, 2013).

    Phrase-based continuous translation models werefirst proposed in (Schwenk et al., 2006) and re-


  • cently further developed in (Schwenk, 2012; Le etal., 2012). The models incorporate a principled wayof estimating translation probabilities that robustlyextends to rare and unseen phrases. They achievesignificant Bleu score improvements and yield se-mantically more suggestive translations. Althoughwide-reaching in their scope, these models are lim-ited to fixed-size source and target phrases and sim-plify the dependencies between the target words tak-ing into account restricted target language modellinginformation.

    We describe a class of continuous translationmodels called Recurrent Continuous TranslationModels (RCTM) that map without loss of generalitya sentence from the source language to a probabil-ity distribution over the sentences in the target lan-guage. We define two specific RCTM architectures.Both models adopt a recurrent language model forthe generation of the target translation (Mikolov etal., 2010). In contrast to other n-gram approaches,the recurrent language model makes no Markov as-sumptions about the dependencies of the words inthe target sentence.

    The two RCTMs differ in the way they condi-tion the target language model on the source sen-tence. The first RCTM uses the convolutional sen-tence model (Kalchbrenner and Blunsom, 2013) totransform the source word representations into a rep-resentation for the source sentence. The source sen-tence representation in turn constraints the genera-tion of each target word. The second RCTM intro-duces an intermediate representation. It uses a trun-cated variant of the convolutional sentence model tofirst transform the source word representations intorepresentations for the target words; the latter thenconstrain the generation of the target sentence. Inboth cases, the convolutional layers are used to gen-erate combined representations for the phrases in asentence from the representations of the words in thesentence.

    An advantage of RCTMs is the lack of latentalignment segmentations and the sparsity associatedwith them. Connections between source and targetwords, phrases and sentences are learnt only implic-itly as mappings between their continuous represen-tations. As we see in Sect. 5, these mappings of-ten carry remarkably precise morphological, syntac-tic and semantic information. Another advantage is

    that the probability of a translation under the modelsis efficiently computable requiring a small numberof matrix-vector products that is linear in the lengthof the source and the target sentence. Further, trans-lations can be generated directly from the probabil-ity distribution of the RCTM without any externalresources.

    We evaluate the performance of the models in fourexperiments. Since the translation probabilities ofthe RCTMs are tractable, we can measure the per-plexity of the models with respect to the referencetranslations. The perplexity of the models is signifi-cantly lower than that of IBM Model 1 and is> 43%lower than the perplexity of a state-of-the-art variantof the IBM Model 2 (Brown et al., 1993; Dyer etal., 2013). The second and third experiments aim toshow the sensitivity of the output of the RCTM IIto the linguistic information in the source sentence.The second experiment shows that under a randompermutation of the words in the source sentences,the perplexity of the model with respect to the refer-ence translations becomes significantly worse, sug-gesting that the model is highly sensitive to wordposition and order. The third experiment inspectsthe translations generated by the RCTM II. Thegenerated translations demonstrate remarkable mor-phological, syntactic and semantic agreement withthe source sentence. Finally, we test the RCTMson the task of rescoring n-best lists of translations.The performance of the RCTM probabilities joinedwith a single word penalty feature matches the per-formance of the state-of-the-art translation systemcdec that makes use of twelve features includingfive alignment-based translation models (Dyer et al.,2010).

    We proceed as follows. We begin in Sect. 2 bydescribing the general modelling framework under-lying the RCTMs. In Sect. 3 we describe the RCTMI and in Sect. 4 the RCTM II. Section 5 is dedicatedto the four experiments and we conclude in Sect. 6.1

    2 Framework

    We begin by describing the modelling frameworkunderlying RCTMs. An RCTM estimates the proba-bility P (f|e) of a target sentence f = f1, ..., fm beinga translation of a source sentence e = e1, ..., ek. Let

    1Code and models available at


  • us denote by fi:j the substring of words fi, ..., fj . Us-ing the following identity,

    P (f|e) =m


    P (fi|f1:i1, e) (1)

    an RCTM estimates P (f|e) by directly computingfor each target position i the conditional probabilityP (fi|f1:i1, e) of the target word fi occurring in thetranslation at position i, given the preceding targetwords f1:i1 and the source sentence e. We see thatan RCTM is sensitive not just to the source sentencee but also to the preceding words f1:i1 in the targetsentence; by doing so it incorporates a model of thetarget language itself.

    To model the conditional probability P (f|e), anRCTM comprises both a generative architecture forthe target sentence and an architecture for condition-ing the latter on the source sentence. To fully cap-ture Eq. 1, we model the generative architecture witha recurrent language model (RLM) based on a re-current neural network (Mikolov et al., 2010). Theprediction of the i-th word fi in a RLM depends onall the preceding words f1:i1 in the target sentenceensuring that conditional independence assumptionsare not introduced in Eq. 1. Although the predic-tion is most strongly influenced by words closelypreceding fi, long-range dependencies from acrossthe whole sentence can also be exhibited. The con-ditioning architectures are model specific and aretreated in Sect. 3-4. Both the generative and con-ditioning aspects of the models deploy continuousrepresentations for the constituents and are trainedas a single joint architecture. Given the modellingframework underlying RCTMs, we now proceed todescribe in detail the recurrent language model un-derlying the generative aspect.

    2.1 Recurrent Language ModelA RLM models the probability P (f) that the se-quence of words f occurs in a given language. Letf = f1, ..., fm be a sequence of m words, e.g. a sen-tence in the target language. Analogously to Eq. 1,using the identity,

    P (f) =m


    P (fi|f1:i1) (2)

    the model explicitly computes without simpli-fying assumptions the conditional distributions


    I O

    fi P(f )i+1



    fi-1 P(f )i fi+1 P(f )i+2


    h h hi-1 i i+1

    Figure 1: A RLM (left) and its unravelling to depth 3(right). The recurrent transformation is applied to the hid-den layer hi1 and the result is summed to the represen-tation for the current word fi. After a non-linear transfor-mation, a probability distribution over the next word fi+1is predicted.

    P (fi|f1:i1). The architecture of a RLM comprisesa vocabulary V that contains the words fi of thelanguage as well as three transformations: an in-put vocabulary transformation I Rq|V |, a re-current transformation R Rqq and an outputvocabulary transformation O R|V |q. For eachword fk V , we indicate by i(fk) its index in Vand by v(fk) R|V |1 an all zero vector with onlyv(fk)i(fk) = 1.

    For a word fi, the result of I v(fi) Rq1 isthe input continuous representation of fi. The pa-rameter q governs the size of the word representa-tion. The prediction proceeds by successively ap-plying the recurrent transformation R to the wordrepresentations and predicting the next word at eachstep. In detail, the computation of each P (fi|f1:i1)proceeds recursively. For 1 < i < m,

    h1 = (I v(f1)) (3a)hi+1 = (R hi + I v(fi+1)) (3b)oi+1 = O hi (3c)

    and the conditional distribution is given by,

    P (fi = v|f1:i1) =exp (oi,v)Vv=1 exp(oi,v)


    In Eq. 3, is a nonlinear function such as tanh. Biasvalues bh and bo are included in the computation. Anillustration of the RLM is given in Fig. 1.

    The RLM is trained by backpropagation throughtime (Mikolov et al., 2010). The error in the pre-dicted distribution calculated at the output layer is


  • backpropagated through the recurrent layers and cu-mulatively added to the errors of the previous predic-tions for a given number d of steps. The procedure isequivalent to standard backpropagation over a RLMthat is unravelled to depth d as in Fig. 1.

    RCTMs may be thought of as RLMs, in whichthe predicted distributions for each word fi are con-ditioned on the source sentence e. We next definetwo conditioning architectures each giving rise to aspecific RCTM.

    3 Recurrent Continuous TranslationModel I

    The RCTM I uses a convolutional sentence model(CSM) in the conditioning architecture. The CSMcreates a representation for a sentence that is pro-gressively built up from representations of the n-grams in the sentence. The CSM embodies a hierar-chical structure. Although it does not make use of anexplicit parse tree, the operations that generate therepresentations act locally on small n-grams in thelower layers of the model and act increasingly moreglobally on the whole sentence in the upper layersof the model. The lack of the need for a parse treeyields two central advantages over sentence modelsthat require it (Grefenstette et al., 2011; Socher etal., 2012). First, it makes the model robustly appli-cable to a large number of languages for which accu-rate parsers are not available. Secondly, the transla-tion probability distribution over the target sentencesdoes not depend on the chosen parse tree.

    The RCTM I conditions the probability of eachtarget word fi on the continuous representation of thesource sentence e generated through the CSM. Thisis accomplished by adding the sentence representa-tion to each hidden layer hi in the target recurrentlanguage model. We next describe the procedure inmore detail, starting with the CSM itself.

    3.1 Convolutional Sentence Model

    The CSM models the continuous representation ofa sentence based on the continuous representationsof the words in the sentence. Let e = e1...ek bea sentence in a language and let v(ei) Rq1 bethe continuous representation of the word ei. LetEe Rqk be the sentence matrix for e defined by,

    Ee:,i = v(ei) (5)

    (K M)* :,1

    M M M:,1 :,2 :,3the cat sat on the mat


    E eK 2

    K 3

    L 3

    K i :,1 K i :,2 K i



    Figure 2: A CSM for a six word source sentence e and thecomputed sentence representation e. K2,K3 are weightmatrices and L3 is a top weight matrix. To the right, aninstance of a one-dimensional convolution between someweight matrix Ki and a generic matrix M that could forinstance correspond to Ee2. The color coding of weightsindicates weight sharing.

    The main component of the architecture of the CSMis a sequence of weight matrices (Ki)2ir that cor-respond to the kernels or filters of the convolutionand can be thought of as learnt feature detectors.From the sentence matrix Ee the CSM computes acontinuous vector representation e Rq1 for thesentence e by applying a sequence of convolutionsto Ee whose weights are given by the weight matri-ces. The weight matrices and the sequence of con-volutions are defined next.

    We denote by (Ki)2ir a sequence of weightmatrices where each Ki Rqi is a matrix of icolumns and r = d

    2Ne, where N is the length of

    the longest source sentence in the training set. Eachrow of Ki is a vector of i weights that is treated asthe kernel or filter of a one-dimensional convolution.Given for instance a matrix M Rqj where thenum...


View more >