Using Monolingual Data in Neural Machine Translation: a Systematic Study
Franck Burlot François Yvon


Simulations of Majorana spin flips in an antihydrogen trap
    F. Robicheaux
    December 30, 2023
==========================================================




  Neural Machine Translation (MT) has radically changed the way systems are developed. A
  major difference with the previous generation (Phrase-Based MT) is the way monolingual
  target data, which often abounds, is used in these two paradigms. While Phrase-Based MT
  can seamlessly integrate very large language models trained on billions of sentences, the
  best option for Neural MT developers seems to be the generation of artificial parallel
  data through back-translation - a technique that fails to fully take advantage
  of existing datasets.
  
  
  In this paper, we conduct a systematic study of back-translation, comparing alternative
  uses of monolingual data, as well as multiple data generation procedures. Our findings
  confirm that back-translation is very effective and give new explanations as to why
  this is the case. We also introduce new data simulation techniques that are almost as
  effective, yet much cheaper to implement.




§ INTRODUCTION 


The new generation of Neural Machine Translation (NMT) systems is known to be extremely
data hungry <cit.>. Yet, most existing NMT training pipelines fail to
fully take advantage of the very large volume of monolingual source and/or parallel data
that is often available. Making a better use of data is particularly critical in domain
adaptation scenarios, where parallel adaptation data is usually assumed to be small in
comparison to out-of-domain parallel data, or to in-domain monolingual texts. This
situation sharply contrasts with the previous generation of statistical MT engines
<cit.>, which could seamlessly integrate very large amounts of non-parallel 
documents, usually with a large positive effect on translation quality.

Such observations have been made repeatedly and have led to many innovative techniques to
integrate monolingual data in NMT, that we review shortly. The most successful approach to
date is the proposal of <cit.>, who use monolingual target texts to
generate artificial parallel data via backward translation (BT). This technique has since
proven effective in many subsequent studies.

It is however
very computationally costly, typically requiring to translate large sets of data. Determining 
the “right” amount (and quality) of BT data is another open issue, but we observe that
experiments reported in the literature only use a subset of the available monolingual 
resources. This suggests that standard recipes for BT might be sub-optimal.

This paper aims to better understand the strengths and weaknesses of
BT and to design more principled techniques to improve its effects. More
specifically, we seek to answer the following questions: since there are many ways to generate
pseudo parallel corpora, how important is the quality of this data for MT performance? Which
properties of back-translated sentences actually matter for MT quality? Does
BT act as some kind of regularizer <cit.>?
Can BT be efficiently simulated? Does BT data play the
same role as a target-side language modeling, or are they complementary? BT is often
used for domain adaptation: can the effect of having more in-domain data be sorted out
from the mere increase of training material <cit.>? 
For studies related to the impact of varying the size of BT data, we refer the readers to the recent work of <cit.>.




To answer these questions, we have reimplemented several strategies to use monolingual
data in NMT 


and have run experiments on two
language pairs in a very controlled setting (see § <ref>). Our main
results (see § <ref> and § <ref>)
suggest promising directions for efficient domain adaptation with cheaper techniques
than conventional BT.



































§ EXPERIMENTAL SETUP 




 §.§ In-domain and out-of-domain data 













We are mostly interested with the following training scenario: a large out-of-domain parallel corpus, and limited monolingual in-domain data. We 
focus here on the Europarl domain, for which we have ample
 data in several languages, and use as in-domain training data the
 Europarl corpus[Version 7, see <www.statmt.org/europarl/>.] <cit.> for two translation directions:
English→German and English→French. As we study the benefits of monolingual data, most of our experiments only use the target side of this corpus. The rationale for choosing this domain is to  (i)
to perform large scale comparisons of synthetic and natural parallel corpora; (ii) to study the
effect of BT in a well-defined domain-adaptation scenario. 
For both language pairs, we use the Europarl tests from 2007 and 2008[<www.statmt.org/wmt08>.]
for evaluation purposes, keeping test 2006 for development. When measuring out-of-domain
performance, we will use the WMT newstest 2014.





 §.§ NMT setups and performance 


Our baseline NMT system implements the attentional encoder-decoder
approach <cit.> as implemented in Nematus
<cit.> on 4 million out-of-domain parallel sentences.
For French we use samples from News-Commentary-11 and Wikipedia from WMT 2014 shared translation task,
as well as the Multi-UN <cit.> and EU-Bookshop <cit.> corpora.
For German, we use samples from News-Commentary-11, Rapid, Common-Crawl (WMT 2017) and Multi-UN
(see table <ref>).
Bilingual BPE units <cit.> are learned with 50k merge operations, yielding
vocabularies of about respectively 32k and 36k for English→French and 32k and 
44k for English→German. 






Both systems use 512-dimensional word embeddings and a single hidden layer with 1024 cells. They are
optimized using Adam <cit.> and early stopped according to the validation
performance. Training lasted for about three weeks on an Nvidia K80 GPU card.

Systems generating back-translated data are trained using the same out-of-domain corpus,
where we simply exchange the source and target sides. They are further documented in 
§ <ref>. 

For the sake of comparison, we also train a system that has access to a large batch of in-domain
parallel data following the strategy often referred to as “fine-tuning”:
upon convergence of the baseline model, we resume training
with a 2M sentence in-domain corpus mixed with an equal amount of randomly selected
out-of-domain natural sentences, with the same architecture and training parameters,
running validation every 2000 updates with a patience of 10.
Since BPE units are selected based only on the out-of-domain
statistics, fine-tuning is performed on sentences that are slightly longer (ie. they
contain more units) than for the initial training. This system defines an upper-bound of the
translation performance and is denoted below as .



Our baseline and topline results are in Table <ref>, where we
measure translation performance using BLEU <cit.>,
BEER <cit.> (higher is better) and characTER <cit.>
(smaller is better). As they are trained from much smaller amounts of data than current
systems, these baselines are not quite competitive to today's best system, but still represent
serious baselines for these datasets. Given our setups, fine-tuning with in-domain natural data improves 
BLEU by almost 4 points for both translation directions on in-domain tests; it also improves, albeit by a 
smaller margin, the BLEU score of the out-of-domain tests.

































§ USING ARTIFICIAL PARALLEL DATA IN NMT 


A simple way to use monolingual data in MT is to turn it into synthetic parallel data and let
the training procedure run as usual <cit.>. In this
section, we explore various ways to implement this strategy. We first reproduce results of
<cit.> with BT of various qualities, that we then analyze thoroughly. 









 §.§ The quality of Back-Translation 




  §.§.§ Setups

BT requires the availability of an MT system in the reverse translation direction. We consider here
three MT systems of increasing quality:


  * : this is a very poor SMT system trained using only 50k parallel
  sentences from the out-of-domain data, and no additional monolingual data. For
  this system as for the next one, we use Moses <cit.> out-of-the-box, computing
  alignments with Fastalign <cit.>, with a minimal pre-processing (basic tokenization). 
  This setting provides us with a
  pessimistic estimate of what we could get in low-resource conditions.
  
  
  
  
  

  * : these are much larger SMT systems, which use the same
  parallel data as the baseline NMTs (see § <ref>) and all the
  English monolingual data available for the WMT 2017 shared tasks, totalling
  approximately 174M sentences. These systems are strong, yet relatively cheap
  to build.

  * : these are the best NMT systems we could train, using
  settings that replicate the forward translation NMTs.




Note that we do not use any in-domain (Europarl) data to train these systems.
Their performance is reported in Table <ref>, where we observe
a 12 BLEU points gap between the worst and best systems (for both languages).

As noted eg. in <cit.>,
artificial parallel data obtained through forward-translation (FT) can also prove
advantageous and we also consider a FT system (): in this case the
target side of the corpus is artificial and is generated using the baseline NMT applied to
a natural source.



  §.§.§ BT quality does matter


Our results (see Table <ref>) replicate the findings of <cit.>: large gains can be
obtained from BT (nearly +2 BLEU in French and German); better artificial data
yields better translation systems. Interestingly, our best Moses system is almost as good
as the NMT and an order of magnitude faster to train. Improvements obtained with the bad system
are much smaller; contrary to the better MTs, this system is even detrimental for the out-of-domain test.

Gains with forward translation are significant, as in <cit.>, albeit
about half as good as with BT, and result in small improvements for the in-domain
and for the out-of-domain tests. Experiments combining forward and backward translation
(), each using a half of the available artificial data, do not outperform
the best BT results.

We finally note the large remaining difference between BT data and natural data, even though they 
only differ in their source side. This shows that at least in our domain-adaptation settings, BT
does not really act as a regularizer, contrarily to the findings of
<cit.>. Figure <ref> displays the learning curves
of these two systems. We observe that  improves quickly
in the earliest updates and then stays horizontal, whereas 
continues improving, even after 400k updates. Therefore BT
does not help to avoid overfitting, it actually encourages it, which may be due “easier”
training examples (cf. § <ref>).





 §.§ Properties of back-translated data 


Comparing the natural and artificial sources of our parallel data wrt. several
linguistic and distributional properties, we observe that (see Fig. <ref> - <ref>):



  (i) artificial sources are on average shorter than natural ones: when using BT, cases where the source is
  shorter than the target are rarer; cases when they have the same length are more frequent.

  (ii) automatic word alignments between artificial sources tend to be more
  monotonic than when using natural sources, as measured by the average Kendall τ
  of source-target alignments <cit.>: for French-English the respective
  numbers are 0.048 (natural) and 0.018 (artificial); for German-English 0.068 and
  0.053. Using more monotonic sentence pairs turns out to be a facilitating factor for NMT, 
  as also noted by <cit.>.

  (iii) 


  syntactically, artificial sources are simpler than real data;
  We observe significant differences in the distributions of tree depths.[Parses were automatically computed with CoreNLP <cit.>.]
  

 
  (iv) distributionally, plain word occurrences in artificial sources are more concentrated; this also translates into
   both a slower increase of the number of types wrt. the number of sentences and a smaller 
   number of rare events. 




















The intuition is that properties (i) and (ii) should help translation as compared to natural
source, while property (iv) should be detrimental. We checked (ii) by building systems with only 10M
words from the natural parallel data selecting these data either randomly or based on the regularity of their
word alignments. Results in Table <ref> show that the latter is much preferable for the 
overall performance. This might explain that the mostly monotonic BT from Moses are 
almost as good as the fluid BT from NMT and that both boost the baseline.








§ STUPID BACK-TRANSLATION 


We now analyze the effect of using much simpler data generation schemes,
which do not require the availability of a backward translation engine. 




 §.§ Setups

We use the following cheap ways to generate pseudo-source texts:


  * : in this setting, the source side is a mere copy of the target-side
  data. Since the source vocabulary of the
  NMT is fixed, copying the target sentences
  can cause the occurrence of OOVs.
  To avoid this situation, <cit.> decompose
  the target words into source-side units to make the copy look like
  source sentences. Each OOV found in the copy is split
  into smaller units until all the resulting chunks are
  in the source vocabulary.

  * : another way to integrate copies without having to deal with
  OOVs is to augment the source vocabulary with a copy of the target vocabulary. In this
  setup, <cit.> ensure that both vocabularies never overlap by marking the target
  word copies with a special language identifier. Therefore the English word resume
  cannot be confused with the homographic French word, which is marked @fr@resume.

  * : instead of using actual copies, we replace each word with
  “dummy” tokens. We use this unrealistic setup to observe the training over noisy
  and hardly informative source sentences.


We then use the procedures described in § <ref>, except that the pseudo-source
embeddings in the  setup are pretrained for
three epochs on the in-domain data, while all remaining parameters are frozen. This prevents 
random parameters from hurting the already trained model.



 §.§ Copy+marking+noise is not so stupid


We observe that the  setup has only a small
impact on the English-French system, for which the
baseline is already strong. This is less true for
English-German where simple copies yield a significant 
improvement. Performance drops for both language pairs in the
 setup.



We achieve our best gains with the  setup, which is the best way to use a copy of the target (although
the performance on the out-of-domain tests is at most the same as the baseline).
Such gains may look surprising, since the NMT model does not need to learn to translate but only
to copy the source. This is indeed what happens: to confirm
this, we built a fake test set having identical source and target side (in French). The average cross-entropy for this test set is 0.33, very close to 0, to be compared with an average cost of 58.52 when we process an actual source (in English). This means that the model
has learned to copy words from source to target with no difficulty, even for sentences not seen in training. 
A follow-up question is whether training a copying
task instead of a translation task limits the improvement: would the NMT learn better if the task
was harder? To measure this, we introduce noise in the target sentences copied onto the
source, following the procedure of <cit.>: it deletes random words and 
performs a small random permutation of the remaining words. Results
(+ Source noise) show no difference for the French in-domain test sets, but bring the
out-of-domain score to the level of the baseline. Finally, we observe a significant
improvement on German in-domain test sets, compared to the baseline (about +1.5 BLEU).
This last setup is even almost as good as the  condition
(see § <ref>) for German. This shows that learning to reorder
and predict missing words can more effectively serve our purposes
than simply learning to copy.








§ TOWARDS MORE NATURAL PSEUDO-SOURCES 


Integrating monolingual data into NMT can be as easy as copying the target into
the source, which already gives some improvement; adding noise makes things even better. 
We now study ways to make pseudo-sources look more like natural data, using the
framework of Generative Adversarial Networks (GANs) <cit.>, an idea
borrowed from <cit.>[Our implementation is available at
<https://github.com/franckbrl/nmt-pseudo-source-discriminator>]. 

















 §.§ GAN setups




In our setups, we use a marked target copy, viewed as a fake source, which a
generator encodes so as to fool a discriminator trained to distinguish a fake from
a natural source. Our architecture contains two distinct encoders, one for the
natural source and one for the pseudo-source. The latter acts as the generator (G) in
the GAN framework, computing a representation of the pseudo-source that is then input
to a discriminator (D), which has to sort natural from artificial encodings. 
D assigns a probability of a sentence being natural.





During training, the cost of the discriminator is computed over two batches, one with
natural (out-of-domain) sentences x and one with (in-domain) pseudo-sentences x^'.
The discriminator is a bidirectional-Recurrent Neural Network (RNN) of dimension 1024.
Averaged states are passed to a single feed-forward
layer, to which a sigmoid is applied. It inputs encodings of natural (E(x)) and
pseudo-sentences (G(x^')) and is trained to optimize:





    J^(D) =     - 1/2𝔼_x∼ p_reallog D(E(x)) 
        - 1/2𝔼_x^'∼ p_pseudolog (1 - D(G(x^')))




G's parameters are updated to maximally fool D, thus the loss J^(G):

    J^(G) = - 𝔼_x^'∼ p_pseudolog D(G(x^'))


Finally, we keep the usual MT objective.  (s is a real or pseudo-sentence):

    J^(MT) = log p(y | s) =  - 𝔼_s∼ p_alllogMT(s)

We thus need to train three sets of parameters: θ^(D), θ^(G) and θ^(MT) (MT parameters), with
θ^(G)∈θ^(MT). 
The pseudo-source encoder and embeddings are updated wrt. both J^(G) and J^(MT).
Following <cit.>, θ^(G) is updated only when D's
accuracy exceeds 75%. On the other hand, θ^(D) is not updated
when its accuracy exceeds 99%. At each update, two batches are generated for each
type of data, which are encoded with the real or pseudo-encoder.  The encoder outputs
serve to compute J^(D) and J^(G). Finally, the pseudo-source is
encoded again (once G is updated), both encoders are plugged into the translation model
and the MT cost is back-propagated down to the real and pseudo-word embeddings.
Pseudo-encoder and discriminator parameters are pre-trained for 10k updates.
At test time, the pseudo-encoder is ignored and inference is run as usual.





 §.§ GANs can help


Results are in Table <ref>, assuming the same fine-tuning procedure as above.
On top of the  setup, our GANs
do not provide any improvement in both language
pairs, with the exception of a small improvement for English-French
on the out-of-domain test, which we understand as a sign
that the model is more robust to domain variations, just like
when adding pseudo-source noise. When combined with noise,
the French model yields the best performance we could
obtain with stupid BT on the in-domain tests, at least in terms of BLEU and BEER.
On the News domain, we remain close to the baseline level,
with slight improvements in German.

A first observation is that this method brings stupid BT models closer to conventional BT, at a greatly reduced computational cost.
While French still remains 0.4 to 1.0 BLEU below very good backtranslation, both approaches are in the same ballpark for German

- may be because BTs are better for the former system than for the latter.

Finally note that the GAN architecture has two differences with basic : (a) a distinct encoder for real and pseudo-sentence; (b) a different training regime for these encoders. To sort out the effects of (a) and (b), we reproduce the GAN setup with BT sentences, instead of copies. Using a separate encoder for the pseudo-source in the  setup can be detrimental to performance  (see Table <ref>): translation degrades in French for all metrics.
Adding GANs on top of the pseudo-encoder was not able to make up for the degradation observed in French, but allowed the German system to slightly
outperform .

Even though this setup is unrealistic and overly costly, it shows that GANs are actually helping even good systems.



§ USING TARGET LANGUAGE MODELS 


In this section, we compare the previous methods with the use of a target side Language Model (LM).




Several proposals exist in the literature to integrate LMs in NMT:
for instance, <cit.> strengthen the decoder by integrating an extra, source
independent, RNN layer in a conventional NMT architecture. 
Training is performed either with parallel, or monolingual data. In the latter case, word prediction only relies on the
source independent part of the network. 



 §.§ LM Setup

We have followed <cit.> and
reimplemented[Our implementation is part of the Nematus toolkit (theano branch):
<https://github.com/EdinburghNLP/nematus/blob/theano/doc/deep_fusion_lm.md>]
their  technique. It
requires to first independently learn a RNN-LM on the in-domain target
data with a cross-entropy objective; then to train the optimal combination 
of the translation and the language models by adding
the hidden state of the RNN-LM as an additional input to the softmax layer of the
decoder. 

Our RNN-LMs are trained using dl4mt[<https://github.com/nyu-dl/dl4mt-tutorial>]
with the target side of the parallel data and
the Europarl corpus (about 6M sentences for both French and German),
using a one-layer GRU with the same dimension as the MT decoder
(1024). 








 §.§ LM Results





Results are in Table <ref>.



They show that  hardly improves
the Europarl results, while we obtain about +0.6 BLEU 
over the baseline on newstest-2014 for both languages.
 differs from stupid BT in that the model
is not directly optimized on the in-domain data, but uses the
LM trained on Europarl to maximize the likelihood
of the out-of-domain training data. Therefore, no specific
improvement is to be expected in terms of domain adaptation,
and the performance increases in the more general domain.
Combining  and  + noise + GANs
brings slight improvements on the German in-domain test sets,
and performance out of the domain remains near the baseline level.















§ RE-ANALYZING THE EFFECTS OF BT 




As a follow up of previous discussions, we analyze the effect of BT on the internals of the network.
Arguably, using a copy of the target sentence instead of a natural source should not be helpful
for the encoder, but is it also the case with a strong BT? What are the effects on the attention model?



 §.§ Parameter freezing protocol


To investigate these questions, we run the same fine-tuning using the , 
 and  setups. Note that except for the last one, 
all training scenarios have access to same target training data. We intend
to see whether the overall performance of the NMT system degrades when we
selectively freeze certain sets of parameters, meaning that they are not updated during fine-tuning.



 §.§ Results

BLEU scores are in Table <ref>. 
The  setup is hardly impacted by selective updates: updating
the only decoder leads to a degradation of at most 0.2 BLEU.  
For , we were not able to freeze the source embeddings, since these are initialized when fine-tuning begins and
therefore need to be trained. We observe that freezing the encoder and/or the attention
parameters has no impact on the English-German system, whereas it slightly degrades the
English-French one.  This suggests that using artificial sources, even of the poorest quality, has a
positive impact on all the components of the network, which makes another big difference
with the LM integration scenario. 

The largest degradation is for , where the model is prevented from learning from
informative source sentences, which leads to a decrease of 0.4 to over 1.0 BLEU. We
assume from these experiments that BT impacts most of all the decoder, and learning to
encode a pseudo-source, be it a copy or an actual back-translation, only marginally helps to
significantly improve the quality. Finally, in the  setup,
freezing the decoder does not seem to harm learning with a natural source.







§ RELATED WORK


The literature devoted to the use of monolingual data is large, and quickly expanding. We already alluded to several possible ways to use such data: using back- or forward-translation or using a target language model. The former approach is mostly documented in <cit.>, and recently analyzed in <cit.>, which focus on fully artificial settings as well as pivot-based artificial data; and <cit.>, which studies the effects of increasing the size of BT data. The studies of <cit.> also consider forward translation and <cit.> expand these results to domain adaptation scenarios. Our results are complementary to these earlier studies.
 
As shown above, many alternatives to BT exist. The most obvious is to use target LMs <cit.>, as we have also done here; but attempts to improve the encoder using multi-task learning also exist <cit.>.


This investigation is also related to recent attempts to consider supplementary data with a valid
target side, such as multi-lingual NMT <cit.>, where source
texts in several languages are fed in the same encoder-decoder architecture, with
partial sharing of the layers. This is another realistic scenario where additional resources can be used 
to selectively improve parts of the model.

Round trip training is another important source of inspiration, as it can be viewed as a way to use BT to perform  semi-unsupervised <cit.> or unsupervised <cit.> training of NMT. The most convincing attempt to date along these lines has been proposed by <cit.>, who propose to use GANs to mitigate the difference between artificial and natural data.



§ CONCLUSION 


In this paper, we have analyzed various ways
to integrate monolingual data in an NMT framework, focusing
on their impact on quality and domain adaptation. While confirming the effectiveness of BT, our study also
proposed significantly cheaper ways to improve the baseline performance,
using a slightly modified copy of the target, instead of its full BT.
When no high quality BT is available, using GANs to
make the pseudo-source sentences closer to natural source sentences
is an efficient solution for domain adaptation.

To recap our answers to our initial questions: the quality of BT actually matters for NMT (cf. § <ref>) and it seems that, even though artificial source are lexically less diverse and syntactically complex than real sentence, their monotonicity is a facilitating factor. We have studied cheaper alternatives and found out that copies of the target, if properly noised (§ <ref>), and even better, if used with GANs, could be almost as good as low quality BTs (§ <ref>): BT is only worth its cost when good BT can be generated. Finally, BT seems preferable to integrating external LM - at least in our data condition (§ <ref>). Further experiments with larger LMs are needed to confirm this observation, and also to evaluate the complementarity of both strategies. More work is needed to better understand the impact of BT on subparts of the network (§ <ref>). 

In future work, we plan to investigate other cheap ways to generate artificial data. The experimental setup we proposed
may also benefit from a refining of the data selection strategies to focus on the most useful monolingual sentences. 







acl_natbib_nourl



