# Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for
English to Indian Languages

Baban Gain  0000-0001-8673-7078 ,  Dibyanayan Bandyopadhyay  ,  Samrat
Mukherjee  ,  Chandranath Adak  0000-0002-9085-2770 and  Asif Ekbal  Indian
Institute of Technology, PatnaDepartment of Computer Science &
EngineeringBihtaBiharIndia801106<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Utilizing multimodal information in Neural Machine Translation (NMT) has
demonstrated its effectiveness in improving translation outcomes. Previous
studies predominantly focused on low-resource scenarios, employing multimodal
translation data as the sole training corpus. Here, incorporating image
features aids in disambiguating unclear words, gender pronouns, deixis,
ellipsis scenarios, etc. However, exploring the transfer of knowledge from
pre-trained unimodal NMT systems remains largely unexplored. In this study, we
investigate the impact of multimodal context in Multimodal NMT (MMT) atop a
robust multilingual unimodal baseline. We specifically explore the importance
of image features when the model is pre-trained on large-scale unimodal data
and has acquired the ability to disambiguate tokens within sentences.
Surprisingly, our findings reveal that images are plausibly redundant when
leveraging an extensive pre-trained unimodal NMT system. Moreover, we
introduce synthetic noise into the dataset to investigate whether the
contextual information from images can assist the model in mitigating the
effects of textual noise. Notably, our observations indicate that a multimodal
model outperforms a text-based model by a slight margin in noisy settings.
However, even the model utilizing random images surpasses the text-based
baseline in noisy conditions. We present our research findings on translating
from English to three Indian languages, namely Hindi, Bengali, and Malayalam.
Significantly, our results surpass the state-of-the-art benchmarks on each
dataset by a considerable margin, emphasizing our contributions and findings.
Furthermore, our study uncovers an intriguing aspect, i.e., the impact of
visual context varies with the noise level in the source text. Specifically,
the absence of visual context yields the best performance for non-noisy
translations. However, in cases with lower noise levels, cropped image
features produce optimal results, while full image features are more effective
in high-noise settings. This observation highlights the crucial role of visual
context, particularly in noisy scenarios, thereby shedding light on an
unexplored research direction, i.e., Noisy Neural Machine Translation in
multimodal settings. Overall, our research provides novel insights into the
usage of multimodal context in NMT, extending beyond low-resource scenarios.
Our findings underscore the importance of considering the interplay between
visual and textual information, paving the way for enhanced translation
systems in noisy environments.

context aware translation, multi-encoder, chat translation

††ccs: Computing methodologies Machine translation

##  1\. Introduction

Unimodal Neural Machine Translation (UNMT) systems generate good-quality
outputs in most domains and languages. However, these models cannot translate
ambiguous words needing additional information to disambiguate. This issue is
amplified when the dataset is noisy, containing spelling and grammatical
mistakes. Further, UNMT models fail to generate context-dependent translation.
Multimodal Neural Machine Translation (MMT) is very important in the era of
social media.

In MMT, an image is used as context to assist the translation process. For
example, in a sentence ”The color of the court is red.”, the word court can be
interpreted in multiple ways. One of them is a playground for indoor games
like tennis or badminton, whereas another interpretation could indicate a
judicial court. While there exist several techniques to tackle MMT, most of
them are applied to high-resource language pairs like English-German or
English-French. Further, those languages belong to the same linguistic family
and use the same latin script, and often share common words. Translating
between distant languages, e.g., English-Hindi, is more challenging, where the
number of shared words is quite lesser. Further, Hindi sentences are gender-
dependent, while English is not (Comrie, 2009). For example, the translation
of ”I am going to market.” will be different based on if the subject is male
or female. There have been a few attempts to use multimodal information to
translate sentences from English to Hindi. However, only a few of them tried
to take advantage of large-scale unimodal pre-training. Most of the existing
methods either do not use large-scale unimodal data at all or use it for very
limited purposes like training word embedding. While multimodal translation is
distinct from unimodal translation, training trained encoders and decoders
with uni-modal data could help to achieve better translation. Although object
tags are useful for generating better translation, only using object tags
without additional information from the image might not be ideal due to the
following reasons:

Figure 1. Example of combined Hindi, Bengali, and Malayalam dataset

  * •

Only object tags (such as a _person, ball, tree,_ etc.) are connected to the
source; other details about the objects and images that could help with
translation are not used.

  * •

If there is not any additional information about the objects, their locations,
or their interactions, the model may use object tags misleadingly. The model
will not be able to grasp whether the subject is male or female, for instance,
if there is another male person in the image and the subject is female, as
both are present in the list of objects, regardless of their action.

  * •

For every detected object, only one tag with the best confidence is chosen.
The misclassified tags may mislead the NMT model because the selection of a
tag is discrete, and the object detection model is susceptible to errors.

  * •

Due to the object detection model’s inadequate training, many objects may
remain unclassified.

The aforementioned factors prevented object tag-based models from consistently
improving the results (Gupta et al., 2021). Therefore, it is preferable to use
and incorporate visual features that have been extracted from pre-trained
image models, e.g., Vision Transformer (ViT), VGG, ResNet (He et al., 2015),
etc. When source texts are hidden, some probing tasks demonstrate how MMT
models can retrieve missing data, such as color, person, etc., from visual
features. It is neither a realistic strategy nor a guarantee of success in
scenarios involving incomplete information. The model can learn that some
information is missing by masking a portion of the source and assigning a
corresponding mask token. This information can then be retrieved from the
image. There are tokens at the proper location for probing tasks. Any mask
token will not be present in real-world datasets, where some of the
information is missing from the source text, and it may be challenging to
infer that the information is insufficient. To summarize, efforts have been
made to make use of visual features in the MT system directly. The majority of
them only use multimodal data, which is scarcely available. Additionally,
other techniques that rely on strong baselines only use image object tags when
fine-tuning. In this study, we attempt to investigate the circumstances where
the visual feature has a positive impact on MMT systems. We use large pre-
trained models as a baseline such that the models are robust enough to
distinguish words from intra-sentence context mostly. We observe that images
have a negligible contribution in non-noisy settings. We further observe that
the models are able to utilize cropped image features in low-noisy settings,
and full image features are more effective in high-noisy settings. Our study
indicates that visual features’ impact on the model performance increases with
noise in the source text. This finding may aid in noisy caption translation
from social media data. The major contributions of this paper are briefly
mentioned below.

(i) Noisy MMT: We develop our systems for noisy MMT. To the best of our
knowledge, this is the earliest attempt of its kind to translate noisy
utterances in multimodal settings. We observe the increased positive impact of
visual context in the presence of noise in the dataset.

(ii) Multilingual MMT: We perform multilingual multimodal fine-tuning, as well
as use a multilingual pre-trained model. This is the first attempt at the
multilingual scenario that includes Indic languages.

(iii) Probing the Need of Image: By substituting the relevant images
corresponding to the text with random images, we inspected to determine the
effectiveness of the employed relevant image. This is the primary attempt of
its kind, which carries out such probing in Indian languages.

(iv) Strong UNMT Baseline: We use the best off-the-shelf UNMT model for fine-
tuning. This is crucial to understand if the multimodal methods achieve better
results due to extracted information from the corresponding images, or a lack
of competency of the employed text-processing module to handle ambiguity.

(v) Enhanced Visual Features in Indian Settings: We extracted rich visual
features from ViT instead of the CNN-based features used in the contemporary
Indic MMT methods.

(vi) State-of-the-Art Results: This work mainly attempts to comprehend the
impact of visual features on multimodal English to Indic language translation
in the presence of noise. Please note that our primary focus has been to
develop the MT system in noisy settings rather than achieving only state-of-
the-art (SOTA) performance. However, our model obtained SOTA results for all
three target Indic languages (Hindi, Bengali, and Malayalam) while English was
the source language. We make our codes publicly
available111https://github.com/babangain/indicMMT.

The rest of the paper is organized as follows. In Section 2, we discuss
related MMT methods along with models for Indic languages and noisy text
translation. Thereafter, Section 3 explains the employed dataset and the noise
addition procedure. Subsequently, Section 4 mentions our methodology, and
Section 5 discusses results with observations. Finally, Section 6 concludes
this paper.

##  2\. Related Works

There is a scarcity of large-scale datasets for source-target-image triplets.
However, there are several attempts to tackle challenges in MMT.

###  2.1. Multimodal Translation

One of the earliest attempts for MMT was to integrate visual features with
shared attention mechanism (Caglayan et al., 2016). However, it did not result
in performance improvement. Other early attempts include the concatenation of
text features with global and regional visual features (Huang et al., 2016) to
generate Multimodal Translation. Here, the regional features are treated as a
token of the input sentence. Subsequently, a multi-task-based training
framework (Elliott and Kádár, 2017) was employed with a shared encoder where
the tasks included standard NMT tasks and visual feature reconstruction. The
following attempts to tackle challenges included consideration of visual
features as pseudo-text features (Calixto and Liu, 2017). First, they
projected visual feature vectors into source text embedding space and used
them as the first and/or last word of the source sentence. Further, they use
them to initialize the hidden state of the encoder instead of using a zero
vector. Finally, they use the projected image representations and encoder
output to initialize hidden representations of a decoder. They use
bidirectional RNN as the encoder. Since visual features are needed in very
specific cases, the model tends to learn to ignore visual features, resulting
in ignoring them, even when necessary. Therefore, a visual attention grounding
mechanism (Zhou et al., 2018) was adopted for MMT that links the visual
semantics with the corresponding textual semantics.

In order to enable the MMT systems to adequately handle multimodal
information, a translate and refine approach (Ive et al., 2019) was adopted
for MMT, where the first stage decoder generates translation on textual
information and second stage decoder refines it with the help of the visual
features. To improve the interaction between textual and visual features, a
latent variable was added to model (Calixto et al., 2019). This latent
variable can be seen as a stochastic embedding which is used by the decoder
for translation, as well as it can be used to predict image features.
Attention (Vaswani et al., 2017a) based multi-modal interactions are widely
used in MMT. This includes cross-attention between regional image features and
decoder self-attention output (Zhao et al., 2022).

The multimodal information generally helped to achieve better results.
However, it was observed that many of the improvements were not related to the
presence of visual features in MMT. This indicated a need to probe how and why
MMT models generate better translation compared to UNMT models. In a probing
task (Li et al., 2022), the words that included color information (red, green,
etc.) were masked. Further, the words indicating characters and nouns were
also replaced with mask tokens from the source text to simulate a situation
where information is insufficient in the source text. They proposed a
Selective Attention based transformer that is able to recover the masked
information from the image at the generated translation better than other
existing models. Similarly, it was observed that models can integrate the
visual modality if they are complementary rather than redundant (Caglayan et
al., 2019). Further, they observe that visual features increase the robustness
of NMT by acting as input noise.

###  2.2. Multimodal Translation on Indian Languages

The early attempts for MMT for Indian languages relied on synthetic data
(Dutta Chowdhury et al., 2018), which was generated by using a text-based
PBSMT model. Then, they use the projected image feature for hidden
representations of encoder and decoder (Calixto and Liu, 2017) to generate
Multimodal MT. Initializing the hidden representation of the decoder generated
better results compared to that of the encoder. Subsequently, it was observed
that using multimodal methods achieves better results (Sanayai Meetei et al.,
2019; Laskar et al., 2019) compared to UNMT or image captioning. Doubly
Attentive decoder (Calixto et al., 2017) architecture, which was proven to
generate a better translation for other languages, was adopted for English-
Hindi (Laskar et al., 2020, 2021a). They use pre-trained word embedding
learned from IITB Corpus (Kunchukuttan et al., 2018) to initialize the models
and augment datasets by using Giza++ Tool (Och and Ney, 2003), to obtain
phrase pairs. Further, the pre-trained VGG19 (Simonyan and Zisserman, 2014)
model is used to extract image features.

It was observed that using text-based pre-trained models achieved better
results for Hindi VG Test set compared to the multimodal model trained only on
multimodal dataset (Gain et al., 2021b). However, it obtained lower results
for the Challenge set. Further, the text-based model was trained on a larger
bilingual corpus, which could be a major contributing factor to better
results. Multimodal Transformer architecture was adopted for English-Hindi
(Shi and Yu, 2022), which improved the results compared to its UNMT baseline.
There are other works that utilized multiple captions from the dataset to
train the models (Singh et al., 2021). It was observed that multiple captions
introduce lexical diversity, which helps to generate robust translation.

There is a very limited number of studies that use large-scale pre-training. A
model was pre-trained on IITB corpus (Kunchukuttan et al., 2018), and it was
observed that large-scale pre-training is more important for better
translation in contrast to image features (Parida et al., 2019b). In another
study, the multimodal features are found to be less relevant as a good pre-
trained model can handle ambiguity issues (Bisht and Solanki, 2022). A pre-
trained mBART model was used to initialize the weights (Gupta et al., 2021).
Then they used unimodal data for the first stage of fine-tuning. Finally, they
concatenated object tags with source text before last-stage fine-tuning. Those
methods are entirely unimodal except for the usage of Object detection models
for object tag extraction at data pre-processing stages. Thereafter, a similar
method was adopted for English-Bengali (Parida et al., 2021). Inspired from
(Grönroos et al., 2018), a sequence-to-sequence model was used for MMT (Parida
et al., 2022). In this method, the input is the concatenated source and target
side, and the output is object tags, which act as dummy image features on
unimodal datasets. They follow the object tag concatenation-based method,
which resulted in huge BLEU gains compared to the text-only model. However,
the text-only models were not trained on very large corpus in required
languages. This makes them unable to disambiguate sentences properly.

There were some attempts in the reverse direction, where Spatial attention was
used for image features (Meetei et al., 2023) for Hindi to English Multimodal
News Translation. In the meantime, some MMT methods were proposed for other
low-resource Indian languages. A transliteration-based phrase augmentation
method was adopted for English-Assamese (Laskar et al., 2023) with Assamese
VisualGenome dataset (Laskar et al., 2021b) and English-Bengali (Laskar et
al., 2022a). A similar method was used for English-Hindi (Laskar et al.,
2022b), and this method was able to generate better results compared to UNMT.
A comparison of results was performed on the Mizo VisualGenome dataset
(Khenglawt et al., 2022) with Transformer, BiLSTM, and PBSMT models
(Lalrempuii et al., 2021; Pathak et al., 2018; Thihlum et al., 2020), where
the transformer outperforms the other models. However, it was observed that
NMT-based models generate a poor-quality translation for longer sentences. A
multilingual multimodal NMT system was introduced for English and Dravidian
languages (Chakravarthi et al., 2019), which generated better results compared
to UNMT and single-language MMT. All these methods assumed the presence of
clean and grammatical training data, which could not be very suitable for
real-world use cases.

###  2.3. Context-Aware Translation

Apart from using multimodal information as context, there have been several
attempts to use unimodal information as a context. For example, the previous
and next sentences can serve as a useful context for document translation.
Similarly, previous sentences in a conversation can serve as a context for
chat translation. The context is generally incorporated by concatenation
(Tiedemann and Scherrer, 2017; Gain et al., 2021c) or using a different
encoder for context. Hierarchical Attention Networks can achieve better
results (Miculicich et al., 2018) compared to context-agnostic NMT. It was
observed that performance gains from multi-encoder models cannot be entirely
attributed to the contextual information. Setting the weight of the context to
zero during testing does not have a significant negative effect on the
results(Li et al., 2020; Gain et al., 2022a). Similarly, training and testing
with random context also improve the results (Appicharla et al., 2023). The
standard metrics for NMT were unable to capture the improvements at the
discourse level. (Fernandes et al., 2023) proposed a discourse-aware framework
for performance evaluation.

###  2.4. Noisy Neural Machine Translation

The presence of noise in the data has adverse effects on the translation
quality (Belinkov and Bisk, 2018; Khayrallah and Koehn, 2018). Early attempts
for Noisy MT relied on Normalization of text (Wang and Ng, 2013). Due to the
lack of parallel corpora for Noisy NMT, most of the research relies on
synthetically generated noise. This method helped to achieve better results in
translating erroneous speech transcripts (Xie et al., 2017; Sperber et al.,
2017). Subsequently, the noise was added with texts for different datasets
with multiple types of synthetic noise (Karpukhin et al., 2019; Cheng et al.,
2018). Generally, noisy text translation can be viewed as a domain adaptation
task. Fine-tuning on noisy data improves the robustness of the model (Helcl et
al., 2019; Berard et al., 2019; Dabre and Sumita, 2019). Other methods include
the construction of source and target side adversarial examples (Cheng et al.,
2019) for noisy NMT. Subsequently, Insertion and Deletion predictors were
added to the NMT, which improved results on noisy inputs (Wang et al., 2021).
It was observed that multi-tasking with translation and denoising objectives
helps to generate robust translation (Passban et al., 2021). Further, they
observed improvements after adding synthetic noise to the dataset and using
Dual Channel Decoding. Noisy Text Translation is a mostly unexplored area for
Indian languages. It was observed that fine-tuning noisy data improves the
results for Conversational NMT (Gain et al., 2022b). However, since the
training datasets were synthetically generated, the improvements are mostly
due to domain adaptation.

##  3\. Dataset Details

We use VisualGenome (VG) datasets containing Bengali (Sen et al., 2022), Hindi
(Parida et al., 2019a), and Malayalam (Parida and Bojar, 2021) sentences for
fine-tuning. The train, valid, test, and challenge subsets of each of them
contain 29230, 998, 1595, and 1400 sentences, respectively. Sentences with
potential ambiguity (such as ”court”) that may or may not be resolved by
textual information made up the challenge subset. Each row in a dataset
contains the following fields:

  * •

Source text: The utterance in English.

  * •

Target text: The utterance in Hindi/ Bengali/ Malayalam.

  * •

Image ID: The filename of the image that is relevant to this utterance.

  * •

Coordinates of image: The dataset offers the bounding box of the particular
area of the image that an utterance is based on.

Table 1. Statistics of Hin: Hindi, Beng: Bengali, and Mala: Malayalam
VisualGenome datasets

Subset | #Sentences | #Average Words per Utterance
---|---|---
English | Hin | Beng | Mala
Train | 28930 | 4.95 | 5.03 | 3.94 | 3.70
Valid | 998 | 4.93 | 4.99 | 3.94 | 3.63
Test | 1595 | 4.92 | 4.92 | 4.02 | 3.57
Challenge | 1400 | 5.85 | 6.17 | 4.76 | 4.32

A sample of the dataset is presented in Figure 1, and we describe some
additional dataset statistics in Table 1. We frequently noticed that the image
size within the box’s boundaries is insufficient. More information might be
available if the entire image were used rather than just the cropped area
contained within the bounding box.

(Gain et al., 2021a) observed that using cropped image feature of ResNet-50
yields better results compared to the full image. However, their observation
is inconclusive as the difference is marginal in most experimental settings,
and using full images achieved slightly better results in one setting.
Therefore, we perform our experiments in both setups.

###  3.1. Noisy Data Generation

We generate noise in the datasets as they appear in social media. Social media
captions frequently misspell words with two identical consecutive characters,
write words without vowels, and omit articles (a, an, the). Since our work
intends to find if the visual context can help to translate noisy text, we did
not add noise to the images.

####  3.1.1. Low Noise

We iterate through all the words in a sentence and remove them if we encounter
an article with 0.2 probability. If the word is non-article or article but not
removed in the previous step, we look for vowel characters in the word and
remove them with 0.1 probability. Finally, we check if the word contains the
same letters at two consecutive positions. In that case, we remove one of them
with 0.2 probability. We started with 0.3 probability for each noise
generation operation and randomly selected 20 samples from the dataset. We
engaged some human annotators who are active on social media to rate noised
sentences in terms of naturalness from 1 to 5. A rating of 5 indicates that
these types of noisy sentences often appear in social media, and a rating of 1
indicates that the appearances of these types of noisy sentences are quite
unlikely. We keep reducing the probability of each type of noise by 0.1 till
we reach an average rating of at least 4.5. The statistics of added noise with
respect to the actual sentences are reported in Table 2. As we acquire the
noisy version of the data through subjective human judgment, we use the BLEU
(Papineni et al., 2002) metric to report n-gram matches as a measure of
similarity. Similarly, TER (Snover et al., 2006) indicates the number of edits
required to obtain the original source from the corrupted text. chrF2
(Popović, 2015, 2016) is a character-level F-score metric. To maintain a
consistent distribution of noise throughout training, validation, and
evaluation, we introduce noise to all the available subsets of the data. In
low noise settings, the resulting BLEU scores typically fall within the range
of 65.9 to 67.5, while chrF2 scores range from 87.0 to 87.7; a lower TER
score, in this case, indicates a lower amount of noise, which ranges from 16.2
to 16.5 in low noise settings.

Table 2. Statistics of the source text after addition of noise when compared
with original source text

Dataset | Low Noise | High Noise
---|---|---
BLEU | chrF2 | TER | BLEU | chrF2 | TER
Train | 67.5 | 87.3 | 16.2 | 38.1 | 71.1 | 34.9
Valid | 66.9 | 87.1 | 16.4 | 36.7 | 70.6 | 35.5
Test | 67.0 | 87.0 | 16.5 | 36.9 | 70.6 | 35.3
Challenge | 65.9 | 87.7 | 16.5 | 37.3 | 71.9 | 36.0

####  3.1.2. High Noise

We remove articles with 0.3 probability. Unlike low noise, we do not make
further modifications to articles if they are not removed because we are
already adding other noises in high amounts. Then, we remove vowels and repeat
characters with 0.3 probability each. In scenarios with high noise levels, the
achieved BLEU scores usually lie between 36.7 to 38.1, and the corresponding
chrF2 scores range from 70.6 to 71.9. A lower TER score in such cases suggests
a reduced amount of noise, which typically falls within the range of 34.9 to
36.0 in high-noise settings.

###  3.2. Pre-Processing

We use langid.py (Lui and Baldwin, 2012), a language identification tool to
detect if any sentence pair contains the wrong language. We found that there
exist some alignment issues with the corpus, like the presence of source
sentences in the target field of the dataset. Some human annotators manually
annotated those instances by translating them into the target language, taking
the corresponding image into consideration. Further, the source data was
unavailable in a few instances of Hindi VisualGenome. Since the Bengali
VisualGenome is aligned with Hindi VisualGenome, we copy the corresponding
source sentence from Bengali VisualGenome. Finally, while Malayalam
VisualGenome contains the same set of sentences, image pairs are Hindi and
Bengali VG; they are not aligned. We align them to match the order of Bengali
VG. Aligning the datasets helps to reduce memory usage, as visual features can
be common between all languages. Finally, we follow the same pre-processing
steps that are performed to train the pre-trained model. Since Hindi, Bengali,
and Malayalam are written in different scripts, we convert all target side
data to the Devanagari script by a publicly available tool (Kunchukuttan,
2020). We use Hindi, Bengali, and Malayalam VisualGenome datasets for fine-
tuning all our models. For cropped experiments, we crop image portions within
bounding boxes. We extract (cropped as well as full) image features from
vit_base_patch16_224 model (Kolesnikov et al., 2021).

##  4\. Methodology

In the literature, previous probing methods focused on the usability of images
when some information from the source was deprived of the model. For example,
the words with _color, person_ , etc., which can be available in the image,
are masked. Even though the models function well in those circumstances,
masking words in the source sentence explicitly tell the model that some
information is hidden. This forces the model to look for pertinent data in the
image. However, in most situations, there will not be explicit mask tokens
that indicate that a word is absent. On the other hand, simply dropping the
words (without mask token) and expecting the models to retrieve it does not
align with the MT objective. For example, in the source sentence chinese
characters on a red vehicle, if we remove the word red, then chinese
characters on a vehicle is also a valid sentence, and the model should not
attempt to add information related to color in the generated translation.
Doing this would lead to undesirable information in the generated sentence,
which is absent in the source. Therefore, instead of providing any information
if the input sentence is noisy, we try to determine whether the models can use
information from images in the presence of noise in the source text.

###  4.1. Baseline

Figure 2. Selective Attention Architecture for Multimodal MT. (softcopy after
zooming-in exhibits better display)

We use a transformer-4x model222https://ai4b-public-nlu-
nlg.objectstore.e2enetworks.net/en2indic.zip trained on Samanantar (Ramesh et
al., 2022) corpus containing 49.6 million sentence pairs between English and
11 Indian languages as our baseline. The model consists of six encoder and six
decoder layers. We initialize weights from this model for all our experiments
and use the same dictionary for byte-pair-encoding. For English-Malayalam
Challenge set, it achieves a 26.3 BLEU score, whereas the SOTA system (Parida
et al., 2022; Gupta et al., 2021) achieves a 20.4 BLEU score. This outperforms
SOTA by 5.9 BLEU points even before fine-tuning on any data. After fine-tuning
the model on only source-target sentence pairs on respective VisualGenome
data, we observe that it outperforms all existing multimodal models by a large
margin. Considering the additional overhead of handling images and extracting
visual features, we attempt to find if they are worth it when we can simply
use a text-based model only and achieve better results since text-only data is
available in large quantities.

###  4.2. Unimodal Fine-tuning

First, we follow pre-processing steps as mentioned in subsection 3.2 and
combine data from Hindi, Bengali, and Malayalam VG datasets. Then, we
initialize the encoder and decoder weights from the baseline model (including
word embedding) and continue training. This model surpasses all existing
baselines on all languages and both Test and Challenge subsets by large
margins. This model does not use any information from images. Unimodal data
are widely available, whereas multimodal datasets only have a small number of
examples. If multimodal models are not trained on unimodal data, we cannot
achieve excellent results.

###  4.3. Multimodal Fine-tuning with Selective Attention

For multi-modal translation, we follow the Selective Attention architecture as
introduced in (Li et al., 2022). The architecture of the model has been shown
in Figure 2. For a given data point of ¡image, source sentence, target
sentence¿ in the training dataset, at first, we obtain the word and positional
embedding for the given source text. The Positional encoding describes the
location or position of a token in a sequence in such a way that each position
is assigned a unique representation. In Transformers positional encoding, each
position is mapped to a vector. As an example, for an input sequence of length
L, the positional encoding of the $k$th object within this sequence is given
by sine and cosine functions of varying frequencies as follows:

(1)
$P(k,2i)={sin}\left(\frac{k}{n^{2i/d}}\right)~{};~{}P(k,2i+1)={cos}\left(\frac{k}{n^{2i/d}}\right);$

where, $n$ is a user-defined scalar, and $i$ is used for mapping the indices
of the column. The obtained representation is forwarded through the encoder
layer of the Transformer (Vaswani et al., 2017b). Each encoder layer consists
of three major components: Multi-Head Attention (MHA) layers, Fully Connected
layers and the Add & Norm layer. MHA is a module for attention mechanisms that
simultaneously runs through multiple attention mechanisms. After that, the
independent attention outputs are combined and transformed linearly into the
expected dimension. Intuitively, having multiple attention heads makes it
possible to focus on different parts of the sequence simultaneously (Vaswani
et al., 2017b).

After the self-attention mechanism has computed the attention scores between
each input token and every other token in the sequence, each token’s
representation is passed through the feedforward network. The output of the
feedforward network is then added back to the original input representation of
the token, which helps to capture more complex relationships between tokens in
the sequence. The Add & Norm layer acts as a residual connection and performs
normalization of the outputs at the corresponding sub-layer.

To obtain the representation $H_{img}$ of the corresponding input image, we
pass the image through the pre-trained Vision Transformer. It is one of the
recent architectures for image classification, which employ a Transformer-like
architecture over patches of the image. Any image is first split into fixed-
size patches. Each patch is then linearly embedded, after which the position
embeddings are added, and the resultant vector is fed to a standard
Transformer encoder.

Then, we add an attention layer, where $H_{text}$ is the ¡query¿ and $H_{img}$
is used as ¡key and value¿ (Kolesnikov et al., 2021). We perform selective
attention on both the $H_{img}$ and $H_{text}$, where $H_{img}$ acts as ¡key
and value¿ pair and $H_{text}$ acts as a query for the computation of the
attention as follows.

(2)
$H^{\textrm{attn}}_{img}={softmax}\left(\frac{H_{text}Q({H_{img}K)^{\textrm{T}}}}{\sqrt{d_{k}}}\right)H_{img}V;$

$Q$, $K$, and $V$ are the layer-specific query, key, and value vectors.
Subsequently, text-encoder output $H_{text}$ and selective-attention output
$H^{\textrm{attn}}_{img}$ are combined with gated fusion as follows to obtain
the final encoder representation $Enc_{out}$.

(3)
$\begin{split}Enc_{out}=(1-\lambda)~{}H_{text}+\lambda~{}H^{attn}_{img}~{};~{}\\\
\lambda={sigmoid}\left(W_{t}{H_{text}}+W_{i}{H^{attn}_{img}}\right);\end{split}$

$\lambda$ indicates the importance of visual features compared to text
features. Here, $W_{t}$ and $W_{i}$ are trainable parameters.

Finally, $Enc_{out}$ is forwarded to the decoder. In addition to the encoder
sublayers, the decoder contains an MHA layer for Encoder-Decoder cross-
attention. This layer is similar to encoder-decoder attention in standard
transformer architecture, except it attends to the output of gated fusion
instead of the output of the text encoder.

We observe that the model was able to achieve only up to 0.25 BLEU
improvements in some subsets compared to the text-only model, and it decreased
for other test sets by up to 1.38 BLEU. The results indicate that the images
are not very useful for generating better translation in high-resource
settings. Therefore, we experiment with another architecture for Multimodal
Translation.

Figure 3. Multimodal Transformer Architecture for Multimodal MT. (softcopy
after zooming-in exhibits better display)

###  4.4. Multimodal Transformer

Table 3. Results of the methods with large-scale pre-training

Method | Image Type | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---
Test | Chal | Test | Chal | Test | Chal | Test | Chal
SOTA | Object Tags | 44.64 | 51.66 | 43.9 | 32.9 | 41.00 | 20.4 | 43.18 | 34.99
IndicTrans | None | 34.74 | 46.32 | 26.68 | 28.91 | 28.62 | 26.30 | 30.01 | 33.84
\+ Text | None | 45.79 | 56.72 | 50.08 | 47.78 | 51.38 | 40.76 | 49.08 | 48.42
\+ SelAttn | Crop | 46.04 | 56.36 | 48.70 | 47.82 | 51.56 | 39.76 | 48.77 | 47.98
\+ SelAttn | Full | 45.30 | 56.03 | 49.74 | 47.48 | 51.55 | 39.49 | 48.86 | 47.67
\+ MMtrans | Crop | 45.47 | 55.61 | 49.03 | 47.32 | 52.40 | 40.40 | 48.97 | 47.78
\+ MMtrans | Full | 45.10 | 56.59 | 50.45 | 47.24 | 52.00 | 39.77 | 49.18 | 47.87
Test: Evaluation Subset; Chal: Challenge Subset; Text: Text-only finetune
SelAttn: Selective Attention; MMtrans: Multimodal Transformer
SOTA results are from (Parida et al., 2022; Gupta et al., 2021)

We also employ a multimodal transformer (Yao and Wan, 2020) architecture for
the task of multimodal machine translation. The source text features
$x^{text}\in\mathds{R}^{m\times d}$ are obtained by enriching the source text
embedding by the positional embeddings at first. The corresponding image’s
features $x^{img}$ are extracted by passing the input image through the Vision
Transformer. Since the image and text features belong to different embedding
spaces, we project visual features into the text embedding dimension by a
learnable projection matrix, $W^{img}$ to obtain
$x^{img}W^{img}\in{\mathds{R}^{n\times d}}$. Following this, the features are
concatenated with each other to obtain a final input representation $x_{com}$
in $R^{(m+n)\times d}$, where projected visual features are
$x^{img}W^{img}\in{\mathds{R}^{n\times d}}$.

In the succeeding phases, we use the combined feature $x_{com}$ as the ¡key
and value¿ vector for the attention computation in the transformer encoder.
The text representation is used as the query vector in the text encoder. The
output of the final encoder from the last encoder layer is forwarded as keys
and values, while the target text representation is passed as the query vector
to the MHA module in the decoder module. The encoder and decoder architectures
are inspired by the traditional transformer architecture itself. The detail of
the architecture is described in Figure 3.

(4)
$attn_{ij}=\sum_{j=1}^{m}{softmax}\left(\frac{x^{com}_{i}Q{(x^{text}_{j}K)^{\textrm{T}}}}{\sqrt{d_{k}}}\right)x^{text}_{j}V;$

where, $attn_{ij}$ is the output attention between $x^{com}_{i}$ and
$x^{text}_{j}$. Here, $Q$, $K$, and $V$ are the layer-specific query, key, and
value vectors. We obtain the final encoder representation at the last encoder
layer and forward the output to the decoder.

Analogous to the results obtained using the Selective Attention architecture,
the Multimodal Transformer architecture was able to achieve only 0.10 BLEU
improvements on the test set, on average. However, for the case of the
Challenge set, the results actually deteriorated, with a drop of around 0.7
BLEU score.

##  5\. Experiments

This section discusses the results with non-noisy, low-noisy, and high-noisy
settings.

###  5.1. Non-Noisy Settings

Table 4. Difference in results obtained by Multimodal models and Text-based model under Non-Noisy setting |  | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---
Architecture | Image Feat | Eval | Chal | Eval | Chal | Eval | Chal | Eval | Chal
SelAttn | Crop | 0.25 | -0.36 | -1.38 | 0.04 | 0.18 | -1.00 | -0.31 | -0.44
SelAttn | Full | -0.49 | -0.69 | -0.34 | -0.30 | 0.17 | -1.27 | -0.22 | -0.75
MMtrans | Crop | -0.32 | -1.11 | -1.05 | -0.46 | 1.02 | -0.36 | -0.11 | -0.64
MMtrans | Full | -0.69 | -0.13 | 0.37 | -0.54 | 0.62 | -0.99 | 0.10 | -0.55
Green/Positive indicates that the MMT model achieved better results than the
UNMT model.

We report our results in Table 3 under non-noisy setups. In this setting, both
training and testing are done with the original dataset. Text-only baseline
achieved 49.08 and 48.42 BLEU scores. While these scores are superior than all
the existing models, it is due to large-scale pre-training. Selective
attention architecture performs inferior to text-only baseline in non-noisy
setting. Although it achieves better results in some subsets, the improvements
are insignificant. Subsequently, we report the comparison of the multimodal
model with respect to the unimodal model in Table 4. We observe that the image
has an overall negative effect on the results. Surprisingly, it performs worse
on the challenge subset, which was constructed with ambiguous words. However,
the presence of an ambiguous word does not necessarily make a sentence
ambiguous, as the proper meaning of the word can be inferred from the
surrounding words within the sentence.

Table 4 indicates that the images are mostly redundant when used on top of a
strong baseline under non-noisy settings. Our observation is similar to (Gupta
et al., 2021), where object tags were concatenated with source tags and
trained on top of the mBART model. They achieved 0.52 BLEU improvement on the
test set over the text-only model. However, BLEU decreased slightly by 0.06
points on the challenge subset for English-Hindi. Although (Parida et al.,
2022) used similar methods and observed huge improvements over text-based
models, their baseline was much weaker. Our text-based baseline (without fine-
tuning on multimodal data) performs better than their fine-tuned model in some
subsets in terms of BLEU score. We found that images do not have a clear
positive impact on the results under non-noisy settings. We investigate if the
visual context has a positive impact when part of the source text is deprived
or distorted via the addition of noise. Specifically, we add noise types that
are predominant in social media text such that multimodal models could be
useful for text translation in social media.

###  5.2. Low Noise Settings

The results for low noise settings are reported in Table 3. Selective
attention with cropped image features surpassed the text-based models on three
subsets (out of six). It is to be noted that the improvements were on the test
Subset, where words were not so ambiguous. It generated lower scores in the
challenge subset than the text-based model, which contains ambiguous words.
This indicates that text-based models are very good at handling most ambiguity
when trained on large data.

Table 5. Results of the methods in low-noise settings

Method | Image Type | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---
Test | Chal | Test | Chal | Test | Chal | Test | Chal
Text | None | 44.03 | 50.82 | 45.29 | 42.67 | 47.70 | 34.76 | 45.67 | 42.75
SelAttn | Crop | 44.20 | 50.62 | 46.07 | 42.33 | 49.34 | 34.19 | 46.54 | 42.38
SelAttn | Full | 43.48 | 51.03 | 45.24 | 41.72 | 47.83 | 34.32 | 45.52 | 42.36
MMtrans | Crop | 43.25 | 50.42 | 45.11 | 42.20 | 48.48 | 34.08 | 45.61 | 42.23
MMtrans | Full | 42.80 | 50.68 | 44.87 | 41.12 | 47.91 | 33.82 | 45.19 | 41.87
Test: Evaluation subset; Chal: Challenge subset; Text: Text-only fine-tune;
SelAttn: Selective Attention;
MMtrans: Multimodal Transformer.

Table 6. Difference in results obtained by multimodal models and UNMT model under noisy settings |  |  | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---|---
Architecture | Image Feat | Noise | Eval | Chal | Eval | Chal | Eval | Chal | Eval | Chal
SelAttn | Crop | Low | 0.17 | -0.20 | 0.78 | -0.34 | 1.64 | -0.57 | 0.87 | -0.37
SelAttn | Full | Low | -0.55 | 0.21 | -0.05 | -0.95 | 0.13 | -0.44 | -0.15 | -0.39
MMtrans | Crop | Low | -0.78 | -0.40 | -0.18 | -0.47 | 0.78 | -0.68 | -0.06 | -0.52
MMtrans | Full | Low | -1.23 | -0.14 | -0.42 | -1.55 | 0.21 | -0.94 | -0.48 | -0.88
SelAttn | Crop | High | 0.07 | -0.51 | -0.11 | -0.78 | -0.27 | -1.19 | -0.1 | -0.82
SelAttn | Full | High | -0.04 | -0.52 | 0.44 | -0.17 | 0.44 | -0.14 | 0.28 | -0.27
Green/Positive indicates that the MMT model achieved better results than the UNMT model. |
Table 7. Results of the methods in high-noise settings

Method | Image Type | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---
Test | Chal | Test | Chal | Test | Chal | Test | Chal
Text | None | 41.44 | 45.28 | 42.49 | 36.56 | 44.38 | 29.35 | 42.77 | 37.06
SelAttn | Crop | 41.51 | 44.77 | 42.38 | 35.78 | 44.11 | 28.16 | 42.67 | 36.24
SelAttn | Full | 41.40 | 44.76 | 42.93 | 36.39 | 44.82 | 29.21 | 43.05 | 36.79
Test: Evaluation subset; Chal: Challenge subset; Text: Text-only fine-tune;
SelAttn: Selective Attention;
MMtrans: Multimodal Transformer.

We report the difference between the multimodal models and the unimodal model
on rows 1-4 of Table 6. The selective attention model with cropped image
feature outperformed the text-only model by 0.87 BLEU on average on the test
set. While multimodal transformer architecture performed competitively with
the text-only baseline in some subsets, the overall results were lower than
that of the selective attention model on both the challenge and test sets as
depicted in Table 5. We observe that using only cropped image features has
only a positive effect on the evaluation set (row 1) on low noise settings.
Similarly, the cropped image features have a higher positive impact on the
Malayalam test set as well as a lesser overall negative impact than full
images in multimodal transformer architecture. This suggests the superiority
of cropped image features under low noise settings.

###  5.3. High Noisy Settings

Since the Multimodal Transformer model was unable to score the best results in
any of the subsets in low-noise settings, we primarily focused on Selective
Attention and text-only models. We report the results in Table 7 and the
difference with respect to the unimodal model on rows 5-6 of Table 6. Here,
using the cropped image was not found to be very effective compared to the
text-based baseline. Selective Attention architecture with cropped images was
able to beat the text-only baseline on only one test set out of six, where the
BLEU improvement was only 0.07 points. Using visual features from the full
image was more effective than using cropped image features. This is the
opposite for Low Noise settings. This indicates that the model utilizes visual
features more in high-noise settings.

###  5.4. Probing with Random Images

Table 8. Comparison of using Random Images vs. UNMT under noisy setup |  |  | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---|---
Architecture | Image Feat | Noise | Eval | Chal | Eval | Chal | Eval | Chal | Eval | Chal
SelAttn | Crop | Low | 0.26 | -0.70 | 1.13 | -0.79 | 1.27 | -0.81 | 0.89 | -0.77
SelAttn | Full | Low | -0.40 | -0.36 | -0.17 | -1.22 | -0.93 | -2.05 | -0.50 | -1.21
MMtrans | Crop | Low | -0.69 | -0.32 | -0.18 | -0.49 | 0.48 | -0.08 | -0.13 | -0.30
MMtrans | Full | Low | -0.79 | 0.12 | -0.72 | -1.41 | 1.01 | -0.60 | -0.16 | -0.63
SelAttn | Crop | High | -0.24 | -0.25 | -0.32 | -0.82 | -0.51 | -1.29 | -0.36 | -0.78
SelAttn | Full | High | 0.13 | -0.41 | 0.24 | -0.49 | 0.52 | -0.81 | 0.30 | -0.57
Green/Positive indicates that random images achieved better results than the
unimodal model.
Table 9. Comparison of using Random Images vs. Actual Images under Noisy setup |  |  | Hindi | Bengali | Malayalam | Average
---|---|---|---|---|---|---
Architecture | Image Feat | Noise | Eval | Chal | Eval | Chal | Eval | Chal | Eval | Chal
SelAttn | Crop | Low | -0.09 | 0.50 | -0.35 | 0.45 | 0.37 | 0.24 | -0.02 | 0.40
SelAttn | Full | Low | -0.15 | 0.57 | 0.12 | 0.27 | 1.06 | 1.61 | 0.35 | 0.82
MMtrans | Crop | Low | -0.09 | -0.08 | 0.00 | 0.02 | 0.30 | -0.60 | 0.07 | -0.22
MMtrans | Full | Low | -0.44 | -0.26 | 0.30 | -0.14 | -0.80 | -0.34 | -0.32 | -0.25
SelAttn | Crop | High | 0.31 | -0.26 | 0.21 | 0.04 | 0.24 | 0.10 | 0.26 | -0.04
SelAttn | Full | High | -0.17 | -0.11 | 0.20 | 0.32 | -0.08 | 0.67 | -0.02 | 0.30
Green/Positive indicates that actual images achieved better results than
random images.

In the previous sections, it seems that the models can extract and use
relevant information from images, namely cropped images under low-noise and
full images under high-noise settings. However, we investigate if the
contribution to the improvement of results actually comes from relevant
information on the visual features. We perform probing by replacing one image
with another randomly chosen image from the same dataset. We attempt to find
out if doing so would affect the results negatively. However, we achieve
similar results when we use random images during testing for low-noise
settings. From Table 9, Selective Attention architecture with random images
achieved similar results on the test set while getting -0.40 BLEU on the
challenge set compared to actual images on cropped features. On the other
hand, using random images on full images has a larger impact reducing BLEU by
0.35 and 0.82 on Test and Challenge subsets, respectively. This can be due to
the fact that there is enough textual context to handle low noisy situations,
as the text-only model performs well when fine-tuned on noised data.
Similarly, for high-noise, the results are almost similar with respect to
actual images. We compare the model with random features with respect to text-
based baseline in Table 8 and found similar patterns as Table 6. This
indicates that improvements that are achieved with actual images can be
achieved with random images. This is indicative of the model utilizing the
visual features as noise during training instead of using it as complementary
useful information. Therefore, the image, in the form of noise, makes the
model more robust and has minimal effects when correct images are not provided
during inference.

###  5.5. Results and Observations

Figure 4. Example of the Annotation Process; This example is obtained from the
Challenge Subset of Bengali VG. It is to be noted that the reference is wrong
since the image indicates that character refers to words in the banner.
However, the reference is referring the word character to protagonist (of
movies, stories, etc.)

We attempt to find out if the dataset is context-aware enough to facilitate
the models to learn from images and if the test sets are context-aware to
truly influence the results if the models are able to generate image-aware
translation. We build an annotation web portal to check the adequacy, fluency,
and image awareness of the dataset by manual evaluation. We provide the
annotators with the source, target, and image. We ask the annotator to rate
adequacy and fluency on a scale of three - _good, medium, and bad_. Further,
we ask the annotators if they think the image is needed to generate the
translation. The answer should be _No_ if they would translate it the same
even if the image were absent. The answer should be _Yes_ if they would have
translated it differently or be in confusion without the availability of the
image. The answer should be _maybe_ when they would have translated similarly
anyway, but the image helped to clear the confusion. Finally, the answer is
_not reflected_ when they believe that the translation could be better with
the information from the image, but the same is not reflected in the
reference.

Table 10. Details of annotation for quality and image awareness of the
datasets

|  | Challenge Set (%) | Test Set (%) | Train Set (%)
---|---|---|---|---
Attribute | Rating | Hindi | Bengali | Hindi | Bengali | Hindi | Bengali
| Good | 82 | 74 | 92 | 88 | 86 | 64
Adequacy | Medium | 8 | 18 | 8 | 8 | 14 | 32
| Bad | 10 | 8 | 0 | 4 | 0 | 4
| Good | 86 | 74 | 80 | 72 | 62 | 48
Fluency | Medium | 14 | 14 | 20 | 22 | 30 | 42
| Bad | 0 | 12 | 0 | 6 | 8 | 10
| Yes | 6 | 0 | 0
Need of Image | Maybe | 4 | 4 | 10
| No | 84 | 94 | 90
| Not Reflected | 6 | 2 | 0

We report our observations in Table 10. While the dataset is mostly fluent and
adequate, usually, the source text is enough to translate it. In the train
set, 90% of the examples do not require an image to translate correctly, while
in 10%, it would have been somewhat helpful but can be translated without it.
This prevents the models from getting proper training signals to use
multimodal information effectively. In the challenge set of VisualGenome (VG)
datasets, only 6% of the dataset would require the image to translate it,
whereas 4% would have been somewhat helpful. This is 0% and 4% for the test
set. There exist many examples where an image is necessary to translate a
sentence. However, the reference does not reflect appropriate image-aware
translation. This occurs in 6% and 2% of the examples in Challenge and Test
sets, respectively. We generate this report on the first 50 examples of Train,
Challenge, and Test Sets of Hindi VG and Bengal VG. This indicates that only a
small percentage of examples require images for correct translation. However,
the usage of examples that are difficult to translate would encourage the
models to use visual features effectively, even when it is trained on a large
uni-modal corpus. However, to the best of our knowledge, there are no such
data publicly available.

###  5.6. Experimental Setup

We try to keep the hyper-parameters the same in every model for a fair
comparison. We set the learning rate to $3\times 10^{-5}$, which performed the
best for the baseline model (Ramesh et al., 2022) during fine-tuning. We set
warmup updates to 4000 and the initial learning rate to $10^{-7}$ with inverse
Square root as the learning rate scheduler. Due to the small size of the
dataset, we validate and save checkpoints at frequent intervals at multiples
of 500 steps or the end of an epoch. We continue training with Early Stopping
till validation loss does not improve for five consecutive checkpoints. We use
_Label Smoothed Cross Entropy_ as the loss function and set 0.1 as the
_smoothing_value_. We set the _batch_size_ to 256 tokens and
_update_frequency_ to 2. Effectively, this makes the _batch_size_ of 512
tokens per step. We use a single GeForce RTX 2080 Ti with 11 GB GPU RAM. We
use Ubuntu 18.04.1 LTS x86_64 for all our experiments. We perform memory-
efficient FLOP16 training and use the fairseq (Ott et al., 2019) library for
our implementations. We set 42 as the _random_seed_ and 0.7 as the
_keeping_probability_ of dropout. During testing, we used beam search with a
_beam_width_ of 5.

##  6\. Conclusion

Multimodal NMT can be useful in translating social media content. Previous
works suggest that multimodal models generate better translation compared to
unimodal NMT. It does not always hold true, specifically when the NMT model is
trained on a large corpus. We observe that images are often redundant. We show
that two well-known MMT models could not consistently outperform a strong
text-based baseline. We have seen that the visual context is valuable when the
source text is noisy. Experimentally, the cropped image features achieved the
best results in low-noise settings, and full image features attained the best
performance in high-noise settings. This indicates that the impact of visual
features increases with an increment in the source text’s noise level.
However, we found that the improvements were not due to useful information
from the images, as we achieved similar results with random image features. We
show that the models need to use images more effectively, even when noise is
introduced in the text, making it less adequate. This could be changed with
the usage of datasets where the image is necessary, enabling models to learn
to use visual features effectively. In the future, we would like to explore
the effectiveness of multilingual multimodal NMT for some other languages and
datasets under noisy settings.

## References

  * (1)
  * Appicharla et al. (2023) Ramakrishna Appicharla, Baban Gain, Santanu Pal, and Asif Ekbal. 2023.  A Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation.  arXiv:2308.06063 [cs.CL]
  * Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018.  Synthetic and Natural Noise Both Break Neural Machine Translation.  arXiv:1711.02173 [cs.CL]
  * Berard et al. (2019) Alexandre Berard, Ioan Calapodescu, and Claude Roux. 2019\.  Naver Labs Europe’s Systems for the WMT19 Machine Translation Robustness Task. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_. Association for Computational Linguistics, Florence, Italy, 526–532.  https://doi.org/10.18653/v1/W19-5361
  * Bisht and Solanki (2022) Paritosh Bisht and Arun Solanki. 2022.  Exploring Practical Deep Learning Approaches for English-to-Hindi Image Caption Translation Using Transformers and Object Detectors. In _Applications of Artificial Intelligence and Machine Learning_ , Bhuvan Unhelker, Hari Mohan Pandey, and Gaurav Raj (Eds.). Springer Nature Singapore, Singapore, 47–60.
  * Caglayan et al. (2016) Ozan Caglayan, Walid Aransa, Yaxing Wang, Marc Masana, Mercedes García-Martínez, Fethi Bougares, Loïc Barrault, and Joost van de Weijer. 2016.  Does Multimodality Help Human and Machine for Translation and Image Captioning?. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_. Association for Computational Linguistics, Berlin, Germany, 627–633.  https://doi.org/10.18653/v1/W16-2358
  * Caglayan et al. (2019) Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. 2019.  Probing the Need for Visual Context in Multimodal Machine Translation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4159–4170.  https://doi.org/10.18653/v1/N19-1422
  * Calixto and Liu (2017) Iacer Calixto and Qun Liu. 2017.  Incorporating Global Visual Features into Attention-based Neural Machine Translation.. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, 992–1003.  https://doi.org/10.18653/v1/D17-1105
  * Calixto et al. (2017) Iacer Calixto, Qun Liu, and Nick Campbell. 2017.  Doubly-Attentive Decoder for Multi-modal Neural Machine Translation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, 1913–1924.  https://doi.org/10.18653/v1/P17-1175
  * Calixto et al. (2019) Iacer Calixto, Miguel Rios, and Wilker Aziz. 2019.  Latent Variable Model for Multi-modal Translation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 6392–6405.  https://doi.org/10.18653/v1/P19-1642
  * Chakravarthi et al. (2019) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Bernardo Stearns, Arun Jayapal, Sridevy S, Mihael Arcan, Manel Zarrouk, and John P McCrae. 2019.  Multilingual Multimodal Machine Translation for Dravidian Languages utilizing Phonetic Transcription. In _Proceedings of the 2nd Workshop on Technologies for MT of Low Resource Languages_. European Association for Machine Translation, Dublin, Ireland, 56–63.  https://aclanthology.org/W19-6809
  * Cheng et al. (2019) Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019.  Robust Neural Machine Translation with Doubly Adversarial Inputs. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 4324–4333.  https://doi.org/10.18653/v1/P19-1425
  * Cheng et al. (2018) Yong Cheng, Zhaopeng Tu, Fandong Meng, Junjie Zhai, and Yang Liu. 2018.  Towards Robust Neural Machine Translation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Melbourne, Australia, 1756–1766.  https://doi.org/10.18653/v1/P18-1163
  * Comrie (2009) Bernard Comrie. 2009\.  _The world’s major languages_.  Routledge.
  * Dabre and Sumita (2019) Raj Dabre and Eiichiro Sumita. 2019.  NICT’s Supervised Neural Machine Translation Systems for the WMT19 Translation Robustness Task. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_. Association for Computational Linguistics, Florence, Italy, 533–536.  https://doi.org/10.18653/v1/W19-5362
  * Dutta Chowdhury et al. (2018) Koel Dutta Chowdhury, Mohammed Hasanuzzaman, and Qun Liu. 2018\.  Multimodal Neural Machine Translation for Low-resource Language Pairs using Synthetic Data. In _Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP_. Association for Computational Linguistics, Melbourne, 33–42.  https://doi.org/10.18653/v1/W18-3405
  * Elliott and Kádár (2017) Desmond Elliott and Ákos Kádár. 2017.  Imagination Improves Multimodal Translation. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Asian Federation of Natural Language Processing, Taipei, Taiwan, 130–141.  https://aclanthology.org/I17-1014
  * Fernandes et al. (2023) Patrick Fernandes, Kayo Yin, Emmy Liu, André Martins, and Graham Neubig. 2023.  When Does Translation Require Context? A Data-driven, Multilingual Exploration. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Toronto, Canada, 606–626.  https://aclanthology.org/2023.acl-long.36
  * Gain et al. (2022a) Baban Gain, Ramakrishna Appicharla, Soumya Chennabasavaraj, Nikesh Garera, Asif Ekbal, and Muthusamy Chelliah. 2022a.  Investigating Effectiveness of Multi-Encoder for Conversational Neural Machine Translation. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 949–954.  https://aclanthology.org/2022.wmt-1.90
  * Gain et al. (2022b) Baban Gain, Ramakrishna Appicharla, Soumya Chennabasavaraj, Nikesh Garera, Asif Ekbal, and Muthusamy Chelliah. 2022b.  Low Resource Chat Translation: A Benchmark for Hindi–English Language Pair. In _Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)_. Association for Machine Translation in the Americas, Orlando, USA, 83–96.  https://aclanthology.org/2022.amta-research.7
  * Gain et al. (2021a) Baban Gain, Dibyanayan Bandyopadhyay, and Asif Ekbal. 2021a.  Experiences of Adapting Multimodal Machine Translation Techniques for Hindi. In _Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021)_. INCOMA Ltd., Online (Virtual Mode), 40–44.  https://aclanthology.org/2021.mmtlrl-1.7
  * Gain et al. (2021b) Baban Gain, Dibyanayan Bandyopadhyay, and Asif Ekbal. 2021b.  IITP at WAT 2021: System description for English-Hindi Multimodal Translation Task. In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_. Association for Computational Linguistics, Online, 161–165.  https://doi.org/10.18653/v1/2021.wat-1.18
  * Gain et al. (2021c) Baban Gain, Rejwanul Haque, and Asif Ekbal. 2021c.  Not All Contexts are Important: The Impact of Effective Context in Conversational Neural Machine Translation. In _2021 International Joint Conference on Neural Networks (IJCNN)_. 1–8.  https://doi.org/10.1109/IJCNN52387.2021.9534444
  * Grönroos et al. (2018) Stig-Arne Grönroos, Benoit Huet, Mikko Kurimo, Jorma Laaksonen, Bernard Merialdo, Phu Pham, Mats Sjöberg, Umut Sulubacak, Jörg Tiedemann, Raphael Troncy, and Raúl Vázquez. 2018.  The MeMAD Submission to the WMT18 Multimodal Translation Task. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_. Association for Computational Linguistics, Belgium, Brussels, 603–611.  https://doi.org/10.18653/v1/W18-6439
  * Gupta et al. (2021) Kshitij Gupta, Devansh Gautam, and Radhika Mamidi. 2021\.  ViTA: Visual-Linguistic Translation by Aligning Object Tags. In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_. Association for Computational Linguistics, Online, 166–173.  https://doi.org/10.18653/v1/2021.wat-1.19
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015\.  Deep Residual Learning for Image Recognition.  arXiv:1512.03385 [cs.CV]
  * Helcl et al. (2019) Jindřich Helcl, Jindřich Libovický, and Martin Popel. 2019.  CUNI System for the WMT19 Robustness Task. In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_. Association for Computational Linguistics, Florence, Italy, 539–543.  https://doi.org/10.18653/v1/W19-5364
  * Huang et al. (2016) Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016.  Attention-based Multimodal Neural Machine Translation. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_. Association for Computational Linguistics, Berlin, Germany, 639–645.  https://doi.org/10.18653/v1/W16-2360
  * Ive et al. (2019) Julia Ive, Pranava Madhyastha, and Lucia Specia. 2019\.  Distilling Translations with Visual Awareness. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 6525–6538.  https://doi.org/10.18653/v1/P19-1653
  * Karpukhin et al. (2019) Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019.  Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation. In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_. Association for Computational Linguistics, Hong Kong, China, 42–47.  https://doi.org/10.18653/v1/D19-5506
  * Khayrallah and Koehn (2018) Huda Khayrallah and Philipp Koehn. 2018.  On the Impact of Various Types of Noise on Neural Machine Translation. In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_. Association for Computational Linguistics, Melbourne, Australia, 74–83.  https://doi.org/10.18653/v1/W18-2709
  * Khenglawt et al. (2022) Vanlalmuansangi Khenglawt, Sahinur Rahman Laskar, Riyanka Manna, Partha Pakray, and Ajoy Kumar Khan. 2022\.  Mizo Visual Genome 1.0 : A Dataset for English-Mizo Multimodal Neural Machine Translation. In _2022 IEEE Silchar Subsection Conference (SILCON)_. 1–6.  https://doi.org/10.1109/SILCON55242.2022.10028882
  * Kolesnikov et al. (2021) Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. 2021.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
  * Kunchukuttan (2020) Anoop Kunchukuttan. 2020\.  The IndicNLP Library.  https://github.com/anoopkunchukuttan/indic_nlp_library/blob/master/docs/indicnlp.pdf.
  * Kunchukuttan et al. (2018) Anoop Kunchukuttan, Pratik Mehta, and Pushpak Bhattacharyya. 2018\.  The IIT Bombay English-Hindi Parallel Corpus. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_. European Language Resources Association (ELRA), Miyazaki, Japan.  https://aclanthology.org/L18-1548
  * Lalrempuii et al. (2021) Candy Lalrempuii, Badal Soni, and Partha Pakray. 2021\.  An Improved English-to-Mizo Neural Machine Translation.  _ACM Trans. Asian Low-Resour. Lang. Inf. Process._ 20, 4, Article 61 (may 2021), 21 pages.  https://doi.org/10.1145/3445974
  * Laskar et al. (2022a) Sahinur Rahman Laskar, Pankaj Dadure, Riyanka Manna, Partha Pakray, and Sivaji Bandyopadhyay. 2022a.  English to Bengali Multimodal Neural Machine Translation using Transliteration-based Phrase Pairs Augmentation. In _Proceedings of the 9th Workshop on Asian Translation_. International Conference on Computational Linguistics, Gyeongju, Republic of Korea, 111–116.  https://aclanthology.org/2022.wat-1.14
  * Laskar et al. (2021a) Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji, Darsh Kaushik, Partha Pakray, and Sivaji Bandyopadhyay. 2021a.  Improved English to Hindi Multimodal Neural Machine Translation. In _Proceedings of the 8th Workshop on Asian Translation (WAT2021)_. Association for Computational Linguistics, Online, 155–160.  https://doi.org/10.18653/v1/2021.wat-1.17
  * Laskar et al. (2020) Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji, Partha Pakray, and Sivaji Bandyopadhyay. 2020\.  Multimodal Neural Machine Translation for English to Hindi. In _Proceedings of the 7th Workshop on Asian Translation_. Association for Computational Linguistics, Suzhou, China, 109–113.  https://aclanthology.org/2020.wat-1.11
  * Laskar et al. (2023) Sahinur Rahman Laskar, Bishwaraj Paul, Partha Pakray, and Sivaji Bandyopadhyay. 2023.  English-Assamese Multimodal Neural Machine Translation using Transliteration-based Phrase Augmentation Approach.  _Procedia Computer Science_ 218 (2023), 979–988.  https://doi.org/10.1016/j.procs.2023.01.078 International Conference on Machine Learning and Data Engineering.
  * Laskar et al. (2021b) Sahinur Rahman Laskar, Bishwaraj Paul, Siddharth Paudwal, Pranjit Gautam, Nirmita Biswas, and Partha Pakray. 2021b.  Multimodal Neural Machine Translation for English–Assamese Pair. In _2021 International Conference on Computational Performance Evaluation (ComPE)_. 387–392.  https://doi.org/10.1109/ComPE53109.2021.9752181
  * Laskar et al. (2022b) Sahinur Rahman Laskar, Rahul Singh, Md Faizal Karim, Riyanka Manna, Partha Pakray, and Sivaji Bandyopadhyay. 2022b.  Investigation of English to Hindi Multimodal Neural Machine Translation using Transliteration-based Phrase Pairs Augmentation. In _Proceedings of the 9th Workshop on Asian Translation_. International Conference on Computational Linguistics, Gyeongju, Republic of Korea, 117–122.  https://aclanthology.org/2022.wat-1.15
  * Laskar et al. (2019) Sahinur Rahman Laskar, Rohit Pratap Singh, Partha Pakray, and Sivaji Bandyopadhyay. 2019.  English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning. In _Proceedings of the 6th Workshop on Asian Translation_. Association for Computational Linguistics, Hong Kong, China, 62–67.  https://doi.org/10.18653/v1/D19-5205
  * Li et al. (2020) Bei Li, Hui Liu, Ziyang Wang, Yufan Jiang, Tong Xiao, Jingbo Zhu, Tongran Liu, and Changliang Li. 2020\.  Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 3512–3518.  https://doi.org/10.18653/v1/2020.acl-main.322
  * Li et al. (2022) Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022.  On Vision Features in Multimodal Machine Translation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 6327–6337.  https://doi.org/10.18653/v1/2022.acl-long.438
  * Lui and Baldwin (2012) Marco Lui and Timothy Baldwin. 2012.  langid.py: An Off-the-shelf Language Identification Tool. In _Proceedings of the ACL 2012 System Demonstrations_. Association for Computational Linguistics, Jeju Island, Korea, 25–30.  https://aclanthology.org/P12-3005
  * Meetei et al. (2023) Loitongbam Sanayai Meetei, Salam Michael Singh, Alok Singh, Ringki Das, Thoudam Doren Singh, and Sivaji Bandyopadhyay. 2023.  Hindi to English Multimodal Machine Translation on News Dataset in Low Resource Setting.  _Procedia Computer Science_ 218 (2023), 2102–2109.  https://doi.org/10.1016/j.procs.2023.01.186 International Conference on Machine Learning and Data Engineering.
  * Miculicich et al. (2018) Lesly Miculicich, Dhananjay Ram, Nikolaos Pappas, and James Henderson. 2018.  Document-Level Neural Machine Translation with Hierarchical Attention Networks. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 2947–2954.  https://doi.org/10.18653/v1/D18-1325
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A Systematic Comparison of Various Statistical Alignment Models.  _Computational Linguistics_ 29, 1 (2003), 19–51.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019\.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_. Association for Computational Linguistics, Minneapolis, Minnesota, 48–53.  https://doi.org/10.18653/v1/N19-4009
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  Bleu: a Method for Automatic Evaluation of Machine Translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311–318.  https://doi.org/10.3115/1073083.1073135
  * Parida and Bojar (2021) Shantipriya Parida and Ondřej Bojar. 2021.  Malayalam Visual Genome 1.0.  http://hdl.handle.net/11234/1-3533 LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
  * Parida et al. (2019a) Shantipriya Parida, Ondřej Bojar, and Satya Ranjan Dash. 2019a.  Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation.  _Computación y Sistemas_ 23, 4 (2019), 1499–1505.  Presented at CICLing 2019, La Rochelle, France.
  * Parida et al. (2019b) Shantipriya Parida, Ondřej Bojar, and Petr Motlicek. 2019b.  Idiap NMT System for WAT 2019 Multimodal Translation Task. In _Proceedings of the 6th Workshop on Asian Translation_. Association for Computational Linguistics, Hong Kong, China, 175–180.  https://doi.org/10.18653/v1/D19-5223
  * Parida et al. (2021) Shantipriya Parida, Subhadarshi Panda, Satya Prakash Biswal, Ketan Kotwal, Arghyadeep Sen, Satya Ranjan Dash, and Petr Motlicek. 2021\.  Multimodal Neural Machine Translation System for English to Bengali. In _Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021)_. INCOMA Ltd., Online (Virtual Mode), 31–39.  https://aclanthology.org/2021.mmtlrl-1.6
  * Parida et al. (2022) Shantipriya Parida, Subhadarshi Panda, Stig-Arne Grönroos, Mark Granroth-Wilding, and Mika Koistinen. 2022.  Silo NLP’s Participation at WAT2022. In _Proceedings of the 9th Workshop on Asian Translation_. International Conference on Computational Linguistics, Gyeongju, Republic of Korea, 99–105.  https://aclanthology.org/2022.wat-1.12
  * Passban et al. (2021) Peyman Passban, Puneeth Saladi, and Qun Liu. 2021.  Revisiting Robust Neural Machine Translation: A Transformer Case Study. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 3831–3840.  https://doi.org/10.18653/v1/2021.findings-emnlp.323
  * Pathak et al. (2018) Amarnath Pathak, Partha Pakray, and Jereemi Bentham. 2018\.  English–Mizo Machine Translation using neural and statistical approaches.  _Neural Computing and Applications_ (2018), 1–17.
  * Popović (2015) Maja Popović. 2015\.  chrF: character n-gram F-score for automatic MT evaluation. In _Proceedings of the Tenth Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Lisbon, Portugal, 392–395.  https://doi.org/10.18653/v1/W15-3049
  * Popović (2016) Maja Popović. 2016\.  chrF deconstructed: beta parameters and n-gram weights. In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_. Association for Computational Linguistics, Berlin, Germany, 499–504.  https://doi.org/10.18653/v1/W16-2341
  * Ramesh et al. (2022) Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2022.  Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 145–162.  https://doi.org/10.1162/tacl_a_00452
  * Sanayai Meetei et al. (2019) Loitongbam Sanayai Meetei, Thoudam Doren Singh, and Sivaji Bandyopadhyay. 2019.  WAT2019: English-Hindi Translation on Hindi Visual Genome Dataset. In _Proceedings of the 6th Workshop on Asian Translation_. Association for Computational Linguistics, Hong Kong, China, 181–188.  https://doi.org/10.18653/v1/D19-5224
  * Sen et al. (2022) Arghyadeep Sen, Shantipriya Parida, Ketan Kotwal, Subhadarshi Panda, Ondřej Bojar, and Satya Ranjan Dash. 2022.  Bengali Visual Genome: A Multimodal Dataset for Machine Translation and Image Captioning.  In _Intelligent Data Engineering and Analytics_. Springer, 63–70.
  * Shi and Yu (2022) Xiayang Shi and Zhenqiang Yu. 2022.  Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language.  _Mathematical Problems in Engineering_ 2022 (2022).
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  https://doi.org/10.48550/ARXIV.1409.1556
  * Singh et al. (2021) Salam Michael Singh, Loitongbam Sanayai Meetei, Thoudam Doren Singh, and Sivaji Bandyopadhyay. 2021\.  Multiple Captions Embellished Multilingual Multi-Modal Neural Machine Translation. In _Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021)_. INCOMA Ltd., Online (Virtual Mode), 2–11.  https://aclanthology.org/2021.mmtlrl-1.2
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006.  A Study of Translation Edit Rate with Targeted Human Annotation. In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_. Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA, 223–231.  https://aclanthology.org/2006.amta-papers.25
  * Sperber et al. (2017) Matthias Sperber, Jan Niehues, and Alex Waibel. 2017\.  Toward Robust Neural Machine Translation for Noisy Input Sequences. In _Proceedings of the 14th International Conference on Spoken Language Translation_. International Workshop on Spoken Language Translation, Tokyo, Japan, 90–96.  https://aclanthology.org/2017.iwslt-1.13
  * Thihlum et al. (2020) Zaitinkhuma Thihlum, Vanlalmuansangi Khenglawt, and Somen Debnath. 2020.  Machine Translation of English Language to Mizo Language. In _2020 IEEE International Conference on Cloud Computing in Emerging Markets (CCEM)_. 92–97.  https://doi.org/10.1109/CCEM50674.2020.00028
  * Tiedemann and Scherrer (2017) Jörg Tiedemann and Yves Scherrer. 2017.  Neural Machine Translation with Extended Context. In _Proceedings of the Third Workshop on Discourse in Machine Translation_. Association for Computational Linguistics, Copenhagen, Denmark, 82–92.  https://doi.org/10.18653/v1/W17-4811
  * Vaswani et al. (2017a) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017a.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.  https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Vaswani et al. (2017b) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017b.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Wang and Ng (2013) Pidong Wang and Hwee Tou Ng. 2013.  A Beam-Search Decoder for Normalization of Social Media Text with Application to Machine Translation. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Atlanta, Georgia, 471–481.  https://aclanthology.org/N13-1050
  * Wang et al. (2021) Tao Wang, Chengqi Zhao, Mingxuan Wang, Lei Li, Hang Li, and Deyi Xiong. 2021\.  Secoco: Self-Correcting Encoding for Neural Machine Translation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 4639–4644.  https://doi.org/10.18653/v1/2021.findings-emnlp.396
  * Xie et al. (2017) Ziang Xie, Sida I. Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y. Ng. 2017.  Data Noising as Smoothing in Neural Network Language Models. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=H1VyHY9gg
  * Yao and Wan (2020) Shaowei Yao and Xiaojun Wan. 2020.  Multimodal Transformer for Multimodal Machine Translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4346–4350.  https://doi.org/10.18653/v1/2020.acl-main.400
  * Zhao et al. (2022) Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara, and Chenhui Chu. 2022.  Region-attentive multimodal neural machine translation.  _Neurocomputing_ 476 (2022), 1–13.  https://doi.org/10.1016/j.neucom.2021.12.076
  * Zhou et al. (2018) Mingyang Zhou, Runxiang Cheng, Yong Jae Lee, and Zhou Yu. 2018\.  A Visual Attention Grounding Neural Model for Multimodal Machine Translation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 3643–3653.  https://doi.org/10.18653/v1/D18-1400