Image search using multilingual texts: a cross-modal learning approach between image and text
Maxime Portaz Hicham Randrianarivo Adrien Nivaggioli Estelle Maudet Christophe Servan Sylvain Peyronnet











Image search using multilingual texts: a cross-modal learning approach between image and text
    Maxime Portaz

Qwant Research

m.portaz@qwant.com





Hicham Randrianarivo

Qwant Research

h.randrianarivo@qwant.com
Adrien Nivaggioli

Qwant Research

a.nivaggioli@qwant.com
Estelle Maudet

Qwant Research

e.maudet@qwant.com
Christophe Servan

Qwant Research

c.servan@qwant.com
Sylvain Peyronnet

Qwant Research

s.peyronnet@qwant.com

    December 30, 2023
=======================================================================================================================================================================================================================================================================================================================================================





Multilingual (or cross-lingual) embeddings represent several languages in a unique vector space. Using a common embedding space enables for a shared semantic between words from different languages.
In this paper, we propose to embed images and texts into a unique distributional vector space, enabling to search images
by using text queries expressing information needs related to the (visual) content of images, as well as using image similarity.
Our framework forces the representation of an image to be similar to the representation of the text that describes it. Moreover, by using multilingual embeddings we ensure that words from two different languages have close descriptors and thus are attached to similar images.
We provide experimental evidence of the efficiency of our approach by experimenting it on two datasets: coco <cit.> and Multi30K <cit.>.






§ INTRODUCTION









Deep Neural Networks have shown improvement in state-of-the-art in different tasks, such as image classification <cit.>, word embeddings <cit.>, text classification <cit.> or image retrieval <cit.>. 
They can be used to embed data into features vectors that were primarily used for information retrieval.
On texts, evolution of these embeddings (namely multilingual embeddings) made it possible to solve multilingual tasks such as cross-lingual classification of texts.
On images, they allow comparison based on image content, with text or image queries.

Cross-modal networks are using recurrent and convolutional networks together in order to embed texts and images in a common vector space.
This research area is heavily studied and we present a short related work in section <ref>.




While many approaches embed visual and semantic information together, they are most of the time limited to only one language.
In this paper, we propose a novel approach for multilingual text and image embeddings (section <ref>).
Our method uses cnn to extract Image information and aligned multilingual Word embeddings and rnn to produce a text representation.

With this framework, we provide image and text embeddings that can be trained to produce comparable features. We can thus retrieve images from text, and texts from an image, with a multilingual representation.

More precisely, we propose two approaches. The first uses bv embeddings (see <cit.>) and improves on the state-of-the-art for English when trained on another language.

The second uses muse (see <cit.>). It enables recognition for several languages with only one model, with a slight performance drawback. More precisely, by using muse aligned embeddings in 30 different languages, we can retrieve images with languages never seen by the model.

The major lesson learned with respect to our method is that it provides close to the state-of-the-art performance in English-only context, but enables the use of multilingual datasets to improve results.
We show in section <ref> that adding a new language during the training phase improves image retrieval for other languages.
The experiments show
that the approach based on bv embeddings gives a 3.35 increase in performance on the coco dataset, and a 15.15 increase on Multi30K.
By using muse, we are able to encode more languages in the same model. The experiments for that approach show that adding other languages gives a small decrease in performance for English, but increases the recall for a multilingual environment. Indeed, we obtain 49.38 recall@10 on Multi30K dataset for image retrieval from captions in 4 languages.


















































§ RELATED WORK












 §.§ Text embeddings

The use of word representation is an important step when it comes to search for information from text documents.
In order to perform this task, we want to be able to extract meaningful embeddings from words.
One useful property of word embeddings is that words with a similar meaning must have representations with a close distance.



A lot of people work on this task but the most popular methods are w2v <cit.> and FastText <cit.>
These methods are simplified methods of the neural language model proposed by Bengio et al. <cit.> with several tricks to boost performance.








































 §.§ Multilingual word embeddings.

Word embeddings can be used in multilingual tasks (e.g. machine translation or cross-lingual document classification) by training a model independently for each language.
However, the representations will be in separate vector spaces, which means that the same words in different languages will most likely have different representations.

There are several methods to solve this problem. One consists in training both models independently and then to learn a mapping from one representation to the other  <cit.>. It is also possible to constrain the training to keep the representations of similar words close to each other <cit.>. Finally, the training can be performed jointly using parallel corpora <cit.>.



In the latter category, the bv approach <cit.>
tries to predict words based on the inner context of the sentence like w2v does but also uses words in the source sentence to predict words in the target sentence (and conversely). Thus, for each update in w2v,  bv performs 4 updates: source to source, source to target, target to target and target to source. This leads to a common representation for the two languages.








Recently, muse <cit.> proposes to learn a mapping from several word embeddings trained independently.
This approach enables mapping between word embeddings from different languages.





 §.§ Cross-modal representation


In order to provide queries as sentences or as images, the image embeddings and the text embeddings must be comparable, i.e. in the same representation space.
Recent works have shown the possibility to learn text and image representation simultaneously <cit.>.
They rely on cross-modal networks, that are able to extract information from images and read the caption describing it.
Those networks use word embeddings followed by a rnn to encode sentence embeddings in the same space as the image embeddings, extracted with a cnn.

cnn methods provide ways to encode images in meaning full embeddings.
Prior works considered image similarity based on the categories <cit.>.
Recent approaches <cit.> use ResNet <cit.> as CNN image features extractor.
For the text part on the network, they use w2v or Skip-Thought <cit.> word embeddings, followed by a multi-layer rnn to encode the sentence.

Those methods are multi-branches networks.
The loss function has to be a comparative loss, with a similarity function.
The similarity function is generally estimated using the Euclidean distance, or the cosine similarity.
To train network with this type of loss function, we use a Siamese network <cit.>.
Siamese networks have been extended to triplet networks with three branches in order to give better results <cit.>.
Triplet is composed of an anchor image, an example of a similar image (positive image) and a dissimilar example (negative image).


Those methods enable to learn complex image representation with few examples, as it is possible to select the best triplet example for the training <cit.>.
Our model is based on triplet networks, with each branch based on text or on images, interchangeably.






















§ MULTILINGUAL JOINT TEXT IMAGE EMBEDDING



We present a model for multilingual and cross-modal (joint learning of text and image representation) embeddings.
We propose to embed images and sentences from different languages in the same space [-1 , 1 ]^d, where the distance between two elements (image or sentence) is inversely proportional to their similarity.

To do this, we train a triplet network <cit.> that compares an image to two sentences: one that describes the image, another that does not (See figure <ref>).




 §.§ Overview












Our framework enables to take advantage of the availability of multilingual corpora in order to learn a cross-modal representation between texts and images.
We present a pipeline that learns a common representation between texts in different languages and images.
This common representation enables to compare image and texts using similarity measure for fast information retrieval.

The figure <ref> illustrate the different step of the method to extract a representation from texts or images.
We show that we can learn a model in several languages and obtain state-of-the-art results on several retrieval tasks.
We also show that our method can improve state-of-the-art methods and generalize on languages the model have not seen during the training but which are model by muse.
Our method is composed of two paths.

One path extract information from an image using a feature extractor like ResNet. Then a feature pooling method <cit.> enables to extract a signature from the features. We use the Weldon pooling method which automatically selects areas with highest and lowest activation and computes the image signature using these areas.

The second path computes an embedding of the input text using muse. This method enables to compute close embeddings for words with close meanings in different languages. A rnn is then used to extract a representation from the set of embeddings.

One interesting property of the Weldon pooling is that it produces a mapping between the highest response in the image path and the most significant words in the text path.
This property enables to extract an accurate representation between text and images, enabling to search precisely images with text and vice versa.




 §.§ Multilingual Sentence Embeddings





To embed the sentence, our model first relies on individual word embeddings.
As we want to embed every sentence from every language in a unique vector space, we use word embeddings aligned in different languages.
The usual method consists in using bv <cit.>.  bv aligns two languages in the same space by learning the word embeddings on the two languages simultaneously.

There are 4 languages in our captions: English, French, German, Czech.
On the one hand, we propose to use pre-trained embeddings from muse.  muse is a multilingual extension of FastText <cit.> that embeds and aligns 30 languages in a single vector space. On the other hand, we propose to jointly use bv and muse approaches in order to enhance our multilingual representation.

The main idea is to train independently several bilingual word embeddings, in which, one of the languages is English.
Then, we learn a mapping between the different English representations (from the several bilingual word embeddings) to maximize the link between the bilingual representations.
For instance, one can train two bilingual word embedding models like English-French and English-Czech representations, apply the muse approach on the two English parts.
From the two bilingual representation (English-French and English-Czech), we obtain a third one: French-Czech.

As shown on figure <ref>, we combine sentences from different languages by using word embedding models from different languages according to the approach described previously.



Those word embeddings being in the same space we can use a multi-layer rnn to learn a sentence embedding.
This network is composed of 4 layers of sru <cit.>, with a dropout after each layer.
The goal of this rnn is to encode a vector of word embeddings of size E into a Sentence Embedding Space ℝ^d.
Lastly, we normalize the output of the rnn to obtain an embedding of the sentence in [-1 , 1 ]^d.





 §.§ Joint embedding



The visual path of our network is similar to the one used by Engilberge et al. <cit.>.
It starts with a ResNet152 <cit.>, on which we replaced its last fully connected layer (usually used for classification) with a Weldon pooling layer <cit.>. This layer pulls the regions with the maximum activation in the network, i.e. the regions of interest, and gives us an embedding of the image, a vector of dimension d'.
Finally, this vector goes through a fully connected layer that normalizes it, which aims to obtain an embedding of the image in [-1 , 1 ]^d.

Both pipelines are learned simultaneously, each image is paired with two sentences, one that describes the image, the other that doesn't.
The architecture of the model is shown on figure <ref>.
The two outputs are compared using a cosine similarity, which is equivalent to the inner product as both embeddings are normalized.

We use a triplet loss  <cit.> to converge correctly and increase our performances.
This loss enables us to compare the relative similarity between the image and both sentences: the corresponding sentence should be closer to the image than the unrelated one.



The Figure <ref> presents the model with a triplet of one image and two captions.
The sentence Y describes the image, and the caption Z is an unrelated caption.
The triplet loss is shown in equation <ref>, with x, y, and z being respectively the embeddings of X, Y, and Z. α is the minimum margin between the similarity of the correct caption and the unrelated caption. During the training, it was set to 0.2.


    loss(x,y,z) = max(0, α - x · y + x · z)





 §.§ Training


We train the ResNet on a classification task. This enables the cnn to learn the extraction of interesting image features. We used a ResNet-152 pretrained on ImageNet  <cit.> dataset, which provides a large collection of images, over 1 million, for 1000 categories.
The last layer of the ResNet, that was used for classification, is removed and replaced by a Weldon pooling, followed by a randomly initialized fully connected layer with a dropout regularization.

For the text pipeline, we use pre-trained w2v, FastText, bv and muse word embeddings.
We then freeze the cnn and region pooling of the network and train the rnn and the fully connected layer. This enables to project the embeddings of both sentence and image into a common space.
Finally, we fine-tune the entire network.

As shown by Schroff et al. <cit.>, the triplets used to train the model should be carefully selected.
Indeed, by using easy triplets, i.e. triplets on which the network performs already well, the network learns almost nothing.
As proposed previously in  <cit.>, we aim to focus on the hardest triplets only, i.e. the hardest ones to differentiate by the network.

Instead of looking for the best triplet throughout the entire dataset at each iteration, we stay in the current batch.
For each image and its corresponding sentence (i,s), we select the closest non-similar image to s and the closest non-similar sentence to i.

The following equation shows the loss computation over the batch B composed of image and caption pairs (i,s):


    ∑_i,s ∈ B (  max_z ∈ U_i loss(i,s,z) + max_z ∈ D_i loss(s,i,z)        )


Where D_i represents every image in the batch B that differs from i.
U_i represents every sentence unrelated to the image i, inside the batch B, in every language.
Each batch can contain several captions in multiple languages corresponding to the same image.
This enables the selection of the best example inside each training batch.




 §.§ Visualization


To enhance the recall evaluation made previously, we propose some visual evaluation.
The figure <ref> shows the five closest images for the same sentence in French and German.
These images come from the Google Semantic Caption dataset, which contains 3000000 images.






We show, in Figure <ref>, the maximum of activation of the network, given different words.



We observe the activation zone of the cnn depending on the word used for the rnn.
The network responds mostly in the same way for words from different languages.
Some differences appear with less common words like Mantel in German, with a noisier activation than for the FrenchManteau or the English Jacket.












§ EXPERIMENTS
  

In this section, we present our experimental protocol: hardware/software setup, datasets used and numerical results. We evaluate and compare our method with state-of-the-art approaches by using classic metrics.




 §.§ Experimental setup


All the experiments were done on an NVIDIA DGX-1[<https://www.nvidia.com/>].
Training the network on coco and Multi30k take roughly two days on 4 V100 GPUs with 16GB of RAM, for each experiment.
We used Facebook implementation of FastText[<https://fasttext.cc/>] to compute word embeddings.
Finally, we rely on Pytorch[<https://pytorch.org/>] for deep learning implementation.




 §.§ Datasets


To train and evaluate our model, we used three datasets of images with their corresponding captions.
The first dataset is coco  <cit.>. It contains 123287 images with 5 English captions per image. We used the val split from Karpathy et al. <cit.> (113287 train, 5000 validation and 5000 test images) to train and evaluate our model on English sentences.

To train our model on other languages, we used the Multi30K <cit.> dataset, containing 31014 images with captions in French, German, and Czech. 29000 are kept for training, 1014 for validation and 1000 for testing.
Lastly, for evaluation purposes we used Google's Conceptual Captions[<https://ai.google.com/research/ConceptualCaptions>] dataset, containing 3154240 captioned images.






 §.§ Evaluation method


We evaluate the quality of our results using recall@k, which is
the proportion of relevant images found in the top-k returned images for a given query.
We evaluate caption retrieval with images as queries using recall at 1, 5 and 10.
This means that we verify if the sentence corresponding to an image is in the first, fifth, or tenth closest results.
For image retrieval, each caption is evaluated in the same way. The presence of the image corresponding to a sentence is verified in the first, fifth, or tenth closest results.

The caption retrieval test is made in batches of 1000 images and caption pairs, using the coco dataset.


In the Multi30K dataset, every images have a caption in each language. The recall is computed across languages.












 §.§ Results and analysis


We perform four different experiments. 
The aim of the first and second experiments is to verify the performance of our model with English captions, depending on the word embeddings used.
They measure respectively the caption and the image recall in English on coco dataset.

Finally, in experiment 3 and 4, we evaluate the model with caption and image recall, on the Multi30K dataset, with captions in different languages.




  
Experiment 1.

The models are trained on the coco dataset for English and on Multi30K dataset for French, German and Czech.
We use the coco dataset for evaluation.




The table <ref> shows the caption retrieval recall on coco dataset.
The first two lines show the state-of-the-art results.
The second pair of lines present the results of our model, with w2v and FastText embeddings used as the baseline.
We can see that our model is close to the dsve method <cit.> while the w2v method is slightly worst, as the representation power of the word embedding is reduced.

The bv English-French method is used in English and on both languages simultaneously.
If trained only on English, i.e. only on the coco dataset like the two previous methods, it shows performance similar to the one of the state-of-the-art.
This means training using bv does not weaken the English representation. 
When trained on English and French together, the recall is increased by 3.35, going from 65.58 to 67.78.
We can also see an improvement for recall@5 and recall@10, with respectively 1.17 and 0.85 of increase.
This implies that the similarity learning with French captions increases English recognition when using bv.

To verify if we can generalize this result to a larger number of languages, we used muse  aligned for 30 languages.
Using the Multi30K dataset, we can also train the model on German (de) and Czech (cs).

First of all, when training with muse for English only, we can see a sharp decrease in performance, with a recall going from 66.08 to 63.1.
By comparing the model trained with w2v, we obtain similar results.
This could come from the fact that both muse and w2v embeddings do not have representation for out of vocabulary words like the FastText ones.
Moreover, rare words have much more chance to be wrongly projected because of space transformation.
When we train the model with additional languages, we can see a slight decrease in performance in English.
The maximum decrease is 1.01 for recall@10, but it is counter-balanced by an increase of 0.29 for the recall@1.




  
Experiment 2.

Given a sentence, in any language, we evaluate the rank of the corresponding image.
The evaluation is again made by batches of 1000.
The results are presented in table <ref>.

The first two lines of the table present the state-of-the-art results, with w2v and FastText embeddings.
We can see similar results as in the previous experiment.
With bv, we have results close to the FastText embeddings when training only in English.
This time, the recall is better with an increase of 2.68 for recall@1.
When trained with English and French, the recall@1 is increased by 3.65.
This implies, again, that we can improve performance by learning on an additional language.

Our model is able to use the multi-language representing the effectiveness of muse embeddings.
We train the model with English, and different combinations of French, German and Czech.
On English only, we have similar results to the w2v approach.
When adding new languages, we can see a decrease in performance for English.
We obtain a maximum decrease of 2.62 for recall@1 when the models saw English, French, German and Czech.








  
Experiment 3.


The model is trained with English only, then with English and French (en+fr), with English, French, and German (en+fr+de) and with English, French, German and Czech (all).
We can see a decrease in performance when adding French that is not present with other languages.
Otherwise, every time we add a new language the recall for this language logically increase.
The best performance is achieved with English+French+German+Czech, with an increase of 6.42 for multilingual retrieval.








  
Experiment 4:

With bv embeddings, we learn two languages at the same time, and test retrieval on one or two of these languages.
Results are shown in table <ref>.
Trained in English alone, the model gives worse performance than muse for languages not seen previously.
For example, with English-German bv and a model trained only in English, and test on German, we obtain only 22.96 recall@10, where muse embeddings obtain 44.18.
But when train on English and French, we obtain 55.22 recall, an increase of 26.39 compared to muse.
With German and English training, we have an increase of 15.16 on English only recall, with a recall of 61.44.
Meaning that, once again, learning a new language with bv enables better results in English, as the same kind of results is visible with French as well.










































































§ CONCLUSION




We presented a novel approach for multilingual text and image embeddings.
While the method provides close to the state-of-the-art performance in English-only context, its main advantage is that it enables the use of multilingual dataset in order to improve the performance.
We showed that using a new language during the training process improve image retrieval for other languages.

Our method uses a cnn to extract image information. It also uses aligned multilingual word embeddings and a rnn to produce text representations.
This way, it provides image and text embeddings that can be trained to produce comparable features.
We demonstrate that we can use this network to retrieve an image from a text and text from an image, with a multilingual representation.

We evaluated our method on the coco dataset for English-only results and shown that using bv embeddings enables the use of another language in order to improve the performance.
The obtained improvement is a 3.35 increase in performance on the coco dataset, and a 15.15 increase on the Multi30K dataset.
By using muse embeddings, we are able to embed more languages in the same model.
We showed that adding other languages decrease performance for English, but increase the recall in a multilingual environment.
For image retrieval from a caption in 4 languages, we obtain a 49.38 recall@10 on the Multi30K dataset.




ieee






