<p>It is still not fully understand what multilingual BERT does and why it works. Recently there were two papers (first <a href="https://arxiv.org/pdf/1906.01502.pdf" rel="nofollow noreferrer">from June</a>, second <a href="https://arxiv.org/pdf/1911.03310.pdf" rel="nofollow noreferrer">from November</a>) that play around with this a little bit.</p>

<p>From the papers, it seems that the vectors tend to cluster according to languages (and even language families), so it is super easy to classify the language. This is the clustering that is shown <a href="https://arxiv.org/pdf/1911.03310.pdf" rel="nofollow noreferrer">in the paper</a>:</p>

<p><a href="https://i.stack.imgur.com/gHGJy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/gHGJy.png" alt="enter image description here"></a></p>

<p>Because of that you can subtract the mean of the language from the representation and end up with a somehow cross-lingual vector that both the papers show that can be used for cross-lingual sentence retrieval.</p>

<p>Also, it seems a thousand parallel sentences (e.g., in both languages) are enough to learn a projection between the languages. Note that they did not use the <code>[CLS]</code> vector, but they mean-pooled the vectors for individual subwords.</p>
Multilingual Bert sentence vector captures language used more than meaning - working as interned?
<p>Playing around with BERT, I downloaded the Huggingface Multilingual Bert and entered three sentences, saving their sentence vectors (the embedding of <code>[CLS]</code>), then translated them via Google Translate, passed them through the model and saved their sentence vectors.</p>

<p>I then compared the results using cosine similarity.</p>

<p>I was surprised to see that each sentence vector was pretty far from the one generated from the sentence translated from it (0.15-0.27 cosine distance) while different sentences from the same language were quite close indeed (0.02-0.04 cosine distance).</p>

<p>So instead of having sentences of similar meaning (but different languages) grouped together (in 768 dimensional space ;) ), dissimilar sentences of the same language are closer. </p>

<p>To my understanding the whole point of Multilingual Bert is inter-language transfer learning - for example training a model (say, and FC net) on representations in one language and having that model be readily used in other languages.  </p>

<p>How can that work if sentences (of different languages) of the exact meaning are mapped to be more apart than dissimilar sentences of the same language?  </p>

<p>My code:</p>

<pre><code>import torch

import transformers
from transformers import AutoModel,AutoTokenizer

bert_name="bert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(bert_name)
MBERT = AutoModel.from_pretrained(bert_name)

#Some silly sentences
eng1='A cat jumped from the trees and startled the tourists'
e=tokenizer.encode(eng1, add_special_tokens=True)
ans_eng1=MBERT(torch.tensor([e]))

eng2='A small snake whispered secrets to large cats'
t=tokenizer.tokenize(eng2)
e=tokenizer.encode(eng2, add_special_tokens=True)
ans_eng2=MBERT(torch.tensor([e]))

eng3='A tiger sprinted from the bushes and frightened the guests'
e=tokenizer.encode(eng3, add_special_tokens=True)
ans_eng3=MBERT(torch.tensor([e]))

# Translated to Hebrew with Google Translate
heb1='חתול קפץ מהעץ והבהיל את התיירים'
e=tokenizer.encode(heb1, add_special_tokens=True)
ans_heb1=MBERT(torch.tensor([e]))

heb2='נחש קטן לחש סודות לחתולים גדולים'
e=tokenizer.encode(heb2, add_special_tokens=True)
ans_heb2=MBERT(torch.tensor([e]))

heb3='נמר רץ מהשיחים והפחיד את האורחים'
e=tokenizer.encode(heb3, add_special_tokens=True)
ans_heb3=MBERT(torch.tensor([e]))


from scipy import spatial
import numpy as np

# Compare Sentence Embeddings

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_heb1[1].data.numpy())

print ('Eng1-Heb1 - Translated sentences',result)


result = spatial.distance.cosine(ans_eng2[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Eng2-Heb2 - Translated sentences',result)

result = spatial.distance.cosine(ans_eng3[1].data.numpy(), ans_heb3[1].data.numpy())

print ('Eng3-Heb3 - Translated sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_heb1[1].data.numpy(), ans_heb2[1].data.numpy())

print ('Heb1-Heb2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Heb1-Heb3 - Similiar sentences',result)

print ("\n---\n")

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng2[1].data.numpy())

print ('Eng1-Eng2 - Different sentences',result)

result = spatial.distance.cosine(ans_eng1[1].data.numpy(), ans_eng3[1].data.numpy())

print ('Eng1-Eng3 - Similiar sentences',result)

#Output:
"""
Eng1-Heb1 - Translated sentences 0.2074061632156372
Eng2-Heb2 - Translated sentences 0.15557605028152466
Eng3-Heb3 - Translated sentences 0.275478720664978

---

Heb1-Heb2 - Different sentences 0.044616520404815674
Heb1-Heb3 - Similar sentences 0.027982771396636963

---

Eng1-Eng2 - Different sentences 0.027982771396636963
Eng1-Eng3 - Similar sentences 0.024596810340881348
"""
</code></pre>

<p>P.S.</p>

<p>At least the Heb1 was closer to Heb3 than to Heb2.
This was also observed for the English equivalents, but less so. </p>