generate a dataset structure such that writers are sampled equally from the
languages we consider. That is, one fifth of the data are English-language
writers, one fifth are German-language writers, etc. However, we did not
ensure that all countries in which these languages are spoken are taken into
account. Lists of writers for the five languages were taken from Wikipedia:

  * •

English (American authors only):
https://de.wikipedia.org/wiki/Liste_amerikanischer_Schriftsteller

  * •

German: https://en.wikipedia.org/wiki/List_of_German-language_authors

  * •

Italian: https://en.wikipedia.org/wiki/List_of_Italian_writers

  * •

Dutch: https://en.wikipedia.org/wiki/List_of_Dutch-language_writers

  * •

Swedish: https://en.wikipedia.org/wiki/List_of_Swedish-language_writers

The list of Swedish-language writers had 186 entries and was the shortest.
Therefore, we randomly sampled 186 writers from each of the lists (without
replacement) and used those $186\times 5=930$ datapoints to compose the
dataset.

### Companies

The companies dataset tests for the headquarters locations of different
companies. Similar to writers, we try to cover five different countries (US,
Germany, Italy, Netherlands, Sweden), such that each of the languages we work
with is the dominant language in one of them. Each datapoint consists of a
company, e.g. $d=($ Volvo AB $)$, and the set of correct answers contains all
relevant variations in the city name, e.g. $A_{d}=\\{$ Gothenburg, Göteborg,
Gotemburgo, Gotenburg$\\}$. We took the 100 largest companies for each of
these countries from different lists on the internet:

  * •

US:
https://en.wikipedia.org/wiki/List_of_largest_companies_in_the_United_States_by_revenue

  * •

Germany:
https://de.wikipedia.org/wiki/Liste_der_gr%C3%B6%C3%9Ften_Unternehmen_in_Deutschland_(Wertsch%C3%B6pfung)

  * •

Italy: https://www.value.today/headquarters/italy

  * •

Netherlands: https://www.value.today/headquarters/netherlands

  * •

Sweden: https://www.stockblogs.se/sveriges-storsta-foretag/

If possible, we extracted both company and headquarters location from these
lists. When no location was given, we searched for it online. In total, the
dataset contains $100\times 5$ datapoints.

##  Appendix B Sense generation prompts

### Simple facts

For all simple facts datasets, except arithmetics, only the task instructions
(corresponding to the templates in Table 1) need to be translated, since the
input data does not change between languages. The prompt for translating is
“Please translate the following text into [LANGUAGE]:\n[TEXT]”. The prompt for
paraphrasing is “Please paraphrase the following text:\n[TEXT]”. The
arithmetics input data consists of spelled-out numbers, which have to be
translated as well. In the case of paraphrasing, these spelled-out numbers are
not paraphrased but remain in their original version. In the case of
translation, the model is instructed to translate each number separately using
the translation prompt above.

### Benchmark data

We use the model to generate alternative senses, treating the task instruction
and the input data separately. The prompt for translating is “Please translate
the following text into [LANGUAGE]:\n[TEXT]”. [LANGUAGE] is replaced by the
target language and [TEXT] by the instruction (for translating instructions)
or each datapoint from the benchmark (for translating input data). For
Belebele, it was necessary to explicitly instruct the model to translate
everything without answering the question. The prompt for paraphrasing differs
depending on whether task instructions or input data are paraphrased. The
prompt for paraphrasing the task instruction is “Please paraphrase the
following text:\n[TEXT]”. The prompt for paraphrasing the input data from the
benchmarks is task-specific to help preserve the structure of the original
task prompt:

  * •

PAWS: “Please paraphrase the following two sentences (separately). Reply only
with the paraphrased text and do not add any additional comments: \n[TEXT].”

  * •

XNLI: “Please paraphrase the following premise and hypothesis (separately).
Reply only with the paraphrased text and do not add any additional comments:
\n[TEXT].”

  * •

COPA: Please paraphrase the following premise and two alternatives
(separately). Reply only with the paraphrased text and do not add any
additional comments: \n[TEXT].”

  * •

Belebele: “Please paraphrase the following text passage, question, and
multiple-choice answer options (separately). Make sure to paraphrase
everything, including the passage and reply only with the paraphrased text and
do not add any additional comments:\n[TEXT].”

##  Appendix C Task instructions and alternative senses

### Simple facts

LABEL:tab:instructions_simple_facts shows the original English ($en$) task
instructions for the simple facts datasets as well as the model’s paraphrases
($en^{P}$) and translations ($de^{T}$, $it^{T}$, $nl^{T}$, $sv^{T}$) thereof.
Native speakers of the corresponding languages judged the paraphrases and
translations to be mostly accurate, although they tend to stay very close to
the English original. In some cases, this tendency leads to some formal
mistakes. For example, the Dutch instruction for arithmetics is “Wat is
[NUMBER1] plus [NUMBER2]? Antwoord alstublieft alleen met het juiste nummer
[…]”), where “Hoeveel is [NUMBER1] plus [NUMBER2]? Antwoord alstublieft alleen
met het juiste getal […]” would be more correct. In addition, there is a
grammatical mistake in the Swedish translation for elements, where the
definitive article of “the atomic number” should be expressed by a suffix on
the noun “atomnummer”, resulting in “atomnumret”.

Table 5: Simple facts task instructions. Task | Language |  Instruction
---|---|---
arithmetics | $en$ |  What is [NUMBER1] plus [NUMBER2]? Please reply with only the correct number (in numerical form) and no additional words.
$en^{P}$ |  What is the sum of [NUMBER1] and [NUMBER2]? Please respond with only the correct numerical answer and no extra words.
| $de^{T}$ |  Was ist [NUMBER1] plus [NUMBER2]? Bitte antworten Sie nur mit der korrekten Zahl (in numerischer Form) und ohne zusätzliche Wörter.
| $it^{T}$ |  Quanto fa [NUMBER1] più [NUMBER2]? Si prega di rispondere solo con il numero corretto (in forma numerica) e senza parole aggiuntive.
| $nl^{T}$ |  Wat is [NUMBER1] plus [NUMBER2]? Antwoord alstublieft alleen met het juiste nummer (in numerieke vorm) en geen extra woorden.
| $sv^{T}$ |  Vad är [NUMBER1] plus [NUMBER2]? Vänligen svara endast med det korrekta numret (i numerisk form) och inga ytterligare ord.
elements-from-element | $en$ |  What is the atomic number of the chemical element [ELEMENT]? Please reply with the number only and do not use any additional words.
$en^{P}$ |  Please provide the atomic number of the chemical element [ELEMENT] using only the number and no extra words.
$de^{T}$ |  Was ist die Ordnungszahl des chemischen Elements [ELEMENT]? Bitte antworten Sie nur mit der Zahl und verwenden Sie keine zusätzlichen Wörter.
| $it^{T}$ |  Qual è il numero atomico dell’elemento chimico [ELEMENT]? Si prega di rispondere solo con il numero e di non utilizzare altre parole aggiuntive.
| $nl^{T}$ |  Wat is het atoomnummer van het chemisch element [ELEMENT]? Antwoord alstublieft alleen met het nummer en gebruik geen extra woorden.
| $sv^{T}$ |  Vad är det atomnummer för grundämnet [ELEMENT]? Vänligen svara endast med numret och använd inga ytterligare ord.
elements-from-position | $en$ |  What is the atomic number of the chemical element in period [PERIOD] and group [GROUP]? Please reply with the number only and do not use any additional words.
$en^{P}$ |  Please provide the atomic number of the element in period [PERIOD] and group [GROUP]. Respond with only the number and no extra words.
$de^{T}$ |  ”Was ist die Ordnungszahl des chemischen Elements in Periode [PERIOD] und Gruppe [GROUP]? Bitte antworten Sie nur mit der Zahl und verwenden Sie keine zusätzlichen Wörter.
| $it^{T}$ |  Qual è il numero atomico dell’elemento chimico nel periodo [PERIOD] e nel gruppo [GROUP]. Si prega di rispondere solo con il numero e di non utilizzare altre parole aggiuntive.
| $nl^{T}$ |  Wat is het atoomnummer van het chemisch element in periode [PERIOD] en groep [GROUP]. Antwoord alstublieft alleen met het nummer en gebruik geen extra woorden.
| $sv^{T}$ |  Vad är det atomnummer för det kemiska elementet i period [PERIOD] och grupp [GROUP]. Vänligen svara endast med numret och använd inga ytterligare ord.
olympics-100m | $en$ |  Who won the [MEDAL] medal in the [GENDER] 100 meters at the [YEAR] Summer Olympics? Please reply with the name only and do not use any additional words.
$en^{P}$ |  Please provide the name of the athlete who won the [MEDAL] medal in the [GENDER] 100 meters at the [YEAR] Summer Olympics, using only the name and no extra words.
| $de^{T}$ |  Wer hat die [MEDAL]-Medaille im [GENDER]-100-Meter-Lauf bei den Olympischen Sommerspielen [YEAR] gewonnen? Bitte antworten Sie nur mit dem Namen und verwenden Sie keine zusätzlichen Wörter.
| $it^{T}$ |  Chi ha vinto la medaglia [MEDAL] nei 100 metri [GENDER] alle Olimpiadi estive del [YEAR]? Si prega di rispondere solo con il nome e di non utilizzare altre parole aggiuntive.
| $nl^{T}$ |  Wie heeft de [MEDAL] medaille gewonnen op de 100 meter voor [GENDER] tijdens de Zomerspelen van [YEAR]? Antwoord alstublieft alleen met de naam en gebruik geen extra woorden.
| $sv^{T}$ |  Vem vann [MEDAL] medaljen i [GENDER] 100 meter vid sommar-OS [YEAR]? Vänligen svara med namnet endast och använd inga ytterligare ord.
olympics-downhill | $en$ |  Who won the [MEDAL] medal in the [GENDER] downhill competition at the [YEAR] Winter Olympics? Please reply with the name only and do not use any additional words.
$en^{P}$ |  Please provide the name of the athlete who won the [MEDAL] medal in the [GENDER] downhill competition at the [YEAR] Winter Olympics, without using any extra words.
| $de^{T}$ |  Wer hat die [MEDAL]-Medaille im [GENDER]-Abfahrtsrennen bei den Olympischen Winterspielen [YEAR] gewonnen? Bitte anworten Sie nur mit dem Namen und verwenden Sie keine zusätzlichen Wörter.
| $it^{T}$ |  Chi ha vinto la medaglia [MEDAL] nella gara di discesa libera [GENDER] alle Olimpiadi invernali del [YEAR]? Per favore, rispondi solo con il nome e non utilizzare altre parole aggiuntive.
| $nl^{T}$ |  Wie heeft de [MEDAL] medaille gewonnen in de [GENDER] afdaling wedstrijd op de [YEAR] Olympische Winterspelen? Antwoord alstublieft alleen met de naam en gebruik geen extra woorden.
| $sv^{T}$ |  Vem vann [MEDAL] medaljen i [GENDER] störtloppstävling vid vinter-OS [YEAR]? Vänligen svara med namnet endast och använd inga ytterligare ord.
writers | $en$ |  In what year was the writer [AUTHOR] born? Please reply with the correct year only and do not use any additional words.
$en^{P}$ |  What is the birth year of the author [AUTHOR]? Please respond with only the correct year and avoid using extra words.
| $de^{T}$ |  In welchem Jahr wurde der Schriftsteller / die Schriftstellerin [AUTHOR] geboren? Bitte antworten Sie nur mit dem korrekten Jahr und verwenden Sie keine zusätzlichen Wörter.
| $it^{T}$ |  In che anno è nato lo scrittore / è nata la scrittrice [AUTHOR]? Per favore, rispondi solo con l’anno corretto e non utilizzare altre parole aggiuntive.
| $nl^{T}$ |  In welk jaar is de schrijver [AUTHOR] geboren? Uw antwoord moet alleen bestaan uit het juiste jaartal.
| $sv^{T}$ |  I vilket år föddes författaren [AUTHOR]? Ditt svar ska bara bestå av det korrekta året.
companies | $en$ |  In what city does [COMPANY] have its headquarters? Please reply only with the name of the city and no additional words.
$en^{P}$ |  Where is the headquarters of [COMPANY] located? Please respond with only the city name, without any extra words.
| $de^{T}$ |  In welcher Stadt hat [COMPANY] seinen Hauptsitz? Bitte antworten Sie nur mit dem Namen der Stadt und ohne zusätzliche Wörter.
| $it^{T}$ |  In quale città ha sede [COMPANY]? Si prega di rispondere solo con il nome della città e senza parole aggiuntive.
| $nl^{T}$ |  In welke stad heeft [COMPANY] zijn hoofdkantoor? Antwoord alstublieft alleen met de naam van de stad en geen extra woorden.
| $sv^{T}$ |  I vilken stad har [COMPANY] sitt huvudkontor? Vänligen svara endast med stadens namn och inga ytterligare ord.

### Benchmark data

LABEL:tab:instructions_benchmarks lists the original English ($en$) task
instructions for the benchmark datasets as well as the model’s paraphrases
($en^{P}$) and translations ($de^{T}$, $it^{T}$, $nl^{T}$, $sv^{T}$) thereof.
Native speakers of the corresponding languages judged the paraphrases and
translations to be generally accurate but some sentences contained minor
mistakes or aspects that the native speakers would have translated
differently. Points that were mentioned are that 1) the model translates
“premise” to “presupposto” in Italian (COPA and XNLI) even though “premessa”
is more appropriate and 2) the repeated use of “noch” in the Dutch XNLI
instruction is incorrect and the correct sentence should end with something
like “als de premisse de hypothese noch impliceert nog tegenspreekt”.

Table 6: Benchmark data task instructions. Task | Language |  Instruction
---|---|---
paws | $en$ |  Do the following two sentences have the same meaning?
Sentence 1: “[SENTENCE1]”
Sentence 2: “[SENTENCE2]”
Please reply with a single word, either “yes” or “no”.
$en^{P}$ |  Are the meanings of the following two sentences the same?
Sentence 1: “[SENTENCE1]”
Sentence 2: “[SENTENCE2]”
Please respond with either “yes” or “no”.
| $de^{T}$ |  Haben die folgenden beiden Sätze die gleiche Bedeutung?
Satz 1: “[SENTENCE1]”
Satz 2: “[SENTENCE2]”
Bitte antworten Sie mit einem einzigen Wort, entweder “ja” oder “nein”.
| $it^{T}$ |  Le seguenti due frasi hanno lo stesso significato?
Frase 1: “[SENTENCE1]”
Frase 2: “[SENTENCE2]”
Rispondi con una sola parola, “sì” o “no”.
| $nl^{T}$ |  Hebben de volgende twee zinnen dezelfde betekenis?
Zin 1: “[SENTENCE1]”
Zin 2: “[SENTENCE2]”
Antwoord alstublieft met één woord, ofwel “ja” ofwel “nee”.
| $sv^{T}$ |  Har de följande två meningarna samma betydelse?
Mening 1: “[SENTENCE1]”
Mening 2: “[SENTENCE2]”
Svara med ett enda ord, antingen “ja” eller “nej”.
xnli | $en$ |  Given the following premise and hypothesis, please identify whether the premise entails the hypothesis, contradicts the hypothesis, or neither of the two.
Premise: “[PREMISE]”
Hypothesis: “[HYPOTHESIS]”
Please reply with a single word: “entailment” if the premise entails the
hypothesis, “contradiction” if the premise contradicts the hypothesis, and
“neutral” if the premise neither entails nor contradicts the hypothesis.
$en^{P}$ |  Please determine if the premise and hypothesis are related.
Premise: “[PREMISE]”
Hypothesis: “[HYPOTHESIS]”
If the premise supports the hypothesis, indicate “entailment”. If the premise
contradicts the hypothesis, indicate “contradiction”. If there is no clear
relationship between the two, indicate “neutral”.
| $de^{T}$ |  Angesichts der folgenden Prämisse und Hypothese, bitte identifizieren Sie, ob die Prämisse die Hypothese impliziert, der Hypothese widerspricht oder weder das eine noch das andere.
Prämisse: “[PREMISE]”
Hypothese: “[HYPOTHESIS]”
Bitte antworten Sie mit einem einzigen Wort: “Implikation”, wenn die Prämisse
die Hypothese impliziert, “Widerspruch”, wenn die Prämisse der Hypothese
widerspricht, und “neutral”, wenn die Prämisse weder die Hypothese impliziert
noch ihr widerspricht.
| $it^{T}$ |  Dato il seguente presupposto e ipotesi, per favore identifica se il presupposto implica l’ipotesi, contraddice l’ipotesi o né implica né contraddice l’ipotesi.
Presupposto: “[PREMISE]”
Ipotesi: “[HYPOTHESIS]”
Per favore rispondi con una sola parola: “implicazione” se il presupposto
implica l’ipotesi, “contraddizione” se il presupposto contraddice l’ipotesi e
“neutrale” se il presupposto né implica né contraddice l’ipotesi.
| $nl^{T}$ |  Gegeven de volgende premisse en hypothese, identificeer alstublieft of de premisse de hypothese impliceert, de hypothese tegenspreekt, of geen van beide.
Premisse: “[PREMISE]”
Hypothesis: “[HYPOTHESIS]”
Antwoord alstublieft met één woord: “implicatie” als de premisse de hypothese
impliceert, “tegenspraak” als de premisse de hypothese tegenspreekt, en
“neutraal” als de premisse noch de hypothese impliceert noch tegenspreekt.
| $sv^{T}$ |  Givet följande premiss och hypotes, vänligen ange om premissen innebär hypotesen, motsäger hypotesen eller varken innebär eller motsäger hypotesen.
Premiss: “[PREMISE]”
Hypotes: “[HYPOTHESIS]”
Vänligen svara med ett enda ord: “innebär” om premissen innebär hypotesen,
“motsäger” om premissen motsäger hypotesen och “neutral” om premissen varken
innebär eller motsäger hypotesen.
copa | $en$ |  Given the following premise, which of the two alternatives is more plausible?
Premise: “[PREMISE]”
Alternative 1: “[CHOICE1]”
Alternative 2: “[CHOICE2]”
Please answer with a single word: “Alternative-1” if alternative 1 is more
plausible and “Alternative-2” if alternative 2 is more plausible.
$en^{P}$ |  Based on the provided premise, which of the two options is more likely?
Premise: “[PREMISE]”
Option 1: “[CHOICE1]”
Option 2: “[CHOICE2]”
Please respond with either “Option-1” if option 1 is more likely, or
“Option-2” if option 2 is more likely.
| $de^{T}$ |  Angesichts der folgenden Prämisse, welche der beiden Alternativen ist plausibler?
Prämisse: “[PREMISE]”
Alternative 1: “[CHOICE1]”
Alternative 2: “[CHOICE2]”
Bitte antworten Sie mit einem einzigen Wort: “Alternative-1”, wenn Alternative
1 plausibler ist, und “Alternative-2”, wenn Alternative 2 plausibler ist.
| $it^{T}$ |  Dato il seguente presupposto, quale delle due alternative è più plausibile?
Presupposto: “[PREMISE]”
Alternativa 1: “[CHOICE1]”
Alternativa 2: “[CHOICE2]”
Per favore, rispondi con una sola parola: “Alternativa-1” se l’alternativa 1 è
più plausibile e “Alternativa-2” se l’alternativa 2 è più plausibile.
| $nl^{T}$ |  Gegeven de volgende premisse, welke van de twee alternatieven is waarschijnlijker?
Premisse: “[PREMISE]”
Alternatief 1: “[CHOICE1]”
Alternatief 2: “[CHOICE2]”
Antwoord alstublieft met één woord: “Alternatief-1” als alternatief 1
waarschijnlijker is en “Alternatief-2” als alternatief 2 waarschijnlijker is.
| $sv^{T}$ |  Givet följande premiss, vilket av de två alternativen är mer troligt?
Premiss: “[PREMISE]”
Alternativ 1: “[CHOICE1]”
Alternativ 2: “[CHOICE2]”
Svara med ett enda ord: “Alternativ-1” om alternativ 1 är mer troligt och
“Alternativ-2” om alternativ 2 är mer troligt.
belebele | $en$ |  [PASSAGE]

[QUESTION]

Option A: [ANSWER1]
Option B: [ANSWER2]
Option C: [ANSWER3]
Option D: [ANSWER4]

Please reply with “A”, “B”, “C”, or “D” to indicate the correct answer. Your
reply should be a single letter and should not contain any additional words.
$en^{P}$ |  [PASSAGE]

[QUESTION]

A) [ANSWER1]
B) [ANSWER2]
C) [ANSWER3]
D) [ANSWER4]

Please respond with the letter corresponding to the correct answer choice.
Your response should be a single letter and should not include any extra
words.
| $de^{T}$ |  [PASSAGE]

[QUESTION]

Option A: [ANSWER1]
Option B: [ANSWER2]
Option C: [ANSWER3]
Option D: [ANSWER4]

Antworten Sie bitte mit “A”, “B”, “C”, oder “D”, um die richtige Antwort
anzugeben. Ihre Antwort sollte nur ein einzelner Buchstabe sein und keine
zusätzlichen Wörter enthalten.
| $it^{T}$ |  PASSAGE]

[QUESTION]

Opzione A: [ANSWER1]
Opzione B: [ANSWER2]
Opzione C: [ANSWER3]
Opzione D: [ANSWER4]

Rispondi con “A”, “B”, “C” o “D” per indicare la risposta corretta. La tua
risposta deve essere una singola lettera e non deve contenere parole
aggiuntive.
| $nl^{T}$ |  [PASSAGE]

[QUESTION]

Optie A: [ANSWER1]
Optie B: [ANSWER2]
Optie C: [ANSWER3]
Optie D: [ANSWER4]

Antwoord alstublieft met “A”, “B”, “C” of “D” om het juiste antwoord aan te
geven. Uw antwoord moet uit één letter bestaan en mag geen extra woorden
bevatten.
| $sv^{T}$ |  [PASSAGE]

[QUESTION]

Alternativ A: [ANSWER1]
Alternativ B: [ANSWER2]
Alternativ C: [ANSWER3]
Alternativ D: [ANSWER4]

Vänligen svara med “A”, “B”, “C”, eller “D” för att ange det korrekta svaret.
Ditt svar ska vara en enda bokstav och får inte innehålla några ytterligare
ord.

##  Appendix D Accuracy scores

### Simple facts

Table 7: Accuracy (%) on the simple fact datasets, with 95% confidence intervals. | arithmetics | elements | olympics | writers | companies | total
---|---|---|---|---|---|---
| - | elem | pos | 100m | downhill | - | - | avg
en | 99.4±1.0 | 100.0±nan | 37.8±10.0 | 55.4±8.1 | 37.6±9.4 | 76.2±2.9 | 78.2±3.8 | 73.5
enP | 98.6±1.4 | 100.0±nan | 42.2±10.0 | 54.1±8.1 | 31.6±9.4 | 76.2±2.8 | 76.0±3.8 | 72.2
deT | 45.2±4.6 | 98.9±5.6 | 40.0±10.0 | 50.7±8.1 | 35.0±9.4 | 76.8±2.8 | 75.4±3.8 | 68.7
itT | 44.0±4.4 | 100.0±nan | 36.7±10.0 | 51.4±8.1 | 35.0±9.4 | 75.3±2.9 | 73.6±4.0 | 67.4
nlT | 42.4±4.4 | 100.0±nan | 36.7±10.0 | 52.0±8.1 | 35.0±8.5 | 76.8±2.9 | 73.2±4.2 | 67.7
svT | 19.2±3.6 | 100.0±nan | 41.1±11.1 | 50.7±8.1 | 33.3±8.5 | 74.3±2.9 | 71.8±4.0 | 65.0

### Benchmark data

Table 8: Accuracy (%) on the benchmark datasets, with 95% confidence intervals. | paws | xnli | copa | belebele | avg
---|---|---|---|---|---
en | 75.6±1.9 | 43.7±1.4 | 84.4±3.4 | 85.9±2.3 | 72.4
enP | 67.6±2.1 | 53.5±1.4 | 82.2±3.4 | - | -
deT | 64.3±2.1 | 50.0±1.4 | 85.6±3.2 | 81.2±2.7 | 70.3
itT | 75.1±2.0 | 56.4±1.4 | 86.6±3.2 | 81.0±2.7 | 74.8
nlT | 71.9±2.0 | 50.9±1.4 | 83.4±3.4 | 79.0±2.7 | 71.3
svT | 55.9±2.2 | 47.0±1.4 | 89.2±2.8 | 79.1±2.8 | 67.8

##  Appendix E Accuracy based on containment versus exact match

On the simple facts datasets, the model is instructed to reply with the
correct entity (and no additional words), which we then use to quantify
consistency. Hence, it is important that the model actually follows that
instruction across all senses. Otherwise, it could be that the model replies
with “Friedrich Schiller was born in 1759” when prompted for a writer in
English but “1759” when prompted in German. While a failure to follow the
instruction in one language but not the other could be considered an unwanted
inconsistency, the meaning of both answers is arguably the same, and we would
like to differentiate between both cases.

Figure 9: Containment score minus exact match score across tasks and senses.

If the model replies correctly but not in one word, the response contains the
right answer but does not exactly match it. Figure 9 shows the distribution of
the difference in accuracy based on containment versus exact match. The scores
for companies and writers are calculated separately for each language-specific
subgroup of samples (i.e. US companies, German companies, …) to obtain more
detailed information. In most cases, the “containment” score is not at all or
only slightly higher than the exact match score. The only exception occurs for
Dutch companies when prompted with $en^{P}$, with a 7% difference in accuracy.
This mismatch arises because the model – while otherwise replying with only
the city name – always responds with a full sentence when the correct answer
is “The Hague” (e.g. “The headquarters of Shell PLC is located in The
Hague.”). Thus, except for this curious case, inconsistencies can largely not
be attributed to a failure to express a response in the correct form.

##  Appendix F Consistency scores

### Simple facts

Table 9: Consistency (%) on the simple fact datasets. | arithmetics | elements | olympics | writers | companies | total
---|---|---|---|---|---|---
| - | elem | position | 100m | downhill | - | - | avg
id | 100.0 | 100.0 | 91.1 | 90.5 | 88.9 | 87.1 | 97.4 | 92.8
enP | 99.0 | 100.0 | 77.8 | 82.4 | 61.5 | 82.7 | 91.2 | 86.0
deT | 45.0 | 98.9 | 71.1 | 80.4 | 76.9 | 81.2 | 89.4 | 81.7
itT | 44.2 | 100.0 | 70.0 | 75.7 | 70.1 | 81.2 | 86.4 | 79.8
nlT | 42.2 | 100.0 | 67.8 | 83.8 | 64.1 | 79.1 | 88.8 | 79.8
svT | 19.4 | 100.0 | 70.0 | 83.8 | 59.8 | 77.3 | 84.8 | 76.2

### Benchmark data

Table 10: Consistency (%) on the benchmark datasets. | paws | xnli | copa | belebele | avg
---|---|---|---|---|---
id | 95.2 | 96.0 | 96.8 | 97.1 | 96.3
enP | 76.5 | 56.3 | 85.2 | - | -
deT | 74.7 | 51.2 | 88.8 | 84.7 | 74.8
itT | 82.2 | 57.5 | 85.8 | 85.1 | 77.6
nlT | 82.4 | 73.1 | 91.0 | 85.8 | 83.1
svT | 67.9 | 82.8 | 86.0 | 83.3 | 80.0

##  Appendix G Examples of inconsistent responses

Table 11: Examples of inconsistencies for simple facts. We report the first ten inconsistent samples per dataset and sense. Task | Senses |  Examples
---|---|---
arithmetics | ($en$ | $en$) |  -
($en$ | $en^{P}$) |  (540 | 340), (1770 | 1778), (237 | Two hundred thirty-seven.), (1173 | One thousand one hundred seventy-three), (1013 | One thousand thirteen.)
| ($en$ | $de^{T}$) |  (778 | 678), (618 | 1008), (926 | 526), (115 | 915), (924 | 524), (1535 | 1035), (1693 | 1689), (1437 | 1337), (1151 | 1248), (1248 | 1448)
| ($en$ | $it^{T}$) |  (778 | 678), (858 | 788), (1471 | 1437), (926 | 836), (924 | 923), (577 | 577 + 300 = 877), (1535 | 1335), (1693 | 1683), (1437 | 1497), (1151 | 1051)
| ($en$ | $nl^{T}$) |  (778 | 678), (965 | 865), (858 | 958), (926 | 726), (115 | 109), (1535 | 935), (1693 | 1689), (1437 | 1338), (1151 | 846), (1248 | 848)
| ($en$ | $sv^{T}$) |  (778 | 784 + 94 = 878), (965 | 929), (858 | 792), (1471 | 1465), (926 | 733 + 163 = 896), (1277 | 1170), (1304 | 645), (924 | 923), (577 | 577 + 300 = 877), (1535 | 825)
elements- | ($en$ | $en$) |  -
from- | ($en$ | $en^{P}$) |  -
element | ($en$ | $de^{T}$) |  (114 | 9)
| ($en$ | $it^{T}$) |  -
| ($en$ | $nl^{T}$) |  -
| ($en$ | $sv^{T}$) |  -
elements-from-position | ($en$ | $en$) |  (22 | 20), (13 | 31), (17 | 107), (107 | 104), (106 | 46), (86 | 14), (33 | 51), (17 | 53)
($en$ | $en^{P}$) |  (16 | 8), (19 | 37), (36 | 26), (28 | 39), (13 | 31), (23 | 55), (22 | 38), (45 | 46), (33 | 51), (16 | 34)
($en$ | $de^{T}$) |  (12 | 4), (16 | 8), (19 | 11), (35 | 17), (36 | 26), (28 | 39), (48 | 30), (33 | 15), (38 | 12), (22 | 23)
| ($en$ | $it^{T}$) |  (2 | 1), (12 | 4), (13 | 5), (7 | 15), (16 | 8), (35 | 23), (28 | 35), (28 | 40), (48 | 40), (13 | 31)
| ($en$ | $nl^{T}$) |  (12 | 4), (13 | 5), (16 | 8), (21 | 13), (35 | 23), (28 | 39), (28 | 40), (48 | 40), (33 | 15), (38 | 12)
| ($en$ | $sv^{T}$) |  (2 | 1), (13 | 5), (16 | 8), (17 | 9), (21 | 23), (36 | 26), (28 | 39), (28 | 40), (48 | 40), (13 | 31)
olympics-100m | ($en$ | $en$) |  (Charley Paddock | Harold Abrahams), (Arthur Jonath | Eddie Tolan), (Lloyd LaBeach | Herbert McKenley), (Lloyd LaBeach | Herbert McKenley), (Herb McKenley | Hector Hogan), (Ben Johnson | Calvin Smith), (Kim Collins | Justin Gatlin), (Ethel Smith | Elizabeth Robinson), (Shirley Strickland | Marjorie Jackson), (Shirley Strickland | Marlene Mathews)
($en$, $en^{P}$) |  (Francis Lane | Frank Lane), (Fay Moulton | Frank Jarvis), (Nathaniel Cartmell | Reggie Walker), (Nate Cartmell | Reggie Walker), (Arthur Porritt | Percy Williams), (Arthur Jonath | Percy Williams), (Barney Ewell | Herb McKenley), (Lloyd LaBeach | Herb McKenley), (Lloyd LaBeach | Herb McKenley), (Herb McKenley | Hector Hogan)
| ($en$ | $de^{T}$) |  (Francis Lane | Frank Lane), (Nate Cartmell | Reggie Walker), (Arthur Porritt | Arthur Jonath), (Arthur Porritt | Arthur Jonath), (Arthur Jonath | Eddie Tolan), (Arthur Jonath | Eddie Tolan), (Arthur Jonath | Ralph Metcalfe), (Lloyd LaBeach | Barney Ewell), (Lloyd LaBeach | Herb McKenley), (Enrique Figuerola | Bob Hayes)
| ($en$ | $it^{T}$) |  (Fay Moulton | Frank Castle), (Nathaniel Cartmell | Reginald Walker), (Nate Cartmell | Reginald Walker), (Charley Paddock | Harold Abrahams.), (Arthur Porritt | Percy Williams.), (Arthur Porritt | Arthur Jonath), (Arthur Jonath | Eddie Tolan.), (Arthur Jonath | Eddie Tolan.), (Arthur Jonath | Ralph Metcalfe.), (Lloyd LaBeach | Barney Ewell.)
| ($en$ | $nl^{T}$) |  (Fay Moulton | Frank Waller), (Arthur Porritt | Arthur Jonath), (Arthur Porritt | Arthur Jonath), (Lloyd LaBeach | Barney Ewell), (Lloyd LaBeach | Herb McKenley), (Herb McKenley | Thane Baker.), (Enrique Figuerola | Edwin Roberts.), (Valeriy Borzov | Valeri Borzov), (Ben Johnson | Calvin Smith.), (Ato Boldon | Maurice Greene)
| ($en$ | $sv^{T}$) |  (Francis Lane | Frank Lane), (Ralph Craig | Donald Lippincott), (Arthur Porritt | Arthur Jonath), (Lloyd LaBeach | Barney Ewell.), (Lloyd LaBeach | Herb McKenley), (Valeriy Borzov | Valeri Borzov), (Ben Johnson | Calvin Smith.), (Linford Christie | Carl Lewis), (Ato Boldon | Maurice Greene.), (Kim Collins | Asafa Powell)
olympics-downhill | ($en$ | $en$) |  (Egon Zimmermann | Guy Périllat), (Franz Klammer | Bernhard Russi), (Franck Piccard | Franz Heinzer), (Didier Défago | Didier Defago), (Beat Feuz | Kjetil Jansrud), (Hedy Schlunegger | Trude Beiser-Jochum), (Andrea Mead-Lawrence | Trude Beiser-Jochum), (Trude Beiser-Jochum | Hanni Wenzel), (Christl Haas | Christine Goitschel), (Brigitte Oertli | Vreni Schneider)
($en$ | $en^{P}$) |  (Zeno Colò | Andreas Molterer), (Christian Pravda | Anton Sailer), (Christian Pravda | Andreas), (Guy Périllat | Jean Vuarnet.), (Egon Zimmermann | Jean-Claude Killy), (Bernhard | Franz Klammer), (Leonhard Stock | Bill Johnson), (Anton Steiner | Bill Johnson), (Franck Piccard | Kjetil André Aamodt), (Hans Knauss | Hermann Maier.)
| ($en$ | $de^{T}$) |  (Egon Zimmermann | Guy Périllat), (Bernhard | Bernhard Russi), (Anton Steiner | Peter Müller), (Franz | Franz Heinzer), (Hermann Maier | Lasse Kjus), (Hans Knauss | Hermann Maier), (Antoine Dénériaz | Michael Walchhofer), (Kjetil André Aamodt | Michael Walchhofer), (Bode Miller | Aksel Lund Svindal), (Kjetil Jansrud | Aksel Lund Svindal)
| ($en$ | $it^{T}$) |  (Egon Zimmermann | Jean-Claude Killy.), (Egon Zimmermann | Guy Périllat.), (Bernhard | Bernhard Russi.), (Leonhard Stock | Peter Wirnsberger.), (Franz | Franz Heinzer.), (Tommy Moe | Markus Wasmeier.), (Hans Knauss | Lasse Kjus.), (Fritz Strobl | Lasse Kjus.), (Antoine Dénériaz | Michael Walchhofer.), (Kjetil André Aamodt | Michael Walchhofer.)
| ($en$ | $nl^{T}$) |  (Zeno Colò | Andrea Mead-Lawrence), (Egon Zimmermann | Guy Périllat), (Egon Zimmermann | Guy Périllat), (Bernhard | Bernhard Russi), (Leonhard Stock | Peter Müller), (Leonhard Stock | Peter Müller), (Franz | Franz Heinzer), (Franz Heinzer | Franck Piccard), (Franck Piccard | Franz Heinzer), (Tommy Moe | Patrick Ortlieb)
| ($en$ | $sv^{T}$) |  (Egon Zimmermann | Guy Périllat), (Egon Zimmermann | Guy Périllat.), (Bernhard | Bernhard Russi), (Anton Steiner | Bill Johnson), (Franz | Franz Heinzer.), (Franck Piccard | Franz Heinzer), (Tommy Moe | Markus), (Hermann Maier | Lasse Kjus.), (Hans Knauss | Lasse Kjus.), (Antoine Dénériaz | Fritz Strobl.)
writers | ($en$ | $en$) |  (1978 | 1977), (1903 | 1912), (1911 | 1901), (1965 | 1968), (1982 | 1984), (1975 | 1980), (1962 | 1939), (1880 | 1891), (1930 | 1936), (1851 | 1871)
($en$ | $en^{P}$) |  (1952 | 1944), (1978 | 1977), (1940 | 1925), (1992 | I’m sorry, but I don’t have access to personal information about individuals unless it has been shared with me in the course of our conversation.), (1903 | 1912), (1945 | 1939), (1956 | 1961), (1935 | 1923), (1955 | 1949)
| ($en$ | $de^{T}$) |  (1952 | 1949), (1978 | 1977), (1932 | 1941), (1955 | 1953), (1903 | 1922), (1943 | 1956), (1940 | 1939), (1956 | 1961), (1935 | 1923), (1911 | 1901), (1965 | 1962)
| ($en$ | $it^{T}$) |  (1992 | 1985), (1929 | 1932), (1903 | 1921), (1945 | 1935), (1940 | 1943), (1956 | 1961), (1935 | 1923), (1965 | 1962), (1982 | 1986), (1975 | 1969)
| ($en$ | $nl^{T}$) |  (1978 | 1977), (1940 | 1910), (1992 | 1987), (1903 | 1922), (1945 | 1950), (1941 | 1935), (1940 | Het juiste jaartal van de geboorte van schrijver Jeannette Howard Foster is niet beschikbaar.), (1980 | Ik heb geen informatie over een schrijver genaamd Eric San Juan.), (1956 | 1961), (1935 | 1923)
| ($en$ | $sv^{T}$) |  (1943 | 1938), (1941 | 1932), (1952 | 1949), (1978 | 1976), (1932 | Bob McGrath föddes år 1932.), (1992 | Jag beklagar, men jag har ingen information om författaren Zach Hughes och när han föddes.), (1903 | 1921), (1890 | 1871), (1963 | Jag beklagar, men jag kan inte hitta information om författaren Susan Wrights födelseår.), (1943 | 1956)
companies | ($en$ | $en$) |  (Dear user, the headquarters of Ford Motor Company is located in Dearborn. | Dear user, Ford Motor Company has its headquarters in Dearborn.), (Mayfield Village | Cleveland), (Berlin | Munich), (Frankfurt | Mannheim), (Milan | Genoa), (Rome | Milan), (Rome | Milan), (Hilversum | Amsterdam), (Hoofddorp | Hague), (Apeldoorn | Heerenveen)
($en$ | $en^{P}$) |  (Issaquah | Seattle), (North Chicago | Chicago), (Fort Worth | Dallas), (Cologne | Frankfurt), (Stuttgart | Munich), (Salzgitter | Salzgitter AG is headquartered in Salzgitter.), (Jena | Oberkochen), (Nuremberg | Frankfurt), (San Donato Milanese | Milan), (Bergamo | Stezzano)
| ($en$ | $de^{T}$) |  (Issaquah | Seattle), (Dear user, the headquarters of Ford Motor Company is located in Dearborn. | Dear user, Ford Motor Company has its headquarters in Dearborn.), (Purchase | New York City), (Kenilworth | New York), (Mayfield Village | Cleveland), (Fort Worth | Dallas), (Cologne | Frankfurt), (Selm | Lünen), (Munich | Ehningen), (Stuttgart | Ismaning)
| ($en$ | $it^{T}$) |  (Irving | Houston), (Chesterbrook | Philadelphia), (Issaquah | Seattle), (Dublin | Dublino), (Bloomfield | Philadelphia), (Dear user, the headquarters of Ford Motor Company is located in Dearborn. | Detroit), (Purchase | New York City.), (White Plains | New York.), (Chicago | Seattle), (North Chicago | Chicago.)
| ($en$ | $nl^{T}$) |  (Minnetonka | Minneapolis), (Issaquah | Seattle), (Dear user, the headquarters of Ford Motor Company is located in Dearborn. | Dear user, Ford Motor Company heeft zijn hoofdkantoor in Dearborn.), (Deerfield | De stad waar Walgreens Boots Alliance Inc. zijn hoofdkantoor heeft, is Deerfield.), (Purchase | New York City), (White Plains | New York City), (Mayfield Village | Cleveland), (Fort Worth | Dallas), (Cologne | Frankfurt), (Selm | Lünen)
| ($en$ | $sv^{T}$) |  (Minnetonka | Minneapolis.), (Dear user, the headquarters of Ford Motor Company is located in Dearborn. | Dear user, Ford Motor Company’s headquarters is located in Dearborn.), (Chicago | San Francisco.), (Purchase | New York.), (White Plains | Hamilton), (Austin | San Francisco.), (Fremont | Toronto), (North Chicago | Chicago), (Stamford | St. Louis), (Mayfield Village | Cleveland.)

##  Appendix H Accuracy scores for ablation experiments

Table 12: Accuracy scores for the ablation experiments. We consider ablations in which we translate/paraphrase only the instruction (columns $I$) or only the input data (columns $X$). | paws | xnli | copa | belebele
---|---|---|---|---
| $I$ | $X$ | $I$ | $X$ | $I$ | $X$ | $I$ | $X$
en | 75.6 | 43.7 | 84.4 | 85.9
enP | 71.8 | 69.7 | 54.2 | 44.5 | 88.2 | 80.6 | 87.4 | -
deT | 62.5 | 78.7 | 49.2 | 38.0 | 88.0 | 82.6 | 86.2 | 80.0
itT | 72.8 | 76.8 | 57.2 | 38.4 | 91.0 | 81.8 | 86.0 | 78.0
nlT | 68.7 | 78.9 | 48.1 | 36.8 | 86.4 | 84.8 | 85.0 | 79.2
svT | 59.1 | 76.1 | 48.3 | 37.1 | 93.0 | 81.4 | 85.6 | 79.8

##  Appendix I Genbench evaluation card

Our work uses generalisation across senses to assess task understanding in
LLMs. In Figure 10, we provide the GenBench eval card (Hupkes et al., 2023) of
our experiments.

Motivation
---
Practical Cognitive Intrinsic Fairness $\boxtimes$
Generalisation type
Compositional Structural Cross Task Cross Language Cross Domain Robust- ness
$\boxtimes$ $\boxtimes$
Shift type
Covariate Label Full Assumed $\boxtimes$
Shift source
Naturally occuring Partitioned natural Generated shift Fully generated
$\boxtimes$
Shift locus
Train–test Finetune train–test Pretrain– train Pretrain– test $\boxtimes$
Figure 10: Our experiments assess cross-lingual generalisation for natural
corpora, in pretrained LLMs, to assess LLM task understanding.