\begin{table*}
\centering
\footnotesize
\begin{tabular}{rcccccc|c}
\toprule
& \multicolumn{2}{c}{\bf Lexical} & \multicolumn{2}{c}{\bf Entities} & \multicolumn{2}{c}{\bf Random} & \bf FRMT \\
\bf Exemplars & pt-BR & pt-PT & pt-BR & pt-PT & pt-BR & pt-PT & pt\\\midrule
0 & 0.0 (-1.0) & 0.0 (-1.2) & 0.0 (-1.4) & 0.0 (-1.6) & 0.1 (-0.8) & 0.0 (-1.3) & 0.0 (1.2)\\
1 & 51.3 (77.8) & 38.2 (74.4) & 56.5 (80.8) & 47.5 (78.5) & 53.7 (\textbf{76.6}) & 45.1 (73.8) & 48.4 (77.0) \\
2 & 51.8 (77.7) & 38.8 (75.0) & 57.3 (80.8) & 46.9 (78.5) & 52.9 (76.1) & 45.0 (74.4) & 48.5 (77.1) \\
5 & 52.8 (77.9) & 39.6 (75.0) & 57.9 (80.8) & 46.8 (78.5) & 54.5 (76.4) & 45.2 (74.6) & 49.1 (77.2) \\
10 & \textbf{54.0} (\textbf{78.7}) & \textbf{40.6} (\textbf{75.3}) & \textbf{58.3} (\textbf{80.9}) & \textbf{48.2} (\textbf{78.9}) & \textbf{54.6} (76.5) & \textbf{46.2} (\textbf{74.9}) & \textbf{50.0} (\textbf{77.5}) \\\midrule
& zh-CN & zh-TW & zh-CN & zh-TW & zh-CN & zh-TW & zh\\\midrule
0 & 0.0 (0.9) & 0.0 (3.2) & 0.0 (0.8) & 0.1 (6.5) & 0.1 (1.6) & 0.3 (4.3) & 0.1 (2.3)\\
1 & 37.2 (70.1) & 25.7 (67.5) & 47.4 (74.2) & 37.7 (73.0) & 39.7 (67.0) & 32.3 (65.5) & 36.3 (69.5)\\
2 & \textbf{37.7} (70.5) & \textbf{27.0} (\textbf{67.8}) & 48.3 (74.8) & 38.5 (\textbf{73.3}) & 39.3 (\textbf{67.1}) & 32.2 (\textbf{65.7}) & \textbf{36.9} (\textbf{69.8})\\
5 & 37.2 (70.2) & 26.4 (67.2) & \textbf{48.4} (74.7) & \textbf{39.0} (73.2) & \textbf{39.9} (66.8) & \textbf{32.4} (65.4) & \textbf{36.9} (69.6)\\
10 & 37.6 (\textbf{70.7}) & 25.5 (67.2) & 48.1 (\textbf{74.9}) & 38.6 (72.6) & 38.9 (66.9) & 30.4 (64.7) & 36.2 (69.5)\\
\bottomrule
\end{tabular}
\caption{FRMT dev set results of PaLM 540B, when varying the number of exemplars, shown in the format: BLEU~(BLEURT)\@. Across both languages, even one exemplar is sufficient for strong results. In Portuguese, increasing to $10$ exemplars gives marginal additional gains.}
\label{tab:palm_exemplars}
\end{table*}
\subsection{Data sampling method}\label{sec:data_sampling}
FRMT seeks to capture region-specific linguistic differences, as well as potential distractors in the form of entities that are strongly associated with one region, without necessarily having a different linguistic rendering (e.g., Lisbon vs.~S\~{a}o Paulo). To this end, we divide the dataset into three buckets (\texttt{lexical}, \texttt{entity}, \texttt{random}), each containing human translations of sentences extracted from different sets of English Wikipedia articles.\footnote{
As Wikipedia data source we use the training split of {\tt wiki40b} (v1.3.0) by \citet{wiki40b}, available at \scriptsize{\url{https://www.tensorflow.org/datasets/catalog/wiki40b}}.}

\textbf{BLEU \citep{papineni-etal-2002-bleu}:} This metric is based on token $n$-gram precision. We use \texttt{sacrebleu.corpus\_bleu} from \citet{post-2018-call}.

\textbf{chrF \citep{popovic-2015-chrf}:} This metric is based on F1 using character $n$-grams. We use \texttt{sacrebleu.corpus\_chrf} from \citet{post-2018-call}.

\textbf{BLEURT \citep{sellam-etal-2020-bleurt}:} This is a learned, model-based metric. While the original paper showed that BLEURT has good correlation with human judgments of translation quality, we note that BLEURT has not, to our knowledge, been evaluated with respect to human judgments of \textit{region-specific} translation quality. We use the authors' released checkpoint.

\textbf{BLEURT-D\{3,6,12\} \citep{sellam-etal-2020-bleurt}:} These are distilled versions of BLEURT that are less resource-intensive to run. They have 3, 6, and 12 layers, respectively. As above, we use checkpoints released by the authors.

Researchers sharing FRMT results should \textbf{report per-bucket BLEU and lexical accuracy} metrics on the test set, as shown in Tables \ref{tab:results_auto} and \ref{tab:lexical_accuracy}. These metrics are efficient to calculate with our provided evaluation scripts.

Our next three baseline models are different-sized versions of PaLM \citep{chowdhery2022palm}, a large language model that has demonstrated remarkable zero-shot and few-shot performance on a variety of tasks, often without needing any fine-tuning. Because it was pre-trained on a multilingual corpus, we include it as a baseline, serving as a representative of the class of recent large decoder-only causal language models including GPT-3 \citep{brown-etal-22-gpt3} and OPT-175B \citep{zhang-etal-22-opt}. We call these models \textbf{PaLM 540B}, \textbf{PaLM 62B}, and \textbf{PaLM 8B}, referring to their approximate parameter counts. The prompt for these models is larger than for the M4 Prompts models and is exemplar-based. It begins with ``Translate the following texts from English to X'', where ``X'' is the name of the language variety. This is followed by ten exemplars selected randomly from the \texttt{lexical} bucket,\footnote{The model has a fixed input sequence length, which includes the prompt, and a fixed output sequence length. Through rejection sampling, we ensure that the ten exemplars are short enough to leave at least 128 tokens for the input text, to match the 128 tokens allotted to the output.} where each exemplar is put on two lines: the first line contains the English text, prefixed by ``English:'', and the second containing the translation in the target variety, prefixed by the variety's name. At the end of the prompt, we show the model the input text and the language variety prefix, and decode from the model, taking everything up to the next newline as the output (or the end of the line if no newline was generated).

To test sensitivity to the number and choice of exemplars, we evaluate PaLM 540B while varying the set of exemplars used.  Table \ref{tab:palm_exemplars} shows the effect of ablating the number of exemplars in the range $0$--$10$.  We observe that a single exemplar is sufficient to achieve strong results, and gains from additional exemplars are marginal.
{"value": "0", "type": "Other"} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "1", "type": "Other"} {"value": "51.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "38.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "56.5", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "47.5", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "53.7", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "45.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "48.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "2", "type": "Other"} {"value": "51.8", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "38.8", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "57.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "46.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "52.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "45.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "48.5", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "5", "type": "Other"} {"value": "52.8", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "39.6", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "57.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "46.8", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "54.5", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "45.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "49.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "10", "type": "Other"} {"value": "54.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "40.6", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "58.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "48.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "54.6", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-BR", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "46.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random pt-PT", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "50.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0", "type": "Other"} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "0.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "0"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "1", "type": "Other"} {"value": "37.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "25.7", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "47.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "37.7", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "39.7", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "32.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "36.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "1"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "2", "type": "Other"} {"value": "37.7", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "27.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "48.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "38.5", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "39.3", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "32.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "36.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "2"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "5", "type": "Other"} {"value": "37.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "26.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "48.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "39.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "39.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "32.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "36.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "5"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "10", "type": "Other"} {"value": "37.6", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "25.5", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Lexical zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "48.1", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "38.6", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Entities zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "38.9", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-CN", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "30.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev Random zh-TW", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "36.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT dev zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "delta BLEU", "experimental settings": {"Exemplars": "10"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}}