\begin{table}
\centering
\footnotesize
\begin{tabular}{lcc}
\toprule
\textbf{Model} & \textbf{pt} & \textbf{zh} \\
\midrule
Gold & 98.6 & 94.4 \\
\midrule
UR & 50.4 & 50.6 \\
M4-UR & 51.2 & 50.9 \\
M4-Prompts & 66.7 & 50.0 \\
M4-Prompts FT & 66.7 & 51.0 \\
PaLM 8B & 85.0 & 69.0 \\
PaLM 62B & 90.4 & 70.8 \\
PaLM 540B & \textbf{93.2} & \textbf{83.6} \\
Online & 50.0 & 50.0 \\
\bottomrule
\end{tabular}
\caption{Lexical accuracy on FRMT test. PaLM outperforms other approaches, while region-agnostic models like Online are guaranteed 50\%.}
\label{tab:lexical_accuracy}
\end{table}
\section{Evaluation Metrics}\label{sec:metrics}
While human judgments are the gold standard for evaluating machine-generated texts, collecting them can be difficult, time-consuming, and expensive. Therefore, generally the best practical method for fast iteration on benchmarks is the use of automatic metrics that are shown to correlate well with human judgments. We hypothesize that common reference-based MT evaluation metrics might have differing sensitivities to regional differences, and so we conduct a human evaluation of several baseline models (see \S\ref{sec:results_human}) and compute correlation of several automatic metrics with the human judgments. We also explore two additional metrics specifically for region-awareness: a modified version of our human metric, and an automatic lexical accuracy metric.

\subsection{Human evaluation}\label{sec:human_eval_mqm}
To obtain the highest fidelity human ratings, we use the expert-based 
Multidimensional Quality Metrics (MQM) evaluation framework proposed by \citet{Freitag2021ExpertsEA}. This is the same framework recommended by the WMT'21 Evaluation Campaign \citep{freitag-etal-2021-results}. For our evaluation, expert raters are shown a chunk of 10 contiguous English sentences from our test set with the corresponding translations from one combination of model (or human) and target region. Raters then identify errors in the translations, assigning a category and severity to each; see \citet{Freitag2021ExpertsEA} for details. Due to cost constraints, we evaluate 25\%{} of the test set, evenly distributed across our three evaluation buckets. Each chunk is rated by three raters.

\subsection{Lexical accuracy}\label{sec:lexical_accuracy}
To assess a model's ability to select lexical forms appropriate to the target region, we define a lexical accuracy metric.  As discussed in section \S\ref{sec:data_sampling}, sentences in our \texttt{lexical} bucket are chosen from Wikipedia articles containing specific words we expect to have distinct translations in the regions in question.  For instance, we include source sentences from the English Wikipedia article ``Bus'' in the Portuguese \texttt{lexical} bucket, as the word for bus is distinct in Brazil and Portugal (\textit{ônibus} vs.~\textit{autocarro}).  As the expected output forms are known ahead of time, we can directly measure the rate at which a model selects region-appropriate variants.

In Mandarin, there is an added complexity of script choice.  Mainland Mandarin is typically written in the ``simplified'' Han script, while Taiwan Mandarin is written in ``traditional'' script.  To disentangle lexical choice from script choice, we define lexical accuracy in a script-agnostic manner.  For example, for the word \textit{pineapple}, if the target region is Taiwan, we count both the (expected) traditional and (unexpected) simplified forms of the Taiwan variant \textit{fènglí} (\trad{鳳梨} and \simp{凤梨}) as correct, while counting both script forms of the Mainland variant \textit{bōluó} (\simp{菠萝} and \trad{菠蘿}) as incorrect.  This ensures that even if a model is outputting the wrong script, it will be rewarded or penalized based on its lexical choices. This also prevents the possibility of ``gaming'' our metric by only using the lexical forms and script of a single region.

Researchers sharing FRMT results should \textbf{report per-bucket BLEU and lexical accuracy} metrics on the test set, as shown in Tables \ref{tab:results_auto} and \ref{tab:lexical_accuracy}. These metrics are efficient to calculate with our provided evaluation scripts.

Our next three baseline models are different-sized versions of PaLM \citep{chowdhery2022palm}, a large language model that has demonstrated remarkable zero-shot and few-shot performance on a variety of tasks, often without needing any fine-tuning. Because it was pre-trained on a multilingual corpus, we include it as a baseline, serving as a representative of the class of recent large decoder-only causal language models including GPT-3 \citep{brown-etal-22-gpt3} and OPT-175B \citep{zhang-etal-22-opt}. We call these models \textbf{PaLM 540B}, \textbf{PaLM 62B}, and \textbf{PaLM 8B}, referring to their approximate parameter counts. The prompt for these models is larger than for the M4 Prompts models and is exemplar-based. It begins with ``Translate the following texts from English to X'', where ``X'' is the name of the language variety. This is followed by ten exemplars selected randomly from the \texttt{lexical} bucket,\footnote{The model has a fixed input sequence length, which includes the prompt, and a fixed output sequence length. Through rejection sampling, we ensure that the ten exemplars are short enough to leave at least 128 tokens for the input text, to match the 128 tokens allotted to the output.} where each exemplar is put on two lines: the first line contains the English text, prefixed by ``English:'', and the second containing the translation in the target variety, prefixed by the variety's name. At the end of the prompt, we show the model the input text and the language variety prefix, and decode from the model, taking everything up to the next newline as the output (or the end of the line if no newline was generated).

Table \ref{tab:results_auto} shows the performance of our baseline models on the automated metrics BLEU and BLEURT\@. The \textbf{``FRMT'' metric} is a summary of per-language performance, calculated as the geometric mean across regions of the arithmetic mean across buckets. Recall, while BLEU scores serve as a lightweight point of comparison, we recommend additionally reporting the more costly BLEURT where possible, as this correlates more strongly with human judgments.

Table \ref{tab:lexical_accuracy} shows \textbf{lexical accuracy} performance, assessing whether specific terms receive region-appropriate translations. Here, the PaLM models outperform alternatives by a wide margin. As even the smallest PaLM model has more than $2\times$ the parameters of our trained baselines (3.7B parameters each), this suggests that model capacity is a key ingredient for learning to use region-specific terminology in a few-shot manner. However, there is still a significant gap compared to human performance.
{"value": "98.6", "type": "Result", "training data/set": "xx", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "Gold", "model settings": {"xx": "yy"}} {"value": "94.4", "type": "Result", "training data/set": "xx", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "Gold", "model settings": {"xx": "yy"}} {"value": "50.4", "type": "Result", "training data/set": "mC4 OPUS", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "UR", "model settings": {"xx": "yy"}} {"value": "50.6", "type": "Result", "training data/set": "mC4 OPUS", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "UR", "model settings": {"xx": "yy"}} {"value": "51.2", "type": "Result", "training data/set": "a mixture of monolingual and parallel data from 112 languages mined from the web", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "M4-UR", "model settings": {"xx": "yy"}} {"value": "50.9", "type": "Result", "training data/set": "a mixture of monolingual and parallel data from 112 languages mined from the web", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "M4-UR", "model settings": {"xx": "yy"}} {"value": "66.7", "type": "Result", "training data/set": "a mixture of monolingual and parallel data from 112 languages mined from the web", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "M4-Prompts", "model settings": {"xx": "yy"}} {"value": "50.0", "type": "Result", "training data/set": "a mixture of monolingual and parallel data from 112 languages mined from the web", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "M4-Prompts", "model settings": {"xx": "yy"}} {"value": "66.7", "type": "Result", "training data/set": "a mixture of monolingual and parallel data from 112 languages mined from the web", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "M4-Prompts FT", "model settings": {"xx": "yy"}} {"value": "51.0", "type": "Result", "training data/set": "a mixture of monolingual and parallel data from 112 languages mined from the web", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "M4-Prompts FT", "model settings": {"xx": "yy"}} {"value": "85.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "PaLM 8B", "model settings": {"xx": "yy"}} {"value": "69.0", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "PaLM 8B", "model settings": {"xx": "yy"}} {"value": "90.4", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "PaLM 62B", "model settings": {"xx": "yy"}} {"value": "70.8", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "PaLM 62B", "model settings": {"xx": "yy"}} {"value": "93.2", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "83.6", "type": "Result", "training data/set": "a multilingual corpus", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "PaLM 540B", "model settings": {"xx": "yy"}} {"value": "50.0", "type": "Result", "training data/set": "xx", "test data/set": "FRMT pt", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "Online", "model settings": {"xx": "yy"}} {"value": "50.0", "type": "Result", "training data/set": "xx", "test data/set": "FRMT zh", "task": ["Few-Shot Region-Aware Machine Translation", "FRMT", "Machine Translation"], "metric": "lexical accuracy", "experimental settings": {"xx": "yy"}, "model": "Online", "model settings": {"xx": "yy"}}