\begin{table}\label{tab:corpus_stats}
\centering
\footnotesize
\begin{tabular}{ccrr}
\toprule
\textbf{Bucket} & \textbf{Split} & \textbf{Portuguese} & \textbf{Mandarin} \\
\midrule
\multirow{3}{3.4em}{Lexical} & Exemplar & 118 & 173\\
& Dev & 848 & 524\\ 
& Test & 874 & 538\\ 
\midrule
\multirow{3}{3.4em}{Entities} & Exemplar & 112 & 104\\
& Dev & 935 & 883\\ 
& Test & 985 & 932\\ 
\midrule
\multirow{3}{3.4em}{Random} & Exemplar & 111 & 111\\
& Dev & 744 & 744\\ 
& Test & 757 & 757\\ 
\midrule
\multirow{3}{3.4em}{Total} & Exemplar & 341 & 388\\
& Dev & 2527 & 2151\\ 
& Test & 2616 & 2227\\ 
\bottomrule
\end{tabular}
\caption{Number of sentence pairs by bucket, split, and language, as well as cross-bucket totals. Note, the \texttt{random} bucket contains the same English source sentences across the Portuguese and Mandarin sets.}
\end{table}
\begin{abstract}
We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of interest, including lexically distinct terms and distractor terms. We explore automatic evaluation metrics for FRMT and validate their correlation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guidelines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: \url{https://bit.ly/frmt-task}.
\end{abstract}

\section{Related Work}
Previous work on textual ``style transfer'' or ``attribute rewriting'' is related in trying to control fine-level stylistic features of generated text. Earlier work in this space leverages supervised parallel data \cite{jhamtani-etal-2017-shakespearizing}. Later work assumes labeled but non-parallel training data \cite{shen17style, li-etal-2018-delete, niu18multitask}, or foregoes training-time labels entirely, as in our setting, relying only on few-shot exemplars provided at inference time \cite{xu2020variational, riley-etal-2021-textsettr, garcia2021towards}. However, there is broad agreement that style transfer evaluation protocols are far from ideal \cite{pang-gimpel-2019-unsupervised, briakou-etal-2021-review, hu2022tst}, due to the underspecification of stylistic attributes (e.g.~formality, sentiment) and the lack of standardization across studies. Region-aware translation addresses these issues, providing an excellent test-bed for exploring few-shot attribute control---MT evaluation methods are relatively mature, and many regional language varieties can be sufficiently delineated for the task.

\section{FRMT Dataset}\label{sec:new-dataset}
We introduce the FRMT dataset for evaluating the quality of few-shot region-aware machine translation. The dataset covers two regions each for Portuguese (Brazil and Portugal) and Mandarin (Mainland and Taiwan).
Our overall approach for constructing the dataset is to sample English sentences from Wikipedia and acquire professional human translations into the target regional varieties.
Final quality control is done through manual evaluation by an independent set of translators, using the MQM protocol \cite{Freitag2021ExpertsEA} that we also employ to evaluate system translation quality.

\subsection{Data sampling method}\label{sec:data_sampling}
FRMT seeks to capture region-specific linguistic differences, as well as potential distractors in the form of entities that are strongly associated with one region, without necessarily having a different linguistic rendering (e.g., Lisbon vs.~S\~{a}o Paulo). To this end, we divide the dataset into three buckets (\texttt{lexical}, \texttt{entity}, \texttt{random}), each containing human translations of sentences extracted from different sets of English Wikipedia articles.\footnote{
As Wikipedia data source we use the training split of {\tt wiki40b} (v1.3.0) by \citet{wiki40b}, available at \scriptsize{\url{https://www.tensorflow.org/datasets/catalog/wiki40b}}.}

{\bf Entities}:  We manually select a balanced number of entities per language region such that they are strongly associated with one region, relying primarily on world knowledge and following Wikipedia hyperlinks.
Selection is done within each of a few broad entity types defined at the outset: people, locations, organizations, attractions/infrastructure, and other.
To mitigate availability bias in picking entities, we also took inspiration from automatically extracted entity mentions most distinctive (by log-odds) of either top-level web domain across the mC4 corpus \cite{xue-etal-2021-mt5}, for {\tt .pt} vs.~{\tt .br}, and {\tt .cn} vs.~{\tt .tw}.
The final selection comprises 38 Mandarin-focused and 34 Portuguese-focused entities.
Again we extract 100 source sentences from the beginning of the English Wikipedia article about each selected entity.

{\bf Random}: 
To include more naturally distributed phenomena, we also sampled 100 articles at random from the combined set of 28k articles appearing in Wikipedia's collections of ``featured'' or ``good''  articles.\footnote{
\scriptsize{\url{https://en.wikipedia.org/wiki/Wikipedia:Good_articles/all}} and
\scriptsize{\url{https://en.wikipedia.org/wiki/Wikipedia:Featured_articles}} (as of 2021-12-15).
}
We accept the inherent selection bias of such curated collections in exchange for improved text quality and factual accuracy.
Here, we can extract less text from more articles, taking up to 20 contiguous sentences from the start
of a randomly chosen section within each article.
The latter step counteracts the somewhat formulaic article structure that is common to the Wikipedia genre.
Unlike the other two buckets, this one features one common set of sentences to be translated into all four target variants.

\subsection{Corpus statistics}
For each bucket, we split our data into exemplar, development (dev), and test data. The exemplars are intended to be the only pairs where the region label is shown to the model, such as via few-shot or in-context learning \cite{brown-etal-22-gpt3}.

Table~\ref{tab:corpus_stats} reports the number of released sentence pairs for each combination of bucket, split, and language. For each document in our dataset, all sentences from that document appear only in a single split---this ensures that a system cannot ``cheat'' by memorizing word-region associations from the exemplars, or by overfitting to words and entities while hill-climbing on the validation set.
{"value": "118", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Lexical Exemplar", "dataset features": {"xx": "yy"}} {"value": "173", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Lexical Exemplar", "dataset features": {"xx": "yy"}} {"value": "848", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Lexical Dev", "dataset features": {"xx": "yy"}} {"value": "524", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Lexical Dev", "dataset features": {"xx": "yy"}} {"value": "874", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Lexical Test", "dataset features": {"xx": "yy"}} {"value": "538", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Lexical Test", "dataset features": {"xx": "yy"}} {"value": "112", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Entities Exemplar", "dataset features": {"xx": "yy"}} {"value": "104", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Entities Exemplar", "dataset features": {"xx": "yy"}} {"value": "935", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Entities Dev", "dataset features": {"xx": "yy"}} {"value": "883", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Entities Dev", "dataset features": {"xx": "yy"}} {"value": "985", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Entities Test", "dataset features": {"xx": "yy"}} {"value": "932", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Entities Test", "dataset features": {"xx": "yy"}} {"value": "111", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Exemplar", "dataset features": {"xx": "yy"}} {"value": "111", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Exemplar", "dataset features": {"xx": "yy"}} {"value": "744", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Dev", "dataset features": {"xx": "yy"}} {"value": "744", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Dev", "dataset features": {"xx": "yy"}} {"value": "757", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Test", "dataset features": {"xx": "yy"}} {"value": "757", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Test", "dataset features": {"xx": "yy"}} {"value": "341", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Exemplar", "dataset features": {"xx": "yy"}} {"value": "388", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Exemplar", "dataset features": {"xx": "yy"}} {"value": "2527", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Dev", "dataset features": {"xx": "yy"}} {"value": "2151", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Dev", "dataset features": {"xx": "yy"}} {"value": "2616", "type": "Data Stat.", "dataset": "Portuguese", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Test", "dataset features": {"xx": "yy"}} {"value": "2227", "type": "Data Stat.", "dataset": "Mandarin", "attribute name": ["Number of sentence pairs", "# sentence pairs"], "sub-set/group name": "Random Test", "dataset features": {"xx": "yy"}}