11institutetext: Chair of Data Science, University of Passau, Germany

# LLMs in the Loop: Leveraging Large Language Model Annotations for Active
Learning in Low-Resource Languages

Nataliia Kholodna  Sahib Julka Corresponding author<EMAIL_ADDRESS>Mohammad Khodadadi  Muhammed Nurullah Gumus  Michael Granitzer

###### Abstract

Low-resource languages face significant barriers in AI development due to
limited linguistic resources and expertise for data labeling, rendering them
rare and costly. The scarcity of data and the absence of preexisting tools
exacerbate these challenges, especially since these languages may not be
adequately represented in various NLP datasets. To address this gap, we
propose leveraging the potential of LLMs in the active learning loop for data
annotation. Initially, we conduct evaluations to assess inter-annotator
agreement and consistency, facilitating the selection of a suitable LLM
annotator. The chosen annotator is then integrated into a training loop for a
classifier using an active learning paradigm, minimizing the amount of queried
data required. Empirical evaluations, notably employing GPT-4-Turbo,
demonstrate near-state-of-the-art performance with significantly reduced data
requirements, as indicated by estimated potential cost savings of at least
42.45 times compared to human annotation. Our proposed solution shows
promising potential to substantially reduce both the monetary and
computational costs associated with automation in low-resource settings. By
bridging the gap between low-resource languages and AI, this approach fosters
broader inclusion and shows the potential to enable automation across diverse
linguistic landscapes.

###### Keywords:

Natural Language Processing Named Entity Recognition Foundation Models Active
Learning Low-resource Languages Large Language Models.

##  1 Introduction

Large Language Models (LLMs) like Claude 3 Opus [3] and GPT-4 [20], have
demonstrated exceptional performance across various Natural Language
Processing (NLP) tasks, particularly in high-resource languages such as
English [18]. However, their effectiveness in low-resource languages remains
underexplored due to limited comprehensive datasets for evaluation [5]. This
research gap arises from both the scarcity of data and the high cost
associated with human annotation, which is crucial for training and evaluating
models in these languages [19, 9].

Active learning (AL) strategies offer a promising solution by iteratively
selecting informative samples to optimize resource utilization and accelerate
model learning [15, 17]. This motivates a key research question: Can the
combination of state-of-the-art LLMs, effective prompting, and active learning
strategies enable resource-efficient data annotation in low-resource
languages? Although there have been some advances in tackling the difficulties
of data annotation and model training in this context [21], automating these
processes through foundation models is still largely unexplored.

With an overarching objective to minimize costs and resources, we introduce a
novel methodology that integrates foundation models into an active learning
framework to enhance the annotation process. This study investigates the
application of our proposed methodology specifically in Named Entity
Recognition (NER) tasks within low-resource language contexts, using African
languages as a focal point for experimentation. The findings from this study
hold potential for extending the methodology to other low-resource languages,
thereby advancing progress in NLP across various domains. Concretely, our
contributions can be listed as follows:

  * •

We conduct a comparative evaluation of various popular LLMs (GPT-4-Turbo,
Claude 3 Opus, Sonnet [3], Gemini 1.0 [24], Mistral 7b [14], Llama 2 70b [26],
and Starling-LM 7b [33]) for NER tasks in low-resource African languages,
analyzing their performance differences in accuracy and consistency of
annotation.

  * •

We investigate common NER annotation challenges faced by LLMs, including token
skipping and output parsing. We assess the impact of prompt quality and batch
annotation on output accuracy.

  * •

We introduce a novel methodology to quantify data leakage by measuring
potential data contamination.

  * •

We quantify the potential cost savings achieved by integrating LLMs into an
active learning framework for data annotation, demonstrating significant
reductions in cost and data requirements compared to traditional human-
annotation methods111Open source code can be found at:
https://anonymous.4open.science/r/llms-in-the-loop-4E32..

##  2 LLMs in the Loop

Large Language Models have rapidly advanced, with prompting methods like In-
context learning (ICL) enabling LLMs to perform well in few-shot and zero-shot
settings across various language tasks [6, 29, 16, 23, 32]. This includes
information extraction [28, 27] and even achieving a performance that is often
comparable to or exceeds that of human annotators for popular languages on
crowdsourcing tasks [11, 7, 34, 13, 4]. Although preliminary work suggests
that LLMs could act as active annotators for tasks such as NER [31], their
performance in low-resource languages remains a critical research gap.

We test the applicability of these models as ad-hoc annotators in an AL
paradigm. This is aimed at satisfying two of our main objectives: a) reducing
annotation costs, and b) saving computational resources.

Active learning is based on selecting the most informative data points for
training, enabling the task-specific model to learn more efficiently from a
smaller dataset. By sampling data with high uncertainty in prediction, the aim
is to maximize learning with a minimal amount of data.

We propose a novel approach for querying LLMs for NER labels using batching of
samples (cf. Section 3.2). This results in the same prompt being used for
annotation of multiple samples, effectively expediting the process and
reducing the querying cost as fewer tokens are expended. We adopt an iterative
active learning approach (cf. Figure 1) for fine-tuning a NER model on the
MasakhaNER 2.0 dataset [1].

Figure 1: Overview of our methodology. The process involves selecting the most
informative samples from the training set, and querying the LLM with a pre-
defined prompt template to obtain annotations. The problem-specific classifier
is then trained with these queried annotations and evaluated on the unseen
test set.

MasakhaNER 2.0 [1] is the largest human-annotated NER dataset for African
languages that contains text data from 20 languages widely spoken in Sub-
Saharan Africa. Despite these languages having a considerable number of
speakers they are underrepresented in NLP research. The dataset contains a
comprehensive collection of entities, including people, places, organizations,
and dates. The percentage of annotated entities relative to all tokens ranges
from 6.5 % for Bambara to 16.2 % for Shona language.

Our methodology consists of the following core steps:

  1. 1.

Initial Model Training: We fine-tune the AfroXLMR-mini model [2] on a small
subset (5 %) of the MasakhaNER 2.0 dataset, establishing a baseline
performance. AfroXLMR-mini is a compact multilingual language model that is
based on the BERT [8] architecture, fine-tuned specifically for African
languages.

  2. 2.

Active Learning Loop: We iteratively execute the following:

_Uncertainty Quantification:_ Using predictions from the model, we calculate
uncertainty at the token level using entropy:

$H(i)=-\sum_{c=1}^{C}p(c|i)\log p(c|i),$ (1)

where for each token i, $p(c|i)$ is the probability of class c, with C as the
total number of entity classes. We then compute the average sample entropy:

$H_{\text{avg}}(\text{sample})=\frac{1}{|\text{sample}|}\sum_{i\in\text{sample}}H(i)$
(2)

_Sample Selection:_ We rank samples by descending entropy and select the top 5
% of the most uncertain samples for annotation with the best performing LLM of
all compared. We add new annotations to the initial training subset and
retrain the AfroXLMR-mini model on the expanded dataset.

##  3 Experiments

###  3.1 Foundation Model Selection

Our experimental setup hinges on the initial choice of a suitable foundation
model. In low-resource settings, where data are scarce and of high value, an
accurate annotator is crucial for high-quality data labeling, particularly for
NER.

NER presents a significant challenge for LLMs because it demands not only
accurate labeling of each token in the text without omissions but also
requires the answers to be returned in a specific format, making it a more
complex task than, for example, classification, where the model is tasked with
providing a categorical value.

We evaluated several LLMs as potential candidates, viz. GPT-4-Turbo [version
"gpt-4-0125-preview"], Google Gemini 1.0 pro [version "gemini-1.0-pro-
vision-001"], Llama 2 70B, Mistral 7B, and Starling-LM-7b-alpha. While
Anthropic’s Claude 3 Sonnet and Claude 3 Opus were assessed in a smaller-scale
benchmark, their high operational costs precluded inclusion in our final
analysis. Candidate selection is performed using the following criteria: a)
adherence to the desired output format, b) correctness of the assigned labels,
and c) labeling consistency and inter-annotator agreement (IAA). We elaborate
on them in the subsequent subsections.

####  3.1.1 Representative Data Subset Sampling for LLM Evaluation

To assess the performance of LLM annotation on the MasakhaNER 2.0 dataset, we
conducted experiments on a stratified subset of its languages. We selected
three relatively common languages from MasakhaNER 2.0’s expanded set: isiZulu
(zul), Bambara (bam), and Setswana (tsn). For contrast, we also included two
least common languages: Ghomala’ (bbj) and Fon (fon) [1]. For each language,
we begin with 50 samples selected to reflect the dataset’s entity distribution
and diversity. We devise a sampling strategy (cf. Algorithm 0.1) that ensures
a representative benchmark for the evaluation of LLM performance, with
variations in the number and type of entities.

1 Procedure  _EntitySampling(_$D$ : dataset_)_

$D^{\prime}\leftarrow\emptyset$ ;

// Initialize empty set

2 foreach  _$s\in D$_ :

$entities\leftarrow\text{NER}(s)$ ;

// Count named entities

3 if  _$entities\neq\emptyset$_ :

4 $p\leftarrow\frac{|entities|}{|s|}$;

5 if  _$0.05\leq p\leq 0.50$_ :

6 $D^{\prime}\leftarrow D^{\prime}\cup\\{s\\}$;

7

8

9

$\text{SortDescending}(D^{\prime})$ ;

// Sort by entity count

$numRecords\leftarrow|D^{\prime}|$ ;

// Total number of records in the dataset

$\mu\leftarrow\frac{numRecords}{2}$ ;

// Mean of the normal distribution

$\sigma\leftarrow\frac{numRecords}{4}$ ;

// Standard deviation of the normal distribution

$w\leftarrow[]$ ;

// Initialize weight array

10 foreach  _$s_{i}\in D^{\prime}$_ :

11 $x_{i}\leftarrow\text{index of }s_{i}\text{ in }D^{\prime}$;

12
$w_{i}\leftarrow\exp\left(-\frac{(x_{i}-\mu)^{2}}{2\cdot\sigma^{2}}\right)$;

;

// Weight for $s_{i}$ using $\mathcal{N}(\mu,\sigma)$

$w\leftarrow w\cup\\{w_{i}\\}$ ;

// Append weight to the array

13

14 $w\leftarrow\text{Normalize}(w)$;

;

// Normalize weights

15 $D_{\text{balanced}}\leftarrow\text{WeightedSample}(D^{\prime},w)$;

16 return $D_{\text{balanced}}$;

17

Algorithm 0.1 Entity-based Balanced Sampling Approach

For our annotation setting, we filtered the samples according to the
proportions of the entities. We set a minimum threshold of 5 % to ensure a
sufficient presence of named entities for training. An upper threshold of 50 %
was chosen to focus the sampled subset on linguistically rich examples with
meaningful entity interactions. Through dataset analysis, we found that
samples with higher proportions of entities frequently consisted of simple
lists, lacking contextual information crucial for robust model learning,
leading to our choice of the upper threshold.

(a) Initial entity distribution in the Bambara dataset.

(b) Entity distribution in 50 records sampled from Bambara dataset.

Figure 2: Entity distribution in a) the overall Bambara dataset and b) in 50
sampled records using Algorithm 0.1. Non-entities are excluded from the chart.

Our balanced sampling approach closely preserves the entity distribution of
the original dataset, enhancing the representativeness and reliability of
subsequent results derived using this subset.

####  3.1.2 Querying LLMs

We employ a few-shot prompting approach [6] to extract entity labels from
large language models. Our flexible prompt template includes placeholders for
the language, annotation examples, and sample tokens. This allows us to tailor
the prompt to each language within the MasakhaNER 2.0 dataset for NER
annotation.

The prompt directs the LLM to label each token, providing examples in the
target language. It specifies the desired output format and set of labels. To
ensure alignment with the MasakhaNER 2.0 dataset, we explicitly instruct the
LLM to follow the MUC-6 annotation
guidelines222https://cs.nyu.edu/~grishman/muc6.html, the standard used by the
original annotators of the dataset. These guidelines offer a consistent
framework for recognizing and classifying named entities, such as persons,
organizations, and locations.

In Figure 3, we present a condensed form of the few-shot prompt for NER
annotation utilized in our research, and the full form of the prompt is shown
in Appendix 0.A.

Your task is to label entities in a given text written in {language}. Use the
following labels for annotation: […] Your task is to provide a list of named
entity labels, where each label corresponds to a token. […] in format [token,
label]. {examples} Note: […] • Follow MUC-6 (Message Understanding
Conference-6) Named Entity Recognition (NER) annotation guidelines. […] This
is your sentence: {sentence}. Figure 3: Condensed version of the full prompt
template with non-essential details abbreviated as […].

####  3.1.3 Correct Output Format

Accurate data labeling in the correct format is crucial for automated use
without manual intervention. To measure the proportion of correctly formatted
labels, we repeat the annotation of the sampled records 10 times using
multiple LLMs. We then compute labeling errors.

Two primary issues arise when querying LLMs for entity labels, namely, a)
complete absence of a response and b) omission errors, where the model skips
tokens that should be labeled. Omission errors also include the creation of
extra tokens, leading to a mismatch between the number of predicted labels and
tokens in the data sample.

As shown in Table 1, omission errors occur across all models and most
languages in our experimental setup. GPT-4-Turbo performs best, with no more
than 3 out of 50 samples per language exhibiting an omission error in at least
one annotation attempt. Llama 2 70B, Starling-LM, and Mistral 7B demonstrate
the poorest results, with significant mismatches between predicted labels and
token counts. This indicates inconsistent output and difficulty in reliably
following prompt instructions. The Gemini 1.0 Pro model was unsuitable for
this task due to its frequent inability to generate tokens by providing empty
response, therefore preventing the assessment of omission errors.

Table 1: Number of samples out of 50 with at least one omission error or missing prediction (output). Dashes (–) indicate that we could not calculate amount of omission errors because there was no response from the LLM. Model | Empty Predictions | Omission Error
---|---|---
| zul | bam | tsn | fon | bbj | zul | bam | tsn | fon | bbj
GPT-4-Turbo | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 1 | 0 | 2
Gemini 1.0 | 50 | 50 | 50 | 17 | 15 | – | – | – | 4 | 1
Llama 2 70B | 0 | 2 | 2 | 1 | 0 | 12 | 15 | 18 | 38 | 13
Starling-LM | 0 | 2 | 0 | 1 | 0 | 10 | 24 | 5 | 15 | 7
Mistral 7B | 0 | 6 | 0 | 6 | 0 | 20 | 25 | 20 | 28 | 6

Table 1 shows that while GPT-4-Turbo consistently provided annotations for all
records in all annotation attempts, Gemini 1.0 failed to return annotations
for all samples in isiZulu (zul), Bambara (bam) and Setwana (tsn) subsets. In
this case, Gemini 1.0 was returning an empty response.

Data samples with the aforementioned labeling issues are excluded from further
analysis.

####  3.1.4 Inter Annotator Agreement

As LLMs exhibit stochastic behavior when generating output [30], we measure
the Inter-Annotator Agreement to assess the labeling consistency of these
models. We proceed with the assumption that an effective model should label
data consistently – indicative of ‘no doubt’ in its annotations – thereby
achieving a high IAA score.

To evaluate IAA, the creators of the MasakhaNER 2.0 dataset utilized Fleiss’
Kappa [10] at the entity level [1]. Following this approach, we compare IAA
between model-annotated and human-annotated data. Instead of relying on
multiple human annotators, we instruct the model to reannotate the same
samples 10 times. Additionally, we adjust the temperature setting to 0.1 to
reduce randomness and limit creativity in the model’s outputs.

Table 2: Inter-annotator agreement (Fleiss’ kappa) for model reannotations. IAA is measured for a representative subset of 50 samples and 10 reannotation attempts. For comparison, we also include IAA of human annotators from [1] for the whole dataset. Model | zul | bam | tsn | fon | bbj
---|---|---|---|---|---
GPT-4-Turbo | 0.979 | 0.94 | 0.932 | 0.898 | 0.923
Gemini 1.0 | – | – | – | 0.928 | 0.966
Llama 2 70B | 0.836 | 0.88 | 0.867 | 0.595 | 0.749
Starling-LM | 0.867 | 0.849 | 0.914 | 0.88 | 0.89
Mistral 7B | 0.907 | 0.877 | 0.922 | 0.83 | 0.906
Human annotation [1] | 0.953 | 0.98 | 0.962 | 0.941 | 1.0

As illustrated in Table 2, GPT-4-Turbo and Gemini 1.0 demonstrate the highest
IAA scores for the sampled subset of records, which are comparable to human
IAA scores across the entire dataset. Specifically, in the isiZulu (zul)
subset, GPT-4-Turbo achieved a higher IAA score compared to that of human
annotators. These results suggest that in some cases, LLMs are capable of
reaching IAA scores comparable to, or even surpassing, those of human
annotators.

####  3.1.5 Consistency

To evaluate annotation quality and address model response variability, we
introduce a metric to compute the consistency of prediction accuracy, referred
to simply as consistency. This metric quantifies how frequently a model’s
predictions match the ground truth labels for each token over multiple
reannotation attempts. Consistency, in this context, indicates the model’s
ability to produce accurate predictions reliably across multiple inferences on
a single data sample. We posit that an effective model should demonstrate
certainty and correctness in its labeling, reflected in a high consistency
score.

We calculate the consistency score by comparing predicted labels against
ground truth labels for each token. This involves querying the same data
sample multiple times and determining the proportion of instances where each
token’s predicted label matches its ground truth counterpart. We then average
these per-sample scores to obtain the overall consistency for a data subset.

Cases with missing annotations or mismatches between predicted label count and
data sample tokens are treated as complete mismatches for all ground truth
labels. Thus, our consistency score also incorporates instances of omission
errors.

Table 3: Consistency of prediction accuracy: it measures the model’s ability to generate accurate annotations for a data sample across single or multiple inference attempts. It was evaluated using a representative subset of 50 records. Here, ’10 iterations’ refers to ten re-annotation attempts for a single data sample, while ’1 iteration’ indicates a single annotation attempt. Model | zul | bam | tsn | fon | bbj
---|---|---|---|---|---
10 iterations
GPT-4-Turbo | 93.5 | 90.0 | 89.7 | 86.9 | 86.7
Gemini 1.0 | 0.0 | 0.0 | 0.0 | 61.2 | 62.6
Llama 2 70B | 17.9 | 18.1 | 14.7 | 15.8 | 55.5
Starling-LM | 66.6 | 54.1 | 63.6 | 70.9 | 76.6
Mistral 7B | 54.3 | 40.5 | 42.7 | 44.8 | 73.8
1 iteration
GPT-4-Turbo | 93.48 | 91.35 | 89.7 | 86.89 | 85.91
Claude 3 Sonnet | 92.23 | 93.21 | 86.2 | 66.44 | 88.65
Claude 3 Opus | 97.86 | 94.64 | 92.87 | 95.1 | 89.2

As demonstrated in Table 3, the annotations produced by GPT-4-Turbo for 10
iterations exhibit significantly higher consistency compared to those obtained
from other LLMs. Meanwhile, Llama 2 70B provides most inconsistent results,
suggesting that this LLM may not be well suited for low-resource NER tasks.

Due to limited resources, we evaluate Anthropic’s Claude 3 Sonnet and Claude 3
Opus using 50 samples and just one annotation attempt. As we calculate the
number of times the label is predicted correctly, with only one annotation
attempt, consistency is equivalent to accuracy.

Table 3 demonstrates that Claude 3 Opus outperforms GPT-4-Turbo, achieving
greater accuracy. However, due to significantly higher costs for Claude 3, we
opt to continue with GPT-4-Turbo for our next experiments. Claude 3 Opus
incurs a cost that is 1.5 times greater than that of GPT-4-Turbo for input
tokens, and 2.5 times greater for output tokens.

####  3.1.6 LLM Evaluation Results

For subsequent experiments, GPT-4-Turbo was selected due to its consistently
superior performance in our initial evaluations. Across these evaluations, it
demonstrated the highest annotation accuracy, correct formatting, and the
lowest rate of omission errors. Additionally, for the most popular languages
within our test set, GPT-4-Turbo’s performance was comparable to IAA scores
achieved by human annotators.

While Claude 3 Opus outperformed GPT-4-Turbo on a smaller-scale benchmark,
this needs to be assessed with more iterations for reliable validation.

Gemini 1.0 achieved higher IAA scores than GPT-4-Turbo for two less
represented languages in the MasakhaNER 2.0 dataset - Ghomala’ (bbj) and Fon
(fon). However, Gemini 1.0’s frequent empty responses warrant further
investigation to understand potential limitations when handling specific
annotation queries.

Llama 2 70B, Mistral 7B, and Starling-LM provided the least consistent
results. The high number of omission errors with them suggests potential
difficulty in handling complex annotation tasks. Interestingly, Starling-LM
outperforms the other two models despite having fewer parameters.

###  3.2 Effect of Prompt Design and Querying LLMs in batches

Before executing the experiments described in our methodology, we assess the
potential impact that a carefully designed prompt template can have on the
accuracy of annotations. In this section, we evaluate the consistency of
annotations provided by Gemini 1.0 across a sample of 50 records, each
subjected to 10 reannotation attempts. This evaluation contrasts our standard
prompt template (cf. Appendix 0.A), against a more concise version of the
template that omits certain details. This comparison aims to discern how the
level of detail in prompt templates affects annotation quality.

For the less concise prompt, we removed the detailed explanation of the output
format but kept examples and the remaining parts intact. The following parts
were removed from the prompt:

  * •

"You will receive a list of tokens as the value for the ‘input’ key and text
language as the value for the ‘language’ key in a JSON dictionary."

  * •

"The input tokens are provided in a list format and represent the text."

  * •

"Your task begins now!"

As can be inferred from table 4, providing additional details regarding the
task, as well as the input and output formats, appears to enhance the accuracy
of data annotation.

Table 4: Comparison of consistency for Gemini 1.0 using our default and short template. Prompt type | bbj | swa
---|---|---
Default prompt | 69.99 | 93.87
Shortened prompt | 60.36 | 89.40

To minimize the usage of computational resources when annotating data using
GPT-4-Turbo, we adopt a novel approach by querying the model in batches.
Instead of processing just one sentence at a time, we combine two sentences in
the same prompt and instruct the model to provide NER annotations for both
samples simultaneously.

To evaluate this approach and compare the processing of individual records to
batches of samples, we compute consistency scores using the methodology
outlined in section 3.1. We evaluated two models: GPT-4-Turbo on isiZulu and
Gemini 1.0 on a Swahili subset containing 50 records that are reannotated 10
times. As illustrated in table 5, batch annotation does not significantly
reduce the consistency of results. Therefore, we opted to adopt this approach
for further use.

Table 5: Consistency for annotating one sample at a time compared to a batch of samples. Method | zul - GPT-4-Turbo | swa - Gemini 1.0
---|---|---
One sample | 93.53 | 93.86
Batch of samples | 93.1 | 91.78

###  3.3 Data Contamination

To ensure the validity of the MasakhaNER 2.0 dataset for our experiments, we
assessed the potential contamination of GPT-4-Turbo’s training data with this
dataset. We employ two methods:

a) _Few-Shot In-Context Learning (ICL)_ : Following Golchin et al. (2024)
[12], we use GPT-4 ICL to detect if the model can reproduce text from
MasakhaNER 2.0. This approach leverages the assumption that if an LLM’s
training data includes a specific dataset, the LLM can complete a partial text
sample from that dataset with an exact or near-exact match. We also include
CoNLL-2003 [25] and WikiANN [22] datasets for comparison.

b) _Contamination Score:_ To address the binary limitation of the ICL method,
we introduce a ‘contamination score’. We randomly select 30 records across
multiple languages from each dataset (details in the following paragraph).
GPT-4-Turbo is then tasked with identifying the source dataset of each record.
The corresponding prompt template is shown in fig. 4.

Identify the source (multilingual) NER dataset for this sample. Respond with
the dataset name alone. {sentence} Figure 4: Prompt template for instructing
the LLM to identify the source dataset of a given data sample. The placeholder
{sentence} is replaced with the actual data sample for this task. For
multilingual datasets, the term ‘multilingual’ is specified in the prompt,
while it is omitted for monolingual datasets.

The contamination score is the average proportion of correct identifications
across three experimental runs.

For multilingual datasets (WikiANN, MasakhaNER 2.0), we sample 10 records each
from three randomly selected languages (Bihari, Estonian, Slovak for WikiANN;
isiXhosa, Ewe, Chichewa for MasakhaNER 2.0). We set the temperature parameter
to 0 in GPT-4 and GPT-4-Turbo to minimize response randomness.

For the ICL method, we report (cf. Table 6) the number of exact or near-exact
matches found by GPT-4 (as per Golchin et al.) and verified manually.

Table 6: Data Contamination Evaluation. We report the data contamination score and its standard deviation. The second and third rows display the number of exact and near-exact matches, respectively, evaluated as per [12]. In case the human judgement about matches differs from that of GPT-4 judgement, the display format shows the GPT-4 judgment followed by the human judgment in parentheses. Method |  CoNLL- 2003 |  WikiANN Multilingual |  WikiANN English |  MasakhaNER 2.0
---|---|---|---|---
Contamination Score | $0.94\pm 0.02$ | $0.81\pm 0.13$ | $0.7\pm 0.1$ | $0.04\pm 0.02$
Exact matches [12] | 0 | 0 | 3 | 0
Near-exact matches [12] | 2(3) | 0(1) | 1 | 0

Our findings suggest that GPT-4-Turbo training data might be contaminated with
$4\leavevmode\nobreak\ \%\pm 2\leavevmode\nobreak\ \%$ of the MasakhaNER 2.0
dataset, indicating minimal influence on the overall results, in contrast to
the CoNLL-2003 and WikiANN datasets, which show significantly higher levels
(94 % and 81 % respectively) of potential contamination.

###  3.4 Active Learning

In this segment, we compare the performance of GPT-4-Turbo generated
annotations against manual annotations within an active learning setting. Our
baseline is a task-specific model trained on the entire manually annotated
ground truth data.

Due to significant class imbalance, our primary metric is entity-class
accuracy, excluding non-entities where the model excels due to their
overrepresentation. We do not report F1 computed at the entity level, where
partial detection counts as a mismatch, as our preprocessing further splits
entities into subtokens, complicating complete entity span detection.
Accuracy, on the other hand, directly assesses the model’s ability to identify
and classify entities, even when an entity extends across multiple subtokens.

We establish a baseline accuracy score by training the model on the entire
Bambara dataset (82 % accuracy) [1]. We then fine-tune the model using active
learning with ground truth annotations. In each iteration, we use entropy-
based uncertainty sampling (cf. Section 2) to select an additional 5 % of the
dataset. We repeat the active learning process but instead utilize annotations
generated by GPT-4-Turbo, fine-tuning the task-specific model in each
iteration.

Our active learning approach with ground truth annotations reaches the
baseline performance after utilizing only 20 % of the data and surpasses it
after 30 %. In contrast, using GPT-4-Turbo annotations, we achieve 76 %
accuracy but don’t fully reach the baseline within 5 iterations (cf. Figure
5). This difference may be due to the lower average accuracy of GPT-4-Turbo
annotations (84.5(9) % compared to manual ground truth). We observe similar
trends for isiZulu (cf. Figure 6). Active learning with manual ground truth
approaches the baseline, while using GPT-4-Turbo annotations with 30 % of the
data falls relatively short. It is to be noted that since the baseline here is
with human annotations, the ground truth will always be more numerically
accurate, and there will always be a disparity when comparing annotations from
other sources.

Figure 5: Accuracy (without non-entities) for Bambara test set achieved by
using ground truth (left) and GPT-4-Turbo annotations (right) in our active
learning framework. X-axis denotes the percentage of the dataset used for
active learning iteration, and red dashed line represents our baseline -
simple AfroXLMR-mini training using 100 % of the dataset without active
learning. Figure 6: Accuracy (without non-entities) for isiZulu test set
achieved by using ground truth (left) and GPT-4-Turbo annotations (right) in
our active learning framework. X-axis denotes the percentage of the dataset
used for active learning iteration, and red dashed line represents our
baseline - simple AfroXLMR-mini training using 100 % of the dataset without
active learning.

To assess the cost comparison between human annotation and GPT-4-Turbo
annotation for the Bambara and isiZulu train datasets, we based our estimates
on the price per 200 annotated sentences as detailed in [1]. The estimated
cost for human annotation for Bambara train set is USD 233.5, and for isiZulu
train set is USD 292.5 per annotator. These costs are then multiplied by the
number of annotators as indicated in the study.

Annotating 30 % of the MasakhaNER 2.0 dataset with GPT-4-Turbo, in comparison,
is significantly cheaper than human annotation, costing approximately USD 16.5
per language. This makes our approach approximately 42.45 times less expensive
for Bambara and 53.18 times less expensive for isiZulu.

Our approach demonstrates the potential of GPT-4-Turbo for cost-effective
dataset annotation. Although there is an accuracy trade-off compared to ground
truth, the substantial cost reduction makes it a compelling option, especially
in resource-constrained settings. GPT-4-Turbo’s performance is promising, and
with continued training, we expect the accuracy and language coverage of
state-of-the-art LLMs to improve further, making automated data labelling an
increasingly viable alternative.

##  4 Conclusion

In this work, we analyzed foundation models in the context of low-resource NER
data annotation. Our findings highlight the intricate balance of trade-offs
and potential benefits associated with using LLMs for this task. Evaluating
various LLMs revealed performance variations, with GPT-4-Turbo demonstrating
consistent and accurate annotations, while others showed strengths in specific
contexts. This emphasizes the necessity of selecting appropriate models based
on task requirements.

Common challenges across LLMs were identified, including token skipping and
the need for refinement in generating correctly formatted outputs. Addressing
these challenges through further training or advanced prompt engineering could
enhance model reliability.

Although models like GPT-4-Turbo may exhibit slight accuracy reductions
compared to human annotations, it is crucial to note that the baseline for
comparison here is human annotations themselves. Consequently, human
annotations are inherently expected to be numerically superior in accuracy.
Additionally, human annotations are not infallible due to subjectivity and
potentially outdated annotation guidelines. This makes minor accuracy
reductions from LLM annotations less significant, especially considering the
substantial cost savings they offer.

Demonstrating GPT-4-Turbo’s cost-effectiveness for data annotation, our study
indicates potential cost savings of at least 42.45 times compared to human
annotation. This is particularly compelling in resource-constrained settings,
where minor accuracy trade-offs may be acceptable given the significant
benefits. Furthermore, GPT-4-Turbo’s promising performance and the ongoing
advancement of LLMs suggest even greater potential for their cost-effective
integration in the future.

Furthermore, in this work, we introduced a novel methodology to assess data
leakage and quantify the potential for data contamination in LLM training
datasets. This is to assess whether the results from our experiments could be
extrapolated to similar unseen low-resource language domains.

In conclusion, our investigation into the use of foundation models for data
annotation in low-resource African languages reveals the transformative
potential of LLMs in NLP. Despite the challenges, our study demonstrates that
LLMs, particularly through active learning approaches, offer a cost-effective
and efficient solution for NER tasks. As a future work, exploring the use of
Claude 3 Opus may yield noticeable improvements, as it outperformed
GPT-4-Turbo in our smaller-scale benchmark.

## References

  * [1] David Ifeoluwa Adelani et al.  “MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition”  In _Conference on Empirical Methods in Natural Language Processing_ , 2022  URL: https://api.semanticscholar.org/CorpusID:253098583
  * [2] Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach and Dietrich Klakow  “Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning”  In _Proceedings of the 29th International Conference on Computational Linguistics_ Gyeongju, Republic of Korea: International Committee on Computational Linguistics, 2022, pp. 4336–4349  URL: https://aclanthology.org/2022.coling-1.382
  * [3] Anthropic  “Introducing the next generation of Claude” Accessed: March 13, 2024, 2024  URL: https://www.anthropic.com/news/claude-3-family
  * [4] Yejin Bang et al.  “A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity”, 2023  arXiv:2302.04023 [cs.CL]
  * [5] Rishi Bommasani et al.  “On the Opportunities and Risks of Foundation Models”, 2022  arXiv:2108.07258 [cs.LG]
  * [6] Tom Brown et al.  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [7] Cheng-Han Chiang and Hung-yi Lee  “Can large language models be an alternative to human evaluations?”  In _arXiv preprint arXiv:2305.01937_ , 2023
  * [8] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, 2019  arXiv:1810.04805 [cs.CL]
  * [9] P. Ferragina and Ugo Scaiella  “Fast and Accurate Annotation of Short Texts with Wikipedia Pages”  In _IEEE Software_ 29, 2010, pp. 70–75  DOI: 10.1109/MS.2011.122
  * [10] Joseph Fleiss  “Measuring Nominal Scale Agreement Among Many Raters”  In _Psychological Bulletin_ 76, 1971, pp. 378–  DOI: 10.1037/h0031619
  * [11] Fabrizio Gilardi, Meysam Alizadeh and Maël Kubli  “ChatGPT outperforms crowd workers for text-annotation tasks”  In _Proceedings of the National Academy of Sciences_ 120.30 National Acad Sciences, 2023, pp. e2305016120
  * [12] Shahriar Golchin and Mihai Surdeanu  “Time Travel in LLMs: Tracing Data Contamination in Large Language Models”, 2024  arXiv:2308.08493 [cs.CL]
  * [13] Xingwei He et al.  “Annollm: Making large language models to be better crowdsourced annotators”  In _arXiv preprint arXiv:2303.16854_ , 2023
  * [14] Albert Q. Jiang et al.  “Mistral 7B”, 2023  arXiv:2310.06825 [cs.CL]
  * [15] Mahnoosh Kholghi, Laurianne Sitbon, G. Zuccon and Anthony N. Nguyen  “Active learning reduces annotation time for clinical concept extraction”  In _International journal of medical informatics_ 106, 2017, pp. 25–31  DOI: 10.1016/j.ijmedinf.2017.08.001
  * [16] Takeshi Kojima et al.  “Large language models are zero-shot reasoners”  In _Advances in neural information processing systems_ 35, 2022, pp. 22199–22213
  * [17] Hongsen Liao, Li Chen, Yibo Song and Hao Ming  “Visualization-Based Active Learning for Video Annotation”  In _IEEE Transactions on Multimedia_ 18, 2016, pp. 2196–2205  DOI: 10.1109/TMM.2016.2614227
  * [18] Humza Naveed et al.  “A Comprehensive Overview of Large Language Models”  In _ArXiv_ abs/2307.06435, 2023  DOI: 10.48550/arXiv.2307.06435
  * [19] Aurélie Névéol, R. Dogan and Zhiyong Lu  “Semi-automatic semantic annotation of PubMed queries: A study on quality, efficiency, satisfaction”  In _Journal of biomedical informatics_ 44 2, 2011, pp. 310–8  DOI: 10.1016/j.jbi.2010.11.001
  * [20] OpenAI et al.  “GPT-4 Technical Report”, 2023  arXiv:2303.08774 [cs.CL]
  * [21] Nicholas Pangakis, Samuel Wolken and Neil Fasching  “Automated annotation with generative ai requires validation”  In _arXiv preprint arXiv:2306.00176_ , 2023
  * [22] Afshin Rahimi, Yuan Li and Trevor Cohn  “Massively Multilingual Transfer for NER”  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ Florence, Italy: Association for Computational Linguistics, 2019, pp. 151–164  URL: https://www.aclweb.org/anthology/P19-1015
  * [23] Mirac Suzgun et al.  “Challenging big-bench tasks and whether chain-of-thought can solve them”  In _arXiv preprint arXiv:2210.09261_ , 2022
  * [24] Gemini Team et al.  “Gemini: A Family of Highly Capable Multimodal Models”, 2023  arXiv:2312.11805 [cs.CL]
  * [25] Erik F. Tjong Kim Sang and Fien De Meulder  “Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition”  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , 2003, pp. 142–147  URL: https://aclanthology.org/W03-0419
  * [26] Hugo Touvron et al.  “Llama 2: Open Foundation and Fine-Tuned Chat Models”, 2023  arXiv:2307.09288 [cs.CL]
  * [27] Shuhe Wang et al.  “Gpt-ner: Named entity recognition via large language models”  In _arXiv preprint arXiv:2304.10428_ , 2023
  * [28] Xiang Wei et al.  “Zero-shot information extraction via chatting with chatgpt”  In _arXiv preprint arXiv:2302.10205_ , 2023
  * [29] Shunyu Yao et al.  “React: Synergizing reasoning and acting in language models”  In _arXiv preprint arXiv:2210.03629_ , 2022
  * [30] Xingdi Yuan et al.  “Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation”, 2022  arXiv:2209.11000 [cs.CL]
  * [31] Ruoyu Zhang et al.  “Llmaaa: Making large language models as active annotators”  In _arXiv preprint arXiv:2310.19596_ , 2023
  * [32] Ruiqi Zhong, Kristy Lee, Zheng Zhang and Dan Klein  “Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections”  In _arXiv preprint arXiv:2104.04670_ , 2021
  * [33] Banghua Zhu et al.  “Starling-7B: Improving LLM Helpfulness & Harmlessness with RLAIF”, 2023
  * [34] Caleb Ziems et al.  “Can large language models transform computational social science?”  In _Computational Linguistics_ MIT Press One Broadway, 12th Floor, Cambridge, Massachusetts 02142, USA …, 2024, pp. 1–55

##  Appendix 0.A Prompt Template for Querying Foundation Model

To query LLMs we utilize a flexible prompt format, tailored for each language
in MasakhaNER2.0 [1] dataset.

Figure 7 illustrates the prompt template used to query the foundation models
for NER labels. The prompt template includes descriptions of the annotation
labels, details of the required output format, and corresponding examples.

Your task is to label entities in a given text written in {language}. Use the
following labels for annotation: • "O": Represents words that are not part of
any named entity. • "B-PER": Indicates the beginning of a person’s name. •
"I-PER": Used for tokens inside a person’s name. • "B-ORG": Marks the
beginning of an organization’s name. • "I-ORG": Tokens inside an
organization’s name. • "B-LOC": Marks the beginning of a location (place)
name. • "I-LOC": Tokens inside a location name. • "B-DATE": Marks the
beginning of a date entity. • "I-DATE": Tokens inside a date entity. You will
receive a list of tokens as the value for the ’input’ key and text language as
the value for the ’language’ key in a JSON dictionary. Your task is to provide
a list of named entity labels, where each label corresponds to a token. Output
the tokens with their corresponding named entity labels in a JSON format,
using the key ’output’. ’output’ should contain a list of tokens and their
entity labels in format [token, label]. {examples} Note: • The input tokens
are provided in a list format and represent the text. • Important: the output
should be a list with the same length as the input list, where each element
corresponds to the named entity label for the corresponding token. Do not
change the order of tokens and do not skip them. • The named entity labels are
case-sensitive, so please provide them exactly as specified ("B-PER", "I-LOC",
etc.). • Follow MUC-6 (Message Understanding Conference-6) Named Entity
Recognition (NER) annotation guidelines. Your task begins now!
\- Output JSON only. Enclose all tokens and tags in double brackets.
This is your sentence: {sentence}. Figure 7: Prompt template for querying
foundation model.

The prompt includes placeholders ({language}, {examples}, {sentence}) which
are substituted with the specific language name, examples, and the data sample
for annotation, respectively.

##  Appendix 0.B Annotation Examples for Querying Foundation Model

For each language featured in MasakhaNER2.0, we generate specific annotation
examples. Two records are randomly chosen from the MasakhaNER2.0 [1] test set
for each language: the first record contains no named entities, while the
second includes at least one named entity. These examples are then structured
as a single string for incorporation in place of the {examples} placeholder.
Figure 8 shows an example of annotation for the Bambara language.

Figure 8: Annotation examples for querying foundation model.


# Finding Memo:
Extractive Memorization in Constrained Sequence Generation Tasks

Vikas Raunak Arul Menezes
Microsoft Azure AI
Redmond, Washington
<EMAIL_ADDRESS>
###### Abstract

Memorization presents a challenge for several constrained Natural Language
Generation (NLG) tasks such as Neural Machine Translation (NMT), wherein the
proclivity of neural models to memorize noisy and atypical samples reacts
adversely with the noisy (web crawled) datasets. However, previous studies of
memorization in constrained NLG tasks have only focused on counterfactual
memorization, linking it to the problem of hallucinations. In this work, we
propose a new, inexpensive algorithm for extractive memorization (exact
training data generation under insufficient context) in constrained sequence
generation tasks and use it to study extractive memorization and its effects
in NMT. We demonstrate that extractive memorization poses a serious threat to
NMT reliability by qualitatively and quantitatively characterizing the
memorized samples as well as the model behavior in their vicinity. Based on
empirical observations, we develop a simple algorithm which elicits non-
memorized translations of memorized samples from the same model, for a large
fraction of such samples. Finally, we show that the proposed algorithm could
also be leveraged to mitigate memorization in the model through finetuning. We
have released the code to reproduce our results at
https://github.com/vyraun/Finding-Memo.

Finding Memo:
Extractive Memorization in Constrained Sequence Generation Tasks

Vikas Raunak Arul Menezes Microsoft Azure AI Redmond, Washington
<EMAIL_ADDRESS>

##  1 Introduction

Previous studies Arpit et al. (2017); Feldman (2020); Zhang et al. (2021a)
have shown that neural networks capture regular patterns in the training data
(generalization) while simultaneously fitting noisy and atypical samples using
brute-force (memorization). For constrained Natural Language Generation tasks
such as Neural Machine Translation (NMT), which rely heavily on noisy (web
crawled) data for training high-capacity neural networks, this creates an
inherent reliability problem. For example, memorizations could manifest
themselves in the form of catastrophic translation errors on specific samples
despite high average model performance Raunak et al. (2021). It is also likely
that the memorization of a specific sample could corrupt the translations of
samples in its vicinity. Therefore, exploring, quantifying and alleviating the
impact of memorization is of critical importance for improving the reliability
of such systems.

Yet, most of the work on memorization in natural language processing (NLP) has
focused either on classification Zheng and Jiang (2022) or on unconstrained
generation tasks, predominantly language modeling Carlini et al. (2021); Zhang
et al. (2021b); Kharitonov et al. (2021); Chowdhery et al. (2022); Tirumala et
al. (2022); Tänzer et al. (2022); Haviv et al. (2022). In this work, we fill a
gap in the literature by developing an analogue of extractive memorization for
constrained sequence generation tasks in general and NMT in particular. Our
main contributions are:

  1. 1.

We propose a new, inexpensive algorithm for studying extractive memorization
in constrained sequence generation tasks and use it to characterize
memorization in NMT.

  2. 2.

We demonstrate that extractive memorization poses a serious threat to NMT
reliability by quantitatively and qualitatively analyzing the memorized
samples and the neighborhood effects of such memorization. We also demonstrate
that the memorized instances could be used to generate errors in disparate
systems.

  3. 3.

Based on an analysis of the neighborhood effects of memorization, we develop a
simple memorization mitigation algorithm which produces non-memorized (higher
quality) outputs for a large fraction of memorized samples.

  4. 4.

We show that the outputs produced by the memorization mitigation algorithm
could also be used to directly impart corrective behavior into the model
through finetuning.

Repetitions | Total Samples | Memorized | Ratio (%) | Perturb Prefix | Perturb Suffix | Perturb Start
---|---|---|---|---|---|---
1 | 100,000 | 174 | 0.17 | 17.58 % | 43.24 % | 12.29 %
2 | 100,000 | 317 | 0.32 | 11.67 % | 62.84 % | 4.98 %
3 | 5,381 | 17 | 0.32 | 28.42 % | 49.52 % | 18.82 %
4 | 1,885 | 5 | 0.26 | 27.40 % | 34.00 % | 8.00 %
5 | 976 | 7 | 0.72 | 26.67 % | 70.00 % | 11.42 %
1-5 | 208,242 | 520 | 0.25 | 16.65 % | 51.65 % | 8.00 %

Table 1: Quantifying Extractive Memorization: Number of Memorized Samples
(using Algorithm 1) and Neighborhood Effects of Memorization (using Algorithm
2) across different training data frequency buckets.

##  2 Related Work

Our work is concerned with the phenomenon of memorization in constrained
natural language generation in general and NMT in particular. The main
challenge in analyzing memorization is to determine which samples have been
memorized by the model during training. There exist two key algorithms to
elicit memorized samples, each yielding a distinctive operational definition
of memorization:

  1. 1.

Counterfactual Memorization: Feldman and Zhang (2020) study label memorization
and propose to estimate the memorization value of a training sample by
training multiple models on different random subsets of the training data and
then measuring the deviation in the sample’s classification accuracy under
inclusion/exclusion. This definition of memorization was further extended to
arbitrary performance measures by Raunak et al. (2021) to study memorization
in NMT and by Zhang et al. (2021b) to study memorization in language models.
However, a practical limitation of analysis based on this definition is the
prohibitive computational cost (multiple model trainings) associated with
computing memorization values for each training sample.

  2. 2.

Extractive Memorization: Carlini et al. (2021) propose a data-extraction based
definition of memorization to study memorization in language models. Therein,
a training string $s$ is extractable if there exists a prefix $c$ that could
exactly generate $s$ under an appropriate sampling strategy (e.g. greedy
decoding). This definition has the benefit of being computationally
inexpensive, although it doesn’t have any existing analogue for constrained
natural language generation tasks such as NMT.

In the next section, we define extractive memorization for constrained
sequence generation tasks and apply it to NMT, in section 4 we estimate the
neighborhood effect of such memorizations and in section 5 we propose a simple
algorithm for recovering correct translations of memorized samples.

##  3 Extractive Memorization

We present our definition of extractive memorization as Algorithm 1. Analogous
to extractive memorization in language models Carlini et al. (2021), this
definition labels an input sentence (source) as being memorized if its
transduction (translation) could be replicated exactly with a prefix
considerably shorter than the length of the full input sentence (source),
under greedy decoding. Operationally, we set prefix ratio threshold ($p$) to
$0.75$.

Data: Trained NMT Model $T$, Training Dataset $S$, Prefix Ratio Threshold $p$

Result: Memorized Samples $M$, Prefix Lengths $L$

Greedily Translate Sources in $S$ using $T$;

$M_{1}$ = Sources with translations matching References;

Greedily Translate Prefixes of Sources in $M_{1}$ using $T$;

$M_{2}$ = Sources with Prefixes producing References;

for  _$M_{2}^{i}$ in $M_{2}$_ do

$n$ = Length of the Source $M_{2}^{i}$ ;

$l$ = Length of Smallest Prefix producing the Ref;

if  _ $\frac{l}{n}\leq p$ _ then

Add $M_{2}^{i}$ to $M$ and Add $l$ to $L$;

Algorithm 1 Extractive Memorization in NMT

Next, we apply this definition of memorization on a strong Transformer-Big
Vaswani et al. (2017) baseline trained on the 48.2M WMT20 En-De parallel
corpus Barrault et al. (2020). We describe the dataset, model and training
details in Appendix A.

Provenance | Source |  Translation
---|---|---
Training Data | Why study in Peru? Spanish Courses |  Warum in Peru studieren?
Perturb Suffix | Why study in Peru? University Courses |  Warum in Peru studieren?
Perturb Suffix | Why study in Peru? Short Courses |  Warum in Peru studieren?
Perturb Suffix | Why study in Peru? Summer Courses |  Warum in Peru studieren?
Perturb Prefix | You study in Peru? Spanish Courses |  Sie studieren in Peru? Spanischkurse
Perturb Prefix | Advanced study in Peru? Spanish Courses |  Weiterbildung in Peru? Spanischkurse
Table 2: Memorization Example: An example to illustrate the phenomenon of
Memorization (elicited using Algorithm 1) and the ensuing Neighborhood Effect
of such memorization (measured using Algorithm 2). This WMT20 English to
German training sample is memorized with a prefix length ratio of 0.67 ($\leq$
0.75).

Qualitatively, we observe that the memorized samples detected by Algorithm 1
mostly consist of low-quality samples – templatized source sentences and noisy
translations. To analyze the results quantitatively, similar to Carlini et al.
(2022), we bucket the training data pairs in terms of their repetitions in the
training data. Owing to the sparsity of data with greater than 5 repetitions
we report results in the range of 1-5 repetitions. Further, for repetition
values 1 and 2, we select 100K random samples for analysis. We observe two key
results:

Repetitions vs Memorization: Table 1 shows that the percentage of samples
memorized is higher for repeated training samples, compared to samples present
only once in the training data, with a Pearson’s correlation coefficient of
0.778 between the number of repetitions and memorizations.

Quality of Memorized Samples: Table 3 shows that both the quality of memorized
samples, measured using COMET-QE Rei et al. (2020), a state-of-the-art Quality
Estimation model for MT as well as their lexical diversity, measured using
Type-to-Token Ratio (TTR) Vanmassenhove et al. (2021); Gehrmann et al. (2022)
is worse when compared to the total samples.

Rep. | T-COM | M-COM | T-TTR | M-TTR
---|---|---|---|---
1 | 42.91 | 32.81 | 9.81 | 44.87
2 | 60.63 | 62.56 | 8.85 | 36.92
3 | 73.19 | 37.31 | 20.03 | 68.97
4 | 70.18 | 61.92 | 27.72 | 78.85
5 | 68.41 | 46.65 | 33.90 | 73.47
1-5 | 54.79 | 51.57 | 7.64 | 38.72

Table 3: Comparison of COMET-QE $\uparrow$ and TTR $\downarrow$ for Memorized
(M) Samples vs Total (T) Samples.

The above two results are similar to the results in language modeling Carlini
et al. (2022) and serve to demonstrate the utility of Algorithm 1 in analyzing
extractive memorization in NMT.

##  4 Neighborhood Effect of Memorization

To measure the neighborhood effect of memorization, we generate new source
sentences in the vicinity of the memorized samples through perturbations and
test whether they generate the same output under greedy decoding. Table 2
shows a memorized training sample from Section 3, alongside translations
generated from perturbations at different positions in the source. The
perturbations (substitutions) were generated using BERT-Cased Devlin et al.
(2019). We define suffix positions and prefix positions based on the recorded
prefix lengths ($L$) in Algorithm 1. Specifically, we use Algorithm 2, which
generates new sources in the neighborhood of a memorized input source by
perturbing its tokens at prefix ($P$), suffix ($S$) or the beginning ($B$)
positions and then computes how many such new sources still translate to the
same memorized output. The resulting effect measure $N$ is higher if the
sources still produce the same memorized output under perturbations at
different positions. The intuition behind Algorithm 2 is that we want to
explore meaningful source sequences around the memorized source sentence.
Therefore, we make changes to only a single position at a time, to keep the
generated sequence close to the original sequence and substitute that position
with a word that fits well in the particular context. Our ‘neighbour’
definition is loosely inspired by the differential privacy literature Dwork et
al. (2014), where a neighbour is used to mean datasets differing in only one
item.

Data: Memorized Samples $M$, Prefix Lengths $L$, Masked Language Model $W$,
Candidates $K$

Result: Perturbed Sources $S^{{}^{\prime}}$, Effect Measure $N$

for  _$M_{i}$ , $L_{i}$ in $M$, $L$ _ do

$n$ = Length of the Source $M_{i}$;

$P_{i}$ = [1,…,$L_{i}-1$], $S_{i}$ = [$L_{i}+1$,…, $n$], $B_{i}$ = [1];

for  _$j$ in $P_{i}$, $S_{i}$, $B_{i}$ _ do

Generate $K$ substitutes at $M_{i}[j]$ using $W$;

Translate each new Source and Add to $S^{{}^{\prime}}$;

$N$ = Fraction of $S^{{}^{\prime}}$ Generating Memorized Outputs;

Algorithm 2 Estimating Neighborhood Effect

Results: The last 3 columns of Table 1 present the results of applying
Algorithm 2 on memorized sources ($K$=5). We find that considerable
percentages of perturbations still yield the same translations, and that this
effect is highly position-dependent. For example, for the unique memorized
samples (repetitions = 1), perturbing the suffix produces no changes in
translations for 43.2% of the new sources; while for prefix perturbations the
same memorized output is produced only 17.6%. Further, perturbing the first
token of the source is highly successful in generating a different (non-
memorized) output. The results demonstrate that a single memorization may
therefore be able to corrupt the translations of multiple input sequences in
its vicinity.

Table 1 and 2 show that while the translations of memorized sources frequently
remain invariant under suffix perturbations, translations under prefix-
perturbations do change more frequently. We hypothesize that this results from
the model switching away from its memorization mode, owing to a change in the
input prefix used for memorization.

##  5 Generating Non-Memorized Outputs

The results in section 4 (and table 2) show that the model is able to generate
non-memorized translations quite frequently if memorized source’s start token
is perturbed. We demonstrate that this fact could be exploited to recover the
non-memorized translations of a memorized sample from the same model with a
surprisingly high-frequency. We call this task Memorization-Mitigation and
present a simple technique to do so in Algorithm 3.

Data: Trained NMT Model $T$, Memorized Samples $M$, Recovery Symbol $X$

Result: Memorized Samples with New Outputs $F$

for  _$S_{i}$ in $M$_ do

$R_{i}$ = Translate $S_{i}$ using $T$;

$S_{i}^{aug}$ = $X$ \+ $S_{i}$;

$R_{i}^{aug}$ = Translate $S_{i}^{aug}$ with $T$;

if  _ $R_{i}^{aug}\neq R_{i}$ and $R_{i}^{aug}$ starts with $X$ _ then

Strip $X$ from $R_{i}^{aug}$, Add ($S_{i}$, $R_{i}^{aug}$ ) to $F$;

Algorithm 3 Generating Non-Memorized Outputs

Source: Victor Emmanuel II of Italy Translation (Reference): Viktor Emanuel
II. Perturbed Source: ! Victor Emmanuel II of Italy Translation: ! Viktor
Emanuel II von Italien Stripped Output: Viktor Emanuel II von Italien

Figure 1:  Generating Non-Memorized Output The example shows how the inclusion
of an ‘isolated’ symbol early in the input prefix elicits a non-memorized
translation from the model using Algorithm 3.

Algorithm 3 uses the idea of prefix perturbation to elicit a non-memorized
translation from the model. As a perturbation, it appends a symbol $X$ to the
source sentence, which is chosen such that it could be translated in
isolation, without semantically altering the source. We find that new
translations for 65.2% of the memorized samples in Table 1 (339/520) could be
recovered using Algorithm 3 with symbol $X$ set to ‘!’. We find that the
results vary with different choices of symbol $X$ and report the best result
after trying 5 different symbols. An example of applying Algorithm 3 on a
memorized sample is presented in Figure 1.

Further, we find that the new (non-memorized) translations are of much higher
quality than the memorized outputs. Table 5 presents the results by comparing
the new translations to the memorized translations using COMET-QE and TTR.

Set | COMET-QE $\uparrow$ | TTR $\downarrow$
---|---|---
Memorized | 54.57 | 57.16
Non-Memorized | 84.00 | 40.47

Table 4: Quality of Non-Memorized Outputs: Comparing the Quality of Algorithm
3 outputs against Memorized References using COMET-QE and Char-Ratio.

##  6 Mitigating Memorization in the Model

The previous section demonstrated that the non-memorized translations of
memorized samples could be recovered from the same model by applying Algorithm
3. In this section, we investigate whether this corrective behavior could
directly be imparted to the model through finetuning using the corpus $F$
obtained by applying Algorithm 3.

Measurement | Base | Finetuned
---|---|---
WMT20 Test BLEU | 32.9 | 33.5
Memorized COMET-QE | 54.57 | 68.94

Table 5: Post-Finetuning Quality: Comparing Model Quality on WMT20 Test and
the Memorized Set.

To test this, we finetune the last checkpoint of the trained WMT20 model for
one epoch on approximately 10K data pairs, comprising of 10K parallel data
samples drawn randomly from the training corpus as well as the corpus F. We
compare the model prior to and post finetuning, in terms of both general
performance on the WMT20 test set with BLEU Papineni et al. (2002); Post
(2018) and on the corpus $F$ with COMET-QE. Table 5 shows that given the non-
memorized translations, such corrective behavior could be imparted to the
model without impacting general quality.

Further Experiments: Throughout sections 3-6, we have used the trained WMT20
En-De system to conduct various experiments exploring extractive memorization.
However, the reported phenomena are quite general in nature, observable across
different language pairs, systems and datasets; the results for other
experiments are presented in Appendix D. Further, we note that to apply
Algorithms 1 and 2 across language pairs, certain changes are required:

  1. 1.

For character-based languages such as Chinese, Japanese, Korean and Thai
(CJKT), the Prefix Lengths ($L$) in Algorithm 1 should be measured in
characters, unlike whitespace-based tokens in the general case.

  2. 2.

For non-English source language, multilingual BERT (or other comparable
multilingual MLMs) should be used for generating the Neighbours
($S^{{}^{\prime}}$) in Algorithm 2.

In the next two sections, we delve into the implications of extractive
memorization and its apparent data-dependency, towards NMT reliability.

##  7 Transferable Memorization Attacks

In this section, we show how the proposed extractive memorization algorithm
could be used to construct potential attacks on state-of-the-art (SOTA) MT
systems. Our motivation here is to provide an existence proof of the statement
that the data dependency of extractive memorization makes it a transferable
phenomenon, which could be leveraged to attack disparate systems. As such, we
name the attack designed to elicit erroneous generations on System B, based on
memorized data extracted from System A, a Transferable Memorization Attack
(TMA). Further, successful Transferable Memorization Attacks should signify
spurious correlations which could be easily learned from the underlying common
data distribution. Therefore, TMA could be characterized as a data poisoning
attack Biggio et al. (2012) and its existence would point to shared
vulnerabilities in NMT systems.

Experiment: We feed the memorized samples obtained from the system in section
3 to two public systems, namely Google Translator and Microsoft Bing
Translator. We find that it is indeed possible to elicit erroneous, memorized
translations from these SOTA systems using the memorized inputs identified
from our WMT20 model. We present two such examples in Table 8 in Appendix B.

##  8 Discussion and Open Questions

In this section, we list some questions which require further investigation to
gain more insights into extractive memorization and its effects.

### Representations of Memorized Samples:

Our hypothesis to explain the differential sensitivity of memorized samples to
perturbation positions posits prefixes as memorization triggers. However,
further investigations from a representational perspective are required to
validate this hypothesis as well as to study how such non-robust memorized
representations are composed within longer sentences Raunak et al. (2019);
Dankers et al. (2022a, b).

### Reference-Free Extractions:

Algorithm 1 could also be applied in a reference-free manner by treating the
output of the full input sentence as the reference for the prefixes and
testing a large number of inputs Raunak et al. (2022). Further research is
required into determining the efficacy of such reference-free extraction and
its effectiveness in generating transferable memorization attacks.

### Counterfactual vs Extractive Memorization:

In the case of language models, Zhang et al. (2021b) show that the samples
elicited by counterfactual and extractive memorization algorithms exhibit
different characteristics, with rarity vs templaticity being a prominent
difference mode. However, further quantitative analysis is required to examine
their differences & similarities in the context of constrained sequence
generation tasks.

### Effects in Multilingual Systems:

Extractive memorization manifests in the form of spurious correlation based
overgenerations learned by the model and may be more prominent in multilingual
models owing to extra triggers Gu et al. (2019).

##  9 Conclusions

In this work, we developed the idea of extractive memorization for constrained
sequence generation tasks and quantitatively demonstrated that such
memorization poses a real threat to NMT reliability. We also proposed an
algorithm to generate non-memorized outputs for such samples. To the best of
our knowledge, our work is the first investigation of extractive memorization
for constrained sequence generation tasks in general. We hope that our work
serves as a useful step towards further research on extractive memorization in
constrained sequence generation tasks & NMT.

##  10 Acknowledgements

We thank Matt Post and Marcin Junczys-Dowmunt for helpful early discussions.
We thank Huda Khayrallah, Matt Post, and Hai Pham for providing detailed
feedback on the original manuscript.

##  11 Limitations

Throughout the work, we emphasized on exploring and mitigating memorization
without the help of data filtering techniques and did not compare
memorizations for models trained under different data-filtering algorithms. We
believe this direction to be very relevant but orthogonal to our work in this
paper. Further, since our work presents the first algorithm on memorization
mitigation, we believe it doesn’t represent an optimal approach for the task.
For example, one immediate extension of the algorithm would be to use multiple
symbols instead of just one symbol.

## References

  * Arpit et al. (2017) Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. 2017.  A closer look at memorization in deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 233–242. PMLR.
  * Barrault et al. (2020) Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors. 2020.  _Proceedings of the Fifth Conference on Machine Translation_. Association for Computational Linguistics, Online.
  * Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012.  Poisoning attacks against support vector machines.  In _Proceedings of the 29th International Coference on International Conference on Machine Learning_ , ICML’12, page 1467–1474, Madison, WI, USA. Omnipress.
  * Carlini et al. (2022) Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022.  Quantifying memorization across neural language models.
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021.  Extracting training data from large language models.  In _30th USENIX Security Symposium (USENIX Security 21)_ , pages 2633–2650.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.
  * Dankers et al. (2022a) Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022a.  The paradox of the compositionality of natural language: A neural machine translation case study.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4154–4175, Dublin, Ireland. Association for Computational Linguistics.
  * Dankers et al. (2022b) Verna Dankers, Christopher Lucas, and Ivan Titov. 2022b.  Can transformer be too compositional? analysing idiom processing in neural machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3608–3626, Dublin, Ireland. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dwork et al. (2014) Cynthia Dwork, Aaron Roth, et al. 2014.  The algorithmic foundations of differential privacy.  _Foundations and Trends® in Theoretical Computer Science_ , 9(3–4):211–407.
  * Feldman (2020) Vitaly Feldman. 2020.  Does learning require memorization? a short tale about a long tail.  In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_ , pages 954–959.
  * Feldman and Zhang (2020) Vitaly Feldman and Chiyuan Zhang. 2020.  What neural networks memorize and why: Discovering the long tail via influence estimation.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 2881–2891. Curran Associates, Inc.
  * Gehrmann et al. (2022) Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez-Beltrachini, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Nikolaev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang, Yixin Liu, and Yufang Hou. 2022.  Gemv2: Multilingual nlg benchmarking in a single line of code.
  * Gu et al. (2019) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O.K. Li. 2019.  Improved zero-shot neural machine translation via ignoring spurious correlations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1258–1268, Florence, Italy. Association for Computational Linguistics.
  * Haviv et al. (2022) Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2022\.  Understanding transformer memorization recall through idioms.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _Proceedings of ACL 2018, System Demonstrations_ , pages 116–121, Melbourne, Australia. Association for Computational Linguistics.
  * Kharitonov et al. (2021) Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. 2021.  How bpe affects memorization in transformers.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A Call for Clarity in Reporting BLEU Scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Qi et al. (2018) Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018\.  When and why are pre-trained word embeddings useful for neural machine translation? In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 529–535, New Orleans, Louisiana. Association for Computational Linguistics.
  * Raunak et al. (2019) Vikas Raunak, Vaibhav Kumar, and Florian Metze. 2019.  On compositionality in neural machine translation.
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021.  The curious case of hallucinations in neural machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1172–1183, Online. Association for Computational Linguistics.
  * Raunak et al. (2022) Vikas Raunak, Matt Post, and Arul Menezes. 2022.  Salted: A framework for salient long-tail translation error detection.  arXiv.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006\.  A study of translation edit rate with targeted human annotation.  In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_ , pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
  * Tänzer et al. (2022) Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022.  Memorisation versus generalisation in pre-trained language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7564–7578, Dublin, Ireland. Association for Computational Linguistics.
  * Tirumala et al. (2022) Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022\.  Memorization without overfitting: Analyzing the training dynamics of large language models.
  * Vanmassenhove et al. (2021) Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. 2021.  Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2203–2213, Online. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need.  In _Advances in Neural Information Processing Systems 30_ , pages 5998–6008.
  * Zhang et al. (2021a) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2021a.  Understanding deep learning (still) requires rethinking generalization.  volume 64, page 107–115, New York, NY, USA. Association for Computing Machinery.
  * Zhang et al. (2021b) Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. 2021b.  Counterfactual memorization in neural language models.
  * Zheng and Jiang (2022) Xiaosen Zheng and Jing Jiang. 2022.  An empirical study of memorization in NLP.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6265–6278, Dublin, Ireland. Association for Computational Linguistics.

##  Appendix A En-De Dataset and Model Details

We used the WMT20 Barrault et al. (2020) parallel training dataset with the
dataset statistics presented in Table 6. A joint vocabulary of 32K was learnt
using Sentencepiece Kudo and Richardson (2018) on a 10M random sample of the
training dataset.

Data Source | Sentence Pairs
---|---
Europarl | 1,828,521
ParaCrawl | 34,371,306
Common Crawl | 2,399,123
News Commentary | 361,445
Wiki Titles | 1,382,625
Tilde Rapid | 1,631,639
WikiMatrix | 6,227,188
Total | 48,201,847
Table 6: WMT20 Parallel Training Data

The trained model is a Transformer-Big with the hyperparameters described
exactly in Vaswani et al. (2017). The model was trained for 300K updates using
Marian Junczys-Dowmunt et al. (2018). The metrics BLEU, ChrF2, TER Papineni et
al. (2002); Popović (2015); Snover et al. (2006) for the trained model on the
WMT20 validation and test sets (under beam = $1$) as measured using SacreBLEU
Post (2018) are presented in Table 7, alongside reference-based COMET Rei et
al. (2020) scores.

Metric | BLEU | ChrF2 | TER | COMET
---|---|---|---|---
Validation | 37.5 | 63.9 | 51.5 | 56.50
Test | 32.9 | 61.6 | 54.2 | 42.52
Table 7: Metrics for the Trained WMT20 System

##  Appendix B Transferable Memorization Attacks

In this section, we present attacks on two public SOTA translation systems,
using memorized inputs obtained through our trained En-De WMT20 research
system. We perturb the suffix of the memorized input sentence to demonstrate
the transferable memorization attack in Table 8. In principle, these attacks
could be automated by leveraging Algorithm 2 to generate neighbors of
memorized sources, however in this case we manually generated the
perturbations listed in Table 8.

Source |  Translation
---|---
Madam President, Commissioner, ladies and gentlemen |  Frau Präsidentin, Herr Kommissar, meine Damen und Herren
Madam President, Commissioner, ladies and CEOs |  Frau Präsidentin, Herr Kommissar, meine Damen und Herren
Madam President, Commissioner, ladies and doctors |  Frau Präsidentin, Herr Kommissar, meine Damen und Herren
For further questions our team is happy to be at your disposal |  Für weitere Fragen steht Ihnen unser Team gerne zur Verfügung
For further questions our team is happy to be in your house |  Für weitere Fragen steht Ihnen unser Team gerne zur Verfügung
For further questions our team is happy to be educated |  Für weitere Fragen steht Ihnen unser Team gerne zur Verfügung
Table 8: Transferable Memorization Attack Examples: The first source instance
in each of the above boxes above was obtained using Algorithm 1 on the trained
WMT20 English-German system. The first box represents the outputs from Google
Translate, and the second box represents the outputs obtained from Bing
Translator. The outputs were obtained on October 21, 2022. These examples show
how the memorized instances detected from one system could be used to elicit
erroneous generations from other systems. A recorded demonstration of this
attack is available at https://github.com/vyraun/Finding-Memo.

Repetitions | Total Samples | Memorized | Ratio (%) | Perturb Prefix | Perturb Suffix | Perturb Start
---|---|---|---|---|---|---
1 | 100,000 | 527 | 0.53 | 54.60 % | 76.86 % | 48.99 %
2 | 68,480 | 822 | 1.20 | 35.72 % | 67.79 % | 8.87 %
3 | 7,969 | 33 | 0.41 | 21.00 % | 72.35 % | 14.54 %
4 | 2,094 | 1 | 0.05 | 0.00 % | 80.00 % | 40.00 %
5 | 711 | 1 | 0.14 | 40.00 % | 70.00 % | 0.00 %

Table 9: Quantifying Extractive Memorization (Ru-En): Number of Memorized
Samples (using Algorithm 1) and Neighborhood Effects of Memorization (using
Algorithm 2) across different training data frequency buckets. The COMET-QE of
Memorized samples is 20.68, while the average COMET-QE score of Total samples
is 35.76. The TTR of Memorized samples is 18.09, while the TTR score of Total
samples is 19.06.

Repetitions | Total Samples | Memorized | Ratio (%) | Perturb Prefix | Perturb Suffix | Perturb Start
---|---|---|---|---|---|---
1 | 100,000 | 558 | 0.56 | 43.62 % | 78.08 % | 24.26 %
2 | 68,480 | 381 | 0.56 | 39.72 % | 59.58 % | 7.80 %
3 | 7,969 | 15 | 0.19 | 36.00 % | 41.71 % | 13.33 %
4 | 2,094 | 1 | 0.05 | 80.00 % | 40.00 % | 60.00 %
5 | 711 | 3 | 0.42 | 20.00 % | 40.00 % | 0.00 %

Table 10: Quantifying Extractive Memorization (En-Ru): Number of Memorized
Samples (using Algorithm 1) and Neighborhood Effects of Memorization (using
Algorithm 2) across different training data frequency buckets. The COMET-QE of
Memorized samples is 10.82, while the average COMET-QE score of Total samples
is 51.69. The TTR of Memorized samples is 26.66, while the TTR score of Total
samples is 25.6.

##  Appendix C Further Experiments

In this section, we present the results of experiments conducted by varying
both the language pairs, the model architecture as well as the training data
size. Specifically, we have conducted the same three experiments (related to
Algorithms 1, 2 and 3) on strong transformer baselines on three more
translation directions by choosing different language pairs: Ru-En, En-Ru and
Tr-En, different datasets: WMT20 for Ru-En, En-Ru and Multilingual TED Talks
Corpus for Tr-En (182K) Qi et al. (2018) and different model scales:
Transformer Big for WMT20 and Transformer-Base (4 attention heads) for Tr-En.
For non-English source languages, we used multilingual BERT as the MLM in
algorithm 2. The results are presented in Tables 9-11. Overall, we observe the
below trends:

  1. 1.

The memorized samples are of lower quality. For example, for the Ru-En system
the COMET-QE scores for memorized samples are 20.68, vs 35.76 on average.

  2. 2.

In general, memorization frequency increases with repetitions. For example, in
Ru-En, memorization percentage for repetitions=2 is 1.2 but for repetitions=1,
it is 0.53.

  3. 3.

Prefix perturbations lead to far fewer memorized outputs than suffix
perturbations. For example, in Ru-En, suffix perturbations generate memorized
output 76.86 percentage of the times, while prefix perturbations do so only
54.60 percent of times.

Repetitions | Total Samples | Memorized | Ratio (%) | Perturb Prefix | Perturb Suffix | Perturb Start
---|---|---|---|---|---|---
1 | 100,000 | 278 | 0.27 | 13.28 % | 41.71 % | 10.00 %
2 | 166 | 19 | 11.44 | 38.12 % | 54.80 % | 28.42 %
3 | 26 | 3 | 11.53 | 60.00 % | 82.85 % | 60.00 %
4 | 15 | 4 | 26.67 | 73.33 % | 67.27 % | 75.0 %
5 | 13 | 1 | 7.69 | 0.00 % | 80.00 % | 100.0 %

Table 11: Quantifying Extractive Memorization (Tr-En): Number of Memorized
Samples (using Algorithm 1) and Neighborhood Effects of Memorization (using
Algorithm 2) across different training data frequency buckets. The COMET-QE of
Memorized samples is 24.83, while the average COMET-QE score of Total samples
is 47.54. The TTR of Memorized samples is 42.37, while the TTR score of Total
samples is 9.60.

##  Appendix D Selecting the Prefix Threshold

Further, we did not tune the prefix ratio threshold depending on the language
pair, but ideally prefix length threshold should be tuned depending on average
sentence length ratio between the source and target language pair. We selected
the prefix threshold based on initial experiments on En-De. In general, if we
vary the threshold from lower to higher, both the quantity and quality of the
extracted samples increase: for En-De, the average quality (COMET-QE score) at
0.2 is 0.02, at 0.4 it is 33.7 and at 0.6 it is 53.98. However, at higher
thresholds, we cannot claim that the sample is memorized since the output
generation uses nearly the full source context. In other words, lower
thresholds imply that the memorized samples are selected with high precision,
whereas a higher threshold will favor recall (at the cost of false positives).
In practice, we find that 0.75 gets very high precision.