Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First, as a proxy for the memorization _capacity_ of a model, we use its ability to fit random, non-systematic mappings.

The evidence to be evaluated is:Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

In the current experiment, this is not strictly necessary, since we map the input examples to random labels.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization.

Some of the tasks introduced above require that the studied models are binary classifiers (e.g., learning non-systematic mappings SS 2.1).

To use LM as a classifier, we associate new tokens (which never occur in the training data) with the target labels and append them to the input strings, after the eos tokens.

While this is a non-standard way of training classifiers, it (i) reflects some of the interesting cases where language models are used as universal zero-shot learners (Brown et al., 2020), (ii) allows us to compare LM's memorization capabilities with other architectures directly.

Memorizing random labels

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this experiment we are only interested in training data memorization, hence we allow all models to overfit and do not apply any early stopping.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\).

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords. This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

To investigate the redundancy hypothesis, we experiment with learning on randomly generated incompressible data and examine whether a similar behavior persists.5 We generate random data by enumerating all \(2^{l}\) binary \((V=\{0,1\})\) sequences of length \(l\) and randomly assign each sequence to one of two classes, \(\{-1,+1\}\). To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7 For instance, say that the current vocabulary has a token "cat". We then iterate over the corpus and replace half of its occurrences with "cat1" and the other half with "cat2" (making sure that those do not happen in the prior vocabulary). Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

A flip side of this conjecture is that a more expressive attention improves memorization.

increasing the number of attention heads improve random sequence memorization.

We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt.

With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016).

Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

However, our experiment on random artificial data (Section 4.1) indicates that the reported effect holds even when the data is not compressible in the information-theoretic sense.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization. What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

we firstly group all possible pairs of 0s and 1s into new tokens (00, 01, 10, 11) and use them to encode the initial strings (e.g., "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1" becomes "00 00 00 00 00 01"). We recursively repeat this procedure, obtaining a sequence of views of the same dataset, at each step growing the vocabulary size quadratically (2, 4, 16, and 256) and reducing the string length by two. In this process, all tokens remain uniformly distributed. Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I find that it is well-stated and accurately reflects the evidence presented. The paper extensively tests and validates the use of random label memorization as a proxy for memorization capacity through multiple experiments and controls. The evidence shows that:
1. Models' ability to fit random labels directly correlates with their memorization capabilities
2. This relationship holds true across different architectures and experimental setups
3. The authors validate this proxy through complementary experiments (membership inference, question-answer recovery)
4. They control for confounding factors like parameter count and data compression

The claim is conservative in scope, using "proxy" rather than making stronger causal claims.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First, as a proxy for the memorization _capacity_ of a model, we use its ability to fit random, non-systematic mappings.

The review comment to be evaluated is:

Pros:
1. The experiments are comprehensive. The authors design three different types of experiments to thoroughly compare the memorization ability differences between three different models (Encoder, LM, MLM).
2. The paper explores the relationship between the memorization and generalization capabilities of the model, and shows that generalization is not directly at odds with memorization, which can inspire future model design.

Cons:
1. The authors focus on three candidate causes, which the authors claimed are principal (side-)effects of applying BPE: (i) removing redundancy in the data (due to compression), (ii) increase in the number of the unique units used to represent the data, or (iii) reducing the length of the training sequences. Other factors that are closely related to the BPE vocabulary may be also important, such as the subword frequency. Can the authors explain why the three factors examined in this paper are more likely the causes than other factors (e.g., subword frequency).
2. Although the paper presents several interesting conclusions, they can not be directly used to determine a suitable vocabulary size in practical tasks, where the demand of the model's memorization is unknowable.

Concerns:
1. The conclusions in this study are only validated on the SNLI dataset (PAQ is only used for the training data recovery experiment), which may threat the universality of the findings. Experiments on other datasets and tasks (e.g. the machine translation task where Transformer was invented) are necessary.

The evidence to be evaluated is:Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

In the current experiment, this is not strictly necessary, since we map the input examples to random labels.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization.

Some of the tasks introduced above require that the studied models are binary classifiers (e.g., learning non-systematic mappings SS 2.1).

To use LM as a classifier, we associate new tokens (which never occur in the training data) with the target labels and append them to the input strings, after the eos tokens.

While this is a non-standard way of training classifiers, it (i) reflects some of the interesting cases where language models are used as universal zero-shot learners (Brown et al., 2020), (ii) allows us to compare LM's memorization capabilities with other architectures directly.

Memorizing random labels

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this experiment we are only interested in training data memorization, hence we allow all models to overfit and do not apply any early stopping.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\).

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords. This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

To investigate the redundancy hypothesis, we experiment with learning on randomly generated incompressible data and examine whether a similar behavior persists.5 We generate random data by enumerating all \(2^{l}\) binary \((V=\{0,1\})\) sequences of length \(l\) and randomly assign each sequence to one of two classes, \(\{-1,+1\}\). To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7 For instance, say that the current vocabulary has a token "cat". We then iterate over the corpus and replace half of its occurrences with "cat1" and the other half with "cat2" (making sure that those do not happen in the prior vocabulary). Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

A flip side of this conjecture is that a more expressive attention improves memorization.

increasing the number of attention heads improve random sequence memorization.

We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt.

With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016).

Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

However, our experiment on random artificial data (Section 4.1) indicates that the reported effect holds even when the data is not compressible in the information-theoretic sense.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization. What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

we firstly group all possible pairs of 0s and 1s into new tokens (00, 01, 10, 11) and use them to encode the initial strings (e.g., "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1" becomes "00 00 00 00 00 01"). We recursively repeat this procedure, obtaining a sequence of views of the same dataset, at each step growing the vocabulary size quadratically (2, 4, 16, and 256) and reducing the string length by two. In this process, all tokens remain uniformly distributed. Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "as a proxy for the memorization capacity of a model, we use its ability to fit random, non-systematic mappings" appears to be well-stated. The evidence strongly supports this approach through:

1. Comprehensive experiments showing how models memorize random labels
2. Multiple validation methods (membership inference, question-answer recovery)
3. Controlled experiments ruling out alternative explanations
4. Clear correlation between memorization capacity and random mapping performance
5. Careful experimental design controlling for variables like model size and parameters

The claim is conservative in scope and accurately reflects what the evidence demonstrates, without overgeneralization or exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First, as a proxy for the memorization _capacity_ of a model, we use its ability to fit random, non-systematic mappings.

The review comment to be evaluated is:

This work is placed on the border for me as it is investigating well crafted question and does some empirical validation of multiple hypotheses. On the other hand, there is not much of discussion on what can we do with long sequences when we require memorization? Authors provide very high-level paths such as 'Our findings can provide guidance for tuning the learned data structures', but I would be very curious to read more details. I think it would nicely fit as findings paper rather than work with the novel contribution.

When it comes to strengths, I like the construction of memorization properties which allows to quantify the memorization by using the proposed probing tasks. In addition, authors tried to provide many details on specific hyper-parameters they used, although I left some comments below where more details would be welcome.

Regarding weaknesses: while the memorization properties are clearly outlined, the construction of model parameterization in probing tasks is written on very high-level without any discussion of why this specific method of casting the model as classifier is chosen and how it may or may not provide necessary information towards research questions about memorization, this is mainly addressing section 3.3).
In addition, authors mention 'SOTA \ state of the art' several times during the work while they do not much relate to the actual SOTA models. Please correct me here if I am wrong, but considered models are not in the same ballpark as SOTA models. I want to stress that the latter fact is not a bad thing, but then there is no need to relate with SOTA then.

Below I have noted multiple comments where it was hard to me to get all the details or where I felt the minor change / addon would improve the presentation:

Page 3, *"At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5)."* : This is very specific design choice, why did you choose it? Model conditioned on the q gives the entire distribution on sequence-level and beam search is very sensitive to the beam size? I see an alternative as doing unbiased sampling and computing expected similarity with the answer. Some discussion on that would help to clarify this.

Bottom of page 3: you used word *setups* to refer to both tasks and models (right?) which is a bit confusing.

Section 3.3: as I wrote above, in my opinion this section needs to be larger as it defines very important design of the model parameterization, right now it is not even clear enough what are 'target class' in the context of vocabulary tokens? I believe explaining this would improve the presentation of that part a lot.

Page 5: *achieves 100% train accuracy* : in cases of MLM, was acc measured only on task class prediction or on other masked tokens as well?

Page 5: *Hence, in this experiment we used a 1-layer*:  Could you explain why you chose to reduce the model size? In my understanding the task itself might be too easy so the other way is to alter the underlying task (since you specifically construct it).

Page 5: *which was enough for the training loss to converge*: Converge to what? Was there an early stopping criteria on loss or acc?

Page 8: *Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with “duplicated” tokens would also be easier to memorize.* : In your proposed construction the frequency distribution of tokens doesn't change relatively to each other, right?. Is it the same for BPE learning when number of merges is growing? Would be great to show this, otherwise it is hard to buy this statement.

The evidence to be evaluated is:Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

In the current experiment, this is not strictly necessary, since we map the input examples to random labels.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization.

Some of the tasks introduced above require that the studied models are binary classifiers (e.g., learning non-systematic mappings SS 2.1).

To use LM as a classifier, we associate new tokens (which never occur in the training data) with the target labels and append them to the input strings, after the eos tokens.

While this is a non-standard way of training classifiers, it (i) reflects some of the interesting cases where language models are used as universal zero-shot learners (Brown et al., 2020), (ii) allows us to compare LM's memorization capabilities with other architectures directly.

Memorizing random labels

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this experiment we are only interested in training data memorization, hence we allow all models to overfit and do not apply any early stopping.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\).

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords. This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

To investigate the redundancy hypothesis, we experiment with learning on randomly generated incompressible data and examine whether a similar behavior persists.5 We generate random data by enumerating all \(2^{l}\) binary \((V=\{0,1\})\) sequences of length \(l\) and randomly assign each sequence to one of two classes, \(\{-1,+1\}\). To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7 For instance, say that the current vocabulary has a token "cat". We then iterate over the corpus and replace half of its occurrences with "cat1" and the other half with "cat2" (making sure that those do not happen in the prior vocabulary). Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

A flip side of this conjecture is that a more expressive attention improves memorization.

increasing the number of attention heads improve random sequence memorization.

We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt.

With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016).

Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

However, our experiment on random artificial data (Section 4.1) indicates that the reported effect holds even when the data is not compressible in the information-theoretic sense.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization. What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

we firstly group all possible pairs of 0s and 1s into new tokens (00, 01, 10, 11) and use them to encode the initial strings (e.g., "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1" becomes "00 00 00 00 00 01"). We recursively repeat this procedure, obtaining a sequence of views of the same dataset, at each step growing the vocabulary size quadratically (2, 4, 16, and 256) and reducing the string length by two. In this process, all tokens remain uniformly distributed. Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim that model memorization capacity can be measured by its ability to fit random, non-systematic mappings is generally well-stated but has some minor overstatement. The evidence strongly supports using random label memorization as one proxy for capacity, showing clear patterns across multiple experiments and controls. However, the claim could be more precise by:
1) Acknowledging this is just one of several proxies (they also use membership inference and direct recovery)
2) Noting that random mapping ability may not fully capture all aspects of memorization capacity

The evidence demonstrates this metric's utility while the authors maintain appropriate caveats about its limitations.

<score>2</score>

Justification:
- The experimental evidence strongly validates using random label memorization as a proxy, with consistent results across architectures, vocabulary sizes, and controls
- Authors appropriately acknowledge limitations, stating "While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization"
- Reviewer comments focus on other aspects and don't dispute this methodological choice
- The claim avoids strong causal language, using "proxy" rather than definitive statements
- Multiple figures (1, 4, 5) demonstrate clear empirical support for random label memorization as a meaningful measure
- Authors validate the approach through controlled experiments ruling out alternative explanations

The minor overstatement comes from not explicitly positioning this as one of several complementary measures used in the paper, but this is a relatively minor issue given the careful experimental validation.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

First, as a proxy for the memorization _capacity_ of a model, we use its ability to fit random, non-systematic mappings.

The review comment to be evaluated is:

This paper as the name suggests, tries to figure out whether and through which ways bpe can affect Transformer’s memorization capacity. It evaluates Transformer’s memorization under these 4 settings: memorizing random synthetic data, memorizing random labeled natural language data, recognizing training data with lower output entropy, and training data recovery QA. Experiments show that if we have more merge times for bpe (larger vocabulary size), the model’s performance on all the four settings would improve on 3 architectures, which shows that bpe indeed can affect the model’s memorization capacity. Larger the vocabulary size is, the better the model would perform on memorizing. Then, the paper tries to figure out why more merges in bpe would lead to better memorization. By excluding two other hypotheses, the paper concludes that more merges in bpe results in shorter input sequence and that makes memorizing easier for transformers.

The evidence to be evaluated is:Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

In the current experiment, this is not strictly necessary, since we map the input examples to random labels.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization.

Some of the tasks introduced above require that the studied models are binary classifiers (e.g., learning non-systematic mappings SS 2.1).

To use LM as a classifier, we associate new tokens (which never occur in the training data) with the target labels and append them to the input strings, after the eos tokens.

While this is a non-standard way of training classifiers, it (i) reflects some of the interesting cases where language models are used as universal zero-shot learners (Brown et al., 2020), (ii) allows us to compare LM's memorization capabilities with other architectures directly.

Memorizing random labels

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this experiment we are only interested in training data memorization, hence we allow all models to overfit and do not apply any early stopping.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\).

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords. This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

To investigate the redundancy hypothesis, we experiment with learning on randomly generated incompressible data and examine whether a similar behavior persists.5 We generate random data by enumerating all \(2^{l}\) binary \((V=\{0,1\})\) sequences of length \(l\) and randomly assign each sequence to one of two classes, \(\{-1,+1\}\). To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7 For instance, say that the current vocabulary has a token "cat". We then iterate over the corpus and replace half of its occurrences with "cat1" and the other half with "cat2" (making sure that those do not happen in the prior vocabulary). Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

A flip side of this conjecture is that a more expressive attention improves memorization.

increasing the number of attention heads improve random sequence memorization.

We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt.

With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016).

Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

However, our experiment on random artificial data (Section 4.1) indicates that the reported effect holds even when the data is not compressible in the information-theoretic sense.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization. What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

we firstly group all possible pairs of 0s and 1s into new tokens (00, 01, 10, 11) and use them to encode the initial strings (e.g., "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1" becomes "00 00 00 00 00 01"). We recursively repeat this procedure, obtaining a sequence of views of the same dataset, at each step growing the vocabulary size quadratically (2, 4, 16, and 256) and reducing the string length by two. In this process, all tokens remain uniformly distributed. Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim "First, as a proxy for the memorization capacity of a model, we use its ability to fit random, non-systematic mappings" is well-stated (score of 1). The evidence strongly supports using random label memorization as a proxy for memorization capacity through:

1. Extensive experimental validation showing consistent patterns across multiple architectures
2. Careful controls including fixed parameter counts, random incompressible data tests, and vocabulary size variations
3. Thorough investigation of alternative hypotheses
4. Clear demonstration that random label memorization correlates with other memorization metrics (membership inference, QA recovery)
5. Strong theoretical justification linking random mapping ability to storage capacity

The claim is conservatively worded as using this metric as a "proxy" rather than making stronger assertions about it being a definitive measure.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Next, we study the _preference_ for memorization when generalization is possible.

The evidence to be evaluated is:To quantify the memorization capabilities and preferences of NLP models, we use three different setups, with which we aim to cover different facets of what one can call training data memorization. Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization. As this is typically the case with natural data, we thus need a different strategy to assess memorization in more realistic scenarios. To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

However, in this experiment we use the true labels of the dataset, rather than using the random labels of the previous setup, allowing us to consider both generalization and memorization.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data.

At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5).

In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently. The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords.

In our experiments, we vary the size of the subword vocabulary used to represent the data. In turn, the number of the learned parameters that are present in the embedding layer changes, too. It is not unreasonable to expect that the memorization capabilities of a model are impacted by this growth of the number of the learned parameters.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

If a model achieves training accuracy above or equal to 0.999, the training is stopped earlier.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed. We see that for all three models, LM, MLM, and Encoder, the dependency is very consistent:

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We believe this indicates two important points: (i) generalization is not directly at odds with memorization, and (ii) there is a level of granularity that allows better memorization _and_ better generalization.

In the current section, we focus on why this might be the case, and further investigate this observed effect. We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7

Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

To summarize, the above experiments allowed us to rule out the alternative explanations (redundancy and vocabulary growth hypotheses) and _we conclude that reduction of the sequence length is the primary factor of the observed memorization effect_.

While the generalization abilities of state-of-the-art models in NLP have been quite extensively studied in the recent past (Finegan-Dollak et al., 2018; Hupkes et al., 2018; Lake and Baroni, 2018; Keysers et al., 2019; Korrel et al., 2019; Mul and Zuidema, 2019; Raunak et al., 2019; Saxton et al., 2019; Kim and Linzen, 2020; Dankers et al., 2021; Vig and Belinkov, 2019; Dubois et al., 2020; Hupkes et al., 2020; Kharitonov and Chaabouni, 2020) comparatively little is known about what factors impact how such models _memorize_.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_. We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This raises the next question: if we increase the power of attention (e.g., by growing number of heads) would the memorization be improved? To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

_Can the increased skeweness of the token distribution explain the increased memorization?_ To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "we study the preference for memorization when generalization is possible" appears overstated. While the study does examine memorization capabilities under different conditions, including when generalization is possible (e.g., in the membership inference and QA experiments), it doesn't directly measure or compare "preference" for memorization versus generalization. The evidence shows how different factors affect memorization ability, but doesn't establish that models actively "prefer" to memorize. The study focuses more on memorization capacity and behavior rather than preference.

<score>4</score>

Justification:

1. The evidence shows experiments measuring memorization capabilities under different conditions:
- Random label memorization
- Membership inference attacks
- Question-answer recovery

2. While some experiments use real data where generalization is possible, the study primarily examines factors affecting memorization ability (vocabulary size, sequence length, attention heads) rather than measuring preference between memorization and generalization.

3. The authors themselves acknowledge this more limited scope: "...we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit"

4. Direct statements like "if we increase the power of attention... would the memorization be improved?" indicate the focus was on understanding memorization capacity rather than preference.

5. The term "preference" implies an active choice or bias toward memorization over generalization, which isn't supported by the experimental evidence presented.

The claim would be better stated as "we study memorization capabilities and behavior when generalization is also possible" to more accurately reflect the scope and findings of the research.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Next, we study the _preference_ for memorization when generalization is possible.

The review comment to be evaluated is:

Pros:
1. The experiments are comprehensive. The authors design three different types of experiments to thoroughly compare the memorization ability differences between three different models (Encoder, LM, MLM).
2. The paper explores the relationship between the memorization and generalization capabilities of the model, and shows that generalization is not directly at odds with memorization, which can inspire future model design.

Cons:
1. The authors focus on three candidate causes, which the authors claimed are principal (side-)effects of applying BPE: (i) removing redundancy in the data (due to compression), (ii) increase in the number of the unique units used to represent the data, or (iii) reducing the length of the training sequences. Other factors that are closely related to the BPE vocabulary may be also important, such as the subword frequency. Can the authors explain why the three factors examined in this paper are more likely the causes than other factors (e.g., subword frequency).
2. Although the paper presents several interesting conclusions, they can not be directly used to determine a suitable vocabulary size in practical tasks, where the demand of the model's memorization is unknowable.

Concerns:
1. The conclusions in this study are only validated on the SNLI dataset (PAQ is only used for the training data recovery experiment), which may threat the universality of the findings. Experiments on other datasets and tasks (e.g. the machine translation task where Transformer was invented) are necessary.

The evidence to be evaluated is:To quantify the memorization capabilities and preferences of NLP models, we use three different setups, with which we aim to cover different facets of what one can call training data memorization. Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization. As this is typically the case with natural data, we thus need a different strategy to assess memorization in more realistic scenarios. To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

However, in this experiment we use the true labels of the dataset, rather than using the random labels of the previous setup, allowing us to consider both generalization and memorization.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data.

At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5).

In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently. The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords.

In our experiments, we vary the size of the subword vocabulary used to represent the data. In turn, the number of the learned parameters that are present in the embedding layer changes, too. It is not unreasonable to expect that the memorization capabilities of a model are impacted by this growth of the number of the learned parameters.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

If a model achieves training accuracy above or equal to 0.999, the training is stopped earlier.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed. We see that for all three models, LM, MLM, and Encoder, the dependency is very consistent:

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We believe this indicates two important points: (i) generalization is not directly at odds with memorization, and (ii) there is a level of granularity that allows better memorization _and_ better generalization.

In the current section, we focus on why this might be the case, and further investigate this observed effect. We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7

Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

To summarize, the above experiments allowed us to rule out the alternative explanations (redundancy and vocabulary growth hypotheses) and _we conclude that reduction of the sequence length is the primary factor of the observed memorization effect_.

While the generalization abilities of state-of-the-art models in NLP have been quite extensively studied in the recent past (Finegan-Dollak et al., 2018; Hupkes et al., 2018; Lake and Baroni, 2018; Keysers et al., 2019; Korrel et al., 2019; Mul and Zuidema, 2019; Raunak et al., 2019; Saxton et al., 2019; Kim and Linzen, 2020; Dankers et al., 2021; Vig and Belinkov, 2019; Dubois et al., 2020; Hupkes et al., 2020; Kharitonov and Chaabouni, 2020) comparatively little is known about what factors impact how such models _memorize_.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_. We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This raises the next question: if we increase the power of attention (e.g., by growing number of heads) would the memorization be improved? To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

_Can the increased skeweness of the token distribution explain the increased memorization?_ To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim "Next, we study the preference for memorization when generalization is possible" appears to be a well-stated transition statement that accurately reflects what follows in the evidence. The paper indeed examines memorization behavior when generalization is possible through multiple experimental setups, including membership inference attacks with true labels and question-answering tasks. The claim is modest and precise, avoiding overstatement about what will be shown. The evidence demonstrates that they thoroughly investigated memorization in contexts where generalization was possible, particularly examining how models balance memorization and generalization capabilities.

<score>1</score>

Justification:
1. The evidence shows comprehensive experimental investigation of memorization in scenarios allowing generalization:
- Using true labels (not just random labels) to allow both memorization and generalization
- Testing on natural question-answering tasks
- Examining membership inference attacks with real data

2. The claim makes no exaggerated assertions about findings or conclusions, serving purely as an accurate transitional statement describing what will be studied.

3. The evidence and experiments directly support the modest scope of the claim.

4. The reviewer comments note the comprehensive nature of the experiments and validate that the paper explores the relationship between memorization and generalization capabilities.