Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our main experimental finding is that, _across all architectures and tasks, the choice of modeling units strongly affects the memorization behavior of the models, with large-cardinality BPE vocabularies greatly facilitating memorization_.

The evidence to be evaluated is:The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords. In particular, for the SNLI dataset (used in random label memorization and membership inference), we apply BPE with \((0.5,1,5,10,20)\times 10^{3}\) steps, resulting in vocabulary sizes of 611, 1097, 4943, 9574, and 18336, respectively. For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively. In our experiments, we vary the size of the subword vocabulary used to represent the data.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data.

In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning. Later, as the vocabulary size grows, in accordance with in our thought experiment, tokens become increasingly more unique to sequences which boosts the train accuracy. In contrast, the line obtained by growing the vocabulary via BPE shows a monotonic, consistent growth for all vocabulary sizes.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows. Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

In our analysis in SS5, we conjecture that the observed effect is caused by the fact that large-cardinality BPE vocabulary reduces pressure on attention to learn complex patterns due to having less tokens to attend to.

What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy.

We can expect that with the increased number of merges, BPE-learned subwords will start approximating words more closely and the frequency distribution of tokens will approach that of words. In turn, the latter follows an extremely skewed Zipf distribution (Zipf, 1949). Figure 6(a) supports this prediction: increasing the number of merges forces the token distribution to become more peaky.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim that "across all architectures and tasks, the choice of modeling units strongly affects the memorization behavior of the models, with large-cardinality BPE vocabularies greatly facilitating memorization" is well-stated and accurately reflects the evidence. The paper provides comprehensive experimental results showing consistent patterns across multiple architectures (Encoder, LM, MLM), tasks (random label memorization, membership inference, question-answer recovery), and control conditions. All figures demonstrate clear monotonic relationships between vocabulary size and memorization metrics. The authors also thoroughly investigated and ruled out alternative explanations through controlled experiments.

<score>1</score>

Justification:
1. Comprehensive evidence across architectures:
- Figures 1, 2, and 3 show consistent patterns for all tested architectures
- Results hold true for both "vanilla" and parameter-controlled conditions

2. Multiple memorization metrics:
- Random label fitting (Fig 1)
- Membership inference (Fig 2)
- Training data recovery (Fig 3)
- Results replicated with different sampling methods (Fig 6)

3. Controlled experiments ruling out alternative explanations:
- Parameter count controlled experiments
- Token frequency distribution tests (Fig 7)
- Random string experiments (Fig 4)

4. Careful methodology:
- Systematic variation of BPE steps
- Error bars/uncertainty reported
- Multiple control conditions
- Alternative hypotheses systematically tested

The claim is cautious and precise in its wording, accurately reflecting the strength and consistency of the evidence without overstating implications.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our main experimental finding is that, _across all architectures and tasks, the choice of modeling units strongly affects the memorization behavior of the models, with large-cardinality BPE vocabularies greatly facilitating memorization_.

The review comment to be evaluated is:

Pros:
1. The experiments are comprehensive. The authors design three different types of experiments to thoroughly compare the memorization ability differences between three different models (Encoder, LM, MLM).
2. The paper explores the relationship between the memorization and generalization capabilities of the model, and shows that generalization is not directly at odds with memorization, which can inspire future model design.

Cons:
1. The authors focus on three candidate causes, which the authors claimed are principal (side-)effects of applying BPE: (i) removing redundancy in the data (due to compression), (ii) increase in the number of the unique units used to represent the data, or (iii) reducing the length of the training sequences. Other factors that are closely related to the BPE vocabulary may be also important, such as the subword frequency. Can the authors explain why the three factors examined in this paper are more likely the causes than other factors (e.g., subword frequency).
2. Although the paper presents several interesting conclusions, they can not be directly used to determine a suitable vocabulary size in practical tasks, where the demand of the model's memorization is unknowable.

Concerns:
1. The conclusions in this study are only validated on the SNLI dataset (PAQ is only used for the training data recovery experiment), which may threat the universality of the findings. Experiments on other datasets and tasks (e.g. the machine translation task where Transformer was invented) are necessary.

The evidence to be evaluated is:The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords. In particular, for the SNLI dataset (used in random label memorization and membership inference), we apply BPE with \((0.5,1,5,10,20)\times 10^{3}\) steps, resulting in vocabulary sizes of 611, 1097, 4943, 9574, and 18336, respectively. For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively. In our experiments, we vary the size of the subword vocabulary used to represent the data.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data.

In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning. Later, as the vocabulary size grows, in accordance with in our thought experiment, tokens become increasingly more unique to sequences which boosts the train accuracy. In contrast, the line obtained by growing the vocabulary via BPE shows a monotonic, consistent growth for all vocabulary sizes.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows. Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

In our analysis in SS5, we conjecture that the observed effect is caused by the fact that large-cardinality BPE vocabulary reduces pressure on attention to learn complex patterns due to having less tokens to attend to.

What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy.

We can expect that with the increased number of merges, BPE-learned subwords will start approximating words more closely and the frequency distribution of tokens will approach that of words. In turn, the latter follows an extremely skewed Zipf distribution (Zipf, 1949). Figure 6(a) supports this prediction: increasing the number of merges forces the token distribution to become more peaky.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, I find this claim to be well-stated and supported by comprehensive evidence. The research demonstrates consistent effects of BPE vocabulary size on memorization across multiple experiments (random label memorization, membership inference, question-answer recovery) and different architectures (Encoder, LM, MLM). The graphs clearly show monotonic relationships between vocabulary size and memorization metrics. The authors systematically ruled out alternative explanations through controlled experiments. While the reviewer points out limited dataset coverage (mainly SNLI), the core finding about BPE vocabulary's effect on memorization is robustly supported by the evidence presented.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our main experimental finding is that, _across all architectures and tasks, the choice of modeling units strongly affects the memorization behavior of the models, with large-cardinality BPE vocabularies greatly facilitating memorization_.

The review comment to be evaluated is:

This work is placed on the border for me as it is investigating well crafted question and does some empirical validation of multiple hypotheses. On the other hand, there is not much of discussion on what can we do with long sequences when we require memorization? Authors provide very high-level paths such as 'Our findings can provide guidance for tuning the learned data structures', but I would be very curious to read more details. I think it would nicely fit as findings paper rather than work with the novel contribution.

When it comes to strengths, I like the construction of memorization properties which allows to quantify the memorization by using the proposed probing tasks. In addition, authors tried to provide many details on specific hyper-parameters they used, although I left some comments below where more details would be welcome.

Regarding weaknesses: while the memorization properties are clearly outlined, the construction of model parameterization in probing tasks is written on very high-level without any discussion of why this specific method of casting the model as classifier is chosen and how it may or may not provide necessary information towards research questions about memorization, this is mainly addressing section 3.3).
In addition, authors mention 'SOTA \ state of the art' several times during the work while they do not much relate to the actual SOTA models. Please correct me here if I am wrong, but considered models are not in the same ballpark as SOTA models. I want to stress that the latter fact is not a bad thing, but then there is no need to relate with SOTA then.

Below I have noted multiple comments where it was hard to me to get all the details or where I felt the minor change / addon would improve the presentation:

Page 3, *"At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5)."* : This is very specific design choice, why did you choose it? Model conditioned on the q gives the entire distribution on sequence-level and beam search is very sensitive to the beam size? I see an alternative as doing unbiased sampling and computing expected similarity with the answer. Some discussion on that would help to clarify this.

Bottom of page 3: you used word *setups* to refer to both tasks and models (right?) which is a bit confusing.

Section 3.3: as I wrote above, in my opinion this section needs to be larger as it defines very important design of the model parameterization, right now it is not even clear enough what are 'target class' in the context of vocabulary tokens? I believe explaining this would improve the presentation of that part a lot.

Page 5: *achieves 100% train accuracy* : in cases of MLM, was acc measured only on task class prediction or on other masked tokens as well?

Page 5: *Hence, in this experiment we used a 1-layer*:  Could you explain why you chose to reduce the model size? In my understanding the task itself might be too easy so the other way is to alter the underlying task (since you specifically construct it).

Page 5: *which was enough for the training loss to converge*: Converge to what? Was there an early stopping criteria on loss or acc?

Page 8: *Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with “duplicated” tokens would also be easier to memorize.* : In your proposed construction the frequency distribution of tokens doesn't change relatively to each other, right?. Is it the same for BPE learning when number of merges is growing? Would be great to show this, otherwise it is hard to buy this statement.

The evidence to be evaluated is:The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords. In particular, for the SNLI dataset (used in random label memorization and membership inference), we apply BPE with \((0.5,1,5,10,20)\times 10^{3}\) steps, resulting in vocabulary sizes of 611, 1097, 4943, 9574, and 18336, respectively. For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively. In our experiments, we vary the size of the subword vocabulary used to represent the data.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data.

In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning. Later, as the vocabulary size grows, in accordance with in our thought experiment, tokens become increasingly more unique to sequences which boosts the train accuracy. In contrast, the line obtained by growing the vocabulary via BPE shows a monotonic, consistent growth for all vocabulary sizes.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows. Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

In our analysis in SS5, we conjecture that the observed effect is caused by the fact that large-cardinality BPE vocabulary reduces pressure on attention to learn complex patterns due to having less tokens to attend to.

What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy.

We can expect that with the increased number of merges, BPE-learned subwords will start approximating words more closely and the frequency distribution of tokens will approach that of words. In turn, the latter follows an extremely skewed Zipf distribution (Zipf, 1949). Figure 6(a) supports this prediction: increasing the number of merges forces the token distribution to become more peaky.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about BPE vocabulary size affecting memorization behavior appears well-supported across multiple experiments and architectures. The authors demonstrate consistent patterns through:
- Random label memorization experiments
- Membership inference attacks
- Question-answer recovery tasks
- Controls for parameter counts
- Investigation of alternative hypotheses

The evidence shows clear, monotonic relationships between vocabulary size and memorization across different setups, with thorough experimental validation and careful consideration of confounding factors. The claim's wording appropriately reflects the strength of the evidence without overstating the findings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our main experimental finding is that, _across all architectures and tasks, the choice of modeling units strongly affects the memorization behavior of the models, with large-cardinality BPE vocabularies greatly facilitating memorization_.

The review comment to be evaluated is:

This paper as the name suggests, tries to figure out whether and through which ways bpe can affect Transformer’s memorization capacity. It evaluates Transformer’s memorization under these 4 settings: memorizing random synthetic data, memorizing random labeled natural language data, recognizing training data with lower output entropy, and training data recovery QA. Experiments show that if we have more merge times for bpe (larger vocabulary size), the model’s performance on all the four settings would improve on 3 architectures, which shows that bpe indeed can affect the model’s memorization capacity. Larger the vocabulary size is, the better the model would perform on memorizing. Then, the paper tries to figure out why more merges in bpe would lead to better memorization. By excluding two other hypotheses, the paper concludes that more merges in bpe results in shorter input sequence and that makes memorizing easier for transformers.

The evidence to be evaluated is:The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords. In particular, for the SNLI dataset (used in random label memorization and membership inference), we apply BPE with \((0.5,1,5,10,20)\times 10^{3}\) steps, resulting in vocabulary sizes of 611, 1097, 4943, 9574, and 18336, respectively. For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively. In our experiments, we vary the size of the subword vocabulary used to represent the data.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

as the vocabulary size grows, the models become universally more successful in fitting random labels. Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data.

In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning. Later, as the vocabulary size grows, in accordance with in our thought experiment, tokens become increasingly more unique to sequences which boosts the train accuracy. In contrast, the line obtained by growing the vocabulary via BPE shows a monotonic, consistent growth for all vocabulary sizes.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows. Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

In our analysis in SS5, we conjecture that the observed effect is caused by the fact that large-cardinality BPE vocabulary reduces pressure on attention to learn complex patterns due to having less tokens to attend to.

What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy.

We can expect that with the increased number of merges, BPE-learned subwords will start approximating words more closely and the frequency distribution of tokens will approach that of words. In turn, the latter follows an extremely skewed Zipf distribution (Zipf, 1949). Figure 6(a) supports this prediction: increasing the number of merges forces the token distribution to become more peaky.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim is well-supported by comprehensive experimental data. The paper demonstrates consistent results across:
1. Random label memorization experiments
2. Membership inference tests
3. Question-answer recovery tasks
4. Control experiments ruling out alternative hypotheses

The graphs consistently show improved memorization with larger BPE vocabularies across different architectures and experimental setups. The authors carefully controlled for confounding variables and systematically investigated alternative explanations. The claim's wording "strongly affects" and "greatly facilitating" accurately reflects the magnitude and consistency of the demonstrated effects without exaggeration.

<score>1</score>

Justification:
1. The evidence shows clear, monotonic improvements in memorization across all tested architectures as vocabulary size increases (Figures 1-4)
2. The effect persists when controlling for:
   - Number of parameters (Figures 1b, 2b)
   - Token frequency distribution (Figure 7)
   - Sampling vs beam search generation (Figure 6)
3. Alternative hypotheses were systematically tested and ruled out through controlled experiments
4. Results are quantitatively significant - e.g., Encoder accuracy improving from 87% to 98% with vocabulary size increase
5. The claim's scope is appropriately limited to observed effects on memorization behavior
6. The reviewer comment aligns with and supports the main findings about BPE's effects on memorization

The claim does not overstate the findings and accurately reflects the strength and breadth of the experimental evidence.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This observation holds even when we control for the number of trainable parameters.

The evidence to be evaluated is:To avoid this confounding factor, we complement our study with experiments where we control for the change in the number of embedding parameters. To do so, we replace the embedding layer by a combination of an embedding and a fully-connected layer. This way, we can change the dimensionality of the input & output token embeddings and control the number of the learned parameters while maintaining the rest of the model isolated from any changes.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2).

In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

In this experiment, for small vocabulary sizes, the total number of learned parameters is nearly constant.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Is the increase perhaps _due to the increase in the number of trainable parameters?_

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

We used the standard Transformer LM models of fairseq that are specified by the --arch=transformer_1m_big (large) and --arch=transformer_1m (base). We use fixed (sinusoidal) positional embeddings. The base architecture has 6 layers with 8 attention heads, embedding dimensionality of 512 and FFN dimensionality of 2048. The large architecture has 12 layers, 16 attention heads, embedding dimension of 1024 and FFN dimension of 2048.

In experiments where we control the number of learned parameters, input token embedding size is no longer attached to the embedding size within the Transformer model (which remains identical to the vanilla, non-controlled model).

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "This observation holds even when we control for the number of trainable parameters" appears to be well-stated. Multiple experiments explicitly controlled for parameters, including:

1. Using embedding layer + fully-connected layer combinations to maintain constant parameters
2. Figures showing results for both "vanilla" and parameter-controlled models
3. Clear persistence of memorization effects in parameter-controlled conditions (Fig 1b, 2b, 3b)
4. Additional controls ruling out confounding factors like token frequency distribution

The evidence thoroughly supports the claim through multiple experimental approaches and clear visualization of results, without overstating the findings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This observation holds even when we control for the number of trainable parameters.

The review comment to be evaluated is:

Pros:
1. The experiments are comprehensive. The authors design three different types of experiments to thoroughly compare the memorization ability differences between three different models (Encoder, LM, MLM).
2. The paper explores the relationship between the memorization and generalization capabilities of the model, and shows that generalization is not directly at odds with memorization, which can inspire future model design.

Cons:
1. The authors focus on three candidate causes, which the authors claimed are principal (side-)effects of applying BPE: (i) removing redundancy in the data (due to compression), (ii) increase in the number of the unique units used to represent the data, or (iii) reducing the length of the training sequences. Other factors that are closely related to the BPE vocabulary may be also important, such as the subword frequency. Can the authors explain why the three factors examined in this paper are more likely the causes than other factors (e.g., subword frequency).
2. Although the paper presents several interesting conclusions, they can not be directly used to determine a suitable vocabulary size in practical tasks, where the demand of the model's memorization is unknowable.

Concerns:
1. The conclusions in this study are only validated on the SNLI dataset (PAQ is only used for the training data recovery experiment), which may threat the universality of the findings. Experiments on other datasets and tasks (e.g. the machine translation task where Transformer was invented) are necessary.

The evidence to be evaluated is:To avoid this confounding factor, we complement our study with experiments where we control for the change in the number of embedding parameters. To do so, we replace the embedding layer by a combination of an embedding and a fully-connected layer. This way, we can change the dimensionality of the input & output token embeddings and control the number of the learned parameters while maintaining the rest of the model isolated from any changes.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2).

In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

In this experiment, for small vocabulary sizes, the total number of learned parameters is nearly constant.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Is the increase perhaps _due to the increase in the number of trainable parameters?_

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

We used the standard Transformer LM models of fairseq that are specified by the --arch=transformer_1m_big (large) and --arch=transformer_1m (base). We use fixed (sinusoidal) positional embeddings. The base architecture has 6 layers with 8 attention heads, embedding dimensionality of 512 and FFN dimensionality of 2048. The large architecture has 12 layers, 16 attention heads, embedding dimension of 1024 and FFN dimension of 2048.

In experiments where we control the number of learned parameters, input token embedding size is no longer attached to the embedding size within the Transformer model (which remains identical to the vanilla, non-controlled model).

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "This observation holds even when we control for the number of trainable parameters" appears to be well-stated. The evidence shows multiple experiments specifically designed to control for parameter count, including:

1. Using a combination of embedding and fully-connected layers to isolate changes
2. Multiple figures comparing "vanilla" vs parameter-controlled models (Figures 0-3)
3. Clear persistence of memorization effects in parameter-controlled conditions
4. Results from both artificial and natural data scenarios

The evidence directly supports the claim without exaggeration, showing the effect remains even when carefully controlling for parameter count through various experimental approaches.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

This observation holds even when we control for the number of trainable parameters.

The review comment to be evaluated is:

This work is placed on the border for me as it is investigating well crafted question and does some empirical validation of multiple hypotheses. On the other hand, there is not much of discussion on what can we do with long sequences when we require memorization? Authors provide very high-level paths such as 'Our findings can provide guidance for tuning the learned data structures', but I would be very curious to read more details. I think it would nicely fit as findings paper rather than work with the novel contribution.

When it comes to strengths, I like the construction of memorization properties which allows to quantify the memorization by using the proposed probing tasks. In addition, authors tried to provide many details on specific hyper-parameters they used, although I left some comments below where more details would be welcome.

Regarding weaknesses: while the memorization properties are clearly outlined, the construction of model parameterization in probing tasks is written on very high-level without any discussion of why this specific method of casting the model as classifier is chosen and how it may or may not provide necessary information towards research questions about memorization, this is mainly addressing section 3.3).
In addition, authors mention 'SOTA \ state of the art' several times during the work while they do not much relate to the actual SOTA models. Please correct me here if I am wrong, but considered models are not in the same ballpark as SOTA models. I want to stress that the latter fact is not a bad thing, but then there is no need to relate with SOTA then.

Below I have noted multiple comments where it was hard to me to get all the details or where I felt the minor change / addon would improve the presentation:

Page 3, *"At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5)."* : This is very specific design choice, why did you choose it? Model conditioned on the q gives the entire distribution on sequence-level and beam search is very sensitive to the beam size? I see an alternative as doing unbiased sampling and computing expected similarity with the answer. Some discussion on that would help to clarify this.

Bottom of page 3: you used word *setups* to refer to both tasks and models (right?) which is a bit confusing.

Section 3.3: as I wrote above, in my opinion this section needs to be larger as it defines very important design of the model parameterization, right now it is not even clear enough what are 'target class' in the context of vocabulary tokens? I believe explaining this would improve the presentation of that part a lot.

Page 5: *achieves 100% train accuracy* : in cases of MLM, was acc measured only on task class prediction or on other masked tokens as well?

Page 5: *Hence, in this experiment we used a 1-layer*:  Could you explain why you chose to reduce the model size? In my understanding the task itself might be too easy so the other way is to alter the underlying task (since you specifically construct it).

Page 5: *which was enough for the training loss to converge*: Converge to what? Was there an early stopping criteria on loss or acc?

Page 8: *Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with “duplicated” tokens would also be easier to memorize.* : In your proposed construction the frequency distribution of tokens doesn't change relatively to each other, right?. Is it the same for BPE learning when number of merges is growing? Would be great to show this, otherwise it is hard to buy this statement.

The evidence to be evaluated is:To avoid this confounding factor, we complement our study with experiments where we control for the change in the number of embedding parameters. To do so, we replace the embedding layer by a combination of an embedding and a fully-connected layer. This way, we can change the dimensionality of the input & output token embeddings and control the number of the learned parameters while maintaining the rest of the model isolated from any changes.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords. Hence, in this experiment we used a 1-layer version for that model type with embedding and FFN sizes of 256 & 512.

We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2).

In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

In this experiment, for small vocabulary sizes, the total number of learned parameters is nearly constant.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Is the increase perhaps _due to the increase in the number of trainable parameters?_

We show that while this growth does play a role, the effect persists when we ensure that the number of trained parameters remains constant when the granularity of the vocabulary grows.

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits. This therefore calls for a more careful consideration of how many subwords to use for a particular task, a decision that, in practical scenarios, is rarely thoroughly motivated or investigated. In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

We used the standard Transformer LM models of fairseq that are specified by the --arch=transformer_1m_big (large) and --arch=transformer_1m (base). We use fixed (sinusoidal) positional embeddings. The base architecture has 6 layers with 8 attention heads, embedding dimensionality of 512 and FFN dimensionality of 2048. The large architecture has 12 layers, 16 attention heads, embedding dimension of 1024 and FFN dimension of 2048.

In experiments where we control the number of learned parameters, input token embedding size is no longer attached to the embedding size within the Transformer model (which remains identical to the vanilla, non-controlled model).

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim "This observation holds even when we control for the number of trainable parameters" appears to be well-stated. The evidence shows:

1. Multiple experiments explicitly controlling for parameters through an embedding+FFN layer design
2. Clear visualization in Figures 1(b), 2(b), and 3(b) showing the memorization effects persist with fixed parameters
3. Detailed methodology explaining how parameters were controlled
4. Consistent results across different model architectures and tasks

The claim is measured and directly supported by the experimental evidence without exaggeration. While the reviewer raises some valid methodological questions, these don't directly challenge this specific claim about parameter control.

<score>1</score>