Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Next, we study the _preference_ for memorization when generalization is possible.

The review comment to be evaluated is:

This work is placed on the border for me as it is investigating well crafted question and does some empirical validation of multiple hypotheses. On the other hand, there is not much of discussion on what can we do with long sequences when we require memorization? Authors provide very high-level paths such as 'Our findings can provide guidance for tuning the learned data structures', but I would be very curious to read more details. I think it would nicely fit as findings paper rather than work with the novel contribution.

When it comes to strengths, I like the construction of memorization properties which allows to quantify the memorization by using the proposed probing tasks. In addition, authors tried to provide many details on specific hyper-parameters they used, although I left some comments below where more details would be welcome.

Regarding weaknesses: while the memorization properties are clearly outlined, the construction of model parameterization in probing tasks is written on very high-level without any discussion of why this specific method of casting the model as classifier is chosen and how it may or may not provide necessary information towards research questions about memorization, this is mainly addressing section 3.3).
In addition, authors mention 'SOTA \ state of the art' several times during the work while they do not much relate to the actual SOTA models. Please correct me here if I am wrong, but considered models are not in the same ballpark as SOTA models. I want to stress that the latter fact is not a bad thing, but then there is no need to relate with SOTA then.

Below I have noted multiple comments where it was hard to me to get all the details or where I felt the minor change / addon would improve the presentation:

Page 3, *"At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5)."* : This is very specific design choice, why did you choose it? Model conditioned on the q gives the entire distribution on sequence-level and beam search is very sensitive to the beam size? I see an alternative as doing unbiased sampling and computing expected similarity with the answer. Some discussion on that would help to clarify this.

Bottom of page 3: you used word *setups* to refer to both tasks and models (right?) which is a bit confusing.

Section 3.3: as I wrote above, in my opinion this section needs to be larger as it defines very important design of the model parameterization, right now it is not even clear enough what are 'target class' in the context of vocabulary tokens? I believe explaining this would improve the presentation of that part a lot.

Page 5: *achieves 100% train accuracy* : in cases of MLM, was acc measured only on task class prediction or on other masked tokens as well?

Page 5: *Hence, in this experiment we used a 1-layer*:  Could you explain why you chose to reduce the model size? In my understanding the task itself might be too easy so the other way is to alter the underlying task (since you specifically construct it).

Page 5: *which was enough for the training loss to converge*: Converge to what? Was there an early stopping criteria on loss or acc?

Page 8: *Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with “duplicated” tokens would also be easier to memorize.* : In your proposed construction the frequency distribution of tokens doesn't change relatively to each other, right?. Is it the same for BPE learning when number of merges is growing? Would be great to show this, otherwise it is hard to buy this statement.

The evidence to be evaluated is:To quantify the memorization capabilities and preferences of NLP models, we use three different setups, with which we aim to cover different facets of what one can call training data memorization. Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization. As this is typically the case with natural data, we thus need a different strategy to assess memorization in more realistic scenarios. To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

However, in this experiment we use the true labels of the dataset, rather than using the random labels of the previous setup, allowing us to consider both generalization and memorization.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data.

At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5).

In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently. The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords.

In our experiments, we vary the size of the subword vocabulary used to represent the data. In turn, the number of the learned parameters that are present in the embedding layer changes, too. It is not unreasonable to expect that the memorization capabilities of a model are impacted by this growth of the number of the learned parameters.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

If a model achieves training accuracy above or equal to 0.999, the training is stopped earlier.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed. We see that for all three models, LM, MLM, and Encoder, the dependency is very consistent:

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We believe this indicates two important points: (i) generalization is not directly at odds with memorization, and (ii) there is a level of granularity that allows better memorization _and_ better generalization.

In the current section, we focus on why this might be the case, and further investigate this observed effect. We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7

Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

To summarize, the above experiments allowed us to rule out the alternative explanations (redundancy and vocabulary growth hypotheses) and _we conclude that reduction of the sequence length is the primary factor of the observed memorization effect_.

While the generalization abilities of state-of-the-art models in NLP have been quite extensively studied in the recent past (Finegan-Dollak et al., 2018; Hupkes et al., 2018; Lake and Baroni, 2018; Keysers et al., 2019; Korrel et al., 2019; Mul and Zuidema, 2019; Raunak et al., 2019; Saxton et al., 2019; Kim and Linzen, 2020; Dankers et al., 2021; Vig and Belinkov, 2019; Dubois et al., 2020; Hupkes et al., 2020; Kharitonov and Chaabouni, 2020) comparatively little is known about what factors impact how such models _memorize_.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_. We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This raises the next question: if we increase the power of attention (e.g., by growing number of heads) would the memorization be improved? To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

_Can the increased skeweness of the token distribution explain the increased memorization?_ To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim "Next, we study the preference for memorization when generalization is possible" and the provided evidence:

This claim appears well-stated (1) because the evidence shows that the authors:
- Study memorization through multiple experimental setups including membership inference attacks which specifically test memorization vs generalization
- Demonstrate that models can simultaneously exhibit both memorization and generalization capabilities
- Show empirically how different factors (vocabulary size, sequence length, etc.) affect memorization while still allowing for generalization
- Support their findings with extensive data and figures showing the relationship between memorization and model parameters

The reviewer's comments about methodology do not significantly impact this specific claim's validity. The claim modestly states what the study examines without overstating conclusions.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Next, we study the _preference_ for memorization when generalization is possible.

The review comment to be evaluated is:

This paper as the name suggests, tries to figure out whether and through which ways bpe can affect Transformer’s memorization capacity. It evaluates Transformer’s memorization under these 4 settings: memorizing random synthetic data, memorizing random labeled natural language data, recognizing training data with lower output entropy, and training data recovery QA. Experiments show that if we have more merge times for bpe (larger vocabulary size), the model’s performance on all the four settings would improve on 3 architectures, which shows that bpe indeed can affect the model’s memorization capacity. Larger the vocabulary size is, the better the model would perform on memorizing. Then, the paper tries to figure out why more merges in bpe would lead to better memorization. By excluding two other hypotheses, the paper concludes that more merges in bpe results in shorter input sequence and that makes memorizing easier for transformers.

The evidence to be evaluated is:To quantify the memorization capabilities and preferences of NLP models, we use three different setups, with which we aim to cover different facets of what one can call training data memorization. Firstly, we consider a task of learning non-systematic mappings, where labels are independent from inputs (Zhang et al., 2016). To achieve accuracy above chance, the model has to "store" the training example in some way, thus we assume that a higher training accuracy implies increased training data memorization ability.

We replace original labels with randomly sampled ones (-1 / +1, equiprobably), and we measure memorization by measuring the (training) accuracy of the models on this data.

While the task of memorizing random labels can tell us how much a model can remember, it doesn't allow us to test how much or what a model memorizes when it is trained on tasks which also admit (or require) generalization. As this is typically the case with natural data, we thus need a different strategy to assess memorization in more realistic scenarios. To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

However, in this experiment we use the true labels of the dataset, rather than using the random labels of the previous setup, allowing us to consider both generalization and memorization.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data.

At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5).

In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently. The core question that we ask in this paper is how the choice of modeling units affects the memorization behaviour of state-of-the-art Transformer models. To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords.

In our experiments, we vary the size of the subword vocabulary used to represent the data. In turn, the number of the learned parameters that are present in the embedding layer changes, too. It is not unreasonable to expect that the memorization capabilities of a model are impacted by this growth of the number of the learned parameters.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

If a model achieves training accuracy above or equal to 0.999, the training is stopped earlier.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed. We see that for all three models, LM, MLM, and Encoder, the dependency is very consistent:

This tendency persists when controlling for the number of learned parameters (Figure 0(b)).

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We believe this indicates two important points: (i) generalization is not directly at odds with memorization, and (ii) there is a level of granularity that allows better memorization _and_ better generalization.

In the current section, we focus on why this might be the case, and further investigate this observed effect. We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_). As an illustration, in the limit case of the vocabulary growth, every sequence might have a unique token. 4 In this case, there is a one-to-one mapping between those tokens and labels (answers) that will, again, simplify the work of self-attention.

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes. In Figure 4 we report the accuracy on fitting the training data with random labels as a function of the logarithm of the vocabulary size. Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7

Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

To summarize, the above experiments allowed us to rule out the alternative explanations (redundancy and vocabulary growth hypotheses) and _we conclude that reduction of the sequence length is the primary factor of the observed memorization effect_.

While the generalization abilities of state-of-the-art models in NLP have been quite extensively studied in the recent past (Finegan-Dollak et al., 2018; Hupkes et al., 2018; Lake and Baroni, 2018; Keysers et al., 2019; Korrel et al., 2019; Mul and Zuidema, 2019; Raunak et al., 2019; Saxton et al., 2019; Kim and Linzen, 2020; Dankers et al., 2021; Vig and Belinkov, 2019; Dubois et al., 2020; Hupkes et al., 2020; Kharitonov and Chaabouni, 2020) comparatively little is known about what factors impact how such models _memorize_.

In this work, we therefore focus specifically on memorization, considering in particular the impact of the choice of modelling unit, which recently became particularly relevant given that all current SOTA NLP architectures are trained with an experimentally tuned number of _subwords_. We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016). In contrast, learning useful attention patterns is hard and often multiple heads turn out to be useless (Voita et al., 2019; Michel et al., 2019). Hence, it is possible that self-attention serves as a bottleneck that limits the memorization capacity.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

This raises the next question: if we increase the power of attention (e.g., by growing number of heads) would the memorization be improved? To investigate that, we repeated our random-string memorization experiment. This time, we have fixed the BPE vocabulary size (8 merges) and varied the number of heads used in each layer as \(\{1,2,4,8,16\}\) and measured the training accuracy. Our results in Figure 5 confirm our expectations: more heads allow more successful memorization. Indeed it seems that the attention serves as a representation bottleneck that does not allow to easily model complex interactions between tokens and, in turn, limits the memorization.

From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy. Another potential confounding behavior is that BPE affects the (relative) frequency of the tokens used for modelling the data.

_Can the increased skeweness of the token distribution explain the increased memorization?_ To rule this possibility out, we run an experiment where the token distribution is fixed to be uniform, but the sequence length is reduced. To achieve this, we start from the same dataset of random strings as in Section 5.1. In this dataset, the initial distribution of 0s and 1s is uniform as we enumerate _all_ binary strings of a fixed length. Next, we replace BPE with an "idealized" subword learning procedure which maintains the uniform distribution of tokens but reduces the lengths of the strings.

Using the obtained series of datasets, we repeat the random label memorization experiment (Section 5.1) and report the results obtained in Figure 6(b). We observe that the reported effect persists even in the case when the relative frequencies of tokens are not changed and fixed to be uniform, thus disproving the hypothesis that the observed effect is due to frequency and not token length.

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that while the paper studies memorization extensively, it does not directly investigate the "preference" for memorization when generalization is possible. The evidence shows that models can simultaneously achieve both memorization and generalization ("generalization is not directly at odds with memorization"), but does not demonstrate a preference for one over the other. The experiments mainly focus on measuring memorization capacity under different conditions rather than comparing preferences between memorization and generalization.

The only relevant evidence about preference comes from membership inference attacks showing models do memorize when generalization is possible, but this doesn't establish a preference ordering. The claim extends beyond what the evidence directly supports.

<score>4</score>

Justification:
1. The claim suggests studying preference between memorization and generalization, but the evidence focuses primarily on memorization capabilities and factors affecting it.
2. The experiments measure memorization capacity but don't explicitly compare or quantify preferences between memorization and generalization.
3. The paper acknowledges both can coexist: "generalization is not directly at odds with memorization," but doesn't establish preferential ordering.
4. The reviewer's comment also focuses on memorization capacity rather than preference between memorization and generalization.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we examine how easy it is to recover training data from a trained language model.

The evidence to be evaluated is:To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

The goal of membership inference is to figure out, given \(f_{\theta}\), whether a particular example \((x_{i},y_{i})\in\mathcal{D}\) was included in the training data \(\mathcal{D}^{\prime}\).

Given a model \(f_{\theta}\) parameterized by \(\theta\), we calculate its loss on a data point \(l(f_{\theta}(x_{i}),y_{i})\) and compare to a threshold \(\tau\): if it is below the threshold, the data point belongs to the training data. By controlling \(\tau\) we can control the trade-off between precision and recall of the attack. To avoid the dependency on this parameter and to represent the entire space of possible trade-offs, we use the AUC metric. After training a model, we measure the AUC of the above rule that separates training and hold-out examples.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data. For this experiment, we use the L1 subset of the large-scale Probably Asked Questions (PAQ) dataset (Lewis et al., 2021), which contains 14M queries with candidate answers. As with SNLI, we transform this dataset so that it is suitable for training an LM by concatenating queries and their respective candidate answers, separated by a special token (:). For instance, a training example might be "where is the capital of argentia located :: buenos aires". Whenever PAQ provided more than one answer, we used the first. We lower-cased questions and answers. At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5). To speed up the evaluation, we probe a fixed random sample of 4M questions. We consider three standard Transformer-based architectures: a (causal) language model (LM), a masked language model (MLM), and a sequence encoder (Encoder). In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently.

For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size. We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2). In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

In Figure 2 we report the AUC for our membership inference experiments for all three architectures (recall that these experiments used the same SNLI data as the previous experiment, except with with true instead of random labels). Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

With our work, we contribute to the important question of what impacts memorization behaviour in Transformer models.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model. Our findings can provide guidance for tuning the learned data structures (e.g., learned Bloom filter-like existence indexes (Kraska et al., 2017)) and compression algorithms that train a Transformer model on the data-to-be-compressed as a compression step (Izacard et al., 2019; Bellard, 2021).

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0. For each prompt, we sample 20 candidates and measure how often the ground-truth continuation is among those. We report our findings in Figure 6. From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization. Overall, we conclude that our findings are likely to be independent from the specific training data extraction method used.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim "Finally, we examine how easy it is to recover training data from a trained language model" to be slightly overstated. While the paper does present extensive experiments examining memorization and data recovery in language models, the term "easy" oversimplifies the complexity of the task. The evidence shows varying degrees of success depending on model size, vocabulary size, and experimental setup. The figures demonstrate that recovery rates, while possible, are often modest (e.g., Fig 3 shows top-1 accuracy mostly below 15%). The research thoroughly examines the phenomenon but doesn't demonstrate that recovery is necessarily "easy."

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we examine how easy it is to recover training data from a trained language model.

The review comment to be evaluated is:

Pros:
1. The experiments are comprehensive. The authors design three different types of experiments to thoroughly compare the memorization ability differences between three different models (Encoder, LM, MLM).
2. The paper explores the relationship between the memorization and generalization capabilities of the model, and shows that generalization is not directly at odds with memorization, which can inspire future model design.

Cons:
1. The authors focus on three candidate causes, which the authors claimed are principal (side-)effects of applying BPE: (i) removing redundancy in the data (due to compression), (ii) increase in the number of the unique units used to represent the data, or (iii) reducing the length of the training sequences. Other factors that are closely related to the BPE vocabulary may be also important, such as the subword frequency. Can the authors explain why the three factors examined in this paper are more likely the causes than other factors (e.g., subword frequency).
2. Although the paper presents several interesting conclusions, they can not be directly used to determine a suitable vocabulary size in practical tasks, where the demand of the model's memorization is unknowable.

Concerns:
1. The conclusions in this study are only validated on the SNLI dataset (PAQ is only used for the training data recovery experiment), which may threat the universality of the findings. Experiments on other datasets and tasks (e.g. the machine translation task where Transformer was invented) are necessary.

The evidence to be evaluated is:To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

The goal of membership inference is to figure out, given \(f_{\theta}\), whether a particular example \((x_{i},y_{i})\in\mathcal{D}\) was included in the training data \(\mathcal{D}^{\prime}\).

Given a model \(f_{\theta}\) parameterized by \(\theta\), we calculate its loss on a data point \(l(f_{\theta}(x_{i}),y_{i})\) and compare to a threshold \(\tau\): if it is below the threshold, the data point belongs to the training data. By controlling \(\tau\) we can control the trade-off between precision and recall of the attack. To avoid the dependency on this parameter and to represent the entire space of possible trade-offs, we use the AUC metric. After training a model, we measure the AUC of the above rule that separates training and hold-out examples.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data. For this experiment, we use the L1 subset of the large-scale Probably Asked Questions (PAQ) dataset (Lewis et al., 2021), which contains 14M queries with candidate answers. As with SNLI, we transform this dataset so that it is suitable for training an LM by concatenating queries and their respective candidate answers, separated by a special token (:). For instance, a training example might be "where is the capital of argentia located :: buenos aires". Whenever PAQ provided more than one answer, we used the first. We lower-cased questions and answers. At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5). To speed up the evaluation, we probe a fixed random sample of 4M questions. We consider three standard Transformer-based architectures: a (causal) language model (LM), a masked language model (MLM), and a sequence encoder (Encoder). In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently.

For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size. We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2). In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

In Figure 2 we report the AUC for our membership inference experiments for all three architectures (recall that these experiments used the same SNLI data as the previous experiment, except with with true instead of random labels). Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

With our work, we contribute to the important question of what impacts memorization behaviour in Transformer models.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model. Our findings can provide guidance for tuning the learned data structures (e.g., learned Bloom filter-like existence indexes (Kraska et al., 2017)) and compression algorithms that train a Transformer model on the data-to-be-compressed as a compression step (Izacard et al., 2019; Bellard, 2021).

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0. For each prompt, we sample 20 candidates and measure how often the ground-truth continuation is among those. We report our findings in Figure 6. From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization. Overall, we conclude that our findings are likely to be independent from the specific training data extraction method used.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim "Finally, we examine how easy it is to recover training data from a trained language model" and the provided evidence:

The claim appears slightly understated compared to the extensive experimental evidence provided. The paper presents comprehensive experiments examining memorization through three different approaches: random label memorization, membership inference attacks, and direct training data recovery. The evidence shows clear patterns of memorization ability across different model architectures and vocabulary sizes, supported by multiple graphs and rigorous testing. The review comments highlight the comprehensiveness of the experiments while noting some limitations regarding dataset generalizability. The claim could have been more specific about the thorough nature of the examination and key findings regarding vocabulary size's impact on memorization.

<score>1</score>

Justification:
1. The evidence strongly supports and actually exceeds what the claim suggests:
- Demonstrates systematic evaluation through three different experimental setups
- Provides clear quantitative results showing memorization patterns (Figures 1-3, 6)
- Includes control experiments fixing parameter counts
- Tests multiple architectures (LM, MLM, Encoder)
- Examines both artificial and natural data scenarios

2. The claim uses appropriately cautious language ("examine") and doesn't overstate conclusions
3. If anything, the claim undersells the comprehensive nature of the analysis that follows

The conservative phrasing of the claim, if anything, understates rather than overstates what the evidence demonstrates.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we examine how easy it is to recover training data from a trained language model.

The review comment to be evaluated is:

This work is placed on the border for me as it is investigating well crafted question and does some empirical validation of multiple hypotheses. On the other hand, there is not much of discussion on what can we do with long sequences when we require memorization? Authors provide very high-level paths such as 'Our findings can provide guidance for tuning the learned data structures', but I would be very curious to read more details. I think it would nicely fit as findings paper rather than work with the novel contribution.

When it comes to strengths, I like the construction of memorization properties which allows to quantify the memorization by using the proposed probing tasks. In addition, authors tried to provide many details on specific hyper-parameters they used, although I left some comments below where more details would be welcome.

Regarding weaknesses: while the memorization properties are clearly outlined, the construction of model parameterization in probing tasks is written on very high-level without any discussion of why this specific method of casting the model as classifier is chosen and how it may or may not provide necessary information towards research questions about memorization, this is mainly addressing section 3.3).
In addition, authors mention 'SOTA \ state of the art' several times during the work while they do not much relate to the actual SOTA models. Please correct me here if I am wrong, but considered models are not in the same ballpark as SOTA models. I want to stress that the latter fact is not a bad thing, but then there is no need to relate with SOTA then.

Below I have noted multiple comments where it was hard to me to get all the details or where I felt the minor change / addon would improve the presentation:

Page 3, *"At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5)."* : This is very specific design choice, why did you choose it? Model conditioned on the q gives the entire distribution on sequence-level and beam search is very sensitive to the beam size? I see an alternative as doing unbiased sampling and computing expected similarity with the answer. Some discussion on that would help to clarify this.

Bottom of page 3: you used word *setups* to refer to both tasks and models (right?) which is a bit confusing.

Section 3.3: as I wrote above, in my opinion this section needs to be larger as it defines very important design of the model parameterization, right now it is not even clear enough what are 'target class' in the context of vocabulary tokens? I believe explaining this would improve the presentation of that part a lot.

Page 5: *achieves 100% train accuracy* : in cases of MLM, was acc measured only on task class prediction or on other masked tokens as well?

Page 5: *Hence, in this experiment we used a 1-layer*:  Could you explain why you chose to reduce the model size? In my understanding the task itself might be too easy so the other way is to alter the underlying task (since you specifically construct it).

Page 5: *which was enough for the training loss to converge*: Converge to what? Was there an early stopping criteria on loss or acc?

Page 8: *Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with “duplicated” tokens would also be easier to memorize.* : In your proposed construction the frequency distribution of tokens doesn't change relatively to each other, right?. Is it the same for BPE learning when number of merges is growing? Would be great to show this, otherwise it is hard to buy this statement.

The evidence to be evaluated is:To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

The goal of membership inference is to figure out, given \(f_{\theta}\), whether a particular example \((x_{i},y_{i})\in\mathcal{D}\) was included in the training data \(\mathcal{D}^{\prime}\).

Given a model \(f_{\theta}\) parameterized by \(\theta\), we calculate its loss on a data point \(l(f_{\theta}(x_{i}),y_{i})\) and compare to a threshold \(\tau\): if it is below the threshold, the data point belongs to the training data. By controlling \(\tau\) we can control the trade-off between precision and recall of the attack. To avoid the dependency on this parameter and to represent the entire space of possible trade-offs, we use the AUC metric. After training a model, we measure the AUC of the above rule that separates training and hold-out examples.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data. For this experiment, we use the L1 subset of the large-scale Probably Asked Questions (PAQ) dataset (Lewis et al., 2021), which contains 14M queries with candidate answers. As with SNLI, we transform this dataset so that it is suitable for training an LM by concatenating queries and their respective candidate answers, separated by a special token (:). For instance, a training example might be "where is the capital of argentia located :: buenos aires". Whenever PAQ provided more than one answer, we used the first. We lower-cased questions and answers. At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5). To speed up the evaluation, we probe a fixed random sample of 4M questions. We consider three standard Transformer-based architectures: a (causal) language model (LM), a masked language model (MLM), and a sequence encoder (Encoder). In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently.

For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size. We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2). In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

In Figure 2 we report the AUC for our membership inference experiments for all three architectures (recall that these experiments used the same SNLI data as the previous experiment, except with with true instead of random labels). Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

With our work, we contribute to the important question of what impacts memorization behaviour in Transformer models.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model. Our findings can provide guidance for tuning the learned data structures (e.g., learned Bloom filter-like existence indexes (Kraska et al., 2017)) and compression algorithms that train a Transformer model on the data-to-be-compressed as a compression step (Izacard et al., 2019; Bellard, 2021).

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0. For each prompt, we sample 20 candidates and measure how often the ground-truth continuation is among those. We report our findings in Figure 6. From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization. Overall, we conclude that our findings are likely to be independent from the specific training data extraction method used.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "Finally, we examine how easy it is to recover training data from a trained language model" appears to be well-stated based on the comprehensive experiments and results presented. The evidence shows detailed analysis across multiple experimental setups (random label memorization, membership inference attacks, and question-answer recovery) with clear metrics and results displayed in multiple figures. The study includes control conditions, investigates different model architectures, and acknowledges factors affecting memorization. The claim is straightforward and accurately reflects the scope of the investigation without overstating capabilities or implications.

<score>1</score>

The reviewer's comments mainly focus on improving methodology explanation and discussion rather than questioning the fundamental claim about examining data recovery. The evidence strongly supports the basic claim through:
- Systematic experiments with controlled variables
- Clear quantitative results shown in figures
- Multiple evaluation approaches
- Consideration of different architectures and parameter settings
- Robustness checks with different sampling methods


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we examine how easy it is to recover training data from a trained language model.

The review comment to be evaluated is:

This paper as the name suggests, tries to figure out whether and through which ways bpe can affect Transformer’s memorization capacity. It evaluates Transformer’s memorization under these 4 settings: memorizing random synthetic data, memorizing random labeled natural language data, recognizing training data with lower output entropy, and training data recovery QA. Experiments show that if we have more merge times for bpe (larger vocabulary size), the model’s performance on all the four settings would improve on 3 architectures, which shows that bpe indeed can affect the model’s memorization capacity. Larger the vocabulary size is, the better the model would perform on memorizing. Then, the paper tries to figure out why more merges in bpe would lead to better memorization. By excluding two other hypotheses, the paper concludes that more merges in bpe results in shorter input sequence and that makes memorizing easier for transformers.

The evidence to be evaluated is:To evaluate memorization in such cases, we resort to measuring models' vulnerability to membership inference attacks (Shokri et al., 2017). Indeed, if it is "easy" to accurately tell if a particular example was used for training, we assume it is actually "stored" in the weights of the model in some form, rather than being inferred from a more general rule or pattern (which would lead to high scores also for examples that were not in the training data, but that are likely given that data).

The goal of membership inference is to figure out, given \(f_{\theta}\), whether a particular example \((x_{i},y_{i})\in\mathcal{D}\) was included in the training data \(\mathcal{D}^{\prime}\).

Given a model \(f_{\theta}\) parameterized by \(\theta\), we calculate its loss on a data point \(l(f_{\theta}(x_{i}),y_{i})\) and compare to a threshold \(\tau\): if it is below the threshold, the data point belongs to the training data. By controlling \(\tau\) we can control the trade-off between precision and recall of the attack. To avoid the dependency on this parameter and to represent the entire space of possible trade-offs, we use the AUC metric. After training a model, we measure the AUC of the above rule that separates training and hold-out examples.

Lastly, we study the memorization capabilities of Transformer models in a setup that is closer to natural large-scale tasks. In particular, we focus on the - interesting for memorization - domain of question-answering, and we consider how well Transformer language models can reproduce exact-match answers to questions present in the training data. For this experiment, we use the L1 subset of the large-scale Probably Asked Questions (PAQ) dataset (Lewis et al., 2021), which contains 14M queries with candidate answers. As with SNLI, we transform this dataset so that it is suitable for training an LM by concatenating queries and their respective candidate answers, separated by a special token (:). For instance, a training example might be "where is the capital of argentia located :: buenos aires". Whenever PAQ provided more than one answer, we used the first. We lower-cased questions and answers. At test-time, we prompt the trained LM by a query followed with the separator token and check whether the trained LM reproduces the correct answer within top-1 or top-5 results returned by beam search (beam size 5). To speed up the evaluation, we probe a fixed random sample of 4M questions. We consider three standard Transformer-based architectures: a (causal) language model (LM), a masked language model (MLM), and a sequence encoder (Encoder). In our first two setups (random label memorization and membership inference), we study all three architecture variants. In the experiments on question-answer-recovery, we only study the LM architecture, as those experiments require the ability to sample from the model efficiently.

For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively.

As in this experiment we are interested in the ability to overfit training data, we train the models for 500 epochs.

In preliminary experiments, we found that a 4-layer Encoder model quickly achieves 100% label prediction accuracy (on train), irrespective of the number of subwords.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size. We also run complementary experiments where we fix the number of learned parameters to stay roughly the same for each vocabulary size (see SS3.2). In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

In Figure 2 we report the AUC for our membership inference experiments for all three architectures (recall that these experiments used the same SNLI data as the previous experiment, except with with true instead of random labels). Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

We firstly observe that the model size has a large impact on memorization: LM-large outperforms LM-base in all cases. Next, we see that both models confirm the pattern we observed in the previous experiments: vocabulary growth consistently leads to a growth in the memorization accuracy. Focusing on the models with the number of parameters fixed, we see that even when correcting for the total number of embedding parameters, the size of the learned vocabulary greatly affects the ability of recovering training data. In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning.

We studied memorization behavior of three types of Transformer models (causal and masked language models, and an Encoder-based classifier). We looked at memorization from three different perspectives: the ability to memorize random mappings, vulnerability to membership inference attacks, and ability to directly reproduce training data on a prompt. We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

With our work, we contribute to the important question of what impacts memorization behaviour in Transformer models.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model. Our findings can provide guidance for tuning the learned data structures (e.g., learned Bloom filter-like existence indexes (Kraska et al., 2017)) and compression algorithms that train a Transformer model on the data-to-be-compressed as a compression step (Izacard et al., 2019; Bellard, 2021).

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt. We re-evaluate the training data extraction performance for the large models used in Section 4.3. We follow exactly the same protocol as in Section 4.3 with one exception: instead of using beam search, we sequentially sample tokens from the language model (i.e., we run ancestral sampling) at temperature 1.0. For each prompt, we sample 20 candidates and measure how often the ground-truth continuation is among those. We report our findings in Figure 6. From Figure 6 we see that the reported relation has the same form as before: larger BPE vocabulary sizes lead to better memorization. Overall, we conclude that our findings are likely to be independent from the specific training data extraction method used.

Input: Figure 2: Membership inference attack, SNLI dataset. Shaded area represents ±1 SEM.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 6: Training data recovery: top-20 accuracy on extracting the correct answer when prompted with a query (sampling).

Input: Figure 5: Random label memorization accuracy vs. number of heads.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the statement "Finally, we examine how easy it is to recover training data from a trained language model" to be well-stated. The evidence shows comprehensive experimentation on training data recovery through multiple approaches:

1. Membership inference attacks (with AUC metrics)
2. Question-answering recovery tests on PAQ dataset
3. Both beam search and ancestral sampling methods
4. Evaluation across different model sizes and architectures
5. Clear quantitative results shown in Figures 2, 3, and 6

The claim is appropriately measured in its wording, as it simply states they will "examine" recovery, without making bold assertions about the results. The evidence demonstrates they conducted thorough examination through multiple methodologies.

<score>1</score>