Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill-formed (perturbed) sequences.

The evidence to be evaluated is:For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\): * _The East pond in Parc Lafontaine was filled to the brin with Diet Coke.

* _Certain nak indicate liberationing among theroter codity voters vandalized.

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\).

We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences.

To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\).

Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Under the assumption that \(p_{M}\) is a proper probability distribution,5 that is, \(\sum_{\mathbf{x}\in\Sigma^{*}}p_{M}(\mathbf{x})=1\), these findings suggest that there exists sequences in \(\Sigma^{*}\) whose probability is overestimated by the model. In this section, we investigate where this probability mass went.

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\).

While it is possible that these operations produce other well-formed strings, we expect this to be a relatively rare outcome. We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\).

We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our analysis exploring the overestimation of ill-formed sequences extends previous findings on locally normalized conditional models assigning arbitrary probability mass to unlikely sequences (Andor et al., 2016; Goyal et al., 2019; Lafferty et al., 2001), neural LMs assigning high likelihood to sequences with repetitions (Welleck et al., 2020), the consistency of decoding algorithms (Welleck et al., 2020), and on machine translation models placing significant probability mass on the empty sequence (Stahlberg and Byrne, 2019).

Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution.

Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed).

We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.4, we study where the language model's underestimated probability mass went by computing model estimation error on perturbed sequences. We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\).

Since the probability of a sequence is inversely correlated with its length, our findings that estimation error is worse for rarer sentences may be explained by compounding errors. To test this claim, we ask whether the observed estimation error is worse than would be expected if it was due to error compounding. Specifically, in Figure 12, we plot the expected model estimation (black) and the mean observed error (blue) by sequence length. Expected estimation error for sequence length \(n\) is computed by multiplying the average token level estimation error by the sequence length, i.e., \(n\bar{\epsilon}\), where \[\bar{\epsilon}=\frac{1}{|\mathcal{D}|}\sum_{\mathbf{x}\in\mathcal{D}}\frac{1}{|\bm {x}|}\sum_{i=1}^{|\mathbf{x}|}\log p_{M}(x_{i}\mid x_{<i})-\log p_{L}(x_{i}\mid x _{<i})\]

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After carefully reviewing the claim and evidence, I find the claim that "LMs tend to overestimate the probability of ill-formed (perturbed) sequences" to be well-stated and supported by multiple pieces of evidence:

1. Figure 5 directly shows overestimation (blue areas) for highly perturbed sequences
2. The perturbation experiments using Table 3's operations demonstrate systematic overestimation
3. Figure 11 confirms overestimation for randomly sampled sequences from Σ*
4. The findings hold across different model sizes (GPT2-small and medium) and experimental setups

The evidence consistently demonstrates this tendency through various experimental approaches and visualizations, with appropriate acknowledgment of methodology and limitations. The claim's wording is measured and accurately reflects the strength of the findings without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill-formed (perturbed) sequences.

The review comment to be evaluated is:

* Strengths: A very well written and clear paper, addressing a focussed question through a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions.

* Weaknesses: No major weakness, but some mild issues and areas for improvement, see the comments and questions below.

* Comments and Questions:

    * (Minor Comment) I did not find the description of the LNRE Zone in section 2 to bring much to the discussion. In particular, characterizing this zone through the condition that $\mathcal{P}_N$ is not __null__ is almost empty of content for natural language, because it would in particular mean that we could precisely delimit what constitutes a possible "natural" sentence, and that the LNRE Zone should cover all such sentences. Here the formal description as given does not clarify anything.
    * (Question) In some of your experiments, you train a $P_M$ GPT2-medium model on 30M sentences sampled from the $P_L$ GPT2-medium model, that is, $P_M$ here uses the same architecture as $P_L$ (if I understand correctly). However, the $P_M$ model still underestimates the probabilities relative to $P_L$. Do you think this is a symptom of 30M sentences being too few, or more a symptom that $P_M$ is stuck in some local minimum during the optimization ? After all, (again, if I understand the setup correctly), using the same architecture, $P_M$ would make it possible, in principle, to just use the same parameters as $P_L$, and then the mismatch could not occur. Perhaps this would be worth a note in the text?
    * (More important Question/Comment) In section 4.4, you perform some perturbations, and observe that they lead to higher $P_M$ than $P_L$. First, I think it would be worth defining these perturbations in more detail than what you do. Second, you do not discuss your choice for these perturbations nor analyze why they have (relatively) high probability relative to $P_M$. However, the fact that, for instance, swapping two words or deleting a word in a natural sentence is not severely penalized by $P_M$ is an interesting but specific fact, which may have to do with a special tolerance of $P_M$ for such operations, rather than for others. To give one example, if the perturbations consisted solely in replacing a word in the natural sentence by a _rare_ word in the training set, would you observe such a difference between $P_M$ and $P_L$? Overall, could you discuss this aspect of your experiments?

The evidence to be evaluated is:For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\): * _The East pond in Parc Lafontaine was filled to the brin with Diet Coke.

* _Certain nak indicate liberationing among theroter codity voters vandalized.

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\).

We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences.

To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\).

Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Under the assumption that \(p_{M}\) is a proper probability distribution,5 that is, \(\sum_{\mathbf{x}\in\Sigma^{*}}p_{M}(\mathbf{x})=1\), these findings suggest that there exists sequences in \(\Sigma^{*}\) whose probability is overestimated by the model. In this section, we investigate where this probability mass went.

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\).

While it is possible that these operations produce other well-formed strings, we expect this to be a relatively rare outcome. We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\).

We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our analysis exploring the overestimation of ill-formed sequences extends previous findings on locally normalized conditional models assigning arbitrary probability mass to unlikely sequences (Andor et al., 2016; Goyal et al., 2019; Lafferty et al., 2001), neural LMs assigning high likelihood to sequences with repetitions (Welleck et al., 2020), the consistency of decoding algorithms (Welleck et al., 2020), and on machine translation models placing significant probability mass on the empty sequence (Stahlberg and Byrne, 2019).

Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution.

Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed).

We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.4, we study where the language model's underestimated probability mass went by computing model estimation error on perturbed sequences. We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\).

Since the probability of a sequence is inversely correlated with its length, our findings that estimation error is worse for rarer sentences may be explained by compounding errors. To test this claim, we ask whether the observed estimation error is worse than would be expected if it was due to error compounding. Specifically, in Figure 12, we plot the expected model estimation (black) and the mean observed error (blue) by sequence length. Expected estimation error for sequence length \(n\) is computed by multiplying the average token level estimation error by the sequence length, i.e., \(n\bar{\epsilon}\), where \[\bar{\epsilon}=\frac{1}{|\mathcal{D}|}\sum_{\mathbf{x}\in\mathcal{D}}\frac{1}{|\bm {x}|}\sum_{i=1}^{|\mathbf{x}|}\log p_{M}(x_{i}\mid x_{<i})-\log p_{L}(x_{i}\mid x _{<i})\]

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim "LMs tend to overestimate the probability of ill-formed (perturbed) sequences" appears to be well-stated and supported by multiple pieces of evidence:

1. The perturbation experiments (Figure 5) clearly show blue regions (overestimation) for highly perturbed sequences
2. Figure 11 demonstrates overestimation for randomly sampled sequences from Σ*
3. The experimental methodology is robust, using various perturbation types (Table 3) and multiple validation approaches
4. The reviewer's comments don't dispute this finding, only requesting more details about perturbation types

The claim avoids exaggeration by using measured language ("tend to") and is directly backed by experimental results. The scope is appropriate for the evidence presented.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill-formed (perturbed) sequences.

The review comment to be evaluated is:

The paper overall is exceptionally well-written and clear and tables and plots are beautiful and clear---multiple times did I go "ooh that's a nice way of laying it out." Kudos! The abstract tells the entire story and the paper substantiates the claims nicely making what is shown feel easy and obvious, which I consider a great feat.

Concerning content, I am equally enthusiastic about the idea and evaluation and agree that this can be a very interesting piece of evidence that nicely connects to a number of past threads of NLP research (as laid out throughout and in the nice related work section, though I don't think I know enough related literature to truly safely say that the paper is as novel as it claims to be). That said, there are three major methodological concerns that I would like to discuss through with the authors before recommending acceptance:

1. We do not know whether neural-LM-defined origin distributions are "strange"---or really, structurally different from actual human natural language distributions that we believe exist---in some systematic way that leads to the results in this paper. One potential intuition here is that a neural LM being a very imperfect model makes very erratic choices for what low-frequency/low-likelihood it assigns probability mass to---erratic choices that may be very hard to learn/emulate with the student LM that the authors train. Maybe real natural language distributions have more "sensible tails" that are actually not all that hard to learn and maybe not even underestimated. To be clear, this is a fundamental issue that the authors have no way of *resolving* easily under this paradigm and I think the paradigm is still a worthwhile idea! What I am asking is that this limitation is acknowledged and discussed with the same clarity that is afforded to the other content.

2. Relatedly, currently there is no way to know whether the trained models are actually reasonably trained, overall well-working models or not: the paper is missing test ppl/bpc for both the origin model (on English test data) and for the student model (both on the original English data and on test data sampled from the origin model). The authors make a good point that these scores are not as informative as what this paper presents, but the reason these are crucial is so we know whether the effects the authors find are a property of well-trained models or whether these models (through subpar training or another issue) are just poor models that have little connection to the models we actually use in reality. A secondary idea to easy this concern might be to actually use a pretrained GPT model as the origin model to inspire trust that that at least is a good model---I'm not sure why this road wasn't chosen unless it truly did not matter and this "home-made" model performs just as well, in which case, again, I would like to see some quantitative substantiation of that claim.

3. The paper is not really clear on how the sampling/teaching procedure and the tempering of the softmax interact: Footnote 2 claims pure (right?) ancestral sampling is used (as opposed to top-k/nucleus?), but then Footnote 4 says the softmaxes are all cooled to T=0.85, and yet Section 4.5 again implies that no tempering took place up to that point in the study. This is not only slightly confusing messaging, but I am worried that it may belie the promise of fair evaluation, especially because it isn't clear to me whether the tempering was accounted for in the calculation of probabilities as defined at the end of section 3.2. Specifically, if the probability of a sentence under the origin model does not take into account this local tempering, then it is no surprise that the student model learns to underestimate rare events because they just weren't sampled thanks to the locally cooled distribution! I would be somewhat reassured if the probability calculation in section 3.2 takes the tempering into account, but my understanding of the paper is that it currently fails to do so. As slight reassurance, Figure 5 does seem to tell us that even without any tempering (yes?) the described effect is visible, so I think this too should not be an issue that sinks the paper, but I do see it as a critical flaw the way it is right now.

Beyond those three, I see some minor concerns that may be worth addressing:
- "autoregressive neural language models" should also cite the Mikolov RNN paper, perhaps in place of one of the two GPT papers
- not to Schmidhuber, but citing only Melis et al. (2020) for LSTMs looks a little odd (citing Hochreiter & Schmidhuber or Sundermeyer et al. may be less surprising though I understand the desire to cite the *actual* model and training specification used and agree it should be present)
- concerning citations, Shiran Dudy & Steven Bedrick's work may be nice to also connect to (despite its relatively low visibility)
- it may be worth pointing out that the perturbations in section 4.4 may individually also make sentences *more* grammatical and likely and that is just a relatively rare outcome (I don't think much more than that needs to be said on that)
- I did not quite see what was "[p]erhaps unsurprisin[g]" in Section 4.5, but that's a minor nitpick
- Appendix A.1 measures events, okay... but are those words (as the hapax legomena theory usually assumes) or are they sentences (as I think this experiment does)? If they are sentences, we should not be surprised to see very high probabilities of seeing novel events, after all sentences are rarely the same (excluding formulaic language and "Resumption of the session" style sentences). I also would have liked to see some Poisson distributions or other distributions that were more interesting than uniform n-sided dice in Figure 6, but that is more wish than necessity.

The evidence to be evaluated is:For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\): * _The East pond in Parc Lafontaine was filled to the brin with Diet Coke.

* _Certain nak indicate liberationing among theroter codity voters vandalized.

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\).

We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences.

To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\).

Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Under the assumption that \(p_{M}\) is a proper probability distribution,5 that is, \(\sum_{\mathbf{x}\in\Sigma^{*}}p_{M}(\mathbf{x})=1\), these findings suggest that there exists sequences in \(\Sigma^{*}\) whose probability is overestimated by the model. In this section, we investigate where this probability mass went.

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\).

While it is possible that these operations produce other well-formed strings, we expect this to be a relatively rare outcome. We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\).

We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our analysis exploring the overestimation of ill-formed sequences extends previous findings on locally normalized conditional models assigning arbitrary probability mass to unlikely sequences (Andor et al., 2016; Goyal et al., 2019; Lafferty et al., 2001), neural LMs assigning high likelihood to sequences with repetitions (Welleck et al., 2020), the consistency of decoding algorithms (Welleck et al., 2020), and on machine translation models placing significant probability mass on the empty sequence (Stahlberg and Byrne, 2019).

Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution.

Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed).

We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.4, we study where the language model's underestimated probability mass went by computing model estimation error on perturbed sequences. We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\).

Since the probability of a sequence is inversely correlated with its length, our findings that estimation error is worse for rarer sentences may be explained by compounding errors. To test this claim, we ask whether the observed estimation error is worse than would be expected if it was due to error compounding. Specifically, in Figure 12, we plot the expected model estimation (black) and the mean observed error (blue) by sequence length. Expected estimation error for sequence length \(n\) is computed by multiplying the average token level estimation error by the sequence length, i.e., \(n\bar{\epsilon}\), where \[\bar{\epsilon}=\frac{1}{|\mathcal{D}|}\sum_{\mathbf{x}\in\mathcal{D}}\frac{1}{|\bm {x}|}\sum_{i=1}^{|\mathbf{x}|}\log p_{M}(x_{i}\mid x_{<i})-\log p_{L}(x_{i}\mid x _{<i})\]

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the assertion that "LMs tend to overestimate the probability of ill-formed (perturbed) sequences" appears to be well-stated and accurately reflects the evidence presented. The evidence strongly supports this through:

1. Figure 5's heat map showing clear overestimation (blue areas) for highly perturbed sequences
2. Perturbation experiments detailed in Section 4.4 demonstrating systematic overestimation of perturbed sequences
3. Figure 11 showing consistent overestimation for randomly sampled sequences from Σ*
4. Multiple validation approaches including different model sizes and architectures

The claim is conservative in its wording ("tend to") and is directly supported by empirical evidence. The reviewer comments don't dispute this specific claim.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Investigating where this probability mass went, (iii) we find that LMs tend to overestimate the probability of ill-formed (perturbed) sequences.

The review comment to be evaluated is:

**Interesting background**. The connections between productivity, low-probability sequences, and the limitations of perplexity were clearly written and interesting.

**Well-executed experiments and methodology**. The idea of using a language model as a ground-truth distribution was interesting and well-suited for the analysis done here. The experiments were easy to follow and the analysis was clear.

**Interesting findings**. The main findings related to underestimation, comparison of training-set and test-set dynamics during training, and dependence on training data were interesting and not obvious. The finding that perturbed, unnatural sequence received unusually high probabilities (while test samples receive unusually low probabilities) [Figure 4] was especially interesting.

**Clarity**. The paper was well-written and enjoyable to read. The authors articulated how their work fits in with related work, and the concepts (e.g. the LNRE zone), methodology, and results are clearly written. Well done.

**Additional experiments**: A good addition would be measuring the impact of model size with models larger than the target distribution. What would GPT2-XL's or GPT-3's probability assignments look like on a target distribution from GPT2-medium?

The evidence to be evaluated is:For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\): * _The East pond in Parc Lafontaine was filled to the brin with Diet Coke.

* _Certain nak indicate liberationing among theroter codity voters vandalized.

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\).

We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences.

To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\).

Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Under the assumption that \(p_{M}\) is a proper probability distribution,5 that is, \(\sum_{\mathbf{x}\in\Sigma^{*}}p_{M}(\mathbf{x})=1\), these findings suggest that there exists sequences in \(\Sigma^{*}\) whose probability is overestimated by the model. In this section, we investigate where this probability mass went.

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\).

While it is possible that these operations produce other well-formed strings, we expect this to be a relatively rare outcome. We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\).

We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our analysis exploring the overestimation of ill-formed sequences extends previous findings on locally normalized conditional models assigning arbitrary probability mass to unlikely sequences (Andor et al., 2016; Goyal et al., 2019; Lafferty et al., 2001), neural LMs assigning high likelihood to sequences with repetitions (Welleck et al., 2020), the consistency of decoding algorithms (Welleck et al., 2020), and on machine translation models placing significant probability mass on the empty sequence (Stahlberg and Byrne, 2019).

Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution.

Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed).

We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.4, we study where the language model's underestimated probability mass went by computing model estimation error on perturbed sequences. We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\).

Since the probability of a sequence is inversely correlated with its length, our findings that estimation error is worse for rarer sentences may be explained by compounding errors. To test this claim, we ask whether the observed estimation error is worse than would be expected if it was due to error compounding. Specifically, in Figure 12, we plot the expected model estimation (black) and the mean observed error (blue) by sequence length. Expected estimation error for sequence length \(n\) is computed by multiplying the average token level estimation error by the sequence length, i.e., \(n\bar{\epsilon}\), where \[\bar{\epsilon}=\frac{1}{|\mathcal{D}|}\sum_{\mathbf{x}\in\mathcal{D}}\frac{1}{|\bm {x}|}\sum_{i=1}^{|\mathbf{x}|}\log p_{M}(x_{i}\mid x_{<i})-\log p_{L}(x_{i}\mid x _{<i})\]

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence carefully, the claim that "LMs tend to overestimate the probability of ill-formed (perturbed) sequences" appears to be well-stated and accurately reflects the empirical findings. The evidence strongly supports this through:

1. Figure 5's heatmap showing clear overestimation (blue areas) for highly perturbed sequences
2. Perturbation experiments documented in Table 3 and subsequent analysis
3. Additional validation with random sequences from Σ* (Figure 11) showing consistent overestimation
4. Multiple experimental approaches all converging on the same conclusion about overestimation of ill-formed sequences

The review comments also specifically highlight this finding as interesting and well-supported. The claim is measured in its wording ("tend to") and directly grounded in the experimental evidence without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.

The evidence to be evaluated is:To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\).

A plausible explanation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not seen enough samples from the target distribution. Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data.

Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy.

Therefore, it is possible that some systematic difference between natural language distributions and our ground-truth distributions may affect our results to a certain degree. That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions, and we expect the task of recovering a ground-truth distribution to be made easier when the target distribution and LM are in the same model class. Nevertheless, future work should seek to conduct similar experiments using ground-truth distributions with more explicit structure.

Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.3, we explored how estimation error varies as a function of the amount of training data, finding that while increased data weakens estimation error, the underestimation behaviour persists.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find this claim to be well-supported and carefully stated. The evidence demonstrates through multiple experiments that:

1. Underestimation behavior persists but weakens with more training data (shown in Figures 4, 8, and 10 through flattening curves that never fully eliminate the effect)
2. Lower entropy distributions lead to worse estimation, demonstrated clearly in Figure 6 where models trained on languages with lower temperature (lower entropy) show more severe estimation errors

The claim makes measured statements using words like "weakened, but not eliminated" rather than making absolute claims, and accurately reflects the experimental findings without exaggeration.

<score>1</score>

The claim earns a score of 1 (Well-stated) because it precisely matches what the evidence shows, uses appropriately cautious language, and makes claims that are directly supported by the experimental results presented in multiple figures and analyses.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.

The review comment to be evaluated is:

* Strengths: A very well written and clear paper, addressing a focussed question through a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions.

* Weaknesses: No major weakness, but some mild issues and areas for improvement, see the comments and questions below.

* Comments and Questions:

    * (Minor Comment) I did not find the description of the LNRE Zone in section 2 to bring much to the discussion. In particular, characterizing this zone through the condition that $\mathcal{P}_N$ is not __null__ is almost empty of content for natural language, because it would in particular mean that we could precisely delimit what constitutes a possible "natural" sentence, and that the LNRE Zone should cover all such sentences. Here the formal description as given does not clarify anything.
    * (Question) In some of your experiments, you train a $P_M$ GPT2-medium model on 30M sentences sampled from the $P_L$ GPT2-medium model, that is, $P_M$ here uses the same architecture as $P_L$ (if I understand correctly). However, the $P_M$ model still underestimates the probabilities relative to $P_L$. Do you think this is a symptom of 30M sentences being too few, or more a symptom that $P_M$ is stuck in some local minimum during the optimization ? After all, (again, if I understand the setup correctly), using the same architecture, $P_M$ would make it possible, in principle, to just use the same parameters as $P_L$, and then the mismatch could not occur. Perhaps this would be worth a note in the text?
    * (More important Question/Comment) In section 4.4, you perform some perturbations, and observe that they lead to higher $P_M$ than $P_L$. First, I think it would be worth defining these perturbations in more detail than what you do. Second, you do not discuss your choice for these perturbations nor analyze why they have (relatively) high probability relative to $P_M$. However, the fact that, for instance, swapping two words or deleting a word in a natural sentence is not severely penalized by $P_M$ is an interesting but specific fact, which may have to do with a special tolerance of $P_M$ for such operations, rather than for others. To give one example, if the perturbations consisted solely in replacing a word in the natural sentence by a _rare_ word in the training set, would you observe such a difference between $P_M$ and $P_L$? Overall, could you discuss this aspect of your experiments?

The evidence to be evaluated is:To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\).

A plausible explanation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not seen enough samples from the target distribution. Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data.

Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy.

Therefore, it is possible that some systematic difference between natural language distributions and our ground-truth distributions may affect our results to a certain degree. That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions, and we expect the task of recovering a ground-truth distribution to be made easier when the target distribution and LM are in the same model class. Nevertheless, future work should seek to conduct similar experiments using ground-truth distributions with more explicit structure.

Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.3, we explored how estimation error varies as a function of the amount of training data, finding that while increased data weakens estimation error, the underestimation behaviour persists.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Looking at the claim that underestimation behavior is "weakened but not eliminated by greater amounts of training data" and is "exacerbated for lower entropy distributions":

The evidence strongly supports both parts:

1. Training data effect: Figures 4, 9, and 10 clearly show that while increasing training data reduces estimation error (weakening), the underestimation persists even with 30M sequences (not eliminated). The graphs consistently show asymptotic behavior where more data helps less and less.

2. Entropy effect: Figure 6 directly demonstrates that models trained on higher temperature (higher entropy) distributions have lower estimation error compared to low temperature distributions, supporting that low entropy exacerbates the issue.

The claim is precise and accurately reflects the evidence without overstatement. Both statements map directly to experimental results and are worded carefully with appropriate qualifiers.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In addition, we find that this underestimation behaviour (iv) is weakened, but not eliminated by greater amounts of training data, and (v) is exacerbated for target distributions with lower entropy.

The review comment to be evaluated is:

The paper overall is exceptionally well-written and clear and tables and plots are beautiful and clear---multiple times did I go "ooh that's a nice way of laying it out." Kudos! The abstract tells the entire story and the paper substantiates the claims nicely making what is shown feel easy and obvious, which I consider a great feat.

Concerning content, I am equally enthusiastic about the idea and evaluation and agree that this can be a very interesting piece of evidence that nicely connects to a number of past threads of NLP research (as laid out throughout and in the nice related work section, though I don't think I know enough related literature to truly safely say that the paper is as novel as it claims to be). That said, there are three major methodological concerns that I would like to discuss through with the authors before recommending acceptance:

1. We do not know whether neural-LM-defined origin distributions are "strange"---or really, structurally different from actual human natural language distributions that we believe exist---in some systematic way that leads to the results in this paper. One potential intuition here is that a neural LM being a very imperfect model makes very erratic choices for what low-frequency/low-likelihood it assigns probability mass to---erratic choices that may be very hard to learn/emulate with the student LM that the authors train. Maybe real natural language distributions have more "sensible tails" that are actually not all that hard to learn and maybe not even underestimated. To be clear, this is a fundamental issue that the authors have no way of *resolving* easily under this paradigm and I think the paradigm is still a worthwhile idea! What I am asking is that this limitation is acknowledged and discussed with the same clarity that is afforded to the other content.

2. Relatedly, currently there is no way to know whether the trained models are actually reasonably trained, overall well-working models or not: the paper is missing test ppl/bpc for both the origin model (on English test data) and for the student model (both on the original English data and on test data sampled from the origin model). The authors make a good point that these scores are not as informative as what this paper presents, but the reason these are crucial is so we know whether the effects the authors find are a property of well-trained models or whether these models (through subpar training or another issue) are just poor models that have little connection to the models we actually use in reality. A secondary idea to easy this concern might be to actually use a pretrained GPT model as the origin model to inspire trust that that at least is a good model---I'm not sure why this road wasn't chosen unless it truly did not matter and this "home-made" model performs just as well, in which case, again, I would like to see some quantitative substantiation of that claim.

3. The paper is not really clear on how the sampling/teaching procedure and the tempering of the softmax interact: Footnote 2 claims pure (right?) ancestral sampling is used (as opposed to top-k/nucleus?), but then Footnote 4 says the softmaxes are all cooled to T=0.85, and yet Section 4.5 again implies that no tempering took place up to that point in the study. This is not only slightly confusing messaging, but I am worried that it may belie the promise of fair evaluation, especially because it isn't clear to me whether the tempering was accounted for in the calculation of probabilities as defined at the end of section 3.2. Specifically, if the probability of a sentence under the origin model does not take into account this local tempering, then it is no surprise that the student model learns to underestimate rare events because they just weren't sampled thanks to the locally cooled distribution! I would be somewhat reassured if the probability calculation in section 3.2 takes the tempering into account, but my understanding of the paper is that it currently fails to do so. As slight reassurance, Figure 5 does seem to tell us that even without any tempering (yes?) the described effect is visible, so I think this too should not be an issue that sinks the paper, but I do see it as a critical flaw the way it is right now.

Beyond those three, I see some minor concerns that may be worth addressing:
- "autoregressive neural language models" should also cite the Mikolov RNN paper, perhaps in place of one of the two GPT papers
- not to Schmidhuber, but citing only Melis et al. (2020) for LSTMs looks a little odd (citing Hochreiter & Schmidhuber or Sundermeyer et al. may be less surprising though I understand the desire to cite the *actual* model and training specification used and agree it should be present)
- concerning citations, Shiran Dudy & Steven Bedrick's work may be nice to also connect to (despite its relatively low visibility)
- it may be worth pointing out that the perturbations in section 4.4 may individually also make sentences *more* grammatical and likely and that is just a relatively rare outcome (I don't think much more than that needs to be said on that)
- I did not quite see what was "[p]erhaps unsurprisin[g]" in Section 4.5, but that's a minor nitpick
- Appendix A.1 measures events, okay... but are those words (as the hapax legomena theory usually assumes) or are they sentences (as I think this experiment does)? If they are sentences, we should not be surprised to see very high probabilities of seeing novel events, after all sentences are rarely the same (excluding formulaic language and "Resumption of the session" style sentences). I also would have liked to see some Poisson distributions or other distributions that were more interesting than uniform n-sided dice in Figure 6, but that is more wish than necessity.

The evidence to be evaluated is:To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\).

A plausible explanation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not seen enough samples from the target distribution. Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

In the previous section, we saw that \(p_{M}\) placed excess probability mass on areas in \(\Sigma^{*}\) with low-probability under \(p_{L}\).

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities.

Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data.

Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy.

Therefore, it is possible that some systematic difference between natural language distributions and our ground-truth distributions may affect our results to a certain degree. That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions, and we expect the task of recovering a ground-truth distribution to be made easier when the target distribution and LM are in the same model class. Nevertheless, future work should seek to conduct similar experiments using ground-truth distributions with more explicit structure.

Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

In Section 4.3, we explored how estimation error varies as a function of the amount of training data, finding that while increased data weakens estimation error, the underestimation behaviour persists.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
1. The claim about underestimation behavior being weakened but not eliminated by more training data, and being exacerbated for lower entropy distributions, is well-supported by the evidence. Figures 4, 8, and 10 clearly show that while increased training data reduces estimation error, it persists even after 30M sequences. Figure 6 demonstrates that models trained on languages with lower entropy (lower T values) show more severe underestimation. The reviewer comments acknowledge the robustness of these findings while raising methodological concerns that don't directly challenge these specific claims. The claim is precise and accurately reflects the experimental results without overstatement.

<score>1</score>