Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution.

The review comment to be evaluated is:

The paper overall is exceptionally well-written and clear and tables and plots are beautiful and clear---multiple times did I go "ooh that's a nice way of laying it out." Kudos! The abstract tells the entire story and the paper substantiates the claims nicely making what is shown feel easy and obvious, which I consider a great feat.

Concerning content, I am equally enthusiastic about the idea and evaluation and agree that this can be a very interesting piece of evidence that nicely connects to a number of past threads of NLP research (as laid out throughout and in the nice related work section, though I don't think I know enough related literature to truly safely say that the paper is as novel as it claims to be). That said, there are three major methodological concerns that I would like to discuss through with the authors before recommending acceptance:

1. We do not know whether neural-LM-defined origin distributions are "strange"---or really, structurally different from actual human natural language distributions that we believe exist---in some systematic way that leads to the results in this paper. One potential intuition here is that a neural LM being a very imperfect model makes very erratic choices for what low-frequency/low-likelihood it assigns probability mass to---erratic choices that may be very hard to learn/emulate with the student LM that the authors train. Maybe real natural language distributions have more "sensible tails" that are actually not all that hard to learn and maybe not even underestimated. To be clear, this is a fundamental issue that the authors have no way of *resolving* easily under this paradigm and I think the paradigm is still a worthwhile idea! What I am asking is that this limitation is acknowledged and discussed with the same clarity that is afforded to the other content.

2. Relatedly, currently there is no way to know whether the trained models are actually reasonably trained, overall well-working models or not: the paper is missing test ppl/bpc for both the origin model (on English test data) and for the student model (both on the original English data and on test data sampled from the origin model). The authors make a good point that these scores are not as informative as what this paper presents, but the reason these are crucial is so we know whether the effects the authors find are a property of well-trained models or whether these models (through subpar training or another issue) are just poor models that have little connection to the models we actually use in reality. A secondary idea to easy this concern might be to actually use a pretrained GPT model as the origin model to inspire trust that that at least is a good model---I'm not sure why this road wasn't chosen unless it truly did not matter and this "home-made" model performs just as well, in which case, again, I would like to see some quantitative substantiation of that claim.

3. The paper is not really clear on how the sampling/teaching procedure and the tempering of the softmax interact: Footnote 2 claims pure (right?) ancestral sampling is used (as opposed to top-k/nucleus?), but then Footnote 4 says the softmaxes are all cooled to T=0.85, and yet Section 4.5 again implies that no tempering took place up to that point in the study. This is not only slightly confusing messaging, but I am worried that it may belie the promise of fair evaluation, especially because it isn't clear to me whether the tempering was accounted for in the calculation of probabilities as defined at the end of section 3.2. Specifically, if the probability of a sentence under the origin model does not take into account this local tempering, then it is no surprise that the student model learns to underestimate rare events because they just weren't sampled thanks to the locally cooled distribution! I would be somewhat reassured if the probability calculation in section 3.2 takes the tempering into account, but my understanding of the paper is that it currently fails to do so. As slight reassurance, Figure 5 does seem to tell us that even without any tempering (yes?) the described effect is visible, so I think this too should not be an issue that sinks the paper, but I do see it as a critical flaw the way it is right now.

Beyond those three, I see some minor concerns that may be worth addressing:
- "autoregressive neural language models" should also cite the Mikolov RNN paper, perhaps in place of one of the two GPT papers
- not to Schmidhuber, but citing only Melis et al. (2020) for LSTMs looks a little odd (citing Hochreiter & Schmidhuber or Sundermeyer et al. may be less surprising though I understand the desire to cite the *actual* model and training specification used and agree it should be present)
- concerning citations, Shiran Dudy & Steven Bedrick's work may be nice to also connect to (despite its relatively low visibility)
- it may be worth pointing out that the perturbations in section 4.4 may individually also make sentences *more* grammatical and likely and that is just a relatively rare outcome (I don't think much more than that needs to be said on that)
- I did not quite see what was "[p]erhaps unsurprisin[g]" in Section 4.5, but that's a minor nitpick
- Appendix A.1 measures events, okay... but are those words (as the hapax legomena theory usually assumes) or are they sentences (as I think this experiment does)? If they are sentences, we should not be surprised to see very high probabilities of seeing novel events, after all sentences are rarely the same (excluding formulaic language and "Resumption of the session" style sentences). I also would have liked to see some Poisson distributions or other distributions that were more interesting than uniform n-sided dice in Figure 6, but that is more wish than necessity.

The evidence to be evaluated is:We begin by briefly characterizing why distributions with a large number of rare events (LNRE) emerge in natural language, and why these events pose challenges for LMs.

Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes. Because of this property, many of the distributions in natural language--particularly the distribution over the sequences of a language--are characterized by a heavy-tail of rare events.

The LNRE zone for natural language appears to be very large, and it seems likely that \(\mathcal{P}_{N}\) will remain greater than \(0\) for samples of natural language many orders of magnitude larger than all the data currently available for training LMs.1 In the LNRE zone, it is difficult to obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation (MLE). Accounting for this enormous amount of novelty is thus a central challenge in language modeling.

In the LNRE zone, \(p_{M}\) is defined over a support containing a very large set of sequences which have never occurred in a training corpus (or equivalently, have all occurred with frequency 0), and which take on a very wide array of differing probabilities. For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\):

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\). However, since \(p_{\text{English}}\) and it's underlying generative process are unknown, it is not possible to explicitly evaluate how closely instance-level probability estimates align. As a proxy, the mean perplexity of the model on a holdout set of sequences \(\mathcal{D}\) is typically used, which measures whether the model, on average, assigns high likelihood to unseen instances. This measure does not, however, tell us whether instance-level estimates align with their true counterparts, nor is it necessarily indicative of performance on rare, idiosyncratic events in \(\mathcal{D}\).

The following section introduces a methodology to overcome these limitations. We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences. Specifically, we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model \(L\). The distribution over sequences induced by a sampling scheme from \(L\), denoted \(p_{L}\), is then our artificial language. We expect a model \(M\) of this artificial language to assign probabilities \(p_{M}(\mathbf{x})\) to sequences \(\mathbf{x}\) which match the target probabilities \(p_{L}(\mathbf{x})\) of \(\mathbf{x}\) under \(p_{L}\). To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

Using this generative model \(L\), we define the target distribution over sequences--the artificial language--as the distribution induced by an ancestral sampling scheme from \(L\). Thus, we draw instances \(\mathbf{x}=(x_{1},\dots,x_{|\mathbf{x}|})\) from our language \(p_{L}\) by recursively sampling from the conditional distribution over tokens at each time step: \(x_{t}\sim p_{L}(\cdot\mid x_{<t})\) where \(x_{1}=\textsc{BOS}\) and \(x_{t}=\textsc{EOS}\). All experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from \(L\) with softmax temperature \(T=0.85\).3 In Section 4.5, we explore the effects of different values of \(T\) when sampling from \(p_{L}\).

Given an artificial language \(p_{L}\), the task of \(M\)--the language model--is to define a distribution \(p_{M}\) whose sequence-level probability estimates closely align with the sequence-level probabilities given by \(p_{L}\). We refer to any deviation from this desiderata as model estimation error. To quantify the model estimation error for a sequence \(\mathbf{x}\), we take the difference between the sequence's log probability under \(M\) and its true log probability under \(L\): \[\text{error}(\mathbf{x})=\log p_{M}(\mathbf{x})-\log p_{L}(\mathbf{x}) \tag{2}\] This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\). In practice, we train \(M\) on a set of sequences \(\mathcal{D}_{\text{train}}\) sampled from \(p_{L}\), and compute model estimation error on a separate set of sequences \(\mathcal{D}_{\text{test}}\) sampled from \(p_{L}\).

We study the estimation performance of two neural LM architectures: the Transformer (Vaswani et al., 2017) and the LSTM (Melis et al., 2020).

Since these models are in the same model class as our artificial target language \(p_{L}\), we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown.

We begin by exploring model estimation error on a fixed training set \(\mathcal{D}_{\text{train}}\) of 1M sequences sampled from \(p_{L}\).

Following training, we sample a test set \(\mathcal{D}_{\text{test}}\) of 500,000 sequences from \(p_{L}\), and score each sequence under both the model distribution \(p_{M}\) and the true language distribution \(p_{L}\). From this, we obtain a set of probability estimates: \(\mathcal{S}_{\text{test}}=\left\{\left.p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle \right.\mathbf{x}\in\mathcal{D}_{\text{test}}\right\}\). If the model \(M\) perfectly models the language \(L\), then for each \(\left\langle p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle\in\mathcal{S}_{\text{test}}\) we would expect \(p_{L}(\mathbf{x})=p_{M}(\mathbf{x})\). Figures 2(A) and 2(B) visualize this relationship with the \(x\)- and \(y\)-axes denoting the true and model estimated sequence probabilities respectively, and a dashed line representing equality. To compare probability estimates, we represent the set \(\mathcal{S}_{\text{test}}\) in the form of a joint histogram over this coordinate space. Histogram bins are shaded based on the number of tuples which lie in the coordinate range they define. Importantly, any deviation of this histogram from the identity line indicates that the model distorts the shape of the distribution of the language. To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\). Once computed, we sort each set of probability tuples \(\mathcal{S}_{\text{train}}^{(i)}\) and \(\mathcal{S}_{\text{test}}^{(i)}\) by their target sequence probabilities \(p_{L}(\mathbf{x})\), and split the probability tuples into 50 equally-sized bins. We plot estimation curves in Figure 3: Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

We again split test sequences by their true probability, with darker lines denoting estimation trends for less probable target sequences. The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution. In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\). Sequence perturbation operations are shown in Table 3.

We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

Here we modulate the shape of the sequence space defined by \(p_{L}\) to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that \(p_{L}\) allocates probability mass over \(\Sigma^{*}\), we control the entropy of the conditional distributions at each generation step \(t\) by dividing the pre-softmax logits by a temperature value \(T\).6Footnote 6: Formally, for the \(i\)th component of the length \(K\) pre-softmax logits \(\mathbf{x}\), this operation is given as: \(\texttt{softmax}(\mathbf{x})_{i}=\frac{\exp(\frac{\mathbf{x}}{t})}{\sum_{j=1}^{K}\exp \left(\frac{\mathbf{x}}{T}\right)}\)

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities. These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across \(\Sigma^{*}\).

We take a more fine-grained approach, proposing a methodology which draws off of instance-level evaluation schemes (Zhong et al., 2021) and the experimental control afforded by artificial corpora (White and Cotterell, 2021; Papadimitriou and Jurafsky, 2020). Indeed, closely related to our work is Kulikov et al. (2021)'s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling.

In this work, we introduce a controlled methodology to evaluate instance-level LM performance on this set of individually rare but collectively frequent events. We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMs on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language. Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data. Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution. Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy. Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWebText corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine-tuned on 1.5M sequences from the OpenWebText corpus.

Analogously to the experiment in Section 4.1, we train a randomly-initialized GPT2-medium model on 1M sequences sampled from the fine-tuned model. In the center of Figure 7, we visualize mean model estimation error for \(50,000\) test sequences as a function of true sequence probability. Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed). Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

To provide further evidence that these results hold for other values of \(T\), we conduct similar experiments as in Section 4.3 with an artificial language \(p_{L}\) defined by an ancestral sampling scheme with \(T=1.00\). Specifically, we train GPT2-medium and an GPT2-large on a total of 30M sequences sampled from \(p_{L}\), and we compute model estimation error on a set of withheld test sequences at each training iteration.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

Given this length, we sample \(l\) tokens from \(\Sigma^{*}\) and concatenate all tokens to form a sequence. We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\). Autoregressive neural language models decompose the joint distribution \(p(\mathbf{x})\) over sequences into a series of conditional distributions \(p(x_{i}\mid x_{<i})\). Generating a sequence of length \(n\), then, involves estimating \(n\) conditional distributions.

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length. This suggests that estimation error for longer (and typically rarer) sequences is not solely due to error compounding. In section 2, we formally defined the LNRE zone as the range of values of \(N\) for which there is non-zero probability of sampling a novel event at the \(N+1\)th draw. Here we introduce a frequentist estimator for this probability. This in turn allows us to empirically verify if a sample exists in the LNRE zone. Suppose we have a set of \(N\) events \(\mathcal{D}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\) drawn from some generative process \(\phi\). Given \(\mathcal{D}\), we aim to obtain an empirical estimate for the potential productivity \(\hat{\mathcal{P}}_{N}\): the amount of probability allocated to unseen events as a function of \(N\). We can do so using the Good-Turing estimate for the probability of an event given its frequency (Good, 1953). Specifically, let \(f(\mathbf{x},\mathcal{D})\) be a function which returns the frequency of the event \(\mathbf{x}\) in \(\mathcal{D}\). Let \(N_{m}\) denote the number of types (unique events) in \(\mathcal{D}\) for which \(f(\mathbf{x},\mathcal{D})=m\). Good-Turing says that for large \(N\), the probability of the event \(\mathbf{x}\) given that it has occurred with frequency \(m\) in our sample \(\mathcal{D}\) of size \(N\) is equal to \[\hat{P}(\mathbf{x}\mid f(\mathbf{x},\mathcal{D})=m)=\frac{(m+1)}{N}\frac{N_{m+1}}{N_{m}} \tag{3}\]

We apply this method to a subset of OpenWebText, a popular language modeling corpus. In Figure 13, we plot the the empirical estimate of \(\mathcal{P}_{N}\) as function of the sample size \(N\) for \(n\)-grams sampled from a subset of OpenWebText. Particularly for \(n\)-grams with \(n\geq 3\), we find that there is significant probability of sampling a previously unseen event, even for \(N>10,000,000\).

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 1: GPT2 sequence probability estimates plotted against the true sequence probabilities. Neural LMs underestimate the probability of sequences drawn from the language they are trained to model.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input: Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 0.70.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence presented, I have carefully evaluated the claim "In this work, we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution."

This claim appears to be well-stated (score of 1) because:
1. The paper thoroughly presents a detailed methodology using artificial languages to evaluate LM performance on rare sequences
2. The evidence includes extensive experiments, visualizations, and analyses demonstrating how the methodology enables precise measurement of LM probability estimates
3. The results clearly show systematic patterns in how LMs estimate probabilities in the distribution tail
4. The reviewer's comments validate the clarity and substantiation of the methodology

The claim accurately represents what was accomplished without exaggerating the scope or implications of the work.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution.

The review comment to be evaluated is:

**Interesting background**. The connections between productivity, low-probability sequences, and the limitations of perplexity were clearly written and interesting.

**Well-executed experiments and methodology**. The idea of using a language model as a ground-truth distribution was interesting and well-suited for the analysis done here. The experiments were easy to follow and the analysis was clear.

**Interesting findings**. The main findings related to underestimation, comparison of training-set and test-set dynamics during training, and dependence on training data were interesting and not obvious. The finding that perturbed, unnatural sequence received unusually high probabilities (while test samples receive unusually low probabilities) [Figure 4] was especially interesting.

**Clarity**. The paper was well-written and enjoyable to read. The authors articulated how their work fits in with related work, and the concepts (e.g. the LNRE zone), methodology, and results are clearly written. Well done.

**Additional experiments**: A good addition would be measuring the impact of model size with models larger than the target distribution. What would GPT2-XL's or GPT-3's probability assignments look like on a target distribution from GPT2-medium?

The evidence to be evaluated is:We begin by briefly characterizing why distributions with a large number of rare events (LNRE) emerge in natural language, and why these events pose challenges for LMs.

Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes. Because of this property, many of the distributions in natural language--particularly the distribution over the sequences of a language--are characterized by a heavy-tail of rare events.

The LNRE zone for natural language appears to be very large, and it seems likely that \(\mathcal{P}_{N}\) will remain greater than \(0\) for samples of natural language many orders of magnitude larger than all the data currently available for training LMs.1 In the LNRE zone, it is difficult to obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation (MLE). Accounting for this enormous amount of novelty is thus a central challenge in language modeling.

In the LNRE zone, \(p_{M}\) is defined over a support containing a very large set of sequences which have never occurred in a training corpus (or equivalently, have all occurred with frequency 0), and which take on a very wide array of differing probabilities. For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\):

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\). However, since \(p_{\text{English}}\) and it's underlying generative process are unknown, it is not possible to explicitly evaluate how closely instance-level probability estimates align. As a proxy, the mean perplexity of the model on a holdout set of sequences \(\mathcal{D}\) is typically used, which measures whether the model, on average, assigns high likelihood to unseen instances. This measure does not, however, tell us whether instance-level estimates align with their true counterparts, nor is it necessarily indicative of performance on rare, idiosyncratic events in \(\mathcal{D}\).

The following section introduces a methodology to overcome these limitations. We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences. Specifically, we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model \(L\). The distribution over sequences induced by a sampling scheme from \(L\), denoted \(p_{L}\), is then our artificial language. We expect a model \(M\) of this artificial language to assign probabilities \(p_{M}(\mathbf{x})\) to sequences \(\mathbf{x}\) which match the target probabilities \(p_{L}(\mathbf{x})\) of \(\mathbf{x}\) under \(p_{L}\). To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

Using this generative model \(L\), we define the target distribution over sequences--the artificial language--as the distribution induced by an ancestral sampling scheme from \(L\). Thus, we draw instances \(\mathbf{x}=(x_{1},\dots,x_{|\mathbf{x}|})\) from our language \(p_{L}\) by recursively sampling from the conditional distribution over tokens at each time step: \(x_{t}\sim p_{L}(\cdot\mid x_{<t})\) where \(x_{1}=\textsc{BOS}\) and \(x_{t}=\textsc{EOS}\). All experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from \(L\) with softmax temperature \(T=0.85\).3 In Section 4.5, we explore the effects of different values of \(T\) when sampling from \(p_{L}\).

Given an artificial language \(p_{L}\), the task of \(M\)--the language model--is to define a distribution \(p_{M}\) whose sequence-level probability estimates closely align with the sequence-level probabilities given by \(p_{L}\). We refer to any deviation from this desiderata as model estimation error. To quantify the model estimation error for a sequence \(\mathbf{x}\), we take the difference between the sequence's log probability under \(M\) and its true log probability under \(L\): \[\text{error}(\mathbf{x})=\log p_{M}(\mathbf{x})-\log p_{L}(\mathbf{x}) \tag{2}\] This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\). In practice, we train \(M\) on a set of sequences \(\mathcal{D}_{\text{train}}\) sampled from \(p_{L}\), and compute model estimation error on a separate set of sequences \(\mathcal{D}_{\text{test}}\) sampled from \(p_{L}\).

We study the estimation performance of two neural LM architectures: the Transformer (Vaswani et al., 2017) and the LSTM (Melis et al., 2020).

Since these models are in the same model class as our artificial target language \(p_{L}\), we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown.

We begin by exploring model estimation error on a fixed training set \(\mathcal{D}_{\text{train}}\) of 1M sequences sampled from \(p_{L}\).

Following training, we sample a test set \(\mathcal{D}_{\text{test}}\) of 500,000 sequences from \(p_{L}\), and score each sequence under both the model distribution \(p_{M}\) and the true language distribution \(p_{L}\). From this, we obtain a set of probability estimates: \(\mathcal{S}_{\text{test}}=\left\{\left.p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle \right.\mathbf{x}\in\mathcal{D}_{\text{test}}\right\}\). If the model \(M\) perfectly models the language \(L\), then for each \(\left\langle p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle\in\mathcal{S}_{\text{test}}\) we would expect \(p_{L}(\mathbf{x})=p_{M}(\mathbf{x})\). Figures 2(A) and 2(B) visualize this relationship with the \(x\)- and \(y\)-axes denoting the true and model estimated sequence probabilities respectively, and a dashed line representing equality. To compare probability estimates, we represent the set \(\mathcal{S}_{\text{test}}\) in the form of a joint histogram over this coordinate space. Histogram bins are shaded based on the number of tuples which lie in the coordinate range they define. Importantly, any deviation of this histogram from the identity line indicates that the model distorts the shape of the distribution of the language. To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\). Once computed, we sort each set of probability tuples \(\mathcal{S}_{\text{train}}^{(i)}\) and \(\mathcal{S}_{\text{test}}^{(i)}\) by their target sequence probabilities \(p_{L}(\mathbf{x})\), and split the probability tuples into 50 equally-sized bins. We plot estimation curves in Figure 3: Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

We again split test sequences by their true probability, with darker lines denoting estimation trends for less probable target sequences. The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution. In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\). Sequence perturbation operations are shown in Table 3.

We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

Here we modulate the shape of the sequence space defined by \(p_{L}\) to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that \(p_{L}\) allocates probability mass over \(\Sigma^{*}\), we control the entropy of the conditional distributions at each generation step \(t\) by dividing the pre-softmax logits by a temperature value \(T\).6Footnote 6: Formally, for the \(i\)th component of the length \(K\) pre-softmax logits \(\mathbf{x}\), this operation is given as: \(\texttt{softmax}(\mathbf{x})_{i}=\frac{\exp(\frac{\mathbf{x}}{t})}{\sum_{j=1}^{K}\exp \left(\frac{\mathbf{x}}{T}\right)}\)

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities. These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across \(\Sigma^{*}\).

We take a more fine-grained approach, proposing a methodology which draws off of instance-level evaluation schemes (Zhong et al., 2021) and the experimental control afforded by artificial corpora (White and Cotterell, 2021; Papadimitriou and Jurafsky, 2020). Indeed, closely related to our work is Kulikov et al. (2021)'s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling.

In this work, we introduce a controlled methodology to evaluate instance-level LM performance on this set of individually rare but collectively frequent events. We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMs on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language. Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data. Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution. Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy. Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWebText corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine-tuned on 1.5M sequences from the OpenWebText corpus.

Analogously to the experiment in Section 4.1, we train a randomly-initialized GPT2-medium model on 1M sequences sampled from the fine-tuned model. In the center of Figure 7, we visualize mean model estimation error for \(50,000\) test sequences as a function of true sequence probability. Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed). Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

To provide further evidence that these results hold for other values of \(T\), we conduct similar experiments as in Section 4.3 with an artificial language \(p_{L}\) defined by an ancestral sampling scheme with \(T=1.00\). Specifically, we train GPT2-medium and an GPT2-large on a total of 30M sequences sampled from \(p_{L}\), and we compute model estimation error on a set of withheld test sequences at each training iteration.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

Given this length, we sample \(l\) tokens from \(\Sigma^{*}\) and concatenate all tokens to form a sequence. We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\). Autoregressive neural language models decompose the joint distribution \(p(\mathbf{x})\) over sequences into a series of conditional distributions \(p(x_{i}\mid x_{<i})\). Generating a sequence of length \(n\), then, involves estimating \(n\) conditional distributions.

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length. This suggests that estimation error for longer (and typically rarer) sequences is not solely due to error compounding. In section 2, we formally defined the LNRE zone as the range of values of \(N\) for which there is non-zero probability of sampling a novel event at the \(N+1\)th draw. Here we introduce a frequentist estimator for this probability. This in turn allows us to empirically verify if a sample exists in the LNRE zone. Suppose we have a set of \(N\) events \(\mathcal{D}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\) drawn from some generative process \(\phi\). Given \(\mathcal{D}\), we aim to obtain an empirical estimate for the potential productivity \(\hat{\mathcal{P}}_{N}\): the amount of probability allocated to unseen events as a function of \(N\). We can do so using the Good-Turing estimate for the probability of an event given its frequency (Good, 1953). Specifically, let \(f(\mathbf{x},\mathcal{D})\) be a function which returns the frequency of the event \(\mathbf{x}\) in \(\mathcal{D}\). Let \(N_{m}\) denote the number of types (unique events) in \(\mathcal{D}\) for which \(f(\mathbf{x},\mathcal{D})=m\). Good-Turing says that for large \(N\), the probability of the event \(\mathbf{x}\) given that it has occurred with frequency \(m\) in our sample \(\mathcal{D}\) of size \(N\) is equal to \[\hat{P}(\mathbf{x}\mid f(\mathbf{x},\mathcal{D})=m)=\frac{(m+1)}{N}\frac{N_{m+1}}{N_{m}} \tag{3}\]

We apply this method to a subset of OpenWebText, a popular language modeling corpus. In Figure 13, we plot the the empirical estimate of \(\mathcal{P}_{N}\) as function of the sample size \(N\) for \(n\)-grams sampled from a subset of OpenWebText. Particularly for \(n\)-grams with \(n\geq 3\), we find that there is significant probability of sampling a previously unseen event, even for \(N>10,000,000\).

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 1: GPT2 sequence probability estimates plotted against the true sequence probabilities. Neural LMs underestimate the probability of sequences drawn from the language they are trained to model.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input: Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 0.70.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, this claim appears to be well-stated and accurately reflects what the paper demonstrates. The authors clearly outline their methodology of using a language model as ground truth to evaluate instance-level performance on rare events. The evidence shows comprehensive experiments, including training various models on sequences sampled from artificial languages, analyzing estimation errors, and conducting perturbation studies. The review comments also support that this was a "well-executed" methodology with "clear analysis." The claim modestly states they "introduce a controlled methodology" without overstating its significance or making grandiose claims about its implications.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since the true distribution of any natural language is in practice unknown, we use a Transformer LM trained on natural data as a generative model to define target artificial languages for which we can exactly compute sequence probabilities.

The evidence to be evaluated is:In the context of language production, a language user has the ability to produce, at any given point in their linguistic lifespan, an utterance which they have never produced before. This creativity is the result of the generative property of productivity, which states that on the basis of finite linguistic experience, a language user can produce and comprehend an unbounded number of grammatically acceptable utterances (Chomsky, 1975, 1955). Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes. Because of this property, many of the distributions in natural language--particularly the distribution over the sequences of a language--are characterized by a heavy-tail of rare events.

A model \(M\) of the language \(L\) attempts to define a distribution \(p_{M}\) which closely resembles the true distribution of the language \(p_{L}\). In a locally-normalized autoregressive model, this distribution is defined by assigning probabilities to variable length sequences \(\mathbf{x}\) via a chain-rule decomposition:

For \(p_{M}\) to perfectly approximate \(p_{L}\), we expect \(p_{M}(\mathbf{x})=p_{L}(\mathbf{x})\) for all \(\mathbf{x}\in\Sigma^{*}\), where \(\Sigma^{*}\) is the set of all strings of finite length (the Kleene closure of \(\Sigma\)).

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\). However, since \(p_{\text{English}}\) and it's underlying generative process are unknown, it is not possible to explicitly evaluate how closely instance-level probability estimates align.

Specifically, we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model \(L\). The distribution over sequences induced by a sampling scheme from \(L\), denoted \(p_{L}\), is then our artificial language.

To define a generative model \(L\), we train a randomly-initialized GPT2-medium on 1.5M sentences sampled from the OpenWebText corpus (Gokaslan and Cohen, 2019).2

Using this generative model \(L\), we define the target distribution over sequences--the artificial language--as the distribution induced by an ancestral sampling scheme from \(L\). Thus, we draw instances \(\mathbf{x}=(x_{1},\dots,x_{|\mathbf{x}|})\) from our language \(p_{L}\) by recursively sampling from the conditional distribution over tokens at each time step: \(x_{t}\sim p_{L}(\cdot\mid x_{<t})\) where \(x_{1}=\textsc{BOS}\) and \(x_{t}=\textsc{EOS}\). All experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from \(L\) with softmax temperature \(T=0.85\).3 In Section 4.5, we explore the effects of different values of \(T\) when sampling from \(p_{L}\). Table 2 shows three sequences sampled from this distribution.

Given an artificial language \(p_{L}\), the task of \(M\)--the language model--is to define a distribution \(p_{M}\) whose sequence-level probability estimates closely align with the sequence-level probabilities given by \(p_{L}\).

In practice, we train \(M\) on a set of sequences \(\mathcal{D}_{\text{train}}\) sampled from \(p_{L}\), and compute model estimation error on a separate set of sequences \(\mathcal{D}_{\text{test}}\) sampled from \(p_{L}\).

Since these models are in the same model class as our artificial target language \(p_{L}\), we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown.

Following training, we sample a test set \(\mathcal{D}_{\text{test}}\) of 500,000 sequences from \(p_{L}\), and score each sequence under both the model distribution \(p_{M}\) and the true language distribution \(p_{L}\). From this, we obtain a set of probability estimates: \(\mathcal{S}_{\text{test}}=\left\{\left.p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle \right.\mathbf{x}\in\mathcal{D}_{\text{test}}\right\}\). If the model \(M\) perfectly models the language \(L\), then for each \(\left\langle p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle\in\mathcal{S}_{\text{test}}\) we would expect \(p_{L}(\mathbf{x})=p_{M}(\mathbf{x})\).

Our previous experiment trained languages models on a set of 1M sequences. A plausible explanation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not seen enough samples from the target distribution. Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

Relative change in estimation error eventually fluctuates around 0 (minimal change) for the majority of the distribution. Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution. In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

Under the assumption that \(p_{M}\) is a proper probability distribution,5 that is, \(\sum_{\mathbf{x}\in\Sigma^{*}}p_{M}(\mathbf{x})=1\), these findings suggest that there exists sequences in \(\Sigma^{*}\) whose probability is overestimated by the model.

We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\). Note that we use as LM the GPT2-medium model from the previous section (trained on 30M sequences). Up to this point, the target artificial language \(p_{L}\) was given as the distribution induced by an ancestral sampling scheme with softmax \(T=0.85\) from the generative model \(L\).

Here we modulate the shape of the sequence space defined by \(p_{L}\) to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that \(p_{L}\) allocates probability mass over \(\Sigma^{*}\), we control the entropy of the conditional distributions at each generation step \(t\) by dividing the pre-softmax logits by a temperature value \(T\).6Footnote 6:

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages.

These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across \(\Sigma^{*}\).

We take a more fine-grained approach, proposing a methodology which draws off of instance-level evaluation schemes (Zhong et al., 2021) and the experimental control afforded by artificial corpora (White and Cotterell, 2021; Papadimitriou and Jurafsky, 2020). Indeed, closely related to our work is Kulikov et al. (2021)'s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling.

We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMs on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language.

That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions, and we expect the task of recovering a ground-truth distribution to be made easier when the target distribution and LM are in the same model class.

In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWebText corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine-tuned on 1.5M sequences from the OpenWebText corpus.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed).

We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

To provide further evidence that these results hold for other values of \(T\), we conduct similar experiments as in Section 4.3 with an artificial language \(p_{L}\) defined by an ancestral sampling scheme with \(T=1.00\). Specifically, we train GPT2-medium and an GPT2-large on a total of 30M sequences sampled from \(p_{L}\), and we compute model estimation error on a set of withheld test sequences at each training iteration.

As an alternative way to explore how underestimation varies with increasing data, we fine-tune Huggface's pre-trained GPT2-small, -medium and -large models on the set of 1M sequences used in Section 4.1. Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\). Autoregressive neural language models decompose the joint distribution \(p(\mathbf{x})\) over sequences into a series of conditional distributions \(p(x_{i}\mid x_{<i})\).

Suppose we have a set of \(N\) events \(\mathcal{D}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\) drawn from some generative process \(\phi\). Given \(\mathcal{D}\), we aim to obtain an empirical estimate for the potential productivity \(\hat{\mathcal{P}}_{N}\): the amount of probability allocated to unseen events as a function of \(N\). We can do so using the Good-Turing estimate for the probability of an event given its frequency (Good, 1953). Specifically, let \(f(\mathbf{x},\mathcal{D})\) be a function which returns the frequency of the event \(\mathbf{x}\) in \(\mathcal{D}\). Let \(N_{m}\) denote the number of types (unique events) in \(\mathcal{D}\) for which \(f(\mathbf{x},\mathcal{D})=m\). Good-Turing says that for large \(N\), the probability of the event \(\mathbf{x}\) given that it has occurred with frequency \(m\) in our sample \(\mathcal{D}\) of size \(N\) is equal to \[\hat{P}(\mathbf{x}\mid f(\mathbf{x},\mathcal{D})=m)=\frac{(m+1)}{N}\frac{N_{m+1}}{N_{m}} \tag{3}\]

We apply this method to a subset of OpenWebText, a popular language modeling corpus. In Figure 13, we plot the the empirical estimate of \(\mathcal{P}_{N}\) as function of the sample size \(N\) for \(n\)-grams sampled from a subset of OpenWebText. Particularly for \(n\)-grams with \(n\geq 3\), we find that there is significant probability of sampling a previously unseen event, even for \(N>10,000,000\).

While we include the perplexity values for our models on sentences of English, this is not to claim that our ground-truth models are meant to define a distribution which closely resembles the distribution of English. \begin{table}
\begin{tabular}{l l} \hline \hline \(\mathbf{x}\) & \(\log p_{L}(\mathbf{x})\) \\ \hline “we’re very excited to have the opportunity to help them,” he says. & \(-29.3811\) \\ so what’s going to happen? & \(-17.4128\) \\ to me, the fisheries are in the midst of a global financial crisis. & \(-41.4835\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Sample of sequences drawn from our artificial language.

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Table 2: Sample of sequences drawn from our artificial language.

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Table 4: PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng).

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input: Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 0.70.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the claim is well-stated and accurately reflects what the researchers actually did. The evidence clearly shows they used a Transformer LM (GPT2-medium) trained on natural data (OpenWebText corpus) to create artificial languages where exact sequence probabilities could be computed. The methodology is thoroughly documented with specific training details, sampling procedures, and experimental validation. The claim doesn't overstate the approach or findings, and directly aligns with the implemented methodology.

<score>1</score>

Justification:
- The evidence directly supports the exact process described in the claim
- Training details are explicitly documented: "train a randomly-initialized GPT2-medium on 1.5M sentences sampled from the OpenWebText corpus"
- They demonstrate ability to compute exact probabilities through tables showing log p_L(x) values for sampled sequences
- The claim is precise and measured in its scope, avoiding unsupported generalizations
- Multiple experiments validate that they could indeed use this approach to compute exact sequence probabilities


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Since the true distribution of any natural language is in practice unknown, we use a Transformer LM trained on natural data as a generative model to define target artificial languages for which we can exactly compute sequence probabilities.

The review comment to be evaluated is:

* Strengths: A very well written and clear paper, addressing a focussed question through a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions.

* Weaknesses: No major weakness, but some mild issues and areas for improvement, see the comments and questions below.

* Comments and Questions:

    * (Minor Comment) I did not find the description of the LNRE Zone in section 2 to bring much to the discussion. In particular, characterizing this zone through the condition that $\mathcal{P}_N$ is not __null__ is almost empty of content for natural language, because it would in particular mean that we could precisely delimit what constitutes a possible "natural" sentence, and that the LNRE Zone should cover all such sentences. Here the formal description as given does not clarify anything.
    * (Question) In some of your experiments, you train a $P_M$ GPT2-medium model on 30M sentences sampled from the $P_L$ GPT2-medium model, that is, $P_M$ here uses the same architecture as $P_L$ (if I understand correctly). However, the $P_M$ model still underestimates the probabilities relative to $P_L$. Do you think this is a symptom of 30M sentences being too few, or more a symptom that $P_M$ is stuck in some local minimum during the optimization ? After all, (again, if I understand the setup correctly), using the same architecture, $P_M$ would make it possible, in principle, to just use the same parameters as $P_L$, and then the mismatch could not occur. Perhaps this would be worth a note in the text?
    * (More important Question/Comment) In section 4.4, you perform some perturbations, and observe that they lead to higher $P_M$ than $P_L$. First, I think it would be worth defining these perturbations in more detail than what you do. Second, you do not discuss your choice for these perturbations nor analyze why they have (relatively) high probability relative to $P_M$. However, the fact that, for instance, swapping two words or deleting a word in a natural sentence is not severely penalized by $P_M$ is an interesting but specific fact, which may have to do with a special tolerance of $P_M$ for such operations, rather than for others. To give one example, if the perturbations consisted solely in replacing a word in the natural sentence by a _rare_ word in the training set, would you observe such a difference between $P_M$ and $P_L$? Overall, could you discuss this aspect of your experiments?

The evidence to be evaluated is:In the context of language production, a language user has the ability to produce, at any given point in their linguistic lifespan, an utterance which they have never produced before. This creativity is the result of the generative property of productivity, which states that on the basis of finite linguistic experience, a language user can produce and comprehend an unbounded number of grammatically acceptable utterances (Chomsky, 1975, 1955). Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes. Because of this property, many of the distributions in natural language--particularly the distribution over the sequences of a language--are characterized by a heavy-tail of rare events.

A model \(M\) of the language \(L\) attempts to define a distribution \(p_{M}\) which closely resembles the true distribution of the language \(p_{L}\). In a locally-normalized autoregressive model, this distribution is defined by assigning probabilities to variable length sequences \(\mathbf{x}\) via a chain-rule decomposition:

For \(p_{M}\) to perfectly approximate \(p_{L}\), we expect \(p_{M}(\mathbf{x})=p_{L}(\mathbf{x})\) for all \(\mathbf{x}\in\Sigma^{*}\), where \(\Sigma^{*}\) is the set of all strings of finite length (the Kleene closure of \(\Sigma\)).

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\). However, since \(p_{\text{English}}\) and it's underlying generative process are unknown, it is not possible to explicitly evaluate how closely instance-level probability estimates align.

Specifically, we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model \(L\). The distribution over sequences induced by a sampling scheme from \(L\), denoted \(p_{L}\), is then our artificial language.

To define a generative model \(L\), we train a randomly-initialized GPT2-medium on 1.5M sentences sampled from the OpenWebText corpus (Gokaslan and Cohen, 2019).2

Using this generative model \(L\), we define the target distribution over sequences--the artificial language--as the distribution induced by an ancestral sampling scheme from \(L\). Thus, we draw instances \(\mathbf{x}=(x_{1},\dots,x_{|\mathbf{x}|})\) from our language \(p_{L}\) by recursively sampling from the conditional distribution over tokens at each time step: \(x_{t}\sim p_{L}(\cdot\mid x_{<t})\) where \(x_{1}=\textsc{BOS}\) and \(x_{t}=\textsc{EOS}\). All experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from \(L\) with softmax temperature \(T=0.85\).3 In Section 4.5, we explore the effects of different values of \(T\) when sampling from \(p_{L}\). Table 2 shows three sequences sampled from this distribution.

Given an artificial language \(p_{L}\), the task of \(M\)--the language model--is to define a distribution \(p_{M}\) whose sequence-level probability estimates closely align with the sequence-level probabilities given by \(p_{L}\).

In practice, we train \(M\) on a set of sequences \(\mathcal{D}_{\text{train}}\) sampled from \(p_{L}\), and compute model estimation error on a separate set of sequences \(\mathcal{D}_{\text{test}}\) sampled from \(p_{L}\).

Since these models are in the same model class as our artificial target language \(p_{L}\), we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown.

Following training, we sample a test set \(\mathcal{D}_{\text{test}}\) of 500,000 sequences from \(p_{L}\), and score each sequence under both the model distribution \(p_{M}\) and the true language distribution \(p_{L}\). From this, we obtain a set of probability estimates: \(\mathcal{S}_{\text{test}}=\left\{\left.p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle \right.\mathbf{x}\in\mathcal{D}_{\text{test}}\right\}\). If the model \(M\) perfectly models the language \(L\), then for each \(\left\langle p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle\in\mathcal{S}_{\text{test}}\) we would expect \(p_{L}(\mathbf{x})=p_{M}(\mathbf{x})\).

Our previous experiment trained languages models on a set of 1M sequences. A plausible explanation for the model's underestimation behaviour on unseen test sequences is therefore that the language model has not seen enough samples from the target distribution. Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

Relative change in estimation error eventually fluctuates around 0 (minimal change) for the majority of the distribution. Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution. In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

Under the assumption that \(p_{M}\) is a proper probability distribution,5 that is, \(\sum_{\mathbf{x}\in\Sigma^{*}}p_{M}(\mathbf{x})=1\), these findings suggest that there exists sequences in \(\Sigma^{*}\) whose probability is overestimated by the model.

We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\). Note that we use as LM the GPT2-medium model from the previous section (trained on 30M sequences). Up to this point, the target artificial language \(p_{L}\) was given as the distribution induced by an ancestral sampling scheme with softmax \(T=0.85\) from the generative model \(L\).

Here we modulate the shape of the sequence space defined by \(p_{L}\) to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that \(p_{L}\) allocates probability mass over \(\Sigma^{*}\), we control the entropy of the conditional distributions at each generation step \(t\) by dividing the pre-softmax logits by a temperature value \(T\).6Footnote 6:

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages.

These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across \(\Sigma^{*}\).

We take a more fine-grained approach, proposing a methodology which draws off of instance-level evaluation schemes (Zhong et al., 2021) and the experimental control afforded by artificial corpora (White and Cotterell, 2021; Papadimitriou and Jurafsky, 2020). Indeed, closely related to our work is Kulikov et al. (2021)'s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling.

We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMs on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language.

That being said, our experiments in Section 4.5 suggest that it may actually be easier for neural LMs to learn less structured distributions, and we expect the task of recovering a ground-truth distribution to be made easier when the target distribution and LM are in the same model class.

In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWebText corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine-tuned on 1.5M sequences from the OpenWebText corpus.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed).

We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

To provide further evidence that these results hold for other values of \(T\), we conduct similar experiments as in Section 4.3 with an artificial language \(p_{L}\) defined by an ancestral sampling scheme with \(T=1.00\). Specifically, we train GPT2-medium and an GPT2-large on a total of 30M sequences sampled from \(p_{L}\), and we compute model estimation error on a set of withheld test sequences at each training iteration.

As an alternative way to explore how underestimation varies with increasing data, we fine-tune Huggface's pre-trained GPT2-small, -medium and -large models on the set of 1M sequences used in Section 4.1. Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\). Autoregressive neural language models decompose the joint distribution \(p(\mathbf{x})\) over sequences into a series of conditional distributions \(p(x_{i}\mid x_{<i})\).

Suppose we have a set of \(N\) events \(\mathcal{D}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\) drawn from some generative process \(\phi\). Given \(\mathcal{D}\), we aim to obtain an empirical estimate for the potential productivity \(\hat{\mathcal{P}}_{N}\): the amount of probability allocated to unseen events as a function of \(N\). We can do so using the Good-Turing estimate for the probability of an event given its frequency (Good, 1953). Specifically, let \(f(\mathbf{x},\mathcal{D})\) be a function which returns the frequency of the event \(\mathbf{x}\) in \(\mathcal{D}\). Let \(N_{m}\) denote the number of types (unique events) in \(\mathcal{D}\) for which \(f(\mathbf{x},\mathcal{D})=m\). Good-Turing says that for large \(N\), the probability of the event \(\mathbf{x}\) given that it has occurred with frequency \(m\) in our sample \(\mathcal{D}\) of size \(N\) is equal to \[\hat{P}(\mathbf{x}\mid f(\mathbf{x},\mathcal{D})=m)=\frac{(m+1)}{N}\frac{N_{m+1}}{N_{m}} \tag{3}\]

We apply this method to a subset of OpenWebText, a popular language modeling corpus. In Figure 13, we plot the the empirical estimate of \(\mathcal{P}_{N}\) as function of the sample size \(N\) for \(n\)-grams sampled from a subset of OpenWebText. Particularly for \(n\)-grams with \(n\geq 3\), we find that there is significant probability of sampling a previously unseen event, even for \(N>10,000,000\).

While we include the perplexity values for our models on sentences of English, this is not to claim that our ground-truth models are meant to define a distribution which closely resembles the distribution of English. \begin{table}
\begin{tabular}{l l} \hline \hline \(\mathbf{x}\) & \(\log p_{L}(\mathbf{x})\) \\ \hline “we’re very excited to have the opportunity to help them,” he says. & \(-29.3811\) \\ so what’s going to happen? & \(-17.4128\) \\ to me, the fisheries are in the midst of a global financial crisis. & \(-41.4835\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Sample of sequences drawn from our artificial language.

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Table 2: Sample of sequences drawn from our artificial language.

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Table 4: PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng).

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input: Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 0.70.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim that they "use a Transformer LM trained on natural data as a generative model to define target artificial languages for which we can exactly compute sequence probabilities" is well-stated and accurately reflects the evidence. The paper clearly demonstrates:

1. They train GPT2-medium on OpenWebText data to create model L
2. Use this model to generate artificial language samples with known probabilities
3. Successfully compute exact sequence probabilities from this model
4. Use this setup to evaluate other language models' probability estimates

The evidence directly supports all aspects of the claim without exaggeration. The reviewer comments also validate this approach, calling it "well-designed" with "precisely controlled experimental conditions."

<score>1</score>