Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime which is critical in legal settings.

The evidence to be evaluated is:However, more meaningful violations of privacy occur at a low FPR. Methods that optimize solely for AUC can overstress the importance of high TPR at high FPR, a regime inherently protected by plausible deniability. In our work, we report both AUC and numerical results at the FPR deemed acceptable by Carlini et al. (2022a) for ease of comparison.

After retrieving the scaled scores for IN and OUT models, the attacker fits them to two separate Gaussian distributions denoted \(\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{2})\) and \(\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}}^{2})\) respectively. Then, the attacker queries the target model with \((x^{*},y^{*})\) and computes the scaled confidence score of the target model \(\mathrm{conf_{t}}=\phi(f_{\theta}(x^{*})_{y^{*}})\). Finally, the probability of \((x^{*},y^{*})\) being in the training data of \(\theta_{\text{t}}\) is calculated as: \[\frac{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{ 2}))}{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}} ^{2}))}, \tag{2}\] where \(p(\mathrm{conf}\mid\mathcal{N}(\mu,\sigma^{2}))\) calculates the probability of conf under \(\mathcal{N}(\mu,\sigma^{2})\). For the **offline** threat model, the attacker exclusively produces OUT shadow models by training on a set of randomized datasets fully disjoint from the possible sensitive data.

In the offline case, we remove line \(8\) and only use OUT models during the optimization.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

Despite these challenges, we still see improvement over LiRA in the offline case - the AUC score is close to random (\(50.82\%\)) for LiRA here and Canary attacks can improve this to \(54.61\%\).

In Table 2, Canary attacks are able to consistently provide enhancement over different models.

Indeed, we see from Table 2 that models with higher reproducibility do correlate with more improvement for the online scenario, especially between WRN28-10 and ConvMixer, where WRN28-10 has higher reproducibility than ConvMixer (Sompealli et al., 2022).

We see from Figure 2(a), that using more shadow models yields a higher true positive rate when the false positive rate is low. Interestingly, as the number of shadow models initially decreases, the overall performance drops slightly, but such an effect diminishes after the number of shadow models is greater than \(2^{4}\).

As shown in Figure 2(c), the performance of Canary drops very fast after \(\varepsilon=2\). When \(\varepsilon=1\) the TPR\(@1\%\)FPT is slightly lower than when \(\varepsilon=2\), which indicates that the perturbation within \(\varepsilon=1\) might be too small to be effective.

Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

However, in the offline scenario, only CW/CW and pre-softmax logits provide improvements to TPR\(@1\%\)FPR.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample. We also investigate and discuss strategies to make the queries trained on the shadow models transferable to the target model. Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Table 2: Results on Different Models Architecture. Canary attacks are able to consistently outperform LiRA over different models. T@1%F stands for TPR@1%FPR.

Input: Figure 2: Hyperparameter Ablation Experiments. We provide ablation experiments on several crucial hyperparameters: number of shadow models, number of adversarial queries, ε bound, and batch size.

Input: Table 5: Comparison with Random Noise Perturbations. A random noise perturbation in the same ε-ball as the canary does not increase membership success. In this sense, the optimized behavior of the canary attack is crucial.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be moderately overstated. While the evidence shows consistent improvements over LiRA across different models and datasets, particularly in offline scenarios, the magnitude of improvements varies significantly. For instance, in offline scenarios, AUC improvements range from +6.14% to +15.59%, and TPR@1%FPR improvements range from +1.97% to +5.39%. While these are meaningful improvements, describing them as "significantly more accurate" is somewhat exaggerated, especially given some modest gains in certain cases (e.g., online scenarios showing improvements as low as +0.39% TPR@1%FPR for VGG).

<score>3</score>

Justification:
1. The claim's strength:
- The evidence supports that improvements are achieved, particularly in offline scenarios
- Table 2 demonstrates consistent outperformance across different models
- The improvements are especially notable in low false-positive regimes (TPR@1%FPR)

2. Overstatement concerns:
- The term "significantly more accurate" overstates the variable improvements seen across different scenarios
- Some improvements are modest (e.g., +0.42% AUC, +0.47% TPR@1%FPR for ConvMixer in online setting)
- Performance drops rapidly with certain parameters (e.g., ε > 2, as shown in Figure 2(c))
- Random noise comparison (Table 5) shows improvement is specific to optimized attacks, suggesting limitations to generalizability

The claim would be better stated by specifying the range of improvements observed and acknowledging the variability across different scenarios and models.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime which is critical in legal settings.

The review comment to be evaluated is:

Strengths:
1. The idea of using adversarial attacks to optimize the MIA queries is quite novel.
2. Following the recent research in MIAs, the paper uses both AUC and FPR to measure the privacy threats.
3. The Canary attack outperforms the state-of-the-art LiRA attack.
4. The paper investigated practical attack scenarios in online and offline threat models. The effectiveness of the proposed Canary attack has been demonstrated in both threat models.
5. The paper investigates the impact of several critical hyperparameters in Canary attacks.
6. The paper is well-written.

Weaknesses:
1. The optimization process of the query generation introduces additional computational overhead. The attack performance is also related to the number of queries. It would be great to compare the computational cost with LiRA attack. In addition, how do we select perturbation bound ϵ in practice? Does the selection require additional cost?
2. The evaluation only considers differential privacy for privacy protection. The paper could discuss the performance of Canary on other state-of-the-art defenses.

The evidence to be evaluated is:However, more meaningful violations of privacy occur at a low FPR. Methods that optimize solely for AUC can overstress the importance of high TPR at high FPR, a regime inherently protected by plausible deniability. In our work, we report both AUC and numerical results at the FPR deemed acceptable by Carlini et al. (2022a) for ease of comparison.

After retrieving the scaled scores for IN and OUT models, the attacker fits them to two separate Gaussian distributions denoted \(\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{2})\) and \(\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}}^{2})\) respectively. Then, the attacker queries the target model with \((x^{*},y^{*})\) and computes the scaled confidence score of the target model \(\mathrm{conf_{t}}=\phi(f_{\theta}(x^{*})_{y^{*}})\). Finally, the probability of \((x^{*},y^{*})\) being in the training data of \(\theta_{\text{t}}\) is calculated as: \[\frac{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{ 2}))}{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}} ^{2}))}, \tag{2}\] where \(p(\mathrm{conf}\mid\mathcal{N}(\mu,\sigma^{2}))\) calculates the probability of conf under \(\mathcal{N}(\mu,\sigma^{2})\). For the **offline** threat model, the attacker exclusively produces OUT shadow models by training on a set of randomized datasets fully disjoint from the possible sensitive data.

In the offline case, we remove line \(8\) and only use OUT models during the optimization.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

Despite these challenges, we still see improvement over LiRA in the offline case - the AUC score is close to random (\(50.82\%\)) for LiRA here and Canary attacks can improve this to \(54.61\%\).

In Table 2, Canary attacks are able to consistently provide enhancement over different models.

Indeed, we see from Table 2 that models with higher reproducibility do correlate with more improvement for the online scenario, especially between WRN28-10 and ConvMixer, where WRN28-10 has higher reproducibility than ConvMixer (Sompealli et al., 2022).

We see from Figure 2(a), that using more shadow models yields a higher true positive rate when the false positive rate is low. Interestingly, as the number of shadow models initially decreases, the overall performance drops slightly, but such an effect diminishes after the number of shadow models is greater than \(2^{4}\).

As shown in Figure 2(c), the performance of Canary drops very fast after \(\varepsilon=2\). When \(\varepsilon=1\) the TPR\(@1\%\)FPT is slightly lower than when \(\varepsilon=2\), which indicates that the perturbation within \(\varepsilon=1\) might be too small to be effective.

Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

However, in the offline scenario, only CW/CW and pre-softmax logits provide improvements to TPR\(@1\%\)FPR.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample. We also investigate and discuss strategies to make the queries trained on the shadow models transferable to the target model. Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Table 2: Results on Different Models Architecture. Canary attacks are able to consistently outperform LiRA over different models. T@1%F stands for TPR@1%FPR.

Input: Figure 2: Hyperparameter Ablation Experiments. We provide ablation experiments on several crucial hyperparameters: number of shadow models, number of adversarial queries, ε bound, and batch size.

Input: Table 5: Comparison with Random Noise Perturbations. A random noise perturbation in the same ε-ball as the canary does not increase membership success. In this sense, the optimized behavior of the canary attack is crucial.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find the claim "Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime" to be mostly well-stated but slightly overstated.

The evidence shows consistent improvements over LiRA across different models and datasets, particularly in offline scenarios (Table 2 shows improvements of +6.14% to +15.59% AUC). For the low false-positive regime (TPR@1%FPR), improvements are demonstrated but more modest (+1.97% to +5.39% in offline cases). While "significantly more accurate" is supported by the offline scenario data, the magnitude of improvement in the low false-positive regime, while consistent, might not warrant the strong emphasis implied by "especially."

<score>2</score>

Justification:
1. The claim is largely supported by empirical evidence, especially for offline scenarios
2. Tables and figures demonstrate consistent improvements across different models and datasets
3. The emphasis on "especially" for low false-positive regime might slightly overstate the magnitude of improvement in that specific area
4. The review comments support the method's effectiveness while noting some limitations regarding computational overhead and limited defense evaluations
5. The improvement is consistent but varies in magnitude across different scenarios


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime which is critical in legal settings.

The review comment to be evaluated is:

Membership inference attacks are of great interest to the privacy and security community, and the paper proposes a method to improve LiRA by learning the query vectors. I find the proposed method interesting, and of potential interest to the community. Paper is clearly written and is easy to understand.

My only comment is that it would be interesting to see the metrics with more stringent privacy budget (for differential privacy), example: $\varepsilon \in [10,20,\cdots,100]$, it will also add another practical dimension to the paper, proving strong DP protections against membership inference.

The evidence to be evaluated is:However, more meaningful violations of privacy occur at a low FPR. Methods that optimize solely for AUC can overstress the importance of high TPR at high FPR, a regime inherently protected by plausible deniability. In our work, we report both AUC and numerical results at the FPR deemed acceptable by Carlini et al. (2022a) for ease of comparison.

After retrieving the scaled scores for IN and OUT models, the attacker fits them to two separate Gaussian distributions denoted \(\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{2})\) and \(\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}}^{2})\) respectively. Then, the attacker queries the target model with \((x^{*},y^{*})\) and computes the scaled confidence score of the target model \(\mathrm{conf_{t}}=\phi(f_{\theta}(x^{*})_{y^{*}})\). Finally, the probability of \((x^{*},y^{*})\) being in the training data of \(\theta_{\text{t}}\) is calculated as: \[\frac{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{ 2}))}{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}} ^{2}))}, \tag{2}\] where \(p(\mathrm{conf}\mid\mathcal{N}(\mu,\sigma^{2}))\) calculates the probability of conf under \(\mathcal{N}(\mu,\sigma^{2})\). For the **offline** threat model, the attacker exclusively produces OUT shadow models by training on a set of randomized datasets fully disjoint from the possible sensitive data.

In the offline case, we remove line \(8\) and only use OUT models during the optimization.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

Despite these challenges, we still see improvement over LiRA in the offline case - the AUC score is close to random (\(50.82\%\)) for LiRA here and Canary attacks can improve this to \(54.61\%\).

In Table 2, Canary attacks are able to consistently provide enhancement over different models.

Indeed, we see from Table 2 that models with higher reproducibility do correlate with more improvement for the online scenario, especially between WRN28-10 and ConvMixer, where WRN28-10 has higher reproducibility than ConvMixer (Sompealli et al., 2022).

We see from Figure 2(a), that using more shadow models yields a higher true positive rate when the false positive rate is low. Interestingly, as the number of shadow models initially decreases, the overall performance drops slightly, but such an effect diminishes after the number of shadow models is greater than \(2^{4}\).

As shown in Figure 2(c), the performance of Canary drops very fast after \(\varepsilon=2\). When \(\varepsilon=1\) the TPR\(@1\%\)FPT is slightly lower than when \(\varepsilon=2\), which indicates that the perturbation within \(\varepsilon=1\) might be too small to be effective.

Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

However, in the offline scenario, only CW/CW and pre-softmax logits provide improvements to TPR\(@1\%\)FPR.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample. We also investigate and discuss strategies to make the queries trained on the shadow models transferable to the target model. Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Table 2: Results on Different Models Architecture. Canary attacks are able to consistently outperform LiRA over different models. T@1%F stands for TPR@1%FPR.

Input: Figure 2: Hyperparameter Ablation Experiments. We provide ablation experiments on several crucial hyperparameters: number of shadow models, number of adversarial queries, ε bound, and batch size.

Input: Table 5: Comparison with Random Noise Perturbations. A random noise perturbation in the same ε-ball as the canary does not increase membership success. In this sense, the optimized behavior of the canary attack is crucial.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find the claim is moderately overstated. While the evidence shows consistent improvements over LiRA across different models and datasets, particularly in offline scenarios, the magnitude of improvement varies significantly. For example, offline improvements range from +1.97% to +15.59% for TPR@1%FPR across different models. The term "significantly more accurate" is somewhat exaggerated given that some improvements are modest (e.g., online improvements as low as +0.39% TPR@1%FPR for VGG). Additionally, while results improve in low false-positive regimes, the effectiveness drops substantially with stronger privacy protections (ε=2 and lower).

<score>3</score>

Justification:
1. The claim of "significantly more accurate" is partially supported:
- Offline scenarios show notable improvements (Table 2: AUC improvements of +6.14% to +15.59%)
- However, online improvements are more modest (Table 2: AUC improvements of +0.42% to +1.89%)

2. Performance in "low false-positive regime":
- Shows consistent improvements in TPR@1%FPR
- But improvements vary widely (from +0.39% to +5.39% offline)
- Performance deteriorates significantly with stronger privacy settings (Figure 2(c))

3. The reviewer's comment about testing with more stringent privacy budgets suggests potential limitations not fully acknowledged in the claim.

4. Random noise comparison (Table 5) validates the method's effectiveness, but the magnitude of improvement (+2.3% TPR@1%FPR) doesn't fully support "significantly more accurate."


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime which is critical in legal settings.

The review comment to be evaluated is:

Strong point:

Perturbing the target data point to boost the membership inference precision is a novel and interesting idea. The core idea behind is to make use of the perturbed variant of the target data point to facilitate the attack. Even though the loss of the target data point may hardly vary if the target data point is considered to be an in- or out-class sample, the perturbed variants within the epsilon-neighbourhood of the target data point can potentially unveil more information to identify the membership.

Weakness:
1. In Table.2, the AUC and the TPR values (given the FPR level) of membership inference attack do not provide a monotonic decreasing as the reproductivity of the target model decreases (as claimed in the paper, the target models in Table.2 are given in a descending order of the model reproductivity.)

2. On page 8, why does the attack performance become worse if the bound epsilon becomes larger or smaller than 2? Any explanation to the observation?  Is there any potential trade-off related to the choice of epsilon ?

3. More theoretical study regarding the effectiveness of the adversarial query could further consolidate this work. The adversarial perturbation added to the target data point may help show the loss distribution within the epsilon-neighborhood of the target data point. Though the loss on the target data point may not be rich enough for a successful membership inference attack, the difference of the loss distribution may provide a better lens to differentiate two different membership classes. For the moment, the reasonability behind Eq.3 is still not very clear.

The evidence to be evaluated is:However, more meaningful violations of privacy occur at a low FPR. Methods that optimize solely for AUC can overstress the importance of high TPR at high FPR, a regime inherently protected by plausible deniability. In our work, we report both AUC and numerical results at the FPR deemed acceptable by Carlini et al. (2022a) for ease of comparison.

After retrieving the scaled scores for IN and OUT models, the attacker fits them to two separate Gaussian distributions denoted \(\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{2})\) and \(\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}}^{2})\) respectively. Then, the attacker queries the target model with \((x^{*},y^{*})\) and computes the scaled confidence score of the target model \(\mathrm{conf_{t}}=\phi(f_{\theta}(x^{*})_{y^{*}})\). Finally, the probability of \((x^{*},y^{*})\) being in the training data of \(\theta_{\text{t}}\) is calculated as: \[\frac{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{in}},\sigma_{\text{in}}^{ 2}))}{p(\mathrm{conf_{t}}\mid\mathcal{N}(\mu_{\text{out}},\sigma_{\text{out}} ^{2}))}, \tag{2}\] where \(p(\mathrm{conf}\mid\mathcal{N}(\mu,\sigma^{2}))\) calculates the probability of conf under \(\mathcal{N}(\mu,\sigma^{2})\). For the **offline** threat model, the attacker exclusively produces OUT shadow models by training on a set of randomized datasets fully disjoint from the possible sensitive data.

In the offline case, we remove line \(8\) and only use OUT models during the optimization.

In this section, we first show that the Canary attack can reliably improve LiRA results under different datasets and different models for both online and offline settings.

We follow the setting of Carlini et al. (2022) for our main experiment on CIFAR-10 and CIFAR-100 for full comparability. We first train \(65\) wide ResNets (WRN28-10) (Zagoruyko and Komodakis, 2016) with random even splits of \(50000\) images to reach \(92\%\) and \(71\%\) test accuracy for CIFAR-10 and CIFAR-100 respectively. For MNIST, we train \(65\)\(8\)-layer ResNets (He et al., 2016) with random even splits to reach \(97\%\) test accuracy.

The improvement of TPR\(@1\%\)FPR is significant for all datasets. The difference is especially notable for online CIFAR-10, where we achieve a \(4.14\%\) boost over the baseline LiRA (a relative improvement in TPR of \(23\%\)).

On average, Canary attacks are most powerful in the more realistic offline scenario. We gain over \(3\%\) boost on AUC scores on all datasets and over \(2.75\%\) TPR\(@1\%\)FPR boost for CIFAR-10 and CIFAR-100.

Despite these challenges, we still see improvement over LiRA in the offline case - the AUC score is close to random (\(50.82\%\)) for LiRA here and Canary attacks can improve this to \(54.61\%\).

In Table 2, Canary attacks are able to consistently provide enhancement over different models.

Indeed, we see from Table 2 that models with higher reproducibility do correlate with more improvement for the online scenario, especially between WRN28-10 and ConvMixer, where WRN28-10 has higher reproducibility than ConvMixer (Sompealli et al., 2022).

We see from Figure 2(a), that using more shadow models yields a higher true positive rate when the false positive rate is low. Interestingly, as the number of shadow models initially decreases, the overall performance drops slightly, but such an effect diminishes after the number of shadow models is greater than \(2^{4}\).

As shown in Figure 2(c), the performance of Canary drops very fast after \(\varepsilon=2\). When \(\varepsilon=1\) the TPR\(@1\%\)FPT is slightly lower than when \(\varepsilon=2\), which indicates that the perturbation within \(\varepsilon=1\) might be too small to be effective.

Surprisingly, minimizing/maximizing the pre-softmax logits gives us the biggest improvement, even though it does not explicitly suppress the logits for other labels like other objectives do.

However, in the offline scenario, only CW/CW and pre-softmax logits provide improvements to TPR\(@1\%\)FPR.

Differential privacy is still a very effective defense for membership inference attacks, but Canary attacks reliably increase the success chance of membership inference over LiRA. We explore a novel way to enhance membership inference techniques by creating ensembles of adversarial queries. These adversarial queries are optimized to provide maximally different outcomes for the model trained with/without the target data sample. We also investigate and discuss strategies to make the queries trained on the shadow models transferable to the target model. Through a series of experiments, we show that Canary attacks reliably enhance both online and offline membership inference algorithms under three different datasets, four different models, and differential privacy.

Input: Table 2: Results on Different Models Architecture. Canary attacks are able to consistently outperform LiRA over different models. T@1%F stands for TPR@1%FPR.

Input: Figure 2: Hyperparameter Ablation Experiments. We provide ablation experiments on several crucial hyperparameters: number of shadow models, number of adversarial queries, ε bound, and batch size.

Input: Table 5: Comparison with Random Noise Perturbations. A random noise perturbation in the same ε-ball as the canary does not increase membership success. In this sense, the optimized behavior of the canary attack is crucial.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the statement appears somewhat overstated. While the evidence shows improvements in both online and offline scenarios, the degree of "significantly more accurate" needs qualification:

1. The improvements vary considerably across different models and scenarios
2. In online settings, improvements are modest (e.g., +0.39% to +4.14% TPR@1%FPR)
3. In offline settings, while improvements are more substantial (+1.97% to +5.39% TPR@1%FPR), baseline performance is quite low
4. Reviewer points out non-monotonic behavior in results and lack of theoretical understanding
5. The claim doesn't acknowledge limitations or variability in performance

The evidence supports improvement over existing methods, but "significantly more accurate" overstates the consistency and magnitude of improvement, especially given the baseline performance levels.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution.

The evidence to be evaluated is:We begin by briefly characterizing why distributions with a large number of rare events (LNRE) emerge in natural language, and why these events pose challenges for LMs.

Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes. Because of this property, many of the distributions in natural language--particularly the distribution over the sequences of a language--are characterized by a heavy-tail of rare events.

The LNRE zone for natural language appears to be very large, and it seems likely that \(\mathcal{P}_{N}\) will remain greater than \(0\) for samples of natural language many orders of magnitude larger than all the data currently available for training LMs.1 In the LNRE zone, it is difficult to obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation (MLE). Accounting for this enormous amount of novelty is thus a central challenge in language modeling.

In the LNRE zone, \(p_{M}\) is defined over a support containing a very large set of sequences which have never occurred in a training corpus (or equivalently, have all occurred with frequency 0), and which take on a very wide array of differing probabilities. For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\):

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\). However, since \(p_{\text{English}}\) and it's underlying generative process are unknown, it is not possible to explicitly evaluate how closely instance-level probability estimates align. As a proxy, the mean perplexity of the model on a holdout set of sequences \(\mathcal{D}\) is typically used, which measures whether the model, on average, assigns high likelihood to unseen instances. This measure does not, however, tell us whether instance-level estimates align with their true counterparts, nor is it necessarily indicative of performance on rare, idiosyncratic events in \(\mathcal{D}\).

The following section introduces a methodology to overcome these limitations. We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences. Specifically, we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model \(L\). The distribution over sequences induced by a sampling scheme from \(L\), denoted \(p_{L}\), is then our artificial language. We expect a model \(M\) of this artificial language to assign probabilities \(p_{M}(\mathbf{x})\) to sequences \(\mathbf{x}\) which match the target probabilities \(p_{L}(\mathbf{x})\) of \(\mathbf{x}\) under \(p_{L}\). To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

Using this generative model \(L\), we define the target distribution over sequences--the artificial language--as the distribution induced by an ancestral sampling scheme from \(L\). Thus, we draw instances \(\mathbf{x}=(x_{1},\dots,x_{|\mathbf{x}|})\) from our language \(p_{L}\) by recursively sampling from the conditional distribution over tokens at each time step: \(x_{t}\sim p_{L}(\cdot\mid x_{<t})\) where \(x_{1}=\textsc{BOS}\) and \(x_{t}=\textsc{EOS}\). All experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from \(L\) with softmax temperature \(T=0.85\).3 In Section 4.5, we explore the effects of different values of \(T\) when sampling from \(p_{L}\).

Given an artificial language \(p_{L}\), the task of \(M\)--the language model--is to define a distribution \(p_{M}\) whose sequence-level probability estimates closely align with the sequence-level probabilities given by \(p_{L}\). We refer to any deviation from this desiderata as model estimation error. To quantify the model estimation error for a sequence \(\mathbf{x}\), we take the difference between the sequence's log probability under \(M\) and its true log probability under \(L\): \[\text{error}(\mathbf{x})=\log p_{M}(\mathbf{x})-\log p_{L}(\mathbf{x}) \tag{2}\] This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\). In practice, we train \(M\) on a set of sequences \(\mathcal{D}_{\text{train}}\) sampled from \(p_{L}\), and compute model estimation error on a separate set of sequences \(\mathcal{D}_{\text{test}}\) sampled from \(p_{L}\).

We study the estimation performance of two neural LM architectures: the Transformer (Vaswani et al., 2017) and the LSTM (Melis et al., 2020).

Since these models are in the same model class as our artificial target language \(p_{L}\), we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown.

We begin by exploring model estimation error on a fixed training set \(\mathcal{D}_{\text{train}}\) of 1M sequences sampled from \(p_{L}\).

Following training, we sample a test set \(\mathcal{D}_{\text{test}}\) of 500,000 sequences from \(p_{L}\), and score each sequence under both the model distribution \(p_{M}\) and the true language distribution \(p_{L}\). From this, we obtain a set of probability estimates: \(\mathcal{S}_{\text{test}}=\left\{\left.p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle \right.\mathbf{x}\in\mathcal{D}_{\text{test}}\right\}\). If the model \(M\) perfectly models the language \(L\), then for each \(\left\langle p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle\in\mathcal{S}_{\text{test}}\) we would expect \(p_{L}(\mathbf{x})=p_{M}(\mathbf{x})\). Figures 2(A) and 2(B) visualize this relationship with the \(x\)- and \(y\)-axes denoting the true and model estimated sequence probabilities respectively, and a dashed line representing equality. To compare probability estimates, we represent the set \(\mathcal{S}_{\text{test}}\) in the form of a joint histogram over this coordinate space. Histogram bins are shaded based on the number of tuples which lie in the coordinate range they define. Importantly, any deviation of this histogram from the identity line indicates that the model distorts the shape of the distribution of the language. To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\). Once computed, we sort each set of probability tuples \(\mathcal{S}_{\text{train}}^{(i)}\) and \(\mathcal{S}_{\text{test}}^{(i)}\) by their target sequence probabilities \(p_{L}(\mathbf{x})\), and split the probability tuples into 50 equally-sized bins. We plot estimation curves in Figure 3: Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

We again split test sequences by their true probability, with darker lines denoting estimation trends for less probable target sequences. The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution. In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\). Sequence perturbation operations are shown in Table 3.

We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

Here we modulate the shape of the sequence space defined by \(p_{L}\) to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that \(p_{L}\) allocates probability mass over \(\Sigma^{*}\), we control the entropy of the conditional distributions at each generation step \(t\) by dividing the pre-softmax logits by a temperature value \(T\).6Footnote 6: Formally, for the \(i\)th component of the length \(K\) pre-softmax logits \(\mathbf{x}\), this operation is given as: \(\texttt{softmax}(\mathbf{x})_{i}=\frac{\exp(\frac{\mathbf{x}}{t})}{\sum_{j=1}^{K}\exp \left(\frac{\mathbf{x}}{T}\right)}\)

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities. These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across \(\Sigma^{*}\).

We take a more fine-grained approach, proposing a methodology which draws off of instance-level evaluation schemes (Zhong et al., 2021) and the experimental control afforded by artificial corpora (White and Cotterell, 2021; Papadimitriou and Jurafsky, 2020). Indeed, closely related to our work is Kulikov et al. (2021)'s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling.

In this work, we introduce a controlled methodology to evaluate instance-level LM performance on this set of individually rare but collectively frequent events. We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMs on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language. Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data. Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution. Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy. Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWebText corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine-tuned on 1.5M sequences from the OpenWebText corpus.

Analogously to the experiment in Section 4.1, we train a randomly-initialized GPT2-medium model on 1M sequences sampled from the fine-tuned model. In the center of Figure 7, we visualize mean model estimation error for \(50,000\) test sequences as a function of true sequence probability. Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed). Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

To provide further evidence that these results hold for other values of \(T\), we conduct similar experiments as in Section 4.3 with an artificial language \(p_{L}\) defined by an ancestral sampling scheme with \(T=1.00\). Specifically, we train GPT2-medium and an GPT2-large on a total of 30M sequences sampled from \(p_{L}\), and we compute model estimation error on a set of withheld test sequences at each training iteration.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

Given this length, we sample \(l\) tokens from \(\Sigma^{*}\) and concatenate all tokens to form a sequence. We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\). Autoregressive neural language models decompose the joint distribution \(p(\mathbf{x})\) over sequences into a series of conditional distributions \(p(x_{i}\mid x_{<i})\). Generating a sequence of length \(n\), then, involves estimating \(n\) conditional distributions.

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length. This suggests that estimation error for longer (and typically rarer) sequences is not solely due to error compounding. In section 2, we formally defined the LNRE zone as the range of values of \(N\) for which there is non-zero probability of sampling a novel event at the \(N+1\)th draw. Here we introduce a frequentist estimator for this probability. This in turn allows us to empirically verify if a sample exists in the LNRE zone. Suppose we have a set of \(N\) events \(\mathcal{D}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\) drawn from some generative process \(\phi\). Given \(\mathcal{D}\), we aim to obtain an empirical estimate for the potential productivity \(\hat{\mathcal{P}}_{N}\): the amount of probability allocated to unseen events as a function of \(N\). We can do so using the Good-Turing estimate for the probability of an event given its frequency (Good, 1953). Specifically, let \(f(\mathbf{x},\mathcal{D})\) be a function which returns the frequency of the event \(\mathbf{x}\) in \(\mathcal{D}\). Let \(N_{m}\) denote the number of types (unique events) in \(\mathcal{D}\) for which \(f(\mathbf{x},\mathcal{D})=m\). Good-Turing says that for large \(N\), the probability of the event \(\mathbf{x}\) given that it has occurred with frequency \(m\) in our sample \(\mathcal{D}\) of size \(N\) is equal to \[\hat{P}(\mathbf{x}\mid f(\mathbf{x},\mathcal{D})=m)=\frac{(m+1)}{N}\frac{N_{m+1}}{N_{m}} \tag{3}\]

We apply this method to a subset of OpenWebText, a popular language modeling corpus. In Figure 13, we plot the the empirical estimate of \(\mathcal{P}_{N}\) as function of the sample size \(N\) for \(n\)-grams sampled from a subset of OpenWebText. Particularly for \(n\)-grams with \(n\geq 3\), we find that there is significant probability of sampling a previously unseen event, even for \(N>10,000,000\).

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 1: GPT2 sequence probability estimates plotted against the true sequence probabilities. Neural LMs underestimate the probability of sequences drawn from the language they are trained to model.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input: Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 0.70.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim is well-supported and accurately reflects what the paper presents. The authors clearly describe their methodology of using generative models trained on natural language corpora to create artificial languages where sequence probabilities can be exactly computed. The evidence shows extensive experiments comparing LM probability estimates to target probabilities, with detailed analysis across different architectures, training regimes, and perturbation tests. The methodology is thoroughly explained and controlled, matching what is claimed. The claim avoids overstating the scope or implications of their approach.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

In this work, we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution.

The review comment to be evaluated is:

* Strengths: A very well written and clear paper, addressing a focussed question through a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions.

* Weaknesses: No major weakness, but some mild issues and areas for improvement, see the comments and questions below.

* Comments and Questions:

    * (Minor Comment) I did not find the description of the LNRE Zone in section 2 to bring much to the discussion. In particular, characterizing this zone through the condition that $\mathcal{P}_N$ is not __null__ is almost empty of content for natural language, because it would in particular mean that we could precisely delimit what constitutes a possible "natural" sentence, and that the LNRE Zone should cover all such sentences. Here the formal description as given does not clarify anything.
    * (Question) In some of your experiments, you train a $P_M$ GPT2-medium model on 30M sentences sampled from the $P_L$ GPT2-medium model, that is, $P_M$ here uses the same architecture as $P_L$ (if I understand correctly). However, the $P_M$ model still underestimates the probabilities relative to $P_L$. Do you think this is a symptom of 30M sentences being too few, or more a symptom that $P_M$ is stuck in some local minimum during the optimization ? After all, (again, if I understand the setup correctly), using the same architecture, $P_M$ would make it possible, in principle, to just use the same parameters as $P_L$, and then the mismatch could not occur. Perhaps this would be worth a note in the text?
    * (More important Question/Comment) In section 4.4, you perform some perturbations, and observe that they lead to higher $P_M$ than $P_L$. First, I think it would be worth defining these perturbations in more detail than what you do. Second, you do not discuss your choice for these perturbations nor analyze why they have (relatively) high probability relative to $P_M$. However, the fact that, for instance, swapping two words or deleting a word in a natural sentence is not severely penalized by $P_M$ is an interesting but specific fact, which may have to do with a special tolerance of $P_M$ for such operations, rather than for others. To give one example, if the perturbations consisted solely in replacing a word in the natural sentence by a _rare_ word in the training set, would you observe such a difference between $P_M$ and $P_L$? Overall, could you discuss this aspect of your experiments?

The evidence to be evaluated is:We begin by briefly characterizing why distributions with a large number of rare events (LNRE) emerge in natural language, and why these events pose challenges for LMs.

Productive processes induce a distribution which places non-zero probability on unseen events at all practical sample sizes. Because of this property, many of the distributions in natural language--particularly the distribution over the sequences of a language--are characterized by a heavy-tail of rare events.

The LNRE zone for natural language appears to be very large, and it seems likely that \(\mathcal{P}_{N}\) will remain greater than \(0\) for samples of natural language many orders of magnitude larger than all the data currently available for training LMs.1 In the LNRE zone, it is difficult to obtain accurate estimates of the probability of events using straightforward maximum likelihood estimation (MLE). Accounting for this enormous amount of novelty is thus a central challenge in language modeling.

In the LNRE zone, \(p_{M}\) is defined over a support containing a very large set of sequences which have never occurred in a training corpus (or equivalently, have all occurred with frequency 0), and which take on a very wide array of differing probabilities. For example, while the sequences \(\mathbf{x}_{1},\mathbf{x}_{2}\in\Sigma^{*}\) have likely never occurred in any sample of English, most would agree that \(\mathbf{x}_{1}\) is far more probable than \(\mathbf{x}_{2}\):

For a perfect LM of English, we would expect the estimated probabilities of the sequences \(\mathbf{x}_{1}\) and \(\mathbf{x}_{2}\) to match their probabilities under the true distribution \(p_{\text{English}}\). However, since \(p_{\text{English}}\) and it's underlying generative process are unknown, it is not possible to explicitly evaluate how closely instance-level probability estimates align. As a proxy, the mean perplexity of the model on a holdout set of sequences \(\mathcal{D}\) is typically used, which measures whether the model, on average, assigns high likelihood to unseen instances. This measure does not, however, tell us whether instance-level estimates align with their true counterparts, nor is it necessarily indicative of performance on rare, idiosyncratic events in \(\mathcal{D}\).

The following section introduces a methodology to overcome these limitations. We propose evaluating language model performance on the heavy-tail of rare events via a known probability distribution over sequences. Specifically, we train a Transformer LM on sequences sampled from a corpus of natural language to define a generative model \(L\). The distribution over sequences induced by a sampling scheme from \(L\), denoted \(p_{L}\), is then our artificial language. We expect a model \(M\) of this artificial language to assign probabilities \(p_{M}(\mathbf{x})\) to sequences \(\mathbf{x}\) which match the target probabilities \(p_{L}(\mathbf{x})\) of \(\mathbf{x}\) under \(p_{L}\). To characterize neural LM behaviour on rare events, we train Transformer and LSTM LMs on data sampled from \(p_{L}\), and compare the instance-level probability estimates given by \(p_{M}\) to target probabilities under \(p_{L}\).

Using this generative model \(L\), we define the target distribution over sequences--the artificial language--as the distribution induced by an ancestral sampling scheme from \(L\). Thus, we draw instances \(\mathbf{x}=(x_{1},\dots,x_{|\mathbf{x}|})\) from our language \(p_{L}\) by recursively sampling from the conditional distribution over tokens at each time step: \(x_{t}\sim p_{L}(\cdot\mid x_{<t})\) where \(x_{1}=\textsc{BOS}\) and \(x_{t}=\textsc{EOS}\). All experiments up until Section 4.5 are conducted on the distribution induced by ancestrally sampling from \(L\) with softmax temperature \(T=0.85\).3 In Section 4.5, we explore the effects of different values of \(T\) when sampling from \(p_{L}\).

Given an artificial language \(p_{L}\), the task of \(M\)--the language model--is to define a distribution \(p_{M}\) whose sequence-level probability estimates closely align with the sequence-level probabilities given by \(p_{L}\). We refer to any deviation from this desiderata as model estimation error. To quantify the model estimation error for a sequence \(\mathbf{x}\), we take the difference between the sequence's log probability under \(M\) and its true log probability under \(L\): \[\text{error}(\mathbf{x})=\log p_{M}(\mathbf{x})-\log p_{L}(\mathbf{x}) \tag{2}\] This quantity is the log probability ratio, which measures, in log-space, the number of times more or less likely the sequence \(\mathbf{x}\) is under the language model \(M\). Note that \(\text{error}(\mathbf{x})<0\) indicates that \(M\) underestimates the probability of \(\mathbf{x}\), whereas \(\text{error}(\mathbf{x})>0\) indicates that \(M\) overestimates the probability of \(\mathbf{x}\). In practice, we train \(M\) on a set of sequences \(\mathcal{D}_{\text{train}}\) sampled from \(p_{L}\), and compute model estimation error on a separate set of sequences \(\mathcal{D}_{\text{test}}\) sampled from \(p_{L}\).

We study the estimation performance of two neural LM architectures: the Transformer (Vaswani et al., 2017) and the LSTM (Melis et al., 2020).

Since these models are in the same model class as our artificial target language \(p_{L}\), we expect the task of recovering the ground-truth distribution to be relatively easy compared to the true problem faced in modeling natural language, where both the distribution and the underlying generative process are unknown.

We begin by exploring model estimation error on a fixed training set \(\mathcal{D}_{\text{train}}\) of 1M sequences sampled from \(p_{L}\).

Following training, we sample a test set \(\mathcal{D}_{\text{test}}\) of 500,000 sequences from \(p_{L}\), and score each sequence under both the model distribution \(p_{M}\) and the true language distribution \(p_{L}\). From this, we obtain a set of probability estimates: \(\mathcal{S}_{\text{test}}=\left\{\left.p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle \right.\mathbf{x}\in\mathcal{D}_{\text{test}}\right\}\). If the model \(M\) perfectly models the language \(L\), then for each \(\left\langle p_{L}(\mathbf{x}),p_{M}(\mathbf{x})\right\rangle\in\mathcal{S}_{\text{test}}\) we would expect \(p_{L}(\mathbf{x})=p_{M}(\mathbf{x})\). Figures 2(A) and 2(B) visualize this relationship with the \(x\)- and \(y\)-axes denoting the true and model estimated sequence probabilities respectively, and a dashed line representing equality. To compare probability estimates, we represent the set \(\mathcal{S}_{\text{test}}\) in the form of a joint histogram over this coordinate space. Histogram bins are shaded based on the number of tuples which lie in the coordinate range they define. Importantly, any deviation of this histogram from the identity line indicates that the model distorts the shape of the distribution of the language. To understand the training dynamics underlying the previously reported underestimation, we compute model probability estimates on subsets of \(\mathcal{D}_{\text{train}}\) and \(\mathcal{D}_{\text{test}}\) at the end of each training iteration \(i\). Once computed, we sort each set of probability tuples \(\mathcal{S}_{\text{train}}^{(i)}\) and \(\mathcal{S}_{\text{test}}^{(i)}\) by their target sequence probabilities \(p_{L}(\mathbf{x})\), and split the probability tuples into 50 equally-sized bins. We plot estimation curves in Figure 3: Each curve represents a 50th of the sequences, with darker curves denoting estimation error for sequences with lower target probabilities (rarer sequences). At any given point, then, the distance between estimation curves represents the degree to which estimation error is dependent on the target probability of the sequence.

Here we explore how estimation error varies as a function of the amount of training data. We train GPT2-medium, GPT2-large and an LSTM model in the online "Ideal World" setting (Nakkiran et al., 2020) by sampling, at the beginning of each training iteration, a fresh set of 500,000 sequences from \(p_{L}\), and training \(M\) on this sample. Doing so for 60 iterations, we obtain LMs which have been trained on 30 million sequences. We compute model estimation error on \(\mathcal{D}_{\text{test}}\) at the end of each iteration \(i\).

We again split test sequences by their true probability, with darker lines denoting estimation trends for less probable target sequences. The estimation curves in Figure 4 suggest that while increasing the amount of data in training initially leads to lower estimation error, this reduction eventually asymptotes.

Comparing architectures, we find that the Transformer is significantly more efficient at reducing mean estimation error throughout the distribution. In the previous section, we saw that even when increasing the amount of training data, \(p_{M}\) consistently underestimates the probabilities of sequences sampled from the tail of \(p_{L}\).

To do so, we compute model estimation error on perturbed sequences from \(p_{L}\)--sequences in \(\Sigma^{*}\) which are increasingly far away from the high-probability zones in \(p_{L}\). We build a corpus of perturbed sequences by recursively applying 30 random perturbations to each sequence \(\mathbf{x}\in\mathcal{D}_{\text{test}}\). Formally, the set of sequences at perturbation step \(i\) can be expressed as: \(\mathcal{D}^{(i)}_{\text{perturbed}}=\{\texttt{perturb}(\mathbf{x})\mid\mathbf{x}\in \mathcal{D}^{(i-1)}_{\text{perturbed}}\}\) where \(\texttt{perturb}(\mathbf{x})\) is a function which returns a novel perturbed version of \(\mathbf{x}\), and \(\mathcal{D}^{(0)}_{\text{perturbed}}=\mathcal{D}_{\text{test}}\). Sequence perturbation operations are shown in Table 3.

We score each of these 15M sequences under both the target generative model \(p_{L}\) and the LM \(p_{M}\).

Here we modulate the shape of the sequence space defined by \(p_{L}\) to investigate how estimation error varies with respect to systematic interventions in the target distribution. To adjust the way that \(p_{L}\) allocates probability mass over \(\Sigma^{*}\), we control the entropy of the conditional distributions at each generation step \(t\) by dividing the pre-softmax logits by a temperature value \(T\).6Footnote 6: Formally, for the \(i\)th component of the length \(K\) pre-softmax logits \(\mathbf{x}\), this operation is given as: \(\texttt{softmax}(\mathbf{x})_{i}=\frac{\exp(\frac{\mathbf{x}}{t})}{\sum_{j=1}^{K}\exp \left(\frac{\mathbf{x}}{T}\right)}\)

By increasing the value of \(T\), we increase the entropy of the distributions over next tokens, which in turn, spreads probability mass across a larger number of sequences in \(\Sigma^{*}\). We define four artificial languages with varying \(T\) and train GPT2-medium on an ancestral sample of 1M sequences from each of these artificial languages. Figure 6 visualizes model estimation error by true sequence probability for each model. We find that models trained on languages with increased entropy perform comparatively better than models trained on low entropy languages. Estimation error for models trained on languages with greater \(T\) is less severe, and this holds throughout nearly all target sequence probabilities. These results indicate that neural LMs provide a more accurate approximation of target distributions which spread mass more uniformly across \(\Sigma^{*}\).

We take a more fine-grained approach, proposing a methodology which draws off of instance-level evaluation schemes (Zhong et al., 2021) and the experimental control afforded by artificial corpora (White and Cotterell, 2021; Papadimitriou and Jurafsky, 2020). Indeed, closely related to our work is Kulikov et al. (2021)'s, in which artificial corpora produced by generative models were used to explore mode recovery in neural language modeling.

In this work, we introduce a controlled methodology to evaluate instance-level LM performance on this set of individually rare but collectively frequent events. We use generative models trained on natural language corpora to define a set of artificial languages for which we can exactly compute the probability of sequences. Training LSTM and Transformer LMs on sequences sampled from these artificial languages, our analysis compares the probability estimates given to sequences by the LMs to the target probabilities of sequences under the artificial language. Our results indicate that neural LMs systematically under-represent sequences in the tail of the target distribution, even when increasing the amount of the training data. Investigating where this probability mass went, our perturbation experiments reveal that neural LMs do not tend to overestimate the head of the distribution, but rather overestimate the probability of sequences outside those typical in the target distribution. Comparing model performance on target distributions with varying properties, we find that neural LMs tend to provide more accurate approximations of distributions with greater entropy. Interpreted together, these results indicate that autoregressive neural language models have a tendency to spread probability mass too uniformly across the space of possible sequences.

In our previous experiments, our ground-truth model was given as a Transformer language model trainined on 1.5M sequences from the OpenWebText corpus. Here we explore underestimation behaviour when the ground-truth distribution is given by a pretrained GPT2-medium model fine-tuned on 1.5M sequences from the OpenWebText corpus.

Analogously to the experiment in Section 4.1, we train a randomly-initialized GPT2-medium model on 1M sequences sampled from the fine-tuned model. In the center of Figure 7, we visualize mean model estimation error for \(50,000\) test sequences as a function of true sequence probability. Similarly to our experiments in Section 4.4, we find that the LM underestimates the probability of the majority of sequences, and does so more severely for less probable sequences.

Finally, to ensure that our findings regarding the overestimation of ill-formed sequences hold, we compute model estimation error on random sequences sampled from \(\Sigma^{*}\) (see A.1.4 for details on how these sequences are constructed). Figure 7 center visualizes mean model estimation error as a function of target sequence probability. We find that \(p_{M}\) overestimates the majority of ill-formed sequences, indicating that these findings hold when the ground-truth distribution is defined using a pretrained model.

To provide further evidence that these results hold for other values of \(T\), we conduct similar experiments as in Section 4.3 with an artificial language \(p_{L}\) defined by an ancestral sampling scheme with \(T=1.00\). Specifically, we train GPT2-medium and an GPT2-large on a total of 30M sequences sampled from \(p_{L}\), and we compute model estimation error on a set of withheld test sequences at each training iteration.

Computing estimation error on a set of unseen test sequences, we find, analogously to our experiments on models trained from scratch, that pre-trained models underestimate the probability of the majority of sequences sampled from the target distribution, and do so more severely for rarer sequences (we visualize this in Figure 9). Furthermore, in Figure 10, we increase the amount of fine-tuning data substantially, and we plot test sequence estimation behaviour at the end of each epoch for a pre-trained GPT2-medium model. Once again, increased training data lessens but does eliminate the underestimation behaviour.

We obtain a set of perturbed sequences by (i) sampling a sequence from \(p_{L}\) and then (ii) recursively perturbing this sequence according to the perturbations provided in Table 3. This method provides us with sequences which are increasingly far away from high-probability zones under \(p_{L}\). However, it does so with initial sequences sampled directly from \(p_{L}\), and as a result, produces strings which are edit-adjacent to the high-probability zones (under \(p_{L}\)) in \(\Sigma^{*}\). To ensure that our results hold in other low-probability subspaces, we conduct an analogous experiment on sequences randomly sampled from \(\Sigma^{*}\).

Given this length, we sample \(l\) tokens from \(\Sigma^{*}\) and concatenate all tokens to form a sequence. We then score the sequence under both \(p_{L}\) and \(p_{M}\), and compute model estimation error. We use as artificial language \(p_{L}\) GPT2-medium with \(T=0.85\) and we use as language model \(p_{M}\) a GPT2-medium model trained on 30M sequences ancestrally sampled from \(p_{L}\). Autoregressive neural language models decompose the joint distribution \(p(\mathbf{x})\) over sequences into a series of conditional distributions \(p(x_{i}\mid x_{<i})\). Generating a sequence of length \(n\), then, involves estimating \(n\) conditional distributions.

As shown in Figure 12, observed estimation error for both GPT2-small and GPT2-medium is more severe than expected estimation error as we increase sentence length. This suggests that estimation error for longer (and typically rarer) sequences is not solely due to error compounding. In section 2, we formally defined the LNRE zone as the range of values of \(N\) for which there is non-zero probability of sampling a novel event at the \(N+1\)th draw. Here we introduce a frequentist estimator for this probability. This in turn allows us to empirically verify if a sample exists in the LNRE zone. Suppose we have a set of \(N\) events \(\mathcal{D}=\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\}\) drawn from some generative process \(\phi\). Given \(\mathcal{D}\), we aim to obtain an empirical estimate for the potential productivity \(\hat{\mathcal{P}}_{N}\): the amount of probability allocated to unseen events as a function of \(N\). We can do so using the Good-Turing estimate for the probability of an event given its frequency (Good, 1953). Specifically, let \(f(\mathbf{x},\mathcal{D})\) be a function which returns the frequency of the event \(\mathbf{x}\) in \(\mathcal{D}\). Let \(N_{m}\) denote the number of types (unique events) in \(\mathcal{D}\) for which \(f(\mathbf{x},\mathcal{D})=m\). Good-Turing says that for large \(N\), the probability of the event \(\mathbf{x}\) given that it has occurred with frequency \(m\) in our sample \(\mathcal{D}\) of size \(N\) is equal to \[\hat{P}(\mathbf{x}\mid f(\mathbf{x},\mathcal{D})=m)=\frac{(m+1)}{N}\frac{N_{m+1}}{N_{m}} \tag{3}\]

We apply this method to a subset of OpenWebText, a popular language modeling corpus. In Figure 13, we plot the the empirical estimate of \(\mathcal{P}_{N}\) as function of the sample size \(N\) for \(n\)-grams sampled from a subset of OpenWebText. Particularly for \(n\)-grams with \(n\geq 3\), we find that there is significant probability of sampling a previously unseen event, even for \(N>10,000,000\).

Input: Figure 4: GPT2-medium, GPT2-large and LSTM trained on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 3: Mean model estimation error by training epoch (GPT2-medium). Each line denotes the mean estimation error for a 50th of the sequences; darker lines represent less probable sequences. The shaded area denotes the area in which validation cross-entropy reaches a minimum.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Table 3: PERTURB(x) randomly applies one of these perturbations to x.

Input: Figure 9: Test sequence probability estimates given by pretrained neural LMs fine-tuned on 1M sequences sampled from pL. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 7: Left: Joint histogram of sequence probability estimates for test sequences. Center: Mean model estimation error by true sequence probability for test sequences. Right: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 6: Model estimation error on test sequences as a function of target sequence probability for three different artificial languages. Each line visualizes estimation error for a GPT2-medium model trained to model a language with a specific softmax temperature parameter T .

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 1: GPT2 sequence probability estimates plotted against the true sequence probabilities. Neural LMs underestimate the probability of sequences drawn from the language they are trained to model.

Input: Figure 12: Expected and observed estimation error by sentence length for GPT2-small and GPT2medium.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 10: Pre-trained GPT2-medium fine-tuned on 30M sequences sampled from pL. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in finetuning. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in fine-tuning. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Figure 13: The probability of sampling a novel item (potential productivity P) as a function of sample size N . Many distributions in natural language are characterized by a potentially unbounded amount of novelty.

Input: Figure 11: Mean model estimation error by true sequence probability for sequences randomly sampled from Σ∗.

Input: Figure 5: GPT2-medium estimation behaviour for 15M sequences in Σ∗ across two dimensions: sequence rarity (x-axis) and degree of perturbation (y-axis). The heat map is shaded based on estimation error severity; blue areas indicate overestimation, whereas brown areas indicate underestimation. We also include example sequences from two zones in this sequence space.

Input: Figure 8: GPT2-medium and GPT2-large trained on 30M sequences sampled from pL with T = 1.00. Main plots: Mean estimation error on test sequences as a function of the number of sequences seen in training. Inset plots: Relative change in mean estimation error of test sequences as a function of the number of sequences seen in training. In both cases, each line denotes estimation behaviour for a 50th of the test sequences; darker lines represent less probable sequences.

Input: Table 5: GPT2-medium PP on validation set generated by the artificial language the LM attempts to model (val), and on real sentences sampled from OpenWebText (eng) for models used in Section 4.5

Input: Table 6: Sequences ancestrally sampled from the artificial language generated by GPT2-medium with softmax T = 0.70.

Input: Figure 2: Test sequence probability estimates given by neural LMs. Three left-most figures: The joint histograms of sequence probability estimates. The dotted line denotes the cases in which the model’s estimates perfectly align with the target probability; shading to the right of this line denotes underestimation. Right-most figure: Mean sequence estimation error by target sequence probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim "In this work, we introduce a controlled methodology to explore how LMs estimate the probability of sequences in the heavy-tail of the distribution," I find this to be a well-stated claim that accurately reflects the evidence without exaggeration.

The evidence shows the authors developed a systematic approach using artificial languages with known probability distributions to evaluate language model performance, particularly focusing on rare sequences. The methodology is thoroughly documented with clear experimental setups, detailed analyses across multiple model architectures, and comprehensive evaluations of estimation errors. The included figures and tables demonstrate the controlled nature of their experiments and methodology.

The reviewer's comment affirms this assessment, noting it as "a well-designed set of experiments on synthetic data allowing to precisely control experimental conditions."

<score>1</score>