Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting.

The evidence to be evaluated is:Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns. It should be noted that variational autoencoders (VAEs) also follow the same logic, but because of their limited capacity, which stems from their low dimensional latent space, fail to make the KL-divergence \(D(q\parallel p_{\theta})\) small enough (see Table 3 for the comparisons).

Therefore, early stopping does not necessarily lead to near-distribution generators.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

As it is shown, almost all the methods considerably lag behind our method by a large margin.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

Therefore, it seems reasonable that performance results in the first and last phases of the training procedure would be poor. figure 1 illustrates the mentioned trend (37).

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "existing methods experience up to 20% decrease in performance in the near-distribution setting" appears to be well-stated and supported by the data. Multiple pieces of evidence confirm this:

1. Table 1(b) shows significant performance drops for existing methods in near-ND settings
2. Figure 1 explicitly shows performance drops ranging from 12.8% to 27% for different methods
3. The evidence states "PANDA (34) with 40 percentage points performance drop" and "DeepSAD with roughly 30 percentage points performance decrease"
4. Table 3 demonstrates performance differences of "2% to 20% better AUROCs" when comparing methods

The claim conservatively states "up to 20%" when some methods actually show larger decreases, making this a careful and well-supported statement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting.

The review comment to be evaluated is:

The paper investigates the near anomalies detection and points a significant decrease in performance of state of the art approaches on near anomalies cases. A new solution is proposed: 1) using SDE models trained to generate data from the training distribution. 2) generate a dataset from the trained SDE. 3) train a binary classifier on top of a feature extractor to distinguish between the generated data and real data. 4) use nearest neighbour for detecting anomalies.
Extensive experiments and empirical analysis are carried on. The method shows good results.

The evidence to be evaluated is:Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns. It should be noted that variational autoencoders (VAEs) also follow the same logic, but because of their limited capacity, which stems from their low dimensional latent space, fail to make the KL-divergence \(D(q\parallel p_{\theta})\) small enough (see Table 3 for the comparisons).

Therefore, early stopping does not necessarily lead to near-distribution generators.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

As it is shown, almost all the methods considerably lag behind our method by a large margin.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

Therefore, it seems reasonable that performance results in the first and last phases of the training procedure would be poor. figure 1 illustrates the mentioned trend (37).

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The claim that "existing methods experience up to 20% decrease in performance in the near-distribution setting" is well-supported by the evidence provided:

1. Table 1(b) and Figure 1 show clear performance drops for existing methods in near-distribution detection:
- PANDA shows 40 percentage point drop
- DeepSAD shows 30 percentage point drop
- Several models show 14-27% decreases

2. The evidence consistently demonstrates significant performance degradation across multiple datasets and methods when dealing with near-distribution anomalies compared to standard anomaly detection.

3. The 20% figure appears conservative given some methods show even larger decreases, making this a well-stated claim that accurately reflects the experimental results without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting.

The review comment to be evaluated is:

* There is a large room for improvement in clarity.
    * The paper should be clearer on how exactly the generative model is trained and used. The most important fact, in my opinion, is that the score-based generative model is trained on in-distribution data and terminated prematurely. However, the fact is only briefly mentioned in Page 2, and then mentioned in Figure 2. Given the importance of this information, this point should be described in detail in Section 2.1. Also, other details and design choices, for example, how early the training of the score-based model should be stopped, needs to be elaborated.
    * The experimental setup described in Section 3.2 is difficult to follow. There are several comparison settings but are not well organized.
* Discussion on the differences among diffusion models, GANs, and VAEs in Section 2.2 are not very persuasive. It does not have concrete examples or quantitative arguments. Plus, the argument of Section 3.2 is mostly about generative modeling itself and not about its application to generating auxiliary data in novelty detection.
* When performing CIFAR-10 vs CIFAR-100 experiment, it is more standard and more challenging to incorporate all the classes, instead of using only one class from each dataset. Overall, I am not convinced that the experiment settings used in the paper are standard.
* Replacing a GAN with a diffusion model in the data generator for novelty detection itself is not a very novel contribution to the community.

The evidence to be evaluated is:Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns. It should be noted that variational autoencoders (VAEs) also follow the same logic, but because of their limited capacity, which stems from their low dimensional latent space, fail to make the KL-divergence \(D(q\parallel p_{\theta})\) small enough (see Table 3 for the comparisons).

Therefore, early stopping does not necessarily lead to near-distribution generators.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

As it is shown, almost all the methods considerably lag behind our method by a large margin.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

Therefore, it seems reasonable that performance results in the first and last phases of the training procedure would be poor. figure 1 illustrates the mentioned trend (37).

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review, the claim "We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting" appears to be well-supported by the evidence. The figures and tables, particularly Table 1 and Figure 1, demonstrate performance drops ranging from 12.8% to 27% for various methods in near-distribution settings. Some methods like PANDA show drops of around 19.5%, and CSI shows an 18.2% decrease. The claim conservatively states "up to 20%" which accurately reflects these findings without exaggeration. The comprehensive experimental results across multiple datasets and methods substantiate this claim.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting.

The review comment to be evaluated is:

The idea of this paper is very natural: try to generate near-distribution outliers to augment the original training dataset which only contains normal data (semi-supervised learning), such that we can fine-tune the pre-trained feature extractor by training a binary classifier.

Strengths:
(1) Using generated outliers to improve the accuracy of novelty detection is not new. There have been some similar works which use GANs instead of SDE.  The main contribution of this paper is that they replace the GANs with the diffusion model (which is increasingly popular recently).  The motivation here is that the outliers generated by SDE are “closer“ to the nominal distribution, and then using these "boarder" points  to deal with the near-ND setting.  Even though the pipeline and SDE are all from existing works, the novel combination is meaningful considering the theoretically and experimental results.

(2) The experimental results show that the proposed method considerably improves over existing models (especially for the near-ND setting).  It is not surprising that SDE over-perform GANs for the near-ND setting, but I am surprised that the improvement is so obvious.  It will be great if the code can be provided.

Weakness:
(1) In Section 2.2 (Diffusion models vs. GANs), they try to explain, for near-ND, why choosing diffusion models instead of other generative modes (GAN and VAE), it would be better to include flow-based models in this part. Also,  VAE-related results are not included in figure3.

(2) In Figure3,  density, coverage and FID are used as the metrics to measure the fidelity and diversity.  Could you add a brief introductions/definitions for these metrics? Also, I am confused how to get the "density" from GAN-based models as they are not tractable models.

(3)About Stopping point: In section 4, "the method is robust against the stopping point with at most 2% variation for the reasonable FID scores". However,  from table4,  it is difficult to get this conclusion. what is "reasonable" FID? Actually, the stopping point decide how similar the fake points and the real points. How to decide when to stop ?

Other minor comments:
The "normal distribution" used in this paper sometimes are confusing, which means the distribution of nominal data instead of Gaussian distribution. It would be better to use other notations.

The evidence to be evaluated is:Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns. It should be noted that variational autoencoders (VAEs) also follow the same logic, but because of their limited capacity, which stems from their low dimensional latent space, fail to make the KL-divergence \(D(q\parallel p_{\theta})\) small enough (see Table 3 for the comparisons).

Therefore, early stopping does not necessarily lead to near-distribution generators.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

As it is shown, almost all the methods considerably lag behind our method by a large margin.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

Therefore, it seems reasonable that performance results in the first and last phases of the training procedure would be poor. figure 1 illustrates the mentioned trend (37).

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I can evaluate the claim "We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting."

The evidence strongly supports this claim through:
- Figure 1 showing performance drops of 12.8% to 27% for various methods
- Table 1(b) demonstrating significant decreases in AUROC scores for near-distribution cases
- Multiple models showing substantial performance degradation when dealing with near-distribution samples
- Quantitative results in tables showing consistent performance drops across different datasets

The claim is well-stated and conservative, as some methods actually show drops larger than 20% (e.g., PANDA with 40% drop). The evidence thoroughly validates the magnitude stated in the claim through multiple experiments and metrics.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We first demonstrate existing methods experience up to 20% decrease in performance in the near-distribution setting.

The review comment to be evaluated is:

### Strengths

- The paper is, for the most part, well-written, easy-to-follow, and does a good job of
contextualising and motivating the work.
- The proposed method is simple and intuitive, and seems to perform well empirically
compared with the baselines in both the regular ND setting and the near ND setting, showing a
significant reduction in the performance gap of going from the former to the latter.
the NR to near-NR setting.
- The figures are well-chosen and well-put-together: they convey well the essential points of the
  paper, such as the shortcomings of existing methods and the feasibility of using GANs vs diffusion
  models for creating the synthetic data.
- Evaluation performed using a good range and selection of baseline methods and benchmark datasets.
- Results are aggregated over a good number (10) of replicates.

### Weaknesses
- Despite the authors claiming otherwise, the method is quite sensitive to the stopping criterion in
  the near ND setting with an empirical range of [68.2, 90.0]; in practice, different datasets may
  require wildly different FID thresholds and there is no reliable way of validating this given the
  nature of the problem. Also, I wonder if the number of sampling steps and the choice of sampler have a
  significant effect on the results -- there doesn't seem to be any mention of this.
  Taking this a step further, might it be feasible to train the diffusion model fully and then rely
  upon early-stopping the reverse diffusion process according to the FID score?
- The need to train, and indeed sample, from a diffusion model can be computationally burdensome for
  higher-resolution datasets.
- The description of the SDE model is a little overdone; a much briefer description with a pointing of the
  reader to the original paper would be sufficient. This is especially the case given that, as
  I understand it, the proposed method should work with any diffusion based model (Imagen, Parti,
  DALLE-2, etc.) or are there theoretically some constraints on the form said model and the
  associated sampler should take?
- While this applies to several methods of the same class, the method seems to make certain
assumptions about the distribution of the ID and OOD data that might not hold in the wild.
- While results are aggregated over a good number (10) of replicates no measure of dispersion
  (standard deviation/error) seems to be reported alongside the mean in each case.
- Some typographic and syntactical errors (e.g. 'that unlikely come' (line 1 of the introduction)).

The evidence to be evaluated is:Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns. It should be noted that variational autoencoders (VAEs) also follow the same logic, but because of their limited capacity, which stems from their low dimensional latent space, fail to make the KL-divergence \(D(q\parallel p_{\theta})\) small enough (see Table 3 for the comparisons).

Therefore, early stopping does not necessarily lead to near-distribution generators.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

As it is shown, almost all the methods considerably lag behind our method by a large margin.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

Therefore, it seems reasonable that performance results in the first and last phases of the training procedure would be poor. figure 1 illustrates the mentioned trend (37).

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "existing methods experience up to 20% decrease in performance in the near-distribution setting" appears to be well-supported by the data presented. Table 1(b) and Figure 1 show significant performance drops for multiple methods, with PANDA showing a 40% decrease and DeepSAD showing a 30% decrease. The visual evidence from Figures 10-12 demonstrates the challenge of near-distribution anomalies. However, the 20% figure in the claim is actually conservative compared to the larger drops shown in the evidence, making this a well-stated claim that doesn't overstate the findings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our first contribution is to benchmark eight recent novelty detection methods in the near-ND setting, which consists of ND problems whose normal and anomaly classes are either naturally semantically close, or else synthetically forced to be close.

The evidence to be evaluated is:In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted. In the diffusion models, the gradient of the logarithmic probability density of the data with respect to the input is called the score. A SDE model considers a continuum of distributions that develop over time in accordance with a diffusion process, to corrupt a sample and turn it into pure noise.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process).

\[x_{n-1}-x_{n}=[f(x,t)-g^{2}(t)s_{\theta}(x,t)]dt+g(t)d\hat{w},\quad x_{T}\sim N (0,1), \tag{1}\] where \(x_{n}\) is the data sample at time \(n\), \(f=0\) and \(g=\sigma^{t}\) are called the drift and diffusion coefficients, respectively. \(s_{\theta}(x,t)\) is called the score function, as is expected to estimate \(\nabla_{x}\log p_{t}(x)\).

In this section, we compare our method against the SOTA in the ND and near-ND setups. All the experiments are performed 10 times and the average result is reported.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Our experiments are organized into three categories and are presented separately in sub-tables.

To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND. The datasets that are tested in this setting include FGVC-Aircraft (25), Stanford-Cars (22), White Blood Cell (WBC) (49), and Weather (10). Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets.

While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process.

All of these results highlight the need of benchmarking previous ND methods in the near-ND setup.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models. Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets. Having artifacts, mode-collapse, and undiversified generated samples could be the reasons behind this observation, which are shown in Appendix10. This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection. The key point in OpenGAN is that a tiny additional dataset, containing both in- and outliers, is used as a validation set for the model selection. This, also known as outlier exposure, is necessary due to the GAN unstable training. Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard near-distribution anomalies.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting.

To provide a standard benchmark for the near-ND task there is a need to define a one-class distribution closeness score. Suppose a \(K\)-class training dataset is given from which a normal class \(C_{i}\) is randomly sampled and used to train the model \(\mathcal{M}\). The closest abnormal distribution with respect to the selected normal class for \(\mathcal{M}\) can be defined as \(C_{j}\neq C_{i}\) that minimizes the test time performance when it is considered as the abnormal distribution. We call the performance in the mentioned scenario the bottom-1 score of class \(C_{i}\) with the backbone \(\mathcal{M}\). However, this score depends on the choice of \(\mathcal{M}\), making it specific for every problem setup and method. Therefore, inspired by the CLP criterion introduced in (47) as a measurement of dataset distance, we could use CLP as an alternative closeness score for the novelty detection task. Given a \(K\)-class training dataset, one category is randomly selected as the normal distribution. Then, a supervised classifier \(\mathcal{P}\) is trained on the rest \(K-1\) abnormal categories.

The closeness score of each abnormal class \(i\) with respect to the normal class is obtained as follows: \[\text{Closeness score}_{i}=\sum_{x\in\text{Normal Class}}\mathcal{P}(\hat{y}=i|x). \tag{8}\] The higher the closeness score of an abnormal category, the more similar it is to the normal class.

In section 8, we proposed the closeness score (CLP) to find the closest abnormal class for each class of the normal dataset. In our experiments, we used the ViT-B_16 (pretrained on ImageNet-21K) as the backbone used for extracting the closest abnormal classes based on this criterion. The bottom-1 abnormal class is the class that has the lowest novelty detection performance during the testing phase. Note that the abnormal classes chosen based on the CLP criterion do not necessarily have the worst novelty detection performance; therefore, it is expected for the novelty detection methods to perform better on the CLP criterion. This section aims to investigate how well these two criteria match each other. In the Table 5, for each novelty detection model and every class of the CIFAR-10 dataset, the bottom-1 class is shown. Furthermore, the last row indicates the closest abnormal class selected based on the CLP criterion. As shown in the Table 5, the extracted classes based on the CLP criterion are the same or conceptually similar to the respected bottom-1 classes. Figure 2 also shows a decent correlation between the bottom-1 score and the CLP score, which means both these criteria could be used to extract near-distribution abnormal classes, and that the CLP could be used as a proxy to the bottom-1 criterion. Also, The Pearson correlation coefficient between these two results is 0.843, indicating a strong positive relationship.

Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup.

As can be seen, the images are clearly different semantically from the normal class. Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous.

To further extend our results, we compared the methods on a number of small datasets from different domains: 102 Category Flowers and Caltech-UCSD Birds 200. For each of these datasets, we evaluated the methods using only each of the first 20 classes as normal, and using the entire test set for the evaluation. For FGVC-Aircraft, due to the extreme similarity of the classes to each other, we randomly selected a subset of ten classes from the entire dataset, such that no two classes have the same Manufacturer.

MvTecAD: This dataset contains 15 different industrial products, with normal images of the proper products for training and 1 to 9 types of manufacturing errors as anomalies. The anomalies in MvTecAD are in-class, i.e. the anomalous images come from the same class of the normal images with subtle variations. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input: Table 5: For each model and every class of the CIFAR-10, the anomalous class that minimizes the anomaly detection performance among 100 classes of the CIFAR-100 dataset is reported. The last row is the CIFAR-100 class that achieves the highest CLP score. Note that the latter does not depend on the anomaly detection method.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 4: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 5: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
The evidence strongly supports the claim regarding benchmarking novelty detection methods in the near-ND setting. The paper provides:
- Comprehensive experimental results comparing 8 methods across multiple datasets
- Clear definition and methodology for near-ND problems using both natural and synthetic approaches
- Detailed performance metrics and analysis showing significant drops in detection accuracy for near-distribution anomalies
- Extensive visual evidence demonstrating the subtle differences between normal and near-anomaly samples
- Quantitative results in Table 1 showing performance gaps between regular ND and near-ND settings

The claim is measured and accurately reflects the scope and findings of the work without overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We expect that BPE-like segmentation might play a crucial role in memorization, as it controls the trade-off between the number of primitives a model will have to operate with and the lengths of sequences it must represent.

The review comment to be evaluated is:

This paper as the name suggests, tries to figure out whether and through which ways bpe can affect Transformer’s memorization capacity. It evaluates Transformer’s memorization under these 4 settings: memorizing random synthetic data, memorizing random labeled natural language data, recognizing training data with lower output entropy, and training data recovery QA. Experiments show that if we have more merge times for bpe (larger vocabulary size), the model’s performance on all the four settings would improve on 3 architectures, which shows that bpe indeed can affect the model’s memorization capacity. Larger the vocabulary size is, the better the model would perform on memorizing. Then, the paper tries to figure out why more merges in bpe would lead to better memorization. By excluding two other hypotheses, the paper concludes that more merges in bpe results in shorter input sequence and that makes memorizing easier for transformers.

The evidence to be evaluated is:To answer this question, for every experiment we create various versions of the involved datasets that differ in the number of subwords, by varying the parameters of the BPE process used to create subwords. In particular, for the SNLI dataset (used in random label memorization and membership inference), we apply BPE with \((0.5,1,5,10,20)\times 10^{3}\) steps, resulting in vocabulary sizes of 611, 1097, 4943, 9574, and 18336, respectively. For the larger PAQ dataset, used in the recovery experiment, we get different versions of the dataset by running BPE for \((0.5,1,5,10,15,20)\times 10^{3}\) steps, obtaining vocabulary sizes of 1280, 1784, 5784, 10784, 15784, and 20776, respectively. In our experiments, we vary the size of the subword vocabulary used to represent the data.

It is not unreasonable to expect that the memorization capabilities of a model are impacted by this growth of the number of the learned parameters.

In this section, we discuss our first main finding: that the number of BPE merges influences the extent to which a model memorizes training data. This finding is persistent across the three different experimental setups we described earlier: random label memorization, membership inference and question-answer recovery. In all these experiments, we systematically control the number of merges, which is roughly the same as the vocabulary size.

In Figure 1, we report the accuracy on fitting the training data with random labels as a function of base-2 logarithm of the vocabulary size \(|V|\). In Figure 0(a) we report results for "vanilla" architectures (without an additional FFN layer that allows to control for the number of learned parameters) and in Figure 0(b) we provide results for the models with the number of parameters fixed.

Encoder fits around 87% of the random labels with the subword vocabulary size of 611 and has a training accuracy of more than 98% when the vocabulary size is increased to 18,336. LM and MLM - that have more layers than Encoder - start off with a slightly higher accuracy (94% and 97%, respectively), their accuracy quickly climbs up to 100% when increasing the number of subwords.

Mirroring the results of our previous experiment, we observe a monotonic dependency of the membership attack success rate in function of the size of the vocabulary used: for all architectures, larger vocabulary granularity implies more memorisation.

From Figure 1(c) we see that as the vocabulary size grows, the test accuracy of (M)LM has distinct regions of growth.

In SS4 we established that larger vocabularies, learned by BPE, lead to stronger memorization in Transformer models, even when we control for the number of the learned parameters.

We hypothesise that the primary cause of the observed behavior is that with the growth of the BPE vocabulary the input sequences become shorter (_length hypothesis_). In turn, shorter sequences are easier to memorize as the work of attention is simplified.

At the same time, shorter sequences would shift the responsibility for memorization onto FFN layers, which are known to memorize well (Zhang et al., 2016).

Being a compression algorithm originally, BPE compresses the data and it could be easier to memorize data without redundancies (_redundancy hypothesis_). Second, BPE increases the vocabulary size and it is possible that each individual token becomes more predictive of the label (_vocabulary size hypothesis_).

To study how the chosen level of representation affects the models' memorization capacity, we apply BPE with {0, 4, 8, 16, 32, 64, 128, 256} steps on the generated data. This allows us to start from a binary dataset with sufficiently large sequences and produce a sequence of datasets _of the same storage complexity_ (in bits), but with varied vocabulary sizes.

Again, we observe that for all three architectures the dependency is very consistent: as the vocabulary size grows, the models become universally more successful in fitting random labels.6 We conclude that the increased memorization success is thus not caused by BPE compressing-out redundancies in more natural data.

We start from our earlier random label memorization setup with SNLI data, using the vocabulary learned with 500 BPE operations. To increase the number of tokens in this vocabulary, we randomly replace each unique token's occurrence with either of two new tokens, with equal probability, roughly doubling the vocabulary size. 7 For instance, say that the current vocabulary has a token "cat". We then iterate over the corpus and replace half of its occurrences with "cat1" and the other half with "cat2" (making sure that those do not happen in the prior vocabulary). Thus, if the vocabulary growth alone can explain the observed effect, we will see that the datasets with "duplicated" tokens would also be easier to memorize.

We report results of this experiment using the LM model in Figure 3(b), contrasting it to the BPE-induced memorization results. We see that the resulting curves exhibit considerably different patterns. The line that corresponds to the doubling procedure has a sharp fall in the beginning. Later, as the vocabulary size grows, in accordance with in our thought experiment, tokens become increasingly more unique to sequences which boosts the train accuracy. In contrast, the line obtained by growing the vocabulary via BPE shows a monotonic, consistent growth for all vocabulary sizes.

We have observed a strong and monotonic influence of the BPE vocabulary size on how successful Transformer models are at memorizing training data. With higher granularity vocabulary (i.e., more BPE merge operations), Transformers (a) _can_ memorize random mappings better, hence they have higher memory capacity, and (b) _do memorize_ more of training data in realistic tasks.

Next, we conjecture that the phenomenon is caused by the reduction in the sequence length that correlates with using larger-cardinality BPE vocabularies. It can simplify memorization, as the latter will no longer require learning complex attention patterns "explaining" the data and offset complexity onto fully-connected layers, which are known to be good in memorization (Zhang et al., 2016).

After ruling out the two alternative possibilities, we are left with our initial hypothesis that using higher-cardinality BPE subword vocabularies implies having to manage shorter sentences on average, which in turn enable easier memorization.

Our experiments provide a clear demonstration that both in artificial and natural data, the simple choice of the number of subwords used has a large impact on the extent to which models memorize training data, a fact that is not at all obvious when merely looking at performance on i.i.d. train/test splits.

In cases where a lot of memorization is desirable (e.g., QA-tasks) or undesirable (e.g., public models trained on medical data), the number of subwords could provide an important factor in tuning the amount of memorization that happens inside a model.

In our analysis in SS5, we conjecture that the observed effect is caused by the fact that large-cardinality BPE vocabulary reduces pressure on attention to learn complex patterns due to having less tokens to attend to.

What we observe in this paper is that, when needed, this limitation can be alleviated by (a) increasing granularity of the units that represent the data (e.g., by running BPE), or (b) improving representation power of the self-attention mechanism.

In this Section we investigate whether the relation between the success rate in the training data extraction experiments and the BPE-learned vocabulary size, reported in Section 4.3 and Figure 3, is independent from the way we generate continuations of a prompt.

larger BPE vocabulary sizes lead to better memorization.

In the main text, we considered three factors that accompany growing BPE vocabulary sizes: reduction in sequence length, increased number of tokens, and decrease in the redundancy.

We can expect that with the increased number of merges, BPE-learned subwords will start approximating words more closely and the frequency distribution of tokens will approach that of words. In turn, the latter follows an extremely skewed Zipf distribution (Zipf, 1949).

increasing the number of merges forces the token distribution to become more peaky.

we firstly group all possible pairs of 0s and 1s into new tokens (00, 01, 10, 11) and use them to encode the initial strings (e.g., "0 0 0 0 0 0 0 0 0 0 0 0 0 0 1" becomes "00 00 00 00 00 01"). We recursively repeat this procedure, obtaining a sequence of views of the same dataset, at each step growing the vocabulary size quadratically (2, 4, 16, and 256) and reducing the string length by two.

Input: Figure 3: Training data recovery: top-1 and top-5 accuracy on extracting the correct answer when prompted with a query.

Input: Figure 1: Training accuracy for fitting random labels on SNLI. Shaded area represents ±1 SEM (Standard Error of the Mean).

Input: Figure 4: Analysing potential causes of the increased memorization: fitting random labels on random strings (left) and SNLI (right) datasets. Everywhere shaded area represents ±1 SEM.

Input: Figure 7: Investigating the relative frequency hypothesis.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I assess that it is relatively well-stated. The paper systematically investigates BPE's role in memorization through multiple controlled experiments, ruling out alternative hypotheses. The evidence strongly supports that BPE segmentation affects memorization by controlling sequence length, showing consistent effects across different architectures and experimental setups. The claim appropriately uses hedging language ("might play") and frames it as a trade-off rather than making absolute statements. While the evidence is strong, the cautious wording matches the level of certainty warranted by the findings.

<score>2</score>

Justification:
1. The data shows clear correlations between BPE merge operations/vocabulary size and memorization capability across multiple experiments and architectures
2. The authors systematically ruled out alternative explanations through controlled experiments
3. The paper acknowledges complexity by discussing multiple factors (sequence length, token distribution, etc.)
4. The claim's hedged language ("might play") appropriately reflects scientific uncertainty
5. The evidence supports both aspects mentioned in the claim: impact on number of primitives (vocabulary size) and sequence lengths
6. The review comment confirms the paper's methodical approach in establishing BPE's effects on memorization

The claim could be rated a 1, but I gave it a 2 since some of the causal mechanisms, while strongly supported, still retain some uncertainty that the hedging language appropriately acknowledges.