Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The importance of artifact- and noise-free anomalous samples in fine tuning is due to the fact that deep models tend to learn such artifacts as shortcuts, preventing them from generalization to the true essence of the anomalous samples.

The review comment to be evaluated is:

The idea of this paper is very natural: try to generate near-distribution outliers to augment the original training dataset which only contains normal data (semi-supervised learning), such that we can fine-tune the pre-trained feature extractor by training a binary classifier.

Strengths:
(1) Using generated outliers to improve the accuracy of novelty detection is not new. There have been some similar works which use GANs instead of SDE.  The main contribution of this paper is that they replace the GANs with the diffusion model (which is increasingly popular recently).  The motivation here is that the outliers generated by SDE are “closer“ to the nominal distribution, and then using these "boarder" points  to deal with the near-ND setting.  Even though the pipeline and SDE are all from existing works, the novel combination is meaningful considering the theoretically and experimental results.

(2) The experimental results show that the proposed method considerably improves over existing models (especially for the near-ND setting).  It is not surprising that SDE over-perform GANs for the near-ND setting, but I am surprised that the improvement is so obvious.  It will be great if the code can be provided.

Weakness:
(1) In Section 2.2 (Diffusion models vs. GANs), they try to explain, for near-ND, why choosing diffusion models instead of other generative modes (GAN and VAE), it would be better to include flow-based models in this part. Also,  VAE-related results are not included in figure3.

(2) In Figure3,  density, coverage and FID are used as the metrics to measure the fidelity and diversity.  Could you add a brief introductions/definitions for these metrics? Also, I am confused how to get the "density" from GAN-based models as they are not tractable models.

(3)About Stopping point: In section 4, "the method is robust against the stopping point with at most 2% variation for the reasonable FID scores". However,  from table4,  it is difficult to get this conclusion. what is "reasonable" FID? Actually, the stopping point decide how similar the fake points and the real points. How to decide when to stop ?

Other minor comments:
The "normal distribution" used in this paper sometimes are confusing, which means the distribution of nominal data instead of Gaussian distribution. It would be better to use other notations.

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs.

The main benefit of using a score-based generative model is that in the backward process (Eq. 1), the score function denoises the input gradually that results in a relatively smooth decline in the Frechet Inception Distance (FID) (17) across training epochs. This enables us to stably produce near anomalous samples based on the FID score of the generated outputs, which could be achieved by stopping the training process earlier than the stage where the model achieves its maximum performance.

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs.

Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

The anomalous data that are generated by StyleGAN-ADA contain lots of artifact-full and distorted samples that a practical novelty detector is expected to detect.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets. Having artifacts, mode-collapse, and undiversified generated samples could be the reasons behind this observation, which are shown in Appendix10.

It has recently been shown that contrastive learning can also improve anomaly detection (44), which encourages the model to learn normal features by contrasting positive and negative samples.

Intuitively, abnormal and normal data do not overlap in the feature space because many features model high-level and semantic image properties. So, in the pre-trained feature space, one may classify normal vs. anomaly.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data. We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

It is important to note that the synthesized images were not an effective anomaly in the first training procedure because they are extremely noisy. Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Input: Figure 7: Generated anomalous samples by different generative models on the WBC dataset.

Input: Figure 4: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 6: Generated anomalous samples by different generative models on the FGVC-Aircraft dataset.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim: "The importance of artifact- and noise-free anomalous samples in fine tuning is due to the fact that deep models tend to learn such artifacts as shortcuts, preventing them from generalization to the true essence of the anomalous samples."

Analysis:
The evidence strongly supports this claim through both textual and visual demonstrations. The figures (especially Fig. 6 and 7) clearly show how other generative models produce artifacts and distorted samples compared to the SDE approach. The text explicitly states that "deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics." Additionally, the evidence shows that models with artifacts performed poorly on Weather and MvTecAD datasets, confirming the negative impact of artifacts on generalization.

<score>1</score>

Justification:
1. Visual evidence demonstrates clear differences between artifact-free and artifact-containing samples
2. Experimental results validate the negative impact of artifacts on performance
3. The claim's wording matches the scope and certainty of the evidence
4. The provided reviewer comments support the technical validity without raising concerns about this specific claim
5. The statement acknowledges the underlying mechanism (shortcuts) without overgeneralizing


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The importance of artifact- and noise-free anomalous samples in fine tuning is due to the fact that deep models tend to learn such artifacts as shortcuts, preventing them from generalization to the true essence of the anomalous samples.

The review comment to be evaluated is:

### Strengths

- The paper is, for the most part, well-written, easy-to-follow, and does a good job of
contextualising and motivating the work.
- The proposed method is simple and intuitive, and seems to perform well empirically
compared with the baselines in both the regular ND setting and the near ND setting, showing a
significant reduction in the performance gap of going from the former to the latter.
the NR to near-NR setting.
- The figures are well-chosen and well-put-together: they convey well the essential points of the
  paper, such as the shortcomings of existing methods and the feasibility of using GANs vs diffusion
  models for creating the synthetic data.
- Evaluation performed using a good range and selection of baseline methods and benchmark datasets.
- Results are aggregated over a good number (10) of replicates.

### Weaknesses
- Despite the authors claiming otherwise, the method is quite sensitive to the stopping criterion in
  the near ND setting with an empirical range of [68.2, 90.0]; in practice, different datasets may
  require wildly different FID thresholds and there is no reliable way of validating this given the
  nature of the problem. Also, I wonder if the number of sampling steps and the choice of sampler have a
  significant effect on the results -- there doesn't seem to be any mention of this.
  Taking this a step further, might it be feasible to train the diffusion model fully and then rely
  upon early-stopping the reverse diffusion process according to the FID score?
- The need to train, and indeed sample, from a diffusion model can be computationally burdensome for
  higher-resolution datasets.
- The description of the SDE model is a little overdone; a much briefer description with a pointing of the
  reader to the original paper would be sufficient. This is especially the case given that, as
  I understand it, the proposed method should work with any diffusion based model (Imagen, Parti,
  DALLE-2, etc.) or are there theoretically some constraints on the form said model and the
  associated sampler should take?
- While this applies to several methods of the same class, the method seems to make certain
assumptions about the distribution of the ID and OOD data that might not hold in the wild.
- While results are aggregated over a good number (10) of replicates no measure of dispersion
  (standard deviation/error) seems to be reported alongside the mean in each case.
- Some typographic and syntactical errors (e.g. 'that unlikely come' (line 1 of the introduction)).

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs.

The main benefit of using a score-based generative model is that in the backward process (Eq. 1), the score function denoises the input gradually that results in a relatively smooth decline in the Frechet Inception Distance (FID) (17) across training epochs. This enables us to stably produce near anomalous samples based on the FID score of the generated outputs, which could be achieved by stopping the training process earlier than the stage where the model achieves its maximum performance.

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs.

Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

The anomalous data that are generated by StyleGAN-ADA contain lots of artifact-full and distorted samples that a practical novelty detector is expected to detect.

Interestingly, other models generated samples are harmful to the performance in Weather and MvTecAD datasets. Having artifacts, mode-collapse, and undiversified generated samples could be the reasons behind this observation, which are shown in Appendix10.

It has recently been shown that contrastive learning can also improve anomaly detection (44), which encourages the model to learn normal features by contrasting positive and negative samples.

Intuitively, abnormal and normal data do not overlap in the feature space because many features model high-level and semantic image properties. So, in the pre-trained feature space, one may classify normal vs. anomaly.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data. We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

It is important to note that the synthesized images were not an effective anomaly in the first training procedure because they are extremely noisy. Furthermore, one has to note that current deep models are highly capable of learning artifacts from their input, instead of paying attention to the general semantics of the input.

However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images.

Input: Figure 7: Generated anomalous samples by different generative models on the WBC dataset.

Input: Figure 4: For each class of CIFAR-10, nearest class from CIFAR-100 has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.

Input: Figure 6: Generated anomalous samples by different generative models on the FGVC-Aircraft dataset.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the main argument about deep models learning artifacts as shortcuts is partially supported but somewhat overstated. While the evidence shows that some generative models (StyleGAN-ADA, GAN baselines) produce artifacts that affect model performance, the figures demonstrate that not all approaches suffer equally from this issue. The SDE-based approach produces cleaner samples with fewer artifacts (as shown in Figures 6 and 7). The claim makes a broad generalization about "deep models" without acknowledging that some architectures can effectively avoid artifact-based shortcuts.

Justification:
- The evidence shows artifact issues primarily in specific models (StyleGAN-ADA) but not universally
- Images demonstrate varying quality across different approaches
- The review comments note assumptions about data distributions "that might not hold in the wild"
- The paper suggests solutions exist (SDE approach) rather than this being an unavoidable problem
- The claim would be better stated with qualifications about which types of models are most susceptible

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, our last contribution is to show that fine-tuning simple baseline ND methods with the generated samples to distinguish them from the normal data leads to a performance boost for both ND and near-ND.

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs. In the second step, the pre-trained backbone is fine-tuned by the generated dataset and given normal training samples through optimizing a binary classification loss.

In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process). Running the backward process allows us to seamlessly transform a random noise into the data that could be used for sample generation as follows:

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs. After the training, all the normal embeddings are stored in the memory \(\mathcal{M}\) that is used at the test time for assigning abnormality score to a given input. Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns.

Our empirical results also confirm that various quality metrics, such as coverage and density(28), which were previously proposed to evaluate synthetic images' fidelity and diversity, where larger values that are close to 1.0 indicate better fidelity and diversity, respectively.

Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process. To further validate our method, we adapt OpenGAN, as SOTA OOD detection, to the ND setting and report its near-ND results. Surprisingly, even by employing fake abnormal samples generated by a GAN and being exposed to real outliers, it is not able to achieve a decent performance in the near-ND setting (Table 1 (c)), which again supports the benefits of our proposed approach for generating near distribution abnormal samples.

For each backbone, the performance with and without the feature fine-tuning phase based on the generated data is reported (denoted as "Pre-trained" and "Fine-tuned" in the table), showing a consistent improvement across different architectures. The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection.

We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

* Murase and Fukumizu [2022] Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pseudo anomalous data via latent variables. _arXiv preprint arXiv:2202.10281_, 2022.

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the statement that "fine-tuning simple baseline ND methods with generated samples leads to a performance boost for both ND and near-ND" appears well-supported by the evidence. The tables show consistent improvements across multiple datasets and architectures after fine-tuning. For example, ViT-B_16 showed ~25% AUROC improvement in near-ND and ~8% in ND settings. The evidence demonstrates quantitative performance gains through comprehensive experimental results and ablation studies comparing different backbones and generation methods.

<score>1</score>

Justification:
1. Comprehensive empirical evidence in Tables 1(a-c) demonstrates consistent performance improvements across multiple datasets and architectures
2. Specific performance gains are clearly quantified (e.g., "16%, 25%, and 36% boosts in AUROCs" for different backbones)
3. Comparative analysis with SOTA methods validates the claims
4. Limitations and conditions are acknowledged (e.g., careful selection of generative models needed)
5. The claim avoids overreaching statements and stays focused on demonstrated performance improvements

The evidence strongly supports the claim without exaggeration, making it well-stated (score of 1).


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, our last contribution is to show that fine-tuning simple baseline ND methods with the generated samples to distinguish them from the normal data leads to a performance boost for both ND and near-ND.

The review comment to be evaluated is:

The paper investigates the near anomalies detection and points a significant decrease in performance of state of the art approaches on near anomalies cases. A new solution is proposed: 1) using SDE models trained to generate data from the training distribution. 2) generate a dataset from the trained SDE. 3) train a binary classifier on top of a feature extractor to distinguish between the generated data and real data. 4) use nearest neighbour for detecting anomalies.
Extensive experiments and empirical analysis are carried on. The method shows good results.

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs. In the second step, the pre-trained backbone is fine-tuned by the generated dataset and given normal training samples through optimizing a binary classification loss.

In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process). Running the backward process allows us to seamlessly transform a random noise into the data that could be used for sample generation as follows:

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs. After the training, all the normal embeddings are stored in the memory \(\mathcal{M}\) that is used at the test time for assigning abnormality score to a given input. Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns.

Our empirical results also confirm that various quality metrics, such as coverage and density(28), which were previously proposed to evaluate synthetic images' fidelity and diversity, where larger values that are close to 1.0 indicate better fidelity and diversity, respectively.

Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process. To further validate our method, we adapt OpenGAN, as SOTA OOD detection, to the ND setting and report its near-ND results. Surprisingly, even by employing fake abnormal samples generated by a GAN and being exposed to real outliers, it is not able to achieve a decent performance in the near-ND setting (Table 1 (c)), which again supports the benefits of our proposed approach for generating near distribution abnormal samples.

For each backbone, the performance with and without the feature fine-tuning phase based on the generated data is reported (denoted as "Pre-trained" and "Fine-tuned" in the table), showing a consistent improvement across different architectures. The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection.

We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

* Murase and Fukumizu [2022] Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pseudo anomalous data via latent variables. _arXiv preprint arXiv:2202.10281_, 2022.

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is mostly well-stated but slightly overstated. While the evidence demonstrates clear performance improvements after fine-tuning across multiple architectures and datasets (shown in Table 1 and discussed in detail), the word "simple" in describing the baseline ND methods understates their complexity. The evidence shows comprehensive experiments validating performance boosts in both ND and near-ND settings, with significant improvements (e.g., 16%, 25%, and 36% boosts for different architectures). However, the baseline methods aren't necessarily "simple" as claimed.

<score>2</score>

Justification:
1. The performance improvements are well-documented with specific numbers and extensive experiments
2. Tables show consistent improvements across different architectures and datasets
3. The review comment confirms extensive empirical validation
4. The term "simple" is somewhat misleading given the sophisticated nature of the baseline methods
5. The rest of the claim about performance boost is accurately supported by quantitative evidence


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, our last contribution is to show that fine-tuning simple baseline ND methods with the generated samples to distinguish them from the normal data leads to a performance boost for both ND and near-ND.

The review comment to be evaluated is:

* There is a large room for improvement in clarity.
    * The paper should be clearer on how exactly the generative model is trained and used. The most important fact, in my opinion, is that the score-based generative model is trained on in-distribution data and terminated prematurely. However, the fact is only briefly mentioned in Page 2, and then mentioned in Figure 2. Given the importance of this information, this point should be described in detail in Section 2.1. Also, other details and design choices, for example, how early the training of the score-based model should be stopped, needs to be elaborated.
    * The experimental setup described in Section 3.2 is difficult to follow. There are several comparison settings but are not well organized.
* Discussion on the differences among diffusion models, GANs, and VAEs in Section 2.2 are not very persuasive. It does not have concrete examples or quantitative arguments. Plus, the argument of Section 3.2 is mostly about generative modeling itself and not about its application to generating auxiliary data in novelty detection.
* When performing CIFAR-10 vs CIFAR-100 experiment, it is more standard and more challenging to incorporate all the classes, instead of using only one class from each dataset. Overall, I am not convinced that the experiment settings used in the paper are standard.
* Replacing a GAN with a diffusion model in the data generator for novelty detection itself is not a very novel contribution to the community.

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs. In the second step, the pre-trained backbone is fine-tuned by the generated dataset and given normal training samples through optimizing a binary classification loss.

In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process). Running the backward process allows us to seamlessly transform a random noise into the data that could be used for sample generation as follows:

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs. After the training, all the normal embeddings are stored in the memory \(\mathcal{M}\) that is used at the test time for assigning abnormality score to a given input. Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns.

Our empirical results also confirm that various quality metrics, such as coverage and density(28), which were previously proposed to evaluate synthetic images' fidelity and diversity, where larger values that are close to 1.0 indicate better fidelity and diversity, respectively.

Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process. To further validate our method, we adapt OpenGAN, as SOTA OOD detection, to the ND setting and report its near-ND results. Surprisingly, even by employing fake abnormal samples generated by a GAN and being exposed to real outliers, it is not able to achieve a decent performance in the near-ND setting (Table 1 (c)), which again supports the benefits of our proposed approach for generating near distribution abnormal samples.

For each backbone, the performance with and without the feature fine-tuning phase based on the generated data is reported (denoted as "Pre-trained" and "Fine-tuned" in the table), showing a consistent improvement across different architectures. The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection.

We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

* Murase and Fukumizu [2022] Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pseudo anomalous data via latent variables. _arXiv preprint arXiv:2202.10281_, 2022.

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence, I find this claim partially overstated. While the results do show performance improvements after fine-tuning with generated samples (e.g., 16-36% AUROC improvements for some backbones), several key limitations are not acknowledged:

1. The reviewer points out lack of clarity around the critical training process and design choices
2. The experimental setup is non-standard (e.g., using only one class from CIFAR datasets)
3. Performance gains vary significantly across architectures
4. The novelty of using diffusion models instead of GANs is questioned by reviewers

The claim would be better stated with appropriate caveats about experimental conditions and varying effectiveness across scenarios.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, our last contribution is to show that fine-tuning simple baseline ND methods with the generated samples to distinguish them from the normal data leads to a performance boost for both ND and near-ND.

The review comment to be evaluated is:

The idea of this paper is very natural: try to generate near-distribution outliers to augment the original training dataset which only contains normal data (semi-supervised learning), such that we can fine-tune the pre-trained feature extractor by training a binary classifier.

Strengths:
(1) Using generated outliers to improve the accuracy of novelty detection is not new. There have been some similar works which use GANs instead of SDE.  The main contribution of this paper is that they replace the GANs with the diffusion model (which is increasingly popular recently).  The motivation here is that the outliers generated by SDE are “closer“ to the nominal distribution, and then using these "boarder" points  to deal with the near-ND setting.  Even though the pipeline and SDE are all from existing works, the novel combination is meaningful considering the theoretically and experimental results.

(2) The experimental results show that the proposed method considerably improves over existing models (especially for the near-ND setting).  It is not surprising that SDE over-perform GANs for the near-ND setting, but I am surprised that the improvement is so obvious.  It will be great if the code can be provided.

Weakness:
(1) In Section 2.2 (Diffusion models vs. GANs), they try to explain, for near-ND, why choosing diffusion models instead of other generative modes (GAN and VAE), it would be better to include flow-based models in this part. Also,  VAE-related results are not included in figure3.

(2) In Figure3,  density, coverage and FID are used as the metrics to measure the fidelity and diversity.  Could you add a brief introductions/definitions for these metrics? Also, I am confused how to get the "density" from GAN-based models as they are not tractable models.

(3)About Stopping point: In section 4, "the method is robust against the stopping point with at most 2% variation for the reasonable FID scores". However,  from table4,  it is difficult to get this conclusion. what is "reasonable" FID? Actually, the stopping point decide how similar the fake points and the real points. How to decide when to stop ?

Other minor comments:
The "normal distribution" used in this paper sometimes are confusing, which means the distribution of nominal data instead of Gaussian distribution. It would be better to use other notations.

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs. In the second step, the pre-trained backbone is fine-tuned by the generated dataset and given normal training samples through optimizing a binary classification loss.

In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process). Running the backward process allows us to seamlessly transform a random noise into the data that could be used for sample generation as follows:

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs. After the training, all the normal embeddings are stored in the memory \(\mathcal{M}\) that is used at the test time for assigning abnormality score to a given input. Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns.

Our empirical results also confirm that various quality metrics, such as coverage and density(28), which were previously proposed to evaluate synthetic images' fidelity and diversity, where larger values that are close to 1.0 indicate better fidelity and diversity, respectively.

Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process. To further validate our method, we adapt OpenGAN, as SOTA OOD detection, to the ND setting and report its near-ND results. Surprisingly, even by employing fake abnormal samples generated by a GAN and being exposed to real outliers, it is not able to achieve a decent performance in the near-ND setting (Table 1 (c)), which again supports the benefits of our proposed approach for generating near distribution abnormal samples.

For each backbone, the performance with and without the feature fine-tuning phase based on the generated data is reported (denoted as "Pre-trained" and "Fine-tuned" in the table), showing a consistent improvement across different architectures. The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection.

We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

* Murase and Fukumizu [2022] Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pseudo anomalous data via latent variables. _arXiv preprint arXiv:2202.10281_, 2022.

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim is moderately overstated. While the evidence shows clear performance improvements after fine-tuning across multiple architectures and datasets (e.g., 16-36% AUROC boosts for some backbones), the claim that fine-tuning "leads to a performance boost for both ND and near-ND" is too definitive and lacks important nuance. The evidence indicates improvements are architecture-dependent and vary significantly in magnitude. The review comment supports that while the method works well, there are questions about stopping point selection and robustness. The claim would be better stated with qualifiers about the varying effectiveness across different conditions.

<score>3</score>

Justification:

1. The evidence shows genuine improvements but they are not universal:
- Tables show varying levels of improvement across architectures and datasets
- Some architectures show dramatic gains (e.g., "36% boosts in AUROCs") while others show more modest improvements
- Reviewer notes uncertainty about stopping point selection which affects performance

2. Claim language issues:
- Uses definitive phrasing "leads to a performance boost" without acknowledging variability
- Doesn't mention dependency on architecture choice and stopping point selection
- Doesn't acknowledge limitations or conditions for effectiveness

3. Supporting evidence exists but with caveats:
- Multiple experimental results demonstrate improvements
- Reviewer confirms meaningful contribution but raises questions about robustness
- Early stopping point selection remains somewhat unclear per reviewer comment
- Limited discussion of potential failure cases or limitations

4. More measured language would better reflect the evidence:
- Could specify "can lead to" rather than definitively "leads to"
- Should acknowledge variation across architectures
- Could mention dependence on proper stopping point selection

The claim would be more accurate if it acknowledged these nuances while still highlighting the method's demonstrated benefits.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, our last contribution is to show that fine-tuning simple baseline ND methods with the generated samples to distinguish them from the normal data leads to a performance boost for both ND and near-ND.

The review comment to be evaluated is:

### Strengths

- The paper is, for the most part, well-written, easy-to-follow, and does a good job of
contextualising and motivating the work.
- The proposed method is simple and intuitive, and seems to perform well empirically
compared with the baselines in both the regular ND setting and the near ND setting, showing a
significant reduction in the performance gap of going from the former to the latter.
the NR to near-NR setting.
- The figures are well-chosen and well-put-together: they convey well the essential points of the
  paper, such as the shortcomings of existing methods and the feasibility of using GANs vs diffusion
  models for creating the synthetic data.
- Evaluation performed using a good range and selection of baseline methods and benchmark datasets.
- Results are aggregated over a good number (10) of replicates.

### Weaknesses
- Despite the authors claiming otherwise, the method is quite sensitive to the stopping criterion in
  the near ND setting with an empirical range of [68.2, 90.0]; in practice, different datasets may
  require wildly different FID thresholds and there is no reliable way of validating this given the
  nature of the problem. Also, I wonder if the number of sampling steps and the choice of sampler have a
  significant effect on the results -- there doesn't seem to be any mention of this.
  Taking this a step further, might it be feasible to train the diffusion model fully and then rely
  upon early-stopping the reverse diffusion process according to the FID score?
- The need to train, and indeed sample, from a diffusion model can be computationally burdensome for
  higher-resolution datasets.
- The description of the SDE model is a little overdone; a much briefer description with a pointing of the
  reader to the original paper would be sufficient. This is especially the case given that, as
  I understand it, the proposed method should work with any diffusion based model (Imagen, Parti,
  DALLE-2, etc.) or are there theoretically some constraints on the form said model and the
  associated sampler should take?
- While this applies to several methods of the same class, the method seems to make certain
assumptions about the distribution of the ID and OOD data that might not hold in the wild.
- While results are aggregated over a good number (10) of replicates no measure of dispersion
  (standard deviation/error) seems to be reported alongside the mean in each case.
- Some typographic and syntactical errors (e.g. 'that unlikely come' (line 1 of the introduction)).

The evidence to be evaluated is:In the first step, a fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs. In the second step, the pre-trained backbone is fine-tuned by the generated dataset and given normal training samples through optimizing a binary classification loss.

In order to generate near anomalous samples, a basic diffusion model, called Stochastic Differential Equation (SDE) (43) is adopted.

The model is trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process). Running the backward process allows us to seamlessly transform a random noise into the data that could be used for sample generation as follows:

Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according to the abnormal inputs. After the training, all the normal embeddings are stored in the memory \(\mathcal{M}\) that is used at the test time for assigning abnormality score to a given input. Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become well-structured, thus better representing the normal semantic features.

Hence, we expect the estimated generative distribution \(p_{\theta}\) ends up _sufficiently_ close to \(q\) if the training is prematurely stopped, and therefore generates "near-distribution" patterns.

Our empirical results also confirm that various quality metrics, such as coverage and density(28), which were previously proposed to evaluate synthetic images' fidelity and diversity, where larger values that are close to 1.0 indicate better fidelity and diversity, respectively.

Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during the training process. To further validate our method, we adapt OpenGAN, as SOTA OOD detection, to the ND setting and report its near-ND results. Surprisingly, even by employing fake abnormal samples generated by a GAN and being exposed to real outliers, it is not able to achieve a decent performance in the near-ND setting (Table 1 (c)), which again supports the benefits of our proposed approach for generating near distribution abnormal samples.

For each backbone, the performance with and without the feature fine-tuning phase based on the generated data is reported (denoted as "Pre-trained" and "Fine-tuned" in the table), showing a consistent improvement across different architectures. The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models.

This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need to be carefully selected based on the constraints defined in such domains.

Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, to augment the training set (6; 26; 50; 29; 11; 1). In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection.

We proposed to take a non-adaptive generative modeling that allows controllable generation of anomalous data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

* Murase and Fukumizu [2022] Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pseudo anomalous data via latent variables. _arXiv preprint arXiv:2202.10281_, 2022.

In this section, according to our main setting, we have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such a setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models can eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Figure 2: The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results are reported according to both criteria, i.e. CLP and bottom-1.

Input: Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training is early-stopped. As it is shown, they have very subtle yet semantic differences with the normal data distribution (downleft). After that, using a linear layer, a pre-trained feature extractor is fine-tuned to solve a binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used to compute the k-NN distance at the test time as the novelty score.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence:

The claim about fine-tuning leading to performance improvements is moderately overstated based on several factors:

1. While the evidence shows consistent improvements across architectures after fine-tuning, the claim doesn't acknowledge the limitations noted in the review about sensitivity to stopping criteria and computational burden.

2. The empirical results show varying degrees of improvement (from modest to significant) rather than universal "performance boosts."

3. The reviewer points out missing measures of dispersion/variance in the results, which would help validate the claimed improvements.

4. However, the extensive empirical results in Table 1 and multiple experiments across different architectures do demonstrate clear benefits from the fine-tuning approach.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our method achieves state-of-the-art results in the ND setting, and is especially effective for the near-ND setting, where we improve over existing work by a large margin of up to 8% in AUROC.

The evidence to be evaluated is:In this section, we compare our method against the SOTA in the ND and near-ND setups.

We compare our approach with current top self-supervised and pre-trained feature adaptation methods in the literature (44; 34; 33; 7; 16).

Table 1(a) provides the results of the case, which indicates our superiority. To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used in fine-grained image classification or segmentation, containing hard-to-distinguish object classes. Thus they are suitable for benchmarking near-ND.

Table 1(b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC in the mentioned near-ND setup.

In this setup, each class of CIFAR-10 (\(C_{i}^{10}\)) and its closest class in CIFAR-100 (\(C_{j}^{100}\)) are selected, to represent normal and abnormal classes, respectively, where \(i\) and \(j\) are class numbers that are less than 10 and 100, respectively. Then, the model is trained on the training samples of (\(C_{i}^{10}\)) and tested against the aggregation of (\(C_{j}^{100}\)) and (\(C_{i}^{10}\)) test sets. As it is shown, almost all the methods considerably lag behind our method by a large margin.

Table 1 (c) compares our method with other SOTA methods in detecting such samples. Due to learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting a wide range of anomalies. While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table 1 (a)), they work significantly worse in the near-ND setting.

The results are provided for both the ND and near-ND evaluation setups.

The performance boost is surprisingly large for some backbones such as ResNet-152, ViT-B_16, and R50+ViT-B_16 with roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B_16 and R50+ViT-B_16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the _generality_ of our approach regardless of the setting or backbone.

Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20; 42; 5; 14) with 2% to 20% better AUROCs based on the chosen dataset.

In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data.

The generated data levels up pre-trained distance-based novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it a better choice for the near-ND setup. \begin{table}\end{table}
Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 1: The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ a pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.

Input: Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10, Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusionbased data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage&Density metrics are calculated according to the dataset and generative model. These metrics measure the diversity and fidelity of the samples that are generated by generative models.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 11: Generated anomaly images on the Caltech-UCSD Birds 200 dataset. The images highlighted in green are normal samples.

Input: Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.

Input: Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR10-vs100 setting, as proxies for low- and high-resolution data, respectively.

Input: Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. A considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish a cat image from a car, they are still unable to detect tiger or lion images as anomalies.

Input: Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green are normal samples.

Input: Figure 3: Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metrics “density” and “coverage” (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: Some generated samples by various methods in different datasets as well as their corresponding normal samples are shown. As can be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.

Input: Figure 3: By increasing i, the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the case of i = 1, in which anomalies constitute the near distribution ones, the models’ performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin. 16

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Summary: The claim states the method achieves state-of-the-art results in ND and improves over existing work by up to 8% AUROC in near-ND. The evidence strongly supports this through comprehensive experimental results shown in Tables 1(a), 1(b), and 1(c). The method consistently outperforms SOTA across multiple datasets, with improvements of ~8% in near-ND settings like FGVC-Aircraft (88.7% vs 84.0%) and WBC (91.2% vs 85.1%). The gains are well-documented across different architectures and evaluation settings. The claim accurately reflects the empirical evidence without exaggeration.

<score>1</score>

Justification:
1. Comprehensive benchmarking against multiple SOTA methods
2. Clear evidence of ~8% improvements in near-ND cases through Tables 1(b) and 1(c)
3. Claims are precise and quantifiable, matching actual experimental results
4. Performance gains are consistent across different datasets and architectures
5. No signs of cherry-picking or overstatement - claim matches the demonstrated improvements